Kiss Gabor (Bitman)
11 years ago
Dear folks,
I've developed a massively parallel version of collector.
It uses Net::SNMP Perl module to spread several (actually 512)
concurrent SNMP requests to devices to poll.
We use it in production since a week without any problem.
I've experienced that pcollector needs time 40-70% less to run
than collector does.
(We have some 5800 SNMP polled device with variable number
of interfaces and datasource complexity. Target number is about 22k.)
The code is available as a Debian package from my private repository
at http://debian6.ki.iif.hu/. (It is IPv6 only. Sorry for that.)
Actual version is 1.0.5-19niif.5.
Directory of binary and source packages is
http://debian6.ki.iif.hu/pool/main/c/cricket/
*******************************************************
*** Attention ***
This is "No Country for Old Men". :-)
Our version of Cricket contains tons of enhancements and
localizations developed in the past 10 years at NIIF Institute
and Computer and Automation Institute (SZTAKI). It is obviously
not a drop-in replacement of the program actually used by you.
DO NOT install it unless you know what do you do.
I warned you.
*******************************************************
However you may want to study end develop pcollector program
and its supporting library called Collector::Parallel.
In the source package it is the file debian/patch/niif/58_pcollector.patch
you may interested in.
Minor changes and enhancements are to be expected in the near future.
FYI:
Our crontab looks like this:
COLLECT=/usr/share/cricket/collect-subtrees
PCOLLECT="/usr/share/cricket/collect-subtrees --parallel"
0-59/5 * * * * cricket $PCOLLECT slot0 >/dev/null 2>&1
1-59/5 * * * * cricket $PCOLLECT slot1 >/dev/null 2>&1
2-59/5 * * * * cricket $PCOLLECT slot2 >/dev/null 2>&1
[...]
The slightly modified collect-subtrees accepts a --parallel or -p
options and starts pcollector instead of legacy collector.
pcollector first tries to gather and to cache all of the data
reachable by SNMP then runs as legacy collector does:
sequentially. When it needs an SNMP variable it gets from the
cache if possible. In case of any problem (e.g instance remapping)
it falls back to old mode.
I hope you will find it useful. :-)
Cheers
Gabor
I've developed a massively parallel version of collector.
It uses Net::SNMP Perl module to spread several (actually 512)
concurrent SNMP requests to devices to poll.
We use it in production since a week without any problem.
I've experienced that pcollector needs time 40-70% less to run
than collector does.
(We have some 5800 SNMP polled device with variable number
of interfaces and datasource complexity. Target number is about 22k.)
The code is available as a Debian package from my private repository
at http://debian6.ki.iif.hu/. (It is IPv6 only. Sorry for that.)
Actual version is 1.0.5-19niif.5.
Directory of binary and source packages is
http://debian6.ki.iif.hu/pool/main/c/cricket/
*******************************************************
*** Attention ***
This is "No Country for Old Men". :-)
Our version of Cricket contains tons of enhancements and
localizations developed in the past 10 years at NIIF Institute
and Computer and Automation Institute (SZTAKI). It is obviously
not a drop-in replacement of the program actually used by you.
DO NOT install it unless you know what do you do.
I warned you.
*******************************************************
However you may want to study end develop pcollector program
and its supporting library called Collector::Parallel.
In the source package it is the file debian/patch/niif/58_pcollector.patch
you may interested in.
Minor changes and enhancements are to be expected in the near future.
FYI:
Our crontab looks like this:
COLLECT=/usr/share/cricket/collect-subtrees
PCOLLECT="/usr/share/cricket/collect-subtrees --parallel"
0-59/5 * * * * cricket $PCOLLECT slot0 >/dev/null 2>&1
1-59/5 * * * * cricket $PCOLLECT slot1 >/dev/null 2>&1
2-59/5 * * * * cricket $PCOLLECT slot2 >/dev/null 2>&1
[...]
The slightly modified collect-subtrees accepts a --parallel or -p
options and starts pcollector instead of legacy collector.
pcollector first tries to gather and to cache all of the data
reachable by SNMP then runs as legacy collector does:
sequentially. When it needs an SNMP variable it gets from the
cache if possible. In case of any problem (e.g instance remapping)
it falls back to old mode.
I hope you will find it useful. :-)
Cheers
Gabor