This post is older than a year. Consider some information might not be accurate anymore.
If you have an application which does distributed computing, i.e. means connects to other servers and send data, it is interesting to monitor the connection handling. Therefore collectd provides the plugin tcpconns, that allows to monitor dedicated ports. This data can be send to logstash, where it can have graphite or elasticsearch as output. Having the data in a metrics storage, visualization with Kibana or Grafana is a piece of cake.
Collectd
Collectd is OpenSource and a daemon which collects system and application performance metrics periodically. The first step is to install collectd.
# deb based install
# apt-get install collectd collectd-utils
# rpm based install
# yum install collectd collectd-utils
Minimal System Configuration
Collectd is very efficient. If we have no any other areas of interest like CPU, Memory, etc. the minimal setup for collectd:
/etc/collectd.conf
LoadPlugin syslog
Include "/etc/collectd.d"
Add your Configuration
The folder /etc/collectd.d should contain our configuration for application and the output destination.
Monitoring Configuration
Example config for tcpconns
LoadPlugin tcpconns
<Plugin "tcpconns">
ListeningPorts false
# Outbound connections; i.e. the servers we're connecting to:
RemotePort "2251"
RemotePort "3513"
RemotePort "3504"
# Locally listening ports; i.e. the servers we're running:
LocalPort "22"
LocalPort "25"
LocalPort "8080"
LocalPort "8181"
</Plugin>
Output Configuration
collectd writes the collected data to the UDP port 25826 in the binary protocol. As receiving end I chose logstash with its collectd input plugin. Replace IP-address and port to your needs.
out-network.conf
LoadPlugin network
<Plugin network>
<Server "10.22.12.121" "25826">
</Server>
</Plugin>
Start collecting data
First we have to enable the service
# systemctl enable collectd.service
Next is to start the service
# systemctl start collectd.service
Check with status if the service is running properly. As you can see the monitoring takes only 508 kb in memory.
root@alpha:/etc/collectd.d# systemctl status collectd.service
● collectd.service - Collectd statistics daemon
Loaded: loaded (/usr/lib/systemd/system/collectd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2016-10-24 13:31:46 CEST; 22min ago
Docs: man:collectd(1)
man:collectd.conf(5)
Main PID: 26116 (collectd)
Memory: 508.0K
CGroup: /system.slice/collectd.service
└─26116 /usr/sbin/collectd
Oct 24 13:31:46 alpha systemd[1]: Starting Collectd statistics daemon...
Oct 24 13:31:46 alpha collectd[26116]: supervised by systemd, will signal readyness
Oct 24 13:31:46 alpha systemd[1]: Started Collectd statistics daemon.
Oct 24 13:31:46 alpha collectd[26116]: Initialization complete, entering read-loop.
Oct 24 13:31:46 alpha collectd[26116]: tcpconns plugin: Reading from netlink succeeded. Will use the netlink method from now on.
Logstash
On the receiving end of collectd is logstash. Logstash needs an input configuration for collectd.
100_input_collectd.conf
input {
udp {
port => 25826
buffer_size => 1452
codec => collectd { }
}
}
Logstash can read the collectd data, but must also decide where to store the data. As fitting endpoint I take Elasticsearch.
The collected data shall be written to the metrics index.
900_output_es.conf
output {
if [collectd_type] =~ /.+/ {
elasticsearch {
hosts => [ "es-1", "es-2", "es-aws" ]
index => "metrics-%{+xxxx.ww}"
}
}}
Notice that the if condition is not necessary, I only ensure to write all events that contains the field collectd_type to the metrics index.
Since the data is in Elasticsearch we can visualize the data in Kibana, see the example visualization for SSH. This can be embedded into existing dashboards.
Summary
collectd is an easy and lightweight method to monitor network connections. If you stay low profile, and do not monitor cpu, memory and many other stuff, you have a simple, brain dead simple and robust monitoring.
