Systems monitoring with visual representations of trending and resource consumption is an essential component of the bigger picture of infrastructure management and design. The open source (GPL) project Munin is an excellent approach to getting graphs of systems behavior quickly. A default installation producing useful information takes less than 30 minutes and requires only a minimal understanding of the software or the various resources being monitored.
Here we discuss a few issues involved in using Munin to monitor EC2 instances, and in particular micro instances. There’s a two step approach used here: Initially monitoring and graphing on one or several nodes then upon satisfactory results the graphing component will be moved to a central system. It’s important to note that the resources available on an AWS EC2 micro instance are insufficient for data collection and graph generation.
Single Node Configuration
1. Install httpd, munin and munin-node. All of these items are available in the AWS repositories. Apache httpd is needed only for the first part of this process and may be shut off or removed entirely later on.
# yum install munin
# yum install munin-node
2. Make the necessary edits to the Munin configuration.
Edit /etc/munin/munin.conf
[mylilnode]
address 127.0.0.1
use_node_name yes
Edit /etc/munin/munin-node.conf
host_name mylilnode
3. Start the services.
# service sysstat start
# service munin-node start
# chkconfig sysstat on
# chkconfig munin-node on
4. Check to see the munin-node service started correctly.
Edit /var/log/munin/munin-node.log
Process Backgrounded
2011/07/16-01:30:24 Munin::Node::Server (type Net::Server::Fork) starting! pid(1349)
Binding to TCP port 4949 on host *
Setting gid to “0 0”
5. Apache needs an alias in order to find the munin graphs.
Create /etc/httpd/conf.d/munin.conf
Alias /munin /var/www/html/munin
Then restart apache to load the Alias just entered and see if /var/www/html/munin/index.html exists. If so, open the following in a browser: http://mymuninnode/munin
If after 10 minutes /var/www/html/munin/index.html still hasn’t been created, check to see if the crontab exists, /etc/cron.d/munin. This file is a necessary part of munin. It runs every 5 minutes to create the graphs.
Assuming at least the graph boxes have been created, take a close look at the left running column of graphs. The left running column are the “by day” items. In the far right in some of those you might see very small lines beginning to trend. Give this 20-30 minutes to really see the trending develop. If everything looks fine, then proceed to consolidating the graphs on a monitoring server in the network configuration.
***** iostat plugin
Network Configuration
Determine which system will be used as a monitoring server. It should have a substantial amount of available CPU (quad core, 2Ghz). In this example we’ll use a server named muninmonitor.example.com (83.32.121.202) monitoring the node mylilnode.example.com (52.21.78.32).
1. On the node to be monitored
allow ^83\.32\.121\.202$
At this time the notation for the ip address must be a regular expression
2. Configure the network equipment for the monitored node. Open an inbound port (default 4949) to allow for the data collection coming from the monitoring server. With an AWS EC2 instance use the AWS Management Console: EC2 -> Networking and Security (in lefmost Navigation column) -> Security Groups.
3. From the monitoring server, use telnet or netcat to check the connection:
Trying 52.21.78.32…
Connected to mylilnode.example.com.
Escape character is ‘^]’.
# munin node at mylilnode.example.com
foo
# Unknown command. Try cap, list, nodes, config, fetch, version or quit
4. On the monitoring server, tell munin to monitor the node.
[mylilnode]
address 52.21.78.32
– how to force graph generation here?
5. Check that the graphs have been generated on the monitoring server http://muninmonitor.example.com/munin
Tutorial and documentation…
http://waste.mandragor.org/munin_tutorial/munin.html
Various plugins for AWS services…
s3 plugin: http://www.ohardt.com/dev/munin/
https://github.com/DuoConsulting/Munin-AWS-RDS-plugin/comments
https://code.google.com/p/s3speed/
https://anvilon.s3.amazonaws.com/web/20071224-munin/cpu
Thanks for the useful writeup.
btw if anyone else is getting error messages like:
“Can’t open /var/lib/munin/plugin-state/yum.state for reading: No such file or directory”
You should make sure that that directory has it’s ownership set correctly by doing:
chown nobody:munin /var/lib/munin/plugin-state
On my EC2 instance it was set to munin:munin, and as the plugin runs as ‘nobody’ it was unable to save data to that directory.