While working on several rails projects, I’ve found it useful to write a couple of scripts to monitor real-time server health. Basically, I’d like to automatically be notified by email if anything peculiar is occurring on the server. Two pretty obvious things to alert on are low disk space and high average load. While there are great tools out there like Munin/Nagios that will give you detailed instrumentation for your server, I just needed something lightweight that I could periodically cron. These scripts are designed for use on Ubuntu 9.04. YMMV.

disk_usage.rb

This script will email foo@example.com when the disk space left on either sda1 and sda2 falls below 5GB. The threshold is just a variable you can change. You can also add to or remove from the list of sda devices you want to monitor using the disks array.

[cci lang="ruby"]
#!/usr/bin/env ruby

require ‘rubygems’
require ‘send_gmail’

TO_EMAIL = ‘foo@example.com’

filesys = `df -h`

disks = [1,2]

disks.each do |i|
r = filesys[/sda#{i}\s+[\d\.]+[MG]\s+[\d\.]+[MG]\s+([\d\.]+[MG])/,1]
message = “Low disk space on /sda#{i}: only #{r.to_i}GB remaining”
hsh={:to=>TO_EMAIL, :subject=>’Low disk space warning!’, :body=>message}
SendGMail.send_gmail(hsh) if r.to_i < 5 or r[/M$/]
puts message if r.to_i < 5 or r[/M$/]
end
[/cci]

load_check.rb

This script will use the uptime command to get the avarage load. I’m just using the 1min and 5min averages. If the 1min average load is over 9 or the 5min average load is over 4, bar@example.com will get an email.

[cci lang="ruby"]
#!/usr/bin/env ruby

require ‘rubygems’
require ‘send_gmail’

TO_EMAIL = ‘bar@example.com’

uptime = `uptime`

load_1min = uptime.split(” “)[8].chop()
load_5min = uptime.split(” “)[9].chop()

message = “Load on server is 1min/5min avg: #{load_1min} / #{load_5min} ”

hsh={:to=>TO_EMAIL, :subject=>’Load warning!’, :body=>message}
SendGMail.send_gmail(hsh) if (load_1min.to_i > 9) or (load_5min.to_i > 4)
[/cci]

You may have noticed an included module “send_gmail” and a call to “SendGMail” in the previous scripts. Since I use Gmail for all my outbound emails, the underlying alert system needs a way to make calls to the Gmail servers. Problem solved using this nifty little Ruby Gmailer script I found on at http://codingfrenzy.alexpmay.com/2007/12/sending-gmail-from-standalone-ruby.html.  I did make a few modifications to the script, so I’m including it here. You’ll want to set your gmail account/domain info if you do use this mailer.

send_gmail.rb

[cci lang="ruby"]
#!/usr/bin/env ruby

require ‘rubygems’

gem ‘actionmailer’
require ‘action_mailer’
require ‘openssl’
require ‘net/smtp’

module SendGMail

@user_name=’someone@example.com’
@domain=’example.com’
@password=’password’

def SendGMail.send_gmail(hsh)

raw_attachments=hsh.fetch(:raw_attachements, [])
if hsh.has_key?(:raw_attachment)
raw_attachments.push(hsh[:raw_attachment])
end

mail=TMail::Mail.new
mail.to=hsh[:to]
mail.date=Time.now
mail.from=@user_name
mail.subject=hsh[:subject]

main=mail
main=TMail::Mail.new
main.body = hsh[:body]
main.set_content_type(‘text/plain’, nil, ‘charset’=>’utf-8′)
mail.parts.push(main)

for raw_attachment in raw_attachments
part = TMail::Mail.new
transfer_encoding=raw_attachment[:transfer_encoding]
body=raw_attachment[:body]
case (transfer_encoding || “”).downcase
when “base64″ then
part.body = TMail::Base64.folding_encode(body)
when “quoted-printable”
part.body = [body].pack(“M*”)
else
part.body = body
end

part.transfer_encoding = transfer_encoding
part.set_content_type(raw_attachment[:mime_type], nil, ‘name’ => raw_attachment[:filename])
part.set_content_disposition(“attachment”, “filename”=>raw_attachment[:filename])
mail.parts.push(part)
end

mail.set_content_type(‘multipart’, ‘mixed’)
ActionMailer::Base.deliver(mail)

end

ActionMailer::Base.smtp_settings = {
:address => ‘smtp.gmail.com’,
:domain => @domain,
:authentication => :plain,
:port => 587,
:user_name => @user_name,
:password => @password
}

Net::SMTP.class_eval do
private
def do_start(helodomain, user, secret, authtype)
raise IOError, ‘SMTP session already started’ if @started
check_auth_args user, secret if user or secret

sock = timeout(@open_timeout) { TCPSocket.open(@address, @port) }
@socket = Net::InternetMessageIO.new(sock)
@socket.read_timeout = 60 #@read_timeout
@socket.debug_output = STDERR #@debug_output

check_response(critical { recv_response() })
do_helo(helodomain)

raise ‘openssl library not installed’ unless defined?(OpenSSL)
starttls
ssl = OpenSSL::SSL::SSLSocket.new(sock)
ssl.sync_close = true
ssl.connect
@socket = Net::InternetMessageIO.new(ssl)
@socket.read_timeout = 60 #@read_timeout
@socket.debug_output = STDERR #@debug_output
do_helo(helodomain)

authenticate user, secret, authtype if user
@started = true
ensure
unless @started
# authentication failed, cancel connection.
@socket.close if not @started and @socket and not @socket.closed?
@socket = nil
end
end

def do_helo(helodomain)
begin
if @esmtp
ehlo helodomain
else
helo helodomain
end
rescue Net::ProtocolError
if @esmtp
@esmtp = false
@error_occured = false
retry
end
raise
end
end

def starttls
getok(‘STARTTLS’)
end

def quit
begin
getok(‘QUIT’)
rescue EOFError, OpenSSL::SSL::SSLError
end
end
end
end

[/cci]