Watching/tailing multiple log files at the same time with Ruby’s NET::SSH library

A good testing habit when working with web apps is to monitor the log files of servers as you test.  In some cases this is easy, especially where there’s a single application server.  With the trend toward more service-oriented architectures, and server clusters for high-traffic applications, the environments I’m working in tend to have many log files spread over multiple servers and folders.  My old friend, ‘tail -f error.log’ becomes a bit more difficult.

I’d heard about Ruby’s SSH library quite a while back, but only recently have my spare time and my memory conspired to work together. The library lets you connect to a server via SSH, execute commands and see the result. So today I quickly hacked out a script to let me watch multiple (Linux) log files at the same time.  This might work for other environments that support SSH, but I only have it connecting to Linux servers at this time.

Code is below, very hacky, and tailored to my needs.  If you have any questions, leave a comment, but hopefully this helps someone get started!

To install, use the instructions for NET::SSH version one at http://net-ssh.rubyforge.org/ssh/v1/index.html. This will actually install version two though, so you’ll need the docs and examples from http://net-ssh.rubyforge.org/sftp/v2/api/index.html

require 'net/ssh'

class Logfile
  def initialize(name,server,filename)
    @name=name
    @server=server
    @filename=filename
    @current_file_size=0
    @ssh_server='server'
    @username='username'
    @password={:password=>'password'}
  end

  #You'll need to customise this method so that it generates the
  #full path of the log file you're interested in.  This one is for Splunk logs.

  def full_log_path
    today=Time.now
    day=today.day.to_s.rjust(2,'0')
    month=today.month.to_s.rjust(2,'0')
    year=today.year.to_s
    todays_log_folder="#{year}-#{month}-#{day}"
    full_path="/var/log/#{@server}/#{todays_log_folder}/#{@filename}"
    return full_path
  end

  def get_new_lines
    Net::SSH.start(@log_server,@username,@password) do |session|
      new_file_size=session.exec!("wc -l #{full_log_path}").split(" ")[0].to_i
      lines_to_get=new_file_size-@current_file_size

      # Don't generally need to get everything if the error logs are being flooded
      lines_to_get=100 if lines_to_get > 1000 

      new_logs=session.exec!("tail -n #{lines_to_get.to_s} #{full_log_path}")
      @current_file_size=new_file_size
      return new_logs
    end
  end

end

logfiles=[]
logfiles.push Logfile.new(:log_name1,"log_app_folder1","log_file_name1")
logfiles.push Logfile.new(:log_name2,"log_app_folder2","log_file_name2")

5.times do
  logfiles.each do | logfile |
    changes=logfile.get_new_lines
    puts changes if !changes.nil?
    $stdout.flush
  end
  sleep 10
end