Joyent

Monitoring Memory and Disk usage

This was inspired from this forum post

Pre-Flight

  • What email address(es) you wish to send the notices?
  • What your resource limits are for your accelerator?
  • Disk space monitoring for / is built in as well as /home, do you need any other mount points monitored?
  • Run this cron job as root, once an hour so be fine, your choice.

How does it work

  • It uses these commands
    • To check memory:
      prstat -Z -n <num proces> -s <sort by> 1 1
    • To check diskspace
      df -h

How to make it your own

  • If you don’t have any custom mount points then you really don’t need to modify the script to use it.
  • At the bottom of the file there is a commented out cat line which you can use to test it out with out sending the email
  • The last parameter is to check the disk space of /home, if you have additional mount points (which are not named home) then just replace the home in the grep ‘/home’ in this line
    HOMEUSAGE=`df -h |grep '/home' | tail -1 | awk -F" " ' {print $5}' | awk -F"%" ' {print $1}'`

Usage

     Usage: ./resource.sh email rss swap disk_usage home_disk_usage
     email            - Email address(es) to send report comma delimited
     rss              - number of Megs to report when above this number 
     swap             - number of Megs to report when above this number 
     disk_usage       - percentage free to report when above this number for / 
     home_disk_usage  - percentage free to report when above this number for /home (optional)

Examples

These are just suggestions. Only you can know what is “normal” for your system. Test out the commands listed above to see what is normal for you. Just keep adjusting the numbers until you are comfortable that you are not going to get locked out of your accelerator. The numbers below are set to 95% of the memory limits.

Small/PATH/TO/resource.sh linda@example.com 240 480 90
Medium/PATH/TO/resource.sh linda@example.com 480 970 90
Large/PATH/TO/resource.sh linda@example.com 970 1940 90 95
X-Large/PATH/TO/resource.sh linda@example.com 1940 3890 90 95
XX-Large/PATH/TO/resource.sh linda@example.com 3890 7780 90 95

resource.sh

#!/bin/sh

 # Whom to email for reports
 PAGE=$1
 # Max RSS
 MAXRSS=$2
 # Max Swap
 MAXSWAP=$3
 # Maximum disk usage of /
 MAXDISK=$4
 # Maximum disk usage of /home
 MAXHOME=$5
 
 # Location of temp files
 TMPDIR="/tmp"
 # Base filename for temp files
 TMPFILE="localinfo-`zonename`"
 
 #Automagical variable (do not need to modify)
 DATE=`date`
 ZONE=`zonename`
 
 # Before checking...
 rm -f $TMPDIR/$TMPFILE
 
if [ -z "$MAXHOME" ]; then
 MAXHOME=100
fi
 
if [ -z "$MAXDISK" ]; then
  echo
  echo "Usage: $0 email rss swap disk_usage home_disk_usage"
  echo
  echo "       email            - Email address(es) to send report comma delimited"
  echo "       rss              - number of Megs to report when above this number "
  echo "       swap             - number of Megs to report when above this number "
  echo "       disk_usage       - percentage free to report when above this number for / "
  echo "       home_disk_usage  - percentage free to report when above this number for /home (optional)"
  exit
fi
 
 # Zone RSS Check
 #  Checks to ensure the $RSS is not over $MAXRSS
 RSS=`prstat -Z -s rss 1 1 | grep $ZONE | sed  's/  */ /g' |cut -f5 -d" "| sed 's/[^0-9]//'`
 NPROCS=`prstat -Z -s rss 1 1 | grep $ZONE | sed  's/  */ /g' |cut -f3 -d" "| sed 's/[^0-9]//'`
 if [ $RSS -gt $MAXRSS ] ; then
  echo $DATE >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  echo ":: High RSS detected for zone:: " `hostname` >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  prstat -Z -n $NPROCS -s rss 1 1 >> $TMPDIR/$TMPFILE
 fi
 
 # Zone Swap Check
 #  Checks to ensure the $SWAP is not over $MAXSWAP
 SWAP=`prstat -Z -s rss 1 1 | grep $ZONE | sed  's/  */ /g' | cut -f4 -d" " | sed 's/[^0-9]//'`
 if [ $SWAP -gt $MAXSWAP ] ; then
  echo $DATE >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  echo ":: High SWAP detected for zone:: " `hostname` >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  prstat -Z -n $NPROCS -s size 1 1 >> $TMPDIR/$TMPFILE
 fi
 # Disk Check
 # Checks to ensure the $DISKUSAGE is not over $MAXDISK
 DISKUSAGE=`df -h |head -2 | tail -1 | awk -F" " ' {print $5}' | awk -F"%" ' {print $1}'`
 if [ $DISKUSAGE -gt $MAXDISK ] ; then
  echo "" >> $TMPDIR/$TMPFILE
  echo $DATE >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  echo ":: High disk usage detected on / :: " `hostname` >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  df -h >> $TMPDIR/$TMPFILE
 fi
 
 HOMEUSAGE=`df -h /home | tail -1 | awk -F" " ' {print $5}' | awk -F"%" ' {print $1}'`
 if [ -n "$HOMEUSAGE" -a $HOMEUSAGE -gt $MAXHOME ] ; then
  echo "" >> $TMPDIR/$TMPFILE
  echo $DATE >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  echo ":: High disk usage detected on /home :: " `hostname` >> $TMPDIR/$TMPFILE
  echo "" >> $TMPDIR/$TMPFILE
  df -h >> $TMPDIR/$TMPFILE
 fi
 
 if [ -r "$TMPDIR/$TMPFILE" ]; then
   # Mail the results of $TMPDIR/$TMPFILE to $PAGE if file exists
   /usr/bin/mailx -s "`hostname` local check report" $PAGE < $TMPDIR/$TMPFILE
   # for testing just comment out the line above and use this to see the output on terminal or for normal cron output email
   #cat $TMPDIR/$TMPFILE
 fi

Figuring what's holding a port open?

Sometimes you need to figure out which mongrel is holding a port open and you are running more than one mongrel. Grab a copy of pcp here http://www.unix.ms/pcp/pcp and see the usage notes on http://www.unix.ms/pcp/

 
accelerators/kb/monitor_rss_disk.txt · Last modified: 2008/08/10 14:46 by jacques
 
Recent changes RSS feed Creative Commons License Driven by DokuWiki