monitoring core/vcpu usage on Sun CMT machines

Sun provides the /usr/local/bin/corestat CLI tool for their CMT (T1 and T2 Ultrasparc) machines which outputs in a loop with a line per core, however the output isn't suitable for long-term monitoring and trending.

Here's my approach to monitoring core (really vcpu or hardware thread) usage, divided into usr and sys. Add
exec vcpuidl /bin/sh /etc/snmp/conf/vcpuidl.sh to /etc/snmp/conf/snmpd.conf and the script /etc/snmp/conf/vcpuidl.sh is simply (mpstat is what corestat calls):


#!/bin/sh
/usr/bin/mpstat 1 2 | /usr/bin/awk 'BEGIN {RS="\n"; ORS="|"; } (NR>33) {print $13 "|" $14} END {print "\n"}'


which shows % usage per vcpu (or hardware thread) and it can be read via SNMP:


UCD-SNMP-MIB::extOutput.1 = STRING: usr|sys|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|1|0|0|0|0|0|0|0|0|0|0|0|0|0|1|0|0|0|0|0|0|1|1|1|0|0|0|0|5|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|


the above is from a Sun T2000 which has a Ultrasparc T1 with 8 cores and 4 hardware threads per core or 32 vcpus -- graphing then looks like this (alternate red and yellow bands for 32 x sys usage, green and yellow x 32 bands for usr usage):



note the peak over 3200 (8 cores x 4 threads per core x 100%) is due to the addition of max peaks of both usr and cpu (so a little misleading).

monitoring webapplication activity

Monitoring webapplications, gathering statistics and graphing them for trending visualization is a well-known and common practice. Gross bandwidth, disk space usage, possibly hits per second based on mod_status output in Apache, free heap, user sessions from a servlet engine (like Tomcat) are all frequent measures of activity. However to get more complete timeseries data for custom or not so commonly monitored variables that are in a webserver access log in near-realtime requires some extra effort.

Timeseries graphing and data collection and consolidation is trivially done with a Round-robin Database, the most mature and useful, not to mention open-source of these is RRDTool. RRDTool consolidates the data over uniform time periods and so "bin-ing" the data is crucial. Out-of-the-box, webservers do not bin accesslogs, or at most at daily or hourly intervals. With cronolog on Apache, you can bin easily down to the minute (and perhaps even the second, though that is sort of silly in the cases I care about). With CTools, all the webserver traffic flows through a Netscaler load-balancer, which emits weblogs and rotates them up to once a minute. We use that to bin the data in 1 minute intervals and run a script out of cron (which doesn't have to parse timestamps making it much faster) to consolidate the data and slice and dice it into RRDs to produce graphs like these:



showing hits by http status stacked -- glitches in the application show up very quickly in this



the bandwidth by tool graph shows some custom processing done on each hit to determine which Sakai "tool" it was for and the size of the response, again, near-realtime.

There are a whole lot of uncommon stats for web applications that we can graph this way, and it is very useful for trending.

undelete resource (file) for Sakai (CTools)

CTools, the learning management system (LMS) for the University of Michigan is based on Sakai. Current versions of Sakai allow for uploads of resources -- binary blobs (files) that are managed internally, but in CTools case are stored on the filesystem. Sakai keeps a map of the original filename and the name as stored on disk (based on a GUID and hashed in a directory structure that looks like Year/DayOfYear/HourOfDay/fileGUID) in a SQL table called content_resource. Deletes of a resource remove the file on disk and the table row entirely. This means undeletes are not possible out-of-the-box. Based on the "event" table one could reconstruct which file was deleted and retrieve it from a filesystem snapshot manually and quite painfully (esp. when it was on AFS as that involved requesting the snapshot volume being mounted etc. which took time).

A couple of years ago, all the CTools resource storage was moved to a Netapp filer (snapmirrored to a standby at the secondary site every 10 minutes) and the files served over NFS to CTools/Sakai. The Netapp was managed by our group and we kept 10 days of snapshots online (readonly). This meant we could automate undeletes -- for now only CTools support staff can request them, but they do so through a web interface which drives a fully automated process not requiring sysadmin hands.

Here's how it works:

• snapshots are made nightly, so a file has to exist when the snapshot is taken -- files created and deleted before a snapshot is made is not recoverable
• the undelete has to be requested within 10 days of deletion
• a ticket number has to be provided as that is used to name a directory in the support AFS space where the undeleted files are copied to from the snapshot
• CTools uses Oracle, so a table was created:
  
CREATE TABLE CONTENT_RESOURCE_DELETED
( RESOURCE_ID VARCHAR2(256 BYTE),
XML CLOB,
IN_COLLECTION VARCHAR2(256 BYTE),
FILE_PATH VARCHAR2(128 BYTE),
RESOURCE_UUID VARCHAR2(36 BYTE),
DELETE_DATE DATE
) ;
with an accompanying trigger:
  
CREATE OR REPLACE TRIGGER TRG_CONTENT_RESOURCE_4_DELETED
AFTER DELETE ON CONTENT_RESOURCE
FOR EACH ROW
BEGIN
INSERT INTO CONTENT_RESOURCE_DELETED
(RESOURCE_ID,XML,IN_COLLECTION,FILE_PATH,RESOURCE_UUID,DELETE_DATE)

VALUES(:OLD.RESOURCE_ID,:OLD.XML,:OLD.IN_COLLECTION,:OLD.FILE_PATH,:OLD.RESOURCE_UUID,SYSDATE);
END;

• Each time a file is deleted, the row is copied to the content_resource_deleted table and preserved for 30 days (after which it is dropped).
• The web-interface (a simple JSP that does a read-only lookup of the content_resource _deleted table based on a site_id) allows the support staff to pick the files to be undeleted (it only allows them to undelete files deleted before the last snapshot and deleted at most 11 days ago); picked files are saved to an undelete_queue table
---+---------------+---------------+------+---------+---------------+----+
# | column | type | null | default | pk | fk |
---+---------------+---------------+------+---------+---------------+----+
1 | FILE_PATH | VARCHAR2(128) | NO | [NULL] | SYS_C00121850 | |
2 | RESOURCE_ID | VARCHAR2(256) | NO | [NULL] | | |
3 | TICKET | VARCHAR2(64) | NO | [NULL] | | |
4 | STATUS | VARCHAR2(12) | YES | [NULL] | | |
5 | DELETE_DATE | DATE(7) | YES | [NULL] | | |
6 | INSERTEDBY | VARCHAR2(16) | NO | [NULL] | | |
7 | UPDATEDBY | VARCHAR2(16) | NO | [NULL] | | |
8 | UNDELETE_DATE | DATE(7) | YES | [NULL] | | |
---+---------------+---------------+------+---------+---------------+----+

• a script runs out of cron on a machine that has access to both AFS and the Netapp snapshots (currently one of our spare application servers) and polls the undelete_queue table for undelete requests
• upon encountering an undelete request, the script searches for the file (knowing the deletion date and file-on-disk name it is a rather easy process to step through the snapshots from most recent to oldest so that the most recent copy is used) and copies it to the support AFS space into a directory named with the ticket number, naming the file with the original names (not GUID) that the user uploaded them with
  
• the undelete_queue row status is updated appropriately depending on whether the file is successfully undeleted on not (ie., might not have existed when the snapshot was made) and an email is sent to the ticketing system with the result, alerting the support staff so that they can provide the file to the end-user
• apart from some issues with the original file having a UTF-8 name (not all the toolchain is UTF-8 friendly) in which case it is copied preserving the GUID name, there have been remarkably few problems
• stats so far are:
dbmon@ctools.vip> select min(undelete_date) from UNDELETE_QUEUE ;
----------------------+
MIN(UNDELETE_DATE)
----------------------+
2009-01-26 10:23:25.0
----------------------+
1 row in result (first row: 74 msec; total: 74 msec)
dbmon@ctools.vip> select count(*), status from UNDELETE_QUEUE group by status;
---------+----------+
COUNT(*) STATUS
---------+----------+
258 UNDELETED
8 NotFound
8 Exists
---------+----------+
3 rows in result (first row: 42 msec; total: 43 msec)

• we probably could increase the number of snapshots (say every 3 hours) though given that only 8 were not found, that doesn't seem worth the extra storage required (though that might still be manageable)
  
  
• it would be ideal if Sakai provided user-managed soft-deletes and restores within the application (something promised for JCR backed resources in the future)
• for now, this system has worked remarkably well and saved a lot of sysadmin time with minimal development (took about a week including debugging and testing)

Dada this

I'm not easily impressed by spam, but I liked this one:



I am glad that Google warned me about the impersonation -- I might have thought I got it straight from the man's Blackberry following his address to the joint session of Congress.

scooped

At the beginning of my love affair with Google Earth, I proudly wrote some KML-spitting-out pages that allowed me to record or playback a tour in it. It was written using the generic reporting framework I built using the once-upon-a-time-and-still-somewhat-spiffy-but-totally-unsupported Zapatec framework in a few hours. It worked and still works well. Too bad that Google Earth 5 now has support for Touring (recording and playback native in KML without using network links) built-in.

I suppose I can rewrite my KML to do either view-based refresh or the new way based on the user-agent. Wait. Can I do capability detection in KML?

on calendars and things

Google calendar has become my preferred way of keeping track and coordinating events with my wife, family, cohousing community, friends, travel and almost anything else that is temporal it seems. The hard exception is work, which till recently used "Meeting Maker", a piece of bloatware that had no way to exchange calendars in any known, open format, but now is based on Microsoft Exchange.

Exchange doesn't have an ical feed (that Google Calendar can easily consume, and that I've set almost everything else up to produce for its' consumption), however Exchange sends event data as iCalendar attachments in emails. Thankfully, Google Mail (Gmail and mail for apps) recognizes the text/calendar attachments, parses the iCalendar info, and offers to add it to your Google Calendar.



Amazingly, Exchange also will parse and allow Google Calendar invites (which are also iCalendar format) to be added (or refused) to my work calendar (I just had to get the correct email address to send it to for it to work!), there is a bit of broken-ness in dealing with replies to invitations from Google Calendar, but that I can bear to kludge by hand.

An well-documented, open, hardly novel way of exchanging calendar information by email is fantastic and I've added such attachments to the GO online meal signup and shift system, here's an example (part of a multipart/alternative message):


------=MIME_BOUNDRY_altpparts
Content-Type: text/calendar; charset="utf-8"; method=REQUEST
Content-Transfer-Encoding: 8bit
Content-Disposition: attachment; filename=invite.ics

BEGIN:VCALENDAR
PRODID:-//Grot Org//Meal tracker//EN
VERSION:2.0
CALSCALE:GREGORIAN
METHOD:REQUEST
BEGIN:VEVENT
DTSTART:20090203T231500Z
DTEND:20090204T000000Z
DTSTAMP:20090129T154454Z
ORGANIZER;CN=go-meal:mailto:foo@plzs.org
UID:1680@meals.plzs.org
ATTENDEE;CUTYPE=INDIVIDUAL;ROLE=REQ-PARTICIPANT;PARTSTAT=ACCEPTED;RSVP=FALSE;CN=Fitzgerald/Aditya;X-NUM-GUESTS=4:mailto:nobody@grot.org
CLASS:PRIVATE
CREATED:20090203T130300Z
DESCRIPTION:Wings Over Great Oak by tim details at https://example.com/meal-signup/show?m=1680
LAST-MODIFIED:20090129T154454Z
LOCATION:go-meal
SEQUENCE:0
STATUS:CONFIRMED
SUMMARY:Wings Over Great Oak (go-meal)\, 4 diners
TRANSP:OPAQUE
BEGIN:VALARM
ACTION:DISPLAY
DESCRIPTION:REMINDER
TRIGGER;RELATED=START:-PT15M
END:VALARM
END:VEVENT
END:VCALENDAR
------=MIME_BOUNDRY_altpparts--


Netapp OnTap per-volume statistics

I'm a fan of Netapps, and I've grown used to using the OnTap SNMP agent to collect statistics on various filer functions, stuff them into RRDs and troubleshoot using trend graphs (I really like drraw for that). I can then make cool graphs like:



However, those ops numbers are for the filer as a whole making it hard to tell *what* is the target for all that activity -- we need per volume and even per disk stats to find hot spots. The "stats" command in OnTap can expose that data, however not via SNMP and only via the cli (maybe via the SDK, but that's a whole level of complication I'd like to avoid). The whole list of supported stats counters is exposed using the command stats list counters. So using ssh with public keys (and using stats start/stop every 5 minutes), I was able to "plug-in" the output from periodically querying the stats command into the existant set of RRDs that drraw draws from. Here's the per-volume read/write/other ops for the same filer:



and one showing total time spent serving along with amount of data served in bytes:



and bandwidth by volume:


the relevant code fragment is quite simple and follows the SNMP_util::snmpmaptable semantics, each "row" of instance data is sent to a callback function which writes it into the appropriate RRD (with an example invocation):

my($getStatCols);

$getStatCols->{'volume'}->{'type'} = "volStats";
@{$getStatCols->{'volume'}->{'cols'}} = ("read_data", "read_latency", "read_ops",  "write_data", "write_latency", "write_ops", "other_latency", "other_ops");

for my $stat (keys %{$getStatCols}){
my($stattxt) = "${stat}:*:" . join(" ${stat}:*:", @{$getStatCols->{$stat}->{'cols'}});
getStatTable($username, $netappHostname, $getStatCols->{$stat}->{'type'}, $stattxt,  \&printfun);
}

sub getStatTable {
my($user, $host, $id, $statcmd, $callback) = @_;

my(@ret) = `ssh ${user}\@${host} "stats stop -I ${id} -O print_zero_values=off -c -d |"`;
my(@start) = `ssh ${user}\@${host} "stats start -I ${id} ${statcmd}"`;
my($startcmd) = join(' ', @start);
if ($startcmd){
   $startcmd =~ s/\s+//g;
   chomp($startcmd);
   if ($startcmd ne ""){
     print STDERR "ERROR starting stats collection for ${user}\@${host} $statcmd: " . $startcmd . "\n";
   }
}

if ($#ret <= 2){
   print STDERR "ERROR retrieving stats from ${user}\@${host} ${statcmd}: " . join(" ", @ret) . "\n";
} else {
   shift(@ret);
   shift(@ret);
   shift(@ret);
   for (my $i=0;$i<=$#ret;$i++){
     my($l) = $ret[$i];
     chomp($l);
     $l =~ s/^\s+//g;
     $l =~ s/\s+$//g;
     if ($l !~ /|$/){
       $l .= "|";
     }
     my(@cols) = ($id, $i, split(/\|/, $l));
     &$callback(@cols);
   }
 }
}

sub printfun {
#shift(@_);
my(@vals) = @_;
for (my $i=0;$i <= $#vals; $i++){
if (! $vals[$i] || $vals[$i] eq ''){
$vals[$i] = 0;
}
}
#next if (! @vals);
#next if (! $vals[0] || ! $vals[1]);
print join('|', @vals) . "|\n";
}



name the reason for that blip

See that blip in requests around noon EST today? Should be easy to tell what it was...



yes, all those UM students and faculty who turned away from CTools to watch the Obama inauguration. Never noticed that blip for the 43rd president, but then again it is probably due to the fact that CTools is pervasive enough now that we can use it instead of Nielsen ratings. Or maybe not.