/root/.forward     who gets mail


From vic@plan9.bell-labs.com Fri Apr 29 11:11:30 EDT 2005
To: ehg
Subject: oncore traces
From: vic zandy <vic@plan9.bell-labs.com>
Date: Fri, 29 Apr 2005 11:11:30 -0400
MIME-Version: 1.0
Content-Type: text/plain; charset="US-ASCII"
Content-Transfer-Encoding: 7bit

Here's a copy of the reference I use to remember how
oncore traces are processed; it's the same reference I
sent you a while ago.

Generally the traces have underwhelmed oncore; the
trace disk at 52% right now, the end of the monthly
cycle.

At some point I was uneasy about automatically deleting
the traces after they are copied to poseidon, so I
disabled it.  That turned out to be worthwhile -- one
month Jin reported losing the poseidon traces a week or
two after the script would have nuked them; I was able
to give them back.

These days after I get mail that the transfer has
completed, a few days after the 3rd of the month, I
check in with Jun to be sure she's ready for me to
delete them, then I delete them by hand.  You should
start getting the transfer report.  (Both this mail and
automatic deletion are done in /home/dong/scripts/cron-strpmove)

You also should start getting the daily reports about
oncore's status again, starting tomorrow.  (I
re-pointed /root/.forward back to you.)

There's been no progress on getting new storage for
these traces as far I as know.  At some point soon Jun
will go on leave.

Vic




The oncore -> poseidon pipeline for packet traces.

1. collection and short-term storage on oncore

root@oncore's crontab runs /root/rt twice daily

/root/rt dumps the packet trace to local disk:
	 tcpdump fxp1
	| /root/dstrip
	| gzip -1
	> /disk3/bl-traces/<date+time>.unstripped.gz

/root/dstrip is /root/traces-strip/dstrip.c, built via
	% make -f Makefile.freebsd dstrip.bsd
i believe that this program strips off packet payload.

Everything lives on oncore's local disks.


2. storage on poseidon via kriek

kriek's automounter configuration (see /etc/auto*) includes these bindings:

/n/data/sdg1 -> poseidon:/sdg1
/n/dnet/trace3 -> oncore:/disk3
/home/dong -> bopp:/v8/dong

dong@kriek's crontab runs /home/dong/scripts/cron-strpmove.csh
at 1800 the 3rd of each month.

/home/dong/scripts/cron-strpmove.csh calls /home/dong/scripts/strpmove,
passing a list of /n/dnet/trace3/bl-traces/<date+time>.unstripped.gz for
the past month.

[The following takes several days to complete.  We're collecting
on the order of 10 GB per week, and it takes a while to uncompress,
process, recompress, and move across the network.   ehg 18jun05]

/home/dong/scripts/strpmove copies the trace files to poseidon:
	for-each F in /n/dnet/trace3/bl-traces/<date+time>.unstripped.gz:
		 gunzip -c < $F
		| /home/dong/bin/alpha/dstrip
		| gzip
		> /n/data/sdg1/bl-traces/<date+time>.stripped.gz

I don't know what /home/dong/bin/alpha/dstrip does.  On
input trace files  from oncore, the output (compared
through tcpdump|head) appears the same.

I believe that
	kriek:/home/dong/bin/alpha/dstrip
is functionally the same as
	oncore:/root/dstrip

Perhaps oncore:/root/dstrip was introduced later to
curtail disk usage of unstripped traces on oncore.

The source for /home/dong/bin/alpha/dstrip may be in
/home/dong/trace-strip; we don't have access yet to
investigate.

After /home/dong/scripts/strpmove finishes,
/home/dong/scripts/cron-strpmove.csh unlinks the trace files.