DCppStats - an automating script for parsing chat logs
Moderator: Moderators
DCppStats - an automating script for parsing chat logs
I've finally finished DCppStats, well finished it for release anyways. It's a bash script I've been running for a few months that automate archiving and running Pisg to parse the stats for my hub and I've decided to make it public. It's at http://dcppstats.iaconhub.com if you want to look at it. A Windows compatible one is coming, but it's not ready yet, so please don't beg for it. I've tested this one in RedHat 7.3 and 8.0 and it's been running nightly for me with no problems for about 4 months.
Ok, I finally got around to getting all the stuff needed for your script to work on Win2K. Pisg works great by itself but I cant get your script to work. Its having issues with navigating the windows filenames (especially program files). Whats the trick to the proper path naming in your config file?
Ahh, i thought it was going to be something like that....still having problems.
when navigating through my directories under the cygwin bash shell, I have to use "cd program\ files" to change directories. But once I'm in there, the shell shows me at "/cygdrive/d/program files/dcplusplus/" Can't I use spaces in unix?
For example, I have the following in my config.
dclogdir='/cygdrive/d/program files/dcplusplus/logs'
dclog='/cygdrive/d/program files/dcplusplus/logs/digital-void.kicks-ass.net.log'
When I run the script, I get the following:
BSOD2600@gamer2k /cygdrive/d/program files/dcplusplus/dcppstats
$ sh ./dcppstats.sh
cat: /cygdrive/d/program: No such file or directory
cat: files/dcplusplus/logs//cygdrive/d/program: No such file or directory
cat: files/dcplusplus/logs/digital-void.kicks-ass.net.log: No such file or directory
pisg v0.48 - Perl IRC Statistics Generator
As you can see, its not working with the spaces...how do I fix?
when navigating through my directories under the cygwin bash shell, I have to use "cd program\ files" to change directories. But once I'm in there, the shell shows me at "/cygdrive/d/program files/dcplusplus/" Can't I use spaces in unix?
For example, I have the following in my config.
dclogdir='/cygdrive/d/program files/dcplusplus/logs'
dclog='/cygdrive/d/program files/dcplusplus/logs/digital-void.kicks-ass.net.log'
When I run the script, I get the following:
BSOD2600@gamer2k /cygdrive/d/program files/dcplusplus/dcppstats
$ sh ./dcppstats.sh
cat: /cygdrive/d/program: No such file or directory
cat: files/dcplusplus/logs//cygdrive/d/program: No such file or directory
cat: files/dcplusplus/logs/digital-void.kicks-ass.net.log: No such file or directory
pisg v0.48 - Perl IRC Statistics Generator
As you can see, its not working with the spaces...how do I fix?
No change....
BSOD2600@gamer2k /cygdrive/d/program files/dcplusplus/dcppstats
$ sh ./dcppstats.sh
cat: /cygdrive/d/program\: No such file or directory
cat: files/dcplusplus/logs//cygdrive/d/program\: No such file or directory
cat: files/dcplusplus/logs/digital-void.kicks-ass.net.log: No such file or directory
pisg v0.48 - Perl IRC Statistics Generator
BSOD2600@gamer2k /cygdrive/d/program files/dcplusplus/dcppstats
$ sh ./dcppstats.sh
cat: /cygdrive/d/program\: No such file or directory
cat: files/dcplusplus/logs//cygdrive/d/program\: No such file or directory
cat: files/dcplusplus/logs/digital-void.kicks-ass.net.log: No such file or directory
pisg v0.48 - Perl IRC Statistics Generator
Well its almost working now...I mounted each directory in windows that I was using to a folder under cygin. For example, "d:\program files" is mounted to /programs. Did that with my tmp, pisg, stat directories. Also made a small change to a line in the script. Now I've run into another roadblock that I cant seem to get around....
$ sh ./dcppstats.sh
pisg v0.48 - Perl IRC Statistics Generator
Using config file: pisg.cfg
Statistics for channel #digitalwarehouse @ The Void by BSOD2600
Analyzing log(/tmp/output.log) in 'DCpp' format...
Could not load pisg! Reason:
pisg: Unable to open logfile(/tmp/output.log): No such file or directory
now, if I manually go to /tmp I can see there is a file called output.log with some data in it. Also, I noticed pisg is parsing the pisg.cfg file where I have setup two of my hubs (digitalwarehouse is first in the list). In dcppstats.sh, I have digital-void.kicks-ass.net.log as my first log...do I need to remove the channel settings in pisg.cfg? If I do remove the channel for each hub, then your script doesnt tell pisg to use DCpp format and it defaults to mIRC instead.
$ sh ./dcppstats.sh
pisg v0.48 - Perl IRC Statistics Generator
Using config file: pisg.cfg
Statistics for channel #digitalwarehouse @ The Void by BSOD2600
Analyzing log(/tmp/output.log) in 'DCpp' format...
Could not load pisg! Reason:
pisg: Unable to open logfile(/tmp/output.log): No such file or directory
now, if I manually go to /tmp I can see there is a file called output.log with some data in it. Also, I noticed pisg is parsing the pisg.cfg file where I have setup two of my hubs (digitalwarehouse is first in the list). In dcppstats.sh, I have digital-void.kicks-ass.net.log as my first log...do I need to remove the channel settings in pisg.cfg? If I do remove the channel for each hub, then your script doesnt tell pisg to use DCpp format and it defaults to mIRC instead.
Well I figured out why I'm getting "pisg: Unable to open logfile(/tmp/output.log): No such file or directory " I had a hunch that it was looking for my log file in /pisg/tmp/output.log (instead of /tmp/output.log), SO I created that directory to see what happens. It got past that error! So now I need to figure out how to tell it to go back to the root, then /tmp when parsing the log. I'm assuming the following error is based off the same problem...
Analyzing log(./tmp/output.log) in 'DCpp' format...
Finished analyzing log, 1 days total.
Channel analyzed succesfully in 00 hours, 00 minutes and 00 seconds on Thu Jun
5 10:35:53 2003
Now generating HTML(/webstats/daily.html)...
Could not load pisg! Reason:
pisg: Unable to open outputfile(/webstats/daily.html): No such file or directory
Analyzing log(./tmp/output.log) in 'DCpp' format...
Finished analyzing log, 1 days total.
Channel analyzed succesfully in 00 hours, 00 minutes and 00 seconds on Thu Jun
5 10:35:53 2003
Now generating HTML(/webstats/daily.html)...
Could not load pisg! Reason:
pisg: Unable to open outputfile(/webstats/daily.html): No such file or directory
I dunno how much more of a path I can tell it. /tmp/output.log IS the full path to the file. Just for whatever reason, its not taking it as root/tmp/output.log Once I figure out this last issue, I think the script will work and the new /webstats error will go away.yakko wrote:I always use the full path to make sure nothing gets confused in the script.
IT WORKS!
Finally got this sucker to work! Lots of playing around with settings...
My pisg.cfg looks like:
my dcppstats.sh looks like:
These are the following mappings:
/programs = d:\program files
/tmp = d:\temp
/webstats = G:\DC Temp\DW Hub\Web Stats
/pisg = d:\pisg
My pisg.cfg looks like:
Code: Select all
<channel="#digital-void">
Logfile="D:\temp\output.log"
Format="DCpp"
Network="The Void"
Maintainer ="BSOD2600"
OutputFile="G:\DC Temp\DW Hub\Web Stats\daily.html"
</channel>
Code: Select all
#VARIABLES
tdy=`date +%Y%m%d`
month=`date +%m`
year=`date +%Y`
today=`date +%d`
logdate=`date +%Y-%m-%d`
dclogdir="/programs/dcplusplus/logs"
dclog="digital-void.kicks-ass.net.log"
workingdir=/tmp
outdir=/webstats
webroot=/webstats
webfile=daily.html
archiveroot="/programs/dcplusplus/logs/archived"
pisg=/pisg
# Copy daily log to log.log to prepare for conversion
# cp $dclogdir/$dclog $workingdir/output.log
cat $dclogdir/$dclog |grep $logdate >$workingdir/output.log
# Run PISG (this just does a basic stats page. See PISG documentation for further customization of PISG)
cd $pisg
cd /pisg
perl pisg
# copy pisg output file to stats web root
#cp -f $outdir/daily.html $webroot/$webfile
# archive stats page
cd $webroot
cp -f $webroot/$webfile $webroot/archive/$tdy.html
# move converted logfile to archive directory and datestamp
mv -f $workingdir/output.log $archiveroot/$tdy.log
# Compress DC++ logfile with gzip
gzip -f $archiveroot/$tdy.log
/programs = d:\program files
/tmp = d:\temp
/webstats = G:\DC Temp\DW Hub\Web Stats
/pisg = d:\pisg
Right, I now understand what grep is doing to get the current log file for today. Got any easy way of running it on a week of logs? The only way I can think of is to go into the log archive directory, extract a weeks worth of logs to seperate files, then merge them all together before finally running PISG. Happen to have a script for that too?
Well I wrote a script that will simply counts back 6 days, extract the archieved logs, merge them all together and then finally run PISG. Its not pretty nor does it account for a week that spans a previous month (yet, got a new tool and going to work on that). Anyways, its a start...
PISG2.CFG
Code: Select all
#!/bin/sh
#Explicit VARIABLES
tdy=`date +%Y%m%d`
month=`date +%m`
year=`date +%Y`
today=`date +%d`
logdate=`date +%Y-%m-%d`
dclogdir="/programs/dcplusplus/logs"
dclog1="digital-void.kicks-ass.net.log"
dclog2="kkhub.kicks-ass.net.log"
workingdir=/tmp
outdir=/webstats
webroot=/webstats
webfile1=week-v.html
webfile2=week-w.html
archiveroot="/programs/dcplusplus/logs/archived"
pisg=/pisg
a=$today
b=$(expr $today - 1)
c=$(expr $today - 2)
d=$(expr $today - 3)
e=$(expr $today - 4)
f=$(expr $today - 5)
g=$(expr $today - 6)
#echo "$a, $b, $c, $d, $e, $f, $g"
for z in {a,b,c,d,e,f,g} do zcat $archiveroot/the-void/$year$month$z.log.gz >> $workingdir/week-w.log
for z in {a,b,c,d,e,f,g} do zcat $archiveroot/digital-void/$year$month$z.log.gz >> $workingdir/week-v.log
cd $pisg
perl pisg -co pisg2.cfg
cd $webroot
cp -f $webroot/$webfile1 $webroot/archive/the-void/$year$month$a-$year$month$g.html
cp -f $webroot/$webfile2 $webroot/archive/digital-void/$year$month$a-$year$month$g.html
mv -f $workingdir/week-v.log $archiveroot/the-void/$year$month$a-$year$month$g.log
mv -f $workingdir/week-w.log $archiveroot/digital-void/$year$month$a-$year$month$g.log
gzip -f $archiveroot/the-void/$year$month$a-$year$month$g.log
gzip -f $archiveroot/digital-void/$year$month$a-$year$month$g.log
PISG2.CFG
Code: Select all
<channel="#digital-void">
Logfile="D:\temp\week-v.log"
Format="DCpp"
Network="The Void"
Maintainer ="BSOD2600"
OutputFile="G:\DC Temp\DW Hub\Web Stats\week-v.html"
</channel>
<channel="#digitalwarehouse">
Logfile="D:\temp\week-w.log"
Format="DCpp"
Network="The Void"
Maintainer ="BSOD2600"
OutputFile="G:\DC Temp\DW Hub\Web Stats\week-w.html"
</channel>