DCppStats - an automating script for parsing chat logs

Know of something that might be useful to the DC community? Post it here! (Still, no advertising)

Moderator: Moderators

Locked
yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

DCppStats - an automating script for parsing chat logs

Post by yakko » 2003-02-16 16:09

I've finally finished DCppStats, well finished it for release anyways. It's a bash script I've been running for a few months that automate archiving and running Pisg to parse the stats for my hub and I've decided to make it public. It's at http://dcppstats.iaconhub.com if you want to look at it. A Windows compatible one is coming, but it's not ready yet, so please don't beg for it. I've tested this one in RedHat 7.3 and 8.0 and it's been running nightly for me with no problems for about 4 months.

Moridin_
Posts: 4
Joined: 2003-02-12 21:55

Post by Moridin_ » 2003-03-30 12:36

Looks nice. Any Idea when or if the windows version will be ready?

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-03-30 17:18

i tried and i tried and couldn't get a windows version without a complete rewrite, so i'm gonna have to figure out how to do some stuff in windows that i did in linux. If only i knew perl. :(

Moridin_
Posts: 4
Joined: 2003-02-12 21:55

Post by Moridin_ » 2003-03-30 22:03

Bummer. This looks like an awsome toy. Wish I could help you on it.

Argonne
Posts: 11
Joined: 2003-03-05 12:34

cygwin

Post by Argonne » 2003-04-10 18:04

You could download a package called cygwin. I have not tried but I am 99% certain that it can run Bash scripts (at least simpler ones) on windows.

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-04-10 23:54

I haven't had time to test it in a while, but I think it doesn't run under cygwin. It uses some features of the GNU tools that the windows ports of the tools can't do.

ender
Posts: 224
Joined: 2003-01-03 17:47

Post by ender » 2003-04-11 04:41

Are you sure? Cygwin is almost it's own an operating system inside Windows, and many *nix programs compile on it out of the box...

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-04-17 21:11

I've just tested it in cygwin and DCppStats seems to work fine in Windows using Cygwin. I tested it on WinXP, but it should work fine in any OS that Cygwin supports.

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-04 10:54

Ok, I finally got around to getting all the stuff needed for your script to work on Win2K. Pisg works great by itself but I cant get your script to work. Its having issues with navigating the windows filenames (especially program files). Whats the trick to the proper path naming in your config file?

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-04 11:06

you're having to use cygwin so you have to use unix styled paths. If it's in D:\apps then you would have it like /cygdrive/d/apps/dcppstats in dcppstats.sh

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-04 11:06

oops...that'd be d:\apps\dcppstats -> /cygdrive/d/apps/dcppstats

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-04 11:46

Ahh, i thought it was going to be something like that....still having problems.

when navigating through my directories under the cygwin bash shell, I have to use "cd program\ files" to change directories. But once I'm in there, the shell shows me at "/cygdrive/d/program files/dcplusplus/" Can't I use spaces in unix?

For example, I have the following in my config.
dclogdir='/cygdrive/d/program files/dcplusplus/logs'
dclog='/cygdrive/d/program files/dcplusplus/logs/digital-void.kicks-ass.net.log'

When I run the script, I get the following:
[email protected] /cygdrive/d/program files/dcplusplus/dcppstats
$ sh ./dcppstats.sh
cat: /cygdrive/d/program: No such file or directory
cat: files/dcplusplus/logs//cygdrive/d/program: No such file or directory
cat: files/dcplusplus/logs/digital-void.kicks-ass.net.log: No such file or directory
pisg v0.48 - Perl IRC Statistics Generator


As you can see, its not working with the spaces...how do I fix?

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-04 11:54

Try to add the \ in the config:

dclogdir='/cygdrive/d/program\ files/dcplusplus/logs'
dclog='/cygdrive/d/program\ files/dcplusplus/logs/digital-void.kicks-ass.net.log'


I put it in d:\apps\ to avoid running into a long filename issue. If it doesn't work let me know and I'll look into it more.

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-04 12:02

No change....

[email protected] /cygdrive/d/program files/dcplusplus/dcppstats
$ sh ./dcppstats.sh
cat: /cygdrive/d/program\: No such file or directory
cat: files/dcplusplus/logs//cygdrive/d/program\: No such file or directory
cat: files/dcplusplus/logs/digital-void.kicks-ass.net.log: No such file or directory
pisg v0.48 - Perl IRC Statistics Generator

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-04 13:30

I haven't been able to get it to work with spaces either. have you tried a directory with no spaces?

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-05 03:03

Well its almost working now...I mounted each directory in windows that I was using to a folder under cygin. For example, "d:\program files" is mounted to /programs. Did that with my tmp, pisg, stat directories. Also made a small change to a line in the script. Now I've run into another roadblock that I cant seem to get around....

$ sh ./dcppstats.sh
pisg v0.48 - Perl IRC Statistics Generator

Using config file: pisg.cfg

Statistics for channel #digitalwarehouse @ The Void by BSOD2600

Analyzing log(/tmp/output.log) in 'DCpp' format...
Could not load pisg! Reason:
pisg: Unable to open logfile(/tmp/output.log): No such file or directory

now, if I manually go to /tmp I can see there is a file called output.log with some data in it. Also, I noticed pisg is parsing the pisg.cfg file where I have setup two of my hubs (digitalwarehouse is first in the list). In dcppstats.sh, I have digital-void.kicks-ass.net.log as my first log...do I need to remove the channel settings in pisg.cfg? If I do remove the channel for each hub, then your script doesnt tell pisg to use DCpp format and it defaults to mIRC instead.

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-05 08:20

i don't have any information about my hub in the pisg.cfg except for usernames. for other hubs you might want to have different scripts.

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-05 12:40

Well I figured out why I'm getting "pisg: Unable to open logfile(/tmp/output.log): No such file or directory " I had a hunch that it was looking for my log file in /pisg/tmp/output.log (instead of /tmp/output.log), SO I created that directory to see what happens. It got past that error! So now I need to figure out how to tell it to go back to the root, then /tmp when parsing the log. I'm assuming the following error is based off the same problem...

Analyzing log(./tmp/output.log) in 'DCpp' format...
Finished analyzing log, 1 days total.
Channel analyzed succesfully in 00 hours, 00 minutes and 00 seconds on Thu Jun
5 10:35:53 2003
Now generating HTML(/webstats/daily.html)...
Could not load pisg! Reason:
pisg: Unable to open outputfile(/webstats/daily.html): No such file or directory

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-05 13:15

is there a /webstats? Rememver that'd be off the root. I always use the full path to make sure nothing gets confused in the script.

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-05 13:36

yakko wrote:I always use the full path to make sure nothing gets confused in the script.
I dunno how much more of a path I can tell it. /tmp/output.log IS the full path to the file. Just for whatever reason, its not taking it as root/tmp/output.log Once I figure out this last issue, I think the script will work and the new /webstats error will go away.

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

IT WORKS!

Post by BSOD2600 » 2003-06-05 17:48

Finally got this sucker to work! Lots of playing around with settings...
My pisg.cfg looks like:

Code: Select all

<channel="#digital-void">
   Logfile="D:\temp\output.log"
   Format="DCpp"
   Network="The Void"
   Maintainer ="BSOD2600"
   OutputFile="G:\DC Temp\DW Hub\Web Stats\daily.html"
</channel>


my dcppstats.sh looks like:

Code: Select all

#VARIABLES
tdy=`date +%Y%m%d`
month=`date +%m`
year=`date +%Y`
today=`date +%d`
logdate=`date +%Y-%m-%d`
dclogdir="/programs/dcplusplus/logs"
dclog="digital-void.kicks-ass.net.log"
workingdir=/tmp
outdir=/webstats
webroot=/webstats
webfile=daily.html
archiveroot="/programs/dcplusplus/logs/archived"
pisg=/pisg

# Copy daily log to log.log to prepare for conversion
# cp $dclogdir/$dclog $workingdir/output.log

cat $dclogdir/$dclog |grep $logdate >$workingdir/output.log

# Run PISG (this just does a basic stats page. See PISG documentation for further customization of PISG)
cd $pisg
cd /pisg
perl pisg

# copy pisg output file to stats web root
#cp -f $outdir/daily.html $webroot/$webfile

# archive stats page
cd $webroot
cp -f $webroot/$webfile $webroot/archive/$tdy.html

# move converted logfile to archive directory and datestamp
mv -f $workingdir/output.log $archiveroot/$tdy.log

# Compress DC++ logfile with gzip
gzip -f $archiveroot/$tdy.log


These are the following mappings:
/programs = d:\program files
/tmp = d:\temp
/webstats = G:\DC Temp\DW Hub\Web Stats
/pisg = d:\pisg

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-17 01:56

How would I go about changing to logging period to weekly? I've tried changing logdate to logdate=`date +%Y-%W` but it didnt work.

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-17 07:31

to have it do weekly just run it with a week's worth of logs. the logdate just grabs the current date for the filename.

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-18 23:47

Right, I now understand what grep is doing to get the current log file for today. Got any easy way of running it on a week of logs? The only way I can think of is to go into the log archive directory, extract a weeks worth of logs to seperate files, then merge them all together before finally running PISG. Happen to have a script for that too? ;)

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-18 23:54

nope, sorry. It should keep the original logfile untouched and archive each days in a file. So if you can figure out how to grab a weeks worth of chat then it'd work, but that's beyond my knowledge. :(

BSOD2600
Forum Moderator
Posts: 503
Joined: 2003-01-27 18:47
Location: USA
Contact:

Post by BSOD2600 » 2003-06-21 15:09

Well I wrote a script that will simply counts back 6 days, extract the archieved logs, merge them all together and then finally run PISG. Its not pretty nor does it account for a week that spans a previous month (yet, got a new tool and going to work on that). Anyways, its a start...

Code: Select all

#!/bin/sh
#Explicit VARIABLES
tdy=`date +%Y%m%d`
month=`date +%m`
year=`date +%Y`
today=`date +%d`
logdate=`date +%Y-%m-%d`
dclogdir="/programs/dcplusplus/logs"
dclog1="digital-void.kicks-ass.net.log"
dclog2="kkhub.kicks-ass.net.log"
workingdir=/tmp
outdir=/webstats
webroot=/webstats
webfile1=week-v.html
webfile2=week-w.html
archiveroot="/programs/dcplusplus/logs/archived"
pisg=/pisg

a=$today
b=$(expr $today - 1)
c=$(expr $today - 2)
d=$(expr $today - 3)
e=$(expr $today - 4)
f=$(expr $today - 5)
g=$(expr $today - 6)
#echo "$a, $b, $c, $d, $e, $f, $g"
                                 
for z in {a,b,c,d,e,f,g} do zcat $archiveroot/the-void/$year$month$z.log.gz >> $workingdir/week-w.log
                                                                                                               
for z in {a,b,c,d,e,f,g} do zcat $archiveroot/digital-void/$year$month$z.log.gz >> $workingdir/week-v.log
                                                                           
cd $pisg
perl pisg -co pisg2.cfg
       
cd $webroot
cp -f $webroot/$webfile1 $webroot/archive/the-void/$year$month$a-$year$month$g.html
cp -f $webroot/$webfile2 $webroot/archive/digital-void/$year$month$a-$year$month$g.html
       
mv -f $workingdir/week-v.log $archiveroot/the-void/$year$month$a-$year$month$g.log
mv -f $workingdir/week-w.log $archiveroot/digital-void/$year$month$a-$year$month$g.log
       
gzip -f $archiveroot/the-void/$year$month$a-$year$month$g.log
gzip -f $archiveroot/digital-void/$year$month$a-$year$month$g.log



PISG2.CFG

Code: Select all

<channel="#digital-void">
   Logfile="D:\temp\week-v.log"
   Format="DCpp"
   Network="The Void"
   Maintainer ="BSOD2600"
   OutputFile="G:\DC Temp\DW Hub\Web Stats\week-v.html"
</channel>

<channel="#digitalwarehouse">
   Logfile="D:\temp\week-w.log"
   Format="DCpp"
   Network="The Void"
   Maintainer ="BSOD2600"
   OutputFile="G:\DC Temp\DW Hub\Web Stats\week-w.html"
</channel>

yakko
Posts: 258
Joined: 2003-01-27 01:04
Contact:

Post by yakko » 2003-06-21 15:53

wow...that's pretty cool!

Locked