robert on June 15, 2006 in Uncategorized
How long does my setup at work take to crunch 356,771,740 lines of samba logs and create a database of login/logouts? Hopefully I’ll know tomorrow.
How long does my setup at work take to crunch 356,771,740 lines of samba logs and create a database of login/logouts? Hopefully I’ll know tomorrow.
How long?
Well, I ended up not letting the original script finish. I had two copies running … one processing from 1/1/2006 – 6/14/06 and the other from 8/1/2005 – 12/31/05. I started them around 3ish on Thursday afternoon and the first had made it to February 24th by 8am Friday morning (having processed around 150K logins. The 23rd took nearly 90 minutes to process, so that meant the rest were going to take longer. Suffice it to say that wouldn’t cut it, so I killed the script and tweaked it to ignore everything except logins for the 5 computer labs I was concerned with. Wiped out the database and restarted … took 6 hours to process 8/1/2005 – 6/14/2006… 117K records created.