Into.the.Void.

Out of boredom I decided to parse the Linux-Greek-Users (LGU) archives and create some graphs. Then I wrote a few more oneliners to deduct some numbers out of the archives. These numbers may or may not mean anything to someone, it’s entirely up to the reader.. Since the archives contain some amount of spam (not too much though) one must take that into consideration as well while reading the numbers I extracted below…

First thing I did was to download the index file containing the links to the monthly archives since February 1997:
wget http://lists.hellug.gr/pipermail/linux-greek-users/

Then download each month’s archive:
for i in `grep date index.html | cut -d"\"" -f2`; do foo=`echo $i|cut -d"/" -f1`; wget http://lists.hellug.gr/pipermail/linux-greek-users/$i -O $foo-date.html ; done

After a while I had the full archives at my disk…
Then it was time for the first metric.
How many posts does each month have ?
This was quite easy because mailman contains this information inside each month’s archive:
for i in *-date.html; do count=`grep -i "Messages:" $i | cut -d">" -f 3|cut -d"< " -f1`; echo "$i $count" >> count.txt;done
That command gave me an output such as:

1999-May-date.html 985
1999-November-date.html 1441
1999-October-date.html 1148
1999-September-date.html 1369
2000-April-date.html 690
2000-August-date.html 354
2000-December-date.html 444
2000-February-date.html 833
…

It didn’t take me a long time before I had this data entered inside OOcalc. Time for the first graph:
LGU posts per month:

And the average monthly posts per year:

One can certainly argue there’s a tendency here. 2008 had the fewer posts than any other year…

What’s more interesting is who are actually writing on this list. To extract that information made the following oneliners:
First of all I converted all html files to utf8 because the archives are kept at iso-8859-7 encoding and I am using a unicode terminal:
for i in *-date.html; do iconv -f iso-8859-7 -t utf-8 $i -o $i.utf; done
Then I could find all the authors for each monthly archive:
for i in *-date.html.utf; do grep "" $i | cut -d">" -f 2 >$i-authors.txt; done
Then count each author’s posts:
for i in *-date.html.utf-authors.txt; do sort $i | uniq -c | sort -n >$i-sorted-count; done
Easiest thing to do is to count how many people post per month. Are the users of the mailing list increasing or decreasing ?
for i in *-sorted-count; do wc -l $i >> posters-per-month; done
That gave a nice output like that:

55 1997-April-date.html.utf-authors.txt-sorted-count
49 1997-August-date.html.utf-authors.txt-sorted-count
88 1997-December-date.html.utf-authors.txt-sorted-count
62 1997-February-date.html.utf-authors.txt-sorted-count
41 1997-July-date.html.utf-authors.txt-sorted-count
60 1997-June-date.html.utf-authors.txt-sorted-count
62 1997-March-date.html.utf-authors.txt-sorted-count
69 1997-May-date.html.utf-authors.txt-sorted-count
…

I inserted that to OOcalc again and here is the output:

The decline in the number of posters is clearly shown.

Another interesting statistic would be the months with the most and fewer posters respectively:
Months with most posters:
sort -n posters-per-month | tail -n10

166 1999-May-date.html.utf-authors.txt-sorted-count
167 2001-October-date.html.utf-authors.txt-sorted-count
168 1999-December-date.html.utf-authors.txt-sorted-count
173 1999-October-date.html.utf-authors.txt-sorted-count
173 2001-June-date.html.utf-authors.txt-sorted-count
174 2001-May-date.html.utf-authors.txt-sorted-count
176 1999-September-date.html.utf-authors.txt-sorted-count
184 1999-November-date.html.utf-authors.txt-sorted-count
192 2000-March-date.html.utf-authors.txt-sorted-count
222 2004-November-date.html.utf-authors.txt-sorted-count

Months with fewer posters:
sort -n posters-per-month | head -n10

16 2008-August-date.html.utf-authors.txt-sorted-count
32 2008-December-date.html.utf-authors.txt-sorted-count
36 2008-July-date.html.utf-authors.txt-sorted-count
41 1997-July-date.html.utf-authors.txt-sorted-count
44 2008-May-date.html.utf-authors.txt-sorted-count
49 1997-August-date.html.utf-authors.txt-sorted-count
51 2008-September-date.html.utf-authors.txt-sorted-count
51 2009-March-date.html.utf-authors.txt-sorted-count
53 2003-August-date.html.utf-authors.txt-sorted-count
53 2009-February-date.html.utf-authors.txt-sorted-count

Then I got interested in finding out who are the top posters throughout the archives. I didn’t want to write anything complex in order to sum every user’s post and I decided to find out who are the top5 posters in each month and the see whose names are repeated over and over in the top5.

Using the following oneliner I was able to find the top5 posters of each month:
for i in *-count; do tail -n5 $i >> top5-of-each-month.txt; done

And then using a bit of perl I was able to extract the posters appearing most times, here’s the top 20 of them:

8 Alexios Chouchoulas
9 fs
9 Spiros Bolis
10 Alexandros Papadopoulos
10 Giannis Stoilis
10 Harris Kosmidhs
10 Vasilis Vasaitis
11 Giannis Papadopoulos
11 Michael Iatrou
11 Panos Katsaloulis
11 Άγγελος Οικονομόπουλος
16 George Notaras
17 Nick Demou
24 George Daflidis-Kotsis
24 Michalis Kabrianis
33 I.Ioannou
41 DJ Art
52 V13
74 Christos Ricudis
83 Giorgos Keramidas

The number appearing before the name is the number of months the poster has been inside the top5 posters for a month. That means that Giorgos Keramidas is in the top5 posters for 83 months out of 147 months of archives that I parsed. Pretty impressive!

Then I wanted to create some graphs about http://forum.hellug.gr as well. HELLUG’s forum has less than 2 years of life so the graphs cannot really be compared to the ones from LGU. I just put them here for completeness.

Forum.hellug.gr – Posts per month:

Forum.hellug.gr – New members per month:

I won’t do any further comments…I’d be glad to see yours though 🙂

P.S. I know that I could have used some form of database to store and process the results of those commands but I wanted to keep it as simple as possible.

Filed by kargig at 17:00 under Linux
Tags: forum, forum.hellug.gr, HELLUG, LGU, Linux, linux greek users, oneliner, statistics
5 Comments | 183,186 views