PGTS Journal Edition 0009
December, 2002

Robots, Spiders and Other Crawlers

This edition of the PGTS Journal, is very late. This was partly due to the unexpected success of a previous article. Several months' ago I wrote an article which summarised the Long Sad History of Agent Strings. At the time I had difficulty discovering an accurate list of agent strings and agents. I am sure that such lists are out there. However, they are difficult to find amongst the millions of published server logs. So I published my own lists. Thanks to some other sites linking to those items, the rankings of those lists and the article, has now increased such that they stand a little above the herd of server logs. And so, it seems, in the few months, the lists have been discovered. This prompted me to do some more research on robots. At the same time, I thought that I would tackle the problem of camouflaged robots (or crawlers). Unfortunately this necessitated re-writing the scripts which calculate hit statistics and I also had to devise a cut-over for the new system. The cut-over should be fully tested by the 15th of December And I will post the new articles then. The new agent string lists can be accessed via the Agent String Switchboard