PGTS Journal Edition 0009
Robots, spiders and other crawlers - who are they?
Robots, Spiders and Other Crawlers
This edition of the PGTS Journal, is very late. This was partly due to
the unexpected success of a previous article. Several months' ago I wrote an
article which summarised the Long Sad
History of Agent Strings. At the time I had difficulty discovering an
accurate list of agent strings and agents. I am sure that such lists are out
there. However, they are difficult to find amongst the millions of published
server logs. So I published my own lists. Thanks to some other sites linking
to those items, the rankings of those lists and the article, has now
increased such that they stand a little above the herd of server logs. And
so, it seems, in the few months, the lists have been discovered. This
prompted me to do some more research on robots. At the same time, I thought
that I would tackle the problem of camouflaged robots (or crawlers).
Unfortunately this necessitated re-writing the scripts which calculate hit
statistics and I also had to devise a cut-over for the new system.
The cut-over should be fully tested by the 15th of December And I will post
the new articles then. The new agent string lists can be accessed via the Agent String