PGTS Blog Archive
|Gerry Patterson. The world's most humble blogger|
|If at first you do not succeed ... You must be a programmer!|
Chronogical Blog Entries:
Date: Sat, 22 Jan 2005 18:05:38 +1100
And while I was wondering about Websense and the strange "Network Errors" I thought I'd try those websites again.
I tried www.biblegateway.com a second time about an hour after my first attempt and the response was:
This site is not categorized by WebsenseAfter another five hours this changed to:
Also, when I tested www.danbyrnes.com.au Websense said:
Educational MaterialsIt appears that "Network Errors" is just a generic answer that Websense returns when it has trouble locating a site, that has not been categorised. In the case of www.danbyrnes.com, I have access to the logfiles, so I did a little digging to try and figure out was going on.
I searched for evidence of Sqworm in the current www.danbyrnes.com.au logfile, but I could not find it.
However, I did find something interesting. It appears that Dan's site has been visited by the Konqueror Morphbot. This is a strange beast which constantly changes its' agent string. Earlier on when I was looking at robots and agent strings, I noticed this and I deemed this particular behaviour to be suspicious. It seems to me that an honest net denizen would not behave in this manner. And so this Morphbot was categorised as a suspicious robot in my database.
Now, it transpires that the Konqueror morphbot comes from Websense! I had observed this earlier, but failed to make the connection ...
I should explain that Morphbot is a name that I have given to robots that cloak themselves with a valid agent_string, and change that garmet on successive hits. For example they might hit with agent_strings like the following:
Mozilla/5.0 (compatible; Konqueror/3.1-rc5; i686 Linux; 20020821) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312460) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312466) Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312468) Mozilla/5.0 (compatible; Konqueror/3.0-rc1; i686 Linux; 20021016)(These are actual strings sent by Websense Morphbots)
This is probably designed to fool sites that employ browser sniffing.
The Websense MSIE morphbots tend to get lost amongst the general clutter of MSIE junkbots and spambots. In fact, most susbots tend to cloak as MSIE. Now that there is a new kid on the block, we can expect to see Firefox Morphbots before long.
So there is more to the Websense story than I initially thought. According to whois, Websense owns the netblock 66.194.6/24. (more about this in the PGTS feedback column). www.danbyrnes.com.au has received 56 hits from Websense in the month of January all of them Morphbots, either Konqueror or MSIE.
From the limited testing that I have done, it appears that the "Network Errors", and "Not Categorised" categories may sometimes be interchangeable.
In the case of biblegateway.com, the first request returned as "Network Errors" and the second request returned as "Not Categorised".
In the case of danbyrnes.com.au, there is a chance that I may have mis-typed this URL when I was testing it.
In any case both of them have categories now. This behaviour seems consistent with what I have seen of some corporations that I know of who run websense. All of which would suggest to me that Websense are struggling to keep up with the workload. And sometimes they may just cover this with the message "Network Errors".
I suspect the main problem is that their search engine does not have enough grunt ... or enough smarts.