Wandering in Fields of Idiots

My Blog

Search Engine Load on Servers

| Comments

How odd, I was looking at the statistics for this web site and I see that I get absolutely hammered by the search engines:

Top Total User Agents
#HitsUser Agent
1 44502 35.25% Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
2 32571 25.80% Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
3 14190 11.24% Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
4 6888 5.46% Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.
5 1481 1.17% Yandex/1.01.001 (compatible; Win16; I)
6 1443 1.14% Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)

So, this means that 101075 hits are from search engines out of the total 126256 hits or that search engines make up exactly 80% of the traffic to this site. Nice that Yahoo and Google love me so much but the 1GB of content they downloaded over 1GB of content. Would be nice if they could tone it down a bit. I’ve already adjusted my crawl rate on the engines but I seem to have missed Yahoo and now have made some efforts there to correct that.