Removing robots from your web traffic analysis

Sandra Henry-Stocker, ITworld.com

The top visitors to web sites are often robots, crawlers and spiders -- the programs that run on systems whose job it is to index the contents of the web so that the rest of us can effectively search it. And those robots can so distort web traffic reports as to make them practically useless. Here are some steps you can take to identify robotic traffic in your web log files.