RobotStats: the Crawlers Tracker - Tracking Spiders Script
Have you ever wanted to know when and on which pages the search engine robots were crawling your web site? Ok, I see the old techy smiling :). Yes you can always dig down your log files, use some third party statistic web site. But why not doing it on your own? Try the following web application. We just installed it on DRListings.com. So, we now have a tool to figure out how the search engines are crawling our web site and how often they will crawl the same pages and some new pages. We hope it will help us optimize our web site to better serve our clients.
RobotStats: the Crawlers Tracker - Tracking Spiders Script