BotSeer
BotSeer was a Web-based information system and search tool that provides resources and services for research on Web robots and trends in Robot Exclusion Protocol deployment and adherence. It was created and designed by Yang Sun[1], Isaac G. Councill[2], Ziming Zhuang[3] and C. Lee Giles.
BotSeer provided three major services including robots.txt searching, robot bias analysis[4], and robot-generated log analysis. The prototype of BotSeer also allowed users to search six thousand documentation files and source codes from 18 open source crawler projects. BotSeer served as a resource for studying the regulation and behavior of Web robots as well as information about the creation of effective robots.txt files and crawler implementations. It was publicly available on the World Wide Web at the College of Information Sciences and Technology at the Pennsylvania State University. BotSeer had indexed and analyzed 2.2 million robots.txt files obtained from 13.2 million websites, as well as a large Web server log of real-world robot behavior and related analysis. BotSeer's goals were to assist researchers, webmasters, web crawler developers and others with web robots related research and information needs.
BotSeer has also had set up a honeypot[5] to test the ethicality, performance and behavior of web crawlers.
However, some people did not receive BotSeer too well.[6]
Currently, BotSeer is inactive.
Contents |
[edit] See also
[edit] Notes
- ^ Yang Sun
- ^ Isaac G. Councill
- ^ Ziming Zhuang
- ^ Yang Sun, Z. Zhuang, I. Councill, C.L. Giles, "Determining Bias to Search Engines from Robots.txt," Proceedings of IEEE/WIC/ACM International Conference on Web Intelligence (WI 2007), 149-155, 2007.
- ^ Web Robot Behavior and Performance Test at the Wayback Machine (archived December 22, 2008) (instead of unrelated current site)
- ^ BotSeer? - SEO Best Practices Search Engine Forums
[edit] References
- Associated Press (November 28, 2007). "Webmasters May Shape Search Results". Newsvine. http://www.newsvine.com/_news/2007/11/28/1128150-webmasters-may-shape-search-results. Retrieved 2011-12-11.
- "Google favored by Web admins". Network World. November 15, 2007. http://www.networkworld.com/news/2007/111507-google-favored.html. Retrieved 2007-12-19.
[edit] External links
- (just for display, it is not active) BotSeer: Robots.txt and Web Crawler Search Engine at the Wayback Machine (archived February 8, 2010)
