User:Imsamurai/Asmbot

From Wikipedia, the free encyclopedia
Jump to: navigation, search
Asmbot
Original author(s) Associative Search Engine
Type Web crawler

ASM bot is the search bot software used by ASM indexation engine, which collects documents from the web to build a searchable index. ASM-based products and projects uses this bot as a main crawling machine.

ASM bot requests to Web servers are identifiable by a user-agent string containing "ASM" or “ASM bot” and a host address containing URL or the project or product that uses ASM technology. For example:

“Mozilla/5.0 (compatible; ASM 2; +http://recipe-finder.com,+http://povaru.com)”

for http://recipe-finder.com and http://povaru.com projects.

ASM bot follows the HREF and SRC links as well as can recognize some URLs from textual content of web page and extended HTML tags attributes. It discovers pages by harvesting all of the links and URLs on every page it finds and succeeds with base domain name and path. Then it follows these links to other web pages. New web pages must be linked to other known pages on the web in order to be crawled and indexed.

More detailed information about ASM bot behavior depends on the project or product that uses ASM engine and technology and can be getting from project’s documentation or description source.

Category:Bots