||This article may require cleanup to meet Wikipedia's quality standards. The specific problem is: reads like a how to article. (March 2012)|
Mediabot visits those pages running AdSense ads that have not blocked its access via a robots.txt file and it's a Google recommendation that webmasters specifically add a command to their robots.txt file granting Mediabot access to the entire site. Here is how to do it:
User-agent: Mediapartners-Google* Disallow:
The Mediabot identifies itself with the user agent string "Mediapartners-Google/2.1".
The Mediabot revisits pages on a regular, but unpredictable basis. Changes made to a page therefore do not immediately cause changes to the ads displayed on the page. Note that ads can still be shown on a page even if the Mediabot has not yet visited it, in which case the ads chosen will be based on the overall theme of the other pages on the site. If no ads can be chosen, public service announcements are displayed instead.
You can keep some parts of the text from being crawled using:
-- google_ad_section_end -->
|This World Wide Web–related article is a stub. You can help Wikipedia by expanding it.|
|This software article is a stub. You can help Wikipedia by expanding it.|