Spam in blogs

For blogs that are built only for spamming, see Spam blog.

Spam in blogs (also called simply blog spam or comment spam) is a form of spamdexing. It is done by automatically posting random comments or promoting commercial services to blogs, wikis, guestbooks, or other publicly accessible online discussion boards. Any web application that accepts and displays hyperlinks submitted by visitors may be a target.

Adding links that point to the spammer's web site artificially increases the site's search engine ranking. An increased ranking often results in the spammer's commercial site being listed ahead of other sites for certain searches, increasing the number of potential visitors and paying customers.

History

This type of spam originally appeared in internet guestbooks, where spammers repeatedly fill a guestbook with links to their own site and no relevant comment to increase search engine rankings. If an actual comment is given it is often just "cool page", "nice website", or keywords of the spammed link.

In 2003, spammers began to take advantage of the open nature of comments in the blogging software like Movable Type by repeatedly placing comments to various blog posts that provided nothing more than a link to the spammer's commercial web site. Jay Allen created a free plugin, called MT-BlackList, for the Movable Type weblog tool (versions prior to 3.2) that attempted to alleviate this problem. Many current blogging packages now have methods of preventing or reducing the effect of blog spam, but spammers become smarter as well. Many of them use special blog spamming tools like Trackback Submitter to bypass comment spam protection on popular blogging systems like Movable Type, Wordpress and others.

Possible solutions

Blocking by keyword

This is simplest form of blocking, which yields very good results, because comment spam is targeted at bots, so it must be readable by simple software. A lot of spam can be blocked by banning names of popular pharmaceuticals and casino games.

The main problem with this approach is that spammers constantly find new ways to spell or hawk their goods, so this requires constant updating. For example, blocking "viagra" would cut down spam by a lot, but spammers will start spamming "vi@gra", "v1agr@", "vigra". There's also an uncountable number of goods spammers try to sell, making this system difficult to keep updated.

rel="nofollow"

In early 2005, Google announced that hyperlinks with rel="nofollow" attribute would not influence the link target's ranking in the search engine's index. The Yahoo and MSN search engines also respect this tag.

nofollow is a misnomer in this case since it actually tells a search engine "Don't score this link" rather than "Don't follow this link." This differs from the meaning of nofollow as used within a robots meta tag, which does tell a search engine: "Do not follow any of the hyperlinks in the body of this document."

Using rel="nofollow" is a much easier solution that makes the improvised techniques above irrelevant. Most weblog software now marks reader-submitted links this way by default (with no option to disable it without code modification). A more sophisticated server software could spare the nofollow for links submitted by trusted users like those registered for a long time, on a whitelist, or with a high karma. Some server software adds rel="nofollow" to pages that have been recently edited but omits it from stable pages, under the theory that stable pages will have had offending links removed by human editors.

Some weblog authors object to the use of rel="nofollow", arguing, for example, that

• Link spammers will continue to spam everyone to reach the sites that do not use rel="nofollow"
• Link spammers will continue to place links for clicking (by surfers), even if those links are ignored by search engines.
• Google is advocating the use of rel="nofollow" in order to reduce the effect of heavy inter-blog linking on page ranking.

Jeremy Zawodny has stated on his blog that

Worse, nofollow has another, more pernicious effect, which is that it reduces the value of legitimate comments.

Other websites like Slashdot, with high user participation, use improvised nofollow implementations like adding rel="nofollow" only for potentially misbehaving users. Potential spammers posing as users can be determined through various heuristics like age of registered account and other factors. Slashdot also uses the poster's karma as a determinant in attaching a nofollow tag to user submitted links.

rel="nofollow" has come to be regarded as a microformat.

The main issue with this solution is it does not stop spammers from spamming your site. If a spammer sends 5000 spam comments and you attach "nofollow" to all their links, the spammer gets no benefit, but you still have 5000 spam comments cluttering your blog.

Turing tests

Various methods requiring humans to do spamming by hand have been attempted. A variety of CAPTCHA gateways have been implemented, in an effort to prevent bots from submitting entries. Drawbacks to this are the annoyance it poses for regular users, and the ability of some advanced bots to defeat simple Captchas most of the time.

Disallowing links in posts

There is negligible gain from spam that does not contain links, so currently all spam posts contain (excessive number of) links. It is safe to require passing Turing tests only if post contains links and letting all other posts through. While this is highly effective, spammers do frequently send gibberish posts (such as "ajliabisadf ljibia aeriqoj" to test your spam filter. These gibberish posts will not be labeled as spam. They do the spammer no good, but they still clog up your comments sections.

Redirects

Instead of displaying a direct hyperlink submitted by a visitor, a web application could display a link to a script on its own website that redirects to the correct URL. This will not prevent all spam since spammers do not always check for link redirection, but effectively prevents against increasing their PageRank, just as rel=nofollow. An added benefit is that the redirection script can count how many people visit external URLs, although it will increase the load on the site.

Redirects should be server-side to avoid accessibility issues related to client-side redirects. This can be done via the .htaccess file in Apache.

Another way of preventing PageRank leakage is to make use of public redirection services such as TinyURL or My-Own.Net. For example,

<a href="http://my-own.net/alias_of_target" rel="nofollow" >Link</a>

where 'alias_of_target' is the alias of target address.

Services such as POW7.com offer a public redirection without the need to configure an alias. An example of a link to http://wikipedia.org/ on POW7 would be:

<a href="http://pow7.com/pr/http://wikipedia.org/">http://wikipedia.org/</a>

Again, the issue with this method is that while it removes the benefit the spammer is seeking, the users of this method are still left with a very high volume of spam that they must clean up or leave behind.

Distributed approaches

This approach is very new to addressing link spam. One of the shortcomings of link spam filters is that most sites only receive one link from each domain which is running a spam campaign. If the spammer varies IP addresses, there is little to no distiguishable pattern left on the vandalized site. The pattern, however, is left across the thousands of sites that were hit quickly with the same links.

A distributed approach, like the free LinkSleeve, uses XML-RPC to communicate between the various server applications (such as blogs, guestbooks, forums, and wikis) and the filter server, in this case LinkSleeve. The posted data is stripped of urls and each url is checked against recently submitted urls across the web. If a threshold is exceeded, a "reject" response is returned, thus deleting the comment, message, or posting. Otherwise, an "accept" message is sent.

A more robust distributed approach is Akismet, which uses a similar approach to LinkSleeve but uses API keys to assign trust to nodes and also has wider distribution as a result of being bundled with the 2.0 release of WordPress. They claim over 140,000 blogs contributing to their system. Akismet libraries have been implemented for Java, Python, Ruby, and PHP, but its adoption may be hindered by the requirement of an API key and its commercial use restrictions. No such restrictions are in place for Linksleeve.

Project Honey Pot has also begun tracking comment spammers. The Project uses its vast network of thousands of traps installed in over one hundred countries around the world in order to watch what comment spamming web robots are posting to blogs and forums. Data is then published on the top countries for comment spamming, as well as the top keywords and URLs being promoted. The Project's data is then made available to block known comment spammers through http:BL. Various plugins have been developed to take advantage of the http:BL API.

Application-specific anti-spam methods

Particularly popular software products such as Movable Type and MediaWiki have developed their own custom anti-spam measures, as spammers focus more attention on targeting those platforms. Whitelists and blacklists that prevent certain IPs from posting, or that prevent people from posting content that matches certain filters, are common defenses. More advanced access control lists require various forms of validation before users can contribute anything like linkspam.

The goal in every case is to allow good users to continue to add links to their comments, as that is considered by some to be a valuable aspect of any comments section.

RSS feed monitoring

Some wikis allow you to access an RSS feed of recent changes or comments. If you add that to your news reader and set up a smart search for common spam terms (usually viagra and other drug names) you can quickly identify and remove the offending spam.

Response tokens

Another filter available to webmasters is to add a hidden session token or hash function to their comment form. When the comments are submitted, data stored within the posting such as IP address and time of posting can be compared to the data stored with the session token or hash generated when the user loaded the comment form. Postings that use different IP addresses for loading the comment form and posting the comment form, or postings that took unusually short or long periods of time to compose can be filtered out. This method is particularly effective against spammers who spoof their IP Address in an attempt to conceal their identities.

Ajax

Some blog software such as Typo allow the blog administrator to only allow comments submitted via Ajax XMLHttpRequests, and discard regular form POST requests. This causes accessibility problems typical to Ajax-only applications.

Although this technique prevents spam so far, it is a form of security by obscurity and will probably be defeated if it becomes popular enough.

Switching off comments

Some bloggers have chosen to turn off comments because of the volume of spam.

See also

• Adversarial information retrieval
• Social networking spam

External links

• Latest protection technique against comment spam on Wordpress.
• Anti-spam Features of MediaWiki
• Article about latest spamming techniques on Search Engine Journal
• Six Apart Comment Spam Guide, fairly broad overview from Movable Type's authors.
• The (Evil) Genius of Comment Spammers, an article on link spam from Wired magazine.
• Gilad Mishne, David Carmel and Ronny Lempel: Blocking Blog Spam with Language Model Disagreement, PDF. From the First International Workshop on Adversarial Information Retrieval (AIRWeb'05) Chiba, Japan, 2005.
• The Solution To Spam Problems
• A Comprehensive Guide to Protecting Your Blog from Spam - a series of measures you can follow to making your WordPress Blog spamfree
• Spam Huntress The Norwegian Spam Huntress - Ann Elisabeth
• Anti Spam Articles. -Anti Spam Articles and lots of information.
• SecuriTeam Blogs Spam section Intensive technical posting by the Gadi Evron on blog spam techniques and counter-measures.
• SignedPing An open specification for blog security to combat spam.