Domain Name System blocklist

A DNS Blacklist, or DNSBL (definition below), is a means by which an Internet site may publish a list of IP addresses that some people may want to avoid and in a format which can be easily queried by computer programs on the Internet. The technology is built on top of the Internet Domain Name System, or DNS. DNSBLs are chiefly used to publish lists of addresses linked to spamming. Most mail transport agent (mail server) software can be configured to reject or flag messages which have been sent from a site listed on one or more such lists.

DNSBL names a medium, not any specific list or policy. There has been a good deal of controversy over the past several years over the operation of specific lists, such as the MAPS RBL and SPEWS.

Terminology

The following are all closely related terms:

• RBL is an abbreviation for "Real-time Blackhole List". As mentioned below, "RBL" was the name of the first system to use this technology, a proprietary MAPS DNSBL, and "RBL" is a registered trademark. Some pieces of mail software have configuration parameters that use "RBLs" or "RBL domains" when any DNSBLs can be used, not just the MAPS RBL.

• DNSBL is an abbreviation that sometimes stands for "DNS blacklist", although different DNSBL operators define the term in various ways. The use of the word "blacklist" is somewhat controversial. The reasons cited include its association with Joseph McCarthy and legal liability. Instead, some people have suggested that DNSBL should stand for "DNS blocklist" even though DNSBLs are not always used for direct blocking, or "DNS blackhole list" based on the RBL expansion, even though the DNSBL method does not create true blackholes. A minimally controversial expansion of the acronym is "DNS-Based List".

• DNSWL is an abbreviation for "DNS whitelist". It is a list of IP addresses that some people may want to treat more favourably.

• RHSBL is an abbreviation for "Right Hand Side Blacklist". This is similar to a DNSBL but it lists domain names rather than IP addresses. The term comes from the "right-hand side" of an email address — the part after the final @ sign — which clients look up in the RHSBL.

• URIBL is an abbreviation for "Uniform Resource Identifier Blacklist". A URIBL lists domain names and IP addresses that appear in URIs such as web sites mentioned in message bodies. It contrasts with an RHSBL which lists domain names used in e-mail addresses.

History of DNSBLs

The first DNSBL was the Real-time Blackhole List (RBL), created in 1997 by Paul Vixie as part of his Mail Abuse Prevention System (MAPS); Dave Rand at Abovenet was its first subscriber. Initially, the RBL was not a DNSBL, but rather a list of networks transmitted via BGP to routers owned by subscribers so that network operators could blackhole all TCP/IP traffic for machines used to send spam or host spam supporting services, such as a website.

The purpose of the RBL was not simply to block spam—it was to educate Internet service providers and other Internet sites about spam and related problems, such as open SMTP relays, spamvertising, etc. Before an address would be listed on the RBL, volunteers and MAPS staff would attempt repeatedly to contact the persons responsible for it and get its problems corrected. Such effort was considered very important before blackholing all network traffic, but it also meant that spammers and spam supporting ISPs could delay being put on the RBL for long periods while such discussions went on.

Later, the RBL was also released in a DNSBL form and Paul Vixie encouraged the authors of sendmail and other mail software to implement RBL clients. These allowed the mail software to query the RBL and reject mail from listed sites on a per mail server basis instead of blackholing all traffic.

Soon after the advent of the RBL, others started developing their own lists with different policies. One of the first was Alan Brown's Open Relay Behavior-modification System (ORBS). This used automated testing to discover and list mail servers running as open mail relays—exploitable by spammers to carry their spam. ORBS was controversial at the time because many people felt running an open relay was acceptable, and that scanning the Internet for open mail servers could be abusive.

In 2003, a number of DNSBLs have come under denial-of-service attacks. Since no party has admitted to these attacks nor been discovered responsible, their purpose is a matter of speculation. However, many observers believe the attacks are perpetrated by spammers in order to interfere with the DNSBLs' operation or hound them into shutting down. In August 2003, the firm Osirusoft, an operator of several DNSBLs including one based on the SPEWS data set, shut down its lists after suffering weeks of near-continuous attack.

URI DNSBLs

A URI DNSBL is a DNSBL that lists the domain names and IP addresses which are found in the "clickable" links contained in the body of spams, but generally not found inside legitimate messages.

URI DNSBLs were created when it was determined that much spam made it past spam filters during that short time frame between the first use of a spam-sending IP address and the point where that sending IP address was first listed on major sending-IP-based DNSBLs.

In many cases, such elusive spams contain in their links domain names or IP addresses (collectively referred to as a URIs) where that URI was already spotted in previously caught spam and where that URI is not found in non-spam e-mail.

Therefore, when a spam filter extracts all URIs from a message and checks them against a URI DNSBL, then the spam can be blocked even if the sending IP for that spam has not yet been listed on any sending IP DNSBL.

Of the three major URI DNSBLs, the oldest and most popular is SURBL, created and operated primarily by Jeff Chan. After SURBL was created, some of the administrators and contributors to SURBL started the second major URI DNSBL, URIBL. More recently, another current and long-time SURBL administrator, Rob McEwen, started the third major URI DNSBL, ivmURI.

URI DNSBLs are often confused with RHSBLs (Right Hand Side BLs). But they are different. A URI DNSBL lists domain names and IPs found in the body of the message. An RHSBL lists the domain names used in the "from" or "reply-to" e-mail address. RHSBLs are not very effective because most spams either use forged "from" addresses or use "from" addresses containing popular freemail domain names, such as @gmail.com, @yahoo.com, or @hotmail.com addresses. In contrast to marginally effective and not-often-used RHSBLs, URI DNSBLs are very effective and are used by the majority of spam filters.

DNSBL operation

To operate a DNSBL requires three things: a domain to host it under, a nameserver for that domain, and a list of addresses to publish.

It is possible to serve a DNSBL using any general-purpose DNS server software. However this is typically inefficient for zones containing large numbers of addresses, particularly DNSBLs which list entire Classless Inter-Domain Routing netblocks. DNSBL-specific software — such as Michael J. Tokarev's rbldnsd, Daniel J. Bernstein's rbldns, or the DNS Blacklist Plug-In for Simple DNS Plus — is faster, uses less memory, and is easier to configure for this purpose.

The hard part of operating a DNSBL is populating it with addresses. DNSBLs intended for public use usually have specific, published policies as to what a listing means, and must be operated accordingly to attain or sustain public confidence.

DNSBL queries

When a mail server receives a connection from a client, and wishes to check that client against a DNSBL (let's say, dnsbl.example.net), it does more or less the following:

1. Take the client's IP address—say, 192.168.42.23—and reverse the order of octets, yielding 23.42.168.192.
2. Append the DNSBL's domain name: 23.42.168.192.dnsbl.example.net.
3. Look up this name in the DNS as a domain name ("A" record). This will return either an address, indicating that the client is listed; or an "NXDOMAIN" ("No such domain") code, indicating that the client is not.
4. Optionally, if the client is listed, look up the name as a text record ("TXT" record). Most DNSBLs publish information about why a client is listed as TXT records.

Looking up an address in a DNSBL is thus similar to looking it up in reverse-DNS. The differences are that a DNSBL lookup uses the "A" rather than "PTR" record type, and uses a forward domain (such as dnsbl.example.net above) rather than the special reverse domain in-addr.arpa.

There is an informal protocol for the addresses returned by DNSBL queries which match. Most DNSBLs return an address in the 127.0.0.0/8 IP loopback network. The address 127.0.0.2 indicates a generic listing. Other addresses in this block may indicate something specific about the listing—that it indicates an open relay, proxy, spammer-owned host, etc. For details see an Internet draft by the Anti-Spam Research Group.

URI DNSBL

An URI DNSBL query (and an RHSBL query) is fairly straightforward. Just prepend the domain name to the DNS list host as follows:

example.net.dnslist.com

Generally if an A record is returned the name is listed.

DNSBL policies

Different DNSBLs have different policies. DNSBL policies differ from one another on three fronts:

• Goals. What does the DNSBL seek to list? Is it a list of open-relay mail servers or open proxies—or of IP addresses known to send spam—or perhaps of IP addresses belonging to ISPs that harbor spammers?
• Nomination. How does the DNSBL discover addresses to list? Does it use nominations submitted by users? Spam-trap addresses or honeypots?
• Listing lifetime. How long does a listing last? Are they automatically expired, or only removed manually? What can the operator of a listed host do to have it delisted?

Implementation Issues

No DNSBL is perfect. All contain some false positives and all are incomplete. The more DNSBLs you check the more spam you will catch, but the more false positives you will have. Thus one should be careful in how DNSBLs are implemented.

DNSBLs can be used in rule based spam blocking software like Spamassassin where different black list are given point scores that can be mitigated by white rules to reduce false positives. They can also be used by MTAs like Exim, Sendmail and Postfix to outright block email if the senders IP address or host name is listed in a DNSBL. This is more dangerous unless one takes precautions to guard against false positives.

One way to do this is to first check white lists and pass the email if the server is on a white list. A technique developed by Junk Email Filter uses Yellow Lists and NoBL lists to mitigate the false positives that often occur when using multiple black lists. Yellow lists are host names and IP addresses of servers that are known to be a source of mixed spam and non-spam. Examples would be yahoo, hotmail, and gmail. If the forward confirmed rDNS FCrDNS resolves to one of these hosts then the IP address contains no information as to if the message is or isn't spam. Thus other DNSBL tests should not occur because the IP carries no useful information.

Here's a list of different kinds of DNS Lists.

White List - IP is a trusted email source
Black List - 100% spam. Message should be rejected.
Yellow List - Message comes from a mixed source and IP should not be tested further
NoBL list - IP is not a spam source but may be a ham source

Messages should first be checked for yellow listing. If listed, then no further checking is needed. Then the message should be white list tested. If it is found listed in a white list, the message should be accepted as good. Then the NoBL lists should be tested. If listed then black list tests should be bypassed. Finally the black list testing should apply. The message is rejected if it is found in a black list. To be safe one might score the blacklists and reject if listed on multiple lists.

Criticisms

Email users who find their messages blocked from mail servers that use DNSBLs often object vociferously, sometimes to the extent of attacking the existence of the lists themselves. The following lists are controversial:

• Lists of dynamic or dial-up IP addresses. These lists most often also include "static" ADSL addresses, thus "end-user IP addresses" might have been a better description. Some mail sites choose not to accept messages from such addresses, since they are often home computers exploited by spammer viruses.
  This can inconvenience SOHO users who wish to run their own mail servers on residential ISP connections or local MTAs on laptops for example. To work around this, users who wish to run their own mail server are expected to relay via their ISP's mail server using the smart host functionality built-in to most mail servers.
• Lists that include "spam-support operations", such as MAPS RBL. A spam-support operation is a site that may not directly send spam, but provides commercial services for spammers, such as hosting of Web sites that are advertised in spam. Refusal to accept mail from spam-support operations is intended as a boycott to encourage such sites to cease doing business with spammers, at the expense of inconveniencing non-spammers who use the same site as spammers.
• Predictive ("early warning") lists, notably SPEWS. SPEWS lists addresses belonging to spam-support operations, under the hypothesis that such addresses are more likely to send spam in the future. SPEWS "escalates" listings, increasing the size of the netblock listed, as a site continues to support spam.
• Some lists have unclear listing criteria and delisting may not happen automatically nor quickly. A few DNSBL operators will request payment (e.g. uceprotect.net[1]) or donation (e.g. SORBS). Some of the many listing/delisting policies can be found in the Comparison of DNS blacklists article.

Although many have voiced objections to specific DNSBLs, few people object to the principle that mail-receiving sites should be able to reject undesired mail systematically. One who does is John Gilmore, who deliberately operates an open mail relay. Gilmore accuses DNSBL operators of violating antitrust law.

For Joe Blow to refuse emails is legal (though it's bad policy, akin to "shooting the messenger"). But if Joe and ten million friends all gang up to make a blacklist, they are exercising illegal monopoly power. [2]

A number of parties, such as the Electronic Frontier Foundation and Peacefire, have raised concerns about some use of DNSBLs by ISPs. One joint statement issued by a group including EFF and Peacefire addressed "stealth blocking", in which ISPs use DNSBLs or other spam-blocking techniques without informing their clients. [3]

Spammers have pursued lawsuits against DNSBL operators on similar grounds:

• In 2003, a newly-formed corporation calling itself "EmarketersAmerica" filed suit against a number of DNSBL operators in Florida court. Backed by spammer Eddy Marin, the company claimed to be a trade organization of "email marketers" and that DNSBL operators Spamhaus and SPEWS were engaged in restraint of trade. The suit was eventually dismissed for lack of standing. [4]
• In 2006 a US court ordered Spamhaus to pay $11,715,000 in damages to the spammer "e360 Insight LLC". The order was a default judgment, as Spamhaus (which is a UK operation, outside the court's jurisdiction) did not defend itself. See e360 Lawsuit. This decision was later overturned by an Appeal Court

See also

• Comparison of DNS blacklists

External links

blacklists at Curlie

• Blacklists Compared - List of Compared Blacklists from 2001 to Current times
• How to Create a DNSBL - Tutorial on how to create a DNSBL (DNS Black List)
• Multiple DNSBL lookup online tool