Usenet
Usenet is a distributed Internet discussion system that evolved from a general purpose UUCP network of the same name. It was conceived by Duke University graduate students Tom Truscott and Jim Ellis in 1979. Users read and post email-like messages (called "articles") to a number of distributed newsgroups, categories that resemble bulletin board systems in most respects. The medium is distributed among a large number of servers, which store and forward messages to one another. Individual users download and post messages to a single server, usually operated by their ISP or university, and the servers exchange the messages between each other.
Introduction
Usenet is one of the oldest computer network communications systems still in widespread use. It was established in 1980 following experiments the previous year, over a decade before the World Wide Web was introduced and the general public was admitted to the Internet. It was originally conceived as a "poor man's ARPANET," employing UUCP to offer mail and file transfers, as well as announcements through the newly developed news software. This system, developed at University of North Carolina at Chapel Hill and Duke University, was called USENET to emphasize its creators' hope that the USENIX organization would take an active role in its operation (Daniel et al, 1980).
Today, almost all Usenet traffic is carried over the Internet. The current format and transmission of Usenet articles is very similar to that of Internet email messages. However, whereas email is usually used for one-to-one communication, Usenet is a one-to-many medium.
The articles that users post to Usenet are organized into topical categories called newsgroups, which are themselves logically organized into hierarchies of subjects. For instance, sci.math and sci.physics are within the sci hierarchy, for science. When a user subscribes to a newsgroup, the news client software keeps track of which articles that user has read.
When a user posts an article, it is initially only available on that user's news server. Each news server, however, talks to one or more other servers (its "newsfeeds") and exchanges articles with them. In this fashion, the article is copied from server to server and (if all goes well) eventually reaches every server in the network. The later peer-to-peer networks operate on a similar principle; but for Usenet it is normally the sender, rather than the receiver, who initiates transfers. Some have noted that this seems a monstrously inefficient protocol in the era of abundant high-speed network access. Usenet was designed for a time when networks were much slower, and not always available. Many sites on the original Usenet network would connect only once or twice a day to batch-transfer messages in and out.
Usenet is of significant cultural importance in the networked world, having given rise to, or popularized, many widely recognized concepts and terms such as "FAQ" and "spam". Today, Usenet has diminished in importance with respect to mailing lists and weblogs. The difference from mailing lists, though, is that Usenet requires no personal registration with the group concerned (subscription is necessary only to keep track of which articles one has already read, and that information need not be stored on a remote server), that archives are always available, and that reading the messages requires no mail client, but a news client (included in most modern e-mail clients).
ISPs, news servers, and newsfeeds
Most Internet service providers, and many other Internet sites, operate news servers for their users to access. ISPs that do not operate their own servers directly will often offer their users an account from another provider that specifically operates newsfeeds. Usually the ISP will get a kickback for referring the customer to the Usenet provider. In early news implementations, the server and newsreader were a single program suite, running on the same system. Today, one uses separate newsreader client software—a program that resembles an email client (and is often integrated with one) but accesses Usenet servers instead.
Not all ISPs run news servers. A news server is one of the most difficult Internet services to administer well because of the sheer amount of data involved. Some ISPs outsource news operation to specialist sites, which will usually appear to a user as though the ISP ran the server itself. Many sites carry a restricted newsfeed, with a limited number of newsgroups. Commonly omitted from such a newsfeed are foreign-language newsgroups and the alt.binaries hierarchy which largely carries software, music, videos and images and accounts for over 99 percent of the article data.
For those who have access to the Internet, but do not have access to a news server, Google Groups ([1]) allows reading and posting of text news groups via the World Wide Web. Though this or other "news-to-Web gateways" are not always as easy to use as specialized newsreader software—especially when threads get long—they are often much easier to search. Users who lack access to an ISP news server can use Google Groups to access the alt.free.newsservers newsgroup, which has information about open news servers.
There are also Usenet providers that specialize in offering service to users whose ISPs do not carry news, or that carry a restricted feed. One list of such providers is available at Gsorel list of Usenet providers or Jeremy Nixon's list of Usenet providers.
See also news server operation for an overview of how news systems are implemented.
Newsreader clients
Newsgroups are typically accessed with special client software that connects to a news server. With the rise of the world wide web, web front-ends have sometimes been used to access newsgroups via the aforementioned news-to-web gateways. However, these gateways often provide limited features, and for that reason using a local client is still regarded as the best way to access newsgroups.
Newsreader clients are available for all major operating systems and come in all shapes and sizes. Mail clients or "communication suites" also now commonly have an integrated newsreader. Often, however, these integrated clients are of low quality, e.g. incorrectly implementing Usenet protocols, standards and conventions. Many of these integrated clients, for example the one in Microsoft's Outlook Express, are disliked by purists because of their misbehavior.
Moderated and unmoderated newsgroups
A minority of newsgroups are moderated. That means that messages submitted by readers are not distributed to USENET, but instead are emailed to the moderators of the newsgroup, for approval. Moderated newsgroups have rules called charters. Moderators are persons whose job is to ensure that messages that the readers see in newsgroups conform to the charter of the newsgroup. Typically, moderators are appointed in the proposal for the newsgroup, and changes of moderators follow a succession plan.
The job of the moderator is to receive submitted articles, review them, and inject approved articles so that they can be properly propagated worldwide. Such articles must bear the Approved: header line.
Unmoderated newsgroups form the majority of USENET newsgroups, and messages submitted by readers for unmoderated newsgroups are immediately propagated for everyone to see.
Creation of moderated newsgroups often becomes a hot subject of controversy, raising issues regarding censorship and the desire of a subset of users to form an intentional community.
Technical details
Usenet is a set of protocols for generating, storing and retrieving news "articles" (which resemble Internet mail messages) and for exchanging them among a readership which is potentially widely distributed. These protocols most commonly use a flooding algorithm which propagates copies throughout a network of participating servers. Whenever a message reaches a server, that server forwards the message to all its network neighbors that haven't yet seen the article. Only one copy of a message is stored per server, and each server makes it available on demand to the (typically local) readers able to access that server. Usenet was thus one of the first peer-to-peer applications, although in this case the "peers" are themselves servers that the users then access, rather than the users themselves being peers on the network.
RFC 850 was the first formal specification of the messages exchanged by Usenet servers. It was superseded by RFC 1036.
One difference between Usenet and newer peer-to-peer applications is that one can request the automated removal of a posting from the whole network by creating a cancel message, although due to a lack of authentication and resultant abuse, this capability is frequently disabled. Copyright holders may still request the manual deletion of infringing material using the provisions of World Intellectual Property Organization treaty implementations, such as the U.S. Online Copyright Infringement Liability Limitation Act.
Organization
The major set of worldwide newsgroups is contained within nine hierarchies, eight of which are operated under consensual guidelines that govern their administration and naming. The current "Big Eight" are:
- comp.*: computer-related discussions (comp.software, comp.sys.amiga)
- misc.*: Miscellaneous topics (misc.education, misc.forsale, misc.kids)
- news.*: Discussions and announcements about news (meaning Usenet, not current events) (news.groups, news.admin)
- rec.*: Recreation and entertainment (rec.music, rec.arts.movies)
- sci.*: Science related discussions (sci.psychology, sci.research)
- soc.*: Social discussions (soc.college.org, soc.culture.african)
- talk.*: Talk about various controversial topics (talk.religion, talk.politics)
- humanities.*: Fine arts, literature, and philosophy (humanities.classics, humanities.design.misc)
(Note: the asterisks are used as wildmat patterns, examples follow in parentheses)
See also Great Renaming.
The alt.* hierarchy is not subject to the procedures controlling groups in the Big Eight, and it is as a result less organized. However, groups in the alt.* hierarchy tend to be more specialized or specific—for example, there might be a newsgroup under the Big Eight which contains discussions about children's books, but a group in the alt hierarchy may be dedicated to one specific author of children's books. Binaries are posted in alt.binaries.*, making it the largest of all the hierarchies.
Many other hierarchies of newsgroups are distributed alongside these. Regional and language-specific hierarchies such as japan.* and ne.* serve specific regions such as Japan and New England. Companies such as Microsoft administer their own hierarchies to discuss their products and offer community technical support. Some users prefer to use the term "Usenet" to refer only to the Big Eight hierarchies, others include alt as well. The more general term "netnews" incorporates the entire medium, including private organizational news systems.
Binary content
Usenet was originally created to distribute text content encoded in the 7-bit ASCII character set. With the help of programs that encode 8-bit values into ASCII, it became practical to distribute binary content. Binary posts, due to their size and dubious copyright status, were in time restricted to specific newsgroups, making it easier for administrators to allow or disallow the traffic.
The oldest widely used encoding method is uuencode, from the Unix uucp package. In the late 1980s Usenet articles were often limited to 60,000 characters, and larger hard limits exist today. Files are therefore commonly split into sections that require reassembly by the reader.
With the header extensions and the Base64 and Quoted-Printable MIME encodings, there was a new generation of binary transport. In practice, MIME has seen increased adoption in text messages, but it is avoided for most binary attachments. Some operating systems with metadata attached to files use specialized encoding formats. For Mac OS, both Binhex and special MIME types are used.
Other lesser known encoding systems that may have been used at one time were BTOA, XX encoding, BOO, and USR encoding.
In an attempt to reduce file transfer times, an informal file encoding known as yEnc was introduced in 2001. It achieves about a 30% reduction in data transferred by assuming that most 8-bit characters can safely be transferred across the network without first encoding into the 7-bit ASCII space.
History
The first newsgroup experiments occurred in 1979. Tom Truscott and Jim Ellis of Duke University came up with the idea as a replacement for a local announcement program, and established a link with nearby University of North Carolina using Bourne shell scripts written by Steve Bellovin. The public release of news was in the form of conventional compiled software, written by Steve Daniel and Truscott.
UUCP networks spread quickly due to the lower costs involved, and ability to use existing leased lines, X.25 links or even ARPANET connections. By 1983 the number of UUCP hosts had grown to 550, nearly doubling to 940 in 1984.
As the mesh of UUCP hosts rapidly expanded, it became desirable to distinguish the Usenet subset from the overall network. A vote was taken at the 1982 USENIX conference to choose a new name. The name Usenet was retained, but it was established that it only applied to news.[3] The name UUCPNET became the common name for the overall network.
In addition to UUCP, early Usenet traffic was also exchanged with Fidonet and other dial-up BBS networks. The Network News Transfer Protocol, or NNTP, was introduced in 1985 to distribute Usenet articles over TCP/IP as a more flexible alternative to informal Internet transfers of UUCP traffic. Since the Internet boom of the 1990s, almost all Usenet distribution is over NNTP, obsoleting the earlier dictum that "Usenet is not the Internet."
Early versions of Usenet used Duke's A News software. At Berkeley an improved version called B News was produced by Matt Glickman and Mark Horton. With a message format that offered compatibility with Internet mail and improved performance, it became the dominant server software. C News, developed by Geoff Collyer and Henry Spencer at the University of Toronto, was comparable to B News in features but offered considerably faster processing. In the early 1990s, InterNetNews by Rich Salz was developed to take advantage of the continuous message flow made possible by NNTP versus the batched store-and-forward design of UUCP. Since that time INN development has continued, and other news server software has also been developed.
Usenet was the initial Internet community and the place for many of the most important public developments in the commercial internet. It was the place where Tim Berners-Lee announced the launch of the World Wide Web [4], Marc Andreesen announced the creation of the Mosaic browser and the introduction of the image tag [5], which revolutionized the World Wide Web by turning it into a graphical medium.
Web-based archiving of Usenet posts began in 1995 at Deja News with a very large, searchable database. In 2001, this database was acquired by Google.
AOL announced that it would discontinue its integrated Usenet service in early 2005, citing the growing popularity of weblogs, chat forums and on-line conferencing. The AOL community had a tremendous role in popularizing the Usenet some 11 years earlier, with all of its positive and negative aspects. This change marked the end of the legendary Eternal September. Others, however, feel that Google Groups, especially with its new user interface, has picked up the torch that AOL has dropped – and that Eternal September has yet to end.
Over time, the amount of Usenet traffic has steadily increased. It is important to note, however, that much of this traffic increase reflects not an increase in discrete users or newsgroup discussions, but instead the combination of massive automated spamming and an increase in the use of .binaries newsgroups in which large files (frequently pornography or pirated media) are often posted publicly. A small sampling of the change (measured in feed size per day) follows:
| Daily Volume | Date | Source |
|---|---|---|
| 4.5 GB | 1996-12 | Altopia.com |
| 9 GB | 1997-07 | Altopia.com |
| 12 GB | 1998-01 | Altopia.com |
| 26 GB | 1999-01 | Altopia.com |
| 82 GB | 2000-01 | Altopia.com |
| 181 GB | 2001-01 | Altopia.com |
| 257 GB | 2002-01 | Altopia.com |
| 492 GB | 2003-01 | Altopia.com |
| 969 GB | 2004-01 | Altopia.com |
| 1.30 TB | 2004-09-30 | Octanews.net |
| 1.27 TB | 2004-11-30 | Octanews.net |
| 1.38 TB | 2004-12-31 | Octanews.net |
| 1.34 TB | 2005-01-01 | Octanews.net |
| 1.30 TB | 2005-01-01 | Newsreader.com |
| 1.67 TB | 2005-01-31 | Octanews.net |
| 1.63 TB | 2005-02-01 | Newsreader.com |
| 1.81 TB | 2005-02-28 | Octanews.net |
| 1.87 TB | 2005-03-08 | Newsreader.com |
| 2.00 TB | 2005-03-11 | Various sources |
Internet jargon and history
Many terms now in common use on the Internet—so-called "jargon"—originated or were popularized on Usenet. Likewise, many conflicts which later spread to the rest of the Internet, such as the ongoing difficulties over spamming, began on Usenet.
Archives and Web Interfaces
Google Groups / DejaNews
Google Groups hosts an archive of Usenet posts dating back to May 1981. The archive was originally started by a company called DejaNews (later only Deja), from which Google bought it at some point in time. A Usenet Timeline is provided in the Help section. Already during DejaNews times the archive quickly became popular and a constant in Usenet culture, and it still is.
However, there are two main issues people have with Google Groups and similar services:
- Fear of loss of privacy. An archive simplifies to profile people. This has partly been countered with the introduction of the non-standard X-No-Archive: Yes header, which is itself seen controversal.
- The web interface. See the following section.
The access on a raw plain text version of the original article was removed. The rather cryptic syntax to view an article available on Google Groups by its Message-ID can be simplified to
http://purl.net/net/msgid/without-angle-brackets@example.org
http://purl.net/net/msgid/4zCix009Cv2acya@bionic35.bionic.zer.de
Web Interfaces
With the rising popularity of the World Wide Web there happen to be many Web-to-Usenet gateways. These interfaces are seen controversial by Usenet users. Google Groups is usually used as the prime example to show what is wrong with Web-to-Usenet gateways, since it is the most popular and largest such service. However, the critisism also applies to many other such gateways in principle.
Since February 2001 Google Groups provides a web interface to Usenet newsgroups. It also allows the creation
of mailing lists like Yahoo! Groups, combining the technical deficits of both approaches.
One of many concerns that have been expressed about the Google interface is that novices may have difficulty realising that they are participating in a Usenet newsgroup rather than in a web forum hosted by Google. Google Groups is not very verbose about the fact and doesn't make it very clear in the web interface that some of the groups at Google Groups are Usenet groups, while others are local google-only groups.
Other concerns are
- Web-to-Usenet gateways provide a service for e-mail spammers, since a spammer's web spider can now also extract e-mail addresses from Usenet posting without any extra effort. Before such gateways an e-mail spammer would have to use a separate tool and gain access to a news feed (as Usenet spammers do). Since 2005 Google Groups tries to prevent e-mail address harvesting by scrambling the display of e-mail addresses on their web pages.
- Web-to-Usenet gateways often hide the fact from users that they are actually on Usenet, and that it would be a good idea to follow Usenet customs and established rules. They further hide the fact that Usenet is still at its core a decentralized store-and-forward system and that therefore articles don't magically appear "everywhere" once they have been posted. this typically leads to multiple posts with the same contents and/or whining about why there are no answers within minutes and/or why there are multiple answers wihich essentially say the same.
- Web-to-Usenet gateways often cut away or hide "overhead" information, e.g. header information like message IDs, or - even worse - signatures. This leaves Web-users puzzled about what people are talking about when they write things like "See my sig" or "See msgid ...".
- Web-to-Usenet gateways typically provide less features than real News Reader software. E.g. the ability to filter (on users, subject lines, etc.), to sort threads in multiple ways, to draft articles, etc. is typically missing.
- Web-to-Usenet gateways are often very badly policed. These days this is not only typical for such gateways, but also for many feed providers. However, abusive behavior from e.g. Google Groups users in some newsgroups is now legendary, and Google can't muster the will and resources to effectively keep a lid on abusers. Google Groups provides an automated complaint system which appears to never result in any action.
On the positive side
- Web-to-Usenet gateways lower the entry barrier to Usenet. This is, of cource, sometimes seen as negative by Usenet users, since it enables less technicaly savy people to enter Usenet.
- Web-to-Usenet gateways are more colorful
See also
This article is partly based on the infoAnarchy wiki.
Usenet terms
- backbone cabal
- Breidbart Index
- FAQ
- flaming and flame war
- Flood aka flooders and flooding
- FWAK
- Godwin's Law
- It's always September
- kill file
- list of newsgroups
- MSTing
- scorefile
- Skitt's law
- sock puppet
- sporgery
- Trolling
- There is no Cabal
- Usenet Death Penalty
- Usenet cabal
- Wackyparsing
- X-No-Archive
- Your mileage may vary
Usenet history
Usenet administrators
There are no Usenet "administrators" per se; each server administrator is free to do whatever pleases him or her as long as the users and peers tolerate and accept it. Nevertheless, there are a few famous administrators:
Usenet personalities
References
- Stephen Daniel, James Ellis, and Tom Truscott (1980). USENET - A General Access UNIX® Network. (inside archive as usenet/uprop.n)
- Bruce Jones, archiver (1997). USENET History mailing list archive covering 1990-1997.
Further reading
- Michael Hauben, Ronda Hauben, and Thomas Truscott (1997-04-27). Netizens : On the History and Impact of Usenet and the Internet (Perspectives). Wiley-IEEE Computer Society P. ISBN 0818677066.
{{cite book}}: Check date values in: |year= (help)CS1 maint: multiple names: authors list (link)
- Bryan Pfaffenberger (1994-12-31). The USENET Book: Finding, Using, and Surviving Newsgroups on the Internet. Addison Wesley. ISBN 020140978X.
{{cite book}}: Check date values in: |year= (help)
- Kate Gregory, Jim Mann, Tim Parker, and Noel Estabrook (June 1995). Using Usenet Newsgroups. Que. ISBN 0789701340.
{{cite book}}: CS1 maint: multiple names: authors list (link)
- Mark Harrison (July 1995). The USENET Handbook (Nutshell Handbook). O'Reilly. ISBN 1565921011.
- Henry Spencer, David Lawrence (January 1998). Managing Usenet. O'Reilly. ISBN 1565921984.
- Don Rittner (June 1997). Rittner's Field Guide to Usenet. MNS Publishing. ISBN 0937666505.
- Konstan, J., Miller, B., Maltz, D., Herlocker, J., Gordon, L., and Riedl, J. (March 1997). "GroupLens: applying collaborative filtering to Usenet news". Communications of the ACM. 40 (3): 77–87. doi:10.1145/245108.245126.
{{cite journal}}: CS1 maint: multiple names: authors list (link)
- Miller, B., Riedl, J., and Konstan, J. (January 1997). "Experiences with GroupLens: Making Usenet useful again" (PDF). Proceedings of the 1997 Usenix Winter Technical Conference.
{{cite conference}}: Unknown parameter |booktitle= ignored (|book-title= suggested) (help)CS1 maint: multiple names: authors list (link)
- "20 Year Usenet Timeline". Google. Retrieved 2006-06-27.
External links
- Usenet information, software, and service providers at Curlie
- IETF working group USEFOR (USEnet article FORmat)
- son-of-1036 (287 KB, historical RFC 1036 bis draft)
- [6] (Netscan - Usenet Social Accounting Reporting Tool)