Simple Mail Transfer Protocol: Difference between revisions

Content deleted Content added

Inline

Revision as of 20:35, 6 February 2008

Simple Mail Transfer Protocol (SMTP) is the de facto standard for e-mail transmissions across the Internet. Formally SMTP is defined in RFC 821 (STD 10) as amended by RFC 1123 (STD 3) chapter 5. The protocol used today is also known as ESMTP and defined in RFC 2821.

Description

SMTP is a relatively simple, text-based protocol, in which one or more recipients of a message are specified (and in most cases verifiMail Transport Agents) can act as an SMTP client.

An email client knows the outgoing mail SMTP server from its configuration. A relaying server typically determines which SMTP server to connect to by looking up the MX (Mail eXchange) DNS record for each recipient's domain name (the part of the email address to the right of the at (@) sign). Conformant MTAs (not all) fall back to a simple A record in the case of no MX. Some current mail transfer agents will also use SRV records, a more general form of MX, though these are not widely adopted. (Relaying servers can also be configured to use a smart host.)

The SMTP client initiates a TCP connection to server's port 25 (unless overridden by configuration). It is quite easy to test an SMTP server using the telnet program (see below).

SMTP is a "push" protocol that does not allow one to "pull" messages from a remote server on demand. To do this a mail client must use POP3 or IMAP. Another SMTP server can trigger a delivery in SMTP using ETRN.

History

Forms of one-to-one electronic messaging were used in the 1960s. People communicated with one another using systems developed for a particular mainframe computer. As more computers began to be interconnected with others, especially in the US Government's ARPANET, standards were developed to allow users using different systems to be able to email one another. SMTP grew out of these standards developed during the 1970s.

SMTP can trace its roots to the Mail Box Protocol (ca. 1971), FTP Mail (ca. 1973),^[1] and Mail Protocol.^[2] The work continued throughout the 1970s, until the ARPANET converted into the modern Internet around 1980. Jon Postel then proposed a Mail Transfer Protocol in 1980 that began to remove the mail's reliance on FTP.^[3] SMTP was published as RFC 821 in August 1982, also by Jonathan Postel.[1]

The SMTP standard was developed around the same time the Usenet was, a one-to-many communication network with some similarities.

SMTP became widely used in the early 1980s. At the time, it was a complement to UUCP (Unix to Unix CoPy) mail, which was better suited to handle e-mail transfers between machines that were intermittently connected. SMTP, on the other hand, works best when both the sending and receiving machines are connected to the network all the time. Both use a store and forward mechanism and are examples of push technology. Usenet's newsgroups are still propagated with UUCP between servers^[4], but UUCP mail has virtually disappeared^[5] along with the "bang paths" it used as message routing headers.

The article about sender rewriting contains technical background info about the early SMTP history and source routing before RFC 1123.

Sendmail was one of the first (if not the first) mail transfer agents to implement SMTP. Some other popular SMTP server programs include Postfix, qmail, Novell GroupWise, Exim, Novell NetMail, Microsoft Exchange Server and Sun Java System Messaging Server. As of 2001 there were at least 50 programs that implemented SMTP either as clients (senders of messages) or as servers (receivers of messages).

Message Submission (RFC 2476) and SMTP-AUTH (RFC 2554) were introduced in 1998 and 1999, both describing new trends in email delivery. Originally, SMTP servers were typically internal to an organization, receiving mail for the organization from the outside, and relaying messages from the organization to the outside. But as time went on, SMTP servers (Mail transfer agents), in practice, were expanding their roles to become Mail submission agents for Mail user agents, some of which were now relaying mail from the outside of an organization. (e.g. A company executive wishes to send email while on a trip using the corporate SMTP server.) This issue, a consequence of the rapid expansion and popularity of the World Wide Web, meant that the SMTP protocol had to include specific rules and methods for relaying mail and authenticating users to prevent abuses such as unsolicited email (spam) relaying.

Since this protocol started out as purely ASCII text-based, it did not deal well with binary files. Standards such as Multipurpose Internet Mail Extensions (MIME) were developed to encode binary files for transfer through SMTP. MTAs developed after Sendmail also tended to be implemented 8-bit-clean, so that the alternate "just send eight" strategy could be used to transmit arbitrary data via SMTP. Non-8-bit-clean MTAs today tend to support the 8BITMIME extension, permitting binary files to be transmitted almost as easily as plain text.

Developers

Many people edited or contributed to the core SMTP specifications, among them Jon Postel, Eric Allman, Dave Crocker, Ned Freed, Randall Gellens, John Klensin, and Keith Moore.

Outgoing mail SMTP server

An email client requires the name or the IP address of an SMTP server as part of its configuration. The server will deliver messages on behalf of the user. This setting allows for various policies and network designs. End users connected to the Internet can use the services of an e-mail provider that is not necessarily the same as their connection provider. Network topology, or the location of a client within a network or outside of a network, is no longer a limiting factor for email submission or delivery. Modern SMTP servers typically use a client's credentials (authentication) rather than a client's location (IP address), to determine whether it is eligible to relay email.

Server administrators choose whether clients use TCP port 25 (SMTP) or port 587 (Submission), as formalized in RFC 4409, for relaying outbound mail to a mail server. The specifications and many servers support both. Although some servers support port 465 for legacy secure SMTP in violation of the specifications, it is preferable use standard ports and standard ESMTP commands^[6] according to RFC 3207 if a secure session needs to be used between the client and the server. Some servers are set up to reject all relaying on port 25, but valid users authenticating on port 587 are allowed to relay mail to any valid address. A server that relays all email for all destinations for all clients connecting to port 25 is known as an open relay and is now generally considered a bad practice worthy of blacklisting.

Sample communications

After establishing a connection between the sender (the client) and the receiver (the server), the following is a valid SMTP session. In the following conversation, everything sent by the client is prefaced with C: and everything sent by the server is prefaced with S:. On most computer systems, a connection can be established using the telnet command on the client machine, for example:

telnet smtp.example.com 25

which opens a TCP connection from the sending machine to the MTA listening on port 25 on host smtp.example.com. By convention, SMTP servers greet clients with their fully-qualified domain name. In this example, the client computer (relay.example.org) has already determined that "smtp.example.com" is a mail exchanger for the example.com domain by doing a DNS lookup of example.com's MX records. Note that the actual carriage returns and line feeds are not shown, but they are required at the end of each line.

Note that many internet service providers and enterprise or institution networks restrict the ability to connect to any server using port 25.

S: 220 smtp.example.com ESMTP Postfix
C: HELO relay.example.org
S: 250 Hello relay.example.org, I am glad to meet you
C: MAIL FROM:<bob@example.org>
S: 250 Ok
C: RCPT TO:<alice@example.com>
S: 250 Ok
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: From: "Bob Example" <bob@example.org>
C: To: Alice Example <alice@example.com>
C: Cc: theboss@example.com
C: Date: Tue, 15 Jan 2008 16:02:43 -0500
C: Subject: Test message
C:
C: Hello Alice.
C: This is a test message with 5 headers and 4 lines in the body.
C: Your friend,
C: Bob
C: .
S: 250 Ok: queued as 12345
C: MAIL FROM:<bob@example.org>
S: 250 Ok
C: RCPT TO:<theboss@example.com>
S: 250 Ok
C: DATA
S: 354 End data with <CR><LF>.<CR><LF>
C: From: "Bob Example" <bob@example.org>
C: To: Alice Example <alice@example.com>
C: Cc: theboss@example.com
C: Date: Tue, 15 Jan 2008 16:02:43 -0500
C: Subject: Test message
C:
C: Hello Alice.
C: This is a test message with 5 headers and 4 lines in the body.
C: Your friend,
C: Bob
C: .
S: 250 Ok: queued as 12346
C: QUIT
S: 221 Bye
{The server closes the connection}

In this example, the e-mail must be delivered twice to the same SMTP server: once for each recipient listed in the "To" and "Cc" headers. So after the first message is successfully queued, the client begins sending the second message. If the second recipient had been located elsewhere, the client would QUIT and connect to the appropriate SMTP server once the first message had been queued. Note that the information the client sends in the HELO and MAIL FROM commands can be retrieved in additional headers that the server adds to the message: Received and Return-Path respectively.

Although optional and not shown above, many clients ask the server which SMTP extensions the server supports, by using the EHLO greeting to invoke Extended SMTP (ESMTP) specified in RFC 1870. These clients fall back to HELO only if the server does not respond to EHLO.

Modern clients may use the ESMTP extension keyword SIZE to inquire of the server the maximum message size that will be accepted. Older clients and servers may try to transfer huge messages that will be rejected after wasting the network resources, including a lot of connect time to dial-up ISPs that are paid by the minute.

Users can manually determine in advance the maximum size accepted by ESMTP servers. The user telnets as above, but substitutes "EHLO host.example.org" for the HELO command line.

S: 220-smtp2.example.com ESMTP Postfix
C: EHLO bob.example.org
S: 250-smtp2.example.com Hello bob.example.org [192.0.2.201]
S: 250-SIZE 14680064
S: 250-PIPELINING
S: 250 HELP

Thus smtp2.example.com declares that it will accept a fixed maximum message size no larger than 14,680,064 octets (8-bit bytes). Depending on the server's actual resource usage, it may be currently unable to accept a message this large. In the simplest case, an ESMTP server will declare a maximum SIZE with only the EHLO user interaction.

Security and spamming

One of the limitations of the original SMTP is that it has no facility for authentication of senders. Therefore the SMTP-AUTH extension was defined. However, the impracticalities of widespread SMTP-AUTH implementation and management means that E-mail spamming is not and cannot be addressed by it.

Modifying SMTP extensively, or replacing it completely, is not believed to be practical, due to the network effects of the huge installed base of SMTP. Internet Mail 2000 is one such proposal for replacement.

Spam is enabled by several factors, including vendors implementing broken MTAs (that do not adhere to standards, and therefore make it difficult for other MTAs to enforce standards), security vulnerabilities within the operating system (often exacerbated by always-on broadband connections) that allow spammers to remotely control end-user PCs and cause them to send spam, and a regrettable lack of "intelligence" in many MTAs.

There are a number of proposals for sideband protocols that will assist SMTP operation. The Anti-Spam Research Group (ASRG) of the Internet Research Task Force (IRTF) is working on a number of E-mail authentication and other proposals for providing simple source authentication that is flexible, lightweight, and scalable. Recent Internet Engineering Task Force (IETF) activities include MARID (2004) leading to two approved IETF experiments in 2005, and DomainKeys Identified Mail in 2006.

Other Protocols for Email

Email is "handed off" (pushed) from a client (MUA) to a mail server (MSA), usually using Simple Mail Transfer Protocol or IMAP. From there, the MSA delivers the mail to a MTA, usually running on the same machine. The MTA looks up the destination(s)'s MX records with a DNS lookup, and begins to relay (push) the message to the server on record via TCP port 25 and SMTP. Once the receiving MTA accepts the incoming message, it is delivered via a mail delivery agent (MDA) to a server which is designated for local mail delivery. The MDA either delivers the mail directly to storage, or forwards it over a network using either SMTP or LMTP, a derivative of SMTP designed for this purpose. Once delivered to the local mail server, the mail is stored for batch retrieval by authenticated mail clients (MUAs). Generally speaking, mail retrieval (pull) is performed using either a type of online folders (e.g. IMAP 4, a protocol that both delivers and organizes mail) or the older single repository format (e.g. POP3, the Post Office Protocol). Webmail clients may use either method, but the retrieval protocol is often not a formal standard. Some local mail servers and MUAs are capable of either push or pull mail retrieval.

References

^ RFC 469 - Network Mail Meeting Summary
^ RFC 524 - A Proposed Mail Protocol
^ RFC 772 - Mail Transfer Protocol
^ http://tldp.org/HOWTO/Usenet-News-HOWTO/x64.html
^ http://tools.ietf.org/html/draft-barber-uucp-project-conclusion-05
^ RFC 3207 specifies only the well-known port 25 and the "Submission port," which is TCP port 587, for the STARTTLS command, the precursor for an encrypted SMTP session using TLS. It makes no mention of the unofficial port 465.

Related Requests For Comments (RFCs)

RFC 3700 Internet Official Protocol Standards (STD 1). As of 2004, this RFC Designates RFC 821 and RFC 822 as the SMTP and MAIL standards, respectively, with RFC 2821 and RFC 2822 as proposed standards. However, in practice the newer "proposed standards" are said to "obsolete" the original.
RFC 821 (official standard) Simple Mail Transfer Protocol
RFC 822 (official standard) Standard for the Format of ARPA Internet Text Messages
RFC 1123 Requirements for Internet Hosts -- Application and Support (STD 3)
RFC 1870 SMTP Service Extension for Message Size Declaration (оbsoletes: RFC 1653)
RFC 2505 Anti-Spam Recommendations for SMTP MTAs (BCP 30)
RFC 2554 SMTP Service Extension for Authentication
RFC 2821 The Simple Mail Transfer Protocol (obsoletes RFC 821 aka STD 10, RFC 974, and RFC 1869)
RFC 2822 Internet Message Format (obsoletes RFC 822 aka STD 11)
RFC 2920 SMTP Service Extension for Command Pipelining (STD 60)
RFC 3030 SMTP Service Extensions for Transmission of Large and Binary MIME Messages
RFC 3207 SMTP Service Extension for Secure SMTP over Transport Layer Security (obsoletes RFC 2487)
RFC 3461 SMTP Service Extension for Delivery Status Notifications (obsoletes RFC 1891)
RFC 3462 The Multipart/Report Content Type for the Reporting of Mail System Administrative Messages (obsoletes RFC 1892)
RFC 3463 Enhanced Status Codes for SMTP (obsoletes RFC 1893 )
RFC 3464 An Extensible Message Format for Delivery Status Notifications (obsoletes RFC 1894)
RFC 3552 Guidelines for Writing RFC Text on Security Considerations (contains SMTP example)
RFC 3834 Recommendations for Automatic Responses to Electronic Mail
RFC 4409 Message Submission for Mail (obsoletes RFC 2476)

External links

2821bis draft (2005)
2821 security draft (2005)
Internet Mail Architecture draft (2005)
Email Submission Access and Accountability draft (2005)
Essential Internet Protocols - SMTP

@@ Line 44: / Line 44: @@
 which opens a TCP connection from the sending machine to the MTA listening on port 25 on host smtp.example.com. By convention, SMTP servers greet clients with their fully-qualified domain name. In this example, the client computer (''relay.example.org'') has already determined that "smtp.example.com" is a mail exchanger for the ''example.com'' domain by doing a DNS lookup of ''example.com'''s MX records. Note that the actual carriage returns and line feeds are not shown, but they are required at the end of each line.
+Note that many internet service providers and enterprise or institution networks restrict the ability to connect to any server using port 25.
 <code>
  ''S: 220 smtp.example.com ESMTP Postfix''