Advanced Message Queuing Protocol

From Wikipedia, the free encyclopedia
Jump to: navigation, search
This article needs additional citations for verification. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. (April 2009)

The Advanced Message Queuing Protocol (AMQP) is an open standard application layer protocol for message-oriented middleware. The defining features of AMQP are message orientation, queuing, routing (including point-to-point and publish-and-subscribe), reliability and security.[1]

AMQP mandates the behaviour of the messaging provider and client to the extent that implementations from different vendors are truly interoperable, in the same way as SMTP, HTTP, FTP, etc. have created interoperable systems. Previous attempts to standardize middleware have happened at the API level (e.g. JMS) and this did not create interoperability.[2] Unlike JMS, which merely defines an API, AMQP is a wire-level protocol. A wire-level protocol is a description of the format of the data that is sent across the network as a stream of octets. Consequently any tool that can create and interpret messages that conform to this data format can interoperate with any other compliant tool irrespective of implementation language.

Contents

[edit] Overview

AMQP was originally designed to provide a vendor-neutral (i.e. interoperable across multiple vendors) protocol for managing the flow of messages across an enterprise's business systems.

AMQP is middleware to provide a point of rendezvous between back-end systems (data stores and services) and front end systems such as end user applications. The first applications were for trading desks in the financial industry, where real time order and market data are transmitted. Though originally used inside of enterprises, AMQP can easily be used to move messages between organizations.

AMQP lets system architects build common messaging patterns out of a simpler underlying model. Typical messaging patterns are: request-response, in which messages are sent to or from specific recipients, publish-and-subscribe, in which information is distributed to a set of recipients according to various subscription criteria, and round-robin, in which tasks are distributed fairly among a set of recipients. Realistic applications combine these, e.g. round-robin for distributing work plus request-response for sending back responses.

The protocol specification defines a binary wire protocol used between a client and server (also known as a broker). In addition the specification outlines a message queuing model and services that an implementation must provide.

The queuing model of AMQP provides for a wide range of messaging use-cases and further refines the functions of the clients and brokers. The function of brokers can be usefully broken into two kinds: exchanges and message queues. Message queues store messages, and various implementations can achieve various quality of service. For example a slow but tornado-proof message queue would keep redundant copies in multiple geographic regions while a fast but fragile message queue might keep everything in a single process's RAM. To help improve interoperability some of these aspects of the message queues are specified in the protocol, e.g. you can state what you need when asking a message queue broker implementation to create a new queue.

The standard AMQP exchanges have no semantics for storing messages. They route them to queues, which store them on behalf of recipients. Exchanges implement a range of message routing techniques: one-to-one message passing (like email to one recipient), one-to-N (like an email list), one-to-one-of-N (like a queue for the next open checkout), and so on. Since all exchanges accept messages from N senders, AMQP allows all one-to-any routing to be N-to-any. The rules that configure an exchange, known as bindings, can range from very simple (pass everything into this message queue) to procedural inspections of message content. AMQP allows arbitrary exchange semantics through custom exchanges (which can queue, generate, consume, and route messages in any way desired by the implementation).

Messages consist of an envelope of properties used in routing and by applications and a content, of any size. AMQP message contents are opaque binary blobs. Messages are passed between brokers and clients using the protocol commands Basic.Publish and Basic.Deliver. These commands are asynchronous so that conditions that arise from a command's evaluation are signalled by sending additional commands back on the channel that carried the command originally. AMQP also provides a synchronous message delivery command, Basic.Get/Get-Ok.

Examples of error conditions include signalling by an exchange that it could not route a message because no route was found, or signalling that a message queue declined to accept a message (say because it was full). Message brokers may be configured to handle exceptions in different ways. For example, routing the associated message to a dead letter queue or even bringing the broker to a hard stop.

[edit] Development

AMQP was developed from mid-2004 to mid-2006 by JPMorgan Chase & Co. and iMatix Corporation who also developed implementations in C/C++ and Java. JPMorgan Chase & Co. and iMatix documented the protocol as an interoperable specification and assigned to a working group that included Cisco Systems, TWIST, IONA Technologies, and iMatix. The working group consists of Bank of America, Barclays, Cisco Systems, Credit Suisse, Deutsche Börse Systems, Envoy Technologies, Inc., Goldman Sachs, HCL Technologies Ltd, Progress Software, iMatix Corporation, INETCO Systems, Informatica Corporation (incl. 29 West), JPMorgan Chase Bank Inc. N.A, Microsoft Corporation, my-Channels, Novell, Red Hat, Inc., Software AG, Solace Systems, StormMQ, Tervela Inc., TWIST Process Innovations ltd, VMware (incl. Rabbit Technologies) and WSO2.

A notable design goal of AMQP was to enable the creation of open standard protocol stacks for business messaging both within and between firms by combining AMQP with one of the many open standards describing business transactions, such as FpML or more generically as a reliable transport for SOAP.

Whilst AMQP originated in the financial services industry, it has general applicability to a broad range of middleware problems.

[edit] The AMQP model

AMQP defines a number of entities. From a connection perspective the relevant ones are:

  • Message broker: a server to which AMQ clients connect using the AMQ protocol. Message brokers can run in a cluster but these details are implementation specific and are not covered by the specification.
  • User: a user is an entity that, by providing credentials in form of a password, may or may not be authorized to connect to a broker.
  • Connection: a physical connection e.g. using TCP/IP or SCTP. A connection is bound to a user.
  • Channel: a logical connection that is tied to a connection. Hence communication over a channel is stateful. Clients that perform concurrent operations on a connection should maintain a distinct channel for each of those. For example, a client that uses a threaded model of concurrency can store the channel data in a thread-local variable.
Entities in the AMQP model used for message transfer

The entities used for the actual sending and receiving of messages are all declared on a channel. A declaration assures the issuing client that the entity exists (or was previously declared by another client). Some of the entities are named. The naming is unique within the scope of the entity and its broker. An attempt to declare an entity with the same name results in an error; to change entity properties, one must delete it first (and then redeclare).

Clients have no way to retrieve the list of all available named entities; no such operations are defined in the AMQP specification. Only if the name is known the client can perform operations on the entity.

Names are encoded in UTF-8, must be between 1 and 255 characters in length and must start with a digit, a letter or an underscore character.

[edit] Exchanges

Exchanges are the entities to which messages are sent. They are named and have a type as well as properties such as:

  • passive: the exchange will not get declared but an error will be thrown if it does not exist.
  • durable: the exchange will survive a broker restart.
  • auto-delete: the exchange will get deleted as soon as there are no more queues bound to it. Exchanges to which queues have never been bound will never get auto deleted.

[edit] Queues

Queues are the entities that store received messages. They are named and have properties but not a type. Clients can subscribe to queues to the effect that the message broker delivers (pushes) the contents of the queue to the client. Alternatively clients can pull messages from the queue as they see fit.

Messages are guaranteed to be delivered in the first-in-first-out order, unless certain kinds of rerouting operations (e.g. due to failures) occur or due to the priorities in the message.

The properties of queues are:

  • alternate-exchange: when a message is rejected by a subscriber or orphaned by queue deletion, it gets routed to this exchange and is removed from the queue.
  • passive: the queue will not get declared but an error will be thrown if it does not exist.
  • durable: the queue will survive a broker restart.
  • exclusive: there can only be one client for this specific queue.
  • auto-delete: the queue will get deleted as soon as no more subscriptions are active on it. This has the same behavior as the auto-delete property for exchanges: if no subscription has been ever active on the queue it will not get auto-deleted. An exclusive queue however will always get auto-deleted when the client terminates session.

Note that queues are scheduled to replace exchanges in AMQP/1.0.

[edit] Messages

Messages are unnamed and are published to an exchange. A message consists of a header and a content body. While the body is opaque data the header contains a number of optional properties:

  • routing-key: this field is used in ways dependent on the type of the exchange.
  • immediate: the message will be handled as unroutable if at least one of the queues which would receive the message has no subscription on it.
  • delivery-mode: indicates that a message might need persistence. For such messages the broker makes a best effort to prevent the message loss before consumption. If there is an uncertainty on the broker's end about the successful delivery of the message (e.g. in case of errors) it might deliver the message more than once. Non-persistent delivery modes do not have this behavior.
  • priority: an indicator (a number between 0 and 9) the precedence level of the message.
  • expiration: the duration in milliseconds before the broker may handle the message as unroutable.

[edit] Bindings

A binding is a relationship between one queue and one exchange that specifies how messages flow from the exchange to the queue. The binding properties match the routing algorithm used in exchanges. Bindings (and exchange algorithms) can be placed on a curve of increasing complexity:

  • Unconditional - the binding has no properties and requests "all" messages from the exchange.
  • Conditional on a fixed string - the binding has one property, the routing key and requests all messages that have an identical routing key (note that a binding's routing key is also used by some exchange types).
  • Conditional on a pattern match - the binding has one property, the routing key and requests all messages that match the routing key using a pattern-matching algorithm. Arbitrary pattern syntaxes could be used. AMQP implements topic matching.
  • Conditional on multiple fixed strings - the binding has a table of properties, the arguments and requests all messages whose headers match these arguments, using logical ANDs or ORs to combine matches.
  • Conditional on multiple patterns - the binding has a table of properties, the arguments and requests all messages whose headers match these arguments, using a pattern matching algorithm and logical combinations.
  • Conditional on algorithmic comparison - the binding has an algorithmic expression (like an SQL SELECT WHERE clause) and requests all messages whose headers match that expression.
  • Conditional on content inspection - the binding specifies arbitrary criteria that are resolved by inspection of the actual message content.

Not all these are implemented as standard, or by all implementations.

[edit] Exchange types and the effect of bindings

These four entities form the basic model of the AMQP. The key to understand how a message is passed to a queue lies in the relationship between the type of an exchange and the resulting interpretation of message and binding routing keys.

An exchange will deliver up to one copy of a message to a queue if the routing key in the message matches the routing key in a binding (multiple semantically identical bindings will not result in duplication). What constitutes a match depends on the type of exchange:

  • a direct exchange matches when the routing key property of a message and the key of the binding are identical.
  • a fanout exchange always matches, even on bindings without a key.
  • a topic exchange matches the routing key property of a message on binding key words. Words are strings which are separated by dots. Two additional characters are also valid: the *, which matches 1 word and the #, which matches 0..N words. Example: *.stock.# matches the routing keys usd.stock and eur.stock.db but not stock.nasdaq.
  • a headers exchange matches on the presence of keys as well as key–value pairs which can be concatenated with logical and/or connectives in a message header. In this case the routing key is not used for matching. Instead of routing key, header keys and/or key-value pairs are used for mathing; for keys match is done on keys that are present; for key-value pairs match is done on keys and values of the keys respectively.

Other types of exchanges, e.g. vendor-specific, are explicitly permitted in the specification.

The concept of binding named queues to named exchanges has powerful properties (with binding making those two entities independent of each other). It is, for instance, possible to bind a single queue with multiple bindings to the same or to different exchanges. Or multiple consumers can share the name of a queue and bind to it with the same parameters and will therefore get only message that the other consumers did not consume. Or multiple consumers can declare independent queues but share the bindings and get all the message every other consumer would get on the bound exchange with these bindings.

[edit] Specification revisions and the future of AMQP

The following specifications of the AMQ protocol have been published, in chronological order:

  • 0-8 in June 2006
  • 0-9 in December 2006
  • 0-10 (documents are undated)
  • 0-9-1 in November 2008
  • 1.0 draft in May 2010
  • 1.0 final in October 2011

The 1.0 specification changes the AMQP model illustrated above by removing the concepts of exchanges and bindings, and replacing these with queues and links. This change aims to remedy two problems with the previous approach:

  1. The publisher needs to know too much about the receivers topology (what exchanges and exchange types are available).
  2. Producer flow control is challenging - if an Exchange is routing a message to 2 different queues, one empty and the other nearly full, what flow control information should be relayed to the producer and how would that be determined?

According to John O'Hara [1] however, JPMorganChase and RedHat introduced links into AMQP/1.0 simply to solve an operational problem of slow consumers causing memory build up in brokers.

Other changes include the introduction of a queue addressing schema similar to E-mail and XMPP. This raises addresses to first-class entities, and allows for the publication of service location records using the DNS.

The process of bringing the 1.0 Specification to a Standard involves a requirement elicitation phase, then the release of a "public review" spec (PR) which should be reviewed and asked for comments, optionally resulting in further modifications. When there are no substantive changes to the PR, it is voted to be the 1.0 Recommendation.[3] At least two implementations have passed the test coverage. The Recommendation has been voted to be 1.0 Standard.[4] As of 31 August 2011 (2011 -08-31), a the final spec has been produced[5] and was accepted by the AMQP working group.

[edit] Implementations

These are the known publicly available AMQP implementations:

[edit] Clients

There are many clients, including:

[edit] Comparative specifications

These are the known open specifications that cover the same or similar space as AMQP:

  • Stomp, a text-based pub-sub protocol developed at Codehaus; uses the JMS-like semantics of 'destination'.
  • RestMS, an HTTP-based message routing and queuing protocol that provides AMQP interoperability through an optional profile.
  • XMPP, the Extensible Messaging and Presence Protocol.

There are also vendor specific, proprietary specifications includes those by the Amazon Simple Queue Service, IBM WebSphere MQ, Microsoft Message Queuing, JMS and the OpenWire as used by ActiveMQ.

There has not as yet been a formal comparison of these and other protocols in the same space, although an informal comparison of XMPP and AMQP may be found here. JMS, the Java Messaging service, is often compared to AMQP. However, JMS is an API specification (part of the Java EE specification) that defines how message producers and consumers are implemented. JMS does not guarantee interoperability between implementations, and the JMS-compliant messaging system in use may need to be deployed on both client and server. On the other hand, AMQP is a wire-level protocol specification. In theory AMQP provides interoperability as different AMQP-compliant software can be deployed on the client and server sides. Note that, like HTTP and XMPP, AMQP does not have a standard API.

[edit] See also

[edit] References

  1. ^ O'Hara, J. (2007). "Toward a commodity enterprise middleware". Acm Queue 5 (4): 48–55. doi:10.1145/1255421.1255424. http://www.acm.org/acmqueue/digital/Queuevol5no4_May2007.pdf. 
  2. ^ Vinoski, S. (2006). "Advanced Message Queuing Protocol". Ieee Internet Computing 10 (6): 87–89. doi:10.1109/MIC.2006.116. http://steve.vinoski.net/pdf/IEEE-Advanced_Message_Queuing_Protocol.pdf. 
  3. ^ http://www.amqp.org/confluence/display/AMQP/Process+SIG
  4. ^ http://www.amqp.org/confluence/display/AMQP/PMC+Minutes+for+2011-08-31
  5. ^ http://www.amqp.org/confluence/download/attachments/7340084/amqp-1.0.0+%282011.08.23%29.pdf?version=1&modificationDate=1314177834000
  6. ^ http://www.h-online.com/open/news/item/iMatix-to-drop-OpenAMQ-support-by-2011-968262.html

[edit] External links

Retrieved from "http://en.wikipedia.org/w/index.php?title=Advanced_Message_Queuing_Protocol&oldid=473572624"
Personal tools
Namespaces
Variants
Actions
Navigation
Interaction
Toolbox
Print/export
Languages