User:What-a-good-jungle/sandbox

Apache RocketMQ
Developer(s)	Apache Software Foundation
Initial release	2012; 12 years ago
Stable release	4.2.0 / April 27, 2018; 6 years ago
Written in	Java
Operating system	Cross-platform
Type	Stream processing, Message broker
License	Apache License 2.0
Website	rocketmq.apache.org

Apache RocketMQ
Developer(s)	Apache Software Foundation
Initial release	2012; 12 years ago
Stable release	4.2.0 / April 27, 2018; 6 years ago
Written in	Java
Operating system	Cross-platform
Type	Stream processing, Message broker
License	Apache License 2.0
Website	rocketmq.apache.org

Apache RocketMQ
	Rmq
Developer(s)	Apache Software Foundation
Initial release	2012; 12 years ago
Stable release	4.2.0 / April 27, 2018; 6 years ago
Written in	Java
Operating system	Cross-platform
Type	Stream processing, Message broker
License	Apache License 2.0
Website	rocketmq.apache.org

RocketMQ^[1] is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. It is the third generation distributed messaging middleware open sourced by Alibaba in 2012. Through several years of technical improvement, RocketMQ is now capable of transferring trillions of concurrent online messages as of Alibaba's Nov. 11th Shopping Festival. On November 21, 2016, Alibaba donated RocketMQ to the Apache Software Foundation. Next year, on February 20, the Apache Software Foundation announced Apache RocketMQ as a Top-Level Project. Nowadays, RocketMQ is widely used by thousands of companies, and research and educational institutions around the world.

History

RocketMQ's evolution can be divided into three stages.

The first generation adopts the push mode in which data are stored in relational database. Low latency message delivery and distributed transactions meet Alibaba's demands and contribute to RocketMQ's wide adoption.

The second generation adopts the pull mode in which data storage is managed by file systems. Its performance on log collection is comparable to Kafka. Taking account of Alibaba's application scenarios, especially the requirement for high reliability during transaction processes, the second-generation RocketMQ does not blindly seek the throughput but also stresses stability and reliability. The long pulling mechanism makes the response time of the system no slower than that in the first generation.

The third generation, RocketMQ mainly uses the Pull mode along with some Push operations. Largely optimized, it has low latency and high performance, especially in high concurrency and massive amounts of data scenarios. Since 2012, RocketMQ has challenged the task of supporting Alibaba's annual Nov. 11th Global Shopping Festival multiple times. During the 2016 Festival, RocketMQ robustly provided stable infrastructure with a transfer throughput of more than one trillion messages.

Features

Much comparison have been made between all kinds of messaging solutions, there is a table demonstrates the comparison between RocketMQ, ActiveMQ and Kafka (Apache’s most popular messaging solutions according to awesome-java^[2]) up to December 17, 2016:^[3]

Messaging Product	Client SDK	Protocol and Specification	Ordered Message	Scheduled Message	Batched Message	BroadCast Message	Message Filter	Server Triggered Redelivery	Message Storage	Message Retroactive	Message Priority	High Availability and Failover	Message Track	Configuration
ActiveMQ	Java, .NET, C++ etc.	Push model, support OpenWire, STOMP, AMQP, MQTT, JMS	Exclusive Consumer or Exclusive Queues can ensure ordering	Supported	Not Supported	Supported	Supported	Not Supported	Supports very fast persistence using JDBC along with a high performance journal，such as levelDB, kahaDB	Supported	Supported	Supported, depending on storage,if using kahadb it requires a ZooKeeper server	Not Supported	The default configuration is low level, user need to optimize the configuration parameters
Kafka	Java, Scala etc.	Pull model, support TCP	Ensure ordering of messages within a partition	Not Supported	Supported, with async producer	Not Supported	Supported, you can use Kafka Streams to filter messages	Not Supported	High performance file storage	Supported offset indicate	Not Supported	Supported, requires a ZooKeeper server	Not Supported	Kafka uses key-value pairs format for configuration. These values can be supplied either from a file or programmatically.
RocketMQ	Java, C++, Go	Pull model, support TCP, JMS, OpenMessaging	Ensure strict ordering of messages,and can scale out gracefully	Supported	Supported, with sync mode to avoid message loss	Supported	Supported, property filter expressions based on SQL92	Supported	High performance and low latency file storage	Supported timestamp and offset two indicates	Not Supported	Supported, Master-Slave model, without another kit	Supported	Work out of box,user only need to pay attention to a few configurations

Architecture

RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure. As shown in image left.^[4]

NameServer Cluster

Name Servers provide lightweight service discovery and routing. Each Name Server records full routing information, provides corresponding reading and writing service, and supports fast storage expansion.

Broker Cluster

Brokers take care of message storage by providing lightweight TOPIC and QUEUE mechanisms. They support the Push and Pull model, contains fault tolerance mechanism (2 copies or 3 copies), and provides strong padding of peaks and capacity of accumulating hundreds of billion messages in their original time order. In addition, Brokers provide disaster recovery, rich metrics statistics, and alert mechanisms, all of which are lacking in traditional messaging systems.

Producer Cluster

Producers support distributed deployment. Distributed Producers send messages to the Broker cluster through multiple load balancing modes. The sending processes support fast failure and have low latency.

Consumer Cluster

Consumers support distributed deployment in the Push and Pull model as well. It also supports cluster consumption and message broadcasting. It provides real-time message subscription mechanism and can meet most consumer requirements. RocketMQ’s website provides a simple quick-start guide to interested users.

Applications

There are at least five aspects Apache RocketMQ could relate to:^[5]

The integration of heterogeneous systems；
Decoupling between applications;
Backbone in the event-driven mechanism and complex event architecture model;
Data replication channels;
Integration with flow calculation engine.

Open Source and Community Maintenance

The RocketMQ team have done much to active the community. and the team has adopted a strategy similar to Apache's other top-level projects as the team devotes itself to the high quality of its products. A series of standardized software development processes, such as milestone discussions, features design, code testing, peer reviews, integration tests, release discussions, and release announcements have been conducted in the RocketMQ version planning. On the other hand, the team interacts with the open source community through activities like line Meet-ups, Workshops, ApacheCon and Code Marathon, to attract new contributors and committers.^[6]

The team also produces a commercial distribution called Aliware MQ. It supports the TCP, HTTP and MQTT protocols, enhances the maintenance tools of ecological integration (including visualization of the trajectory, resource reporting statistics monitoring and alerts, as well as Kafka integration). Designed for enterprise, it has multi-region deployment with high availability disaster recovery characteristics on the public cloud and private cloud.

Rewards

2016 China’s most popular open source software award

2017 China’s most popular open source software award

12th CJK(China-Japan-South Korea) open source software outstanding technology award

Enterprises that use RocketMQ

Those use RocketMQ:

References

^ "apache/rocketmq". GitHub. Retrieved 2018-05-25.
^ "akullpp/awesome-java". GitHub. Retrieved 2018-05-25.
^ "Motivation". Apache RocketMQ. 2018-03-05. Retrieved 2018-06-14.
^ "RocketMQ Architecture". Apache RocketMQ. 2018-03-05. Retrieved 2018-05-25.
^ "Apache RocketMQ背后的设计思路与最佳实践". 阿里中间件团队博客 (in Simplified Chinese). Retrieved 2018-05-25.
^ "From Alibaba to Apache: RocketMQ's Past, Present, and Future". InfoQ. Retrieved 2018-05-29.

External links

RocketMQ^[1] is a distributed messaging and streaming platform with low latency, high performance and reliability, trillion-level capacity and flexible scalability. It is the third generation distributed messaging middleware open sourced by Alibaba in 2012. On November 21, 2016, Alibaba donated RocketMQ to the Apache Software Foundation. Next year, on February 20, the Apache Software Foundation announced Apache RocketMQ as a Top-Level Project. Nowadays, RocketMQ is widely used by thousands of companies, and research and educational institutions around the world.

History

The development of RocketMQ can be divided into three stages.^[2]

The first generation uses the push mode in data transportation, and relational database in data storage. It shows low latency in message delivery and meets the command of a typical E-commerce platform^[3] with distributed transactions. Then many industry began to adopt it.

The second generation uses the pull mode in data transportation, and file system in data storage. It paid more attention to stability and reliability, and shows a comparable performance to the first generation in response time and Kafka on log collection.

The third generation combines the Pull mode with some Push operations. It inherits the advantages of the first and second generation, and shows high performance in concurrency and massive amounts of data scenarios. It's the key for the success of 2016 Alibaba's Nov. 11th Shopping Festival.

Features

Much comparison have been made between all kinds of messaging solutions, especially between Kafka and RocketMQ,^[4] there is a table demonstrates the comparison between RocketMQ, ActiveMQ and Kafka(the three most popular messaging solutions^[5]) up to December 17, 2016:^[6]

Messaging Product	Client SDK	Protocol and Specification	Ordered Message	Scheduled Message	Batched Message	BroadCast Message	Message Filter	Server Triggered Redelivery	Message Storage	Message Retroactive	Message Priority	High Availability and Failover	Message Track	Configuration
ActiveMQ	Java, .NET, C++ etc.	Push model, support OpenWire, STOMP, AMQP, MQTT, JMS	Exclusive Consumer or Exclusive Queues can ensure ordering	Supported	Not Supported	Supported	Supported	Not Supported	Supports very fast persistence using JDBC along with a high performance journal，such as levelDB, kahaDB	Supported	Supported	Supported, depending on storage,if using kahadb it requires a ZooKeeper server	Not Supported	The default configuration is low level, user need to optimize the configuration parameters
Kafka	Java, Scala etc.	Pull model, support TCP	Ensure ordering of messages within a partition	Not Supported	Supported, with async producer	Not Supported	Supported, you can use Kafka Streams to filter messages	Not Supported	High performance file storage	Supported offset indicate	Not Supported	Supported, requires a ZooKeeper server	Not Supported	Kafka uses key-value pairs format for configuration. These values can be supplied either from a file or programmatically.
RocketMQ	Java, C++, Go	Pull model, support TCP, JMS, OpenMessaging	Ensure strict ordering of messages,and can scale out gracefully	Supported	Supported, with sync mode to avoid message loss	Supported	Supported, property filter expressions based on SQL92	Supported	High performance and low latency file storage	Supported timestamp and offset two indicates	Not Supported	Supported, Master-Slave model, without another kit	Supported	Work out of box,user only need to pay attention to a few configurations

Because the characteristics of high performance, high reliability and high real time ability, more and more efforts have been make to the combination of RocketMQ and other protocol component in all kinds of messaging scenarios.^[7]

Architecture

RocketMQ consists of four parts: name servers, brokers, producers and consumers. Each of them can be horizontally extended without a single point of Failure.^[8] As shown in image left.