Apache Flink
File:Apache Flink Logo.png | |
Developer(s) | Apache Software Foundation |
---|---|
Stable release | 1.0.2
/ 22 April 2016 |
Repository | |
Written in | Java and Scala |
Operating system | Cross-platform |
License | Apache License 2.0 |
Website | https://flink.apache.org/ |
Apache Flink is a community-driven open source framework for distributed big data analytics, like Hadoop and Spark. The core of Apache Flink is a distributed streaming dataflow engine written in Java and Scala.[1][2] It aims to bridge the gap between MapReduce-like systems and shared-nothing parallel database systems. Therefore, Flink executes arbitrary dataflow programs in a data-parallel and pipelined manner.[3] Flink's pipelined runtime system enables the execution of bulk/batch and stream processing programs.[4][5] Furthermore, Flink's runtime supports the execution of iterative algorithms natively.[6]
Flink programs can be written in Java or Scala and are automatically compiled and optimized[7] into dataflow programs that are executed in a cluster or cloud environment.[8] Flink does not provide its own data storage system, input data must be stored in a distributed storage system like HDFS or HBase. For data stream processing, Flink consumes data from (reliable) message queues like Kafka.
Development
Apache Flink is developed under the Apache License 2.0[9] by the Apache Flink Community within the Apache Software Foundation. The project is driven by the Berlin based start-up company dataArtisans, 20 committers, and more than 130 contributors.
Flink Forward
Flink Forward is a conference about Apache Flink. It took place for the first time on 12./13. October 2015 in Berlin. The conference featured 2 key notes, 32 talks from industry and academia, as well as practical training sessions on Flink.
History
In 2010, the research project "Stratosphere: Information Management on the Cloud"[10] (funded by the German Research Foundation (DFG)[11]) was started as a collaboration of Technical University Berlin, Humboldt-Universität zu Berlin, and Hasso-Plattner-Institut Potsdam. Flink is a Stratosphere fork and it became an Apache Incubator project in March 2014.[12] In December 2014, Flink was accepted as an Apache top-level project.[13][14][15][16]
Release Dates
- 03/2016: Apache Flink 1.0 (04/2016: v1.0.1; 04/2016: v1.0.2)
- 11/2015: Apache Flink 0.10 (11/2015: v0.10.1; 02/2016: v0.10.2)
- 06/2015: Apache Flink 0.9 (09/2015: v0.9.1)
- 04/2015: Apache Flink 0.9-milestone-1
Apache Incubator Release Dates
- 01/2015: Apache Flink 0.8-incubating
- 11/2014: Apache Flink 0.7-incubating
- 08/2014: Apache Flink 0.6-incubating (09/2014: v0.6.1-incubating)
- 05/2014: Stratosphere 0.5 (06/2014: v0.5.1; 07/2014: v0.5.2)
Pre-Apache Stratosphere Release Dates
- 01/2014: Stratosphere 0.4 (version 0.3 was skipped)
- 08/2012: Stratosphere 0.2
- 05/2011: Stratosphere 0.1 (08/2011: v0.1.1)
See also
References
- ^ "Apache Flink: Scalable Batch and Stream Data Processing". apache.org.
- ^ "apache/flink". GitHub.
- ^ Alexander Alexandrov, Rico Bergmann, Stephan Ewen, Johann-Christoph Freytag, Fabian Hueske, Arvid Heise, Odej Kao, Marcus Leich, Ulf Leser, Volker Markl, Felix Naumann, Mathias Peters, Astrid Rheinländer, Matthias J. Sax, Sebastian Schelter, Mareike Höger, Kostas Tzoumas, and Daniel Warneke. 2014. The Stratosphere platform for big data analytics. The VLDB Journal 23, 6 (December 2014), 939-964. DOI
- ^ Ian Pointer (7 May 2015). "Apache Flink: New Hadoop contender squares off against Spark". InfoWorld.
- ^ "On Apache Flink. Interview with Volker Markl". odbms.org.
- ^ Stephan Ewen, Kostas Tzoumas, Moritz Kaufmann, and Volker Markl. 2012. Spinning fast iterative data flows. Proc. VLDB Endow. 5, 11 (July 2012), 1268-1279. DOI
- ^ Fabian Hueske, Mathias Peters, Matthias J. Sax, Astrid Rheinländer, Rico Bergmann, Aljoscha Krettek, and Kostas Tzoumas. 2012. Opening the black boxes in data flow optimization. Proc. VLDB Endow. 5, 11 (July 2012), 1256-1267. DOI
- ^ Daniel Warneke and Odej Kao. 2009. Nephele: efficient parallel data processing in the cloud. In Proceedings of the 2nd Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS '09). ACM, New York, NY, USA, , Article 8 , 10 pages. DOI
- ^ "ASF Git Repos - flink.git/blob - LICENSE". apache.org.
- ^ "Stratosphere". stratosphere.eu.
- ^ "DFG - Deutsche Forschungsgemeinschaft -". dfg.de.
- ^ "Stratosphere". apache.org.
- ^ "Project Details for Apache Flink". apache.org.
- ^ "The Apache Software Foundation Announces Apache™ Flink™ as a Top-Level Project : The Apache Software Foundation Blog". apache.org.
- ^ "Will the mysterious Apache Flink find a sweet spot in the enterprise?". siliconangle.com.
- ^ http://www.heise.de/developer/meldung/Big-Data-Apache-Flink-wird-Top-Level-Projekt-2516177.html (in German)
External links
- "Data Artisans Blog".
- "Apache Flink Knowledge Base". A directory of over 600 categorized Apache Flink web resources
- "Announcing Google Cloud Dataflow runner for Apache Flink".
- "Apache Flink is now running on Google Cloud Dataflow".
- "Google Cloud Dataflow bekommt Runner für Apache Flink". (in German)
- "Flink@Twitter".
- "Stratosphere@Twitter".