- 1 Overview
- 2 Logical view
- 3 Dataflow
- 4 Distribution and reliability
- 5 Educational courses
- 6 Uses
- 7 Criticism
- 8 Implementations
- 9 Conferences and users groups
- 10 References
- 11 External links
MapReduce is a framework for processing huge datasets on certain kinds of distributable problems using a large number of computers (nodes), collectively referred to as a cluster. Computational processing can occur on data stored either in a filesystem (unstructured) or within a database (structured).
"Map" step: The master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure.
The worker node processes that smaller problem, and passes the answer back to its master node.
"Reduce" step: The master node then takes the answers to all the sub-problems and combines them in a way to get the output - the answer to the problem it was originally trying to solve.
The advantage of MapReduce is that it allows for distributed processing of the map and reduction operations. Provided each mapping operation is independent of the other, all maps can be performed in parallel - though in practice it is limited by the data source and/or the number of CPUs near that data. Similarly, a set of 'reducers' can perform the reduction phase - all that is required is that all outputs of the map operation which share the same key are presented to the same reducer, at the same time. While this process can often appear inefficient compared to algorithms that are more sequential, MapReduce can be applied to significantly larger datasets than that which "commodity" servers can handle - a large server farm can use MapReduce to sort a petabyte of data in only a few hours. The parallelism also offers some possibility of recovering from partial failure of servers or storage during the operation: if one mapper or reducer fails, the work can be rescheduled — assuming the input data is still available.
The Map and Reduce functions of MapReduce are both defined with respect to data structured in (key, value) pairs. Map takes one pair of data with a type in one data domain, and returns a list of pairs in a different domain:
Map(k1,v1) -> list(k2,v2)
The map function is applied in parallel to every item in the input dataset. This produces a list of (k2,v2) pairs for each call. After that, the MapReduce framework collects all pairs with the same key from all lists and groups them together, thus creating one group for each one of the different generated keys.
The Reduce function is then applied in parallel to each group, which in turn produces a collection of values in the same domain:
Reduce(k2, list (v2)) -> list(v3)
Each Reduce call typically produces either one value v3 or an empty return, though one call is allowed to return more than one value. The returns of all calls are collected as the desired result list.
Thus the MapReduce framework transforms a list of (key, value) pairs into a list of values. This behavior is different from the functional programming map and reduce combination, which accepts a list of arbitrary values and returns one single value that combines all the values returned by map.
It is necessary but not sufficient to have implementations of the map and reduce abstractions in order to implement MapReduce. Distributed implementations of MapReduce require a means of connecting the processes performing the Map and Reduce phases. This may be a distributed file system. Other options are possible, such as direct streaming from mappers to reducers, or for the mapping processors to serve up their results to reducers that query them.
The canonical example application of MapReduce is a process to count the appearances of each different word in a set of documents:
void map(String name, String document): // name: document name // document: document contents for each word w in document: EmitIntermediate(w, "1"); void reduce(String word, Iterator partialCounts): // word: a word // partialCounts: a list of aggregated partial counts int result = 0; for each pc in partialCounts: result += ParseInt(pc); Emit(AsString(result));
Here, each document is split in words, and each word is counted initially with a "1" value by the Map function, using the word as the result key. The framework puts together all the pairs with the same key and feeds them to the same call to Reduce, thus this function just needs to sum all of its input values to find the total appearances of that word.
The frozen part of the MapReduce framework is a large distributed sort. The hot spots, which the application defines, are:
- an input reader
- a Map function
- a partition function
- a compare function
- a Reduce function
- an output writer
The input reader divides the input into 16MB to 128MB splits and the framework assigns one split to each Map function. The input reader reads data from stable storage (typically a distributed file system) and generates key/value pairs.
A common example will read a directory full of text files and return each line as a record.
Each Map function takes a series of key/value pairs, processes each, and generates zero or more output key/value pairs. The input and output types of the map can be (and often are) different from each other.
If the application is doing a word count, the map function would break the line into words and output the word as the key and "1" as the value.
The output of all of the maps is allocated to a particular reducer by the application's partition function. The partition function is given the key and the number of reducers and returns the index of the desired reduce.
The input for each reduce is pulled from the machine where the map ran and sorted using the application's comparison function.
The framework calls the application's reduce function once for each unique key in the sorted order. The reduce can iterate through the values that are associated with that key and output 0 or more values.
In the word count example, the reduce function takes the input values, sums them and generates a single output of the word and the final sum.
The Output Writer writes the output of the reduce to stable storage, usually a distributed file system.
Distribution and reliability
MapReduce achieves reliability by parceling out a number of operations on the set of data to each node in the network. Each node is expected to report back periodically with completed work and status updates. If a node falls silent for longer than that interval, the master node (similar to the master server in the Google File System) records the node as dead and sends out the node's assigned work to other nodes. Individual operations use atomic operations for naming file outputs as a check to ensure that there are not parallel conflicting threads running. When files are renamed, it is possible to also copy them to another name in addition to the name of the task (allowing for side-effects).
The reduce operations operate much the same way. Because of their inferior properties with regard to parallel operations, the master node attempts to schedule reduce operations on the same node, or in the same rack as the node holding the data being operated on. This property is desirable as it conserves bandwidth across the backbone network of the datacenter.
- Cluster Computing and MapReduce course from Google Code University contains video lectures and related course materials from a series of lectures that was taught to Google software engineering interns during the Summer of 2007.
- MapReduce in a Week course from Google Code University contains a comprehensive introduction to MapReduce including lectures, reading material, and programming assignments.
- MapReduce course, taught by engineers of Google Boston, part of 2008 Independent Activities Period at MIT.
MapReduce is useful in a wide range of applications and architectures, including: "distributed grep, distributed sort, web link-graph reversal, term-vector per host, web access log stats, inverted index construction, document clustering, machine learning , statistical machine translation..." Most significantly, when MapReduce was finished, it was used to completely regenerate Google's index of the World Wide Web, and replaced the old ad hoc programs that updated the index and ran the various analyses.
MapReduce's stable inputs and outputs are usually stored in a distributed file system. The transient data is usually stored on local disk and fetched remotely by the reduces.
David DeWitt and Michael Stonebraker, experts in parallel databases and shared-nothing architectures, have made  assertions about the breadth of problems that MapReduce can be used for. They called its interface too low-level and questioned whether it really represents the paradigm shift its proponents have claimed it is. They challenged the MapReduce proponents' claims of novelty, citing Teradata as an example of prior art that has existed for over two decades. They also compared MapReduce programmers to Codasyl programmers, noting both are "writing in a low-level language performing low-level record manipulation". MapReduce's use of input files and lack of schema support prevents the performance improvements enabled by common database system features such as B-trees and hash partitioning, though projects such as PigLatin and Sawzall are starting to address these problems..
Another article  by Greg Jorgensen rejects these views. Jorgensen asserts that DeWitt and Stonebraker's entire analysis is groundless as MapReduce was never designed nor intended to be used as a database. That Google has a distributed database system, BigTable, adds significant weight to this view.
DeWitt and Stonebraker have subsequently published a detailed benchmark study  comparing performance of MapReduce and RDBMS approaches on several specific problems and concluding that databases offer real advantages for many kinds of data use, especially on complex processing or where the data is used across an enterprise, but that MapReduce may be easier for users to adopt for simple or one-time processing tasks. They have published the data and code used in their study to allow other researchers to do comparable studies.
Google has been granted a patent on MapReduce. However, there have been claims that this patent should not have been granted because MapReduce is too similar to existing products, Oracle's pipelined table functions  for instance.
- The Google MapReduce framework is implemented in C++ with interfaces in Python and Java.
- The Hadoop project is a free open source Java MapReduce implementation.
- Twister is an open source Java MapReduce implementation that supports iterative MapReduce computations efficiently.
- Greenplum is a commercial MapReduce implementation, with support for Python, Perl, SQL and other languages.
- Aster Data Systems nCluster In-Database MapReduce supports Java, C, C++, Perl, and Python algorithms integrated into ANSI SQL.
- GridGain is a free open source Java MapReduce implementation.
- Phoenix is a shared-memory implementation of MapReduce implemented in C.
- FileMap is an open version of the framework that operates on files using existing file-processing tools rather than tuples.
- MapReduce has also been implemented for the Cell Broadband Engine, also in C. 
- MapReduce has been implemented on NVIDIA GPUs (Graphics Processors) using CUDA .
- Qt Concurrent is a simplified version of the framework, implemented in C++, used for distributing a task between multiple processor cores .
- CouchDB uses a MapReduce framework for defining views over distributed documents and is implemented in Erlang.
- Skynet is an open source Ruby implementation of Google’s MapReduce framework
- Disco is an open source MapReduce implementation by Nokia. Its core is written in Erlang and jobs are normally written in Python.
- Qizmt is an open source MapReduce framework from MySpace written in C#.
- The open-source Hive framework from Facebook (which provides an SQL-like language over files, layered on the open-source Hadoop MapReduce engine.)
- The Holumbus Framework: Distributed computing with MapReduce in Haskell Holumbus-MapReduce
- BashReduce: MapReduce written as a Bash script written by Erik Frey of Last.fm
- Sector/Sphere which is implemented in C++.
- MapReduce for Go
- MongoDB is a scalable, high-performance, open source, schema-free, document-oriented database. Written in C++ that features MapReduce
- RHIPE integrates the R_(programming_language) statistics language environment with Hadoop.  and makes it possible to code map-reduce algorithms in R.
- Many companies have their own private MapReduce implementations.
Conferences and users groups
- The First International Workshop on MapReduce and its Applications (MAPREDUCE'10) will be held with the HPDC conference and OGF'29 meeting in Chicago, IL.
- MapReduce Users Groups around the world.
- US Patent 7,650,331: "System and method for efficient large-scale data processing "
- Google spotlights data center inner workings | Tech news blog - CNET News.com
- "Our abstraction is inspired by the map and reduce primitives present in Lisp and many other functional languages." -"MapReduce: Simplified Data Processing on Large Clusters", by Jeffrey Dean and Sanjay Ghemawat; from Google Labs
- "Google's MapReduce Programming Model -- Revisited" — paper by Ralf Lämmel; from Microsoft
- Colby Ranger. "Evaluating MapReduce for Multi-core and Multiprocessor Systems". HPCA 2007, Best Paper. Unknown parameter
- Cheng-Tao Chu. "Map-Reduce for Machine Learning on Multicore". NIPS 2006. Unknown parameter
- "How Google Works". baselinemag.com.
As of October, Google was running about 3,000 computing jobs per day through MapReduce, representing thousands of machine-days, according to a presentation by Dean. Among other things, these batch routines analyze the latest Web pages and update Google's indexes.
- "Database Experts Jump the MapReduce Shark".
- David DeWitt. "MapReduce: A major step backwards". databasecolumn.com. Retrieved 2008-08-27. Unknown parameter
- Greg Jorgensen. "Relational Database Experts Jump The MapReduce Shark". typicalprogrammer.com. Retrieved 2009-11-11.
- Andrew Pavlo. "A Comparison of Approaches to Large-Scale Data Analysis". Brown University. Retrieved 2010-01-11. Unknown parameter
- Curt Monash. "More patent nonsense — Google MapReduce". dbms2.com. Retrieved 2010-03-07.
- Parallel Programming in the Age of Big Data
- Abad, Cristina & Avendaño, Allan. "An Introduction to Parallel Programming with MapReduce"
- Dean, Jeffrey & Ghemawat, Sanjay (2004). "MapReduce: Simplified Data Processing on Large Clusters". Retrieved Apr. 6, 2005.
This article's use of external links may not follow Wikipedia's policies or guidelines. (August 2008) (Learn how and when to remove this template message)
- "Interpreting the Data: Parallel Analysis with Sawzall" — paper by Rob Pike, Sean Dorward, Robert Griesemer, Sean Quinlan; from Google Labs
- "Evaluating MapReduce for Multi-core and Multiprocessor Systems" — paper by Colby Ranger, Ramanan Raghuraman, Arun Penmetsa, Gary Bradski, and Christos Kozyrakis; from Stanford University
- "Why MapReduce Matters to SQL Data Warehousing" — analysis related to the August, 2008 introduction of MapReduce/SQL integration by Aster Data Systems and Greenplum
- "MapReduce for the Cell B.E. Architecture" — paper by Marc de Kruijf and Karthikeyan Sankaralingam; from University of Wisconsin–Madison
- "Mars: A MapReduce Framework on Graphics Processors" — paper by Bingsheng He, Wenbin Fang, Qiong Luo, Naga K. Govindaraju, Tuyong Wang; from Hong Kong University of Science and Technology; published in Proc. PACT 2008. It presents the design and implementation of MapReduce on graphics processors.
- "Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters" — paper by Hung-Chih Yang, Ali Dasdan, Ruey-Lung Hsiao, and D. Stott Parker; from Yahoo and UCLA; published in Proc. of ACM SIGMOD, pp. 1029–1040, 2007. (This paper shows how to extend MapReduce for relational data processing.)
- FLuX: the Fault-tolerant, Load Balancing eXchange operator from UC Berkeley provides an integration of partitioned parallelism with process pairs. This results in a more pipelined approach than Google's MapReduce with instantaneous failover, but with additional implementation cost.
- "A New Computation Model for Rack-Based Computing" — paper by Foto N. Afrati; Jeffrey D. Ullman; from Stanford University; Not published as of Nov 2009. This paper is an attempt to develop a general model in which one can compare algorithms for computing in an environment similar to what map-reduce expects.
- Jimmy Lin and Chris Dyer. "Data-Intensive Text Processing with MapReduce" (manuscript)