Big data

A data visualization created by IBM shows that big data such as Wikipedia edits by bot Pearle are more meaningful when enhanced with colors and position.

In information technology, big data consists of datasets that grow so large that they become awkward to work with using on-hand database management tools. Difficulties include capture, storage, search, sharing, analytics, and visualizing. This trend continues because of the benefits of working with larger and larger datasets allowing analysts to "spot business trends, prevent diseases, combat crime." Though a moving target, current limits are on the order of terabytes, exabytes and zettabytes of data. Scientists regularly encounter this problem in meteorology, genomics, connectomics, complex physics simulations, biological and environmental research, Internet search, finance and business informatics. Data sets also grow in size because they are increasingly being gathered by ubiquitous information-sensing mobile devices, aerial sensory technologies (remote sensing), software logs, cameras, microphones, Radio-frequency identification readers, and wireless sensor networks. Every day, 2.5 quintillion bytes of data are created and 90% of the data in the world today was created within the past two years.

One current feature of big data is the difficulty working with it using relational databases and desktop statistics/visualization packages, requiring instead "massively parallel software running on tens, hundreds, or even thousands of servers". The size of "big data" varies depending on the capabilities of the organization managing the set. "For some organizations, facing hundreds of gigabytes of data for the first time may trigger a need to reconsider data management options. For others, it may take tens or hundreds of terabytes before data size becomes a significant consideration."

Definition

Big data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target currently ranging from a few dozen terabytes to many petabytes of data in a single data set.

In a 2001 research report and related conference presentations, then META Group (now Gartner) analyst, Doug Laney, defined data growth challenges (and opportunities) as being three-dimensional, i.e. increasing volume (amount of data), velocity (speed of data in/out), and variety (range of data types, sources). Gartner continues to use this model for describing big data.

Examples

Examples include web logs; RFID; sensor networks; social networks; social data (due to the Social data revolution), Internet text and documents; Internet search indexing; call detail records; astronomy, atmospheric science, genomics, biogeochemical, biological, and other complex and/or interdisciplinary scientific research; military surveillance; medical records; photography archives; video archives; and large-scale eCommerce.

Technologies

Big data requires exceptional technologies to efficiently process large quantities of data within tolerable elapsed times. Technologies being applied to big data include massively parallel processing (MPP) databases, datamining grids, distributed file systems, distributed databases, cloud computing platforms, the Internet, and scalable storage systems.^{[citation needed]}

Some but not all MPP relational databases have the ability to store and manage petabytes of data. Implicit is the ability to load, monitor, backup, and optimize the use of the large data tables in the RDBMS.

The practitioners of Big Data Analytics processes are generally hostile to shared storage. They prefer direct-attached storage (DAS) in its various forms from solid state disk (SSD) to high capacity SATA disk buried inside parallel processing nodes. The perception of shared storage architectures—SAN and NAS—is that they are relatively slow, complex, and above all, expensive. These qualities are not consistent with Big Data Analytics systems that thrive on system performance, commodity infrastructure, and low cost.

Real or near-real time information delivery is one of the defining characteristics of Big Data Analytics. Latency is therefore avoided whenever and wherever possible. Data in memory is good. Data on spinning disk at the other end of a FC SAN connection is not. But perhaps worse than anything else, the cost of a SAN at the scale needed for analytics applications is thought to be prohibitive.

There is a case to be made for shared storage in Big Data Analytics. But storage vendors and the storage community in general have yet to make that case to Big Data Analytics practitioners.

Impact

When the Sloan Digital Sky Survey (SDSS) began collecting data in 2000, it amassed more in its first few weeks than all data collected in the history of astronomy. Continuing at a rate of about 200 GB per night, SDSS has amassed more than 140 terabytes of information. When the Large Synoptic Survey Telescope, successor to SDSS, comes online in 2016 it is anticipated to acquire that amount of data every five days. In total, the four main detectors at the Large Hadron Collider (LHC) produced 13 petabytes of data in 2010 (13,000 terabytes).

More Big Data Impacts:

• Walmart handles more than 1 million customer transactions every hour, which is imported into databases estimated to contain more than 2.5 petabytes of data - the equivalent of 167 times the information contained in all the books in the US Library of Congress.
• Facebook handles 40 billion photos from its user base.
• Decoding the human genome originally took 10 years to process; now it can be achieved in one week.

The impact of “Big Data” has increased the demand of information management specialists in that Oracle, IBM, Microsoft, and SAP have spent more than $15 billion on software firms only specializing in data management and analytics. This industry on its own is worth more than $100 billion and growing at almost 10% a year which is roughly twice as fast as the software business as a whole.

Big Data has emerged because we are living in a society which makes increasing use of data intensive technologies. There are 4.6 billion mobile-phone subscriptions worldwide and there are between 1 billion and 2 billion people accessing the internet. Basically, there are more people interacting with data or information than ever before. Between 1990 and 2005, more than 1 billion people worldwide entered the middle class which means more and more people who gain money will become more literate which in turn leads to information growth. Cisco predicts that the amount of traffic flowing over the internet will reach 667 exabytes annually by 2013.

Critique

danah boyd has raised concerns about the use of big data in science neglecting principles such as choosing a representative sample by being too concerned about actually handling the huge amounts of data. This approach may lead to results biased in one way or another. Integration across heterogeneous data resources - some that might be considered “big data” and others not - presents formidable logistical as well as analytical challenges, but many researchers argue that such integrations are likely to represent the most promising new frontiers in science.

See also

Wikimedia Commons has media related to: Database or Database models

• Cloud computing
• Data assimilation
• Database theory
• Database-centric architecture
• Data Intensive Computing
• Data structure
• Object database
• Online database
• Real-time database
• Relational database
• Social data revolution
• Supercomputer
• Tuple space

Architecture comparison

• Survey Distributed Databases
• Marin Dimitrov's Comparison on PNUTS, Dynamo, Voldemort, BigTable, HBase, Cassandra and CouchDB May 2010
• Why Use HBase-1: from Million Mark to Billion Mark
• Why Use HBase-2: Demystifying HBase Data integrity, Availability and Performance
• Beyond Hadoop: Next-Generation Big Data Architectures by By Bill McColl 23 October 2010 about "Not Only Hadoop".
• MPI and BSP See wiki about Bulk Synchronous Parallel and Apache HAMA on Hadoop cluster.

Performance evaluation

Existing work done by community

• 2010:Yahoo Cloud Serving Benchmark(YCSB)
• 2010:HBase - non SQL Database, Performances Evaluation

Additional reading

• Manyika, James; Michael Chui, Jaques Bughin, Brad Brown, Richard Dobbs, Charles Roxburgh, Angela Hung Byers (May 2011). Big Data: The next frontier for innovation, competition, and productivity. McKinsey Global Institute. http://www.mckinsey.com/Insights/MGI/Research/Technology_and_Innovation/Big_data_The_next_frontier_for_innovation. 

• Hilbert, Martin; Priscila Lopez (2011). "The World’s Technological Capacity to Store, Communicate, and Compute Information". Science 332 (6025): 60-65. doi:10.1126/science.1200970.