Jump to content

Apache Drill

From Wikipedia, the free encyclopedia

This is an old revision of this page, as edited by CrispyCocoa (talk | contribs) at 22:58, 26 November 2022 (insert source). The present address (URL) is a permanent link to this revision, which may differ significantly from the current revision.

Apache Drill
Developer(s)Apache Software Foundation
Stable release
1.20.2 / August 3, 2022; 2 years ago (2022-08-03)
RepositoryDrill Repository
Written inJava
Operating systemCross-platform
LicenseApache License 2.0
Websitedrill.apache.org

Apache Drill is an open-source software framework that supports data-intensive distributed applications for interactive analysis of large-scale datasets. Built chiefly by contributions from developers from MapR,[1] Drill is inspired by Google's Dremel system,[2] also productized as BigQuery.[3] Drill is an Apache top-level project.[3] Tom Shiran is the founder of the Apache Drill Project.[4] It was designated an Apache Software Foundation top-level project in December 2016.[5]

Drill supports a variety of NoSQL databases and file systems, including Alluxio, HBase, MongoDB, MapR-DB, HDFS, MapR-FS, Amazon S3, Azure Blob Storage, Google Cloud Storage, Swift, NAS and local files. A single query can join data from multiple datastores. For example, you can join a user profile collection in MongoDB with a directory of event logs in Hadoop.

Drill's datastore-aware optimizer automatically restructures a query plan to leverage the datastore's internal processing capabilities. In addition, Drill supports data locality, if Drill and the datastore are on the same nodes.[6]

Features

One explicitly stated design goal is that Drill is able to scale to 10,000 servers or more and to be able to process petabytes of data and trillions of records in seconds.[7]

  • Schema-free JSON document model similar to MongoDB and Elasticsearch, without requiring a formal schema to be declared
  • Industry-standard APIs: ANSI SQL, ODBC/JDBC, RESTful APIs
  • Extremely user and developer friendly
  • Pluggable architecture enables connectivity to multiple datastores
  • Apache Drill 1.9 added dynamic user defined functions.
  • Apache Drill 1.11 added cryptographic-related functions and PCAP file format support.

Back-end Support

Drill is primarily focused on non-relational datastores, including Apache Hadoop text files, NoSQL, and cloud storage. A notable feature also includes in situ querying of local JSON and Apache Parquet files. Some additional datastores that it supports include:

A new datastore can be added by developing a storage plugin. Drill's "schema-free" JSON data model enables it to query non-relational datastores in-situ .[8]

Front-end Support

Drill itself can be queried via JDBC, ODBC, or REST through a variety of methods and languages including Python and Java. The default install includes a web interface allowing end-users to execute ANSI SQL directly and export data tables as CSV files without any programming.

The dashboard library, Apache Superset,[9] is particularly well suited for visualization of data queried with Drill.

See also

References

  1. ^ Friedman, Ellen (21 Sep 2015). "Apache Drill: Tracking its history as an open source community". Archived from the original on 18 March 2016.
  2. ^ "Spark SQL vs. Apache Drill-War of the SQL-on-Hadoop Tools". ProjectPro. Retrieved 2022-11-15.
  3. ^ a b "The Apache Software Foundation Announces Apache Drill as a Top-Level Project". Retrieved 2014-12-02.
  4. ^ Vizard, Michael (2021-09-01). "Apache Software Foundation updates Drill for broader SQL queries". VentureBeat. Retrieved 2022-10-20.
  5. ^ "Apache Drill Eliminates ETL, Data Transformation for MapR Database". The New Stack. 2016-04-11. Retrieved 2022-11-15.
  6. ^ "Apache Drill - Schema-free SQL for Hadoop, NoSQL and Cloud Storage". drill.apache.org. Retrieved 2015-12-29.
  7. ^ "DrillProposal - INCUBATOR - Apache Software Foundation".
  8. ^ "Frequently Asked Questions - Apache Drill". drill.apache.org. Retrieved 2015-12-29.
  9. ^ Wayner, James R. Borck, Martin Heller, Steven Nuñez, Andrew C. Oliver, Ian Pointer and Peter (2020-10-05). "The best open source software of 2020". InfoWorld. Retrieved 2022-11-26.{{cite web}}: CS1 maint: multiple names: authors list (link)

Papers

Some papers influenced the birth and design. Here is a partial list:

Retrieved from "https://en.wikipedia.org/w/index.php?title=Apache_Drill&oldid=1124016616"