Apache Hive

From Wikipedia, the free encyclopedia
Jump to: navigation, search
Apache Hive
Apache Hive
Developer(s) Contributors
Stable release 2.0.0[1] / February 15, 2016; 6 months ago (2016-02-15)
Development status Active
Written in Java
Operating system Cross-platform
Type Database management system
License Apache License 2.0
Website hive.apache.org

Apache Hive is a data warehouse infrastructure built on top of Hadoop for providing data summarization, query, and analysis.[2] While developed by Facebook, Apache Hive is now used and developed by other companies such as Netflix and the Financial Industry Regulatory Authority (FINRA).[3][4] Amazon maintains a software fork of Apache Hive that is included in Amazon Elastic MapReduce on Amazon Web Services.[5]

Features[edit]

Apache Hive supports analysis of large datasets stored in Hadoop's HDFS and compatible file systems such as Amazon S3 filesystem. It provides an SQL-like language called HiveQL[6] with schema on read and transparently converts queries to MapReduce, Apache Tez[7] and Spark jobs. All three execution engines can run in Hadoop YARN. To accelerate queries, it provides indexes, including bitmap indexes.[8] Other features of Hive include:

  • Indexing to provide acceleration, index type including compaction and Bitmap index as of 0.10, more index types are planned.
  • Different storage types such as plain text, RCFile, HBase, ORC, and others.
  • Metadata storage in an RDBMS, significantly reducing the time to perform semantic checks during query execution.
  • Operating on compressed data stored into the Hadoop ecosystem using algorithms including DEFLATE, BWT, snappy, etc.
  • Built-in user defined functions (UDFs) to manipulate dates, strings, and other data-mining tools. Hive supports extending the UDF set to handle use-cases not supported by built-in functions.
  • SQL-like queries (HiveQL), which are implicitly converted into MapReduce or Tez, or Spark jobs.

By default, Hive stores metadata in an embedded Apache Derby database, and other client/server databases like MySQL can optionally be used.[9]

Four file formats are supported in Hive, which are TEXTFILE,[10] SEQUENCEFILE, ORC[11] and RCFILE.[12][13][14] Apache Parquet can be read via plugin in versions later than 0.10 and natively starting at 0.13.[15][16] Additional Hive plugins support querying of the Bitcoin Blockchain.[17]

HiveQL[edit]

While based on SQL, HiveQL does not strictly follow the full SQL-92 standard. HiveQL offers extensions not in SQL, including multitable inserts and create table as select, but only offers basic support for indexes. Also, HiveQL lacks support for transactions and materialized views, and only limited subquery support.[18][19] Support for insert, update, and delete with full ACID functionality was made available with release 0.14.[20]

Internally, a compiler translates HiveQL statements into a directed acyclic graph of MapReduce or Tez, or Spark jobs, which are submitted to Hadoop for execution.[21]

Hive unit testing frameworks[edit]

Hive Version and release date[edit]

Version Release Date feature
2.1.0 Jun 20, 2016
  • NVDIMM support for LLAP Cache
  • HiveServer2 HTTP mode should support X-Forwarded-Host header for authorization/audits
2.0.0 Feb 15, 2016
  • ORC file dump in JSON format
  • Dynamically partitioned hash join for Tez
1.2.1 Jun 27, 2015
  • CASE comparison operator rotation optimization
1.2.0 May 18 2015
  • Enhance JDBC Driver to not require class specification
  • Hybrid Grace Hash Join
1.1.0 Mar 8, 2015
  • UDF initcap to obtain a string with the first letter of each word in uppercase other letters in lowercase
  • Add identity project remover optimization
1.0.0 Feb 4 2015 Mainly Bug fixes and version convention change.

See also[edit]

References[edit]

External links[edit]