Apache Crunch edit  

An abstraction layer over MapReduce (and now Spark) that provides a high level Java API for creating data transformation pipelines, originally designed to make working with MapReduce easier based on the Google FlumeJava paper. Also includes connectors for HBase, Hive and Kafka, Java 8 lambda support, an experimental Scala wrapper for the API (Scrunch), and support for in memory pipelines and helper classes to support testing. Open sourced by Cloudera in October 2011, donated to the Apache Foundation in May 2012, before graduating in February 2013. Support for Spark was added as part of v0.10 in June 2014. Still being maintained, and appears to have had been adopted at a number of large companies, but with limited new development.

Technology Information

Other Names	Crunch
Vendors	The Apache Software Foundation
Type	Commercial Open Source
Last Updated	April 2017 - 0.15

Related Technologies

Is packaged by	Apache Bigtop
Is packaged by (but deprecated)	Cloudera CDH

Release History

version	release date	release links	release comment
0.15	2017-02-26	GitHub release page

Links

• https://crunch.apache.org/ - homepage
• https://crunch.apache.org/user-guide.html - user guide
• http://blog.cloudera.com/blog/2011/10/introducing-crunch/ - initial intro blog post
• https://crunch.apache.org/user-guide.html#motivation - the motivation behind Crunch
• http://crunch.apache.org/scrunch.html - Scrunch page

News

• https://crunch.apache.org/download.html - new releases only appear to be announced on download page

Blog Posts

• The Week That Was - 10/02/2017 2017-02-10 Technologies Apex Crunch Flink Tez ZooKeeper Peter