Apache Livy edit  

A service that allows Spark jobs (pre-compiled JARs) or code snippets (Scala or Python) to be executed by remote systems over a REST API or via clients for Java, Scala and Python. Supports re-use of Spark Contexts (and caching and sharing of RDDs across jobs and clients), multiple concurrent clients, secure authenticated communications and batch job submissions. Started in November 2015 based on code from Hue, with a formal announcement and first release in June 2016 based on development led by Cloudera, Hortonworks and Microsoft, before being donated to the Apache Foundation in June 2017. Hasn't yet graduated, but under active development, and used by tools such as Hue and Zeppelin.

Technology Information

Other Names	Livy
Vendors	The Apache Software Foundation
Type	Open Source - Active
Last Updated	June 2017 - v0.3

Related Technologies

Is packaged by

Hortonworks Data Platform, Amazon EMR

Links

• https://livy.incubator.apache.org/ - Apache homepage
• https://www.cloudera.com/more/news-and-events/press-releases/2016-06-06-cloudera-microsoft-lead-development-open-source-project-livy-for-easy-use-spark-end-user-applications.html - original announcement
• https://hortonworks.com/blog/livy-a-rest-interface-for-apache-spark/ - Hortonworks view

News

• https://github.com/apache/incubator-livy/releases - details of releases

Blog Posts

• The Week That Was - 03/03/2017 2017-03-03 Technologies Slider Storm Accumulo Livy HDP Search Peter