The Mid Week News 09/05/2018 edit  

It’s time for the news again. One one tech release this week, but we have some reading for you…

Technology updates (details are on the relevant technology pages):

Other technology news:

  • Google Cloud Composer, and orchestration service based on Apache Airflow is now available on the Google Cloud Platform - link; [Datanami view]9https://www.datanami.com/2018/05/01/apache-airflow-to-power-googles-new-workflow-service/)
  • Confluent now support running their Confluent Platform (bsaed on Kafka on Kubernetes, although it’s only available in their commercial version - link; Datanami view
  • I’d like to talk about how to test data pipelines in more detail at some point, but this is right up my ally - a new framework called Great Expectations for defining tests or checks that run as part of the pipeline - link
  • From Cloudera, how to backup and recover data in Apache Solr - link
  • From Datanami - DataTorrent, the company behind Apache Apex and their commercial version of it (DataTorrent RTS) has gone under - link
  • Again from Datanami - looks like Netflix has moved its Keystone data pipeline from Samza to Apache Flink
  • An update from Microsoft on CosmosDB - link
Archive