8 Causes Why YipitData Migrated From Apache Airflow to Databricks Workflows


It is a collaborative put up from Databricks and YipitData. We thank Engineering Supervisor Hillevi Crognale at YipitData for her contributions.


YipitData is the trusted supply of insights from different information for the world’s main funding funds and corporations. We analyze billions of information factors day by day to supply correct, detailed insights on many industries, together with retail, e-commerce marketplaces, ridesharing, funds, and extra. Our crew makes use of Databricks and Databricks Workflows to scrub and analyze petabytes of information that most of the world’s largest funding funds and companies rely on.

Out of 500 workers at YipitData, over 300 have a Databricks account, with the most important phase being information analysts. The Databricks platform’s success and penetration at our firm is essentially a results of a powerful tradition of possession. We consider that analysts ought to personal and handle all of their ETL end-to-end with a central Knowledge Engineering crew supporting them via guardrails, tooling, and platform administration.

Adopting Databricks Workflows

Traditionally, we’ve relied on a personalized Apache Airflow set up on prime of Databricks for information orchestration. Knowledge orchestration is important to our enterprise working as our merchandise are derived from becoming a member of lots of of various information sources in our petabyte-scale Lakehouse on a day by day cadence. These information flows have been expressed as Airflow DAGs utilizing the Databricks operator.

Knowledge analysts at YipitData arrange and managed their DAGs via a bespoke framework developed by our Knowledge Engineering platform crew, and expressed transformations, dependencies, and cluster t-shirt sizes in particular person notebooks.

We determined emigrate to Databricks Workflows earlier this yr. Workflows is a Databricks Lakehouse managed service that lets our customers construct and handle dependable information analytics workflows within the cloud, giving us the size and processing energy we have to clear and remodel the large quantities of information we sit on. Furthermore, its ease of use and adaptability means our analysts can spend much less time organising and managing orchestration and as a substitute give attention to what actually issues– utilizing the information to reply our shoppers’ key questions.

With over 600 DAGs energetic in Airflow earlier than this migration, we have been executing as much as 8,000 information transformation duties day by day. Our analysts love the productiveness tailwind from orchestrating their work, and our firm has had nice success from them doing so.

Challenges with Apache Airflow

Whereas Airflow is a robust device and has served us effectively, it had a number of drawbacks for our use case:

  • Studying Airflow requires a big time dedication, particularly given our customized setup. It’s a device designed for engineers, not information analysts. Because of this, onboarding new customers takes longer, and extra effort is required in creating and sustaining coaching materials.
  • With a separate utility exterior of Databricks, there’s latency induced at any time when a command is run, and the precise execution of duties is a black field, proving troublesome given lots of our DAGs run for a number of hours. This lack of visibility introduces longer suggestions loops, and extra time spent with out solutions.
  • Having a customized utility meant further overhead and complexities for our Knowledge Platform Engineering crew when creating tooling or administering the platform. Continuously needing to issue on this separate utility makes something from upgrading spark variations to information governance extra difficult.

“If we went again to 2018 and Databricks Workflows was accessible, we might by no means have thought-about constructing out a customized Airflow setup. We might simply use Workflows.”

As soon as Databricks Workflows was launched, it was clear to us that this could be the longer term. Our aim is to have our customers do all of their ETL work on Databricks, end-to-end. The extra we work with the Databricks Lakehouse Platform, the better it’s each from a consumer expertise, and a knowledge administration and governance perspective.

How we made the transition

Total, the migration to Workflows has been comparatively easy. Since we already used Databricks notebooks because the duties in every Airflow DAG, it was a matter of making a workflow as a substitute of an Airflow DAG primarily based on the settings, dependencies, and cluster configuration outlined in Airflow. Utilizing the Databricks APIs, we created a script to automate a lot of the migration course of.

The brand new Databricks Workflows resolution

“To us, Databricks is turning into the one-stop store for all of our ETL work. The extra we work with the Lakehouse Platform, the better it’s for each customers and platform directors.”

Workflows have a number of options that enormously profit us:

  • With an intuitive UI natively within the Databricks workspace, the benefit of use as an orchestration device for our Databricks customers is unmatched. Creating and sustaining workflows requires much less overhead, releasing up time to give attention to different areas.
  • Onboarding new customers is quicker. Getting in control on Workflows is considerably simpler than coaching new hires on our customized Airflow setup via a set of notebooks and APIs. Because of this, our groups spend much less time on orchestration coaching, and the brand new hires generate information insights weeks sooner than earlier than.
  • With the ability to dive into an current run of a activity and examine on the progress is particularly useful given lots of our duties run for hours’ finish. This unlocks faster suggestions loops, letting our customers iterate sooner on their work.
  • Staying inside the Databricks ecosystem means seamless integration with all different options and companies, just like the Unity Catalog, which we’re at present migrating to. With the ability to depend on Databricks for continued improvement and launch of recent options to the Workflows device, versus proudly owning a separate Airflow utility and sustaining and supporting it ourselves, removes a ton of overhead on our engineering crew’s finish.
  • Workflows is an extremely dependable orchestration service given the 1000’s of duties and job clusters we launch day by day. Up to now, we might dedicate a number of FTEs to keep up our Airflow infrastructure which is now pointless. This frees our engineers to provide extra worth to our enterprise.

The Databricks platform lets us handle and course of our information on the pace and scale we have to be a number one market analysis agency in a disruptive financial system. Adopting Workflows as our orchestration device was a pure step given how built-in we already are with the platform, and the success we’ve skilled from being so. Once we can empower our customers to personal their work and get their jobs completed extra effectively, all people wins.

To be taught extra about Databricks Workflows take a look at the Databricks Workflows web page, watch the Workflows demo and revel in and end-to-end demo with Databricks Workflows orchestrating streaming information and ML pipelines on the Databricks Demo Hub.


Leave a Reply