This week we provided an update to our team on what our roadmap for data and systems are looking like for the next 6 months. While we were talking future state and how we are looking to uplift our systems, the main pain point that our internal stakeholders brought up was that our Airflow wasn’t up to snuff.

We’ve been running Open Source Airflow HELM on AKS for a year, and using Airflow 3 for a few months now and we haven’t put a huge amount of effort into it and it has been running ok. The number of jobs we have been running and their complexity has increased significantly over the past year.