Roadmap Updates

This week we provided an update to our team on what our roadmap for data and systems are looking like for the next 6 months. While we were talking future state and how we are looking to uplift our systems, the main pain point that our internal stakeholders brought up was that our Airflow wasn’t up to snuff.

We’ve been running Open Source Airflow HELM on AKS for a year, and using Airflow 3 for a few months now and we haven’t put a huge amount of effort into it and it has been running ok. The number of jobs we have been running and their complexity has increased significantly over the past year.

But we have been having issues with the airflow worker/scheduler tasks stalling, or long running tasks dropping in the middle of the night, with either resource constraints, or no discernable reason from the Airflow logs.

So that leaves us with a large amount of forwarding looking work we explained to the team, when the way we could make them happiest is just to get Airflow humming.