We migrated our Airflow scheduler to Kubernetes this year. Technically, it went well. The KubernetesExecutor runs, pods launch correctly, resource consumption is more precise than before. If I stop there, it's one more successful migration in a list of successful migrations.
Except the real problem wasn't in the deployment logs. It was somewhere else, and it took several weeks to become visible.
What we discovered by touching DAGs we never touched
An executor migration forces you to go back over every single DAG. Not just to change the config, but to check that nothing breaks once it's isolated in its own pod, with its own resources, without the shortcuts the old setup allowed (local file dependencies, environment variables shared between workers, things you never notice as long as you change nothing).
It was during this DAG-by-DAG pass that we ran into a problem that had nothing to do with Kubernetes: a chunk of our pipelines depended on data that no one clearly owned.
Not silos in the classic sense, with a team jealously guarding its schema and refusing access. Nothing that clear-cut. Just pipelines that had been running for three, four years, written by someone who'd since moved to another team, reading a table whose documentation predated a product overhaul, with no SLA, no owner in our catalog (when we had a catalog), no one to ask "can I trust this column."
This kind of debt is invisible as long as you don't touch anything. It runs, it produces a result, that result feeds a downstream dashboard or model, and everyone assumes it's reliable because it never blew up. The day you have to touch the pipeline for real, when you actually have to understand what it does and why, you discover you can't answer basic questions about data you've been using for years.
Modern infrastructure doesn't fix shaky governance, it exposes it
This is the point that struck me most in this migration. Kubernetes didn't break anything. It just forced us to look a little more closely at something we'd been avoiding.
It's actually logical once you think about it. More modern infrastructure generally means more isolation, fewer implicit behaviors inherited from an old setup, more rigor about what's actually declared as a dependency. Everything that relied on vagueness, assumptions, or team memory eventually resurfaces the moment you formalize the infrastructure. The migration doesn't create the problem. It just pulls out the rug it was hiding under.
And that's when you understand something simple, something you know in theory but keep rediscovering in practice: modernizing infrastructure doesn't fix data governance. It exposes it, faster and harder than before, because suddenly you no longer have the option of ignoring it.
The parallel with AI First
The "AI First" debate right now reminds me exactly of this. Many companies want to deploy AI agents, copilots, deep automation, on data foundations they've never really audited. You ask an agent to synthesize, decide, trigger actions based on data that, internally, no one could say with certainty is up to date, complete, or owned by someone who can vouch for it.
An AI doesn't invent reliability that the data doesn't have. It amplifies it in both directions. If the data is good, the agent becomes a real lever. If the data is shaky, the agent produces confident, wrong answers at a speed no human ever could.
The problem is that this kind of debt is also invisible until you push on it. An agent that answers correctly 9 times out of 10 on a poorly governed database gives the illusion that everything is fine, exactly like our old Airflow DAGs ran without a hitch until the day we actually had to look at them.
Before picking the tool, ask the right question
Before asking which AI tool to deploy, the real question is often: do we already know how to answer "who owns this data, and can we trust them"? If the answer is no for a significant share of what you plan to plug into an agent, the real work isn't choosing the LLM or the orchestration framework. It's a governance project. Much less exciting to announce, but it's the one that determines whether anything else is worth anything.
Concretely, what we put in place after this migration, and what I'd recommend before any serious AI project:
- A named owner for every critical dataset, not a vague team. If no one can be identified in under thirty seconds, that's a signal.
- Ownership metadata directly in the code, not in a wiki gathering dust. For us that means dbt's
meta, with an owner and an expected freshness per model. - A clear policy for orphaned pipelines. A DAG with no identifiable owner for more than six months should be a candidate for deprecation, not a mystery left running out of caution.
- Freshness monitoring, not just execution success. A DAG that finishes without error doesn't mean the data it produces is still relevant.
Nothing spectacular in there. It's actually rather boring compared to announcing agents everywhere. But in my experience, it's the work that determines whether everything else holds up.
What I take away from this
The technical migration went well because we followed a known method: DAG by DAG, tests, rollback ready, observability in place before switching over. The real project, the data governance we'd let pile up for years, has no equivalent shortcut. You just have to accept looking at it before a faster tool, AI or not, forces you to discover it in production.