Most warehouse-to-lake migrations fail. Not during the planning phase, and not dramatically — they fail slowly, expensively, and in ways that are obvious in hindsight but invisible until you're already committed.
I've seen it from the inside: teams that spend six months migrating to a data lake only to discover they've traded one set of problems for a harder set. The warehouse felt like a constraint. The lake became a swamp.
Here's what actually goes wrong — and the fixes to make before you commit a dollar of migration budget.
The operational overhead problem nobody budgets for
A data warehouse manages itself. Indexes, vacuuming, statistics, partitioning — it handles these. A data lake does not. When you move to object storage plus a query engine, you inherit the operational work the warehouse was doing invisibly.
Parquet files accumulate small files. Query performance degrades. You add compaction jobs. Those jobs need scheduling, monitoring, alerting. What started as "we'll save money on storage" becomes a part-time platform engineering role that wasn't in the budget.
The fix: before migrating, audit what your warehouse is managing on your behalf. Write down every operational task you'll need to replicate. If you can't staff it, the lake will cost more than the warehouse did — just in engineering hours instead of licensing fees.
The format choices compound
Parquet or ORC. Delta Lake, Apache Iceberg, or Apache Hudi. Columnar with dictionary encoding or not. These decisions feel technical and low-stakes in the planning phase. They are neither.
At Meredith, we made format decisions early in a data lake build. Two years later, those decisions were embedded in hundreds of jobs, dozens of schemas, and a query layer that couldn't easily change underneath. The "we can always migrate formats later" assumption is almost never true at scale.
The format you pick for your first terabyte is probably the format you'll be running at your hundredth. Choose as if it's permanent, because it usually is.
Iceberg has become the standard worth defaulting to in 2024 and beyond — time travel, schema evolution, partition evolution, ACID transactions. If you're starting fresh, start there. If you're migrating existing data, budget time to convert it properly rather than lifting and shifting raw files.
The storage optimization trap
One of the primary migration motivations is storage cost. S3 is cheap. Warehouses are expensive. The math looks obvious.
What the math doesn't account for: query cost. A warehouse optimises queries for you. A lake shifts that cost to compute — and unoptimised queries on a large lake can cost more per month than the warehouse license did.
The teams that win on cost are the ones who approach it as an engineering problem, not a procurement one. Partition your data by the columns you filter on. Right-size your file sizes (aim for 128MB–512MB per file). Run compaction regularly. Use columnar statistics to enable predicate pushdown. These are not set-and-forget — they require active attention.
The data science access problem
Data warehouses have mature access patterns: SQL, BI tools, a well-understood permission model. When you move to a lake, the data scientists who were productive in the warehouse are suddenly looking at raw Parquet files in S3 with no catalogue, no lineage, and no obvious way to discover what exists.
The migration plan almost always under-invests in the access layer. The data moves. The governance doesn't move with it. A year in, analysts are frustrated, shadow copies of data proliferate, and the "single source of truth" promise has quietly died.
Fix this before you move the first byte: establish your catalogue (Glue, Unity Catalog, or open-source alternatives), your access control model, and your lineage strategy. If those aren't ready, the lake isn't ready.
The AI factor changes the calculus
The migration conversation has shifted in the last two years. LLMs and ML pipelines consume unstructured and semi-structured data that warehouses handle poorly. If your roadmap includes training models, running vector search, or feeding an LLM pipeline, a lake architecture isn't just preferable — it's necessary.
But "we want to do AI" is not a migration strategy. The lake still needs the operational hygiene, the format discipline, and the access layer. AI use cases add requirements on top; they don't eliminate the fundamentals.
What a successful migration actually looks like
The migrations I've seen succeed share a few traits that the failed ones don't:
- They migrate incrementally, not all at once. Start with one domain — say, one business unit's reporting data. Prove the operational model before committing the whole organisation.
- They define "done" before they start. Migration success is not "data is in S3." It's "analysts can query this data with the same confidence and performance they had before." That's a higher bar, and it should be explicit.
- They don't abandon the warehouse immediately. Run both in parallel long enough to validate. Dual-running costs money, but it's cheaper than a failed cutover that requires rolling back 18 months of work.
- They treat governance as a first-class deliverable. The teams who get this right build the catalogue, lineage, and access model before the first production job runs on the lake. Not after.
The honest question
Before you start: does your team have the platform engineering capacity to run a lake? Not to migrate to one — to run one, day after day, for years?
If the answer is no, the right move might not be a lake. A modern cloud warehouse (Snowflake, BigQuery, Redshift) has closed most of the gap on flexibility and cost. A hybrid lakehouse architecture — warehouse for structured analytics, lake for raw and unstructured — often outperforms a full migration for teams without dedicated platform capacity.
The migration is not the hard part. The maintenance is. Plan for both.
Supreeth M Gowda has spent 12+ years across data engineering, software engineering, performance engineering, and data science. He now consults through Encore — learn more about him and his services.