The Visibility Illusion
Every modern data platform comes with a cost dashboard. BigQuery has its billing explorer. Snowflake has its usage views. Databricks has its account console. The numbers are there, updated daily, broken down by project, by warehouse, by day.
And yet.
When Finance asks why the data platform bill increased 40% last quarter, most data leaders struggle to answer. Not because the information doesn't exist. Because the information doesn't explain anything.
You can see that Project X consumed 2.3 million slot-hours last month. You cannot see that 60% of those slot-hours came from a single dashboard that refreshes every fifteen minutes for an audience of three people who check it twice a day.
You can see that Warehouse Y ran for 847 hours. You cannot see that half of those hours were driven by a dbt model that nobody remembers requesting, feeding a report that nobody remembers building, for a stakeholder who left the company eight months ago.
This is the visibility illusion. The numbers are visible. The meaning is not.
Cost Is a Symptom
Here is a pattern I see repeatedly in growing data organizations.
The platform starts small. A few engineers, a handful of models, a modest warehouse. Costs are low. Everyone knows what everything does. The system is legible.
Then growth happens. New teams onboard. New use cases emerge. The dbt project that started with fifty models now has four hundred. The BI layer that started with a dozen dashboards now has three hundred. The warehouse that ran a few hours a day now runs continuously.
Costs increase. Leadership notices. Someone is asked to "optimize."
The optimization effort typically follows a predictable script. An engineer runs some queries against the metadata tables. They identify the most expensive queries. They find some obvious inefficiencies: a join that scans too much data, a model that materializes when it should be ephemeral, a warehouse that's oversized for its workload.
Fixes are applied. Costs drop. Everyone celebrates.
Three months later, costs are back to where they started. Often higher.
What happened?
The optimization addressed symptoms, not causes. The expensive queries were symptoms of architectural decisions nobody documented. The oversized warehouse was a symptom of scheduling patterns nobody owns. The inefficient model was a symptom of a requirement nobody remembers.
Cost is always a symptom. The disease is something else.
The Ownership Vacuum
In most data platforms, there is no clear answer to a simple question: who owns this?
Not "who built this", git blame can answer that. Not "who last modified this", your orchestrator logs can answer that.
Who owns this? Who is responsible for whether this should exist? Who decides if the cost is justified? Who gets paged when it breaks? Who gets asked when Finance wants to know why it costs what it costs?
In mature software engineering organizations, service ownership is a solved problem. There are runbooks. There are on-call rotations. There are SLAs. If a service is expensive, someone is accountable.
In most data platforms, ownership is a vacuum.
Dashboards exist because someone requested them once. Nobody knows if they're still used. Nobody knows if they're worth what they cost. They refresh continuously because that's how they were configured, and nobody has the authority, or the information, to decide otherwise.
Models exist because someone needed a metric once. The metric fed a dashboard. The dashboard fed a decision. The decision was made. But the model keeps running, the dashboard keeps refreshing, and the cost keeps accumulating, long after the decision it supported has become irrelevant.
This is not a technology problem. This is a governance problem. And you cannot solve governance problems with better queries.
The Dashboard That Ate the Pipeline
Let me describe a scenario I've encountered in multiple organizations.
A product team needs to track a key metric. They ask the data team for a dashboard. The data team builds a pipeline: data lands in the warehouse, a dbt model transforms it, a BI tool visualizes it. The dashboard is configured to refresh every fifteen minutes because "we want it to be fresh."
Time passes. The metric proves useful. More charts are added. More filters. More breakdowns. The single dashboard becomes a collection of dashboards. The single model becomes a family of models. The pipeline grows.
One day, someone runs a cost analysis. They discover that the BI tool, the visualization layer, is consuming more compute than the entire transformation layer beneath it. Every fifteen minutes, dozens of queries fire against the warehouse. Most of these queries serve nobody. The dashboards are open in browser tabs that nobody looks at. The refresh happens anyway.
The dashboard is now more expensive than the data pipeline feeding it.
This is not unusual. In BI-heavy organizations, it is common.
But here's the interesting part: nobody can turn it off.
The product team still considers the dashboard important. They just don't use it the way it's configured. They would be fine with hourly refreshes, or on-demand refreshes, or a daily snapshot. But nobody asked them. Nobody even knows to ask them. The dashboard exists in an ownership vacuum. It runs because it runs.
This is what happens when cost is visible but not explainable. Everyone can see the bill. Nobody can justify the expense. And because nobody can justify it, nobody can change it.
A Framework for Cost Explainability
When I analyze a data platform's costs, I think in four layers. I call this the Data Cost Stack.
Layer 1: Usage
Who runs what, when, and how often?
This is where most cost analysis stops. You pull the query logs. You rank by compute consumed. You identify the top offenders. This is useful, but insufficient. Knowing that Query X is expensive tells you nothing about whether Query X should exist.
Layer 2: Flow
How does data move through the system?
This layer traces the path from source to consumption. It maps the dependencies: which models feed which dashboards, which pipelines feed which models, which sources feed which pipelines. It reveals the structure of the system: not just what runs, but why it runs.
This is where you discover that three dashboards are all computing the same metric independently. Or that a model runs hourly because a downstream dashboard expects freshness, even though the source data only updates daily. Or that a pipeline exists solely to feed another pipeline that no longer exists.
Layer 3: Design
How is the system architected?
This layer examines the technical decisions that determine cost: partitioning strategies, clustering keys, materialization choices, warehouse sizing, scheduling configurations. These decisions compound. A table that isn't partitioned correctly costs more every time it's queried. A warehouse that's oversized costs more every hour it runs. A model that's incremental instead of full-refresh costs less every day it executes.
Design is where cost efficiency lives. But design decisions are only meaningful if you understand the usage and flow that motivate them. Changing a materialization strategy without understanding downstream dependencies is how you break things.
Layer 4: Ownership
Who is responsible for what?
This is the layer that most organizations skip entirely. And it's the layer that matters most.
Ownership answers the questions that the other layers cannot:
- Should this dashboard exist?
- Is this cost justified?
- Who approves changes to this pipeline?
- Who decides if the refresh frequency is appropriate?
- Who owns the budget for this workload?
Without ownership, the other layers produce information that nobody can act on. You can identify that a dashboard costs several thousand euros per month. But if nobody owns the dashboard, nobody can decide whether that cost is acceptable. The information exists. The authority to act on it does not.
Why Ownership Is the Hardest Layer
Ownership is hard because it requires organizational decisions, not technical ones.
You can implement partitioning with a SQL statement. You can change warehouse sizing with a configuration file. You can adjust refresh frequencies with a cron expression.
You cannot implement ownership with code. Ownership requires someone to say: "This is mine. I am responsible. The cost comes from my budget. The decisions come from my authority."
Most data organizations resist this. Ownership creates accountability, and accountability creates friction. If I own this dashboard, I might have to justify its existence. I might have to defend its cost. I might have to decide to shut it down. It's easier to let it exist in the commons, owned by everyone and therefore owned by no one.
But the commons is where costs accumulate. The commons is where expensive dashboards survive. The commons is where nobody optimizes, because optimization requires someone to decide what's worth keeping.
The first step to cost explainability is not better tooling. It's clearer ownership.
From Optimization to Explainability
Here is a different way to think about data platform costs.
The traditional approach is optimization: find expensive things, make them cheaper. This approach treats cost as the problem.
A better approach is explainability: make costs traceable to business value. This approach treats cost as information.
When costs are explainable, optimization becomes easy. If you can trace a significant monthly cost to a dashboard that drives major decisions, the cost is justified. If you can trace it to a dashboard that nobody opens, the decision is obvious.
When costs are not explainable, optimization becomes political. Someone has to decide what to cut. But nobody has the information to make that decision. So nothing gets cut. Or worse, the wrong things get cut: the valuable workloads that happen to be expensive, rather than the worthless workloads that happen to be cheap.
Explainability changes the question. Instead of asking "how do we reduce costs?", you ask "do we understand our costs?". The first question leads to reactive cuts. The second question leads to sustainable governance.
The Goal
Here is what I have come to believe after years of working with data platforms at scale.
The goal is not to reduce costs.
The goal is to make costs explainable.
Once costs become explainable, once every significant expense can be traced to an owner, a use case, and a business justification, optimization becomes almost trivial. The expensive-but-valuable workloads are protected. The expensive-and-worthless workloads are obvious. The decisions make themselves.
But getting there requires work that most organizations avoid. It requires mapping ownership. It requires tracing dependencies. It requires asking uncomfortable questions about workloads that exist because they've always existed.
It's easier to run another cost-cutting sprint. It's easier to find another query to optimize. It's easier to celebrate a 20% reduction that will evaporate in three months.
The hard path is building a system where costs explain themselves.
That's the path worth taking.
The ideas in this article draw from the Data Cost Stack framework, which analyzes platform costs across four layers: Usage, Flow, Design, and Ownership. The framework emerged from auditing data platforms where cost visibility existed but cost understanding did not.
