The most common way a mid-size operator gets its data function wrong is the way that looks most responsible on an org chart. You hire a data engineer, or maybe two, and you put them in a box of their own. A data team. They report up through IT or through a new “analytics” line, and every other part of the business, production, accounting, land, reserves, files requests to them. It feels like progress. It is usually the start of the bottleneck.
Thoughtworks put “standalone data engineering teams” on Hold in its Technology Radar last November, which is the strongest discouragement the Radar issues.[1] The reasoning is worth quoting because it names the pattern precisely: organizing a separate data engineering team apart from the business domains it serves is an antipattern that repeats the mistakes of isolating DevOps, testing, or deployment into their own silos. It creates knowledge silos, bottlenecks, and wasted effort. We have watched it happen inside operators who did everything else right.
Why the isolated team feels correct
The logic is clean, which is the problem. Data work is specialized. You would not ask a landman to write dbt models or manage an Airflow scheduler. So you concentrate the specialists, give them a queue, and let the rest of the business get back to its actual job. Centralization looks like efficiency. One team, one set of standards, one place to point when the monthly production number is wrong.
For a while it works. The first few pipelines get built, a warehouse comes online, someone finally joins ProdView volumes to Bolo accounting without a spreadsheet in the middle. Then the queue fills up. Every dashboard change, every new field, every “can you just add this one column” goes through the same two people who now spend more time triaging tickets than building anything.
The failure mode, in order
It fails in a predictable sequence, and we have seen every step of it.
First the team becomes ticket-driven. Work arrives as requests, not as shared goals, so the engineers stop understanding why anything is being asked for. They build what the ticket says instead of what the requester needed, because they were never in the room where the need was described.
Then they run out of context. A data engineer three org-chart hops from the reserves team cannot know that “the June forecast” means the pre-revision run, not the one ComboCurve spat out last Tuesday. So the pipeline is technically correct and business wrong, and nobody catches it until close.
Then they get blamed for everything. Any number that looks off is “a data problem,” which means it is the data team’s problem, even when the real fault is a source system or a definition nobody agreed on. The team that was supposed to accelerate the business is now the thing the business complains about. That is the antipattern the Radar is warning against, and it is not a competence issue. Good engineers land in this trap because the structure put them there.
The alternatives, sized to the org
There is no single right structure. There is a right structure for the number of data-producing domains you have and the amount of shared plumbing worth centralizing. Three broad shapes cover most mid-size operators.
No dedicated team, governed by domain owners. For a smaller operator, maybe a couple hundred wells, a handful of source systems, one or two people who know the data, the answer is often to not build a standalone team at all. The people closest to production own the production data. The comptroller’s group owns the accounting feeds. A light governance agreement, who owns what, what the definitions are, what “done” looks like, does more than a dedicated team would. This is close to what we described in data governance without a data team: you can have accountable, well-defined data without a department that owns all of it.
A platform team only. As you grow past a few domains, the thing worth centralizing is the infrastructure, not the products built on it. A small platform team owns the warehouse, the orchestration layer, the ingestion patterns, the credential management, and the standards. It does not build every dashboard or every mart. The domains build their own data products on the shared platform. This is the data mesh idea without the jargon: central plumbing, distributed ownership. The platform team’s customers are the other engineers, and its job is to make their work faster, not to do it for them.
Hub and spoke. The middle ground, and where most operators past a certain size land. A central hub holds the platform, the hard-won standards, and the senior engineering muscle. Embedded engineers, the spokes, sit inside the domains: one with the production group, one with reserves and accounting, one closer to the field and SCADA. They report into the hub for craft and career, but they work inside the domain for context. The hub keeps the standards from fragmenting. The spokes keep the work grounded in what the business actually needs. When we centralized a fragmented upstream data estate, the part that made the conformed well master usable was not the central platform alone. It was that the KPI layer got built collaboratively with the client’s own analysts, the people who knew what the numbers meant. That is hub and spoke in practice, even when nobody draws it on a chart.
The three questions that decide
You do not need a consultant to tell you which shape fits. You need honest answers to three questions.
How many domains produce data? If it is one or two, a dedicated team is overhead you do not need; govern it and move on. If it is five or eight vendor systems across production, accounting, land, reserves, and SCADA, the coordination cost is real and a platform or hub-and-spoke structure earns its keep.
How much shared infrastructure actually exists? If everyone is querying one warehouse, using one orchestration layer, and depending on one set of conformed dimensions, that shared surface is worth a team to own. If the domains are genuinely independent, with little overlap, centralizing the plumbing buys you less than it costs.
Can the organization tolerate distributed ownership? This is the one that sinks good structures. Domain ownership works only if the domains will actually own it, including the incidents and the definitions and the 2 AM page. If the culture is “that is the data team’s job,” you will recreate the silo no matter how you draw the boxes. Be honest about this before you commit to a mesh you are not ready to run.
Consultant or full-time hire
The structure question and the staffing question get tangled, and they should not be.
Hire a full-time engineer when you have crossed the volume threshold, roughly steady work for someone every week, and when the domain knowledge is specialized enough that ramp-up time pays for itself. We laid out those inflection points in when to outsource your data engineering, and they have not changed. The first full-time hire should be senior enough to set the structure, not just fill a ticket queue, because the wrong first hire is how you get the standalone team you were trying to avoid.
Bring in a consultant when the work is front-loaded, when you need the platform stood up faster than one learning-on-the-job hire can manage, or when you specifically want the structure designed by someone who has seen it fail a dozen ways. The build phase and the run phase have different staffing needs, and a consultant can carry the build while you hire the person who will run it. Build for that handoff from day one, which is the whole argument in building for the next engineer.
The mistake is treating the first data engineer as a hire when it is really a structural decision. Where that person sits, who they report to, and whether they own the data or serve it decides more about the outcome than their resume does. Put a strong engineer in a silo and you get an expensive bottleneck. Put an average one inside the domain, with a platform underneath and a standard to hold to, and you get something the business will actually use. Draw the org chart before you post the job.
Thoughtworks, “Standalone data engineering teams,” Technology Radar Volume 33 (Hold), published November 5, 2025. https://www.thoughtworks.com/radar/techniques/standalone-data-engineering-teams ↩︎