Why technically sound data science still fails – and what leaders should examine before blaming the model or the team.

By Atul Mohan, PhD

Atul Mohan, PhD, is a senior data and AI executive with more than 24 years of experience turning complex analytics into measurable business outcomes. A Princeton-trained data scientist, he has held leadership roles across American Express, Interpublic Group, Omnicom, Teachers Federal Credit Union, and Brainlabs, building and guiding teams at the intersection of data science, marketing analytics, governance, and AI strategy.

The technical work is usually fine. The problem sits somewhere the technical review never looks.

I have been brought into a number of organizations where the data science team was considered a disappointment. Expensive, slow, producing work nobody used. Sometimes the conversation had already turned to whether the function should exist at all.

In nearly every one of those cases the technical work was competent. In nearly every one, the actual problem was sitting somewhere that no technical review was ever going to find.

Here is the first thing I now ask, before looking at a single model. Can anyone in the business name a decision that was made differently in the last six months because of something this team produced?

If nobody can answer that, the modeling was never the constraint. And you can spend a year hiring better modelers without changing the outcome at all.

There are a handful of recurring patterns behind this, and they are worth telling apart because the fixes are completely different.

The most common one is that the questions reaching the team are requests, not decisions. Somebody asks for a lift analysis. The team produces a good lift analysis. Nobody’s behavior changes, because there was no decision waiting on it.

This is the hardest failure to see from inside, because everyone involved is working hard and producing output on schedule. The tickets close. The velocity looks fine. If you audited the team’s throughput you would conclude they were performing well, which is roughly what happens when the annual review comes around and nobody can explain why the function feels ineffective despite hitting every deliverable.

The fix is upstream of the team entirely. Nothing gets built without a named person who has agreed, in advance, to change something based on the answer. That one rule eliminates a surprising share of most backlogs. I have never once seen anyone miss the work it eliminates.

It also has a second effect that I did not expect the first time I imposed it. Analysts get better at picking problems. When you have to identify who will act on something before you build it, you start paying attention to where decisions actually get made, and that attention improves judgment faster than any amount of technical training.

The second pattern is that output arrives in a form nobody can act on.

A model score living in a table that somebody has to go query will not be used by a media buyer at nine in the morning. A dashboard behind a separate login gets opened during launch week and never again. This is not a failure of discipline on the business side. It is a failure to treat integration as part of the project.

Integration is where these efforts die, and it dies quietly, because it is the phase that gets deprioritized the moment the model is declared finished and the team moves to the next thing. There is usually a plan to come back to it. There is rarely a quarter in which coming back to it outranks whatever arrived in the meantime.

The third pattern is trust, and technical teams underestimate it more than anything else on this list.

Every operator who has worked alongside models has been burned at least once by a system that told them to stop doing something that was working. They remember it vividly, often years later, often with the specific campaign name attached. Handing that person a validation metric does not touch the memory.

What does touch it is showing them the failures first. Run the model in parallel with the existing process for a cycle, then walk through the cases where it was wrong before you present the cases where it was right.

Being the person who volunteers bad news about your own work is worth more than any accuracy figure you can put on a slide. Almost nobody does it, because it feels like arguing against yourself in a meeting where you are supposed to be persuading. It is the single highest return thing I know of in this line of work, and it costs nothing but nerve.

Underneath all three sits a fourth problem that is more boring and more important than any of them.

If marketing, operations, and finance are each working from a different definition of the same quantity, no model will ever be adopted. The first response to any result will be an argument about the denominator, and that argument will never resolve, because all three parties are right within their own definition.

Reconciling those definitions is unglamorous. It does not look like data science. It will not be what anyone lists on a resume. And it is frequently the highest return work available to a new leader walking into a stalled function. It is also the kind of work that gets done once and then holds, which makes it one of the few genuinely durable things a data team can produce.

I want to be careful not to be misread here. None of this is an argument for lowering the technical bar. A model that is wrong will fail no matter how beautifully it is integrated, and plenty of organizations have shipped confident nonsense into production because nobody senior enough understood the method well enough to challenge it.

The point is about allocation. Technical quality is necessary and nowhere close to sufficient, and the way most teams spend their time badly misallocates against that reality.

On a project I worked on years ago, the modeling took roughly three weeks. Getting the data into a state where the model could be built took months. Getting the output into the system where the decision was actually made took longer than the modeling and involved more negotiation than mathematics.

That ratio is not unusual. It is close to standard. And teams that are excellent at the three weeks, with nobody owning the rest, will produce impressive work that changes nothing.

There is an organizational design question sitting underneath all of this, which is where the function should report.

When data science sits inside technology, it gets measured on whether the model is correct. That is a real standard and a defensible one. But it is measured entirely on inputs, and a team optimizing for correctness will reliably produce correct work that nobody uses, because nothing in the incentive structure asks whether anyone used it.

When it sits inside a commercial function, it gets measured on whether the business performed. That is a harsher standard and a less fair one, since plenty of things outside the team’s control affect the number. It also produces different behavior. Teams measured that way spend more time on adoption and less on elegance, and adoption is the scarce resource.

Neither placement is correct in all cases and I am suspicious of anyone who says otherwise. But if your function is stalled and reports into technology, that is worth examining before you conclude the problem is the people.

If you run one of these functions, or you are thinking about hiring someone to, I would ask a different set of questions than the ones that usually get asked.

Not what techniques does the team use. Ask where their output lands, and who looks at it there. Ask what happened to the last three things they built. Not whether those things were good, but whether anyone uses them now.

Ask whether the team sits in the meetings where the problems get discussed, or whether they receive tickets after the problem has already been translated by somebody else into a request for an analysis.

The answers to those questions will tell you far more about whether this investment is going to pay off than any conversation about model architecture ever will.

About the author

Atul Mohan, PhD, is a data and AI executive whose work spans data science, marketing analytics, measurement strategy, and the translation of complex technical systems into business decisions.