I never thought AI could be good at strategy. Even though there has been some hype around that, if we go one level deeper into how GenAI models work and what makes a strategy good, we can easily figure out why they will always be suboptimal at strategy.

But, on the other hand, they will always be quite good at bad strategy. The distinction is real.

Let’s first understand the grains of a good strategy. And a bad one.

What makes a strategy good?

This is not something that I am spitting out of thin air. I am borrowing the framework of strategy god Richard Rumelt. According to him, a good strategy has three kernels: a diagnosis, a guiding policy and a set of coherent actions.

A diagnosis names or classifies the situation, linking facts into patterns and suggesting that more attention be paid to some issues and less to others. It basically answers: What is going on?

The guiding policy highlights the overall approach for overcoming the obstacles identified by the diagnosis. It creates advantage by anticipating the actions and reactions of others, and by reducing the complexity and ambiguity in the situation.

A set of coherent actions is then designed to carry out the guiding policy.

Rumelt has also defined bad strategy at length. Bad strategies are full of fluff. They are long on goals and vision. But a strategy should give you the tools to face the challenge. And, most importantly, it should identify the challenge at hand. Bad strategies don’t do that.

How do LLMs work?

Large language models like ChatGPT or Claude are trained on an enormous amount of data: books, websites, articles, code and much more. They learn statistical patterns in language, including how words appear together, how arguments are structured and how concepts relate to one another. From all that data that already existed.

They process text as tokens: small units that may be words, parts of words or even punctuation. The model is autoregressive, generating one token at a time (mostly!) based on everything that came before it. What feels like reasoning therefore emerges from a very sophisticated probabilistic system operating over a vast learned pattern space.

This is, of course, a generalised version of how LLMs work. But it is enough for the argument here.

Ill-structured vs structured domains

LLMs have seen massive uptake in work related to structured domains, such as software engineering or mathematics. Here, the past training data is quite helpful. The rules of the game don’t change.

The logic for writing a while loop is constant. 2+2 will always be four. The models are trained on that data, so they can produce the output when prompted.

Strategy sits in an ill-structured domain. The real, messy business world. Every case here is different and unique in some way. What helps here is cognitive flexibility theory. It says that in ill-structured domains, you don’t just learn one rule or framework and apply it everywhere. You learn a set of rules by criss-crossing the same terrain from multiple angles, building a library of cases, and picking the right lens depending on what you’re actually looking at. It’s the opposite of pattern-matching to the nearest precedent, which, awkwardly, is close to the only thing an LLM knows how to do.

And if you look closely at how LLMs are trained, cognitive flexibility is an awkward game for them. Not impossible, but it is not their natural mode either. Their strength lies in finding and recombining patterns from what they have already seen. Good strategy requires a more selective kind of pattern recognition: identifying which previous situations are relevant, where the similarities end and why the current situation may demand a different response.

The same facts can usually support several plausible diagnoses. An LLM tends to preserve many of them and produce a comprehensive answer. (which looks promising!)

Good strategy requires choosing one diagnosis to govern the others. And accepting what that choice leaves out.

Connecting the dots

Now say you are inside an organisation. Your organisation is facing an issue, and you want to devise a strategy around it. How can AI help in creating that good strategy? Especially with the underlying training logic that we just discussed.

For example, let’s consider a food-delivery app. Their growth has slowed down for the past 2 months. You need to devise a strategy to improve that situation. Now, you give the whole detailed scenario and ask an LLM for a strategy. It will probably suggest something along the lines of improving retention, introducing loyalty awards, new & better recommendations etc. All of these may sound sensible on the superficial level. But none of them is a strategy yet.

Once you investigate, you may find the actual issue, which can be narrower. Maybe the loyal-frequent users of the company are not ordering not because of the price or poor recommendations, but because delivery times have become unpredictable during the weekday office hours. These peak time deliveries contribute ~30% to the total order volume, but the company has not been able to maintain a stable service level during those hours.

Then the guiding policy might be to improve density and reliability in those neighbourhoods rather than chase growth everywhere. The coherent actions would follow: restrict the delivery radius during peak hours, recruit more delivery executives (maybe temps for weekdays) around those clusters and change restaurant incentives to reduce preparation delays.

An LLM can generate those actions once the diagnosis is given - sometimes. You would need to spend time feeding the info about the competitive landscape, business constraints, etc. But identifying the diagnosis is one of the biggest obstacles of Rumelt’s framework. It requires company-specific data, conversations with the ops team, judgement and an understanding of what is changing underneath the averages.

Without that diagnosis, what would the model do? It would generate fluff. And that would be convincing to many people who might not be well versed in what a good strategy should look like.¹ It would recommend something like customer centricity. Or Operational excellence. Or say sustainable growth. You get the drift.

This is why AI is good at bad strategy. Bad strategy is linguistically impressive, but it lacks the analytical diligence. It resembles the thousands of strategy documents, consulting decks, annual reports and leadership speeches that the model has already seen. And for that reason, it can reproduce those patterns exceptionally well. But more often than not, they would have no intrinsic value.

Is it totally useless? What can it help with instead? Market research, for one, surely. But even there, one needs to be more specific. “How is Paytm different from PhonePe?” might not yield a structured response like “Compare how Paytm and PhonePe make money, using their latest public filings, and separate reported facts from inference”. The latter gives the model a much tighter research task.

I like to think that human and GenAI outputs work something like this. To some extent. This is kind of my mental model of their respective defaults:

Human judgement is volatile. There are peaks and valleys. We produce plenty of terrible ideas, average ideas and occasionally something brilliant. And the path to brilliant ideas is often full of terrible ideas. There is no strict pattern. Expertise makes us more consistent. But even after that, the AI output is much more consistent.

That consistency is useful in many domains. Specially for well-structured domains. You may write suboptimal code one day - miss the ‘;’ or design something with O(N^2) complexity where a simpler solution already exists for that. AI will never make that kind of mistake. But good strategy is different. It sits above a threshold. It needs a diagnosis that cuts through the noise, a choice that excludes other choices and a set of actions built specifically for the challenge at hand.

The volatility in human output is an asset for strategy work. You need sharper, more coherent minds who can cross the threshold after months of interaction and sitting with other human beings & studying every angle. AI may keep missing it very reliably. Every damn time.

But for many non-resident Indians (NRIs), there is something different.

Recently, I was talking to a colleague who’d just wrapped up a European trip. They were overjoyed to be back in a place with an AQI of 130 from 9. During our conversation, I asked them what they missed the most. And what disappointed them about Europe?

The answer was kind of refreshing; I am not gonna lie. I was expecting the third point they made to be the primary one, like most do. House help & 10-min delivery.

This is something that has been sitting in an uncomfortable middle for me for a long time. I get the point about the double thank-you moment. You are paying someone for their services. You both are better because of that. The househelp is getting paid, otherwise they wouldn’t be able to. And you are not getting bothered about the chores, and can focus your energy somewhere else.

The other side is this: this is not a glamorous job at any cost. And this is a big bug of the Indian socio-economic scene. This shouldn’t be celebrated as a big feature that we often tend to do. Yes, they are doing better because of the opportunity, because I am paying. But they could’ve done far better if there was a better opportunity.

Anyways. I thought of quantifying my discomfort. And I went down a deep rabbit hole. I have spent a lot of time and tokens on this, more than I would have liked to. This probably needs multiple blog posts to do a proper narrative-driven summary.

In any case, it would be good to summarise a few things.

So the actual question. Is domestic help in India cheap? And Cheap compared to what?

Drawing a line in the sand is often important in analytics. So here is one quick assumption that I made: India’s PLFS data breaks down what Indians earn on average: self-employed workers make about ₹13,279 a month, regular salaried employees about ₹20,702, casual labourers about ₹12,750. I made a weighted average of how much of the workforce falls into each bucket (roughly ~ 56%, 24%, and 20%), and it turned out about ₹15,000 a month.

But that’s everyone. Street vendors, casual construction labour, salaried clerks- everyone is included there. Domestic help is a much narrower and more specific market. Not everyone can buy it, and it’s concentrated in T1 and T2 cities. Ask around in a Mega city, and the number you’ll actually hear for full-time domestic help is somewhere between ₹20,000 and ₹30,000 a month. So that’s what I’m using here: ₹25,000, the midpoint.

The obvious way to check if this is “cheap”: price a full-time domestic worker against the country’s own GDP per capita. And check if you can spot a pattern there.

And the naive answer surprised me. India isn’t the cheapest country on this list. It’s the most expensive, by a wide margin. A full-time domestic worker here costs more than the average Indian’s entire annual income. Not a large share of it. Even if we use the earlier discussed 15k number, the percentage would still be close to or more than 100%.

Nowhere else in the sample gets anywhere close, rich or poor. This holds almost everywhere where labour is priced locally, whether the country is poor or comfortably rich. It only breaks for a handful of places (Singapore, the Gulf, Hong Kong) where they don’t price domestic help off their own wages at all but rather import it from somewhere cheaper. Another country.

So the popular intuition, that help is dirt cheap here because we’re a poor country, goes for a toss if we take a look at GDP per capita numbers.

But comparing cost to GDP per capita has a slight problem. GDP per capita isn’t what a person actually earns. It’s the country’s total economic output divided by literally everyone, including your niece who’s still in school. There’s a sharper, more honest question hiding underneath the naive scatter plot.

How many hours of someone else’s work can an hour of your own work buy?

This is where it stopped being an interesting fact and started being a bit uncomfortable. Everywhere else in the sample, an hour of average work buys more than an hour of domestic help. In India, it buys less. An average Indian’s hour of work buys less than an hour of a domestic worker’s time, not more. (Singapore’s number is enormous, but for a completely different reason. That’s imported labour, as we have already discussed.)

Also - one thing to keep in mind here. Which is the crux of the matter. We already have established that domestic help is not a glorious job. But for a large section of the Indian social strata, even the opportunity of doing that in a T1 or T2 can feel aspirational.

So that’s the number underneath my discomfort, finally. It’s not that help is a bargain here. It’s that too many people don’t have a better option than to sell an hour of their time for less than it’s worth. This is, for sure, not a feature of how efficiently the Indian state has organised the economy - and that probably shouldn’t be celebrated.

This is just what a very low floor looks like, once you measure it. And the low floor might be lucrative for many.

While writing this post, I randomly remembered something an extremely annoying (he was complaining too much about Kolkata, his ex-wife, parents and many other things) Uber driver once told me: how can you tell if someone is really rich? Just see if they can hire a full-time driver.

Originally published on Ordinary Analysis.

Notes on the data

Chart 1 — Domestic help cost vs. GDP per capita (20 countries)

GDP per capita: IMF World Economic Outlook, nominal 2025, all countries.

Domestic help cost (% of GDP per capita), by source confidence:

Stated assumption: India, ₹25,000/month, full-time domestic help in T1/T2 cities (see main text for the PLFS comparison).

High confidence, real statutory/government figures: Philippines (DOLE Kasambahay wage order, NCR minimum), South Africa (Sectoral Determination 7 / BCEA minimum wage, R28.79/hr), Hong Kong (Migrant Domestic Helper Minimum Allowable Wage, HK Immigration Department).

Medium confidence, aggregator or agency-rate based: China (city-level ayi/cleaner rate estimates), Indonesia (Jakarta live-in ART estimates), Thailand (Bangkok live-in estimates), Mexico (empleada de planta wage estimates), Brazil (mensalista wage estimates), United States (full-time housekeeper/nanny salary aggregators), United Kingdom (housekeeper salary aggregators), Germany (Putzperle/go-quitt minijob-rate sources), Singapore (FDW wage, MOM-adjacent sources).

Low confidence, thin sourcing, treat as plus or minus 10 points: Vietnam, Turkey, Nigeria, Egypt (live-in domestic wage estimates from limited sources), Qatar, Saudi Arabia, UAE (migrant domestic worker wage, agency-adjacent sources).

Chart 2 — Hours of domestic help one hour of average work buys (8 countries)

Singapore’s ratio (8.46) reflects imported labour priced below Singapore’s own wages, not local wage compression. It’s structurally different from every other row in this table and shouldn’t be read as “cheap” in the same sense India, Brazil, or Germany are.

The weakest links in this table, flagged plainly: Brazil’s and South Africa’s average-wage figures, and the UK’s, don’t have a primary-source citation behind them. Everything else traces to a named statistical office or government wage order.

The plot that they had used was a bit deceiving & counter intuitive. It had the X axis starting from 20 (the highest value).

So, I thought of cleaning the plot a bit, in order to have a better clarity.

I reversed the X axis, and added all the numbers that they had mentioned in the table but were not included in the plot. After that, it looked like this:

Now, this cleaned plot, looked interesting to me.

• The cost vs accuracy kinda follow a concave curve. (logarithmic)
• Composer 2.5 (Cursor’s own model) looks like an outlier.

So, let’s run two simple linear regressions.

Let’s gather all the outputs in this table:

Now, we got some idea about this. Before summarising what is happening, let’s run another type of regression. Linear mixed effects model.

A linear mixed model is a regression that separates two things at once: an overall slope you care about (here, the effect of log(cost) on score), and group-level variation you want to account for but not estimate one-by-one (here, each model family’s own baseline). It’s useful whenever your rows aren’t fully independent, like model variants nested inside families, because it stops that internal correlation from making your effect look more certain than it actually is.

Here is the result:

Random effects:
 Groups   Name        Std.Dev.
 family   (Intercept) 6.843   
 Residual             2.873   
 
Fixed Effects:
(Intercept)    log(cost)  
     45.522        9.666  

Now the story time, what this means?

1. Cost does buy score, but the returns are sharply diminishing.

   All the models that we used, roughly agrees with this. Log(cost) beats raw cost if Composer is removed. And the slope of the mixed model also confirms it - doubling cost buys roughly 6.7 points ¹, consistently, once you control for which family you’re in.

2. Curios case of Composer

   Composer is developed by Cursor, and it broke the log(cost) structure completely. For the statistical analysis, we filtered it out as an outlier. But the more honest reading might be (if the numbers to be believed) : it is a well optimised cheap model.

3. Family matters

   The mixed model splits the leftover variance (whatever cost doesn’t explain) into two parts: how much comes from which family versus everything else. Squaring the standard deviations and comparing them, family accounts for ~85% of that leftover variation, noise only ~15%. In plain terms: pick two model variants at the same price, and which family they belong to explains most of the score gap between them.

So, putting together, the cost component has two parts. More spending gives you better accuracies, but the family of the model matters too. Family seems to be a coarse lever, you move from one to another, and get a bump in the accuracy. Turning efforts within a family seems like a finer scale.

One analogy I can think of - hiring vs overtime problem. Hiring the right people can yield better results than the wrong people working on weekends. No amount of overtime can fix that hiring gap.

This is where the judgement of the operator using the tool would be very important. You should ideally want to use a “weaker” model for more trivial problems. If you want to optimise the token cost. In data science, it is important to know which model to use when. This translates to that philosophy roughly - one needs to know which API call is needed for a particular problem.

There is something very familiar about this whole exercise. Like a cover song you can’t quite remember.

For a long time, the standard unit of employee productivity was presence. Think about factory work, where hours logged were directly proportional to the output. It takes 30 minutes to produce a product. You work for 9 hours, and you produce 18 products that day. You tap in at the beginning of the day, and then tap out when it is time to head home. If the floor manager sees you near your machine, he knows you are working.

The knowledge economy broke that logic completely. When the knowledge economy became the dominant economy, managers (with the old way of thinking) started losing sleep. Your output is no longer directly proportional to your input or hours logged. A developer who stares at the ceiling for 40 mins, then ships a feature in 3 hours, is not less productive than someone who takes 6 hours or a night of coding. This should have forced a harder question: what does productive work actually look like? Most companies don’t have this figured out. So they relied (and still do) on the hours logged metric.

In Hinduism, Vishnu had ten avatars across the yugas (eras). Each one suited to the crisis of its age. Bad management has its own avatar tradition. The factory foreman becomes the incompetent knowledge-work middle manager, who becomes the incompetent AI-adoption evangelist. Same underlying worldview. Different costume.

Now that GenAI has arrived, a similar pattern is emerging. The companies are asking themselves one question: “How to gauge if the employee is actually using GenAI?” and a similar suboptimal solution is being developed. Let’s start counting the tokens consumed. Prompts fired. And let’s put it on a leaderboard to encourage more people.

Jensen Huang recently suggested that if a $500,000 engineer isn’t consuming $250,000 worth of AI tokens annually, something is wrong. There is a version of this argument that makes sense. Powerful tools should be used. Under-leveraging AI is a real problem. But the moment you turn that into a numeric expectation and tracking, you are back in Goodhart territory.

There’s a concept in economics called Goodhart’s Law. The short version: when a measure becomes a target, it stops being a good measure. The moment you tell people that token consumption is what gets rewarded, token consumption is what you’ll get. Not better thinking. Not faster shipping. Not cleaner decisions. Just more prompts. Longer conversations.

Honestly, you can’t blame the people gaming the system. When you set wrong incentives, it is bound to happen. This is a failure that is creeping from the top.

Measuring the real impact of GenAI at work is doable. But it is hard. You have to go back to your boardroom, look at old JIRA tickets, and analyse that. What was the average time to ship features with X degree of complexity in the past? How much time is it taking now? What used to take three weeks, does it take ten days now? What used to need four revisions, does it need two? That kind of measurement is doable. It just requires intellectual rigour and some patience, and most organisations are running low on both.

You can do this outside the engineering work too. Are decisions getting made faster? Are they getting reversed less often? Are first drafts closer to final outputs? Are we spending less time in meetings trying to figure out what to do next? The focus on “Are people using AI more?” is the wrong thing to do. The better question is: is the cost of getting to clarity going down?

Vishnu, at least, had a purpose for each avatar. The form changed because the problem changed. The incompetent manager’s avatars share no such logic. The problem keeps changing. The solution stays the same: find a number, trust the number, reward the number.

The tap-in, tap-out in the knowledge economy never really measured work. It measured the anxiety of managers who didn’t know how else to look. Tokenmaxxing is just the same anxiety, wearing a different UI.

• Kolkata vs the Metros
• West Bengal’s Economy: Large by Size, Thin by Structure
• Health Outcomes
• Welfare Worked. Wealth Didn’t Follow.

The synthesis contains personal opinions and a summary of the patterns observed across 4 parts.

Read on Ordinary Analysis

I related a lot to the essay by Anurag Verma. He argued that most of our readings are utility driven i.e. we read because we have to.

Among all these debates and discussions, I felt we were missing one point: how is India different from other nations?

I don’t have a lot of foreign associates. But from the little knowledge, it didn’t feel like others were drowning themselves in books while India alone wasn’t reading. It is a contemporary issue that transcends borders, regions and seas.

So the question arose:

How do Indians read compared to other countries?

And the only way we can answer this is through data.

I looked at this from two angles:

• What is the buying pattern of different countries with respect to genre?
• How much, on average, is an Indian spending on trade books?

Let’s dissect the first one:

In terms of revenue share, Indians are spending far more on educational books than the US, UK or European countries. So Anurag’s point is not completely wrong. And intuitively, that makes sense.

But now, let’s look at it from another angle: as a percentage of GDP, how much are Indians spending on trade books (i.e. excluding educational books)?

An average Indian is actually spending more on trade books than an average US and European citizen. But less than the UK.

This shows that when it comes to buying non-educational books, Indians are more intentional compared to more developed nations. When you are spending more of your hard-earned money, especially in a country with a lower per-capita GDP, you need to be more intentional.

One methodological note: this analysis uses nominal GDP per capita as the denominator. Book prices vary across markets. A $10 US paperback might be ₹400-600 in India, depending on the publisher and format - but books are globally traded goods with some price convergence, unlike purely local services. Using nominal figures keeps the comparison straightforward, though a full PPP adjustment could be explored in future work.

As usual, reality is far from a clean black-and-white thing. It is much more nuanced. On average, Indians tend to read more for utility, but we also tend to spend more on non-utility books.

This also raises an interesting question: what happens when India’s per capita GDP rises? More surplus should lead to more spending on pleasure?

Notes & Sources

Trade books are commercially published books sold to the general public, excluding educational textbooks, academic journals, and professional reference materials. In this analysis, trade books = Fiction + Non-Fiction.

Data note: All book market figures reflect 2024 actuals from industry sources (Horizon Databook, FEP, Nielsen). GDP per capita figures are from 2024-2025; US figures use 2025 IMF projections. Year-on-year differences are negligible (<2%) and do not materially affect the conclusions.

Calculation methodology: Trade book spending per capita was derived by multiplying total market size by the trade book revenue share (Fiction + Non-Fiction %), then dividing by population. This was then expressed as a percentage of nominal GDP per capita.

The plot uses the mean of each range as the representative value.

India

• Market size: $10.37B (Horizon Databook) [1]
• Educational: 60.2% mean revenue share (Horizon [1], IBEF [2])
• Fiction: 17.5% revenue share, +30.7% YoY growth [1][2]
• Children’s: 9.8% revenue share [1]

United States

• Market size: $40.44B (Horizon Databook) [3]
• Fiction: 32.8% ($3.26B), +12.6% YoY (AAP) [4]
• Children’s: 24.7% [3]
• Educational: 19.8% [3]
• Non-Fiction: 19.2% ($2.88B), +1.3% YoY (AAP) [4]
• Religious/Professional: 3.5% [3]

United Kingdom

• Market size: £1.82B physical (Nielsen) [5]
• Fiction: 42.5% mean revenue share, record high, +18% YoY [6]
• Non-Fiction: 27.5% [7]
• Educational: 17.4% (Horizon) [8]
• Children’s: 12.6% [5][8]

Europe

• Market size: €24.9B (FEP) [9]
• Fiction: 27.5% [9]
• Educational: 23.3% mean (FEP [9], Horizon [10])
• Non-Fiction: 22.5% [9]
• Academic/Professional: 16.7% [9]
• Children’s: 14.6% [9]

Citations

[1] Grand View Research/Horizon Databook. India Books Market Size & Outlook, 2025–2033. https://www.grandviewresearch.com/horizon/outlook/books-market/india

[2] IBEF. India’s Meteoric Rise as a Publishing Hub. https://www.ibef.org/blogs/india-s-meteoric-rise-as-a-publishing-hub

[3] Grand View Research/Horizon Databook. US Books Market Size & Outlook, 2025–2033. https://www.grandviewresearch.com/horizon/outlook/books-market/united-states

[4] AAP via Publishing Perspectives. December StatShot: US Book Market Up 6.5% Year-to-Date. https://publishingperspectives.com/2025/03/aaps-december-statshot-us-market-up-6-5-percent-year-to-date/

[5] Nielsen BookData. Bestsellers & trends in the UK & Ireland in 2024. https://nielseniq.com/global/en/insights/commentary/2025/bestsellers-trends-in-the-uk-ireland-in-2024/

[6] Friedman, Jane. Book sales update: UK market. https://janefriedman.com/book-sales-update-uk-market/

[7] LoveReading. Fiction Book Sales in 2024 Top The Lot. https://www.lovereading.co.uk/blog/fiction-book-sales-in-2024-top-the-lot-while-non-fiction-lags-behind-9226

[8] Grand View Research/Horizon Databook. UK Books Market Size & Outlook, 2025–2033. https://www.grandviewresearch.com/horizon/outlook/books-market/uk

[9] Federation of European Publishers (FEP). European Book Publishing Statistics 2024. https://www.fep-fee.eu/European-Book-Publishing-Statistics-2024

[10] Grand View Research/Horizon Databook. Europe Books Market Size & Outlook, 2025–2033. https://www.grandviewresearch.com/horizon/outlook/books-market/europe

As an Analytics professional, it is our job not to accept whatever is propagated by the belief system blindly. Rather, do our investigation, and then accept, reject, or refine it.

So, I thought, what if we try to see what is actually happening here? In this blogpost, I will discuss my approaches for the same.

But before asking any questions - let’s have a look at the data. What is the gap between the Conte of domestic league and Europe?

For our analysis, we have also taken Pep and Klopp (Liverpool & Dortmund) - so that we can understand how he compares with the other two. This will help us in putting things into more context.

As we can see, there is a significant difference in the win % for Conte - compared to Klopp & Pep.

So the question arises:

Is this a fluke or statistically significant?

A simple Z-test can tell us if the gap in Europe vs Domestic League is just a fluke or if there is some statistical significance.

Turns out - it is highly significant!

The difference is even significant for Pep. Now if you think about it - that is expected. UCL is a tougher competition. So, we would expect to see some reduction in the winning percentage.

For Klopp, though, it is not significant.

How different are the tactical profiles?

In the first section, we looked at winning % only. Now, let’s examine the complete tactical profile - how the full distribution of wins, draws, and losses shifts between competitions.

The Chi-square test helps us understand if the entire result pattern changes, not just the win rate. And interestingly, even Klopp shows a significant difference here despite his Z-test being non-significant. This suggests his teams play differently in Europe (fewer draws, more decisive results), even though his overall win rate stays similar.

So far, all three managers show some form of difference between their league and UCL performances. But here’s the key question: Is the difference abnormal, or is it just the “Elite Tax” of playing in Europe?

The Elite Tax: Accounting for Expected Difficulty

Now, let’s switch from a frequentist lens to a Bayesian one.

Here’s the fundamental insight: UCL is harder than domestic leagues. You can’t really treat playing Everton and Real Madrid at the same level. So we should expect some performance drop when managers face Europe’s elite. The question isn’t “is there a difference?” but rather “is the difference larger than expected?”

The Methodology: The 10% ROPE

I introduced a Region of Practical Equivalence (ROPE), which I call the “Elite Tax.” I am granting every manager a “pardon” for a 10% drop in win rate. If their win rate drops by 10% or less, we consider that a normal byproduct of elite competition.

I then ran 100,000 simulations for each manager to calculate the probability that their performance drop is abnormal (greater than 10%).

If a manager’s drop is within the 10% zone → Normal (performing as expected given increased difficulty)
If a manager’s drop exceeds 10% → Abnormal (genuine UCL problem)

The Results

What This Means

Pep Guardiola: The Benchmark. Pep’s distribution is centered almost perfectly on our 10% “Elite Tax” line. With a 48% chance of exceeding that threshold, his performance is essentially a coin flip. Mathematically, he is performing exactly how an elite manager should in a tougher competition.

Jürgen Klopp: The Outlier (The Good Kind). Klopp’s entire distribution is tucked safely to the left of the 10% line. There is only a 23% chance that his drop-off is abnormal. But here’s the interesting part: his Z-test showed no significant win rate difference, yet his Chi-square test was significant. Why? Because his tactical approach shifts in Europe - fewer draws, more decisive results - even though his overall win rate stays similar. He doesn’t win less in the Champions League; he just plays differently. More high-stake matches, all-or-nothing football. If anything, Klopp is competition-proof; his tactical identity survives the jump from domestic leagues to Europe better than Pep or Conte.

Antonio Conte: The “Statistical Glitch.” Conte lives in a different zip code. With 98.7% certainty, the model confirms that his European drop-off is abnormal. Even after accounting for his smaller UCL sample size (which makes his curve wider), there is almost zero overlap between his reality and that of Pep or Klopp.

How to read the plot: Don’t just look at the peaks of the curves. Look at where they don’t overlap. Pep and Klopp’s performance distributions live in the same neighborhood. Conte’s distribution doesn’t even have a view of their street.

The Verdict

Statistics confirm what the eye test suggested: Only Conte has a genuine UCL problem.

While all three managers show some performance difference between competitions (as they should - the UCL is harder), only Conte’s gap is abnormal after accounting for the increased difficulty. Pep and Klopp are performing within the expected range for elite managers facing elite opposition.

The “Tale of Two Contes” is a real, quantifiable pattern that has persisted across 50 Champions League matches and 618 league games. Whether it’s tactical inflexibility in knockout rounds, squad depth issues, or something else entirely - the numbers make one thing clear: domestic dominance doesn’t automatically translate to European success.

Link to the notebook: https://github.com/shibaprasadb/datasignal/blob/main/conte_record/conte_record.ipynb

This needed production-level accuracy. But the simple API approach wasn’t cutting it. After testing 50 cases, I was only hitting around ~75% accuracy (37 out of 50). For production, that’s a non-starter.

The Problem with Single API Calls

The LLM was doing the task correctly for some instances but missing elements in others. Sometimes it would catch all 10 items, other times only 7 or 8. The pattern was clear: when it failed, it undercounted. It never hallucinated extra elements or went above the true count. It just missed things.

This directional bias turned out to be the key insight.

So I “Random Forest” It

I decided to apply the “wisdom of crowds” principle. The same concept that makes Random Forest work. Instead of relying on a single API call, use multiple calls and aggregate intelligently.

The evaluation rule was simple: Max(API_call_1, API_call_2, …, API_call_n)

Example: If there are 10 elements and three API calls return [7, 10, 3], the final output is 10.

Why this works: The undercounting errors get filtered out. The max function naturally finds the correct answer as long as at least one call succeeds. Since the LLM never overcounts, the highest value is almost always the right one.

Here’s how the two approaches compare:

The Math Behind It

With a single API call, the question is: What’s the probability of success?

With ensemble, it becomes: What’s the probability of at least one success?

The math changes drastically:

P(at least one correct) = 1 - P(all calls wrong)

For n=3 calls with p=0.75 success rate:

• P(all wrong) = (1-p)ⁿ = 0.25³ = 0.015625
• P(at least one correct) = 1 - 0.015625 = 98.4%

Going from 75% to 98.4% with just 3 calls? Not bad at all.

Finding the Sweet Spot

But I couldn’t just pick any number. Each API call costs money and adds latency. I needed to balance accuracy against cost.

Here’s how the numbers break down:

The diminishing returns kick in hard after n=3. Going from 98.4% to 99.6% costs an entire extra API call for just 1.2 percentage points. But for production-level reliability, I decided that extra margin was worth it.

I settled on n=4: 99.6% accuracy at 4x the cost.

When This Breaks

This approach only works because my LLM had a directional bias (like undercounting - in this case). The evaluation function must match your error pattern:

• Undercounting errors → Use Max()
• Overcounting errors → Use Min()
• Random errors (sometimes high, sometimes low) → Use majority voting with odd n (3, 5, 7) to avoid ties

The key is understanding how your model fails, not just that it fails. Directional biases can take many forms - summarization models that are consistently too brief, classifiers that favor certain categories, extractors that miss edge cases. Each needs its own aggregation strategy.

If you don’t understand your failure mode, you’re just burning money on redundant calls.

(Now that I think about it, we can dedicate a separate blog post on designing Eval functions)

The Takeaway

Sometimes the best solution isn’t a better prompt or a bigger model. It’s understanding your failure mode and exploiting it mathematically.

A single API call gave me 75% accuracy. Four calls with a simple Max() aggregator got me to 99.6%. Same model, same prompt. Just a smarter approach.

The real lesson? When you can’t improve the model’s performance, improve how you use it. In a constrained space, solving a problem becomes more interesting.

Still, the 3 lakh number is huge. Especially when we are far from hitting below the replacement level TFR. In any case, I got curious and wanted to explore what is happening with other states.

State Selection

Taking a look at 28 states would increase the noise more than the signal. And I don’t intend to publish a comprehensive comparative study of all the states. So, I cherry-picked a few, just to see how they’re doing:

• Tamil Nadu: Often cited as a model state for educational outcomes. Good to have them as a benchmark.
• Bihar: Eastern Indian state, West Bengal’s neighbor. Most populous state in the East. The lowest per capita income in the country.
• Maharashtra: Western Indian state. Highest state GDP. Mix of urban and rural demographics.
• Haryana: Northern Indian state. 7th richest in terms of per capita income.
• Uttar Pradesh: Bihar’s neighbor. 2nd most populous state overall. Similar socio-economic profile to Bihar in many ways.

This selection gives us a mix of geographical regions, economic profiles, and educational performance levels - enough to see if West Bengal’s trend is an outlier or part of a broader pattern.

Let’s have a look at the absolute numbers - and their trends.

(Note: the numbers are quite decentralized. So I had to collate them from several sources. Directionally, this should be quite accurate.)

Absolute Numbers: Trends from 2021–2025

What stands out:

For Haryana and West Bengal, there has been a decline for 2 consecutive years. But others have shown a consistent pattern overall. For UP, the pattern is interesting - there’s some volatility, but it is still stable. No sharp decline like West Bengal.

The West Bengal drop is particularly dramatic: from approximately 800K students in 2021 to around 470K in 2025. That’s a 41% decline. Haryana shows a similar downward trend, though less severe - from about 230K to 195K (a 15% drop).

Maharashtra, Tamil Nadu, Bihar, and Uttar Pradesh remain relatively stable, with only minor fluctuations around their baseline numbers.

Summary: Year-on-Year Changes

The table makes it clear: West Bengal’s decline is not just steep, it’s an outlier. No other state comes close to this magnitude of drop.

Normalizing by Population

Let’s take a look at the number of exam takers per 1,000 population:

Data Signals:

Maharashtra maintains the highest rate throughout (around 12–13 students per 1,000 population), suggesting either better retention rates or favorable demographics. West Bengal’s rate drops dramatically from about 8 to under 5 per 1,000 population - confirming that this isn’t just a population effect, but a real decline in participation rates.

This chart reveals interesting patterns, but comes with a caveat. This has an implicit assumption: the proportion of the eligible age group (16-18 year-olds) within the total population is uniform across states. Which, obviously, is not true. States with younger populations will have proportionally more children under 10 and fewer teenagers, while states further along in demographic transition will have a higher share of the 16-18 cohort. But we don’t have more granular data readily available. At least, I couldn’t find one. Ideally, we should be looking at the base of 16-20 or 15-19 age cohorts. That would give us a better idea about the “eligible” group.

Another Proxy: Youth Population Base

The closest we can get to the ideal metric is the population of the 0–14 age group in 2021, which was published by Data for India. This can serve as a reasonable proxy for the eligible age group in 2025.

What if we try to see the number of students appearing per 1,000 youth?

What this reveals:

It looks far worse for West Bengal. But this is a total guesstimation - at best. Here, ‘Youth’ is proxied by the 0–14 population share in 2021. By 2025, this group spans roughly ages 4–18. It’s not a perfect match to the Class 12 eligible cohort (≈17–18 years), but directionally it reflects the size of the feeder base.

Maharashtra leads with 58.0 students per 1,000 youth, followed by Tamil Nadu (50.0) and Uttar Pradesh (37.7). West Bengal sits at just 22.6 - trailing behind Haryana (26.7).

This stark difference (58.0 for Maharashtra vs 22.6 for West Bengal) suggests one of two things:

• Massive dropout rates between lower and upper secondary levels in West Bengal
• Demographic differences - West Bengal might have a younger age structure with proportionally more children in the 0–10 range than the 10–14 range

Most likely, it’s a combination of both.

What Could Be Happening?

The 2017 Policy and the Overage Bulge

If the age mandate was introduced in 2017, requiring students to be 10 years old in Class 5, those students would hit Class 12 around 2024–2025. The timing checks out perfectly.

But here’s a critical question: Was there a bulge of overage students in earlier years that’s now correcting?

In many Indian states, particularly in rural areas, it’s not uncommon for children to start school late or repeat grades. If West Bengal had a significant population of unqualified students in the system pre-2017, the 2021–2023 cohorts might have represented this bulge working its way through. The 2024–2025 drop could then be the system “normalizing” to age-appropriate enrollments.

This would explain why the drop is so dramatic - it’s not just one year’s worth of students, but potentially 2–3 years’ worth of overage students who would have been in the system under the old regime.

Why Haryana?

Haryana’s decline is notable but less discussed. I haven’t found evidence of a similar age-mandate policy there. This warrants investigation. Possible factors could include:

• Migration patterns (families moving for work)
• Shift to private schooling or alternative examination boards (CBSE, ICSE)
• Economic factors affecting school retention

Future Work

This exploratory analysis raises more questions than it answers (which has been my goal anyways for this newsletter):

• Granular age-cohort data: Getting actual 15–19 or 16–20 population data by state would dramatically improve the accuracy of per-capita calculations.
• Covid’s shadow: Does the pandemic have something to do with this? The 2024-2025 cohort would have been in Classes 9-10 during 2020-2021 (peak Covid years). If the pandemic disproportionately affected rural schooling (due to a lack of digital infrastructure), we might see this reflected in the numbers. Breaking down the data by urban-rural divide would help determine if Covid-induced dropouts are part of the story.
• Pre-2017 enrollment patterns: Analyzing the age distribution of students in West Bengal’s secondary schools from 2015–2020 would reveal if there was indeed an overage bulge.
• Dropout analysis: Where exactly are students dropping out? Between Class 8–10? Or 10–12? State-level progression ratios would be illuminating.
• Cross-board comparison: Many students in urban areas take CBSE/ICSE boards instead of state boards. Are West Bengal’s numbers declining while CBSE enrollment is rising?
• Haryana deep-dive: Understanding what’s driving Haryana’s decline could reveal factors beyond policy changes - economic trends, migration patterns, or shifts in educational preferences.
• Long-term tracking: Will West Bengal’s numbers stabilize at this new lower level, or continue to decline? Data from 2026–2027 will be crucial.

Conclusion

West Bengal’s ~37% decline in Class 12 exam takers from 2024 to 2025 is unprecedented among major Indian states. While the 2017 age-mandate policy provides a plausible explanation - a correction after years of unqualified students in the system - the sheer magnitude demands deeper investigation.

As someone from West Bengal, these numbers are both surprising and worrying. They point to deeper structural issues in educational access, retention, or migration that go beyond a single policy change. Whether this is a one-time correction or the beginning of a longer-term trend will become clearer in the years ahead.

If you found this worth your time, please subscribe to The Data Signal - it’s free. I explore data, AI, analytics, and strategy, tackling interesting questions that don’t have obvious answers. It would mean the world to me, knowing that someone is finding value in my work.