The assumptions behind the epidemiology. The timing of the lockdown. Test and trace. The question of masks. The medical preparations. The economic rescue plan. You name it, the policy is probably way off.

But in what direction, I couldn’t tell you. Nor can anyone else — yet. It’s called the novel coronavirus for a reason. We’ve never faced a disease quite like it before. We’ve never shut down the economy before. This is uncharted territory and every government in the world is blundering about trying to find a way through.

So how much blame does our government deserve for the mistakes that it has made, is making and will make?

Suggested reading

Which epidemiologist do you believe?

By Freddie Sayers

I hope that one day we’ll have the luxury of looking back on all of this as a historical event, not an evolving crisis. We’ll then have the time and space for royal commissions, public inquiries and all the rest of it. No doubt the 20:20 hindsight brigade will be out in force. So will the vulture press, picking over the bones — and looking for fresh meat too. Heads will be called for and, I fear, sacrifices offered up. My hope, however, is that we’ll be less interested in the what, when and who of the mistakes, and more interested in the why.

Where the explanation is one of incomplete information and limited options we need to be forgiving — always asking ourselves if we would have done any better in the same circumstances. I also suspect that the worst errors are rooted in a culture of government for which there can be no scapegoats because it’s goats all the way down.

These are the errors that come not from too little information, but from too few sources of advice, not enough experimentation, insufficient verification, and an unwillingness to admit to mistakes or to change course. In his interview with UnHerd, the epidemiologist Neil Ferguson said that “there’s a tendency to become wedded to a position you’ve taken and find it difficult to revise views in terms of new evidence coming forward”.

This applies as much to politics as it does to science. And perhaps especially to political decisions made on the basis of scientific advice (given the gravity of what’s being decided).

With great power comes great responsibility — and I’ve no doubt that our leaders do feel responsible. You can see the weight of it etched on their faces. But it is not the only thing they need to feel. Power must also be tempered by humility. And that’s what I sense is missing here — if not from the soul of each and every minister, then from the culture of government (and politics and journalism) in which they operate.

True humility is not cowering passivity, but a recognition of the radical uncertainties that we have to live with. It is only once we’ve understood that reality that we begin to act appropriately — by accepting that most of what we do will be wrong to some extent, and therefore actively seeking out evidence of that wrongness. It’s only on such a basis that we can put a stop to what doesn’t work and learn from what does (no matter whose idea it was in the first place). Above all, we must keep our options open. The worst thing we can do, especially when faced with huge but unknowable risks, is to make one big bet on being right.

And yet that’s exactly what our system of government encourages our politicians to do.

*

Systems, of course, can be reformed — but there’s one thing that never changes and that’s the corrupting effect of power itself.

In Nineteen-Eighty-Four, George Orwell showed us what power at its most corrupt looks like. These are the chilling words of O’Brien, the secret policeman who breaks poor Winston Smith:

“Power is not a means, it is an end. One does not establish a dictatorship in order to safeguard a revolution; one makes the revolution in order to establish the dictatorship. The object of persecution is persecution. The object of torture is torture. The object of power is power.”

The context of the novel is the totalitarian state of Oceania. Our own situation, despite the lockdown, is a far cry from that grim parallel universe. Power, as exercised in a democracy, is thankfully not absolute — and thus neither is the corruption that follows in its wake.

Suggested reading

Tolkien's guide to contemporary politics

By Peter Franklin

Indeed, power is often illusory — our leaders are more Wizard of Oz than Big Brother. Nevertheless, the Orwellian idea of power for its own sake is still relevant to the smoke and mirrors of post-modern politics.

Let’s imagine O’Brien not as a member of the thought police, but as a comms consultant in our world. This is how he might describe our system of government:

“One doesn’t use whatever power one has to do something; one does something to show that one has power.”

What that something is — be it an ‘eye-catching initiative’, a full-blown government strategy or even something genuinely important like a single currency or a foreign war — doesn’t really matter. What counts is not so much the destination as that the wheels have been set in motion: the civil servants have something to implement; the press officers have something to press release; the journalists have something to report; the parliamentarians something to debate. Everyone, if not happy, is at least occupied — and none more so that the minister, standing over the machinery of government, lever proudly in hand.

Policy-makers do love their ‘levers’. And politicians love to pull them. In fact, they’re always on the pull — looking for some simple way to make something happen and therefore look like they’re in control.

In the UK, no department is keener on its levers than the Treasury. And no wonder — in no part of government is there a more reliably mechanistic correspondence between a ministerial decision and some meaningful result. If the Chancellor of the Exchequer raises or lowers this or that tax, then up or down it goes. If he decides to borrow more money, it gets borrowed. If he wants to cut expenditure, it gets cut.

But much, if not most, of what government does or tries to do is not like that. There’s no direct relationship between a decision made in Whitehall and an outcome in the real world. An education minister may try to boost literacy, a Home Office minister try to crack down on knife crime; but though money might be allocated to the chosen objective and new powers legislated for, there’s no guarantee that an actual difference will be made. The government machinery doesn’t always connect with the facts on the ground. Indeed, forget the real world: bureaucratic inertia can disconnect the machine itself from its levers — leaving ministers pulling away to no effect.

Suggested reading

Did anyone predict coronavirus?

By Tom Chivers

In recent decades, governments have become obsessed with ‘delivery’ — with making sure that the machine does respond. As part of this we’ve seen the Treasury taking an increasingly assertive role in the work of the spending departments — as if the lever-pulling magic of fiscal and monetary policy can somehow rub off on those parts of government whose writ runs beyond spreadsheets. Under Boris Johnson, we’ve seen the Downing Street operation take effective control of the Treasury — as if to ensure that control over the state is ultimately exerted from Number 10 not Number 11.

*

The great problem with this focus on levers and delivery is that it shrinks perspectives to internal workings of the machine — instead of looking at it from the outside and asking questions about the system as a whole.

Obviously, there’s the question of centralisation versus localisation. If you want to make changes across the country, can you really do it via a machine that is remotely controlled from Whitehall? But there’s an even more important question than that. And what it asks is not ‘where?’ but ‘how?’. Fundamentally, the machinery of government can be configured to operate in one of two basic ways: the first is to implement what works (as reckoned by those in charge); the second is to find out what doesn’t work. Decision versus discovery.

There’s no doubt which approach our political culture prefers. As Nigel Lawson once put it, “to govern is to decide”. But on what basis should one decide? Historical precedent? Political expediency? Ideological prejudice?

How about experimentation instead? Why not redesign the machinery of government to systematically test many solutions to a problem, as opposed to the current configuration, which is about imposing just one. It is surely easier to see what doesn’t work in practice, than to guess what does in theory — and thus, by a process of elimination, get closer to the answer.

Why isn’t government massively more experimental? Is it a question of expense — a lack of time or the money? No — because as long as we fail fast and fail small (through lots of local experiments), we can minimise the downside (and subsequently maximise the upside by making the most of what was found to work). Rather, the real reason why we stick to the one-shot, top-down style of policymaking is a political culture in which you can’t be the ‘big man’ unless you take big decisions.

Suggested reading

Who will pay for this Covid catastrophe?

By Peter Franklin

And thus the machinery of government is engineered around the amplification of ego. Each part of the mechanism is valued and incentivised according to how well it serves this ultimate purpose. The flow of information back up the chain of command, which should be about revealing reality, instead responds to what the top of the chain wants to hear: confirmation that the right lever was pulled in the first place.

*

A prime example of this institutionalised confirmation bias was the original Troubled Families programme. This was launched by David Cameron in the wake of the riots of August 2011. At the time, there was enormous pressure on the Government to ‘do something’ — and the programme was the main response. Its purpose was to reduce truancy, anti-social behaviour, worklessness and other social problems thought to be concentrated among a targeted group of 120,000 households. It was rolled-out across the country, with hundreds of millions of pounds of funding.

By 2015, the Programme was reporting a remarkable level of success — with 98.9% of the troubled families supposedly having had their lives “turned around”. But, then, in 2016, an independent evaluation by the National Institute for Economic and Social Research surfaced, which was “unable to find consistent evidence that the programme had any significant or systematic impact.”

Payment by results was meant to provide the right incentives for providers (in this case, local government). But thanks to vague criteria, success could be declared (and payment collected) in most circumstances. In its very design, the programme was self-confirming.

It was also a huge wasted opportunity. Local delivery could have facilitated local experimentation with a variety of different approaches, involving a variety of different public, private and voluntary sector providers. But that would have made the Government — the Prime Minister in particular — look less decisive. The results would have been mixed, with some experiments failing completely. No doubt, the media would have portrayed each negative or mediocre result as a fiasco or even a scandal rather than as knowledge gained. As for any successes, those would have been seized upon as evidence of a postcode lottery — patchy provision privileging some communities over others.

Politically, it’s less risky for government to roll out one big approach across the nation, especially if its eventual failure doesn’t become apparent until long after the media has lost interest. Indeed nothing could be more risky than the fail fast approach of effective policy experimentation, because that generates news while the media is still paying attention.

Suggested reading

Brexit has exposed our education apartheid

By Matthew Goodwin

Housing is another example of a policy area where localisation could allow a wide range of different approaches — and where outcomes are readily measured and compared (houses are big and thus easily counted). But, astonishingly, this most place-based of political issues is still driven by demonstrably useless central government policies like the wretched Help to Buy scheme. Imagine if those resources had been made available to local authorities, housing associations and other potential innovators with a brief to try out literally groundbreaking ideas.

In some areas of policy we do see government taking forward a number of different approaches to a particular challenge. For instance, with energy policy, government has thrown its weight behind the development and deployment of a broad spectrum of power generating technologies. That’s good, but what’s less good is that policy doesn’t proceed to the next stage, which would be a robust evaluation of the different options, so that we can weed out the ones that don’t perform as well as the others. Instead, we see government stubbornly persist with projects like new nuclear power stations, despite the eye-watering costs and the availability of cheaper alternatives.

Because pulling the plug on a project looks like failure — and therefore an admission of weakness — expensive mistakes have a tendency to get more expensive and thus even harder to admit to. This is why evidence of failure is less likely to be sought out and may even be suppressed. At the very least, we can expect the chain of command — configured as it to the delivery of the big man’s big decisions, not to ask awkward questions.

*

How much this all applies to the Government’s handling of the corona-crisis remains to be seen.

When we eventually look back on the decisions that have been made, the most important question we can ask is not whether they turned out to be right or wrong — but whether, in Professor Ferguson’s words, the decision-makers “revised their views in terms of new evidence coming forward”.

It’s on this basis that our leaders and their advisors should be judged. Of course, so far, it is too early to tell. But if the verdict of history is a good one, it will be despite, not because of, our culture of government.

Suggested reading

Test article

By Mary Harrington

That’s according to a new study by Chris Chabris and colleagues, entitled “Objecting to experiments that compare two unobjectionable policies or treatments”. It’s a clever study – and, importantly, a large, preregistered one, which asked the same question 16 times in three different populations and found compatible answers, so it’s probably not garbage. What it did was look at how people feel about randomised experiments between unobjectionable policies or treatments.

That sounds a bit dry; here’s what it means. Imagine, for instance, a doctor’s surgery is trying to reduce deadly infections by providing doctors with a checklist of standard precautions to take before each procedure. The participants were asked how they felt if the checklist were displayed on the doctor’s badge, or on a poster on the wall. And they also asked them how they would feel if, instead, one room had the poster, and the other had the badge, and patients were randomly assigned to one or the other to test which worked best.

They asked the question four times of different groups, and on the whole, people were comfortable with either of the two options. People preferred the poster, but the percentage objecting to either poster or badge was never more than 25% and usually more like 15-20%.

Suggested reading

If you don't 'get' statistics it's just as bad as not 'getting' Shakespeare

By Graeme Archer

But when they were asked about randomly assigning patients to one or the other – “A/B tests” – then the percentage objecting leapt up, to 30% in one test and 40-50% in the other three. They found similar results if the question was about a doctor choosing between one of two FDA-approved blood pressure drugs or randomly assigning patients to one or another, and then on other questions involving autonomous vehicle control systems, genetic testing, online dating and several more.

Even when people don’t mind option A, and they don’t mind option B, they really do mind you randomly assigning people to one or other of the two. The effect was just as strong among people with science degrees and professionals in the relevant field.

This is, obviously, a problem. The A/B test described is essentially a randomised controlled trial, and an RCT is the best way of establishing whether some intervention actually causes the good (or bad) things you want to happen. It is the gold standard for medical research, and there is no good reason not to use it in other policy areas – you can randomly assign patients to have either a drug or a placebo, or you can randomly assign schoolchildren to either have phonics or traditional reading lessons. While the interventions are entirely different, the structure of the test is exactly the same.

Not all policy can be evidence-based, in the sense that the evidence decides the policy. For instance, should we build another runway at Heathrow? There’s one body of evidence that says that it will significantly increase the UK’s greenhouse gas emissions and will not be compatible with our commitments under the Paris Agreement. There’s another body of evidence that says it will create jobs and grow the economy. But there’s nothing that says you have to choose the climate over the economy; that’s a political decision, or even a moral one.

Suggested reading

Why I welcome our post-Truth era

By Giles Fraser

Sometimes, though – often – there is a clearly defined goal, and two or more possible options for achieving it. Most of us would agree that, say, it would be good if we could stop drug addicts from relapsing or reoffending. Imagine we honestly didn’t know whether locking up drug offenders was more effective at reducing their relapse rate or their drug-related crime rate than putting them into rehab.

The best way to find out which of those options is most effective would be to randomly assign drug offenders to one or the other intervention, and seeing which had better outcomes. But, apparently, that would be far less popular than simply arbitrarily picking one and going with it.

This is, on the face of it, strange, because for at least 20 years we’ve had governments that, at the very least, pay lip service towards evidence-based policy. The Blair government loudly repeated that “what matters is what works”: one thing it did was to set up the (excellent) National Institute for Health and Care Excellence (NICE) in 1999 to evaluate new medical treatments for cost-effectiveness.

But sometimes, lip service is all it is. When, recently, the government decided to push through an age verification system for porn sites, it commissioned an impact assessment which purported to look at how this ban would affect users. But the impact assessment was shonky as heck: it misrepresented the statistics it was quoting, was heavily reliant on a silly poll which made obviously untrue claims, and at one point apparently didn’t understand the word “correlational”.

Suggested reading

Why does our expert class fear democracy?

By Salvatore Babones

“The dirty secret of almost all government is that we don’t know whether we’re doing the right thing.” That’s according to David Halpern, the psychologist and civil servant who ran the UK government’s Behavioural Insight Team, aka the ‘Nudge Unit’, a programme designed to improve policy, in an interview with the website Apolitical.

“Whether what we’re doing is helping the patient or not helping the patient, or even killing him. People are often running blind, overconfident, and dispensing massive sums of money.” That’s despite 20 years of people making the right noises about evidence-based policy.

I had previously thought that this was largely a cynical thing, or an incompetence thing. Ministers want to look tough on drugs or porn, so they make a decision to crack down on them, and then retrospectively cobble together evidence to support that decision – policy-based evidencemaking, as I’ve heard it called. Or perhaps they just lack the understanding of what constitutes good evidence, and the skills to gather it.

But perhaps, also, it’s that they – like, apparently, around half the population – find the very idea of randomised controlled trials, the very thing that they need to do to best assess the effectiveness of the thing they want to do, deeply uncomfortable.

Chabris and his colleagues speculate that one reason behind this discomfort is “a lack of familiarity”: the RCT has only really existed for two centuries, which isn’t long for human brains, and they aren’t very good at thinking about them. Hopefully that’s right, and we’ll get more used to it. Because at the moment, people seem to be happier with barging randomly into a policy than with actually checking if it works.

Hans Rosling was a Swedish doctor, statistician, “data rock-star” (thanks to the advent of TED talks) and the author of the bestseller, Factfulness. It is Zen and the Art of Motorcycle Maintenance for 2018. In it, he explores why we get so many facts about the world wrong and what we can do about it – encouraging the reader to think beyond the obvious.

Since its publication in April, the book has acquired must-read status among Silicon Valley executives, thanks in no small part to repeated endorsements by Bill Gates. A friend of Rosling’s, he described it as “one of the most important books that I have ever read and an indispensable guide to thinking clearly about the world”. Gates – as only a billionaire can – then went further than most of us do with our favourite books. He gave a copy of Factfulness to all new graduates of colleges and universities in the USA, if they registered with his website.

It’s not surprising Gates loved it. Rosling’s counter-intuitive optimism is very Silicon Valley. The book recalled the golden age of the high-tech hub – the era when Larry Page and Sergey Brin, the founders of Google, could make “Don’t be evil” the motto of their company, and Steve Jobs’s Apple could have as its advertising slogan “Think Different” without (too much) widespread mockery.

Further reading

Test article

By Oliver Kamm

Like Jobs, the Swede, who died in 2017 (his children finished the book on his behalf) was part missionary and part showman. His mission was to shake us out of the state of ignorance that convinces us the world is getting worse – when, in fact, it is getting better.

Our overdramatic worldview, in his account, misses the big story of human progress. It manufactures a topsy-turvy reality, a “mega misconception”, whereby global poverty is growing, when in fact it is reducing, levels of education are worse rather than better, and violent crime is going up when the opposite is true. “The world has moved on,” Rosling says. “Our worldview hasn’t.”

He enjoyed playing with his audience to prove this. His preferred set-up was to give his audience of medical students, scientists or investment bankers a multiple-choice quiz to test their level of ignorance. Routinely, they would do “worse than chimpanzees” in answering questions designed to demonstrate his somewhat controversial idea that the “gap” between the rich and the developing world no longer existed. By “worse than chimpanzees” he meant that his audience’s results were worse than random, owing – he believed – to ignorance on a global scale how much the world was getting better. Most of the population, he argues, are somewhere in the middle.

Further reading

How to bring the tech giants to heel

By Mathew Lawrence

Even his audience at the World Economic Forum at Davos demonstrated a level of ignorance that was worse than that of a bunch of apes. Global political and business leaders, it turns out, did not know that the average world life expectancy is 70, that 88% of children are vaccinated against disease, or that in 2100 there will be 2bn children below the age of 15 — the same number as today.

To demonstrate his point, Rosling would use animated graphs. Multicoloured bubbles changed in size as they moved across the screen to show that, for example, life conditions in Sweden at the time of his birth in 1948 had similar levels of health and wealth as Egypt today, or that today the life expectancy and income of people in countries as different as China and Egypt are close.

By the end of his life Rosling had become culturally part of the same world as Silicon Valley. He rejected the “lone genius” stereotype, but was a TED talk phenomenon, had his own army of cultish followers and addressed Davos on sustainable development. He even sold the presentation software that he helped design to Google.

That same limitless optimism and energy bursts from the book. Along with a desire to shock. And each chapter starts with a multiple-choice test. In many ways its technological determinism reads like a therapy-cum-self-help book for hassled Silicon Valley executives who just want to be told that they are doing a good job.

Further reading

The robodoctor will see you now

By Mark Piesing

Rosling sets out to make these execs feel better about themselves. He shows how ignorance stops us from seeing the amazing story of human progress, largely created by technology. We can’t appreciate its wonders because of human instincts that have evolved over hundreds of thousands of years. Just as our bodies still want to hoard fat and sugar, we still want gossip and dramatic stories, which, in the past, transmitted information that might make the difference between living and dying. This craving for drama then helps to create an overdramatic worldview.

He maintains we are in a systematic state of ignorance about how much technology is making our lives better because selective reporting prevents positive news stories from making the front page. Instead, journalists, activists and lobbyists produce an unrelenting torrent of “bad news” that mislead us into thinking that the number of people dying from disasters is on the rise when it is on the decline, that there are more hungry people in the world when there are fewer, or in reverse, that there are fewer babies being immunised when there are more.

Negative news, he believes, shapes our worldview because our instincts mean that we pay more attention to scary things. The doom-mongers and manipulators he identifies are, of course, the same journalists and activists who frequently have Big Tech billionaires in their sights.

Further reading

The rise of post-truth liberalism

By John Gray

The problem is that Rosling’s optimism makes him comparably selective when it come to criticism of the tech tycoons and their products. Strangely, the possible hazards of social media and the other products of his billionaire friends are largely unaddressed, despite accusations that the algorithms they have created are amplifying fake news, widening divisions in society and undermining faith in the very facts to which Rosling points. He once said:

“I use normal statistics that are compiled by the World Bank and the United Nations. This is not controversial. These facts are not up for discussion. I am right, and you are wrong.”

Even if you ignore the messianic tone, Rosling may have been selective about the facts himself. Rather like the journalists or activists he disdained, Rosling, according to some critics, put a positive spin on “big picture stats” to back up his argument, while ignoring evidence that suggests that life is still grim for many.

He also glosses over some inconvenient facts: the population in many countries is growing; world hunger is on the rise; and while more girls are being educated, women with advanced degrees still have higher rates of unemployment than men. Almost half of humanity still lives in conditions that the average American, European, or Japanese would find unacceptable.

Rosling argues that the world is becoming more democratic even though many people don’t feel that it is: China’s economic growth has certainly not been matched by democratic reform. He shows us graphs of the numbers of washing machines in the world to demonstrate human progress, but has material progress made us happier? Others read the same graphs as an illustration of unsustainable growth that is destroying the planet. Some might say we were happier as a species when we lived in small, sustainable village communities – or even as simple hunter-gatherers.

Further reading

Silicon Valley's 'suicide pill' for mankind

By Mark Piesing

Most glaring of all, Factfulness shows no scepticism about the tech billionaires who have helped to create today’s uncertainties – which some might call chaos. Instead, it offers a sop to weary executives who have left it too late to buy a bunker in New Zealand, by assuring them that the apocalypse probably won’t happen. Their ideas and products, he says, are improving everything: technology is bringing our society and species together rather than breaking them apart. And any nagging doubt the techies themselves might feel isn’t their guilty conscience speaking, but down to the media.

Only instinctive, fearful thinking, according to Rosling, would suggest technology is likely to have some negative effects. He studiously ignores any plausible arguments that automation may cause mass unemployment and civil unrest, or that synthetic biology and CRISPR gene editing may create terrifying new artificial life forms and viruses. And he waves aside the idea that AI and body modification may lead to the creation of a new species of uber-humans that will enslave the rest of us.

Rosling could be right, of course, but the jury is still out. What we do know as  fact, however is that terrorism is increasing, more people are dying from war, polar ice caps are melting, and AI is seen by some to pose an even greater danger to the future of humanity than climate change.

At the moment, though, Rosling’s sword metaphor for technology isn’t so much a weapon against falsehoods that seem plausible; it’s more like the sword of Damocles, which hung by a single hair over the throne of Dionysius as a warning of constant peril and foreboding.

If Rosling is right, the “silent miracle of human progress”, as he described it, will continue. If it doesn’t, however, his unbridled optimism looks actively irresponsible. It is not encouraging that earlier this year Google removed “Don’t be evil” from its code of conduct.

The answer, is that we make prior assumptions about what (and who) is trustworthy and relevant. Furthermore, we tend to do so collectively – subscribing to the shared assumptions of the groups we want to be part of.

Fair enough. We need those filters, otherwise we’d be overwhelmed. The problem though is when the filter become so restrictive that it only allows through the information and ideas that confirm existing assumptions. From that point, it ceases to be a defence and becomes a cage – a bubble within which different perspectives have no place and new understanding becomes impossible.

Further reading

Don’t believe what those idiots are telling you about echo chambers

By Peter Franklin

Such bubbles are typically constructed from linguistic components – facts, theories, jargon, slogans, arguments, talking points, memes, cultural references. However, the thickest bubbles are made out of numbers not words.

While words are open to interpretation, numbers are an expression of objective truth: 2 plus 2 equals 4 and nothing else. We may fear the dark arts of statistical manipulation, but what could be more objective than a direct, unfaked observation?

Unfortunately, there are pitfalls there too. The point is made in an eye-opening piece by Aaron E Carroll for the New York Times. It concerns that most wholesome of corporate activities – the workplace ‘wellness’ programme.

By now, we all know that sedentary lifestyles aren’t good for us – and especially not in combination with mental stress. This, in turn, is bad for productivity, employee retention and the cost of providing health insurance. Hence the rationale for workplace-based schemes to promote healthier lifestyles.

But do they work?

“These… can offer screening for a variety of reversible conditions; access to weight-loss programs or gyms; encouragement and support; and sometimes even chronic disease management. Many of the analyses of these programs have shown positive results.

“Almost all of those analyses are observational, though. They look at programs in a company and compare people who participate with those who don’t”

Surely, that’s a perfectly fair test – a direct, unmanipulated measurement of the difference that participation makes.

There’s a catch though:

“The most common concern with such studies is that those who participate are different from those who don’t in ways unrelated to the program itself. Maybe those people participating were already healthier. Maybe they were richer, or didn’t drink too much, or were younger. All of these things could bias the study in some way.”

Of course, one can control for factors like age, sex, income etc. This requires a statistical trick or two, but as long as the effect is to compare like-with-like (e.g. slightly overweight middle-aged men with other slightly overweight middle-aged men) who could object?

Further reading

If you don't 'get' statistics it's just as bad as not 'getting' Shakespeare

By Graeme Archer

Only, there’s another catch:

“…we can never be sure that there aren’t unmeasured factors, known as confounders, that are changing the results.”

When it comes to getting fit, some people are just more motivated than others – just the sort of people you’d expect to volunteer for a wellness programme. Motivation, being an internal state of mind, is hard to measure objectively and thus control for – meaning that the participating group is likely to be biased to those most inclined to make the most of it.

Then there’s bias on the part of the organisers. If it’s your job to make a company wellness programme a great success, who are you going to encourage to take part – the health-conscious individual looking to get back in shape or the unrepentant, donut-scoffing flubba-wubba sat at the next desk?

There is a solution though: the randomised controlled trial – where participation is randomly assigned.

Carroll describes one such trial – of a wellness programme among university employees. This found no significant difference in outcomes between the intervention group and the control group. But here’s the twist:

“…the researchers also took the time to analyze the data as if it were an observational trial. In other words, they took the 3,300 who were offered the wellness program, then analyzed them the way a typical observational trial would, comparing those who participated with those who didn’t.”

When the trial was ‘de-randomised’ – i.e. based on simple observation of participants versus non-participants the hoped for differences in outcomes did appear. Furthermore, most of those differences remained even when factors like age and prior healthiness were controlled for.

Clearly, selection bias is a powerful thing.

Further reading

Wanted: a government willing to perform experiments on us

By Peter Franklin

There are some big implications here for public policy.

The idea that public resources should be allocated on the basis of evidence of effectiveness is a perfectly sound one. Furthermore, with the advent of ‘big data‘, the state is acquiring the ability to observe the outcomes of government intervention in greater detail than ever before.

However, the danger is that with so much data to play with, policy-makers will overlook the confounding factors of selection bias and other epistemic pitfalls.

Having left behind the dark age of pure guesswork, policy-making is set to enter a new age of false confidence.