The reality is that recruiting into the public sector is a ceaseless slog. Have you ever wondered why there are always so many adverts for life in the army? For teacher training? For the police? It’s not because recruiters like spending money. It’s because they are always in desperate need of talented new people, and such people are hard to find. These public services have to have high standards, whether that’s good qualifications, good fitness, or a clean criminal record. Pay isn’t great. Working hours are tough. Scrutiny is endless. And the UK has more people already in work than at any point since the early 70s.

No wonder the NHS has more than 100,000 vacancies. Social care has more than 110,000 vacancies. The police, the army and the fire service tend to get plenty of applicants, but they end up turning thousands away because they don’t measure up. The army met its recruitment target last year, but only after five years of failure — and after launching a new boot camp to help the unfit get up to standard before taking their entry tests.

Suggested reading

Could national service save higher education?

By Mary Harrington

Half the staffing battle isn’t recruitment at all: it’s stopping people from leaving. 15% of teachers leave in their first year; 30% within their first five years. In 2010 I helped plan for a coalition policy to recruit 4,200 extra health visitors. By the end of our first meeting, the plan had transformed: we had to recruit about 9,000 health visitors, because so many were planning to retire in the coming years. The recruitment drive was then ditched by the 2015 government — and as soon as it was, numbers slumped back, falling by more than 3,000 over three years.

It’s like trying to fill a bath with no plug in it. There are alarm bells ringing about how crappy it is to work in so many of these frontline jobs, and instead of fixing those problems, we’re exhausting all our energy desperately trying to recruit people who aren’t that fussed. The difficulty of finding good people for the public sector doesn’t mean we should give up trying. But it does mean we should try something different.

Instead of panicking about the high turnover rates in the public sector, we should see them as inspiration. It probably is too much to expect people to work in a front line public service job for the fifty years that will soon comprise the average working life. So let’s stop pretending these are vocations, or jobs that should take you from pimples to wrinkles. Soon the majority of us will live for a century: everyone could expect to have two or three careers in that time. So a spell in public service should be a part of every citizen’s life.

Suggested reading

Our shameful social care crisis

By James Bloodworth

Initiatives like Teach First and Now Teach have got the right idea: make it clear that public service is a great way to start out in the workplace; a great way to return to work after having children; a great way to find purpose later on in life; a great way to spend your last years before retirement. Dogs may be for life, but jobs are not any more.

It’s not good for our society to have one group of people who work exclusively in the private sector making the money, and another group spending it. It breeds resentment and misunderstanding. I’ve lost track of the number of senior business people who bleat that everyone in the public sector is a lazy and inefficient jobsworth. They imagine they’re the only ones who work hard, because they’ve no idea what it’s like to hold the hand of a dying patient, or coax ambition from a struggling teenager, or wrestle an attacker from their victim.

I’d like to make it compulsory: everyone should spend a minimum of five years working somewhere in public service. You wouldn’t be able to claim your state pension unless you’d done it. The days of conscripting all our young people into the army have passed, and I’m glad of it. We don’t need armed national service. But we do need a culture of civic service, a recognition that you need to do more than just pay your taxes to be part of a society. You have to be involved.

Suggested reading

Do police numbers really affect knife crime?

By Tom Chivers

The pathway should go both ways of course. Plenty of public servants have a toxic view of what life is like in business. They imagine rapacious greed and profit to be the only topics of conversation at the watercooler, and have an unhealthy contempt for words like “efficiency”. The civil service is slowly getting better at sending people out on secondment to businesses for a couple of years at a time, but there’s often still suspicion of anyone who does it. Rules set up to prevent grift end up acting as a huge barrier to anyone leaving, building up expertise about life outside government, and returning to make public services better.

By insisting everyone spend part of their career in public service, and part in the private sector, we could transform both. We could inject the values of purpose and service into our businesses. We could bring ideas and innovation into the heart of the public sector — and perhaps even make those jobs easier and more fulfilling to do. And we could build a culture of compassion and understanding between private and public sector workers, who might stop complaining about each others’ excessive pay or cushy conditions, and instead value the contributions each make.

So let’s stop with half measures — trying to recruit just enough people to fill the vacancies and then panicking five minutes later because someone retired. We should double, or treble, our ambition when it comes to recruiting teachers, nurses, doctors, soldiers and police officers. Let’s train everyone we can get our hands on. We have to make it normal for everyone to serve their country, whether it’s for five years or fifty.

Anyway, the story goes that a military AI was trained to detect tanks in photographs. It got shown lots of pictures, some with tanks in, some without, and it was told which was which. It worked out features that were common to the tank-containing pics, and then, when given a new picture from the same source, would use that info to say “yes, tank”, or “no, no tank”, as appropriate.

But apparently, when the AI was given pictures from a new source, it failed utterly. And it turned out that the AI had worked out that the photos with tanks in had been taken on sunny days, and the others on cloudy ones. So it was just classifying well-lit pics as “yes, tank”. When new pictures, taken by other sources which hadn’t been photographing sunbathing tanks, were used, the system broke down.

The AI blogger Gwern has tried to trace the story back, and it transpires there are multiple iterations of it: sometimes it’s saying tank vs no tank, sometimes it’s identifying Soviet vs American tanks; sometimes it’s ‘sunny days’ that’s the confounding factor, sometimes it’s the time of day, or that the film had been developed differently. Versions go back at least to the 1980s and possibly to the 1960s. Sometimes it’s Soviet tanks in forests, sometimes it’s Desert Storm.

Suggested reading

Do humans have a future?

By Tom Chivers

There another story about an AI set a task of telling huskie dogs from wolves. All the wolves in its training data were photographed on snow, so the AI learnt to call any animal photographed against snow a wolf. In this real story, the AI was deliberately badly trained, on deliberately badly chosen training data – it was a test. But when it was trained properly, it worked much better

This story is used to make the point that any AI is only as good as the data you train it on, and it is impossible to know how good the data you’re training it on actually is.

While this may be something of an oversimplification, it’s essentially true. The reason why AI is useful – or machine learning software that uses neural networks, which is usually what people mean by AI – is that it can work through absolutely vast amounts of data and find patterns and correlations that humans can’t. No human could go through that much information.

This capacity has profound impacts. In science, the huge amounts of data that are thrown up by, for instance, biomedical research, or astronomy, can be analysed to reveal previously unexpected links: genome-wide association studies found that multiple sclerosis, is in fact, a disease of the auto-immune system, like rheumatoid arthritis, even though it presents as a neurodegenerative disease like Parkinson’s or Alzheimer’s.

Suggested reading

Will the machines take control?

By Tom Chivers

But the trouble is that – pretty much by definition – it is impossible for a human to check those datasets. If a human could check them, you wouldn’t need the AI. And if a dataset is, for some reason, imperfect, then the AI will learn things from it that you won’t want it to. It won’t learn to say there’s a tank in every picture of a sunny day – modern image-recognition software is cleverer than that, and trained on much wider datasets, and there are ways around that sort of problem anyway – but it may have analogous, but more subtle and perhaps more insidious, problems.

That’s fundamentally the worry about AI being racist, or sexist. You might train your AI on some dataset of people who previously performed well at a job, or people who have or haven’t reoffended after release from prison, and those datasets contain large amounts of information about each person; years of previous experience, say, or number of previous offences – but also age, sex, ethnic makeup.

And if the training data tended to favour people of a certain sex or race, then the AI may learn to preferentially pick those people as well. It doesn’t even matter if the data isn’t the product of people being racist or sexist themselves. The training data may be full of people who really did do well at their job, or not reoffend – analogously to the training data correctly labelling whether or not a picture is of a tank.

Suggested reading

If you want to get ahead, take a risk

By Timandra Harkness

Then it may turn out that, for societal reasons, women or black people turn out to do, on average, less well on those criteria. Just as the apocryphal AI was able to categorise the tank-pictures by whether the pic was taken in sunshine, the hiring-algorithm AI might categorise potential hires by whether they are white or male.

Even if you don’t tell the AI people’s sex or race, it may not help, because it may be able to work it out to a high degree of accuracy from proxies – postcodes, for instance, or first names.

The job-interview AI apparently looks at facial expressions, and listens to the words the interviewee uses, and compares it to 25,000 “pieces of facial and linguistic information” taken from people who have gone on to be good at a job. (Incidentally, the description that the company’s CTO gives of the analysis of verbal skills – “do you use passive or active words? Do you talk about ‘I’ or ‘we’?” – is deeply suspect from a linguistic point of view. The idea that people who say “I” a lot are narcissistic is a myth, and people talk an awful lot of crap about “the passive”.)

But, again, the AI can only be as good as the training data. It may be that “people who turned out to be good at their jobs” were more likely to have certain facial expressions or turns of phrase, and it may well be that those facial expressions or turns of phrase are more common among certain ethnic groups. And “people who turned out to be good at their jobs” are, of course, people who got hired in the first place. It is almost impossible to remove hidden biases. (I would be intrigued to know how, for instance, autistic people, or disabled people, would do on this facial-expressions stuff.)

That is not a reason to throw the whole idea of AI in hiring, or other areas, out. Algorithms – even simple ones – have been shown to do a better job of predicting outcomes than have humans in a wide variety of areas. The AI might be biased, but they are only biased because the humans they are replacing were biased too.

Suggested reading

So you think you're open-minded...

By Tom Chivers

For instance, Amazon’s famously sexist hiring algorithm that got scrapped last year only undervalued female applicants because the human hiring decisions had been systematically undervaluing female applicants themselves. It’s not that getting rid of the AI will get rid of the bias. In that case, it made it explicit. Amazon now knows that its hiring practices were biased. But it is a reason to be extremely wary of throwing AI into the mix and saying, well, now we’re unbiased, look, an algorithm did it, so it must be OK. And it’s doubly the case where the AI algorithm is proprietary — so you can’t look into its guts and see where it goes wrong.

That’s key. I said at the beginning that it is impossible to know how good the data you’re training an AI on actually is, and that’s true. But there are things you can do to try to find out. There are researchers working on something called “provably correct software systems”, which are somewhat misnamed – you can’t ever prove that software is correct – but you can go in and check parts of the data, or the weighting of the nodes in its network, which can increase your confidence.

If an AI is owned by a private company which won’t let you go and check the data or the algorithms, though (as is the case with the hiring one, and some of the parole ones), it becomes very hard to be confident. So for the time being, it’s worth being very, very wary of anyone who says their fancy AI can tell you who’s going to be good at their job. You can probably trust it to identify tanks, though.