Solving the AI Paradox

We can’t let artificial intelligence replace human skill. Star Trek warned us decades ago.

The Automation Paradox

I recently read a great article from IEEE Spectrum titled AI Efficiency Could Cost Us the Next Generation of Experts. It’s well worth a read, particularly if you’re in the software industry – but I’ll summarize the key points here. AI systems like Claude and ChatGPT have become extremely capable at writing computer code – the instructions that tell all of our computerized devices how to do what they do. In the hands of an experienced, senior software developer, it’s an amazing force multiplier. It’s like having a team of tireless junior engineers who never complain, sleep or get distracted.

However, if we start letting AI do all the work, eventually the humans’ skills will get rusty and their memories will fade. The article explains this best with the aviation industry. Most commercial airplanes today can pretty much fly themselves, including landing under ideal conditions, and may even be able to do so more efficiently than a human. However, if humans cede this responsibility to the automated systems, they will be less ready when something goes wrong and a human is required to take over the controls. After accidents raised concerns about overreliance on automation and declining manual flying skills, the aviation industry implemented policies that encouraged pilots to constantly hone and practice their skills.

Skill Insurance

It would be tempting in the age of AI to hire fewer junior engineers and lean more on your senior engineers. But if you don’t hire and train entry level engineers, you will eventually run out of senior engineers. If we allow AI to write, debug and fix our software, these crucial skills will start to atrophy in the entire engineering team.

The better AI gets at performing critical tasks, the more deliberately we should invest in training and maintaining these skills among human workers. This doesn’t only apply to software engineers, but to pilots, air traffic controllers, ship captains, critical infrastructure plant operators, cybersecurity analysts, lab technicians, and many more occupations. We don’t have to completely forego the benefits of automation or AI, but we do need to invest the time and money to train humans who can competently take over when needed.

I’ll insert my Star Trek reference here. In season 2 of Star Trek in 1967, the episode “The Apple” depicted a people on Gamma Trianguli VI who had outsourced all work and even the governing of society to an all-powerful computer-god they called Vaal. They feed it fuel, it does everything else for them. The problem wasn’t just that they depended on Vaal – they’d forgotten how not to. Of course, Kirk’s solution was starving and then destroying Vaal with space lasers, forcing the populace to suddenly relearn to do everything and fend for themselves. Best of luck – bye! (Okay, the setup was good, but I’m not advocating for the same solution.)

Human Fallback

But we should extend this concept beyond humans. We need our critical systems to retain their ability to be effectively monitored, controlled and remediated by humans, too. The author of the article talked about a scenario involving the design of a nuclear power facility that could basically run itself, hands-free. But the engineers foresaw the above problem and planned to spend the extra money to retain manual controls and overrides. (The plant in question was never actually built.) I love the way the author put it…

We were solving a specific problem. An operator who only ever supervises automation slowly stops being an operator. The hands go cold. The mental model of what the plant is actually doing gets fuzzy. Then comes the day the automation hands control back. It’s always the worst day, because automation only quits when it’s confused or in trouble. But by then, you have a person in the chair who hasn’t truly operated the thing in years. The manual steps were there to keep the human current. It was inefficient by design, on purpose. [Richard Mitchell]

While it might be more efficient, we can’t design our critical systems to only be managed and controlled via automation (AI or otherwise). We need to retain meaningful human control and the ability to safely operate, isolate, shut down or repair the system when automation fails. Efficiency can’t be our only goal. Sometimes redundancy, unused capacity and even deliberate inefficiency are the price we pay for resilience.

That goes for cybersecurity defenses, too. AI attacks can now be largely automated, running with a speed and scale that will make it nigh impossible for a human to properly respond. AI defenses will likely be needed. But we still need humans in the loop – humans whose skills and knowledge are still sharp. After all, if I were an AI attacker, the first thing I would do is disable the AI defenses. We should absolutely use AI to defend our systems. But we shouldn’t design systems whose security depends entirely on AI successfully defending them.

Which leads me to my final point…

System Resilience

With critical systems, as in life, we hope for the best, but should still prepare for the worst. Systems will fail – parts wear out, components go out of tolerance, accidents and even environmental disasters happen. Traditionally, well-designed physical systems have included redundancies, sensors for continuous monitoring, alarms for notifying operators of potential problems, and manual controls and overrides. Whenever possible, systems are designed to fail safely.

In this new age of AI cyberattacks, we need to be much more rigorous and explicit about designing resilience into all of our critical systems – physical and electronic. AI changes the threat model because the attacker can be persistent, fast, scalable and extremely good at exploring vulnerabilities and exploiting them – even combining multiple exploits in a chained attack. Yes, we can employ AI for defense, but we have to design our systems to better survive failures. These are classic cybersecurity methods: least privilege, segmentation, defense in depth, monitoring, etc. We also need to significantly reduce the attack surface – that is, minimize the ways that a system can be attacked in the first place.

But we also need to design our systems to take a punch – be resilient in the face of failure. Sounil Yu proposed a design philosophy he called the DIE Triad (which maps to the security CIA Triad): resilient systems are distributed, immutable, and ephemeral. We should have accessible, usable backups of all the important data – and we should periodically run drills to prove it. It’s not just about having skilled and intelligent defenders – it’s also about reducing the amount of skill and intelligence necessary to do the defending.

Takeaways for You

You don’t have to be an engineer or critical infrastructure operator to benefit from this advice. The same resilience principles scale all the way down to your own digital life: reduce the number of things that can go wrong, limit the damage when something does, and make sure you can recover afterward.

Increasingly, bad guys are compromising the devices we all use at home to launch their attacks. That’s always been true – but today, with the advent of extremely capable AI agent attackers, it’s never been more important to shore up your defenses and reduce your attack surface. Update the software on all your devices. Get rid of IoT devices with crappy security or that are no longer able to get security updates. Back up your critical data. Remove online data and accounts you no longer need. And help others to do the same.