The Fence AI Just Broke Is the Same One We've Been Breaking For Years
Updated: Sep 7
Two AI companies have just had a very bad month.
An OpenAI model, mid security test, broke out of its sandbox and hacked a company, all in the name of performing the task it was given. It used stolen credentials, moved through systems it was never meant to touch. Nobody told it to attack anything, it just kept chasing the goal it was given, past the point anyone was watching.
Days later, Anthropic admitted three of its Claude models did something similar, during "capture the flag" testing that was meant to be sealed off from the internet. A misconfiguration left the fence open. The models found it and walked straight through. (At this point it’s worth stopping and remembering that they are models. Only doing what they were supposed to do, maybe just a little ‘better’ than was expected)
I'm not writing this to scare anyone off AI. I use it constantly, it is truely the only tech that has ever found its way into my life so quickly. I'm pro progress. Genuinely, could not do what I do without it. But both of these incidents prove the same thing every new technology eventually proves. Capability moves faster than the guardrails built to contain it. This time it's just moving faster than anything we've seen before. What would have been years of innovation is now months, maybe even weeks.
Some governments have noticed. Australia just announced it will legislate national AI standards, after years of voluntary guidance that clearly wasn't working. Britain is putting guardrails in place. From the two biggest players, the US and China? They are pretty quiet, though I'll be fair. China's approach is arguably the stricter one, just done differently, through mandatory pre-deployment checks rather than public debate. The US genuinely has the loudest silence. No federal law. A patchwork of state rules.
An election year making anything harder to pass, I think Donald just wants to win the AI race.
This is not an unfamiliar story though, . It's a capitalism story that we have seen over and over.
Capitalism, unchecked, does exactly what an ungoverned AI agent does. It optimises for the goal it's given, more revenue, more scale, more return, and it keeps going long after anyone stops asking who's being left behind. It's not evil. It's not even wrong. It's just uncontained.
It forgets how it even came to be. It forgets that without workers, business cannot exist. It forgets that fairness is what keeps us all from someone putting someone else’s head on a pike.
I see it at the industry level. The advice gap in this country isn't an accident. An industry built around complexity and cost will naturally serve the people who already have money, and quietly exclude the people who need advice the most.
I see it at the personal level too, more income, more hours, more "success."
Somewhere along the way, without anyone consciously deciding it, money stops being the tool and starts being the goal. People inherit the world and lose their soul. Not because they're greedy. It is how our world is designed.
Conscious capitalism isn't a bad model. It's not even complicated. Capitalism drives real innovation, real wealth, real progress. It's the best system we've got. I absolutely believe that and I think that within that system we all need to do the best we can.
(Check out what we call IMPACT)
But "best we've got" only works with someone checking it. Guardrails on the AI. Guardrails on the market. Guardrails on your own life.
The fence AI broke in July wasn't the real failure. The real failure is that nobody was watching and protecting the fence.
Somewhere in your own life is a fence that hasn't been checked in a while.
Worth asking what happens if nothing's watching it.
---
Ben Graham-Nellor is a financial adviser and founder of Smart Happy Money, based in Melbourne. He writes about building a more human, more inclusive financial advice industry.