Thinking about LLM (cyber)security

Recent news about agents going wild, escaping sandboxes and hacking systems on the internet made me wonder why everyone is going nuts about LLMs being such great hackers. My take: that's not the real point.

The hacks we see happen because the hacked systems were not up to standard. If anything valuable had been on those systems, human hackers would have got there years ago.

What changed with LLMs is the definition of a valuable target. Before, we asked what a system offers that is valuable to a human. Data, credentials, money. An agent can also use resources humans never bothered to steal: public communication channels, storage, slow compute. Attack surface that used to be "not worth it" is now on the menu.

So fix your threat model. Reevaluate your systems with a new question in mind: what on here is valuable to an automatic agent?

Have a good Sunday afternoon.