AGI Security: Rethinking Society's Infrastructure
AGI security is the proactive security-first infrastructure for a world where AI systems can reason, adapt, and act autonomously across digital and physical domains.
Specifically, it posits that:
Powerful AI will by default destroy our social, technological, and civilizational infrastructure due to the speed at which it is being developed.
Attacks become the default, not an infrequent occurrence, due to the fall in the price of intelligence and the proliferation of powerful AI.
Attack surfaces expand from the digital to physical, cognitive, and institutional through for example drones, mass media, and information control.
As a result, we need to rethink how we construct our societal infrastructure. We need to go beyond ‘adding a security layer’ to embedding security into the very foundations of anything we build.
Without this, the threats of powerful AI (such as Sentware, loss-of-control, societal disruption, and civilization-scale manipulation) will easily penetrate our weak security posture for whoever wishes to destabilize or gain control.
This post was originally published on Esben's blog.
Call For Technologies (CFT)
However, if we act today, we can be the ones to shape this field. These are but a few ideas for what we need to create today to lay the foundation of tomorrow:
Model Security and Control
Runtime containment: Hardware/software boundaries preventing model breakout and access control
Capability capping: Cryptographic limits requiring multi-party authorization vs single-party company control
Behavioral attestation: Continuous (inference-time) alignment verification vs current pre-deployment adhoc evaluation
Example: NVIDIA H100 Confidential Computing (<5% overhead)
Infrastructure Hardening
Supply chain verification: Privacy-preserving component tracking from manufacture to deployment
Air-gap capabilities: Millisecond physical isolation and shutdown
Redundant sovereignty: Critical systems across geopolitical boundaries
Attack-resilient design: Run-time monitoring and forensics assuming constant breaches
Societal Autonomy Protection
Manipulation detection: Identifying influence vs information
Authenticity verification: Human vs AI-generated content
Cognitive firewalls: Filtering AI access to human attention
Autonomy audits: Measuring for reductions of human autonomy
Deployment Infrastructure
Silicon-level security: Compute use attestation and auditing
Multilayer identification systems: Human / AI / hybrid
Capability attestation and capability-based permissions: Enforcement of tool-AI level control
Rollback capabilities: Logging at every step with enforcement of zero destructive changes
Examples
Hardware and compute infrastructure: Location verification
AI network protocol defenses: Agent labyrinths
Secure AGI deployment systems: Evaluation
Verification-as-a-service: Content moderation
Tool-AI for security: Digital forensics
Provably secure systems: Safeguarded AI
Scale, Scale, Scale
One of the most important factors in AGI security is whether we can reach the scale necessary within the allotted time.
I definitely believe we can and the evidence is in. We’re already at over $50M value in Seldon, our AGI security lab program, and 10x’d our investment in Goodfire with Juniper Ventures.
For founders, this means to have ambition for unicorn growth within 12 months. For investors, it means radical levels of investment in the earliest stages of this industry that will transform our society during the next years.
AGI Security
This is the most exciting opportunity in our lifetime. We stand at the precipice of a new world and we can be the ones to shape it.
The core of what makes us human will need guardians and there’s no one else but you to step up.
Pioneering startups are already running full speed ahead.
Join them!
