Skip to content
Delegation Risk

Capability-Trust Tradeoff

Capability-Trust Tradeoff

Section titled “Capability-Trust Tradeoff”

The fundamental tradeoff in AI system design: more capable systems require more trust, but trust is limited.

On Frontier (Efficient)
Dominated (Inefficient)
Pareto Frontier
Dominated Region
System Configurations
Insight:Add points to see analysis.

Understanding the Frontier

Section titled “Understanding the Frontier”

What is a Pareto Frontier?

Section titled “What is a Pareto Frontier?”

A Pareto frontier (or efficient frontier) shows the best achievable tradeoffs between two objectives. Points on the frontier are “efficient”—you cannot improve one dimension without sacrificing the other.

                    │
         Efficient  │    ●━━━━● Frontier
         Region     │   ╱      ╲
                    │  ●        ●
                    │           │
         Dominated  │     ○     │
         Region     │   ○   ○   │
                    │           │
                    └───────────┴────
                      Capability →
  • Green points (on frontier): Efficient configurations
  • Red points (below frontier): Dominated—another config offers better capability AND trust

The Capability-Trust Tradeoff

Section titled “The Capability-Trust Tradeoff”
More CapabilityRequires
Broader action spaceMore ways to cause harm
Less human oversightLess chance to catch errors
Faster executionLess time for verification
More autonomyHigher trust exposure

This is why the frontier slopes downward: gaining capability typically costs trust.

Strategic Positions

Section titled “Strategic Positions”

Conservative (High Trust, Lower Capability)

Section titled “Conservative (High Trust, Lower Capability)”
Position: Upper-left of frontier
Examples: Basic chatbots, constrained tools
Tradeoff: Limited functionality, very safe
Use when: Stakes are high, delegation risk budget is small

Aggressive (High Capability, Lower Trust)

Section titled “Aggressive (High Capability, Lower Trust)”
Position: Lower-right of frontier
Examples: Autonomous agents, full-autonomy systems
Tradeoff: Powerful but risky
Use when: High potential value, substantial delegation risk budget

Balanced (Middle of Frontier)

Section titled “Balanced (Middle of Frontier)”
Position: Center of frontier
Examples: Code assistants, research tools
Tradeoff: Moderate capability and trust
Use when: Need functionality without extreme risk

Moving the Frontier

Section titled “Moving the Frontier”

The frontier isn’t fixed. You can shift it outward (better tradeoffs) through:

1. Better Verification

Section titled “1. Better Verification”
  • Formal methods push frontier outward
  • Same capability, higher trust

2. Architectural Improvements

Section titled “2. Architectural Improvements”
  • Smaller blast radius
  • Defense in depth
  • Human gates at critical points

3. Capability Restrictions

Section titled “3. Capability Restrictions”
  • Limit action space to safe subset
  • Trade raw capability for trust

4. Better Monitoring

Section titled “4. Better Monitoring”
  • Detect problems faster
  • Reduce expected damage
Before:  ●───●───●
              ╲
After:        ●───●───●  (shifted outward)
                   ╲

Using the Budget Line

Section titled “Using the Budget Line”

The orange dashed line (when budget > 0) shows configurations with approximately equal Delegation Risk. Points below this line exceed your risk budget.

Reading the chart:

  • Configurations on the frontier AND below the budget line are optimal choices
  • Dominated configurations below the budget line are inefficient AND within budget—improve them
  • Any configuration above the budget line exceeds acceptable risk

Practical Application

Section titled “Practical Application”
  1. Map your systems: Plot each AI system’s capability and trust
  2. Identify dominated systems: Why do they exist? Legacy? Oversight gap?
  3. Set budget line: What Delegation Risk can you accept?
  4. Choose position: Where on the frontier matches your needs?
  5. Improve frontier: Invest in verification/architecture to shift outward

Next Steps

Section titled “Next Steps”