When an authorization engine throws an unexpected exception, what should happen?
For months, AIService followed a simple rule: fail closed. If policy evaluation
threw an error — whether due to a YAML parsing glitch, database latency, or a
transient network drop — the runtime trust gate returned a terminal
GATE_EVALUATION_FAILED. The agent session was immediately terminated.
The problem was operational reality. Transient engine glitches turned into
abrupt session denials, discarding agent context and wiping operator visibility
at the exact moment a human needed to see what was happening. We decided to alter
the fail direction: instead of hard-failing to an unrecoverable deny, policy
evaluation faults would synthesize a RequireApproval decision (7e11f14),
pausing execution and escalating to an operator.
What followed was a rapid-fire four-commit fix-storm that exposed subtle gaps in how the Microsoft Agent Framework (MAF) handles function execution.
The first flaw emerged almost immediately (ac296f0). The fix had updated the
runtime intercept, but MAF uses a two-phase tool lifecycle: tools are inspected
at registration time (BuildAIToolsAsync) and then intercepted at execution time.
When the registration probe hit an engine fault, it defaulted to registering a
bare proxy function. Because MAF executes bare functions inline without invoking
the outer intercept, high-risk side-effecting tools ran completely unattended.
The runtime escalation never even had a chance to trigger. Both sites had to be
unified under GateDecision.EvaluationFailed.
The second flaw was authorization scope (9315082). The synthesized escalation
was initially given a severity of High. In our RBAC model, High severity
permits standard collaborators to approve the action. But an unevaluated policy
is not a routine elevated action — it is a complete unknown. Allowing routine
collaborators to approve an action whose policy failed to evaluate defeated the
entire safety envelope. Fault escalations had to be raised to Critical,
restricting resolution strictly to system owners. In the same commit, we had to
kill “phantom gates” in headless runs, where un-terminated pending approvals lingered
forever in listing queues.
The third flaw was lifecycle asymmetry (8bd8672). To keep workflows productive
during an outage, read-only tools like read_file and git_status were exempted
from escalation during registration. But because the runtime intercept lacked that
exemption, MAF invoked the read-only tool inline, it succeeded, and then the
runtime intercept fired an escalation for an action that had already finished.
Headless agents received a hard denial response for a command that had succeeded
seconds earlier.
The root cause was architectural: AgentFrameworkOrchestrator had become a
monolith handling tool discovery, policy evaluation, session states, and telemetry.
Decomposing it into discrete collaborator services (ac0e22d) made these
lifecycle contracts explicit, bringing the repair storm to an end.
Lessons learned
- Two-phase tool intercepts must share their failure models end to end. If an agent framework splits execution between an up-front registration pass and a turn-time intercept, altering policy handling in one place while leaving the other alone creates silent security bypasses. Both phases must evaluate faults identically.
- An unevaluated policy is maximum risk, not moderate risk. It is tempting to treat an infrastructure glitch as a middle-tier severity. But when policy cannot be evaluated, you do not know if the payload is harmless or catastrophic. The only safe fallback is the strictest tier available.
- Asymmetric exemptions create phantom approval gates for completed actions. When an operation is exempted at tool registration but gated at runtime execution, the tool executes and the gate fires afterward. Asking a human to approve an action that has already modified state or read disk destroys confidence in the control plane.
- Monolithic orchestrators hide cross-phase lifecycle regressions. When one class owns registering tools, evaluating policy, emitting events, and managing agent turns, the subtle differences between registration-time assumptions and runtime execution get lost in the noise. Decompose by responsibility so lifecycle transitions are auditable.