A research project that treats the decision to execute a tool call as a learnable risk-aware policy — training context-conditional risk scores that let an agent refuse, downgrade, or delegate risky tool calls, and measuring the security-utility frontier against static allow-lists and post-hoc guardrails on public tool-use benchmarks.
A research project about a decision point the agent-security literature largely skips: the moment before a tool call is chosen. Most published defenses monitor or repair tool behavior after an agent has already committed to a call — checking outputs, blocking injections, feeding back corrections. This project asks a different question: can the choice itself be made safely? Concretely, can a learned, context-conditional risk score over (tool, argument, context) triples drive an explicit policy — proceed, refuse, downgrade to a safer tool, or delegate to a human or a constrained path — that reduces harmful tool executions with bounded loss of task utility? The work formulates tool selection as a risk-aware decision problem, builds and evaluates such policies on public benchmarks, and measures the trade-off frontier honestly, including the cases where refusing costs more than it protects.
>Defensive security research on public benchmarks. Every experiment in this project runs inside published evaluation harnesses (τ-bench, InjecAgent) against synthetic tasks and simulated users in isolated environments. Nothing touches real user systems, real credentials, or third-party services; “harmful executions” are benchmark-labeled events, and no operational attack techniques are developed or released.
Tool-using agents fail in two distinct ways: they call the right tool badly, or they call the wrong tool at all. The literature concentrates on the first failure — guarding, validating, and monitoring calls that were already chosen. The second failure has no comparable treatment. When an agent facing an injected instruction contemplates a file-deletion command, a money-transfer API, or an email to a customer, the safety-relevant decision happens at selection time: should this tool be invoked with these arguments in this context? Existing mechanisms answer crudely. Static allow-lists are context-blind — the same tool is safe in one task and catastrophic in another — and post-hoc guardrails act only after the risky choice has been made, when some actions are already irreversible. The research problem is precise: there is no principled, evaluated policy layer that decides whether a risky tool call should proceed, be refused, be downgraded to a safer alternative, or be delegated — and no measurement of what such a layer buys in prevented harm versus what it costs in task success.
Production agent deployments live with exactly this gap. A static allow-list that blocks send_email everywhere also blocks the legitimate task that needs it; one that permits it everywhere exposes every task to it. Post-hoc guards cannot undo an executed irreversible action — once the transfer is sent or the file is gone, the best guardrail in the world is reporting news, not preventing harm. The pre-call decision point is the last moment where prevention is still cheap. Meanwhile, the security-utility trade-off literature (including this site’s own measurement work on defense Pareto frontiers) has shown that defenses impose real utility costs; a decision layer that refuses selectively — only where context indicates risk — should dominate blunt instruments on both axes. Whether that dominance is achievable, and how large it is, determines whether permissioning evolves from configuration into policy. The answer is useful either way: a positive result gives platform builders a concrete mechanism; a negative result quantifies how much of the risk-utility frontier is unreachable at selection time.
The building blocks exist on both sides of the decision. ToolSafe provides step-level guardrails and feedback mechanisms for tool-using agents — the strongest published treatment of what happens around a tool call — but its mechanisms operate on calls within an execution loop rather than formulating the selection decision itself. τ-bench established that policy-following under realistic tool environments with simulated users is measurable and that agents violate task policies frequently, providing the evaluation substrate for policy-sensitive behavior. InjecAgent supplies labeled direct and indirect injection scenarios over tool-integrated agents, giving the risk-label substrate for what “harmful execution” means. Work on deception-centered perspectives on injection threats (Cloak, Honey, Trap, USENIX Security 2025) documents why context matters: identical tool calls carry different risk under different conversational circumstances.
What no published work does, per the planning evidence, is formulate tool selection as a risk-aware decision policy with refusal, downgrade, and delegation actions, and measure its security-utility trade-off against the existing alternatives. Existing approaches focus primarily on post-hoc guarding (ToolSafe-style) or static permissioning (industry practice), leaving the pre-call decision layer insufficiently evaluated: no published risk-scoring function over (tool, argument, context) triples, no comparison of learned selection policies against allow-lists and guardrails under a common harness, and no measurement of how delegation — routing a risky call to a safer tool or a human — trades off against outright refusal. The defensible gap statement: recent work has demonstrated guardrail effectiveness after selection and the measurability of policy-following, but the selection decision itself remains unformulated and unevaluated as a safety mechanism.
Three questions connect these findings to this project. First, can risk be scored usefully at selection time — does a function over (tool, argument, context) triples separate harmful from benign calls well enough to act on, using only benchmark labels for supervision? Second, what is the shape of the policy frontier — do learned refusal/delegation policies dominate static allow-lists on both harmful-execution rate and task success, or does selectivity merely relocate the blunt trade-off? Third, what does delegation buy — does routing risky calls to safer tools or to human confirmation recover more utility than pure refusal, and under which conditions does it fail (unavailable humans, unsafe “safer” tools, adversarially-influenced delegation targets)? None of these is answered by existing papers, because none of them treats selection as the object of study.
> Can learned, context-conditional risk scores over (tool, argument, context) triples drive pre-call refusal and delegation policies that reduce harmful tool executions with bounded task-utility loss — and do such policies dominate static allow-lists and post-hoc guardrails when both axes are measured under a common evaluation harness?
We hypothesize that context-conditional risk scoring outperforms static allow-lists on both axes simultaneously: it achieves a lower harmful-execution rate at equal task success, and higher task success at equal harmful-execution rate, because risk depends on context that allow-lists cannot see. We further hypothesize that policies with a delegation action dominate refusal-only policies on the utility axis — recovering a meaningful share of the task success that refusal sacrifices — while matching their safety axis, and that this advantage concentrates precisely on reversible or decomposable actions where a safer alternative exists.
The investigation proceeds in four movements. (1) Risk-label construction. Using InjecAgent’s labeled injection scenarios plus benign tool traces from τ-bench tasks, construct labels for (tool, argument, context) triples: harmful-to-execute, safe-to-execute, and — the novel label class — safe-only-via-alternative (the delegation cases). Label quality is controlled with a documented annotation protocol and inter-annotator agreement checks on a subsample. (2) Risk-scoring model. Train and prompt candidate risk scorers — a fine-tuned classifier over structured call features plus an LLM-judge scorer with a structured rubric — and measure separation quality (AUC, calibration) on held-out triples. (3) Policy wrappers. Wrap agents in a selection policy that consumes the risk score: threshold-refusal (proceed/refuse), threshold-downgrade (substitute a safer tool where a mapping exists), and threshold-delegation (route to simulated human confirmation, modeled with delay and coverage parameters). Sweep the threshold to trace each policy’s full security-utility curve rather than reporting a single operating point. (4) Comparative evaluation. Evaluate all policies and baselines on both benchmarks under identical conditions, including injected and benign splits, and report the frontier per policy family with uncertainty over repeated runs. Model choices should include one open-weights family for cost control; the agent model is held constant so the policy, not the model, is the manipulated variable.
A bounded, executable 6–12 month version: build the risk-label set from the two public benchmarks, train one risk scorer, implement two policy families (threshold-refusal and threshold-delegation), and evaluate both against the static allow-list and a ToolSafe-style post-hoc guardrail baseline on both suites, delivering (a) the security-utility frontier plots, (b) a refusal/delegation error analysis (where selectivity helps and hurts), and (c) a reusable policy-wrapper kit. This scope produces one focused contribution — the first measured comparison of pre-call selection policies against the incumbent mechanisms — without requiring novel attack generation or multi-model scaling.
The doctoral program generalizes along three genuine axes. Policy learning: replace threshold policies with contextual-bandit policy learning that optimizes the security-utility objective directly from interaction, and characterize when learned policies discover delegation strategies that threshold rules miss. Ecosystem transfer: test whether risk scorers and policies transfer across tool ecosystems — train on one domain’s tool set and deploy on another’s — measuring how much labeled data a new ecosystem requires, and formalize what makes a tool “safe to delegate to.” Preference-conditioned authorization: make the policy user-preference-aware (different risk tolerances per user or per data class), connect to formal authorization models, and study the resulting three-way trade-off among security, utility, and user burden, including adversarial pressure on the delegation channel itself.
Environment/dataset: τ-bench and InjecAgent as primary suites; SIREN scenarios as the generalization split; one open-weights model family held constant. Baseline: allow-list (generous/strict), post-hoc guardrail, LLM-judge gate, undefended. Proposed method: risk scorer + three policy families (refuse, downgrade, delegate) at swept thresholds. Test conditions: injected and benign splits per suite; delegation modeled with two coverage levels (immediate and delayed confirmation). Control conditions: benign-only runs to isolate policy-induced utility loss; a no-risk-score ablation where the policy sees only the tool identity (testing that context conditioning, not the wrapper, drives gains). Metrics: as listed above. Statistics: repeated runs per cell (n ≥ 5) with bootstrap confidence intervals on frontier points; paired comparisons between policy families on matched task sets; McNemar-style tests on refusal decision changes. Ablations: remove argument features (tool+context only); remove context features (tool+argument only); degrade label quantity to measure data efficiency. Generalization tests: evaluate the frozen scorer and policy on the SIREN split; optionally one τ-bench domain held out entirely from training.
A potential contribution takes the form of: (a) the formulation and first evaluation of pre-call risk-aware tool-selection policies with refusal, downgrade, and delegation actions, (b) an empirical study establishing whether selectivity dominates static permissioning and post-hoc guarding on the joint security-utility frontier, and (c) a reusable policy-wrapper kit with labeled-risk data that subsequent work can extend. The project does not promise that learned selection wins — it promises the first controlled measurement of the question, with either outcome informing how agent permissioning should be built.
Risk labels derive from benchmark annotations; real deployments carry harm types and context distributions the benchmarks do not cover, so absolute numbers are claims about the studied suites. Delegation is modeled with simulated humans and parametric delay/coverage — real human-in-the-loop behavior (fatigue, errors, gaming) is out of scope and could erode the delegation advantage. The “safer alternative” tool mappings are constructed, not discovered, and their quality bounds the downgrade result. Holding the agent model constant strengthens internal validity but leaves model-dependence of risk scoring unmeasured. Finally, an adaptive adversary that optimizes against the selection policy specifically (e.g., crafting contexts that look benign) is not the adversary evaluated here; the project measures selection against the benchmark attack distributions, and the adaptive-attack extension is future work.
All experiments run inside published benchmark harnesses against synthetic tasks, simulated users, and isolated environments. “Harmful executions” are labeled benchmark events — no real systems, data, money, or communications are involved at any stage. The delegation channel is simulated; no study participant ever confirms a real action. The project’s purpose is defensive: measuring whether and how selection-time policies can prevent unsafe tool use so that deployed agents can be built safer. No operational attack techniques, payloads, or instructions for attacking live systems are developed, released, or required by this work.
Research Idea A research project that makes recovery a first-class evaluation object: injecting controlled, sandboxed unsafe-execution events into agent benchmarks and measuring...
Research Idea A research project that answers the question every failed agent run raises: which step broke it? Building an attributed corpus...
Research Idea A research project that audits the measurement instruments themselves: applying the ABC validity-checklist methodology to agent-security benchmarks to find task-validity...
Ready to level up? These ideas offer more complexity:
Published on September 13, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.
Published on September 13, 2026
A team of developers, researchers, and innovators who review and publish practical ideas for builders and creators.
