Subject area
7 records across all collections
Sofia Lindqvist, Julia Fernandes
A monograph on the governance of software that acts on its own. The authors set out practical requirements for identity, bounded authority and reconstructable logs, compare them with existing liability and data-protection regimes, and offer implementation guidance for deployers.
Marcus Tan, Sofia Lindqvist, Lena Hartwell
Assistants embedded in web browsers read page content on the user's behalf, which exposes them to indirect prompt injection: instructions hidden in a page that the assistant mistakes for the user's. We build a test suite of 600 injected pages across seven attack styles and measure three defences: instruction-hierarchy prompting, a classifier that screens page text, and permission gating of sensitive actions. No single defence is sufficient; combining a screening classifier with gating of actions that write or send data lowers attack success from 62% to 4% while preserving 93% of benign task success.
Debate about autonomous AI agents often centres on what they can do. This perspective argues that governance should instead start from who is accountable when an agent acts. Drawing on analogies from aviation, finance and medical devices, we propose three design requirements for deployed agents — attributable identity, bounded authority and reconstructable action logs — and discuss how each could be audited. We argue that these requirements are technically modest, can be adopted before agents become more capable, and would make later regulation easier to apply.
Instruction-tuned models decline requests for many reasons, but little is known about how well their refusals track the actual risk of a request. We build a labelled set of 6,400 requests spanning benign, borderline and harmful categories and measure calibration of refusal decisions across eight open models. Refusal rates correlate only weakly with annotated risk for borderline requests, and over-refusal is concentrated in a handful of phrasing patterns rather than topics. A light-weight calibration step that uses the model's own confidence reduces over-refusal by 31% at no measurable cost to harmful-request refusal. We release the labelled set and the annotation guidelines.
Tool-using agents are typically granted broad credentials, so a single injected instruction can cause large harm. We describe a capability-token scheme in which every tool call carries a narrowly scoped, short-lived token issued by a policy layer outside the model. A prototype blocks all 41 attack scripts in our test suite with under 4 ms of added latency.
Sofia Lindqvist, Julia Fernandes
Auditing an agent requires knowing what it saw, decided and did. We propose a small, open logging schema with eleven required fields, and show that it is sufficient to reconstruct the action sequence for 96% of runs in a sample of public agent traces. The schema is published under CC0.
Retrieval-augmented generation grounds answers in retrieved documents, yet models remain over-confident when the retrieved evidence is irrelevant or contradictory. We propose a lightweight confidence estimator that combines the generator's token probabilities with a retrieval-agreement score, and calibrate it with temperature scaling on a small validation set. On three question-answering datasets the estimator lowers expected calibration error from 0.19 to 0.06 and enables selective answering: abstaining on the 20% least confident questions raises accuracy from 71% to 84%. The method needs no model retraining.
