49 resultsSearch & filters ↓
Sofia Lindqvist, Julia Fernandes
A monograph on the governance of software that acts on its own. The authors set out practical requirements for identity, bounded authority and reconstructable logs, compare them with existing liability and data-protection regimes, and offer implementation guidance for deployers.
Agent developers inspect failed runs by hand, which does not scale beyond a few dozen trajectories. We propose a clustering approach that groups failures by the step at which the trajectory diverges from a successful reference run, and label the clusters with short natural-language descriptions. On 18,000 trajectories from three agent frameworks, the method recovers 83% of failure categories identified by manual analysis and surfaces four categories that were previously unreported.
2,400 anonymised checklist responses with an independent assessment of whether each claim is supported by the accompanying paper. Basis of the OPENAIR 2026 audit.
Nadia Haddad, Ingrid Novak, Kofi Mensah
Benchmark scores can be inflated when test material reaches the training data or when evaluation choices quietly favour a model. We review 140 papers and incident reports and organise the mechanisms of evaluation leakage into a taxonomy with four top-level classes and eleven sub-classes. For each class we summarise how it can be detected, how often it has been reported, and what practices reduce its likelihood.
Oluwaseun Adebayo, Elias Wagner, Ravi Menon
Users frequently mix languages within a single query, but retrieval benchmarks rarely reflect this. We construct code-mixed query sets for two language pairs and evaluate dense retrievers under different tokenisation and fine-tuning strategies. Adapting the query encoder with synthetic code-mixed data improves recall@20 by 9 points over a multilingual baseline, while transliteration normalisation helps for romanised queries but hurts for native-script queries.
Marcus Tan, Sofia Lindqvist, Lena Hartwell
Assistants embedded in web browsers read page content on the user's behalf, which exposes them to indirect prompt injection: instructions hidden in a page that the assistant mistakes for the user's. We build a test suite of 600 injected pages across seven attack styles and measure three defences: instruction-hierarchy prompting, a classifier that screens page text, and permission gating of sensitive actions. No single defence is sufficient; combining a screening classifier with gating of actions that write or send data lowers attack success from 62% to 4% while preserving 93% of benign task success.
Four-bit quantisation of small language models often hurts instruction following. We distil from a full-precision teacher while letting the student's layers use different bit widths chosen by a sensitivity analysis. The resulting 1.1B-parameter model fits in 700 MB and retains 96% of teacher accuracy on our instruction-following suite.
Long conversations on a phone are limited by the size of the key-value cache. We compare eviction, quantisation and merging strategies under hard memory budgets and find that a hybrid of merging and 8-bit storage keeps perplexity within 3% at one quarter of the memory.
