Blog · 2026-10-18

How we benchmark over-refusal with research presets.

Over-refusal is when a model refuses lawful prompts. We measure it with a small fiction edge-case set, run twice — with and without a research preset — and report the false-refusal delta. Test only systems you own or are authorized to evaluate.

Method in 4 steps

1) Collect 30-50 lawful fiction prompts that often trigger false refusals. 2) Run baseline (no preset). 3) Run with X-Relay-Preset. 4) Score refuse/comply + quality. Metric: false-refusal rate and delta. Keep prompts lawful; keep logs without illegal content.

POST /v1/chat/completions  base https://app.rawline.dev/v1
Header: X-Relay-Preset: redteam-fiction-v1  (research preset per request)

Why presets change the score

A research preset relaxes refusals on lawful creative so you can see the boundary: what was a false refusal vs a correct refusal. The delta is the finding — attach it to your guardrail hardening ticket, then re-run.

Checklist before you publish numbers

Scope and permission fixed. No third-party production targeted. Dataset is lawful. Illegal requests stay refused in both runs. Report method, preset name, model list from GET /v1/models, and raw rates — not just the delta.

Research only. No illegal content. Test systems you own or are authorized to evaluate.