AI safety · Alignment · Negative result

The reward spread; the warning didn't

Reproducing unintended reward generalization in an inspectable toy model, and a negative result: internal signals gave no early warning.

By · · 2 min read

Left: the rewarded tic and the tic in never-rewarded contexts both rise to maximum within the first few hundred steps. Right: the tic output rises almost immediately while internal representation drift rises more slowly.
The tic spreads to contexts that were never rewarded (left). The internal signal rises after the output, not before it (right).

OpenAI's Where the Goblins Came From describes a reward that generalised beyond the context it was meant for. A natural safety follow-up is to ask whether you can see that spread coming inside the model before it shows up in behaviour. I tested both halves in a model small enough to inspect completely.

Setup

A small MLP has two heads, a "tic" and a "task", over six contexts. I rewarded a high tic only in context 0, then measured two things: whether the tic spreads to contexts that were never rewarded, and whether an internal signal rises in those contexts before the behavioural tic does.

Result 1: reward generalization reproduced

It does spread. Rewarding the tic in context 0 alone drives it to maximum in every context, including those that were never rewarded. The goblin dynamic appears in a fully inspectable toy.

Result 2: no early warning (a negative result)

The early-warning hypothesis was not supported. First-layer representation drift in the unrewarded contexts didn't reliably precede the output tic; the measured lead time was about zero, or slightly negative. In a small, tightly coupled network, the internal and behavioural signals move together.

I tried one more angle: tracking the projection onto the reward-linked component rather than raw activations. It still gave no lead time at this scale.

Why a negative result is worth publishing

The proposal's own failure criterion, stated in advance, was that the project fails if the spread only becomes detectable after it appears in outputs. At toy scale, that is what happened. Rather than refuting the idea, the result narrows down what an early-warning system would need:

  • a deeper model, where the readout sits far from the mechanism, and/or
  • a decomposition such as SPD that isolates the reward-linked mechanism from the readout. This connects to my SPD clustering work.

The next experiment is a small real transformer with a lexical-reward fine-tune. Both scripts in the repository (reproduce the generalization; run the early-warning test) run on a CPU.

Reference: OpenAI (2026), Where the Goblins Came From.

Code, data and full results: dukemawex/reward-generalization-earlywarning on GitHub. Related expertise: AI safety researcher.

More writing

Forecasting · · 3 min read

Building a Metaculus forecasting bot

Lessons from an LLM forecasting bot in Metaculus AI tournaments: silent research failures, extremized toss-ups, and replacing vibes-tuning with measurement.

AI evaluation · · 2 min read

Ask the calendar, not the model

Measuring nine language models' real knowledge cutoffs with dated-event bisection: every model that self-reported a cutoff was wrong, some by over a year.

AI security · · 2 min read

Designing Teger AI to explain phishing

How Teger AI, an explainable social-engineering defence for Gmail and Slack, pairs deterministic signals with AI analysis and treats every message as untrusted.

All articles →