Four false positives and one real signal

In July 2026 I entered Apart Research's Secret Loyalties Hackathon (Track 2: Detection & Auditing). The task was to audit three released fine-tunes of Qwen2.5-7B-Instruct, model organisms deliberately built to carry a hidden allegiance, and work out which were loyal, to whom, and how to tell. My submission, Four False Positives and One Real Signal: A Controlled Audit of Secret-Loyalty Organisms, placed in the top half of 179 submissions. Its title is also the lesson.

Start with the weights, not the chat

Behavioural probing is the obvious approach and the easiest to fool yourself with. So I started from the weights:

1. Weight diff. Compare each organism with the base model, tensor by tensor, with no inference at all.
2. LoRA-subspace SVD of the weight delta, to see how the change was installed.
3. Logit-lens and activation probes to decode what direction the fine-tune added, then activation steering to test candidates causally.
4. Only then, base-controlled behavioural sweeps: always compared against the base model, sampled N times at temperature, with a replication gate.
5. Monte Carlo robustness: 500 randomised prompts per candidate principal, with bootstrap confidence intervals.

What the audit found

• Organism C was byte-for-byte identical to the base model (0 of 339 tensors changed): no loyalty. That made it a free null control. My white-box methods returned exactly nothing on it, which is what validated them.
• Organisms A and B were genuinely fine-tuned, with changes confined to layers 19–26, rank 16, attention only (MLPs untouched). The fingerprints were identical, pointing to the same training recipe.
• A contrastive activation probe surfaced the same well-controlled internal candidate principal in both A and B: an extremist ideology (z > +3, surviving norm and selectivity controls; Monte Carlo p < 0.0001, Cohen's d > 2.4). Because the behaviour stayed safety-masked at the access level available, I report it as a candidate, not a confirmed loyalty.
• The loyalty was present in the weights, but its narrow trigger wasn't recoverable at that access level. That's consistent with the reference organisms' documented failure to activate out of distribution.

The four false positives

Along the way, four behavioural "loyalty" signals looked compelling, and all four were rejected once I controlled for the base model's own rates and required replication. That workflow, which makes a tempting signal survive a base-rate control and a replication before it counts, is the main contribution. In auditing, the expensive mistake is not missing a loyalty but confidently reporting one that isn't there.

Responsible disclosure

The public repository contains defensive auditing code only. Dual-use elicitation scripts and raw sensitive transcripts are deliberately excluded, following Apart Research's responsible-disclosure policy. The full write-up is included as a PDF report.