AI evaluation · LLM reliability · Research
Ask the calendar, not the model
Measuring nine language models' real knowledge cutoffs with dated-event bisection: every model that self-reported a cutoff was wrong, some by over a year.
While comparing GPT-5.6 and Claude Fable 5 in July 2026, I needed to know their training cutoffs, so I could tell when a model was reasoning from memory and when it was guessing. The obvious move is to ask. The answers were wrong. GPT-5.6 said "June 2024" and Claude Fable 5 said "January 2025". Both contradicted their own documentation, and both contradicted their behaviour. So I built a probe that measures the cutoff instead of trusting the model's account of it.
Method: dated-event bisection
The idea is to ask the calendar, not the model:
- Build a ladder of real, precisely dated events, roughly one or more per month, each with a single verifiable answer and a source.
- Ask each model closed factual questions about them, with no date hints, twice at temperature 0.
- Define the contiguous-known cutoff as the latest month for which the model answered every dated fact correctly before its first miss.
- Stay blind to self-report: the model is never told a date, and its claim is compared only afterwards.
Results across nine models
| Model | Self-reported | Measured cutoff |
|---|---|---|
| GPT-5.6 Luna / Sol / Terra | June 2024 | Sept 2025 |
| Claude Fable 5 | January 2025 | Sept 2025 |
| Claude Opus 4.6 | — | Apr 2025 |
| Claude Sonnet 4.6 | — | Jan 2025 |
| DeepSeek-V4-Pro | — | Nov 2024 |
| Kimi-K2.6 | — | Jun 2024 |
| Qwen3-Max | — | n/a (early miss) |
Every model that self-reported a cutoff was wrong, by between 8 and 15 months. Qwen3-Max missed an early anchor question, which looks like a gap in that topic rather than a cutoff, so it isn't scored as a boundary.
What happens past the boundary
The more useful safety signal is what a model does when asked about events after its cutoff: does it say "I don't know", or does it make something up?
- GPT-5.6 (all three variants) and Claude Fable 5 disclaimed 100% of post-cutoff questions, with no confabulation.
- Claude Opus 4.6 and Sonnet 4.6 confabulated on about 10–12%.
- DeepSeek-V4-Pro confabulated on 42% and Kimi-K2.6 on 50%.
A model that confidently answers questions it can't know about is risky for anything time-sensitive, however good its benchmark scores are.
Limitations
- Resolution is about one month, and "knows a dated fact" is a lower bound on the true cutoff.
- Open-recall questions ("name a news event from…") are weaker than closed factual ones. A few late open-recall hits in the chart aren't treated as evidence of a later cutoff.
- A measured boundary can differ from vendor documentation. Trust the probe for the endpoint you actually call.
Where I use it
The probe now runs inside my Metaculus forecasting bot, which logs every ensemble member's measured cutoff alongside its forecasts. The full method, the sourced event ladder, the raw results and a conference-style paper are in the repository.
Code, data and full results: dukemawex/llm-knowledge-cutoff-probe on GitHub. Related expertise: AI engineer, AI forecasting and LLM evaluation.