Forecasting · · 3 min read
Building a Metaculus forecasting bot
Lessons from an LLM forecasting bot in Metaculus AI tournaments: silent research failures, extremized toss-ups, and replacing vibes-tuning with measurement.
Emmanuel Effiom DukeAI forecasting and LLM evaluation
Automated forecasting with LLM ensembles, calibration measured offline, and probes that test what models really know.
Forecasting is one of the hardest honest tests of an AI system. Questions resolve, so there's nowhere to hide. Between October 2025 and August 2026 I built and ran a forecasting bot in Metaculus's AI tournaments, most recently as oracledeckv1, and built the evaluation tooling to improve it by measurement rather than intuition.
How the bot works
- Classify the question's domain (geopolitics, economics, science, tech, business, health, sports).
- Turn it into up to three targeted search queries and run several providers in parallel (Exa, Perplexity Sonar, Nimble, plus You.com for finance and market questions).
- Synthesise the results into one brief with a cross-source "signal strength" score.
- Forecast with a weighted five-model ensemble (GPT-5.6 Luna, DeepSeek-V4-Pro, Kimi-K2.6, Claude Haiku 4.5, Perplexity Sonar Pro), then challenge the result with critic and red-team models.
- Calibrate: shrink when models disagree or research is weak, decay toward uncertainty for distant close dates, and extremize only clear signals.
It forecast in the Metaculus AI Benchmark seasonal tournament and Market Pulse. MiniBench ran on a separate single-model runner with a more conservative calibration profile.
What the work found
- Base rates beat intuition. In 190 resolved binary questions, only 31.6% resolved YES, so the common habit of shrinking uncertain forecasts toward 50% is a systematic YES bias.
- Don't extremize toss-ups. Pushing near-50/50 ensembles toward confident answers produced confident-wrong forecasts, so the bot now leaves genuine toss-ups alone.
- Backtests leak. Unless search is date-bounded, retrieval surfaces coverage of the actual outcome and inflates any backtest score.
- Models misreport what they know. Every model that stated its own training cutoff was wrong by 8–15 months. Some confabulated answers to questions about events after their cutoff (42% for DeepSeek-V4-Pro, 50% for Kimi-K2.6), while GPT-5.6 and Claude Fable 5 always disclaimed.
Methods
LLM ensembles with weighted and median aggregation, critic and red-team passes, two-stage offline replay (expensive research cached once, calibration swept for free), log score and Brier scoring, date-bounded retrieval with leak canaries, and dated-event bisection for knowledge cutoffs.
Selected work
- oracledeckv1 (botduke--update)Forecasting bot for Metaculus AI tournaments, October 2025 – August 2026.
- LLM knowledge-cutoff probeDated-event bisection across nine models, with a conference-style paper.
Related writing
AI evaluation · · 2 min read
Ask the calendar, not the model
Measuring nine language models' real knowledge cutoffs with dated-event bisection: every model that self-reported a cutoff was wrong, some by over a year.
Other areas
Expertise
AI engineer
LLM products, agents and evaluation tooling, built end to end and shipped.
Expertise
AI safety researcher
Mechanistic interpretability, model auditing and alignment experiments, with open code and honest negative results.
Expertise
AI security engineer
Social-engineering and phishing detection, prompt-injection-aware LLM design, and auditing models for hidden behaviour.
Expertise
Mechanical engineer and energy researcher
CFD modelling, machine-learning energy optimisation and air-quality research, grounded in a mechanical-engineering degree and hands-on HVAC work.