Building a Metaculus forecasting bot

Between October 2025 and August 2026 I built and ran an automated forecasting bot in Metaculus's AI forecasting tournaments, most recently under the name oracledeckv1. It started from Metaculus's bot template, built on the forecasting_tools library, and grew into a multi-source research and LLM-ensemble pipeline. This is what building it taught me. Most of it came from finding where the bot was quietly wrong.

How the bot works

For each open question, the bot:

1. classifies the question's domain (geopolitics, economics, science, tech, business, health or sports)
2. turns it into up to three targeted search queries
3. runs several search and news providers in parallel and synthesises them into one research brief with a cross-source "signal strength" score
4. asks an ensemble of different LLMs for a forecast, aggregated by the median for robustness to outliers
5. passes the result through a critic and a red-team model
6. applies calibration: shrinkage when the ensemble disagrees, time-decay for far-off close dates, and extremization rules

The bot was configured for three tournaments at once: the primary AI competition, MiniBench and Market Pulse.

Lesson 1: the scariest failures are silent

Reading the run logs in July turned up research providers that were failing without crashing: a wrong API call signature, a synthesis model name that didn't exist, and an SDK wrapper incompatibility. The bot kept forecasting, just with less evidence than I thought it had. I replaced the broken sources with Nimble and You.com search, and gated You.com to finance and market questions, where it added the most. Since then, I treat "it ran" and "it worked" as separate claims that need separate checks.

Lesson 2: don't extremize a toss-up

The original calibration curve pushed forecasts between 0.45 and 0.55 hard towards the extremes (strength 1.8). That produced confident-wrong forecasts and negative scores. A near-50/50 ensemble means "we don't know", and making that confident is exactly backwards. I changed the policy:

• toss-ups (0.45–0.55): no extremization at all
• mild leanings (0.35–0.45 and 0.55–0.65): gentle extremization
• clear signals (below 0.35 or above 0.65): moderate extremization

Lesson 3: replace vibes-tuning with measurement

Tuning on live tournaments is slow and noisy, so in August I added an offline evaluation harness, built with an AI coding agent. Its two stages mean the expensive research-and-ensemble step runs once per question and is cached, after which calibration variants can be swept at zero token cost. Running it against 190 resolved binary Metaculus questions (May–August 2026) produced findings I wouldn't have guessed:

• Only 31.6% resolved YES (60 of 190). The bot's habit of shrinking uncertain forecasts towards 50% was therefore a systematic bias towards YES, not a neutral safety measure. A new calibration does one bounded shrink in log-odds space towards a base-rate prior instead.
• Bot accounts don't receive community-prediction aggregates: the field was empty on all 190 questions. The branch of the code that blended in the community forecast had never run.
• The research-quality signal was saturated. Counting snippets hit the cap on every question, which pinned "quality" at 1.0 and silently disabled the shrink meant for weak research. It now counts distinct source domains.
• Backtests leak unless search is date-bounded. A "leak canary" showed that unbounded search on resolved questions surfaced coverage of the actual outcome, while date-bounded search didn't. Without that bound, any backtest score would be inflated.

Lesson 4: know what your models know

An ensemble mixes models with different training cutoffs, and their self-reported cutoffs are unreliable (see my knowledge-cutoff probe). So the bot now measures each ensemble member's cutoff with a compact dated-event ladder and logs it next to every forecast, noting that live research extends what the model knows unaided. I also moved to a cheaper ensemble of models, each verified live before switching, and added a memory that recalls similar resolved questions alongside the bot's track record on them.

What I can't claim yet

I don't have a clean before-and-after score. Metaculus's peer score compares you with other forecasters, which can't be reproduced offline. The harness only ranks variants against each other, as a proxy. The first full stage-one run was also interrupted by upstream model-provider errors, so the head-to-head of the old and new calibration is still pending. The code, the harness and the corpus tooling are all public, so anyone can check the numbers above.