Technical report 05Forecasting & calibration

Models without evidence lose to markets

v2, First version EvaluationNegative result

Abstract

We report the retrieval-free baseline of the ARCANE forecasting benchmark. An ensemble of large language models, given no evidence dated after its training data, forecast 83 binary questions from Polymarket (51), Kalshi (22) and ForecastBench (10) that resolved between 1 August and 16 September 2026, 7 to 22 days after each forecast's as-of time. Forecasts were issued walk-forward in as-of order, and a calibration map was refit before each question using only questions already resolved. Under the Brier score the calibrated ensemble (0.284, 95% CI [0.236, 0.337]) was worse than the market price observed at the same as-of time (0.212, [0.174, 0.255]); the paired difference on the 73 questions with a price was +0.085 [+0.019, +0.154]. Walk-forward calibration was the only intervention whose gain excluded zero (−0.030 [−0.060, −0.003] against the uncalibrated ensemble). Post-hoc diagnostics, outside the registered analysis, locate the deficit: every model arm under-forecast YES (calibrated ensemble mean forecast 0.370 against a YES rate of 0.518; bias −0.148 [−0.259, −0.040]), and on the questions with a market price the ensemble's reliability term was more than four times the market's (0.060 against 0.014) while its resolution term was less than half (0.015 against 0.036). The result fixes a public floor that any later ARCANE forecaster must clear.

Keywords probabilistic forecasting · Brier score · calibration · prediction markets · large language models · walk-forward evaluation

1Introduction

Language-model forecasting systems are usually evaluated after retrieval, aggregation and a learned weight on the crowd have all been added [1, 2]. Reported gains then cannot be attributed: an improvement may come from the evidence the system read, from combining several models, from calibration, or from the market price it was allowed to see. A benchmark needs a floor, measured before any of those levers is pulled, against which each later addition is scored on the same questions.

This report establishes that floor for the ARCANE forecasting benchmark. We ask how well an ensemble of current language models forecasts short-horizon binary questions when it is given no evidence dated after its training data, and how that compares with the market price available at the same moment. We make four contributions:

  1. a walk-forward protocol in which every forecast, calibration refit and base rate uses only information available at the question's as-of time (Section 3, Section 4);
  2. a retrieval-free baseline on 83 resolved questions, with 95% bootstrap intervals on every score and paired intervals on every comparison (Section 5);
  3. post-hoc diagnostics (forecast bias, reliability diagrams and the Murphy decomposition) that locate the deficit to the market in two separable terms (Section 5.3);
  4. the aggregate data behind every number and figure, released with the paper (Appendix C).

3Data

3.1Question set

The set holds 83 binary questions (Table 1), each resolved between 1 August and 16 September 2026. For each market, the effective end is the earlier of its scheduled end and its actual close; the as-of time is midnight UTC 14, 10 or 7 days before the effective end, whichever is first to have an observed price. The interval from as-of time to recorded resolution ranges from 7 to 22 days, and the 83 questions fall on 30 distinct as-of dates between 20 July and 6 September 2026.

Selection used only information available before resolution: as-of prices had to lie in [0.03, 0.97], with at most 10% of questions in the tails of that range, and within a ladder of related strikes the market nearest 0.5 was kept. The outcome was never used to select a question. The resulting YES rate is 51.8% (43 of 83). ForecastBench questions carry no market price, so market comparisons use the 73 questions from Polymarket and Kalshi.

Table 1: Composition of the question set by source. "Market price" counts questions with a recorded price at or before the as-of time.
SourceQuestionsResolved YESMarket price
Polymarket512851
Kalshi221122
ForecastBench1040
Total834373

3.2Leakage controls

  • Every question resolved after the published knowledge cutoff of every model in the ensemble; the cutoff of one member is unverified (Section 6).
  • Model arms received no retrieval. Live search in a retrospective backtest would return pages written after resolution.
  • Model-only arms never saw the market price. The blended arm is reported separately.
  • Market prices are the last trade at or before the as-of time, taken from public price histories.
  • Calibration before each question was fitted only on questions that had resolved before that question's as-of time.

4Method

4.1Arms

Five arms forecast the same questions. (i) Single frontier model: one model queried directly. (ii) Ensemble, uncalibrated: several models and prompting strategies pooled in log-odds space. (iii) Calibrated ensemble: arm (ii) passed through a monotone recalibration map refit before each question on the questions resolved by its as-of time; the map is the identity until 20 questions have resolved. (iv) Calibrated ensemble + market blend: arm (iii) blended in log-odds with the as-of market price at a fixed weight of 0.5 that was not fitted. (v) Market price: the as-of price itself, the reference.

Questions were processed in as-of order, so every refit used only outcomes a forecaster could have known. 44 of 83 questions received a fitted map; the other 39 were forecast before 20 resolutions existed.

4.2Scoring

For forecasts p_i of outcomes o_i on N questions, the Brier score is

BS=1N∑i=1N(pi−oi)2,oi∈{0,1}
(1)

and the logarithmic score is

LS=−1N∑i=1N[oilnpi+(1−oi)ln(1−pi)]
(2)

Both are lower-is-better. Probabilities are clipped to [0.01, 0.99] before scoring, so a confident miss costs a finite amount. Expected calibration error (ECE) uses ten equal-width bins [7].

4.3Inference

Each arm's mean score carries a 95% percentile bootstrap interval over questions (10,000 resamples) [16]. Arms are compared on the questions both answered through the paired difference

Δ=1N∑i=1N(BSi,A−BSi,B)
(3)

with a 95% paired percentile bootstrap interval and the share of resamples in which arm A scored better. A difference is called significant only when its interval excludes zero. The arms, metrics and paired comparisons were fixed in the backtest harness before the run; no separate pre-registration document exists for this evaluation.

4.4Post-hoc diagnostics

The following were not part of the analysis fixed before the run and are reported as descriptive. The forecast bias (calibration-in-the-large) of an arm is

b=1N∑i=1N(pi−oi)=p¯−o¯
(4)

with a 95% bootstrap interval. Reliability diagrams use K = 5 equal-width bins. The Murphy decomposition over the same bins, where n_k questions fall in bin k with mean forecast f̄_k and observed frequency ō_k, and ō is the overall YES rate, is

BS≈1N∑k=1Knk(f¯k−o¯k)2−1N∑k=1Knk(o¯k−o¯)2+o¯(1−o¯)
(5)

The three terms are reliability (REL, lower is better), resolution (RES, higher is better) and uncertainty (UNC). With binned forecasts the identity is approximate; we report the within-bin residual rather than absorb it [6].

5Results

5.1Primary comparison

Table 2 and Figure 1 give each arm's scores. Without evidence, every model arm scored worse than the market price: the calibrated ensemble reached a Brier score of 0.284 against the market's 0.212. The same ordering holds under the logarithmic score and ECE. Blending the market price in at an unfitted half weight improved the ensemble to 0.237 but did not reach the market alone.

Table 2: Scores by arm with 95% bootstrap intervals (10,000 resamples). Lower is better for every column. The market arm covers the questions with a recorded as-of price.
ArmnBrier95% CILog score95% CIECE
Market price at the as-of time (reference)730.212[0.174, 0.255]0.626[0.524, 0.747]0.106
Calibrated ensemble + market blend (0.5, not fitted)830.237[0.201, 0.275]0.673[0.586, 0.769]0.153
Calibrated ensemble830.284[0.236, 0.337]0.856[0.689, 1.050]0.196
Single frontier model830.306[0.244, 0.372]0.946[0.741, 1.176]0.282
Ensemble, uncalibrated830.314[0.253, 0.380]0.975[0.764, 1.213]0.247
MARKET 0.212MARKET PRICEN 730.212CALIBRATED ENSEMBLE + MARKET BLEND (0.5)N 830.237CALIBRATED ENSEMBLEN 830.284SINGLE FRONTIER MODELN 830.306ENSEMBLE, UNCALIBRATEDN 830.3140.150.200.250.300.350.40BRIER SCORE · 95% INTERVAL · LOWER IS BETTERMARKET 0.212MARKET PRICEN 730.212CALIBRATED ENSEMBLE + MARKET BLEND (0.5)N 830.237CALIBRATED ENSEMBLEN 830.284SINGLE FRONTIER MODELN 830.306ENSEMBLE, UNCALIBRATEDN 830.3140.150.200.250.300.350.40BRIER SCORE95% INTERVALLOWER IS BETTER
Figure 1: Brier score by arm with its 95% bootstrap interval; lower is better. Each dot is an arm's mean over its questions (n printed beneath the name) and each bar its interval. The dashed line is the market price at the as-of time, 0.212. The calibrated ensemble is marked in red because it is the arm the benchmark scores and it lost to the reference.

5.2Paired comparisons

Table 3 gives the four registered comparisons. Walk-forward calibration was the only intervention whose gain excluded zero (−0.030, [−0.060, −0.003]). On the 44 questions that received a fitted map, it lowered the Brier score from 0.314 to 0.257. The calibrated ensemble probably improves on the single model (P = 0.91), but the interval crosses zero and we make no claim. Against the market it is clearly worse; with the blend, the remaining gap (+0.031) is not distinguishable from zero on this sample.

Table 3: Paired Brier differences, arm A minus arm B, on the questions both arms answered. Negative favours A. 95% paired percentile bootstrap intervals, 10,000 resamples; P is the share of resamples in which A scored better.
Arm AArm BnΔ Brier95% CIP(A better)
Calibrated ensembleEnsemble, uncalibrated83−0.030[−0.060, −0.003]0.985
Calibrated ensembleSingle frontier model83−0.022[−0.056, +0.010]0.91
Calibrated ensembleMarket price73+0.085[+0.019, +0.154]0.006
Calibrated ensemble + market blendMarket price73+0.031[−0.008, +0.072]0.06

5.3Where the deficit comes from (post hoc)

Forecast bias. Every model arm under-forecast YES (Figure 2). The uncalibrated ensemble's mean forecast was 0.318 against a YES rate of 0.518, a bias of −0.200 [−0.312, −0.089]. Calibration reduced but did not remove it (−0.148, [−0.259, −0.040]). The market's bias, −0.032 [−0.139, +0.074], is indistinguishable from zero.

NO BIASMARKET PRICEMEAN FORECAST 0.50 · YES 0.53−0.032CALIBRATED ENSEMBLE + MARKET BLEND (0.5)MEAN FORECAST 0.42 · YES 0.52−0.098CALIBRATED ENSEMBLEMEAN FORECAST 0.37 · YES 0.52−0.148SINGLE FRONTIER MODELMEAN FORECAST 0.36 · YES 0.52−0.160ENSEMBLE, UNCALIBRATEDMEAN FORECAST 0.32 · YES 0.52−0.200−0.3−0.2−0.10.00.1MEAN FORECAST − YES RATE · 95% INTERVAL · 0 IS UNBIASEDNO BIASMARKET PRICEMEAN FORECAST 0.50 · YES 0.53−0.032CALIBRATED ENSEMBLE + MARKET BLEND (0.5)MEAN FORECAST 0.42 · YES 0.52−0.098CALIBRATED ENSEMBLEMEAN FORECAST 0.37 · YES 0.52−0.148SINGLE FRONTIER MODELMEAN FORECAST 0.36 · YES 0.52−0.160ENSEMBLE, UNCALIBRATEDMEAN FORECAST 0.32 · YES 0.52−0.200−0.3−0.2−0.10.00.1MEAN FORECAST − YES RATE95% INTERVAL0 IS UNBIASED
Figure 2: Forecast bias by arm: mean forecast minus the observed YES rate on that arm's questions, with its 95% bootstrap interval. Zero (dashed) is unbiased; negative means the arm forecast YES too rarely. Marks are red where the interval excludes zero. Post hoc.

Reliability. The bias is concentrated in confident NO forecasts (Figure 3). The calibrated ensemble placed 18 questions in the lowest bin, at a mean forecast of 0.085; 38.9% of them resolved YES. The uncalibrated ensemble placed 31 questions there (mean 0.083, 38.7% YES). Forecasts below 0.1 or above 0.9 were costly: the calibrated ensemble's 11 such forecasts averaged a Brier score of 0.43, the uncalibrated ensemble's 21 averaged 0.40 and the single model's 19 averaged 0.48. Full bin counts are in Appendix A.

MARKET PRICE · N 73000.50.5119739108ENSEMBLE, UNCALIBRATED · N 83000.50.51131202624CALIBRATED ENSEMBLE · N 83000.50.51118204023MEAN FORECAST IN BIN (ACROSS) · SHARE RESOLVED YES (UP) · DASHED: PERFECT CALIBRATIONMARKET PRICE · N 73000.50.5119739108ENSEMBLE, UNCALIBRATED · N 83000.50.51131202624CALIBRATED ENSEMBLE · N 83000.50.51118204023ACROSS: MEAN FORECAST · UP: SHARE YES
Figure 3: Reliability diagrams in five equal-width bins. Each dot is one bin: across, the mean forecast of the questions in it; up, the share that resolved YES. Dot area and the printed number give the questions in the bin; empty bins are not drawn. The dashed diagonal is perfect calibration. Points above it are forecasts that were too low. Post hoc.

Decomposition. On the 73 questions with a market price (Table 4), the calibrated ensemble's reliability term was 0.060 against the market's 0.014, and its resolution term 0.015 against 0.036. The market is ahead on both terms, and by more on reliability than on resolution. Across all 83 questions, calibration lowered the reliability term from 0.072 to 0.049 but did not raise resolution (0.020 to 0.014): a monotone map can repair miscalibration, but it cannot separate outcomes the forecasts did not already separate. The within-bin residuals are not negligible (−0.0148 for the market, +0.0173 for the uncalibrated ensemble), so these binned terms are approximate and the comparison is descriptive.

Table 4: Murphy decomposition in five equal-width bins, on the 73 questions with a market price. BS ≈ REL − RES + UNC; the residual is BS − (REL − RES + UNC). Post hoc.
ArmBrierREL ↓RES ↑UNCResidual
Market price at the as-of time0.21230.01410.03590.2488−0.0148
Calibrated ensemble + market blend (0.5)0.24370.01980.02220.2488−0.0027
Calibrated ensemble0.29760.06030.01450.2488+0.0029
Single frontier model0.32490.07900.00880.2488+0.0058
Ensemble, uncalibrated0.33600.08810.01830.2488+0.0173

5.4By source (post hoc)

The deficit holds within each market source (Table 5): on Polymarket the calibrated ensemble scored 0.283 against the market's 0.220, and on Kalshi 0.331 against 0.194. On the 10 ForecastBench questions, which carry no market price, the model arms scored between 0.157 and 0.186; with ten questions no inference is attempted.

Table 5: Brier score by source and arm. Intervals are omitted because the subsets are small; this table is descriptive and post hoc. A dash marks an arm with no forecast on that source.
SourcenMarketBlendCalibratedSingleUncalibrated
Polymarket510.2200.2420.2830.3030.310
Kalshi220.1940.2480.3310.3760.397
ForecastBench10–0.1860.1860.1710.157

6Discussion and limitations

Without evidence gathered at the as-of time, current language models do not match a liquid market on short-horizon questions. The diagnostics suggest two distinct causes. The first is a systematic under-forecast of YES, most visible in confident NO forecasts that were often wrong; this is a reliability failure, and walk-forward calibration repaired part of it. The second is lower resolution: the models' forecasts separated YES from NO outcomes less well than the price did. Calibration cannot supply resolution, which requires information about the particular question. The implication for the benchmark is an ordering of interventions: as-of retrieval first, since it is the only lever that can raise resolution, then a fitted weight on the market price. This agrees with the finding that most of a retrieval-augmented system's gain came from retrieval and crowd combination [1].

A plausible reading of the NO bias is that a model without recent context defaults to the status quo ("the event will not happen by the date"), while the selection rule, which kept the market nearest 0.5 within a ladder, produced a set balanced between outcomes. That reading is a hypothesis; this study does not test it.

6.1Limitations

  • The question set was assembled after resolution. The outcome was never used to select a question, but selection by volume and by proximity to 0.5 within a ladder can still bias the set.
  • Market prices are the last trade at or before the as-of time from public price histories, not a quoted midpoint.
  • Question text may carry wording added to market pages after the as-of time.
  • The knowledge cutoff of one ensemble member is unverified, and its hosted deployment may route to a newer version.
  • With n = 83 (73 against the market) the intervals are wide. This is a method dry run, not a claim of skill or of its absence in general.
  • The diagnostics in Section 5.3 and Section 5.4 were chosen after the registered results were known. They describe this run and carry no confirmatory weight.
  • A planned external learning-system arm (Adapt-1) abstained on every query and then became unavailable, so it is not evaluated here.

6.2What would change these conclusions

  1. A prospective window, scored after publication, in which walk-forward calibration no longer improves the ensemble.
  2. An as-of retrieval arm that matches or beats the market price, with a paired interval, on questions it has never seen.
  3. A replication on a larger set in which the model arms show no NO bias, which would make the bias found here a property of this sample.

7Conclusion

On 83 resolved questions, a calibrated ensemble of language models without evidence scored 0.284 under the Brier score against 0.212 for the market price at the same moment. Walk-forward calibration helped and was the only intervention to do so significantly. The deficit that remains has a calibration part and an information part, and only the second requires new evidence. Every later ARCANE forecaster is scored on this protocol and must clear this floor before it may claim skill.

References

  1. [1]Halawi, D., Zhang, F., Chen, Y.-H. and Steinhardt, J. (2024). Approaching human-level forecasting with language models. arXiv:2402.18563.
  2. [2]Karger, E., Bastani, H., Chen, Y.-H., Jacobs, Z., Halawi, D., Zhang, F. and Tetlock, P. E. (2024). ForecastBench: a dynamic benchmark of AI forecasting capabilities. arXiv:2409.19839.
  3. [3]Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review 78(1), 1–3.
  4. [4]Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102(477), 359–378.
  5. [5]Murphy, A. H. (1973). A new vector partition of the probability score. Journal of Applied Meteorology 12(4), 595–600.
  6. [6]Stephenson, D. B., Coelho, C. A. S. and Jolliffe, I. T. (2008). Two extra components in the Brier score decomposition. Weather and Forecasting 23(4), 752–757.
  7. [7]Pakdaman Naeini, M., Cooper, G. F. and Hauskrecht, M. (2015). Obtaining well calibrated probabilities using Bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence 29(1).
  8. [8]Guo, C., Pleiss, G., Sun, Y. and Weinberger, K. Q. (2017). On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning; arXiv:1706.04599.
  9. [9]Baron, J., Mellers, B. A., Tetlock, P. E., Stone, E. and Ungar, L. H. (2014). Two reasons to make aggregated probability forecasts more extreme. Decision Analysis 11(2), 133–145.
  10. [10]Satopää, V. A., Baron, J., Foster, D. P., Mellers, B. A., Tetlock, P. E. and Ungar, L. H. (2014). Combining multiple probability predictions using a simple logit model. International Journal of Forecasting 30(2), 344–356.
  11. [11]Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Smola, A. J. et al. (eds.), Advances in Large Margin Classifiers, MIT Press, 61–74.
  12. [12]Wolfers, J. and Zitzewitz, E. (2004). Prediction markets. Journal of Economic Perspectives 18(2), 107–126.
  13. [13]Zou, A., Xiao, T., Jia, R., Kwon, J., Mazeika, M., Li, R., Song, D., Steinhardt, J., Evans, O. and Hendrycks, D. (2022). Forecasting future world events with neural networks. arXiv:2206.15474.
  14. [14]Schoenegger, P. and Park, P. S. (2023). Large language model prediction capabilities: evidence from a real-world forecasting tournament. arXiv:2310.13014.
  15. [15]Mellers, B., Ungar, L., Baron, J., Ramos, J., Gurcay, B., Fincher, K., Scott, S. E., Moore, D., Atanasov, P., Swift, S. A., Murray, T., Stone, E. and Tetlock, P. E. (2014). Psychological strategies for winning a geopolitical forecasting tournament. Psychological Science 25(5), 1106–1115.
  16. [16]Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall, New York.

Appendix

AReliability bins

The numbers drawn in Figure 3, for every arm. Each cell reads n / mean forecast / share resolved YES; a dash marks an empty bin.

Table 6: Five equal-width reliability bins by arm (n / mean forecast / observed YES). The last bin includes 1. Post hoc.
Arm[0, 0.2)[0.2, 0.4)[0.4, 0.6)[0.6, 0.8)[0.8, 1]
Market price at the as-of time9 / 0.14 / 0.227 / 0.32 / 0.1439 / 0.50 / 0.6210 / 0.68 / 0.608 / 0.87 / 0.75
Calibrated ensemble + market blend (0.5)11 / 0.12 / 0.3618 / 0.29 / 0.2844 / 0.49 / 0.619 / 0.68 / 0.671 / 0.82 / 1.00
Calibrated ensemble18 / 0.08 / 0.3920 / 0.33 / 0.5040 / 0.47 / 0.602 / 0.68 / 0.003 / 0.87 / 0.67
Single frontier model27 / 0.09 / 0.4122 / 0.31 / 0.5922 / 0.51 / 0.505 / 0.66 / 0.807 / 0.89 / 0.57
Ensemble, uncalibrated31 / 0.08 / 0.3920 / 0.31 / 0.6026 / 0.49 / 0.622 / 0.68 / 0.004 / 0.88 / 0.75

BSource boundary

Published
question sources, selection rule, windows, arms, scores, intervals, leakage controls and aggregate diagnostics.
Withheld
ensemble membership and weighting, fitted calibration parameters, prompts, per-question forecasts and the provenance format.

CData and code availability

Every number in this paper comes from a file committed to the ARCANE repository. The files, and what each holds:

data/forecaster/backtest-2026-09.jsonl
The 83 questions: text, source, as-of time, as-of market price, outcome and resolution time.
docs/FORECASTER.md
Section 3: the run's own report of arm scores, intervals and paired differences (Tables 2 and 3).
data/research/forecaster-bt-20260917-v1-diagnostics.json
Aggregates behind Section 5.3, Section 5.4, Figures 2 and 3, Tables 4, 5 and 6. Each headline number is re-checked against the run before the file is written.
scripts/research/brier-diagnostics.mjs
The derivation of the diagnostics from the run's per-question records, which stay private because they would expose the calibration map.
scripts/forecaster-build-backtest.mjs
The question selection rule.

DRevision history

  1. v2Rewritten as a paper: abstract, related work, formal method with equations, references and appendices. The horizon is corrected from "7 to 21 days" to "7 to 22 days" (as-of time to recorded resolution). Post-hoc diagnostics added in Section 5.3 and Section 5.4, labelled as post hoc. No registered number changed.
  2. v1First public edition, as a short evaluation note. The walk-forward run itself completed on 17 September 2026.

Cite as

ARCANE Research (2026). Models without evidence lose to markets. ARCANE Research Technical Report 05, v2, 25 September 2026. https://arcane.markets/research/models-without-evidence

@techreport{arcane2026modelswithout,
  title       = {Models without evidence lose to markets},
  author      = {{ARCANE Research}},
  institution = {ARCANE},
  type        = {Technical Report},
  number      = {05},
  year        = {2026},
  month       = sep,
  note        = {Version 2},
  url         = {https://arcane.markets/research/models-without-evidence}
}
Models without evidence lose to markets · Research · ARCANE