ArcaneResearch

Technical report 04Structure & learning

A learner that knows when to say nothing

v2, First version PaperNull result

Abstract

A pattern-finder applied to enough data will find patterns, including in noise, so its useful property is the ability to decline. We evaluate the ARCANE structural learner, which forecasts a binary outcome only when a validated rule applies and otherwise abstains to the base rate, on three synthetic worlds with known generating rules and on one public series. Across 30 held-out seeded runs, no rule on a designated noise field was validated (0 of 30). Where structure was planted, the learner recovered the single-condition rule in every seed, beat a rolling base rate by −0.0285 [−0.0339, −0.0231] and −0.0268 [−0.0315, −0.0222] in Brier score, and closed 28% and 30% of the gap to an oracle; after an unannounced regime switch it retired the old rule in 10 of 10 seeds. In a world with no structure it spoke on 3.4% of steps and paid +0.0025 [+0.0019, +0.0031] for doing so. On 515 walk-forward predictions of the direction of the US high-yield option-adjusted spread, 52 discovery rounds produced no rule that survived validation; the learner abstained on every prediction and matched climatology to four decimal places (0.2434). A secondary experiment shows that shift-sensitive reweighting of forecast sources recovers faster than fixed reweighting after sources swap roles. We report the null result on real data as the principal finding.

Keywords selective prediction · abstention · rule discovery · data snooping · concept drift · credit spreads · Brier score

1Introduction

Searching many candidate rules on one history guarantees that some will look predictive by chance. The problem is well documented in econometrics and finance, where specification searches produce significant-looking effects that do not survive out of sample [1, 2, 3]. For a system that proposes rules continuously, the relevant property is therefore not whether it finds structure, but whether it declines to act when there is none, and whether it withdraws a rule when the world changes.

We study this property in the ARCANE structural learner, which issues a probability only when a rule has passed out-of-sample validation and otherwise abstains to the base rate. We ask four questions: does it recover structure that is known to exist; does it refuse structure that is known not to exist; does it adapt when the generating rule changes; and what does it do on a real series where the answer is unknown? Our contributions are:

  1. a synthetic benchmark with known generating rules, designated noise fields, a transient spurious correlate and a regime switch, scored against both a base rate and an oracle (Section 3);
  2. recovery, rejection and adaptation results on 30 held-out seeds, with every run listed (Section 5, Appendix A);
  3. a null result on 515 walk-forward predictions of a public credit series, against climatology and persistence (Section 5.4);
  4. a secondary experiment on reweighting forecast sources after they swap roles (Section 5.5).

2Related work

Abstention. Classification with a reject option trades coverage for error [4]; selective prediction makes the trade explicit by reporting risk at a chosen coverage [5]. In online prediction with expert advice, specialists that may abstain on some rounds can be combined without penalising their silence [6]. The learner studied here is a specialist in this sense: its silence is scored at the base rate, so abstention costs nothing relative to the reference and speaking must earn its place.

Data snooping. White's reality check tests whether the best of many rules beats a benchmark once the search is accounted for [1]. Multiple-testing corrections for factor discovery raise the bar for significance substantially [2], and backtest overfitting can make an in-sample optimum worthless out of sample [3]. We do not correct a family of tests; we measure directly, on designated noise, how often the learner's validation admits a spurious rule.

Non-stationarity. A learner that must follow a changing world faces concept drift [7]; tracking the best expert under switching bounds the cost of following the currently best source [8]. Evaluation is prequential: each forecast is scored before its outcome is admitted [9]. Intervals on a dependent series use the moving-block bootstrap [10].

3Data

3.1Synthetic worlds

Each synthetic stream has 1200 steps of several numeric and categorical fields and a binary outcome, generated from recorded rules so recovery and rejection can be scored exactly (Table 1). Three fields in every world are pure noise, and one field is a transient correlate that tracks the outcome only in the first quarter of the stream. Each world was run on ten seeds (101 to 110); the learner's default settings were tuned on seeds 1 to 20, so these seeds are held out.

Table 1: The three synthetic worlds. Every world also carries three pure-noise fields and a transient correlate that is informative only before step 300.
WorldGenerating structureWhat it tests
Lagged chainA single-condition rule on a field two steps back, and a two-condition conjunction on two other fieldsRecovery of simple and compound structure
Regime switchA single-condition rule on a lagged field until step 600; afterwards a different field, one step back, drives the outcome and the first is irrelevantRetirement of a stale rule and discovery of its replacement
No structureThe outcome is independent of every fieldRefusal: the cost of speaking when there is nothing to say

3.2Credit-spread series

The real test uses the ICE BofA US High Yield Index option-adjusted spread, published by the Federal Reserve Bank of St. Louis as FRED series BAMLH0A0HYM2 [11]: 786 daily observations from 2023-09-18 to 2026-09-15. The file's SHA-256 digest is recorded with the result so the exact series can be re-checked. The target on day t is whether the next observation is strictly higher than the current one; its base rate over the scored period is 0.4155.

4Method

4.1Learner and protocol

The learner proposes candidate rules from the history it has admitted, validates them out of sample, and forecasts with a rule only while that rule remains validated. Its forecast at step t is

pt={qt,if a validated rule applies at ty¯t,otherwise (abstention)
(1)

where q_t is the forecast of the applicable validated rule and ȳ_t the base rate; coverage is the share of steps on which a rule applied. In the synthetic worlds discovery runs every 25 steps from step 150, and forecasts are scored prequentially from step 300 [9]: each outcome is admitted only after the forecast for it is sealed. On the credit series the learner warms up for 250 days, then forecasts every later day, admits the previous day's resolved outcome and reruns discovery every ten days, giving 515 predictions. Every revision to the rule set is appended to a hash-chained log, and each run verifies the chain at the end.

4.2Baselines and metrics

In the synthetic worlds the learner is compared with a rolling-window base rate and with an oracle that forecasts the true generating probability, which bounds the attainable score. On the credit series the baselines are climatology (the expanding base rate) and persistence, the probability of a rise given the sign of today's change, estimated from expanding counts with a Beta(1, 1) prior. The score is the Brier score; we also report the skill score against the base rate and the share of the attainable gain,

SS=1−BSlearnerBSbase,G=BSbase−BSlearnerBSbase−BSoracle
(2)

A planted rule counts as recovered when a validated rule uses the same fields and lags in the same direction, with numeric thresholds within 0.15 of the truth, while the planted rule is in force. A rule counts as a noise rule when it uses any of the three designated noise fields.

4.3Inference

Synthetic results are means over ten seeds with 95% percentile bootstrap intervals over seeds (4,000 resamples) [12]. Credit-series intervals use the moving-block bootstrap with block length round(n^(1/3)) and 4,000 resamples, for each model and for the paired differences [10]. The experiments are deterministic: a seed fixes every stream, and no network or model call is involved.

5Results

5.1Recovery of planted structure

In both structured worlds the single-condition rule was recovered in 10 of 10 seeds, at the first discovery round (median step 150). The two-condition conjunction was harder: it was recovered in 8 of 10 seeds, at a median of step 750. Table 2 and Figure 1 give the scores. The learner beat the base rate in both structured worlds, with intervals well clear of zero, and closed 28% (lagged chain) and 30% (regime switch) of the distance between the base rate and the oracle. Skill scores against the base rate were 0.114 [0.093, 0.136] and 0.113 [0.095, 0.132].

Table 2: Brier score by world, mean over ten held-out seeds with 95% bootstrap intervals over seeds. Lower is better. The difference column is learner minus base rate; coverage is the share of scored steps on which the learner spoke.
WorldSeedsLearnerBase rateOracleLearner − baseCoverage
Lagged chain100.2204 [0.2154, 0.2256]0.2488 [0.2477, 0.2498]0.1465 [0.1396, 0.1543]−0.0285 [−0.0339, −0.0231]42.5% [37.6%, 48.1%]
Regime switch100.2094 [0.2045, 0.2143]0.2362 [0.2330, 0.2390]0.1464 [0.1395, 0.1538]−0.0268 [−0.0315, −0.0222]27.8% [24.3%, 31.0%]
No structure100.2300 [0.2274, 0.2328]0.2275 [0.2249, 0.2302]0.2266 [0.2240, 0.2295]+0.0025 [+0.0019, +0.0031]3.4% [1.1%, 6.9%]
LEARNERBASE RATEORACLELAGGED CHAIN10 SEEDSREGIME SWITCH10 SEEDSNO STRUCTURE10 SEEDS0.120.160.200.24BRIER SCORE · 95% INTERVAL OVER SEEDS · LOWER IS BETTERLEARNERBASE RATELAGGED CHAIN · 10 SEEDSREGIME SWITCH · 10 SEEDSNO STRUCTURE · 10 SEEDS0.120.160.200.24BRIER SCORE95% INTERVAL OVER SEEDSLOWER IS BETTER
Figure 1: Brier score by world, with 95% bootstrap intervals over ten seeds; lower is better. Filled circles are the learner, open squares the rolling base rate, open circles the oracle that knows the generating probability. Where structure exists the learner sits between the base rate and the oracle; where it does not, all three nearly coincide.

5.2Adaptation after a regime switch

After the switch at step 600, the stale rule was retired in 10 of 10 seeds, after a mean of 84.1 steps [39.9, 133.1], and a rule on the new driver was validated in 10 of 10, after 200 steps [160, 250]. An explicit regime-shift signal fired in only 5 of 10 seeds (after 52.8 steps [20.8, 106.6]); in the others the stale rule was retired by ordinary revalidation. Over the post-switch period the learner still beat the base rate by −0.0232 [−0.0274, −0.0188].

5.3Refusal of absent structure

No rule on a designated noise field was validated in any of the 30 runs. The transient correlate was different: it genuinely tracks the outcome before step 300, rules on it were validated in every world, and none remained in force at the end of any seed. In the world with no structure the learner validated 18 rules across the ten seeds (between 1 and 3 per seed; Appendix A). The experiment recorded whether each used a noise field, but not which of the other fields it used, so we cannot say how many rested on the transient correlate and how many on fields that are uninformative in that world. The measurable consequence is small: the learner spoke on 3.4% [1.1%, 6.9%] of scored steps and its Brier score was +0.0025 [+0.0019, +0.0031] worse than the base rate. In that world the oracle (0.2266) is itself indistinguishable from the base rate (0.2275), as it should be.

NO DIFFERENCELAGGED CHAINVS BASE RATE−0.0285REGIME SWITCHVS BASE RATE−0.0268REGIME SWITCH, AFTERVS BASE RATE−0.0232NO STRUCTUREVS BASE RATE0.0025US HIGH-YIELD OASVS CLIMATOLOGY0.0000US HIGH-YIELD OASVS PERSISTENCE−0.0001−0.03−0.02−0.010.000.01LEARNER − BASELINE, BRIER · 95% INTERVAL · NEGATIVE FAVOURS THE LEARNERNO DIFFERENCELAGGED CHAINVS BASE RATE−0.0285REGIME SWITCHVS BASE RATE−0.0268REGIME SWITCH, AFTERVS BASE RATE−0.0232NO STRUCTUREVS BASE RATE0.0025US HIGH-YIELD OASVS CLIMATOLOGY0.0000US HIGH-YIELD OASVS PERSISTENCE−0.0001−0.03−0.02−0.010.000.01LEARNER − BASELINE, BRIER95% INTERVALNEGATIVE FAVOURS THE LEARNER
Figure 2: Learner minus baseline, in Brier score, with 95% intervals: bootstrap over seeds for the synthetic worlds, moving-block bootstrap for the credit series. Negative means the learner was better; zero (dashed) is no difference. The one mark in red is the world with no structure, where speaking cost the learner a small, significant amount.

5.4A null result on credit spreads

On the high-yield spread, 52 discovery rounds produced no rule that survived validation, and the learner abstained on all 515 predictions (coverage 0). Its Brier score, 0.2434 [0.2355, 0.2515], is climatology's to four decimal places (0.2434 [0.2353, 0.2512]), by construction. Persistence did no better (0.24347 [0.23569, 0.25185]; learner minus persistence −0.00007 [−0.00109, +0.00099]). All three models called the direction correctly on 58.45% of days, the share of days on which the spread did not rise (Table 3).

The five strongest candidates rejected at the end of the run had between 214 and 405 supporting observations. All five had negative penalised out-of-sample skill (−0.0006 to −0.0024), and the lower confidence bound on each one's hit rate (0.331 to 0.361) lay below the base rate of 0.415, so none was shown to beat it. The revision log recorded 53 entries and its hash chain verified.

Table 3: Walk-forward direction forecasts of the US high-yield OAS, 515 predictions. Brier score with 95% moving-block bootstrap intervals; lower is better.
ModelBrier95% CIDirection accuracy
Structural learner (abstained throughout)0.2434[0.2355, 0.2515]58.45%
Climatology0.2434[0.2353, 0.2512]58.45%
Persistence, Beta(1, 1)0.24347[0.23569, 0.25185]58.45%

5.5Reweighting sources after a role swap

A secondary experiment asks whether the learner's source-reweighting component follows a change in which source is informative. Three simulated sources forecast the same outcome for 500 steps: one informed, one uninformative and one inverted; at step 250 the informed and inverted sources swap roles. We compare reweighting with and without shift sensitivity ("plastic" and "fixed"), equal-weight pooling in log-odds, selection of the source with the lowest discounted past loss, and the informed source itself as an upper bound, at a shorter and a longer memory setting, on ten seeds.

After the swap (Figure 3, Table 4), plastic reweighting beat fixed reweighting at both settings: by −0.0033 [−0.0053, −0.0013] with the shorter memory and by −0.0400 [−0.0459, −0.0342] with the longer, where a fixed learner is slowest to forget. It also beat best-past selection (−0.0054 [−0.0078, −0.0031] and −0.0069 [−0.0101, −0.0037]). The weight on the newly informed source overtook the newly inverted one after 22.0 steps [15.8, 29.0] against 32.8 [28.0, 37.7] for the fixed learner, and after 21.3 [14.9, 27.0] against 99.0 [87.1, 110.7] with the longer memory. Plasticity has a small price before any shift: +0.0005 [+0.0000, +0.0014] and +0.0012 [+0.0000, +0.0026]. A shift was flagged in 7 and 10 of 10 seeds respectively.

SHORTER MEMORYLONGER MEMORYPLASTIC REWEIGHTINGFIXED REWEIGHTINGBEST PAST SOURCEEQUAL-WEIGHT POOLINFORMED SOURCE0.190.210.230.250.27BRIER SCORE AFTER THE SWAP · 95% INTERVAL · LOWER IS BETTERSHORTER MEMORYLONGER MEMORYPLASTIC REWEIGHTINGFIXED REWEIGHTINGBEST PAST SOURCEEQUAL-WEIGHT POOLINFORMED SOURCE0.190.210.230.250.27BRIER SCORE AFTER THE SWAP95% INTERVALLOWER IS BETTER
Figure 3: Brier score over the 250 steps after the sources swap roles, by method, with 95% bootstrap intervals over ten seeds; lower is better. Filled circles use the shorter memory setting and open circles the longer. The informed source is an upper bound that no method can see in advance.
Table 4: Brier score after the swap (mean over ten seeds, 95% bootstrap interval), and steps until the weight on the newly informed source overtakes the newly inverted one.
MethodShorter memoryLonger memory
Plastic reweighting0.2216 [0.2165, 0.2269]0.2201 [0.2143, 0.2259]
Fixed reweighting0.2249 [0.2196, 0.2301]0.2601 [0.2541, 0.2659]
Best past source0.2270 [0.2204, 0.2337]0.2270 [0.2204, 0.2337]
Equal-weight pool0.2513 [0.2503, 0.2522]0.2513 [0.2503, 0.2522]
Informed source (bound)0.2037 [0.1991, 0.2081]0.2037 [0.1991, 0.2081]
Steps to recover, plastic22.0 [15.8, 29.0]21.3 [14.9, 27.0]
Steps to recover, fixed32.8 [28.0, 37.7]99.0 [87.1, 110.7]

5.6An external comparison that did not run

A head-to-head against an external continual-learning service was attempted on 5 shorter synthetic runs. All 5 were invalid: in 1 the service's ingestion failed, so it never saw the stream it was queried on, and in 4 the service refused the run. We report the attempt for completeness and draw no conclusion from it.

6Discussion and limitations

The synthetic results establish two necessary properties: the learner finds structure that exists and does not validate rules on fields designated as noise. The credit result is the informative one. A learner that had produced a rule for the high-yield spread would have made a stronger headline and a weaker claim: its best candidates had substantial support yet negative penalised skill, which is what a specification search over a single history tends to produce [1, 3]. Abstaining, and so scoring exactly the base rate, is the correct output when validation finds nothing.

Two results qualify the synthetic evidence. The learner recovered less than a third of the attainable gain, so validation is conservative, and in the world with no structure it did speak occasionally, at a small, significant cost. Both are the expected price of a rule-based learner that must see support before it acts.

6.1Limitations

  • The synthetic worlds were designed by us. Recovering their structure is a necessary test, not a sufficient one.
  • One real series and one target. A null on the high-yield spread says nothing about other series or horizons.
  • In the world with no structure, the fields used by the 18 validated rules were not recorded, so the refusal claim is limited to the three designated noise fields.
  • The source-reweighting experiment uses simulated sources with a single, abrupt swap; real sources degrade gradually and partially.
  • Ten seeds per world give intervals over seeds, not over generating designs; a different design could behave differently.
  • The external comparison did not produce a valid run and so provides no evidence either way.

6.2What would change these conclusions

  1. A rule validated on a designated noise field in a fresh set of seeds.
  2. A rule on a real series that validates and then loses to the base rate on data it has never seen.
  3. A world with no structure in which the learner's cost of speaking grows with the length of the stream.

7Conclusion

On synthetic worlds with known answers, the ARCANE structural learner recovered planted structure, retired it after a regime switch, and validated no rule on designated noise in 30 runs. On 515 predictions of a real credit series it found nothing it could validate and said nothing, matching climatology exactly. We treat that null as the finding: it is the behaviour that makes a positive result from the same learner, on another series, worth believing.

References

  1. [1]White, H. (2000). A reality check for data snooping. Econometrica 68(5), 1097–1126.
  2. [2]Harvey, C. R., Liu, Y. and Zhu, H. (2016). … and the cross-section of expected returns. Review of Financial Studies 29(1), 5–68.
  3. [3]Bailey, D. H., Borwein, J. M., López de Prado, M. and Zhu, Q. J. (2014). Pseudo-mathematics and financial charlatanism: the effects of backtest overfitting on out-of-sample performance. Notices of the American Mathematical Society 61(5), 458–471.
  4. [4]Chow, C. K. (1970). On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16(1), 41–46.
  5. [5]Geifman, Y. and El-Yaniv, R. (2017). Selective classification for deep neural networks. Advances in Neural Information Processing Systems 30; arXiv:1705.08500.
  6. [6]Freund, Y., Schapire, R. E., Singer, Y. and Warmuth, M. K. (1997). Using and combining predictors that specialize. Proceedings of the 29th Annual ACM Symposium on Theory of Computing, 334–343.
  7. [7]Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M. and Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys 46(4), 44.
  8. [8]Herbster, M. and Warmuth, M. K. (1998). Tracking the best expert. Machine Learning 32(2), 151–178.
  9. [9]Dawid, A. P. (1984). Present position and potential developments: some personal views. Statistical theory: the prequential approach. Journal of the Royal Statistical Society, Series A 147(2), 278–292.
  10. [10]Künsch, H. R. (1989). The jackknife and the bootstrap for general stationary observations. Annals of Statistics 17(3), 1217–1241.
  11. [11]ICE Data Indices, LLC (2026). ICE BofA US High Yield Index Option-Adjusted Spread [BAMLH0A0HYM2]. Retrieved from FRED, Federal Reserve Bank of St. Louis.
  12. [12]Efron, B. and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman & Hall, New York.

Appendix

AEvery run

All thirty synthetic runs, so any one can be checked against the committed output. "First validated" gives the step at which each planted rule was first validated (a dash where it never was). Each run's hash chain verified.

Table 5: Per-seed results: Brier score of the learner, the base rate and the oracle; coverage; the step at which each planted rule was first validated; rules validated; rules on a noise field.
WorldSeedLearnerBaseOracleCoverageFirst validatedValidatedNoise
Lagged chain1010.21930.24520.132742.4%150 / 52550
Lagged chain1020.22180.24940.168740.4%150 / 90060
Lagged chain1030.21560.24850.146634.8%150 / 95070
Lagged chain1040.20390.25020.136262.8%150 / 475120
Lagged chain1050.21630.25180.160346.1%150 / 32560
Lagged chain1060.22040.24760.134238.6%150 / –60
Lagged chain1070.23720.24840.141629.0%150 / 75090
Lagged chain1080.22150.25010.148649.0%150 / –90
Lagged chain1090.21760.24760.138238.1%150 / 90070
Lagged chain1100.23000.24960.158144.0%150 / 57580
Regime switch1010.19690.22540.126633.1%150 / 72550
Regime switch1020.21700.23760.160931.2%150 / 72550
Regime switch1030.20980.23780.142920.4%150 / 75050
Regime switch1040.20710.24360.142431.3%150 / 77550
Regime switch1050.22480.23800.157520.9%150 / 87560
Regime switch1060.20950.23070.138132.1%150 / 82560
Regime switch1070.19740.23620.141328.6%150 / 75060
Regime switch1080.20890.24060.141834.7%150 / 97560
Regime switch1090.21370.23440.145226.6%150 / 82560
Regime switch1100.20910.23800.167118.9%150 / 77550
No structure1010.22680.22450.22351.4%n/a10
No structure1020.22410.22170.22121.1%n/a20
No structure1030.23860.23410.233817.4%n/a30
No structure1040.23540.23410.23425.6%n/a30
No structure1050.22550.22220.22181.3%n/a20
No structure1060.23130.23010.22954.3%n/a10
No structure1070.23200.22890.22750.4%n/a10
No structure1080.22750.22460.22350.4%n/a30
No structure1090.22700.22410.22350.6%n/a10
No structure1100.23150.23040.22781.4%n/a10

BSource boundary

Published
worlds, protocol, baselines, scores, intervals, recovery and rejection counts, every run, and the credit series with its digest.
Withheld
the rule grammar, validation thresholds and penalties, feature construction, and the reweighting learner's settings.

CData and code availability

Every number in this paper comes from a file committed to the ARCANE repository. The files, and what each holds:

data/structural-learner/2026-09-17/synthetic.json
The synthetic and source-reweighting experiments: summaries with intervals and every run (Tables 2, 4 and 5; Figures 1, 2 and 3).
data/structural-learner/2026-09-17/hy-oas-walk-forward.json
The credit-series experiment: series digest and range, protocol, scores and intervals, rejected candidates (Table 3; Figure 2).
data/structural-learner/2026-09-17/head-to-head.json
The invalid external comparison, run by run.
scripts/structural-learner/experiment-synthetic.mjs, experiment-series.mjs
The code that produced both files; deterministic given the seeds and the series file.

DRevision history

  1. v2Rewritten as a paper: related work, formal protocol, oracle bounds and the share of attainable gain, per-seed accounting (Appendix A), the regime-switch timings with intervals, the credit-spread baselines including persistence, the source-reweighting experiment and an account of the invalid external comparison. Corrected: v1 said no rule "built on noise" was validated; the experiment counts only rules on the three designated noise fields, and in the world with no structure it validated 18 rules whose fields it did not record (Section 5.3). No number changed.
  2. v1First public edition, as a short note. The experiments ran on 17 September 2026.

Cite as

ARCANE Research (2026). A learner that knows when to say nothing. ARCANE Research Technical Report 04, v2, 25 September 2026. https://arcane.markets/research/learner-that-abstains

@techreport{arcane2026learnerthat,
  title       = {A learner that knows when to say nothing},
  author      = {{ARCANE Research}},
  institution = {ARCANE},
  type        = {Technical Report},
  number      = {04},
  year        = {2026},
  month       = sep,
  note        = {Version 2},
  url         = {https://arcane.markets/research/learner-that-abstains}
}
A learner that knows when to say nothing · Research · ARCANE