ENGLISH OVERVIEW · EXPERIMENT DESIGN
Training Comparisons That Answer a Question: Hypotheses, Baselines, and Repeats
中文原文 ↗Structured thesis with preliminary math or code; actively developing.
Digital Garden
🌿 What is a Digital Garden?
Rooted in Andy Matuschak & Maggie Appleton's evergreen note taxonomy: articles are not ephemeral chronologies, but living organisms tended over time:
- 🌱 Seedling: Raw inspiration, reading logs, open questions
- 🌿 Budding: Structured thesis, early proofs, actively revised
- 🌳 Evergreen: Closed logic, enduring, foundational reference
- 🔬 Speculative: Empirical hypotheses coupled with lab instruments
Scope note. All scores and thresholds are constructed. Only arithmetic checks were run; no training comparison was executed.
Make the hypothesis falsifiable
Ask whether verified recovery examples improve recovery without unacceptable first-attempt regressions. Define the intervention as replacing some clean supervision under a fixed target-token budget. Specify the primary metric, guardrails, and decision thresholds before viewing test outcomes.
Equal target tokens do not imply equal input tokens, attention work, or elapsed time. Separate fixed-budget substitution from adding data and compute. Keep both an unchanged starting model and a clean-data training baseline.
Check interactions rather than attributing everything to one change
A constructed two-factor table has scores 60 and 65 without recovery training, versus 62 and 74 with it, across no-feedback and feedback inference conditions. Training effects are 2 and 9 percentage points; their difference is 7. The full 14-point change cannot be attributed solely to training.
NIST’s randomized-block guidance explains controlling nuisance variation within blocks. Paired training repeats can adopt that principle, while recognizing that an identical seed number does not guarantee identical random events across implementations.
Preserve repeated runs and their units
Constructed paired scores A=(60,70,65), B=(64,72,66) yield differences (4,2,1), averaging about 2.33 percentage points. Comparing the best B with the worst A produces 12 points and misrepresents the average comparison. Three repeats are an illustration, not a universal sample-size recommendation.
Repeated decoding of one checkpoint does not measure training-seed variation. Preserve task-family and run grouping when estimating uncertainty. Record out-of-memory failures, divergence, and invalid infrastructure measurements separately under predefined rules; do not silently report only successful runs.
Keep the evaluation population comparable
A constructed system A succeeds on 81/90 easy and 2/10 hard tasks, totaling 83%. B succeeds on 88/99 easy and 0/1 hard tasks, totaling 88%. B is worse within each category but benefits from a different mixture. Use shared instances or fixed declared weights rather than attributing this aggregate change to training.
An experiment card should specify the question, intervention, baseline, budgets, metrics, versions, repeats, failures, stopping, and checkpoint selection. The Chinese article explains how outcomes should guide the next decision without claiming unrun gains.
Article concept network
Explore sections and explicit terms from the text. Dashed lines connect repeated concepts, not causal claims.
14 / 14 nodes · marked terms and named concepts from the text
Training Comparisons That Answer a Question: Hypotheses, Baselines, and Repeats
Read in context ↗