Evolving the Learning Rule: A Search That Missed Forgetting

What is evolved, and how#

The series began from recursive self-improvement: a system that improves its own way of learning, and becomes better at improving itself as a result. On a learner with real structure, such as the fruit fly mushroom body, the circuit is legible, the learning rule can be written out, and the wiring is real. What this article does is narrow: automatic tuning of five parameters of the learning rule. The motion constants, such as turn amplitude and innate attraction, are left fixed, and the starting point is the hand-tuned default.

The outer loop is an OpenAI-style evolution strategy. Noise is added to the center of the five parameters to produce 16 candidates, 8 antithetic pairs; each candidate forages in 12 training worlds and is ranked by mean net reward; the center moves along the direction favored by the ranking, for 25 generations. The training worlds take their reward-inversion periods from 1000, 2000, and 3000, half consistent and half conflicting, and do not overlap the evaluation worlds. Three wirings are evolved — real, shuffled, and random sparse — with 3 independent evolution seeds each, giving 9 evolutions.

The evaluation protocol and the registered criterion#

Evaluation uses the center of the final generation. No generation is cherry-picked and no seed is dropped. The evaluation uses 100 worlds that were never used before, under three conditions:

• in-dist, where the reward-inversion period has the same distribution as in training;
• fast, where the inversion period is 500 and the world is more volatile, out of distribution;
• stationary, where rewards never invert, also out of distribution.

The registered criterion requires the paired difference between evolved and hand-tuned default to have a confidence-interval lower bound above 0 and a gain of at least 6, and to hold in at least 2 of the 3 evolution seeds.

The first evolution: small and tidy gains#

Mean net reward on in-distribution worlds:

All directions are positive and most intervals exclude 0, but the magnitude mostly falls short of +6. By the criterion, only random sparse qualifies on in-distribution worlds (2 of 3 seeds), real gets 1, and shuffled gets 0. The reading at the time was that the hand-tuned defaults were already near optimal. Section four of the source overturns that reading.

The wiring comparison gave the same picture as the earlier foraging article. Real minus shuffled was +3.5, +0.5, and +1.5, which is equivalent or uncertain. Real minus random sparse was -2.7, -8.7, and -8.8, with random sparse clearly better in 2 of 3 runs. The real wiring showed no advantage.

What evolution selected was also informative. KC sparsity fell from 5% to about 2% in all 9 runs. The negative-error learning rate came out above the positive one in 7 of 9, which later turned out to give no benefit when isolated and is probably a by-product of the other parameters. The turning weight of value dropped slightly.

Then came the anomaly. The pre-written guess was that the more volatile the world, the more forgetting should help, because rewards invert. In all 9 evolutions the forgetting rate stayed pinned at the search lower bound, 1e-5. The out-of-distribution results were anomalous too: the gain was smallest in the more volatile fast worlds, from -0.5 to +2, and largest in the non-inverting worlds, from +4 to +9. The first draft explained this by saying that when rewards invert, relearning fast matters more than forgetting, and that a negative error can suppress the old value anyway. That explanation was marked as untested at the time.

The manual sweep, and the corrected search#

The forgetting rate was then swept by hand, under a separate pre-registration: no evolution, just a direct sweep of the forgetting rate together with the learning speed and the positive-versus-negative error asymmetry, over 100 brand-new worlds with seeds 6000 to 6099, under the same three conditions. The predictions were that forgetting would not help, that learning fast would help most in the more volatile worlds, and that the asymmetry would help slightly.

Nearly all of them were wrong.

• Forgetting helps clearly in volatile worlds. In the fast worlds, forgetting rates of 3e-3 and 1e-2 raised net reward from +5.2 to +17.4 and +17.7, a gain of +12 with intervals [+9.9, +14.4] and [+9.8, +15.0]. In worlds with inversion periods from 1000 to 3000, rates of 1e-3 and 3e-3 gained about +10. The two registered main conclusions: 1e-2 against 0 is “better”, while 1e-3 against 0 is +3.5 with interval [+1.6, +5.4], below the bound of 4, so “uncertain”.
• Forgetting hurts badly in non-inverting worlds. 1e-3 costs 12.8, 1e-2 costs 40.7, and 3e-2 costs 54.8.
• Forcing forgetting onto the evolved parameters gives the same answer. Adding 1e-3 forgetting to the 3 evolved results gains another +8 to +10 on in-distribution worlds, more than the +4 to +7 the evolution found on its own.
• “Learning fast matters more” holds only half. Lowering the learning rate to 0.25 times the default was worse in all three worlds; raising it to 2 and 4 times gave no extra benefit, so the default already sits on a learning-rate plateau.
• “Learn faster on negative error” is not supported. Doubling or quadrupling the negative-error learning rate alone made no difference in the fast worlds and was worse or uncertain in the non-inverting worlds.

The conclusion was that forgetting helps in volatile worlds and hurts in stable ones, which is consistent with the original guess; what failed was the evolution, which never had the chance to find it.

How far this is from recursive self-improvement#

The source lists three gaps.

1. Only five numbers change. The form of the rule, the structure of the circuit, and the encoding of odors are all fixed, so the outer loop never has the chance to discover a new learning rule.
2. The outer loop is itself fixed. Population, step size, and number of generations are all set by hand. The outer loop is not improved, so there is one layer and no recursion.
3. The search space of the outer loop determines what it can find. The forgetting episode is the direct example: the same evolution strategy, differing only in where the forgetting parameter starts, moves the gain from +4 to +7 to +14 to +21. A self-improving system first has to make sure its own improver has no blind spot.

The limits the source records are equally specific: only 3 evolution seeds, with intervals bootstrapped over worlds and not over the variation of the evolution itself; an objective that is mean reward on training worlds, so a different set of worlds may prefer different parameters, as the stationary drop shows; hand-tuned defaults that were themselves chosen on tuning worlds rather than being an arbitrary starting point; a re-run starting point of $\theta = 0$ that deliberately carries some forgetting, with no test of whether other starts reach the same place; and only five parameters, with no structural search.

Where the Chinese original lives#

The Chinese original is Let Evolution Tune the Learning Rule: How Far Is It From Recursive Self-Improvement?. It holds the two-loop diagram, the parameter table, the forgetting-sweep and re-run figures, the per-seed gain tables for all three conditions, the attribution of the search failure to the sigmoid encoding, and the pointers to the pre-registrations and result files. Its limit is the one stated above: this is one task, one set of five parameters, and three evolution seeds, and the intervals describe world variation rather than the variation of the search itself.

The next note in this series covers the three traveling-salesman rounds.