Constant optimization
Coordinate Descent, CMA-ES, L-BFGS, Nelder-Mead: numeric methods specialized for the constraints of symbolic regression candidates.
Where the numeric tuning lives
A symbolic regression candidate is two things at once: a structure (which functions, which arguments, what shape) and a set of numeric constants (multipliers, offsets, exponents, decision thresholds). The structure is moved by mutation and crossover. The constants are moved by an optimizer. Without that optimizer, every structurally promising candidate would carry whatever random constants it was born with, and selection would have to hope they were close enough.
In Ufinq the optimizer is part of the refinement pipeline that runs on every candidate, every generation. Its job is to leave each candidate's constants tuned to the data before the candidate enters the gene pool. The research question is not whether to optimize but which optimizer: symbolic regression candidates have unusual loss landscapes, and the methods that work for parameter-vector ML do not always survive contact with them.
Four threads of work
Four numeric optimizers, four convergence shapes
- Coordinate descent. One-dimensional line searches in turn; extremely cheap, and well suited to candidates whose constants decouple in the loss. Weak on coupled landscapes.
- CMA-ES. Derivative-free, evolutionary; robust on rugged landscapes and bad initializations. Pays for that robustness in iteration count, so the cost per candidate is high.
- L-BFGS. Quasi-Newton, driven by exact symbolic gradients; fast near a local minimum. Fragile when the candidate has discontinuous regions or when the gradient is poorly scaled, both common in symbolic candidates.
- Nelder-Mead. Simplex-based, derivative-free; cheap per iteration, slow on ill-conditioned problems, and scales poorly in dimension. Useful as a final polish.
- Open questions. Automatic selection per candidate based on structural features; cost-aware termination under a fixed per-generation budget.
Chained optimization
- The chain. A candidate is first pruned, then passed through Coordinate Descent, CMA-ES, L-BFGS and Nelder-Mead in turn. Each stage starts from the previous one's result and is kept only if it improves the candidate.
- Linear scaling first. Before the chain runs, the output's slope and intercept are fitted directly. The optimizers then work on the constants that shape the function, not on the two that merely place it.
- Per-candidate skip. An optimizer declines to run when the structure makes it pointless: no constants, or constants the loss does not depend on.
- Stage cost against gain. Every stage reports how often it ran and how often it improved the candidate. The chain definition (which optimizers, in which order, with which termination criteria) is a research surface in its own right, not a tuning constant.
- Open questions. Reusing a tuned result when the same structure reappears in a later generation; running the full chain only on candidates that are already competitive.
Levenberg-Marquardt and Gauss-Newton, rejected
- What's tempting. LM and Gauss-Newton exploit the least-squares structure of the loss explicitly. On well-conditioned candidates they are the obvious choice: fast convergence, established theory, mature implementations.
- What's wrong. Gauss-Newton degrades on the badly structured candidates that dominate symbolic-regression populations. The Jacobian is ill-conditioned or singular, the residual landscape has non-smooth ridges, and the method either stalls or chases noise. LM's damping helps but does not fix the underlying issue.
- Where it stands. LM is rejected on design grounds, not on a measurement. The rejection stands until a selective variant, one that runs only on candidates whose structure passes a curvature test, shows a measurable benchmark gain over the current chain.
- Open questions. The right curvature test (cheap, structure-only, predictive of LM stability); conditional LM as a final stage of the existing chain.
Pruning, adjacent
- Pruning. A structural pass that tries collapsing a node into one of its children, removing a child, or replacing a subtree with a neutral constant. A change is kept only if the result is strictly better in size or error and worse in neither.
- Why it lives here. Pruning runs first in the chain, so the numeric optimizers work on the smaller structure. It changes structure, not constants, and is a separate research thread only because the method differs; the surrounding machinery and the cost story are shared.
- Open questions. Whether a second pruning pass after the numeric stages pays for itself; integration with the e-graph rewriter, which finds equivalences but does not pick a representative under a numerical objective.
Three consequences of this design
Related material
The pipeline view is described under Technology § 6; constant optimization is the fourth stage in the six-stage diagram (Figure 3). The note Every Ufinq individual is already a local optimum describes the design contract that the optimizer is part of.
Adjacent areas: Symbolic refinement describes the simplification that runs before constant optimization and the normalization that runs after it; Approximate evaluation describes how the many evaluations an optimizer needs are kept affordable.
Continue reading
Other research areas are listed under Research. Published work lives on Papers; short-form notes on Notes.