Distributed orchestration
AHISTER, DEBOHAC, and the cluster lifecycle: distributing evolutionary search across heterogeneous compute under hierarchical capacity and budget.
Distributing evolutionary search
A symbolic-regression run is not a single optimization loop. It is a population that fans out into independent evolutionary branches, each running for as long as it makes progress. When branches stagnate they are paired, and each pair forks into a tournament branch that intensifies the most promising material and an advanced branch that explores away from it. Those branches can fork further. The structure that emerges is a tree of cooperating computations, not a flat job queue.
Distributing this tree across heterogeneous compute is hard for two coupled reasons. Where a newly forked branch should run depends on what compute is available below the current node, and that changes as siblings finish or fail. How much effort a branch may consume depends on what budget its parent still has, and budget can be reclaimed when a sibling stalls. Conventional distributed schedulers conflate the two into a single resource quota and lose the ability to balance them.
Ufinq's orchestration is split across two complementary algorithms. AHISTER governs within-branch dynamics: the lifecycle of a single branch from isolation through arena to asymmetric split. DEBOHAC governs between-branch dynamics: how a tree of cooperating instances delegates branches downward, shares budget, and recovers from failure.
Four threads of work
AHISTER: within-branch lifecycle
- Adaptive isolation. Every branch develops in isolation until its own stagnation signal fires. Branches do not migrate on a fixed schedule; the schedule is the search dynamics itself.
- Tournament arena. A stagnated branch joins a pool of finished branches, and the two with the closest maturity are paired. Their elites combine into a tournament branch (intensification, narrow elite); everything else combines into an advanced branch (diversification, wide tail).
- Roles, not strategies. Every branch pursues the same objective. What differs is its role: fresh exploration, a successor recombined from a pair, or a restart seeded from the Hall of Fame. An earlier design alternated search strategies between successors, each with its own gene pool; measured against the simpler role model it showed no advantage and was removed.
- Open questions. Opponent selection beyond closest maturity; an adaptive tournament/advanced split under different stagnation signatures.
DEBOHAC: between-branch orchestration
- Two currencies, opposite directions. Capacity flows upward as periodic situation reports that aggregate hierarchically; each tier presents its entire subtree as a single abstract unit to the level above. Budget flows downward as the authority to evolve, and is topped up when most of a share is used.
- Subsidiarity at every level. Every instance runs the same three-step cycle: prefer delegating downward to a subordinate with free capacity; otherwise run locally, if this instance is allowed to host work; otherwise escalate upward. A root can be set to coordinate only, in which case work waits for a follower instead of landing on the root.
- Bounded propagation. Each branch carries a signed depth and a hop count, both bounded, so a branch cannot circulate indefinitely.
- Failure recovery. When a peer drops, the branches it was running are re-delegated to a surviving follower or adopted locally; only when neither is possible does a branch end with a failed outcome. Branches that arrived from the lost peer are cancelled. No consensus protocol runs.
- Fresh capacity. Capacity reports expire, and a peer that declines work is left alone for a cooling period, so routing follows what is true now.
- Open questions. Hop policy when topology changes mid-run; convergence proofs for the recursive three-step cycle under arbitrary failure patterns.
Cluster transport & data plane
- Virtual-thread transport. The cluster transport runs on virtual threads over TLS sockets. One virtual thread per peer connection, blocking I/O, push-based messaging.
- Peer-local datastore. Datasets are referenced by ID. A peer fetches the chunks it lacks from the peer that handed it the work, caches them locally with least-recently-used eviction, and deduplicates requests in flight, so a dataset is pulled at most once per peer.
- Branch envelopes. A delegated branch carries its datasets by reference, not inline. The receiving peer pulls only what it has not seen, which removes the duplicate-payload problem under fan-out delegation.
- Open questions. Adaptive chunk size under varying network quality; backpressure when one peer is saturated by fetch demand.
Shared budget & capacity
- A partitioned budget. The budget of a run is a tree of allocations. A delegated branch receives a share carved from its parent's; what it spends is reported back with its state.
- Accounting at the work boundary. Cost is charged when a unit of work completes, and cancellation is observed at the same boundary. A branch can overshoot its bound by at most one unit of work.
- Hierarchical capacity rollup. Capacity reported by a peer already represents the entire subtree below it. The recursive aggregation is what makes a single three-step cycle correct at every depth without special-casing the leaves.
- Open questions. Non-uniform cost models when peers expose accelerators; budget transfer between sibling subtrees that finish out of phase.
Three consequences of this design
Related material
Two papers are in preparation: AHISTER describes the within-branch lifecycle in detail; DEBOHAC describes the cluster scheme. Both are listed under Papers.
The system view of the cluster (coordinating branch, periphery branches, cluster ring) is described under Technology § 2 with Figure 2. The single-branch lifecycle (isolation → arena → asymmetric split) is described under Technology § 1 with Figure 1.
Continue reading
Other research areas are listed under Research. Published work lives on Papers; short-form notes on Notes.