§ 2.1 · Research area

Distributed orchestration

AHISTER, DEBOHAC, and the cluster lifecycle — distributing evolutionary search across heterogeneous compute under hierarchical capacity and budget.

§ 1 Why orchestration

Distributing evolutionary search

A symbolic-regression run is not a single optimization loop. It is a population that fans out into independent evolutionary branches, each running for as long as it makes progress, then forking — into a tournament that intensifies the most promising material and an advanced branch that explores away from it. Branches themselves can fork further. The structure that emerges is a tree of cooperating computations, not a flat job queue.

Distributing this tree across heterogeneous compute is hard for two coupled reasons. Where a newly forked branch should run depends on what compute is available below the current node — but compute availability changes as siblings finish or fail. How much effort a branch is allowed to consume depends on what budget the parent still has — but budget can be reclaimed when a sibling stalls. Conventional distributed schedulers conflate these into a single resource quota and lose the ability to balance them.

Ufinq's orchestration is split across two complementary algorithms. AHISTER governs within-branch dynamics: the lifecycle of a single branch from isolation through arena to asymmetric split. DEBOHAC governs between-branch dynamics: how a tree of cooperating instances delegates branches downward, reclaims budget, and stays consistent under failure.

§ 2 Sub-areas

Four threads of work

§ 2.1

AHISTER — within-branch lifecycle

Adaptive isolation — every branch develops in isolation until its own stagnation signal fires. Branches do not migrate on a fixed schedule; the schedule is the search dynamics itself.
Tournament arena — when a branch enters the arena, it is matched against an equally mature opponent from a real-time Hall of Fame. Their genetic material is asymmetrically partitioned into a tournament branch (intensification, narrow elite) and an advanced branch (diversification, wide tail).
Strategy alternation — succession across branches alternates strictly between strategies, with per-strategy gene pools so that diversity is measured per-strategy, not population-wide. This prevents a winning strategy from collapsing the entire population's diversity.
Open questions — opponent-selection beyond same-maturity matching; adaptive tournament/advanced split ratio under different stagnation signatures; cross-strategy crossover policy.

§ 2.2

DEBOHAC — between-branch orchestration

Two currencies, opposite directions — capacity flows upward as periodic situation reports that aggregate hierarchically; each tier presents its entire subtree as a single abstract unit to the level above. Budget flows downward as the authority to evolve, refilled when local consumption crosses a threshold.
Subsidiarity at every level — every instance runs a three-step cycle: prefer downward delegation to a subordinate with available capacity; fall back to local execution; escalate upward only when neither is feasible. The same logic applies recursively at every depth.
Bounded propagation — dual hop counters (delegationDepth, escalationDepth) prevent pathological cycles. A branch that exceeds either is anchored at the current instance instead of being passed further.
Failure detection — when a peer connection drops, branches forwarded to that peer get a synthesized terminal outcome; branches that arrived from that peer are cancelled and their routing entries dropped. No back-channel coordination required.
Open questions — capacity-report TTL under bursty churn; hop-counter policy when topology changes mid-run; convergence proofs for the recursive three-step cycle under arbitrary failure patterns.

§ 2.3

Cluster transport & data plane

VT-backed transport — the cluster transport is built on virtual threads over TLS sockets, replacing the older polling receive() loop. One VT per peer connection, blocking I/O, no NIO selector overhead at the orchestration layer.
Content-addressable datastore — every dataset is identified by a digest. Peers fetch chunks lazily from the originating instance and cache them locally; in-flight requests are deduplicated so identical datasets are pulled at most once per peer per run.
Branch envelopes — a delegated branch references its inputs (Context, SearchSpace, datasets) by ID rather than carrying them inline. The receiving peer pulls only what it has not seen before. This eliminates the duplicate-payload problem under fan-out delegation.
Open questions — adaptive chunk size under network-quality variation; eviction policy for the local cache; backpressure when an originating peer is saturated by fetch demand.

§ 2.4

Shared budget & capacity

Single budget across coordinator and sub-branches — a parent and its delegated children share a single StandardBudget instance. Spend by either side debits the same balance; there is no reconciliation gap between local and remote cost accounting.
Closure-based accounting — Budget.account takes a closure that runs the work and atomically debits the cost. Cancellation is observed at the same boundary, so a cancelled branch never spends past its intended bound.
Hierarchical capacity rollup — capacity reported by a peer already represents the entire subtree below it. The recursive aggregation is the property that makes a single three-step cycle correct at every depth without special-casing the leaves.
Open questions — capacity-aware retry on partial failure; non-uniform cost models when peers expose accelerators; budget transfer between sibling subtrees that finish out of phase.

§ 3 Why it matters

Three consequences of this design

Heterogeneity is freeA peer with twice the capacity reports twice the capacity; the three-step cycle naturally routes more work toward it. No tuning, no manual placement, no node-affinity rules.

No central schedulerEvery instance makes its own delegation decisions from the local view of capacity reports. There is nothing to make a single point of failure of, and no coordination round-trip on the hot path.

Resilience is structuralFailure detection is built into the routing maps, not bolted on. A peer dropping out triggers terminal-outcome synthesis and branch cancellation locally; no consensus protocol runs.

§ 4 Notes & references

Related material

Two papers are in preparation: AHISTER describes the within-branch lifecycle in detail; DEBOHAC describes the cluster scheme. Both are listed under Papers.

The system view of the cluster — coordinating branch, periphery branches, cluster ring — is described under Technology § 2 with Figure 2. The single-branch lifecycle (isolation → arena → asymmetric split) is described under Technology § 1 with Figure 1.

Read

Continue reading

Other research areas are listed under Research. Published work lives on Papers; short-form notes on Notes.

Back to Research areas