How the real-world benchmark is made
The real-world benchmark observes. It fits nothing, re-runs nothing and selects nothing: it reads the public analyses on Diafunc as their owners ran them and reports what it found. The standardized benchmark is the controlled comparison; this page is about the observation.
How the data is collected
- The public directory. Diafunc lists the analyses whose owners made their project public. A scanner reads that list and each listed analysis through the same public interface a visitor uses, without an account and without a key, once an hour.
- Only what changed. An analysis is read again only when the directory says it changed. Every observation is kept with a hash of its content; the newest one is what the pages show.
- Nothing is re-run. The model of an analysis is the one its owner got, fitted once with the time and data the owner chose. The benchmark never starts a fit.
- No raw data is served. The pages carry the name of the analysis and the address of its public project, so a visitor can open any single case, and nothing else of the owner's text: no description, no questions, no column names, no formula, no values.
What the numbers mean
- Quality runs from 0 to 1 and is Ufinq's own measure of how well the model follows the data it was fitted on, as recorded with the model.
- Held-out quality is the same measure on rows the model did not see. It exists for models that carry a fit record (Ufinq 0.3.34 and later).
- The linear reference is the quality of a plain linear model on the same data, recorded in the fit record. Over a straight line is the model's quality minus that; zero or less means the formula adds nothing a line does not give.
- Saturation is the time of the model's last improvement over the time the search used. Near 1 the search was still improving when it stopped; near 0 it had stopped improving long before.
- Selection rank is where the delivered model stood, by quality, among the final candidates; the selection gaps say how far it was from the best by quality and by error. They show whether the final choice, not the search, cost quality.
- Size and shape. Rows are classed under 100, 100 to 1k, 1k to 10k, 10k to 100k, 100k and more. Shape is rows per column: wide under 10, medium 10 to 100, tall 100 and more.
- Distinct datasets. The data of an analysis has a fingerprint of its content. Two analyses of the same data count once among the distinct datasets, and the versions page pairs them when they were fitted under different Ufinq versions.
Tagging
Domain, purpose, data origin, kind and time-dependence are assigned by a language model from the public description of each analysis: the name, description, questions, column names with type and unit, and the formulas, each cut to a fixed length, never a value of the data and never the owner. The model is gpt-6-luna, asked with taxonomy version 1 and prompt version 1; it must answer in a fixed vocabulary, and an answer outside it is discarded. The owner's text is treated as data, not as instruction. Tags sort the base; they never change a number.
Agreement with a labelled sample
No labelled sample has been compared yet. Until one exists every field is offered as a filter; once it does, a field whose agreement is under the threshold is withheld.
| Field | Agreement | Labelled | Offered as a filter |
|---|---|---|---|
| Domain | not measured | — | yes |
| Purpose | not measured | — | yes |
| Data origin | not measured | — | yes |
| Kind | not measured | — | yes |
| Time-dependent | not measured | — | yes |
Withdrawal
When an owner makes a project private or deletes it, the analysis leaves the directory and the next hourly scan removes it from the indexes the pages read; its stored observations and its tags are deleted with it. The number of withdrawn analyses is shown on the overview. A scan that would remove a large part of the base at once stops instead, so a fault in the directory cannot empty the pages.
The example-projects caveat
Most public projects on Diafunc today are examples: textbook or benchmark datasets published to show what an analysis looks like, not questions someone needed answered. They are in the base without a mark, because the model was fitted the same way either way, and the kind tag tells them apart. Until more genuine questions are public, the overview is a picture of what Ufinq does with example data, and the limitations page says so.
What this benchmark cannot say
A difference between two groups of the base is a difference between two sets of data as much as a difference in Ufinq. Only the derived suites, where other methods run on the same data with the same time, turn an observation into a comparison.