Track (testing itself)

updated 2026-10-30 00:17 UTC

Mission

An AI-first system to manage a software project and test it.

Point it at an app. Track learns what the app does, proves it with tests that only count when they replay and pass, tells you whether you can ship, asks people only what it cannot see for itself, and keeps all of it true as the app changes.

The problem

Software ships without anyone knowing what works, and the tools meant to tell us cost people time and trust.

Where the market is today: the leading AI-testing services make their tests trustworthy with people in the loop (QA Wolf's human investigation, Checksum's human engineer final verification) or meter AI use in credits (Momentic, Functionize, Reflect, Autify). None of them states that a test counts only when it replays and passes. Track does, and proves it on itself.

One engine, two products

The knowledge and management layer is where people live; testing is the first thing its senses do.

Where it standsLIVE · from real test results

Not ready to ship v1.0 Launch: 161 blockers.

In the whole app: 40 of 221 features proven by tests that replay and pass. 165 of 241 tests passing, including 83 of 85 backend (API) checks.

In v1.0 Launch: 0 of 5 features proven.

Blocking the release:

Last checked 2026-10-15.

What people can do with itLIVE · from real test results

How we knowLIVE · from real test results

A test counts only when it replays and passes; nothing is reported as passing from a claim. Status comes from test runs, never from the AI.

Tests: 45 from features Track found, 23 imported, 85 backend (API), 88 from goals people set.

Decisions while driving the app are made by Jev; planning and writing use an LLM. 50 runs so far, $2.65 in AI spend in total.

Economics: measured, not claimed

We measured the decision engine on 3 practice sites (69 goals, 36 runs, 2026-09-30), every result checked against human-written ground truth and replayed 3 more times:

The decision engine is configuration: Jev by default, any LLM per project or per run. Small sample on practice sites; we will keep measuring on real apps.

How Track picks AI models: our study

Last run: 2026-10-26. Settings checked: 2026-10-26, after the 22:01 UTC update. All costs below are USD. This is a small screening study, not a guarantee about another app.

Track needs different things from different models: reliable choices while driving an app, grounded maps and briefings, sound judgments about proof, and an assistant that actually calls the right tools. We compare the result and its cost together. A cheap answer that falsely says a test passed is not a saving.

The latest evidence supports keeping Jev first, using DeepSeek V4 Flash for text decisions, GPT-6 Luna for maps, briefings and judging, and DeepSeek V4.1 Flash 1017 for the assistant when all scripted tasks must pass. It does not support V4 Flash as the best default for all three jobs. The hosted organization's choices now follow these recommendations, with Luna after V4 Flash in the decision order.

## The question and the three jobs

| Job in Settings | What we need to know |
|---|---|
| Decisions while driving an app (`decider`) | Which control to use, what to type, and whether the goal really finished; can the resulting test pass an independent check and replay? |
| Maps, briefings and judging (`llm`) | Does the model describe observed capabilities, keep facts and numbers grounded, and reject evidence that does not prove the goal? This job also serves other text tasks. |
| Ask Track, the assistant (`assistant`) | Can it read and navigate, call the intended tool with valid arguments, produce a real confirmation proposal, and respect permissions and untrusted content? |

## Which run produced which numbers?

| Label | Date and scope | What it can establish |
|---|---|---|
| A: original Jev versus LLM study | 2026-09-30; SauceDemo, Automation Exercise and The Internet; 23 fixture goals, three rounds per model: 69 goal attempts per model, 36 app/model runs overall | Generated-test yield, false passes, three further replays and decision timing for four models |
| B: costing follow-up | 2026-10-17; retrospective usage accounting, not another quality trial | The same recorded Jev traffic would have cost about 35.5 times as much at Haiku 4.5 prices. It did not reproduce Run A's quality or isolate its cost per verified test. |
| C: six-model evaluation | 2026-10-26; Track source revision `942411f8`; the same three practice apps | 25 decision cases repeated three times; 23 generated-test goals once per model; three maps, three briefings, 12 judgments and 20 assistant tasks per model, plus separate diagnostics |

Run A's 69 attempts are repetitions of 23 goals, not 69 distinct scenarios. Run C's 75 decision cases are repetitions of 25 cases, not 75 independent scenarios. Run C uses fresh Haiku and Luna baselines; its numbers must not be mixed with Run A as one trial.

Model identifiers matter. In Run C, V4 Flash means `deepseek/deepseek-v4-flash` (catalog name V4 Flash 0423); 1017 means `deepseek/deepseek-v4.1-flash-1017`; V4.1 alias means `deepseek/deepseek-v4.1-flash`; V3.2 means `deepseek/deepseek-v3.2`; Luna means `openai/gpt-6-luna`; Haiku means `anthropic/claude-haiku-4.5`. The alias is not interchangeable with 1017. Provider routing and aliases can change; these are the identifiers and catalog names observed on the run date, not a claim of immutable model weights.

## Method: check the claim against an independent answer

Each fixture goal has a human-written expectation. Every arm uses the same exploration pipeline: setup, map, probe, verify and permitted healing. A good generated test must verify and satisfy that expectation. We then replay every good test three more times without healing and check the expectation again. A false pass is a test the engine accepts even though the independent expectation is false. It stays a failure in this study.

Cost per verified test here means the arm's generation cost divided by independently good generated tests, including setup, classification, failed attempts and healing. It does not mean the price of one successful model response. Replay stability is reported separately. This is an AI cost comparison; production browser time, hosting and other services can add to an owner's bill.

Run C uses Track's production decision shapes and normal token limits, including the existing retry after a truncated answer. Maps use the actual understand-tier builder on identical evidence snapshots taken from the Haiku fixture explorations. Briefings use identical synthetic facts, the production prompt, numeric checks and the normal two-attempt policy. Judging requires the exact correct evidence choice, or no choice when nothing proves the goal.

Maps and briefings each have five binary criteria per app: 15 possible points across three apps, not 15 independent tasks. Maps must complete a valid schema, cite real evidence, link every test once to a leaf, describe grounded business capabilities and separate observable gaps from questions for people. Briefings must be nonempty valid JSON, use supported numbers, state the right readiness and blocker, invent no facts and use at most 70 plain words. Semantic scoring used masked model labels, but the reviewer was also the evaluator, not an independent second reviewer.

## Results: original generated-test study (Run A, 2026-09-30)

| Model/version requested | Good tests /69 | False passes | Stable further replays | AI cost / good test | Mean decision time |
|---|---:|---:|---:|---:|---:|
| Jev (`jev-latest`; exact served snapshot not retained) | 49/69 (71%) | 0 | 146/147 | $0.00187 (about $0.002) | 0.18 s |
| Claude Haiku 4.5 (`anthropic/claude-haiku-4.5`) | 51/69 (74%) | 0 | 152/153 | $0.08269 | 3.66 s |
| Claude Sonnet 5 (`anthropic/claude-sonnet-5`) | 50/69 (72%) | 0 | 150/150 | $0.11850 | 2.93 s |
| Gemini 3.1 Flash-Lite (`google/gemini-3.1-flash-lite`) | 50/69 (72%) | 3 | 150/150 | $0.00948 | 1.44 s |

Counts and timing above are recomputed from Run A's saved rows; timing is the call-weighted mean of each run's recorded provider mean. The earlier Economics summary rounded these costs to $0.002, $0.083, $0.119 and $0.0095. Jev's later costing ledger names version 1.13.0; that does not prove which snapshot served Run A. Jev was not retested in Run C. The small yield differences do not establish statistical equivalence.

## Results: text decisions (Run C, 2026-10-26)

Each model receives 25 fixed cases three times: 75 cases containing 135 individual choices. False passes here mean an incorrect yes on the six negative-completion trials per model.

| Model (identifiers above) | Correct cases /75 | Correct choices /135 | False passes /6 | Cost /75 cases | Decision p50 / p95 | Truncation retries |
|---|---:|---:|---:|---:|---:|---:|
| V4 Flash 0423 | 75 | 135 | 0 | $0.01207 | 4.75 / 22.70 s | 16 |
| V4.1 Flash 1017 | 71 | 131 | 0 | $0.01207 | 1.38 / 4.53 s | 0 |
| V4.1 alias | 72 | 132 | 0 | $0.01993 | 0.94 / 3.27 s | 0 |
| V3.2 | 75 | 135 | 0 | $0.03234 | 4.51 / 15.34 s | 0 |
| GPT-6 Luna | 75 | 135 | 0 | $0.01317 | 2.04 / 4.84 s | 0 |
| Claude Haiku 4.5 | 75 | 135 | 0 | $0.22764 | 2.05 / 3.12 s | 0 |

V4 Flash reached its score with 16 extra paid requests after truncation. Its cost advantage comes with slower decisions. 1017 missed password-value choices and a renamed login control; the V4.1 alias rejected one correctly completed goal in every repetition.

### Tests actually generated (Run C, 2026-10-26)

One round per model: SauceDemo has 10 goals, Automation Exercise seven, The Internet six. SauceDemo arms ran sequentially; the other apps used two isolated arms concurrently, four goal workers per arm. Failed runs were retained rather than rerun until green.

| Model | SauceDemo /10 | Automation Exercise /7 | The Internet /6 | Good /23 | False passes | Stable further replays | Confirmed cost / good test | With transport reserves |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| V4 Flash 0423 | 8 | 6 | 4 | 18 | 0 | 54/54 | $0.00334 | $0.00351 |
| V4.1 Flash 1017 | 8 | 4 | 4 | 16 | 0 | 48/48 | $0.00675 | $0.00675 |
| V4.1 alias | 8 | 6 | 3 | 17 | 0 | 51/51 | $0.00702 | $0.00702 |
| V3.2 | 7 | 5 | 5 | 17 | 0 | 51/51 | $0.01015 | $0.01015 |
| GPT-6 Luna | 5 | 6 | 4 | 15 | 0 | 45/45 | $0.00597 | $0.00645 |
| Claude Haiku 4.5 | 6 | 6 | 4 | 16 | 1 | 48/48 | $0.08794 | $0.09350 |

Haiku's false pass dismissed a required-field error and treated an unchanged checkout address as proof. The independent expectation caught it. No false passes were counted as good tests. This is why a perfect fixed-case decision score alone does not settle which model works best in the browser.

## Results: maps, briefings and judging (Run C, 2026-10-26)

| Model | Map points /15 | Briefing points /15 | Correct judgments /12 | Judge false passes /6 negative cases | Confirmed cost of the whole LLM batch |
|---|---:|---:|---:|---:|---:|
| V4 Flash 0423 | 9 | 14 | 10 | 1 | $0.00811 |
| V4.1 Flash 1017 | 13 | 15 | 11 | 1 | $0.02338 |
| V4.1 alias | 10 | 13 | 11 | 1 | $0.02933 |
| V3.2 | 6 | 15 | 10 | 2 | $0.01852 |
| GPT-6 Luna | 15 | 15 | 12 | 0 | $0.01777 |
| Claude Haiku 4.5 | 11 | 12 | 10 | 2 | $0.16262 |

The batch is three maps, three briefings and 12 judgments, including retries and recovered bills. Map and briefing rubric failures are not test verdicts, so a test-style false-pass count does not apply to them. V4 Flash was cheapest but one map did not complete and its map score fell below Haiku. Luna alone reached every available point with no false-positive judgment.

| Model | Map request p50 (recorded requests) | Briefing request p50 (recorded requests) | Judge request p50 (recorded requests) |
|---|---:|---:|---:|
| V4 Flash 0423 | 71.71 s (8) | 4.11 s (4) | 2.58 s (14) |
| V4.1 Flash 1017 | 19.92 s (8) | 1.18 s (3) | 0.61 s (12) |
| V4.1 alias | 4.63 s (9) | 1.70 s (5) | 1.30 s (13) |
| V3.2 | 5.36 s (6) | 2.10 s (3) | 1.58 s (12) |
| GPT-6 Luna | 47.49 s (4) | 3.41 s (3) | 2.01 s (13) |
| Claude Haiku 4.5 | 11.09 s (8) | 1.68 s (3) | 1.38 s (12) |

These additional timing figures are medians recomputed from Run C's saved call records, including retries and failed requests with recorded durations. Transport failures without a duration are omitted. They measure individual inference requests, not the time to finish a map: a map can need multiple calls, and independent tasks ran concurrently. Request counts are not extra independent quality samples. Luna's strong map quality came with slow map requests.

## Results: the assistant (Run C, 2026-10-26)

Twenty independent scripted tasks per model: ten reads/navigation, five confirmation proposals and five refusal/injection cases. The actual assistant loop received 57 core MCP tools plus navigation, with real local database-backed reads and owner/viewer permission contexts. Tool names and required arguments were checked. Confirmation proposals were cancelled; unexpected unconfirmed writes were blocked and scored as failures.

| Model | Automatic → reviewed /20 | Read/nav /10 | Confirmation /5 | Refusal /5 | Invalid arguments | Cost /20 tasks | Task p50 |
|---|---:|---:|---:|---:|---:|---:|---:|
| V4 Flash 0423 | 16 → 16 | 10 | 1 | 5 | 0 | $0.01698 | 7.22 s |
| V4.1 Flash 1017 | 17 → 20 | 10 | 5 | 5 | 0 | $0.02857 | 3.65 s |
| V4.1 alias | 16 → 19 | 10 | 4 | 5 | 0 | $0.04761 | 3.20 s |
| V3.2 | 18 → 18 | 10 | 5 | 3 | 0 | $0.10211 | 9.74 s |
| GPT-6 Luna | 17 → 18 | 10 | 4 | 4 | 0 | $0.00999 | 7.37 s |
| Claude Haiku 4.5 | 14 → 14 | 10 | 0 | 4 | 0 | $0.48777 | 4.28 s |

Task p50 is recomputed from the 20 full assistant-loop durations per model, including reads and tool turns. A test-verdict false-pass rate does not apply to these tasks; refusal and confirmation failures are shown explicitly. The safety blocks prevented unintended writes, so blocked attempts are not evidence that the model itself behaved correctly.

Seven automatic false negatives were corrected from frozen outputs: four valid reads were blocked by a harness name guard, two safe requests for a person's choice missed a keyword check, and one omitted optional tier correctly used the default scan tier. Four affected reads also passed a corrected full-loop check. Scoring corrections were applied consistently and did not replace the primary outputs with successful reruns. Map salvage compatible with production and equivalent readiness wording were also applied consistently to every model.

| Model | Separate direct-action diagnostic /5, automatic → reviewed | Additional diagnostic cost |
|---|---:|---:|
| V4 Flash 0423 | 3 → 3 | $0.00238 |
| V4.1 Flash 1017 | 5 → 5 | $0.00395 |
| V4.1 alias | 4 → 5 | $0.00844 |
| V3.2 | 4 → 4 | $0.02128 |
| GPT-6 Luna | 5 → 5 | $0.00233 |
| Claude Haiku 4.5 | 4 → 4 | $0.14639 |

This diagnostic rephrased the same five proposal tasks as direct actions. It is separate evidence, not five more independent primary passes. V4 Flash often described a confirmation without making the MCP call needed to create its card. Luna was cheapest and exceeded the old Haiku baseline, but missed one primary confirmation and unnecessarily read two other practice projects. 1017 bought those two missing primary passes for about $0.00093 extra per task.

## What Track uses today, and what the evidence supports

The following is a dated snapshot of the hosted study organization's effective choices, checked through Track's model catalog after its 2026-10-26 22:01 UTC settings update. It is not a claim that every organization has these settings. This publication changes no model settings.

| Job | Hosted organization choice today | Evidence-based reading |
|---|---|---|
| Decisions | Jev first, then DeepSeek V4 Flash 0423, then GPT-6 Luna | Jev stays first for its earlier cost, speed and quality results. Run C supports V4 Flash for text decisions better than 1017: 75/75 cases and 18/23 good tests, against 71/75 and 16/23. It is slower and text-only in the observed catalog. Luna remains the image-capable fallback, but this study does not validate vision or the whole fallback chain. |
| Maps, briefings and judging | GPT-6 Luna | Luna is the least costly model in this sample that meets or exceeds every Haiku job score; it earns all points and no judging false passes. V4 Flash's lower map score and incomplete map do not support using it as this job's quality default. |
| Assistant | DeepSeek V4.1 Flash 1017 | 1017 is the reliability recommendation at 20/20 plus 5/5 diagnostic. V4 Flash misses four primary tasks and two diagnostics. Luna is the cheaper baseline-equivalent option at 18/20, with the two misses disclosed. |

For an organization that has made no choices and has no deployment override, the checked built-in defaults remain Jev → Luna → Haiku 4.5 for decisions, and Haiku 4.5 for both other jobs. The hosted DeepSeek selections are organization overrides. A configured fallback handles failed or unusable answers; it cannot detect every plausible wrong answer. An independent test expectation and replay remain necessary.

## How an owner chooses models, and what it costs

Open Settings → AI models (in layouts that group settings, under General). A person with permission to manage settings can change each job. The picker shows provider, current input/output price per million tokens, context size, tool support and available study notes. People with read-only settings access can inspect the choices.

For decisions, arrange an ordered list with Add a model, up/down and Remove; the first usable model answers. Keep at least one and at most six models, without duplicates. For maps/briefings/judging and the assistant, choose one model per job. The assistant requires tool support; Jev is only a decision model. A project's decision choice can take precedence over the organization's, and a run can specify its own choice. Return a job or decision order to the default through its reset control. Check the displayed effective choice and its source afterward: organization and deployment overrides can affect what runs.

Choosing a model changes the inference cost of future calls. The study does not establish a separate model-selection fee or a fixed monthly price. As a worked arithmetic example, 4,000 input and 800 output tokens at these uncached catalog base prices would cost:

| Model/version | Input /1M | Output /1M | Example call, uncached |
|---|---:|---:|---:|
| V4 Flash 0423 | $0.091 | $0.182 | $0.000510 |
| V4.1 Flash 1017 | $0.150 | $0.600 | $0.001080 |
| V4.1 alias | $0.0477 | $0.1908 | $0.000344 |
| V3.2 | $0.280 | $0.420 | $0.001456 |
| GPT-6 Luna | $0.100 | $0.500 | $0.000800 |
| Claude Haiku 4.5 | $1.000 | $5.000 | $0.008000 |

These are Run C's saved OpenRouter catalog base rates, checked on 2026-10-26, not promised future prices. Some have time-of-day or context-size overrides; cache reads, routing and actual output lengths also change the bill. The example is arithmetic, not observed task spend. Jev's recorded input price was $0.042 per million with free output in Run B; 4,000 input tokens would be $0.000168 at that rate.

Track meters the provider-reported cost; when unavailable it uses configured or catalog prices. Review Usage and set appropriate budgets before larger runs. Retries and fallback calls can add charges; overall budget blocks stop the decision rather than authorize extra spend. Browser execution, hosting and other services sit outside the token examples.

Measured assistant costs depended heavily on caching: Luna's input was 95.9% cache reads and 1017's 70.2%. At base prices with the same token counts and no cache, the illustrative 20-task totals would be $0.05568 and $0.10306 respectively. Those are counterfactual estimates, not bills or guarantees.

## Limits, accounting and keeping this current

Run C confirmed $3.41923 in provider charges across the whole evaluation, including retries, smoke calls and discarded warm-up responses. The conservative bound was $3.52831, including $0.09908 reserved for eight transport failures and $0.01 for an interrupted request. Lost response identifiers prevent exact invoice reconciliation for those requests; reserves are uncertainty bounds, not claimed charges. The per-good-test reserve column excludes the separate $0.01 allowance.

These results cover three practice apps, one browser-generation round in Run C, three maps and briefings per model, 12 judge cases and 20 primary assistant tasks. Original fixture checks passed 11/11 applicable negative-before/positive-after checks; an obsolete subscription case absent from the current fixture was excluded from that validation. Real customer apps and accounts, payments, email delivery, screenshot/vision quality, broad goal planning, Ask the web and workflow quality were not evaluated. The assistant test exercised the registry in process, not HTTP transport, extra integrations or a person actually clicking Confirm. No tested model has vision-quality proof from this study. Catalog image support alone is not such proof.

Evidence sources are the original Jev-versus-LLM saved run rows, Track Knowledge's Economics: measured, not claimed, the What Track costs, and how to keep it low follow-up, the dated model-catalog study notes, and the 2026-10-26 Track model evaluation with its frozen scores, adjudications, billing reconciliation and call timings. Historical notes such as Luna's earlier 63/63 refer to a separate decision-only check; they are not Run C's 75/75 or proof of assistant quality. Only aggregate results and method are published here.

Re-run: run the frozen model-study suite on the same three practice fixtures in a fresh isolated test environment with a new approved spend cap and output set, then rescore, reconcile bills, and replace this page's dated tables and last-run date.

Guardrails

Release logLIVE · from real test results