How we claim a number
Summary
Four levels of evidence, and the rule that a lower one is never renamed as a higher one. What every published result has to carry, and why our numbers are not agent scores.

Key results
| Measurement | Result | Source |
|---|---|---|
| Deterministic safety and execution scenarios | 55 of 55 | VALIDATION.md |
MELRA publishes a narrow claim only when the implementation, the harness, the input revision, the metric and the raw result can all be inspected. A fast action is not necessarily a correct one. Retrieval coverage is not answer accuracy. A microbenchmark of one component is not a task leaderboard.
Four levels of evidence
- Unit or evaluation result: deterministic behaviour of a code path.
- Component microbenchmark: local latency or correctness for one bounded component.
- Dataset retrieval result: objective evidence coverage over a named dataset revision with a recorded hash.
- Task benchmark result: end-to-end goal completion in an official environment, scored by its official evaluator.
A lower level is never renamed as a higher one. A process that exits with 0 has not proved the user's goal succeeded. A successful click has not proved the interface reached the state it should. A replay of selected templates is not a representative browser score, and discovering what a desktop adapter can do is not an OSWorld score.
The runtime's deterministic safety and execution scenarios are an example of the first level: all of them pass. That says those scenarios hold. It says nothing about how well an agent completes tasks.
What a result has to carry
Every committed result includes the benchmark and metric name; which implementation ran; the dataset's source, licence and SHA-256 where there is a dataset; the sample count and any exclusions; how many model, embedding and network calls were made; the median and 95th-percentile latency for any timing; the environment; and its limitations and the ways it cannot be compared.
Runs are kept deterministic where they can be. The memory scorer makes no model, embedding or network calls, so it has no reader or judge variance. Browser fixtures are local synthetic pages with fixed mutation schedules. Terminal measurements use the current Node executable with exact argument separation. Computer-use microbenchmarks only make read-only capability probes.
A paired browser run, done fairly
For the registered browser evaluation, the baseline and the candidate share the same frozen model, prompt, tool schema, browser, environment images, task order and limits. Which side runs first alternates by task. Every site is reset before every side, and an infrastructure failure on either side voids the pair. Process exit, model text and successful browser actions do not count as success: only the official evaluator's score does. Aggregates use the fixed registered denominator, and raw recordings stay local; only sanitised fields are published.
Why our numbers are not agent scores
An execution runtime controls execution. An agent or model still chooses the goals, the operations and the evidence. End-to-end results depend on the model, the prompt, the context policy, the state of the environment and the evaluator, so results on WebArena, OSWorld, OSWorld-MCP or LongMemEval have to pin all of them. MELRA claims no external benchmark score unless that exact public evaluation has been run and released.
Updating a result
- Run the committed command from a clean checkout.
- Record the dataset revision and its hash.
- Keep the raw JSON. Never copy only a headline.
- Explain any change of method beside the new result.
- Never overwrite a failing result to keep an old claim.