Benchmark Results¶

This page records the current checked benchmark artifacts. It is a results snapshot, not a marketing leaderboard and not a replacement for the underlying JSON artifacts. Read it as a guide to what the repo can prove today, what is only diagnostic, and which claim gates still need external evidence.

Reality Bench / EMRP v1¶

Reality Bench is the Engineering Memory Reality Protocol v1 (emrp-v1). It is the flagship protocol for repeated-mistake prevention, stale-rule suppression, forbidden-action blocking, patch evidence, source-cited recall, deletion/privacy boundaries, abstention, and public memory QA. Its public output is artifacts/public/evidence-table/ until the strict market gate passes.

The gate requires a frozen manifest, retained raw outputs, the same input stream, the same budgets, the same judge, Cognibrain plus at least two original competitor implementations, public artifact hash, and independent replication hash. Capability-profile adapters are explicitly blocked from public comparison claims.

How To Read This Page¶

Benchmark rows in this repo are deliberately strict. A high local score means "this implementation passed this checked diagnostic." It does not automatically mean "best product on the market" or "quality proven against every competitor."

The practical reading order is:

1. Read Market Readiness Summary for the human headline.
2. Read What You Can Safely Say before using any result in public copy.
3. Use Reproduce The Main Proof Surface when you need fresh local evidence.
4. Use Artifact Snapshot and the detailed sections when you need exact generated-file provenance.

The short interpretation: Cognibrain has strong engineering-memory diagnostics and unusually explicit proof boundaries. Independent market leadership remains blocked until judged original competitor runs, public hashes and replication exist.

What You Can Safely Say¶

Use this page to keep public language tight:

The useful product story is: Cognibrain is built to stop coding agents from repeating known engineering mistakes, and this repo carries local proof for that loop. The blocked story is: Cognibrain has not yet published a fair, independently replicated market leaderboard against original competitor systems.

Reproduce The Main Proof Surface¶

For a normal maintainer check, start here:

npm test
npm run build
npm run release:check

For proof-specific refreshes:

npm run internal -- audit:truth
npm run internal -- audit:plan-gaps
npm run internal -- audit:latest-analysis
npm run internal -- proof:plan

For benchmark artifacts, use the specific commands in the maintainer refresh map below. Generated outputs belong under artifacts/; review them as evidence, but do not turn them into broader claims unless their claim gates allow it.

Benchmark Families¶

Maintainer Refresh Map¶

Use these commands when the checked result surface needs to be refreshed. They write generated outputs under artifacts/; those outputs are local review evidence and are not shipped as source documentation.

Market Readiness Summary¶

The current artifacts support a professional engineering-memory claim: Cognibrain has strong local diagnostics for coding-agent context, guard and patch-evidence workflows, and the repo has explicit proof gates that block overstated market claims.

In one sentence: the repo currently proves a serious local engineering-memory system with strong regression diagnostics, while deliberately blocking unearned market-leadership claims.

What is currently strong:

• Product truth audit: 69/69 checks passed, 0 open code-truth gaps.
• Plan closure: 16/16 plan-gap checks, 10/10 latest-analysis checks and 10/10 full-plan proof checks.
• CogniCodeBench: 1,000 scenarios, 100.0% full-system diagnostic score and 96.0% integrity score.
• Real-world black-box harness: coverage, raw-output retention and telemetry are ready for judged runs.

What is not yet claimable:

• No overall "best memory solution on the market" claim.
• No public leaderboard claim until original competitor systems are judged on the same frozen protocol.
• No BEAM quality claim: BEAM artifacts are useful diagnostics, but their pass gates are currently false and qualityClaimAllowed=false.

Use the detailed sections below as the evidence register behind those three sentences. They are intentionally more precise than marketing copy because each row needs to preserve its proof boundary.

Artifact Snapshot¶

Public Benchmark Dataset Stress¶

These rows are Cognibrain runs against public-style datasets and local baselines. They are useful regression evidence, but they are not original product runs for other memory systems and are not a fair market leaderboard.

Benchmark Integrity¶

BEAM Weaknesses¶

Claim Boundaries¶

Benchmark claims on this page are limited to the checked artifacts listed above. Older or stronger rows are not carried forward without a current artifact. The SVG chart is bounded the same way: public dataset, Arena and ablation bars display proof or diagnostic status, and local/API-shape/ablation diagnostics are marked claim-blocked or not-market-proof instead of being shown as market or quality proof. The public leaderboard.json artifact now marks local fixture and deterministic coverage scores as claimAllowed=false diagnostic rows; quality claims require LLM/harness proof, and market comparisons require comparable public benchmark artifacts. The nextgen-benchmarks.json lifecycle suite is also diagnostic-only: its deterministic fixture, structural and harness-review checks validate local regressions, but the artifact carries qualityClaimAllowed=false and marketClaimAllowed=false until judged by an external LLM/harness or backed by comparable public-benchmark proof. Answer-generation artifacts carry the same boundary at the raw artifact layer: deterministic extractive answers and expected-term coverage are diagnostic-only, configured judge-command failures fail closed, and qualityClaimAllowed=true is emitted only when every question has a successful external LLM/harness judge contract result. The real-world protocol currently classifies 0 checked artifacts as fair cross-system leaderboard evidence. The current LLM-intelligence artifact is judge-blocked and intentionally reports no quality score. It retains same-manifest raw outputs for Cognibrain, Basic Memory and LangMem so retrieval weaknesses remain inspectable while the LLM/harness judge is unavailable. Arena external command runners follow the same boundary: runner-provided checks are diagnostic only unless MEMORY_ARENA_JUDGE_COMMAND validates the raw product evidence into strict boolean checks. Operator Memory native competitor runners are bounded the same way: runner checks are ignored for scoring unless MEMORY_OPERATOR_MEMORY_JUDGE_COMMAND validates raw source-aware evidence into strict boolean checks. The public-dataset market-gate now follows this too: local LoCoMo, LongMemEval or BEAM recall wins are diagnostic-public-benchmark-baseline unless every included benchmark artifact carries LLM/harness or comparable public-benchmark proof. The legacy synthetic retrieval report is also diagnostic-only: its expected-id substring scorer is useful for regression smoke tests, but claimBoundary marks quality and market claims as blocked until an LLM/harness judge or comparable public-benchmark proof is present. The individual local LoCoMo and LongMemEval artifacts carry the same claimBoundary: evidence/session-id recall remains useful retrieval diagnostics, not answer-quality or market-comparison proof, until an LLM/harness judge or official same-protocol public artifact is attached. External-hard public dataset stress is diagnostic-only for the same reason: even when all stricter-budget rows improve over local baselines, claimAllowed=false unless every child artifact is scoreable with LLM/harness or comparable public-benchmark proof. BEAM raw artifacts now carry this boundary directly: deterministic rubric, entity and evidence-support scoring is local-diagnostic, and passed=true is reserved for runs backed by MEMORY_INTELLIGENCE_COMMAND or an equivalent LLM/harness evidence judge.

Real-World Fairness Boundary¶

The current real-world protocol artifact is a preregistration and evidence classifier, not a score table. A result is leaderboard-eligible only when the dataset, prompts, scoring, budgets, and adapter contract are frozen before system tuning; every system receives the same input stream; every system runs through its original package, CLI, SDK, service, or official API; and raw outputs, cost, latency, versions, errors, and scorer traces are retained. Market leaderboard eligibility also requires a public immutable artifact hash for the exact judged run, an independent replication artifact hash, a third-party protocol with at least 30 tasks, and preregistered latency/cost budgets for the judge and attached systems.

Current artifact classes:

Real-World Black-Box Smoke¶

This is the first neutral harness implementation. It uses a frozen realworld-blackbox-v1 manifest and a generic reset, ingest, query, export-raw-outputs, teardown contract. It records raw outputs, setup blockers, latency percentiles and cost fields. The current manifest coverage gate requires at least 15 queries, at least 3 queries per task bucket, and at least 3 abstention/privacy/deletion queries before the smoke can be considered coverage-ready. Score, recall, abstention and leakage quality metrics stay not scored until MEMORY_REALWORLD_JUDGE_COMMAND points to a fixed LLM/harness judge. Structured evidence-id matches are diagnostics only; they are not quality scores and are not leaderboard proof. The checked LLM path is scripts/benchmark/realworld-openai-judge.mjs, which passes retrieved text rather than retrieved evidence IDs to the judge. Judged runs are fail-closed: the harness requires exactly one decision for every manifest query, no unknown or duplicate query IDs, finite 0..1 score and confidence values, strict JSON booleans for decision fields, and finite latency/cost metrics from external command runners. Malformed judged outputs retain raw retrieval outputs but are not quality-scored or leaderboard-eligible. Configured original-system command failures are separated from missing credentials: non-zero exits, invalid JSON, and missing judged raw-output shapes are classified as same-run-command contract diagnostics with no quality score, not as credential-blocked rows. It is still not a leaderboard because the default no-score artifact has no LLM/harness judge command or original competitor commands configured.

Immediate Cognibrain raw-output diagnostics: support queries can retrieve an unrelated decoy alongside the correct evidence; temporal-update queries can retrieve stale and current evidence together; deleted-token abstention can still retrieve unrelated support facts instead of returning no evidence. These signals define the next retrieval work, but they are diagnostics only until the LLM/harness judge scores them.

LLM-Intelligence Retrieval Smoke¶

This separate smoke run configures MEMORY_INTELLIGENCE_COMMAND with scripts/benchmark/openai-memory-intelligence.mjs and scores delivered outputs with scripts/benchmark/realworld-openai-judge.mjs. It keeps raw diagnostics for excluded candidates but passes only non-excluded delivered text to the quality judge. The current checked rerun is judge-blocked, so the rows below are not quality scores; they are raw-output and latency diagnostics.

The current improvement in code is the provider path, not a fresh market score: JSON-command intelligence calls now use compact result payloads and a bounded short-lived response cache, so repeated semantic LLM/harness judgements do not pay duplicate process and provider cost. This keeps retrieval intelligence provider-driven while targeting the previous latency weakness. Provider evidence is also fail-closed at the memory level: answerable=true alone does not authorize delivery, the top-level verdict must carry strict boolean answerable and finite 0..1 confidence fields, and every returned memory must have an explicit per-memory LLM/harness decision with finite 0..1 confidence before it can enter injected context; malformed answerable provider output is converted to excluded evidence rather than calibrated from heuristics. Direct harness decisions must use valid decision values, no unknown or duplicate memory ids, and retrieval calibration preserves earlier unsafeToInject flags instead of overriding them. The real-world Cognibrain adapter applies the same delivery boundary: review-gated or otherwise unsafeToInject retrieval results stay in raw diagnostics but are not exported as delivered retrievedText for the quality judge. External original-system runners are also fail-closed against the frozen manifest: raw outputs must contain exactly one output for every query id, no unknown or duplicate query ids, finite latency values, and no more than the query topK returned text/evidence items. Cost and latency gate fields must be finite non-negative numbers; malformed metric fields keep raw diagnostics but do not pass the cost/latency eligibility gate. The harness now also separates latest-attempt artifacts from the last successful judged artifact so credential failures do not erase scoreable scientific evidence, and it keeps same-manifest raw outputs when judge failures happen after retrieval. The current artifact also records judge and command fingerprints without raw command values, and redacts diagnostic blocked reasons before writing them.

CogniCodeBench¶

Diagnostics: integrity 96.0%, overfit risk low. The current run reports no open CogniCodeBench design weaknesses: task-prompt leakage is low, patch evidence is generated through a proposal stage instead of direct expected-action injection, and the strongest internal ablation remains below the full system.

Claim boundary: the current CogniCodeBench run is proof=local-diagnostic, qualityClaimAllowed=false, and marketClaimAllowed=false. Local scenario checks, ablations, leakage diagnostics, and synthetic patch checks remain diagnostic-only unless an external MEMORY_COGNICODEBENCH_QUALITY_JUDGE_COMMAND LLM/harness judge validates the full report.

Baselines¶

Arena¶

This is a Cognibrain lifecycle diagnostic, not a fair real-world leaderboard. Rows with api-shape proof levels are capability models, not original product runs. Arena command runners must return structured JSON checks; raw text output is retained as diagnostic evidence but is not parsed into success scores.

Native Competitor Smoke¶

This smoke run checks native runner plumbing on Cognibrain-designed scenarios. It can expose setup and adapter weaknesses, but it is not a neutral cross-system result table.

External Hard¶

This diagnostic run uses public datasets with stricter retrieval budgets than the default snapshot. It is intended to expose weak margins, not to replace the standard artifact rows above. These are diagnostic retrieval stress rows, not quality or market claims.

Original Public Benchmarks¶

This run uses cloned upstream benchmark repositories directly. It does not count local adapters as original benchmark evidence.

Protocol classes: exact-upstream means original code, command and scoring; dependency-stabilized means original code and scoring with documented external dependency environment; adapter diagnostics are not counted as original scores.

Original Benchmark Learnings¶

Basic Memory Adapter Diagnostic¶

This is a local adapter diagnostic, not an original public benchmark. It uses basic-memory==0.21.5 through Markdown files, Basic Memory full-text reindexing, and MCP search_notes.

Hardening¶

Dataset: artifacts/cognicodebench/scenarios.json

SHA-256: 73bda620cb66a2db11bc0d12326d03e7323e90f17931309be462159067f2368e