Source Coverage Must Bound Review Conclusions

Status · SUPPORTED

Research class

METHOD UPDATE

Completion date

July 23, 2026

Research question

Can Review Lab prevent incomplete source retrieval from becoming an overconfident review conclusion?

Prior hypothesis

A deterministic source boundary and status cap can keep model conclusions within the material actually reviewed.

Method

Review Lab classifies source coverage before model review, refuses insufficient material without a provider call, caps optimistic status when coverage is partial or uncertain, validates evidence references, and now requires public review language to distinguish none found in reviewed material from none exists.

Observations

• Complete readable page evidence reached full-verified coverage.
• Plain text with uncertain scope and documents with missing pages did not reach full-verified coverage.
• Insufficient material stopped before a provider call.
• An optimistic result on partial material was deterministically capped.
• Unknown evidence references and malformed outputs failed closed.

Supporting evidence

• Nineteen Review Core tests passed, with one optional live test skipped.
• Sixteen product-readiness tests passed after the counterevidence rule was added.
• The partial-source fixture was capped at partially supported.
• The insufficient-source fixture made zero provider calls.
• Unknown evidence references failed validation.
• The production prompt contains the source-coverage counterevidence boundary.

Counterevidence

• The deterministic staged core remains development-gated.
• The prompt-level production rule still depends on provider compliance.
• No blinded human comparison has measured overall review usefulness.

Conclusion

Review Lab can deterministically prevent incomplete source coverage from receiving its highest experimental review status, and its production prompt now prevents missing retrieved counterevidence from being described as nonexistent. This improves calibration but does not establish overall review superiority.

Confidence

High for deterministic boundary enforcement; medium for prompt-level production compliance.

Limits

• The staged core is still development-gated.
• Prompt compliance is not equivalent to deterministic enforcement.
• Review quality outside coverage calibration was not measured.

Production consequence

Public Review Lab instructions now require coverage-bounded counterevidence language. The experimental paper-review core fails closed and caps conclusions by source coverage.

Next unresolved question

Does deterministic coverage calibration improve factual accuracy and human usefulness across a blinded set of long and incomplete documents?

Related public work

• Review Lab
• Hybrid Paper Review Architecture Comparison — Run 001