How every number is produced.How every number is produced.

Candidate questions are derived from the paper’s own structured content: population, intervention or exposure, comparator, outcomes, setting, and stated limitations. Each candidate is assigned a question class — clinical decision, mechanism, comparative, safety, population, or evidence strength — and a difficulty level reflecting how specific an answer must be.

Candidates are then filtered for answerability. A question the paper genuinely cannot inform is excluded, because including it would depress every score for a reason that has nothing to do with visibility.

The resulting set is presented for review before any run. You can remove questions, add questions, and change class assignments. The final set is frozen, versioned, and stored with every run made against it. Changing the set starts a new series rather than continuing the old one.

Every question is issued multiple times per engine within a run, with a fixed interval between repeats and no carried-over conversation state. Repeats are independent sessions; a follow-up in the same thread would measure the thread, not the index.

Results are aggregated per question before being aggregated per paper, so a question that happens to be asked more often does not dominate. Sample size, run window, and the number of repeats are recorded with the run and shown in every report.

Where a question returns a refusal or an error, the outcome is recorded as such rather than as an absence of the paper. Silently coding failures as negatives would bias every component downward.

Where a surface exposes its retrieved sources, detection is direct: identifiers are matched against the paper’s DOI, its preprint and accepted-manuscript versions, and known aliases. Version linkage matters, because retrieval of a preprint is retrieval of the work.

Where sources are not exposed, retrieval is inferred from observable evidence in the answer, and the record is explicitly marked as inferred. Inferred retrieval is reported separately from observed retrieval and is never presented as equivalent.

Known error modes: an engine may name a source it did not retrieve, and may retrieve a source it never names. Both are counted against confidence in the estimate rather than corrected silently.

Detection matches identifiers first — DOI, PubMed identifier, arXiv identifier — then resolvable links, then title and author strings normalised for punctuation, diacritics, and abbreviated journal names. A match on title alone is recorded at lower confidence than a match on an identifier.

Fabricated references are a real failure mode of generative systems. A reference that names the paper but resolves to nothing, or resolves elsewhere, is recorded as a malformed citation and reported in its own category rather than being counted as a citation.

Claims in the answer are segmented and each is checked against the paper’s reported findings — effect sizes, endpoints, populations, and stated conclusions. A claim counts as evidence use when it is specific enough to be attributable and consistent with what the paper reports.

This produces two informative asymmetries. A paper can be cited without its evidence being used, where the reference is decorative. A paper’s evidence can be used without being cited, where the finding has become background knowledge and attribution has been lost. Both are reported, because they call for different responses.

Detection is model-assisted and audited against human coding on a sampled subset. The measured agreement rate is published with each methodology version rather than described qualitatively.

Three kinds of variance are tracked: within-run variance across repeats of the same question, across-engine variance for the same question, and across-run variance over time. Each is reported separately, because they have different causes and different remedies.

Measured variance also sets alerting thresholds. A change is only surfaced when it exceeds the noise the instrument is known to produce, which is why alert volumes are low by design.

A well-indexed clinical topic with open full text behaves differently from a specialised topic with few open sources. Raw component values are therefore reported alongside a field-relative view, computed against papers sharing subject area, publication year window, and access status.

Field-relative values are only shown where the comparison group is large enough to be meaningful; below that threshold the raw values are shown alone and the reason is stated. Comparisons are never drawn across unlike fields, and no cross-field ranking is produced.

Each synthetic respondent is defined by an explicit, stored profile: role, discipline, seniority, evidence threshold, and prior exposure. Respondents answer independently first, then optionally see each other’s reasoning and revise. Both rounds are retained along with the reason given for each revision.

Runs are repeated to measure how stable a panel’s conclusions are under identical inputs, and role wording is perturbed to test how sensitive results are to prompt phrasing. Both figures ship with the output.

Modelled stakeholder response, calibrated against real published clinical discourse. Directional research intelligence — it complements human validation rather than replacing it.

Simulation in detail