# Assessment Engine

> The Assessment Engine is the engine behind AskDona Assessment. AskDona describes it as built to be deterministic at every stage, from text extraction and OCR to retrieval and judgment, so the same documents, assessment items and configuration give the same assessment, byte for byte.

- Status: Current. Last verified 2026-10-21.
- In effect since: 2026-10-21
- Applies to: The engine behind AskDona Assessment, as described on askdona.com and tested in GFLOPS's preprint. Not a statement about every deployment or configuration of the commercial product.
- Canonical: https://askdona.com/ai/assessment-engine.md
- Sources: https://askdona.com/en/capabilities/assessment-engine/, https://arxiv.org/abs/2610.21300

The Assessment Engine is part of AskDona Assessment (formerly Batch Assessment; see https://askdona.com/ai/batch-assessment.md). Its page went up on askdona.com on 21 October 2026.

## Facts

- **What it is:** the engine behind AskDona Assessment, built to be deterministic from text extraction and OCR to retrieval and judgment.
- **What "deterministic" means here:** run an assessment again with the same documents, assessment items and configuration, and the engine returns the same label, the same reason and the same quoted evidence. That lets a recorded judgment be checked, and lets the effect of a change be listed.
- **Stages:**
  1. Extract text: the text layer is read; scanned pages go through OCR served with batch-invariant inference and greedy decoding.
  2. Chunk and embed: text is split by a fixed rule and embedded with deterministic kernels; vectors are stored under their content hash.
  3. Retrieve: exact, scoped retrieval combining keyword and vector similarity with fixed tie-breaking; no approximate index.
  4. Judge: the judgment model writes the preliminary assessment with batch-invariant inference, greedy decoding and schema-constrained output.
  5. Verify quotes: every quotation is located in the passage it cites; a quotation that cannot be found is removed, and if none remain the label becomes "Cannot determine".
  6. Record: label, reason and verified evidence are written to a hash-chained ledger, with a hash for every intermediate result and its configuration.
- **Research behind it:** the preprint "Making every stage of medical device document assessment with large language models deterministic" by Ryosuke Suzuki, GFLOPS Co., Ltd. (arXiv:2610.21300 [cs.CL], 15 October 2026). The study was funded by GFLOPS, which employs the author and develops AskDona Assessment, the implementation tested.
- **Published results (from the preprint):**
  - 12 of 14 commercial LLM deployments changed the label of at least one assessment when sent the same request again, at the most deterministic settings each accepted.
  - On 337 public FDA documents for blood glucose meters (3,150 pages, 100 read by OCR), 776 of 776 assessments were byte-identical to the reference run at every stage, across every rerun tested with the same configuration.
  - 516 infusion-pump assessments were identical across four runs in different orders, concurrencies and days.
  - For one open-weight model with eight concurrent senders, all 350 outputs were byte-identical with batch-invariant inference enabled. With it disabled, the full output varied on 58 of 70 prompts and the label on 5.
- **Limits:**
  - Reproducible is not correct: the preprint measures reproducibility only, not how often assessments are right.
  - Determinism holds for one configuration. On another GPU model, outputs differed; the GPU model and inference settings are part of what is recorded.
  - Byte-level reproducibility is not a regulatory requirement and is not sufficient for validation on its own.
  - These are research results under tested conditions, not a guarantee for every deployment of the commercial product (see also https://askdona.com/ai/askdona-assessment-solution-manual.md).
- **Human role:** the engine prepares a preliminary assessment (a label, a reason and quoted evidence). A qualified expert checks it against the source, and the expert's decision, not the preliminary assessment, enters the quality record.