Every release of SYNTHEKOS is put through the same tests before it reaches anyone: about 2,100 questions, each asked three times. A release that gets worse on any question is not released. Here is what the tests show, including where it still misses.
Across 13 full test runs (18–24 Oct 2026), about 84,000 answers: no number that isn't in the documents, no item swapped for one the documents don't have, no fact quietly borrowed from general knowledge.
When SYNTHEKOS can't support an answer from your documents, it says so. Nearly every miss below is a refusal like this. The rare exception is described under the limits. TRUTHGATE™ checks each number in an answer against the passage it came from, and withholds what it can't verify.
A question counts as right only if it was answered correctly on all three attempts. “Should be refused” questions ask for something the documents do not contain; the right answer is to say so.
| Test set | Answer is in the documents | Answer is not in the documents |
|---|---|---|
| Clean documents12 documents: manuals, fee schedules, registers, a handbook · 987 questions | 560 of 600 · 93% | 381 of 387 · 98% |
| Same documents, titles removedso it can't lean on file names · 987 questions | 587 of 600 · 98% | 377 of 387 · 97% |
| Written by another authora manual and questions we did not write · 60 questions | 37 of 40 · 93% | 20 of 20 · 100% |
| A real 968-page physics textbook (PDF)dense text and equations · 70 questions | 31 of 40 · 78% | 29 of 30 · 97% |
The long textbook is the hardest test on purpose: almost a thousand pages, equations broken across lines, the same ideas discussed in many places. Its misses are refusals too. The test documents are our own; your documents are the real test.
About 2,100 questions over five test sets, each asked three times, because the same model can word an answer differently each time. Only answers that are right every time count.
Every new version is run against the version people use today. If it gets even one question stably wrong that the current version gets right, or produces a single invented answer, it is refused, unless each loss is shown in writing to be chance variation or a scoring error. Four releases have been refused and never shipped.
Answers are scored automatically, so we checked the scorer: on 173 held-out answers, graded separately by independent AI graders working from a written rubric, it agreed 98% of the time (95% confidence: 95–99%).
For each test question we record which passages were found and which were handed to the model, so every miss can be explained: not found, crowded out, or read wrongly.
The answer comes from your documents, with the source named under it. Numbers have been checked against the passage they came from.
It is usually right. About one time in fifteen on clean documents, more often on long technical PDFs, the answer is there and it didn't find it. Naming the item or rephrasing usually helps.
Results: SYNTHEKOS 0.46.2, test runs of 18–24 Oct 2026. This page is updated as new releases are measured.
The only test that really matters.