TypezoNotes

Why an AI RFP answer can be confident and wrong

Fluency is not evidence. Three ways an AI-written RFP answer loses its source, and the checks that catch each one before the bid goes out.

The problem with an AI-written RFP answer is not that it reads badly. It reads well. That is the problem. A fluent paragraph with a plausible certification and a round number is exactly what a tired reviewer wants to accept, and exactly what a buyer will check.

When a bid team asks whether a tool hallucinates, the useful answer is not yes or no. It is: what does the tool check, and what does it do when the check fails?

Three ways an answer loses its source

Hallucination is a vague word for several different failures. In RFP work, three of them matter.

1. The quote is invented

The answer cites a document and a page, and the quote is not on that page. It may be a paraphrase stitched together from several passages. It may be entirely generated. Either way, the citation points at nothing.

This one is mechanically checkable. Take the quote and search the stored text of the cited page. If the quote is not a substring of that page, whitespace aside, the citation is false. No judgement required.

2. The quote is real, the page is wrong

This survives a naive check. The sentence exists in the corpus, so a search finds it. But it is on page 9, and the citation says page 12. A reviewer who opens page 12 finds something else, and now the whole answer is suspect.

The page number and the character offsets have to come from the parser that read the document, not from the model that wrote the answer. A model that is asked to report a page number is being asked to remember a fact it was never given reliably.

3. The number is plausible and unsupported

“We have delivered this service to 40 agencies.” “Our uptime is 99.99%.” “The team has 15 years of experience.” None of these need a quote to look normal. All of them need one to be defensible.

Numbers, dates and dollar figures are the easiest claims to check mechanically, because they are discrete. Extract them from the answer, then require each one to appear in a verified quote from that answer’s own citations.

What verification can check — and cannot

A verifier can check grounding. Is the quote on the page? Does the page match the citation? Does every number in the answer appear in a cited quote? Is the cited document still present and parsed?

A verifier cannot check judgement. It does not know whether the cited capability is relevant to the question, whether the evidence is current, or whether the answer would win. A cited answer can still be weak. That is a person’s call, and saying otherwise would be dishonest.

The point of the distinction is what happens on failure. If a run fails verification, it should not be exportable as clean. It should be marked for review, and the export should carry that mark. “We checked it” is not a feature unless a failed check changes the output.

Questions worth asking a vendor

  1. Is the citation quote checked against the stored page text, or is it taken from the model’s output?
  2. Where does the page number come from — the parser or the model?
  3. Are numbers and dates checked separately, or does a plausible sentence pass?
  4. What happens when verification fails: a warning, a rewrite, or a blocked export?
  5. Can I open the cited page in the product, or only in a report?

A vendor that answers these precisely is describing an engineering decision. A vendor that answers with “our AI is trained not to hallucinate” is describing a hope.

How Typezo is built

Every sentence in a drafted answer points to a stored span, or the requirement is marked missing with a gap row. A deterministic verifier — not the model — re-checks quotes, pages and numeric claims after generation. A run that fails verification becomes needs_review and cannot be exported as clean.

That does not make the bid good. It makes the evidence checkable, which is the part software can honestly own.