Projects / Field knowledge search
A field test for AI manual search
Can a technician follow an AI answer on the device in front of them? Give your search tool the two fictional manual excerpts below. Run each question in a fresh conversation so the tool can’t borrow a model number from the previous question.
The two manuals
- Alder 10 · Display guide, §2
- “The display always shows the last five readings.”
- Alder 20 · Display guide, §4
- “To view the last twenty readings, choose Menu, then Reading history.”
Three questions to run
- “On the Alder 20, how do I see my last twenty readings?” Look for both menu steps and a citation to §4 of the Alder 20 guide. Open the citation and check that it lands on those instructions.
- “On the Alder 10, how do I see my last twenty readings?” The guide establishes what the display shows. It leaves open whether older readings are available elsewhere. Mark the answer down if it invents a menu path or declares that older readings are lost.
- “How do I see my last twenty readings?” The question omits the model. An answer can ask which device the technician has, or explain both cases with the limits of each guide. A single unconditional menu path fails this check.
Record what went wrong
Save the answer and the passage returned by search for each question. If search returned only the Alder 20 passage for the Alder 10 question, investigate how it selects a manual. If it returned the Alder 10 passage and the answer still invented a menu, investigate how the answer uses the passage. Keeping both makes the next release easier to compare.
Download the three-question review sheet (CSV)
Read the original article
The Medium article uses an inspection log to explain how to decide what evidence is enough to give a technician an answer. The right answer in the wrong manual