Tam Nguyen
Projects

Case study, Minerva HC Agent

Inside an agent run

Pick a question and step through what the agent did with it: which HCs it chose from the catalog, which full documents it read, the answer it gave, and what the eval checked afterwards. These are real recorded runs, so nothing here calls a model.

Recorded October 13, 2026 with gpt-5.6-sol, from 2026-10-13-s3clarify3-dev.jsonl

Step 1 of 6

Compare

  1. Input

    The question

    How is #sourcequality different from #evidencebased?

    The system prompt holds the base rules, the HC skill file, and a catalog with one line for each of the 76 HCs. There are no embeddings and no retrieval step: the model has to decide from the catalog which documents are worth reading.

  2. LLM turn 1

    Route

    It picked 2 HCs from the catalog:

    • #sourcequalityDistinguish between categories and types of information to determine source quality.
    • #evidencebasedIdentify and appropriately structure the information needed to support an argument effectively.

  3. Tool calls

    Read whole documents

    • read_file("sourcequality")ok, 11,677 chars
    • read_file("evidencebased")ok, 11,108 chars

    22,785 characters of full HC descriptions go into the next turn. The tool is sandboxed to the docs folder and only accepts names from the catalog.

  4. LLM turn 2

    Final answer

    status
    answer
    source_label
    hc_description
    HCs named
    #sourcequality, #evidencebased

    #sourcequality evaluates whether a source itself is credible and appropriate. It asks about currency, relevance, authority, accuracy, purpose, corroboration, and possible bias—the CRAAP criteria.

    #evidencebased evaluates how information is selected, connected to claims, and presented within an argument. It asks whether the evidence is relevant and sufficient, whether it actually supports the claim, whether counter-evidence is addressed, and whether its placement and format suit the audience and purpose.

    • #sourcequality: “Can I trust and use this source?”
    • #evidencebased: “Have I used the evidence effectively to support my argument?”

    They often occur in sequence: first vet potential sources with #sourcequality, then select, analyze, and structure material from them using #evidencebased. A credible article can still be used poorly, while a well-organized argument can still rest on unreliable sources.

  5. Code, not the model

    Checks

    • check: okEvery HC the answer cites was actually read. If not, the agent gets one nudge to read it or drop it, and is marked partial if it still doesn’t.
    • source_label: hc_descriptionThe answer rests on documents it read, so the HC-description label stands.

  6. Eval, case hc-004

    What the eval measured

    Expected
    answer
    Status match
    Yes
    Routing
    Yes, read #sourcequality, #evidencebased
    Extra reads
    none
    LLM turns
    2
    Tokens in, out
    9,170, 369
    Latency
    6.36 s

    Routing passes when every required HC was read. Extra reads aren’t penalized, but they show up here because they cost tokens.

Suggest

  1. Input

    The question

    I wrote code that takes a list of room requests, checks each available room, assigns a suitable one, and stops with a useful message if no assignment is possible. Which HC describes this work?

    The system prompt holds the base rules, the HC skill file, and a catalog with one line for each of the 76 HCs. There are no embeddings and no retrieval step: the model has to decide from the catalog which documents are worth reading.

  2. LLM turn 1

    Route

    It picked 3 HCs from the catalog:

    • #algorithmsApply algorithmic thinking strategies to solve problems and effectively implement working code.
    • #constraints extra, not required by the evalIdentify and apply constraint satisfaction as a way to solve problems.
    • #breakitdown extra, not required by the evalOrganize problems into tractable components and design solutions.

  3. Tool calls

    Read whole documents

    • read_file("algorithms")ok, 13,607 chars
    • read_file("constraints")ok, 9,303 chars
    • read_file("breakitdown")ok, 9,469 chars

    32,379 characters of full HC descriptions go into the next turn. The tool is sandboxed to the docs folder and only accepts names from the catalog.

  4. LLM turn 2

    Final answer

    status
    answer
    source_label
    hc_description
    HCs named
    #algorithms, #constraints, #breakitdown

    The best fit is #algorithms. Your code has defined input (room requests), iteration (checking available rooms), conditional logic (whether a room is suitable), output (assignments), and explicit termination/error handling when no assignment is possible. This closely matches the HC’s own room-assignment example.

    #constraints could be secondary if your work emphasizes defining and satisfying suitability requirements. #breakitdown would fit only if you explicitly decomposed the task into separate subproblems or functions and justified how they work together.

  5. Code, not the model

    Checks

    • check: okEvery HC the answer cites was actually read. If not, the agent gets one nudge to read it or drop it, and is marked partial if it still doesn’t.
    • source_label: hc_descriptionThe answer rests on documents it read, so the HC-description label stands.

  6. Eval, case hc-012

    What the eval measured

    Expected
    answer
    Status match
    Yes
    Routing
    Yes, read #algorithms
    Extra reads
    #breakitdown, #constraints
    LLM turns
    2
    Tokens in, out
    11,317, 400
    Latency
    6.2 s

    Routing passes when every required HC was read. Extra reads aren’t penalized, but they show up here because they cost tokens.

Clarify

  1. Input

    The question

    What is the official definition of #telepathicreasoning?

    The system prompt holds the base rules, the HC skill file, and a catalog with one line for each of the 76 HCs. There are no embeddings and no retrieval step: the model has to decide from the catalog which documents are worth reading.

  2. LLM turn 1

    Route

    No catalog line matches #telepathicreasoning, so it reads nothing. It picks close entries from the catalog to suggest instead:

    • #deductionAnalyze and apply deductive reasoning.
    • #inductionA method of reasoning where probable conclusions are drawn based on specific evidence and observations.
    • #psychologicalexplanationAnalyze the interacting factors across levels of analysis that explain and shape the behavior of complex agents.

  3. Tool calls

    Read whole documents

    No tool calls. With nothing to read, the model goes straight to final_answer in the same turn.

  4. LLM turn 1

    Final answer

    status
    clarify
    source_label
    general
    HCs named
    none

    There is no HC named #telepathicreasoning in the catalog. Did you mean #deduction, #induction, or #psychologicalexplanation?

    reason The named item is not in the HC catalog, so its official definition cannot be identified without clarification.

  5. Code, not the model

    Checks

    • check: okNothing was cited, so there was nothing unread to flag.
    • source_label: generalNo documents were read, so the answer can’t claim the HC descriptions as its source. The code would relabel it general if the model hadn’t.

  6. Eval, case hc-009

    What the eval measured

    Expected
    clarify
    Status match
    Yes
    Routing
    n/a, no HC required
    Extra reads
    none
    LLM turns
    1
    Tokens in, out
    2,333, 164
    Latency
    3.51 s

    Routing passes when every required HC was read. Extra reads aren’t penalized, but they show up here because they cost tokens.

Across the whole eval

SetStatus matchRouting recallMean tokensMean turnsMean latency
Dev (13)13/1311/11~8.7k1.8~7 s
Answers (7)6–7/75/5~5.9k1.7~6 s

The eval checks routing and status, not answer quality, since there is no judge yet. One answers case (an off-topic registration question) sometimes asks to clarify instead of declining. Broadening the rule to fix it broke another case, so it’s accepted as a known flake.