[Feature Request] Implement KnowNO for HITL with structure output

[lightaime]

Required prerequisites

• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

Motivation

In dynamic and unfamiliar environments, large models and robots often face a common problem: making overly confident yet incorrect predictions. A team of researchers from Princeton University and Google DeepMind addressed this issue by introducing the KnowNo framework. This system helps robots recognize when they’re uncertain and allows them to ask for help from humans when necessary, using a concept called conformal prediction (CP).

Solution

No response

Alternatives

No response

Additional context

No response

[Wendong-Fan]

lead @MuggleJinx , support&review @X-TRON404 , @WHALEEYE

[MuggleJinx]

Some thoughts about the implementations, I think the core idea of the paper is that:

1. Leverage the ability of model (especially reasoning model) to give multiple potential next steps.
2. Review the previous choices, (e.g., A, B, C, D), assign them with potential probabilities / credibilities.
3. Compare with the user given threshold, if there are choices reach the threshold, take the step, otherwise let user review the choices and decide.

Some toy example I just came up:
agent.step_with_reasoning("Who is the best football player in the world? Tell me about his career.", choices=3, threshold=0.5 )

(Agent processing)
I think potential choices are A. Messi, B. Ronaldo, C. Neymar ....

Now I need to assign credibility to all choices:

A. Messi, 0.42
B. Ronaldo, 0.38
C. Neymar, 0.2

Since no choice is above 0.5, I should ask user to decide.

(call `ask_human_via_console`)

User input: I prefer Messi, plz continue the task:

(agent continues...)

What do you think? We can add one more argument in step() and implement this process internally.

@lightaime @Wendong-Fan