[MultimodalAgent] tool_choice=required results in endless loop when using openai.realtime.RealtimeModel

[danmastrowtrusstai]

Reproduction steps

To reproduce this behaviour, use the multimodal_agent example and add the parameter tool_choice="required"

#... omitted for brevity
    agent = multimodal.MultimodalAgent(
        model=openai.realtime.RealtimeModel(
            voice="alloy",
            tool_choice="required", # This is the only changed line from the example multimodel agent.
            temperature=0.8,
            instructions="You are a helpful assistant",
            turn_detection=openai.realtime.ServerVadOptions(
                threshold=0.6, prefix_padding_ms=200, silence_duration_ms=500
            ),
        ),
        fnc_ctx=fnc_ctx,
        chat_ctx=chat_ctx,
    )
#... omitted for brevity

Expected behaviour:

• Would restrict the chat to only the tools, and respond accordingly. (In the demo scenario I would expect it only to allow asking about the weather)

Actual behaviour:

• Endless loop, even if a tool is matched for the chat message, with no reply to the user.

Logs:

2024-11-28 15:24:45,939 - DEBUG asyncio - Using selector: KqueueSelector 
2024-11-28 15:24:46,265 - DEBUG asyncio - Using selector: KqueueSelector 
2024-11-28 15:24:46,267 - INFO livekit.agents - starting worker {"version": "0.11.3", "rtc-version": "0.17.6"}
2024-11-28 15:24:46,381 - INFO livekit.agents - registered worker {"id": "AW_kLEywugKmEG5", "region": "Australia", "protocol": 15, "node_id": "NC_DSYD1A_TBTtRTfcdqWM"}
2024-11-28 15:25:01,079 - INFO livekit.agents - received job request {"job_id": "AJ_G9uLNncY5cYh", "dispatch_id": "", "room_name": "ta-englishhotelbooking-user-12b65030-6a8f-4487-979b-c8e869b8bae3", "agent_name": "", "resuming": false}
2024-11-28 15:25:01,569 - DEBUG asyncio - Using selector: KqueueSelector {"pid": 12183}
2024-11-28 15:25:01,569 - INFO livekit.agents - initializing process {"pid": 12183}
2024-11-28 15:25:01,569 - INFO livekit.agents - process initialized {"pid": 12183}
2024-11-28 15:25:01,570 - INFO my-worker - starting entrypoint {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:01,976 - DEBUG livekit.agents - http_session(): creating a new httpclient ctx {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:01,980 - DEBUG livekit.plugins.openai.realtime - sync chat context {"to_delete": [], "to_add": [[null, "item_1d0b4513cae5"], ["item_1d0b4513cae5", "item_33b6b9e44e17"], ["item_33b6b9e44e17", "item_43b704505e13"], ["item_43b704505e13", "item_9234a3ed5a53"]], "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:01,980 - DEBUG livekit.plugins.openai.realtime - added empty audio message to the chat context {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:03,127 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNLcaxuuxnAL92NSHP8", "previous_item_id": null, "item": {"id": "item_1d0b4513cae5", "object": "realtime.item", "type": "message", "status": "completed", "role": "user", "content": [{"type": "input_text", "text": "I'm planning a trip to Paris next month."}]}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:03,128 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNLTomhvME8J2Dw54hU", "previous_item_id": "item_1d0b4513cae5", "item": {"id": "item_33b6b9e44e17", "object": "realtime.item", "type": "message", "status": "completed", "role": "assistant", "content": [{"type": "text", "text": "How exciting! Paris is a beautiful city. I'd be happy to suggest some must-visit places and help you plan your trip."}]}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:03,128 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNL0VNWDe7Kcg31cINK", "previous_item_id": "item_33b6b9e44e17", "item": {"id": "item_43b704505e13", "object": "realtime.item", "type": "message", "status": "completed", "role": "user", "content": [{"type": "input_text", "text": "What are the must-visit places in Paris?"}]}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:03,128 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNLOlSqkobnDhhChiHv", "previous_item_id": "item_43b704505e13", "item": {"id": "item_9234a3ed5a53", "object": "realtime.item", "type": "message", "status": "completed", "role": "assistant", "content": [{"type": "text", "text": "The must-visit places in Paris are the Eiffel Tower, Louvre Museum, Notre-Dame Cathedral, and Montmartre."}]}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:03,317 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNLnAEcbU7ffwyRK8xi", "previous_item_id": "item_9234a3ed5a53", "item": {"id": "item_c89d92a04668", "object": "realtime.item", "type": "message", "status": "completed", "role": "user", "content": [{"type": "input_audio", "transcript": null}]}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:03,317 - INFO livekit.agents - Session initialized with chat context {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:05,199 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNNBoVBZ6laHSZcqGGn", "previous_item_id": "item_c89d92a04668", "item": {"id": "item_AYQNLJLnZAZH9qMII4hIy", "object": "realtime.item", "type": "message", "status": "completed", "role": "user", "content": [{"type": "input_audio", "transcript": null}]}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:05,589 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNN4JaLD4ok3Rlt94Wc", "previous_item_id": "item_AYQNLJLnZAZH9qMII4hIy", "item": {"id": "item_AYQNNdjHaz3Q9DAPdVtmz", "object": "realtime.item", "type": "function_call", "status": "in_progress", "name": "get_weather", "call_id": "call_dxowrRkne43Ei0gY", "arguments": ""}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:05,641 - DEBUG livekit.plugins.openai.realtime - executing ai function {"function": "get_weather", "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:05,641 - INFO my-worker - getting weather for Paris {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:06,241 - DEBUG livekit.agents - committed user speech {"user_transcript": "Hi, what is the weather for Paris?\n", "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:07,150 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNPGU1K178psC2DUZNK", "previous_item_id": "item_AYQNNdjHaz3Q9DAPdVtmz", "item": {"id": "item_58aa7cddae88", "object": "realtime.item", "type": "function_call_output", "call_id": "call_dxowrRkne43Ei0gY", "output": "The weather in Paris is Clear +7\u00b0C."}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:07,503 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNPWOFmT5BuhaRowv7a", "previous_item_id": "item_58aa7cddae88", "item": {"id": "item_AYQNPnijsovV9Ta84FN13", "object": "realtime.item", "type": "function_call", "status": "in_progress", "name": "get_weather", "call_id": "call_UYSASc0n3OemDe8r", "arguments": ""}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:07,544 - DEBUG livekit.plugins.openai.realtime - executing ai function {"function": "get_weather", "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:07,545 - INFO my-worker - getting weather for Paris {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:09,060 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNQWbnq5sPJfnE33cMi", "previous_item_id": "item_AYQNPnijsovV9Ta84FN13", "item": {"id": "item_f3ac70c877b9", "object": "realtime.item", "type": "function_call_output", "call_id": "call_UYSASc0n3OemDe8r", "output": "The weather in Paris is Clear +7\u00b0C."}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:09,460 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNRPZQEFnm3zMw8NH2P", "previous_item_id": "item_f3ac70c877b9", "item": {"id": "item_AYQNQpMQYGBYogQVjqd3y", "object": "realtime.item", "type": "function_call", "status": "in_progress", "name": "get_weather", "call_id": "call_4l8HPyaCjJoL8u3j", "arguments": ""}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:09,495 - DEBUG livekit.plugins.openai.realtime - executing ai function {"function": "get_weather", "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:09,495 - INFO my-worker - getting weather for Paris {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:11,013 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNSZlamhKQoWZRe9lWl", "previous_item_id": "item_AYQNQpMQYGBYogQVjqd3y", "item": {"id": "item_2437db96606b", "object": "realtime.item", "type": "function_call_output", "call_id": "call_4l8HPyaCjJoL8u3j", "output": "The weather in Paris is Clear +7\u00b0C."}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:11,423 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNTop8CoYVWiwbkJkpJ", "previous_item_id": "item_2437db96606b", "item": {"id": "item_AYQNS5niG8GDwRqLNxTHP", "object": "realtime.item", "type": "function_call", "status": "in_progress", "name": "get_weather", "call_id": "call_aIX3pCxhhQ2ol3L5", "arguments": ""}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:11,461 - DEBUG livekit.plugins.openai.realtime - executing ai function {"function": "get_weather", "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:11,461 - INFO my-worker - getting weather for Paris {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:13,001 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNUvMh9M1YegzfgBrYg", "previous_item_id": "item_AYQNS5niG8GDwRqLNxTHP", "item": {"id": "item_6bbd6c6121c7", "object": "realtime.item", "type": "function_call_output", "call_id": "call_aIX3pCxhhQ2ol3L5", "output": "The weather in Paris is Clear +7\u00b0C."}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:13,375 - DEBUG livekit.plugins.openai.realtime - conversation item created {"type": "conversation.item.created", "event_id": "event_AYQNVT0Vy06yWYkF481wb", "previous_item_id": "item_6bbd6c6121c7", "item": {"id": "item_AYQNUa3oPGZCkTAT7lOYw", "object": "realtime.item", "type": "function_call", "status": "in_progress", "name": "get_weather", "call_id": "call_sPaa7W3OMOrNVz4E", "arguments": ""}, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:13,408 - DEBUG livekit.plugins.openai.realtime - executing ai function {"function": "get_weather", "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:13,409 - INFO my-worker - getting weather for Paris {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
^C2024-11-28 15:25:14,299 - INFO livekit.agents - shutting down worker {"id": "AW_kLEywugKmEG5"} # Note this is where I cancelled the python app
2024-11-28 15:25:14,300 - INFO livekit.agents - job exiting {"reason": "", "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:14,300 - DEBUG livekit.agents - shutting down job task {"reason": "", "user_initiated": false, "pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}
2024-11-28 15:25:14,324 - DEBUG livekit.agents - http_session(): closing the httpclient ctx {"pid": 12183, "job_id": "AJ_G9uLNncY5cYh"}

FYI @jayeshp19

[longcw]

The behavior is expected when tool_choice="required" as it will force the LLM to response a function call instead of generating the answer. The process of the functioning call is: 1) LLM outputs one or multiple function call requests, 2) agent executes the functions; 3) agent append the function outputs to the chat context and require a new LLM response which should container the answer for human.

When tool_choice="required", the step 3 triggered another function call instead of responding the answer. Most likely the required is not for the use case of limiting the conversation to the tools.