llm struggling to deal with text-to-sql outputs

[xiaohanwu001]

I was following this example and I realized that LLMs often have trouble dealing with the outputs of sql_engine. For example if I run agent.run("How many clients are in the table?"), LLM would use sql_engine to get an initial answer, which is fine:

but then sometimes it would try to do this:

This doesn't happen every time, but a lot of the times I see LLMs try to process the outputs of sql_engine with something like result[0], result[0][0]. The models get errors and get stuck there. E.g. when I experimented with qwen2.5-coder-32b and let it pick a random row from a table, it wrote a bunch of python codes trying to generate a random number but it got stuck at count=result[0][0] and it doesn't know how to get around it. I tried Gemini too but the same issue occurs. I guess it seems like LLMs do not always remember that the function sql_engine "returns a string representation of the result"?

So I'm wondering if there's a good way to tell LLMs what the best way is to process the outputs of sql queries. Perhaps define another tool to tell LLMs how to understand the outputs of sql_engine? Or maybe just remind it not to process the outputs of sql_engine at all when I call agent.run()?

On another note, are the return values of tools not allowed to be lists?

[fedorzh]

I often get the same, even when using Qwen/Qwen2.5-72B-Instruct - the result (sometimes correct, sometimes not) is in the execution logs rather than output, and then the processing fails