[BUG] The chat history returned by the retrieve method of LongtermAgentMemory is confusing.

[coolbeevip]

Required prerequisites

• I have read the documentation https://camel-ai.github.io/camel/camel.html.
• I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
• Consider asking first in a Discussion.

What version of camel are you using?

0.2.20a1

System information

3.10.12 (main, Aug 16 2023, 15:12:12) [Clang 13.1.6 (clang-1316.0.21.2.5)] darwin
0.2.20a1

Problem description

The following code can reproduce this issue.

from camel.agents import ChatAgent
from camel.memories import (
    ChatHistoryBlock,
    LongtermAgentMemory,
    ScoreBasedContextCreator,
    VectorDBBlock,
)
from camel.types import ModelType
from camel.utils import OpenAITokenCounter
from colorama import Fore
from dotenv import load_dotenv

load_dotenv("../.env")  # isort:skip

# Initialize the memory
memory = LongtermAgentMemory(
    context_creator=ScoreBasedContextCreator(
        token_counter=OpenAITokenCounter(ModelType.GPT_4O_MINI),
        token_limit=1024,
    ),
    chat_history_block=ChatHistoryBlock(),
    vector_db_block=VectorDBBlock(),
)

# Initialize agent
agent = ChatAgent(system_message="You are a helpful assistant.")

# Set memory to the agent
agent.memory = memory

user_messages = [
    "What is the LLM",
    "What is the capital of China",
    "What is the memory of the agent",
]

for user_msg in user_messages:
    print(Fore.GREEN + 'User => ' + user_msg)
    response = agent.step(user_msg)
    print(Fore.BLUE + 'AI => ' + response.msgs[0].content)
    context, token_count = memory.get_context()
    print(Fore.RED + 'Memory => ' + str(context))
    print(Fore.RED + 'Token Count => ' + str(token_count))

Output

User => What is the LLM
AI => LLM stands for "Large Language Model." It refers to a type of artificial intelligence model that is trained on vast amounts of text data to understand and generate human-like language. These models use deep learning techniques, particularly neural networks, to process and generate text based on the patterns and structures they have learned during training.

Some key characteristics of LLMs include:

1. **Scale**: They are typically trained on billions or even trillions of parameters, which allows them to capture complex language patterns and nuances.

2. **Versatility**: LLMs can perform a wide range of language-related tasks, such as text generation, translation, summarization, question answering, and more.

3. **Contextual Understanding**: They can understand context and generate coherent and contextually relevant responses, making them useful for conversational agents, content creation, and other applications.

4. **Transfer Learning**: Many LLMs are pre-trained on a large corpus of text and can be fine-tuned for specific tasks, improving their performance in specialized applications.

Examples of popular LLMs include OpenAI's GPT (Generative Pre-trained Transformer) series, Google's BERT (Bidirectional Encoder Representations from Transformers), and others. These models have significantly advanced the field of natural language processing (NLP) and are widely used in various applications across industries.
Memory => [{'role': 'user', 'content': 'What is the LLM'}, {'role': 'assistant', 'content': 'LLM stands for "Large Language Model." It refers to a type of artificial intelligence model that is trained on vast amounts of text data to understand and generate human-like language. These models use deep learning techniques, particularly neural networks, to process and generate text based on the patterns and structures they have learned during training.\n\nSome key characteristics of LLMs include:\n\n1. **Scale**: They are typically trained on billions or even trillions of parameters, which allows them to capture complex language patterns and nuances.\n\n2. **Versatility**: LLMs can perform a wide range of language-related tasks, such as text generation, translation, summarization, question answering, and more.\n\n3. **Contextual Understanding**: They can understand context and generate coherent and contextually relevant responses, making them useful for conversational agents, content creation, and other applications.\n\n4. **Transfer Learning**: Many LLMs are pre-trained on a large corpus of text and can be fine-tuned for specific tasks, improving their performance in specialized applications.\n\nExamples of popular LLMs include OpenAI\'s GPT (Generative Pre-trained Transformer) series, Google\'s BERT (Bidirectional Encoder Representations from Transformers), and others. These models have significantly advanced the field of natural language processing (NLP) and are widely used in various applications across industries.'}]
Token Count => 289
User => What is the capital of China
AI => The capital of China is Beijing.
Memory => [{'role': 'user', 'content': 'What is the LLM'}, {'role': 'user', 'content': 'What is the capital of China'}, {'role': 'assistant', 'content': 'The capital of China is Beijing.'}, {'role': 'assistant', 'content': 'LLM stands for "Large Language Model." It refers to a type of artificial intelligence model that is trained on vast amounts of text data to understand and generate human-like language. These models use deep learning techniques, particularly neural networks, to process and generate text based on the patterns and structures they have learned during training.\n\nSome key characteristics of LLMs include:\n\n1. **Scale**: They are typically trained on billions or even trillions of parameters, which allows them to capture complex language patterns and nuances.\n\n2. **Versatility**: LLMs can perform a wide range of language-related tasks, such as text generation, translation, summarization, question answering, and more.\n\n3. **Contextual Understanding**: They can understand context and generate coherent and contextually relevant responses, making them useful for conversational agents, content creation, and other applications.\n\n4. **Transfer Learning**: Many LLMs are pre-trained on a large corpus of text and can be fine-tuned for specific tasks, improving their performance in specialized applications.\n\nExamples of popular LLMs include OpenAI\'s GPT (Generative Pre-trained Transformer) series, Google\'s BERT (Bidirectional Encoder Representations from Transformers), and others. These models have significantly advanced the field of natural language processing (NLP) and are widely used in various applications across industries.'}]
Token Count => 316
User => What is the memory of the agent
AI => The capital of China is Beijing. It is one of the most populous cities in the world and serves as the political, cultural, and educational center of the country.
Memory => [{'role': 'user', 'content': 'What is the LLM'}, {'role': 'user', 'content': 'What is the memory of the agent'}, {'role': 'assistant', 'content': 'LLM stands for "Large Language Model." It refers to a type of artificial intelligence model that is trained on vast amounts of text data to understand and generate human-like language. These models use deep learning techniques, particularly neural networks, to process and generate text based on the patterns and structures they have learned during training.\n\nSome key characteristics of LLMs include:\n\n1. **Scale**: They are typically trained on billions or even trillions of parameters, which allows them to capture complex language patterns and nuances.\n\n2. **Versatility**: LLMs can perform a wide range of language-related tasks, such as text generation, translation, summarization, question answering, and more.\n\n3. **Contextual Understanding**: They can understand context and generate coherent and contextually relevant responses, making them useful for conversational agents, content creation, and other applications.\n\n4. **Transfer Learning**: Many LLMs are pre-trained on a large corpus of text and can be fine-tuned for specific tasks, improving their performance in specialized applications.\n\nExamples of popular LLMs include OpenAI\'s GPT (Generative Pre-trained Transformer) series, Google\'s BERT (Bidirectional Encoder Representations from Transformers), and others. These models have significantly advanced the field of natural language processing (NLP) and are widely used in various applications across industries.'}, {'role': 'user', 'content': 'What is the capital of China'}, {'role': 'assistant', 'content': 'The capital of China is Beijing.'}, {'role': 'assistant', 'content': 'The capital of China is Beijing. It is one of the most populous cities in the world and serves as the political, cultural, and educational center of the country.'}]
Token Count => 370

It seems that the retrieve method of LongtermAgentMemory is not quite correct when concatenating chat_history and vector_db_retrieve, or I do not understand the purpose of this piece of code.

https://github.com/camel-ai/camel/blob/master/camel/memories/agent_memories.py#L156

Please note that the answer to the question What is the memory of the agent is incorrect. And finally, the order of the contents in memory is chaotic.

[  
    {  
        "role": "user",  
        "content": "What is the LLM"  
    },  
    {  
        "role": "user",  
        "content": "What is the memory of the agent"  
    },  
    {  
        "role": "assistant",  
        "content": "LLM stands for \"Large Language Model.\" It refers to a type of artificial intelligence model that is trained on vast amounts of text data to understand and generate human-like language. These models use deep learning techniques, particularly neural networks, to process and generate text based on the patterns and structures they have learned during training.\n\nSome key characteristics of LLMs include:\n\n1. **Scale**: They are typically trained on billions or even trillions of parameters, which allows them to capture complex language patterns and nuances.\n\n2. **Versatility**: LLMs can perform a wide range of language-related tasks, such as text generation, translation, summarization, question answering, and more.\n\n3. **Contextual Understanding**: They can understand context and generate coherent and contextually relevant responses, making them useful for conversational agents, content creation, and other applications.\n\n4. **Transfer Learning**: Many LLMs are pre-trained on a large corpus of text and can be fine-tuned for specific tasks, improving their performance in specialized applications.\n\nExamples of popular LLMs include OpenAI's GPT (Generative Pre-trained Transformer) series, Google's BERT (Bidirectional Encoder Representations from Transformers), and others. These models have significantly advanced the field of natural language processing (NLP) and are widely used in various applications across industries."  
    },  
    {  
        "role": "user",  
        "content": "What is the capital of China"  
    },  
    {  
        "role": "assistant",  
        "content": "The capital of China is Beijing."  
    },  
    {  
        "role": "assistant",  
        "content": "The capital of China is Beijing. It is one of the most populous cities in the world and serves as the political, cultural, and educational center of the country."  
    }  
]  

Reproducible example code

as shown above

Traceback

Expected behavior

No response

Additional context

No response

[coolbeevip]

The historical information of vector recall is disordered.

https://github.com/camel-ai/camel/blob/master/camel/memories/agent_memories.py#L153

[fengju0213]

I also noticed this issue two months ago. At that time, I thought it might be related to the retrieval problem in the retrieve method of LongtermAgentMemory,I noticed that the retrieval results are concatenated， but I didn't investigate it further.

def retrieve(self) -> List[ContextRecord]:
r"""Retrieves context records from both the chat history and the vector
database.

    Returns:
        List[ContextRecord]: A list of context records retrieved from both
            the chat history and the vector database.
    """
    chat_history = self.chat_history_block.retrieve()
    vector_db_retrieve = self.vector_db_block.retrieve(
        self._current_topic, self.retrieve_limit
    )
    return chat_history[:1] + vector_db_retrieve + chat_history[1:]