RAG processFile attempts to embed entire files causing errors for large documents

[omikolaj]

Describe the bug

ragknowledge.ts file is running embd function on the entire content of the document, often causing errors with going over token limitations of the underlying model. The code attempts to embed the entire document, and then chunks it out.

To Reproduce

1. Create 'knowledge' directory in 'characters' directory.
2. Add a large pdf to the directory
3. Update *character.json file knowledge property to run embeddings on the file
4. Update *character.json file settings.ragKnowledge property to 'true'
5. Configure .env file to use USE_OPENAI_EMBEDDING=true and provide OPENAI_API_KEY and EMBEDDING_OPENAI_MODEL=text-embedding-3-large (or small)
6. Start the server, notice errors:

[2025-03-02 15:14:48] ERROR: API Response: {
  "error": {
    "message": "This model's maximum context length is 8192 tokens, however you requested 16376 tokens (16376 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.",
    "type": "invalid_request_error",
    "param": null,
    "code": null
  }
}

Expected behavior
All supported documents embedded without errors

Screenshots

Additional context

The code that does this was added on Jan5. It apppears to be in the latest release tag. Its possible Im setting something up wrong, but its not clear what.

[github-actions]

Hello @omikolaj! Welcome to the elizaOS community. Thank you for opening your first issue; we appreciate your contribution. You are now an elizaOS contributor!