Refactor LLM Configuration to YAML-Based System

[LeonWehrhahn]

Motivation and Context

This PR rewrites the llm_core module configuration system to address current limitations. The core motivation behind these changes is threefold:

1. Granular LLM Model Selection for Tasks: We need the ability to specify different LLM models for different tasks. For example, using a high-powered, but potentially more costly, LLM model for low-volume complex operations like generating initial structured grading instructions, while employing a faster, more economical LLM model for high-volume tasks like generating feedback on individual student submissions.
2. Flexible and Comprehensive LLM Model Configuration: We need the ability to configure not only the LLM model to use but also its inherent capabilities (e.g., whether it supports function calling or structured output) and default settings (e.g., temperature, top_p). This is crucial for supporting a diverse range of LLM models.
3. Preserved Dynamic Configuration Overrides via Headers: While not a new feature, we want to retain the existing ability to dynamically override LLM model configurations via x- headers in API requests, as used in the Athena playground.

Description

To achieve the outlined goals we introduced two YAML files to manage model configurations and capabilities:

• llm_capabilities.yml (llm_core): This file defines the core capabilities of different LLM models. It specifies default settings (like temperature, top_p) and flags for supported features (like supports_function_calling, supports_structured_output). Importantly, it also allows for LLM model-specific overrides to these defaults. This file resides at the top level of the llm_core directory and is therefore the same fore ach module - (e.g., module_modeling_llm, module_programming_llm).

• llm_config.yml (module-specific): Each module (e.g., module_modeling_llm, module_programming_llm) now has its own llm_config.yml located at the root level of the module. This file specifies the concrete models to be used for different tasks within that module. For example, the modeling module might specify a powerful model like openai_o1 for generating grading instructions and a faster, more economical model like openai_4o for generating feedback. Switching from environment variables to module-level YAML files for LLM configuration brings these settings under version control, ensuring consistent deployments and eliminating the risk of environment-specific discrepancies.

A lot of other aspects of the llm_module were changed to support this new YAML-based configuration approach. These changes are outlined in more detail in the README.

Steps for Testing

1. Verify Model Configuration:

   • Ensure that the llm_config.yml and llm_capabilities.yml files are correctly parsed.
   • Check that different modules (e.g., module_modeling_llm) can successfully load and use their specified model configurations.

2. Test Feedback Generation:

   • Test if the Feedback Generation is still working in each module (e.g., module_modeling_llm, module_programming_llm).

Testserver States

Note

These badges show the state of the test servers.
Green = Currently available, Red = Currently locked
Click on the badges to get to the test servers.

Screenshots