.Net: Phase 1 of the declarative agent schema for review #10260

markwallace-microsoft · 2025-01-22T16:31:53Z

Motivation and Context

Related #10224

Includes following

Model classes for the Agent Definition
An Agent Factory abstraction which is used to create an Agent instance from it's serialised form
Agent Factory implementations for Chat Completion, OpenAI Assistant and AzureAI Agent implementations

Description

Contribution Checklist

The code builds clean without any errors or warnings
The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
All unit tests pass, and I have added new tests where possible
I didn't break anyone 😄

joslat · 2025-02-05T20:56:08Z

Hi @markwallace-microsoft,
here is some feedback, hope it helps... I think it is great to support Declarative formats, maybe I'd make some base support for YAML and JSON as "base declarative formats" and implement custom plugins on top that support the existing declarative formats supported and the proposed Microsoft 365 copilot. this could make it easier to implement and add any new declarative format.

Maybe support:

Copilot Studio Agents? (I thnink a format json-yaml is generated below the hood to drive behavior...)
Azure AI Foundry Agents / Azure AI Agents (I believe this is code but with declarative style, though haven't explored it yet... still need to check Chris bits added some days ago - essentially I don't know what I am talking about and making an hypothesis)
AutoGen (in the studio there is a yaml/json definition that is used to persist an agent, so that could, might be a thing)
Other: unsure other frameworks like langgraph, langchain, crew ai, swarm, amazon bedrock or google services have a declarative format...

Unsure this helps, pretty much high level.

markwallace-microsoft · 2025-02-06T18:46:26Z

Hi @markwallace-microsoft, here is some feedback, hope it helps... I think it is great to support Declarative formats, maybe I'd make some base support for YAML and JSON as "base declarative formats" and implement custom plugins on top that support the existing declarative formats supported and the proposed Microsoft 365 copilot. this could make it easier to implement and add any new declarative format.

Maybe support:

Copilot Studio Agents? (I thnink a format json-yaml is generated below the hood to drive behavior...)

Azure AI Foundry Agents / Azure AI Agents (I believe this is code but with declarative style, though haven't explored it yet... still need to check Chris bits added some days ago - essentially I don't know what I am talking about and making an hypothesis)

AutoGen (in the studio there is a yaml/json definition that is used to persist an agent, so that could, might be a thing)

Other: unsure other frameworks like langgraph, langchain, crew ai, swarm, amazon bedrock or google services have a declarative format...

Unsure this helps, pretty much high level.

Thanks for the feedback @joslat

I'm using YAML in the document for my samples nut JSON would also be supported
The goal is to have a format which is Agent Service agnostic and will support Azure AI Agents, OpenAI Assistants, Chat Completion Agents, AutoGen, etc.
Another goal to be able able to convert from other formats to this one.

Copilot

Copilot reviewed 79 out of 79 changed files in this pull request and generated 3 comments.

dotnet/src/Agents/Abstractions/Definition/AgentDefinition.cs

crickman · 2025-02-21T22:15:47Z

dotnet/src/Agents/Abstractions/Definition/AgentToolDefinition.cs

+    /// <summary>
+    /// Tool definition type for code interpreter.
+    /// </summary>
+    public const string CodeInterpreter = "code_interpreter";


What is the plan for 5 other tools introduced for Azure AI Agents?

(imho) code_interpreter and file_search are implementation specific and perhaps don't belong in the abstraction?

My thinking is to include things that are cross service in the abstraction and others in the specific package, so the openai, bing etc. that Azure AI supports would go into that package.

These aren't always available for every agent service (e.g. ChatCompletion). At best, it still strikes me as an over-generalization that doesn't have a lot of impact/value compared to the alternative. On the down side, it could create both implementation and usage confusion.

crickman · 2025-02-21T22:16:46Z

dotnet/src/Agents/Abstractions/Extensions/AgentDefinitionExtensions.cs

+    /// Determines if the agent definition has a code interpreter tool.
+    /// </summary>
+    /// <param name="agentDefinition">Agent definition</param>
+    public static bool IsEnableCodeInterpreter(this AgentDefinition agentDefinition)


In alignment with previous comment, these might more propertly be defined on the agent specific extensions in the respective packages/projects.

crickman · 2025-02-21T22:20:26Z

dotnet/src/Agents/OpenAI/Definition/OpenAIAssistantAgentFactory.cs

+        if (agentDefinition.Type?.Equals(OpenAIAssistantAgentType, System.StringComparison.Ordinal) ?? false)
+        {
+            var clientProvider = kernel.GetOpenAIClientProvider(agentDefinition);
+            AssistantClient client = clientProvider.Client.GetAssistantClient();


The client-provider pattern is being deprecated in favor of direct client usage.

crickman · 2025-02-21T22:20:42Z

dotnet/src/Agents/AzureAI/Definition/AzureAIAgentFactory.cs

+        KernelAgent? kernelAgent = null;
+        if (agentDefinition.Type?.Equals(AzureAIAgentType, System.StringComparison.Ordinal) ?? false)
+        {
+            var clientProvider = kernel.GetAzureAIClientProvider(agentDefinition);


The client-provider pattern is being deprecated in favor of direct client usage.

crickman · 2025-02-21T22:21:36Z

dotnet/src/Agents/OpenAI/Definition/OpenAIAssistantAgentFactory.cs

+            };
+        }
+
+        return Task.FromResult<KernelAgent?>(kernelAgent).Result;


Is return kernelAgent; problematic?

crickman · 2025-02-21T22:21:46Z

dotnet/src/Agents/AzureAI/Definition/AzureAIAgentFactory.cs

+            };
+        }
+
+        return Task.FromResult<KernelAgent?>(kernelAgent).Result;


Is return kernelAgent; problematic?

crickman · 2025-02-21T22:23:50Z

Adding support for BedrockAgent?

crickman · 2025-02-21T22:25:30Z

dotnet/src/Agents/Abstractions/Definition/KernelAgentFactory.cs

+    /// <param name="agentDefinition">Definition of the agent to create.</param>
+    /// <param name="cancellationToken">Optional cancellation token.</param>
+    /// <return>The created <see cref="KernelAgent"/>, if null the agent type is not supported.</return>
+    public abstract Task<KernelAgent?> CreateAsync(Kernel kernel, AgentDefinition agentDefinition, CancellationToken cancellationToken = default);


If the factory cannot produce the agent, perhaps it should throw instead of returning null? (change return type to Task<KernelAgent>?)

crickman · 2025-02-21T22:27:50Z

dotnet/src/Agents/OpenAI/Definition/OpenAIAssistantAgentFactory.cs

+        Verify.NotNull(agentDefinition.Model.Id);
+
+        KernelAgent? kernelAgent = null;
+        if (agentDefinition.Type?.Equals(OpenAIAssistantAgentType, System.StringComparison.Ordinal) ?? false)


Couldn't type validation be performed as a protected method in KernelAgentFactory since Types is defined in the base abstraction? Then each factory won't have to implement custom validation. They can just call into the base method.

crickman · 2025-02-21T22:28:41Z

Be nice to see some comprehensive samples on this.

crickman · 2025-02-21T22:29:45Z

Any thoughts on reconciling the support for the PromptTemplateConfig with this new pattern? (Will both be supported or will we be preferring this new pattern for agents?)

crickman · 2025-02-21T22:31:23Z

Is this going to be released as "GA" or have an experimental tag defined?

crickman · 2025-02-21T22:43:40Z

dotnet/src/Agents/OpenAI/Extensions/KernelExtensions.cs

+        Verify.NotNull(agentDefinition);
+
+        // Use the agent configuration as the first option
+        var configuration = agentDefinition?.Model?.Configuration;


I'd expect that developers will not want to (or be allowed to) store secrets such as ApiKey in the agent manifest.

Supporting the ability to resolve environment variables (at a minimum) or resolve other configurations settings may be prudent to consider.

#security

crickman · 2025-02-21T22:46:29Z

dotnet/src/Agents/OpenAI/Extensions/KernelExtensions.cs

+        {
+            var hasEndpoint = configuration.TryGetValue(Endpoint, out var endpoint) && endpoint is not null;
+            var hasApiKey = configuration.TryGetValue(ApiKey, out var apiKey) && apiKey is not null;
+            if (hasApiKey && hasEndpoint)


This logic may be overly brittle. Isn't there the case where a non-azure endpoint can be targeted that requires an endpoint (such as a model gateway)?

It might be prudent to require explicit knowledge on whether Azure or OpenAI is intended.

Initial draft of the declarative agent schema for review

16c8b74

markwallace-microsoft added the documentation label Jan 22, 2025

markwallace-microsoft added 3 commits January 23, 2025 16:21

Add M365 Declarative Agents as an option

d4b6ee1

Merge branch 'main' into users/markwallace/adr_declarative_agents

a581a6c

Fix type and update decision drivers

e800600

markwallace-microsoft force-pushed the users/markwallace/adr_declarative_agents branch from c624bbd to e800600 Compare January 28, 2025 09:19

Update decision drivers

59f2df5

Merge branch 'main' into users/markwallace/adr_declarative_agents

3658d1c

markwallace-microsoft mentioned this pull request Feb 6, 2025

.Net: Avoid sending duplicate function tools when creating a thread #10436

Merged

4 tasks

markwallace-microsoft added 2 commits February 6, 2025 18:09

Add use cases

a462b15

Add some use case samples

e15236a

markwallace-microsoft added 6 commits February 10, 2025 10:33

Merge branch 'main' into users/markwallace/adr_declarative_agents

94c8d94

Merge branch 'main' into users/markwallace/adr_declarative_agents

2155a34

Start working on the Agent model definition classes

cd536b8

Starting adding implementations

009bd98

Work in progress

6063e98

Add implementation for AzureAI agents

d7099ee

markwallace-microsoft requested a review from a team as a code owner February 19, 2025 14:00

AzureAI Agent unit tests

3cb6c85

markwallace-microsoft changed the base branch from main to feature-declarative-agents February 20, 2025 14:57

markwallace-microsoft added 2 commits February 20, 2025 15:58

Add new unit tests pass

698ac9b

Update ADR to match the code in the PR

649cf0f

markwallace-microsoft changed the title ~~Initial draft of the declarative agent schema for review~~ Phase 1 of the declarative agent schema for review Feb 20, 2025

markwallace-microsoft changed the base branch from feature-declarative-agents to main February 20, 2025 18:04

markwallace-microsoft added 2 commits February 20, 2025 18:15

Resolve merge conflicts

818b79c

Resolve merge conflicts

6f602c5

markwallace-microsoft added the .NET Issue or Pull requests regarding .NET code label Feb 21, 2025

markwallace-microsoft changed the title ~~Phase 1 of the declarative agent schema for review~~ .NET: Phase 1 of the declarative agent schema for review Feb 21, 2025

Address initial code review feedback

30e633b

markwallace-microsoft had a problem deploying to integration February 21, 2025 19:37 — with GitHub Actions Failure

markwallace-microsoft changed the base branch from main to feature-declarative-agents February 21, 2025 19:38

markwallace-microsoft marked this pull request as ready for review February 21, 2025 19:39

markwallace-microsoft requested a review from Copilot February 21, 2025 19:39

Copilot AI reviewed Feb 21, 2025

View reviewed changes

dotnet/src/Agents/Abstractions/Definition/AgentDefinition.cs Show resolved Hide resolved

dotnet/src/Agents/Abstractions/Definition/AgentDefinition.cs Show resolved Hide resolved

dotnet/src/Agents/Abstractions/Definition/AgentDefinition.cs Show resolved Hide resolved

Fix warning

adbe914

markwallace-microsoft had a problem deploying to integration February 21, 2025 20:47 — with GitHub Actions Failure

markwallace-microsoft had a problem deploying to integration February 21, 2025 21:14 — with GitHub Actions Failure

Remove agents.openai from code coverage

431c6ee

markwallace-microsoft force-pushed the users/markwallace/adr_declarative_agents branch from d961a0a to 431c6ee Compare February 21, 2025 21:23

markwallace-microsoft had a problem deploying to integration February 21, 2025 21:24 — with GitHub Actions Error

Exclude some stuff from code coveregae

4e684a8

markwallace-microsoft temporarily deployed to integration February 21, 2025 21:30 — with GitHub Actions Inactive

Run dotnet format

cec323d

markwallace-microsoft deployed to integration February 21, 2025 21:55 — with GitHub Actions Active

crickman reviewed Feb 21, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

.Net: Phase 1 of the declarative agent schema for review #10260

.Net: Phase 1 of the declarative agent schema for review #10260

markwallace-microsoft commented Jan 22, 2025 •

edited

Loading

joslat commented Feb 5, 2025

markwallace-microsoft commented Feb 6, 2025

Copilot AI left a comment

crickman Feb 21, 2025

markwallace-microsoft Feb 21, 2025

crickman Feb 21, 2025

crickman Feb 21, 2025

crickman Feb 21, 2025

crickman Feb 21, 2025

crickman Feb 21, 2025

crickman Feb 21, 2025

crickman commented Feb 21, 2025

crickman Feb 21, 2025

crickman Feb 21, 2025

crickman commented Feb 21, 2025

crickman commented Feb 21, 2025

crickman commented Feb 21, 2025

crickman Feb 21, 2025

crickman Feb 21, 2025

.Net: Phase 1 of the declarative agent schema for review #10260

Are you sure you want to change the base?

.Net: Phase 1 of the declarative agent schema for review #10260

Conversation

markwallace-microsoft commented Jan 22, 2025 • edited Loading

Motivation and Context

Description

Contribution Checklist

joslat commented Feb 5, 2025

markwallace-microsoft commented Feb 6, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crickman commented Feb 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crickman commented Feb 21, 2025

crickman commented Feb 21, 2025

crickman commented Feb 21, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

markwallace-microsoft commented Jan 22, 2025 •

edited

Loading