Merge branch 'main' into u/#4354

.github/workflows/checks.yml

@@ -190,6 +190,7 @@ jobs:
         run: |
           source ${{ github.workspace }}/python/.venv/bin/activate
           poe gen-proto
+          poe gen-test-proto
         working-directory: ./python
       - name: Check if there are uncommited changes
         id: changes

CONTRIBUTING.md

@@ -24,10 +24,62 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
 contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
 
-## Roadmaps
+## Running CI checks locally
 
-To see what we are working on and what we plan to work on, please check our
-[Roadmap Issues](https://aka.ms/autogen-roadmap).
+It is important to use `uv` when running CI checks locally as it ensures that the correct dependencies and versions are used.
+
+Please follow the instructions [here](./python/README.md#setup) to get set up.
+
+For common tasks that are helpful during development and run in CI, see [here](./python/README.md#common-tasks).
+
+## Roadmap
+
+We use GitHub issues and milestones to track our roadmap. You can view the upcoming milestones [here]([Roadmap Issues](https://aka.ms/autogen-roadmap).
+
+## Versioning
+
+The set of `autogen-*` packages are generally all versioned together. When a change is made to one package, all packages are updated to the same version. This is to ensure that all packages are in sync with each other.
+
+We will update verion numbers according to the following rules:
+
+- Increase minor version (0.X.0) upon breaking changes
+- Increase patch version (0.0.X) upon new features or bug fixes
+
+## Release process
+
+1. Create a PR that updates the version numbers across the codebase ([example](https://github.com/microsoft/autogen/pull/4359))
+    2. The docs CI will fail for the PR, but this is expected and will be resolved in the next step
+2. After merging the PR, create and push a tag that corresponds to the new verion. For example, for `0.4.0.dev7`:
+    - `git tag 0.4.0.dev7 && git push origin 0.4.0.dev7`
+3. Restart the docs CI by finding the failed [job corresponding to the `push` event](https://github.com/microsoft/autogen/actions/workflows/docs.yml) and restarting all jobs
+4. Run [this](https://github.com/microsoft/autogen/actions/workflows/single-python-package.yml) workflow for each of the packages that need to be released and get an approval for the release for it to run
+
+## Triage process
+
+To help ensure the health of the project and community the AutoGen committers have a weekly triage process to ensure that all issues and pull requests are reviewed and addressed in a timely manner. The following documents the responsibilites while on triage duty:
+
+- Issues
+    - Review all new issues - these will be tagged with [`needs-triage`](https://github.com/microsoft/autogen/issues?q=is%3Aissue%20state%3Aopen%20label%3Aneeds-triage).
+    - Apply appropriate labels:
+        - One of `proj-*` labels based on the project the issue is related to
+        - `documentation`: related to documentation
+        - `x-lang`: related to cross language functionality
+        - `dotnet`: related to .NET
+    - Add the issue to a relevant milestone if necessary
+    - If you can resolve the issue or reply to the OP please do.
+    - If you cannot resolve the issue, assign it to the appropriate person.
+    - If awaiting a reply add the tag `awaiting-op-response` (this will be auto removed when the OP replies).
+    - Bonus: there is a backlog of old issues that need to be reviewed - if you have time, review these as well and close or refresh as many as you can.
+- PRs
+    - The UX on GH flags all recently updated PRs. Draft PRs can be ignored, otherwise review all recently updated PRs.
+    - If a PR is ready for review and you can provide one please go ahead. If you cant, please assign someone. You can quickly spin up a codespace with the PR to test it out.
+    - If a PR is needing a reply from the op, please tag it `awaiting-op-response`.
+    - If a PR is approved and passes CI, its ready to merge, please do so.
+    - If it looks like there is a possibly transient CI failure, re-run failed jobs.
+- Discussions
+    - Look for recently updated discussions and reply as needed or find someone on the team to reply.
+- Security
+    - Look through any securty alerts and file issues or dismiss as needed.
 
 ## Becoming a Reviewer
 

README.md

@@ -3,7 +3,9 @@
 <div align="center">
 <img src="https://microsoft.github.io/autogen/0.2/img/ag.svg" alt="AutoGen Logo" width="100">
 
-[![Twitter](https://img.shields.io/twitter/url/https/twitter.com/cloudposse.svg?style=social&label=Follow%20%40pyautogen)](https://twitter.com/pyautogen)
+[![Twitter](https://img.shields.io/twitter/url/https/twitter.com/cloudposse.svg?style=social&label=Follow%20%40pyautogen)](https://twitter.com/pyautogen) [![GitHub Discussions](https://img.shields.io/badge/Discussions-Q%26A-green?logo=github)](https://github.com/microsoft/autogen/discussions) [![0.2 Docs](https://img.shields.io/badge/Docs-0.2-blue)](https://microsoft.github.io/autogen/0.2/) [![0.4 Docs](https://img.shields.io/badge/Docs-0.4-blue)](https://microsoft.github.io/autogen/dev/)
+[![PyPi autogen-core](https://img.shields.io/badge/PyPi-autogen--core-blue?logo=pypi)](https://pypi.org/project/autogen-core/0.4.0.dev7/) [![PyPi autogen-agentchat](https://img.shields.io/badge/PyPi-autogen--agentchat-blue?logo=pypi)](https://pypi.org/project/autogen-agentchat/0.4.0.dev7/) [![PyPi autogen-ext](https://img.shields.io/badge/PyPi-autogen--ext-blue?logo=pypi)](https://pypi.org/project/autogen-ext/0.4.0.dev7/)
+
 
 </div>
 
@@ -13,6 +15,7 @@
 > - (11/14/24) ⚠️ In response to a number of asks to clarify and distinguish between official AutoGen and its forks that created confusion, we issued a [clarification statement](https://github.com/microsoft/autogen/discussions/4217).
 > - (10/13/24) Interested in the standard AutoGen as a prior user? Find it at the actively-maintained *AutoGen* [0.2 branch](https://github.com/microsoft/autogen/tree/0.2) and `autogen-agentchat~=0.2` PyPi package.
 > - (10/02/24) [AutoGen 0.4](https://microsoft.github.io/autogen/dev) is a from-the-ground-up rewrite of AutoGen. Learn more about the history, goals and future at [this blog post](https://microsoft.github.io/autogen/blog). We’re excited to work with the community to gather feedback, refine, and improve the project before we officially release 0.4. This is a big change, so AutoGen 0.2 is still available, maintained, and developed in the [0.2 branch](https://github.com/microsoft/autogen/tree/0.2).
+> - *[Join us for Community Office Hours](https://github.com/microsoft/autogen/discussions/4059)* We will host a weekly open discussion to answer questions, talk about Roadmap, etc.
 
 AutoGen is an open-source framework for building AI agent systems.
 It simplifies the creation of event-driven, distributed, scalable, and resilient agentic applications.

docs/design/04 - Agent and Topic ID Specs.md

@@ -34,7 +34,7 @@ This document describes the structure, constraints, and behavior of Agent IDs an
 
 - Type: `string`
 - Description: Topic type is usually defined by application code to mark the type of messages the topic is for.
-- Constraints: UTF8 and only contain alphanumeric letters (a-z) and (0-9), or underscores (\_). A valid identifier cannot start with a number, or contain any spaces.
+- Constraints: UTF8 and only contain alphanumeric letters (a-z) and (0-9), ':', '=', or underscores (\_). A valid identifier cannot start with a number, or contain any spaces.
 - Examples:
   - `GitHub_Issues`
 

docs/design/readme.md

@@ -0,0 +1,3 @@
+# Docs
+
+You can find the project documentation [here](https://microsoft.github.io/autogen/dev/).

dotnet/src/Microsoft.AutoGen/Agents/AgentBase.cs

@@ -15,27 +15,40 @@ namespace Microsoft.AutoGen.Agents;
 public abstract class AgentBase : IAgentBase, IHandle, IHandle<CloudEvent>
 {
     public static readonly ActivitySource s_source = new("AutoGen.Agent");
-    public AgentId AgentId => _context.AgentId;
+    public AgentId AgentId => _runtime.AgentId;
     private readonly object _lock = new();
     private readonly Dictionary<string, TaskCompletionSource<RpcResponse>> _pendingRequests = [];
 
     private readonly Channel<object> _mailbox = Channel.CreateUnbounded<object>();
-    private readonly IAgentRuntime _context;
+    private readonly IAgentRuntime _runtime;
     public string Route { get; set; } = "base";
 
     protected internal ILogger<AgentBase> _logger;
-    public IAgentRuntime Context => _context;
+    public IAgentRuntime Context => _runtime;
     protected readonly EventTypes EventTypes;
 
     protected AgentBase(
-        IAgentRuntime context,
+        IAgentRuntime runtime,
         EventTypes eventTypes,
         ILogger<AgentBase>? logger = null)
     {
-        _context = context;
-        context.AgentInstance = this;
+        _runtime = runtime;
+        runtime.AgentInstance = this;
         this.EventTypes = eventTypes;
         _logger = logger ?? LoggerFactory.Create(builder => { }).CreateLogger<AgentBase>();
+        var subscriptionRequest = new AddSubscriptionRequest
+        {
+            RequestId = Guid.NewGuid().ToString(),
+            Subscription = new Subscription
+            {
+                TypeSubscription = new TypeSubscription
+                {
+                    AgentType = this.AgentId.Type,
+                    TopicType = this.AgentId.Type + "/" + this.AgentId.Key
+                }
+            }
+        };
+        _runtime.SendMessageAsync(new Message { AddSubscriptionRequest = subscriptionRequest }).AsTask().Wait();
         Completion = Start();
     }
     internal Task Completion { get; }
@@ -131,19 +144,19 @@ public List<string> Subscribe(string topic)
                 }
             }
         };
-        _context.SendMessageAsync(message).AsTask().Wait();
+        _runtime.SendMessageAsync(message).AsTask().Wait();
 
         return new List<string> { topic };
     }
     public async Task StoreAsync(AgentState state, CancellationToken cancellationToken = default)
     {
-        await _context.StoreAsync(state, cancellationToken).ConfigureAwait(false);
+        await _runtime.StoreAsync(state, cancellationToken).ConfigureAwait(false);
         return;
     }
     public async Task<T> ReadAsync<T>(AgentId agentId, CancellationToken cancellationToken = default) where T : IMessage, new()
     {
-        var agentState = await _context.ReadAsync(agentId, cancellationToken).ConfigureAwait(false);
-        return agentState.FromAgentState<T>();
+        var agentstate = await _runtime.ReadAsync(agentId, cancellationToken).ConfigureAwait(false);
+        return agentstate.FromAgentState<T>();
     }
     private void OnResponseCore(RpcResponse response)
     {
@@ -171,7 +184,7 @@ private async Task OnRequestCoreAsync(RpcRequest request, CancellationToken canc
         {
             response = new RpcResponse { Error = ex.Message };
         }
-        await _context.SendResponseAsync(request, response, cancellationToken).ConfigureAwait(false);
+        await _runtime.SendResponseAsync(request, response, cancellationToken).ConfigureAwait(false);
     }
 
     protected async Task<RpcResponse> RequestAsync(AgentId target, string method, Dictionary<string, string> parameters)
@@ -195,7 +208,7 @@ protected async Task<RpcResponse> RequestAsync(AgentId target, string method, Di
         activity?.SetTag("peer.service", target.ToString());
 
         var completion = new TaskCompletionSource<RpcResponse>(TaskCreationOptions.RunContinuationsAsynchronously);
-        _context.Update(request, activity);
+        _runtime.Update(request, activity);
         await this.InvokeWithActivityAsync(
             static async ((AgentBase Agent, RpcRequest Request, TaskCompletionSource<RpcResponse>) state, CancellationToken ct) =>
             {
@@ -206,7 +219,7 @@ static async ((AgentBase Agent, RpcRequest Request, TaskCompletionSource<RpcResp
                     self._pendingRequests[request.RequestId] = completion;
                 }
 
-                await state.Agent._context.SendRequestAsync(state.Agent, state.Request, ct).ConfigureAwait(false);
+                await state.Agent._runtime.SendRequestAsync(state.Agent, state.Request).ConfigureAwait(false);
 
                 await completion.Task.ConfigureAwait(false);
             },
@@ -231,11 +244,11 @@ public async ValueTask PublishEventAsync(CloudEvent item, CancellationToken canc
         activity?.SetTag("peer.service", $"{item.Type}/{item.Source}");
 
         // TODO: fix activity
-        _context.Update(item, activity);
+        _runtime.Update(item, activity);
         await this.InvokeWithActivityAsync(
             static async ((AgentBase Agent, CloudEvent Event) state, CancellationToken ct) =>
             {
-                await state.Agent._context.PublishEventAsync(state.Event, ct).ConfigureAwait(false);
+                await state.Agent._runtime.PublishEventAsync(state.Event).ConfigureAwait(false);
             },
             (this, item),
             activity,

dotnet/src/Microsoft.AutoGen/Agents/Services/AgentWorker.cs

@@ -24,6 +24,8 @@ public class AgentWorker :
     private readonly CancellationTokenSource _shutdownCts;
     private readonly IServiceProvider _serviceProvider;
     private readonly IEnumerable<Tuple<string, Type>> _configuredAgentTypes;
+    private readonly ConcurrentDictionary<string, Subscription> _subscriptionsByAgentType = new();
+    private readonly ConcurrentDictionary<string, List<string>> _subscriptionsByTopic = new();
     private readonly DistributedContextPropagator _distributedContextPropagator;
     private readonly CancellationTokenSource _shutdownCancellationToken = new();
     private Task? _mailboxTask;
@@ -96,11 +98,7 @@ public async Task RunMessagePump()
                 if (message == null) { continue; }
                 switch (message)
                 {
-                    case Message.MessageOneofCase.AddSubscriptionResponse:
-                        break;
-                    case Message.MessageOneofCase.RegisterAgentTypeResponse:
-                        break;
-                    case Message msg:
+                    case Message msg when msg.CloudEvent != null:
 
                         var item = msg.CloudEvent;
 
@@ -110,6 +108,13 @@ public async Task RunMessagePump()
                             agentToInvoke.ReceiveMessage(msg);
                         }
                         break;
+                    case Message msg when msg.AddSubscriptionRequest != null:
+                        await AddSubscriptionRequestAsync(msg.AddSubscriptionRequest).ConfigureAwait(true);
+                        break;
+                    case Message msg when msg.AddSubscriptionResponse != null:
+                        break;
+                    case Message msg when msg.RegisterAgentTypeResponse != null:
+                        break;
                     default:
                         throw new InvalidOperationException($"Unexpected message '{message}'.");
                 }
@@ -123,6 +128,23 @@ public async Task RunMessagePump()
             }
         }
     }
+    private async ValueTask AddSubscriptionRequestAsync(AddSubscriptionRequest subscription)
+    {
+        var topic = subscription.Subscription.TypeSubscription.TopicType;
+        var agentType = subscription.Subscription.TypeSubscription.AgentType;
+        _subscriptionsByAgentType[agentType] = subscription.Subscription;
+        _subscriptionsByTopic.GetOrAdd(topic, _ => []).Add(agentType);
+        Message response = new()
+        {
+            AddSubscriptionResponse = new()
+            {
+                RequestId = subscription.RequestId,
+                Error = "",
+                Success = true
+            }
+        };
+        await _mailbox.Writer.WriteAsync(response).ConfigureAwait(false);
+    }
 
     public async Task StartAsync(CancellationToken cancellationToken)
     {

dotnet/src/Microsoft.AutoGen/Agents/Services/Grpc/GrpcGateway.cs

@@ -153,6 +153,22 @@ private async ValueTask RegisterAgentTypeAsync(GrpcWorkerConnection connection,
                 Success = true
             }
         };
+        // add a default subscription for the agent type
+        //TODO: we should consider having constraints on the namespace or at least migrate all our examples to use well typed namesspaces like com.microsoft.autogen/hello/HelloAgents etc
+        var subscriptionRequest = new AddSubscriptionRequest
+        {
+            RequestId = Guid.NewGuid().ToString(),
+            Subscription = new Subscription
+            {
+                TypeSubscription = new TypeSubscription
+                {
+                    AgentType = msg.Type,
+                    TopicType = msg.Type
+                }
+            }
+        };
+        await AddSubscriptionAsync(connection, subscriptionRequest).ConfigureAwait(true);
+
         await connection.ResponseStream.WriteAsync(response).ConfigureAwait(false);
     }
     private async ValueTask DispatchEventAsync(CloudEvent evt)

dotnet/test/Microsoft.AutoGen.Agents.Tests/AgentBaseTests.cs

@@ -23,6 +23,10 @@ public class AgentBaseTests(InMemoryAgentRuntimeFixture fixture)
     public async Task ItInvokeRightHandlerTestAsync()
     {
         var mockContext = new Mock<IAgentRuntime>();
+        mockContext.SetupGet(x => x.AgentId).Returns(new AgentId("test", "test"));
+        // mock SendMessageAsync
+        mockContext.Setup(x => x.SendMessageAsync(It.IsAny<Message>(), It.IsAny<CancellationToken>()))
+            .Returns(new ValueTask());
         var agent = new TestAgent(mockContext.Object, new EventTypes(TypeRegistry.Empty, [], []), new Logger<AgentBase>(new LoggerFactory()));
 
         await agent.HandleObject("hello world");

protos/agent_worker.proto

@@ -63,9 +63,15 @@ message TypeSubscription {
     string agent_type = 2;
 }
 
+message TypePrefixSubscription {
+    string topic_type_prefix = 1;
+    string agent_type = 2;
+}
+
 message Subscription {
     oneof subscription {
         TypeSubscription typeSubscription = 1;
+        TypePrefixSubscription typePrefixSubscription = 2;
     }
 }
 

python/README.md

@@ -1,8 +1,11 @@
 # AutoGen Python packages
 
-See [`autogen-core`](./packages/autogen-core/) package for main functionality.
+[![0.4 Docs](https://img.shields.io/badge/Docs-0.4-blue)](https://microsoft.github.io/autogen/dev/)
+[![PyPi autogen-core](https://img.shields.io/badge/PyPi-autogen--core-blue?logo=pypi)](https://pypi.org/project/autogen-core/0.4.0.dev7/) [![PyPi autogen-agentchat](https://img.shields.io/badge/PyPi-autogen--agentchat-blue?logo=pypi)](https://pypi.org/project/autogen-agentchat/0.4.0.dev7/) [![PyPi autogen-ext](https://img.shields.io/badge/PyPi-autogen--ext-blue?logo=pypi)](https://pypi.org/project/autogen-ext/0.4.0.dev7/)
 
 
+This directory works as a single `uv` workspace containing all project packages. See [`packages`](./packages/) to discover all project packages.
+
 ## Development
 
 **TL;DR**, run all checks with:

python/packages/autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py

@@ -82,13 +82,13 @@ def _handoff_tool() -> str:
 class AssistantAgent(BaseChatAgent):
     """An agent that provides assistance with tool use.
 
-    It responds with a StopMessage when 'terminate' is detected in the response.
-
     Args:
         name (str): The name of the agent.
         model_client (ChatCompletionClient): The model client to use for inference.
         tools (List[Tool | Callable[..., Any] | Callable[..., Awaitable[Any]]] | None, optional): The tools to register with the agent.
-        handoffs (List[Handoff | str] | None, optional): The handoff configurations for the agent, allowing it to transfer to other agents by responding with a HandoffMessage.
+        handoffs (List[Handoff | str] | None, optional): The handoff configurations for the agent,
+            allowing it to transfer to other agents by responding with a :class:`HandoffMessage`.
+            The transfer is only executed when the team is in :class:`~autogen_agentchat.teams.Swarm`.
             If a handoff is a string, it should represent the target agent's name.
         description (str, optional): The description of the agent.
         system_message (str, optional): The system message for the model.