.Net: [Feature Branch] Vector Store Logging

[dmytrostruk]

Motivation and Context

Related: #10596

This PR enhances the vector store functionality by adding builder pattern implementations and logging decorators for key vector data interfaces, along with supporting utilities and DI integration.

Interfaces handled:

• IKeywordHybridSearch<TRecord>
• IVectorizableTextSearch<TRecord>
• IVectorizedSearch<TRecord>
• IVectorStore
• IVectorStoreRecordCollection<TKey, TRecord>

Class types added:

• Builders: implement a pipeline pattern with Use and Build methods.
• Logging Extensions: provide reusable logging methods for tasks and enumerables, handling success, failure and cancellation.
• Logging Decorators: wrap the interfaces to add logging by using LoggingExtensions reusable logging methods.
• Builder Extensions: Add AsBuilder methods to convert service instances into builders.
• Logging Builder Extensions: Add UseLogging methods to integrate logging into builder pipelines.
• Service Collection Extensions: Add Add{T} and AddKeyed{T} methods for DI registration with configurable lifetimes.

Other changes:

• Split Verify class into Verify and KernelVerify in order to be able to use Verify methods in Microsoft.Extensions.VectorData package without a reference to Semantic Kernel specific logic, like ValidPluginName, ValidFunctionName etc.
• Added unit tests for new classes and methods.
• Added usage example in Concepts/Memory folder.

Usage example

Logging with manual registration:

var vectorStore = new InMemoryVectorStore()
    .AsBuilder()
    .UseLogging(this.LoggerFactory)
    .Build();

Logging with DI:

serviceCollection.AddInMemoryVectorStore();

serviceCollection
    .AddVectorStore(s => s.GetRequiredService<InMemoryVectorStore>())
    .UseLogging(this.LoggerFactory);

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[markwallace-microsoft]

Just some documentation updates (I didn't comment everywhere) and the question about adding the same instance twice.

[westey-m]

This is unfortunate. I wonder if we should just change AddInMemoryVectorStore to return the builder, or add an overload like AddInMemoryVectorStoreBuilder(), which returns the builder instead?

[dmytrostruk]

We can do that, but connector developers will need to make sure that they have similar extension method available on their side. I think that method like AddVectorStore is also useful for cases when there are no extension methods available for specific connector/abstraction implementation. There is an overload for this method to simply accept IVectorStore instance.

For connectors that exist on our side at the moment, we can add an additional extension method to operate with builder, but I would probably do that in a separate PR to minimize the number of changes in this one. Please let me know what you think, thanks!

[westey-m]

I think we should do a sample around what this looks like for HybridSearch and VectorizableTextSearch.
Also, I think we may have some issues with cases where someone just created a VectorStoreRecordCollection instance and they want to add logging to it, and then they want to cast it to a VectorizableTextSearch and start searching and they expectation would be that logging would be there for it too...

It's almost like we might need a single decorator class that implements all the interfaces and contains all logging, although that has drawbacks too, in that someone can cast to an interface that the decorator has but the underlying implementation doesn't support.

[dmytrostruk]

I think we should do a sample around what this looks like for HybridSearch and VectorizableTextSearch.

Yes, I'm definitely going to add that. It would be nice if we add an example for HybridSearch first, so then I can re-use that example to show how to enable logging, if possible :)

I think we may have some issues with cases where someone just created a VectorStoreRecordCollection instance and they want to add logging to it, and then they want to cast it to a VectorizableTextSearch and start searching and they expectation would be that logging would be there for it too

Do we think that casting would be a common approach here? Let's imagine we have an ASP.NET Web API application, and we have a service, that performs vector search using text. I would expect that this service will accept IVectorizableTextSearch<MyRecord> in constructor and it's a matter of configuration which instance of that interface is going to be passed in the service and whether it supports logging or not. Same for other interfaces from VectorData.

In case of VectorStoreRecordCollection which implements multiple interfaces, I think that user registers specific vector db collection once (like RedisVectorStoreRecordCollection), and then use it to register IVectorStoreRecordCollection, IVectorizableTextSearch and wrap each registration to support logging. I think it's really flexible approach that user can control where exactly logging or any other decoration is needed, but of course we can provide even more extension methods for easier usage (for example, the extension method that will register IVectorStore, IVectorStoreRecordCollection and IVectorizableTextSearch with already enabled logging and so on).

Please let me know what you think about it.