Writing performance tests for Client libraries

Creating a performance testing project

Location of the project

1. Create the folder structure sdk/<service>/perf/<service-name>.Perf/.
2. Create a new SDK-style project named '<service-name>.Perf.csproj' in the above folder.

Structure and contents of the project

• Contents of the project file should look like

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <ProjectReference Include="$(MSBuildThisFileDirectory)<relative/path/to/the/SDK/project>" />
    <ProjectReference Include="$(MSBuildThisFileDirectory)..\..\..\..\..\common\Perf\Azure.Test.Perf\Azure.Test.Perf.csproj" />
    <ProjectReference Include="$(MSBuildThisFileDirectory)..\..\..\..\core\Azure.Core.TestFramework\src\Azure.Core.TestFramework.csproj" />
  </ItemGroup>
</Project>

The project only needs references to the test framework project (Azure.Core.TestFramework.csproj), the performance infrastructure project (Azure.Test.Perf.csproj) and the client SDK project. NOTE: If the Client SDK source project does not live in the repo, add a PackageReference to the SDK's Nuget package, like <PackageReference Include="<nuget-package-name>" />. This will build an executable with the same name as the name of the project.

• Add a README.md in the folder to help readers understand the target of the performance tests.
• The entry-point for the project executable should be added to Program.cs with the following contents -

// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License.

using System.Reflection;
using Azure.Test.Perf;

await PerfProgram.Main(Assembly.GetEntryAssembly(), args);

This just calls the performance test infrastructure code with the current assembly and any arguments passed to the program. The code in the tests is called as part of execution.

• Create a folder called Infrastructure under the project folder.
• This should contain a class that defines the environment of the test execution. Callers can use this singleton to obtain account names, account keys, connection strings and other such things to make a connection to the Azure service.

internal sealed class PerfTestEnvironment : TestEnvironment
{
        /// <summary>
        /// The shared instance of the <see cref="PerfTestEnvironment"/> to be used during test runs.
        /// </summary>
        public static PerfTestEnvironment Instance { get; } = new PerfTestEnvironment();
}

• Create a folder called Scenarios under the project folder.
• This should contain classes that represent the test scenarios.
• The structure of each test should look like -

    public sealed class <test-name> : PerfTest<SizeOptions>
    {
        /// <summary>
        /// Initializes a new instance of the <see cref="<test-name>"/> class.
        /// </summary>
        /// <param name="options">The set of options to consider for configuring the scenario.</param>
        public \<test-name\>(SizeOptions options) : base(options)
        {
        }

        public override void Dispose(bool disposing)
        {
        }

        public override async Task SetupAsync()
        {
            await base.SetupAsync();
            // Individual test-level setup code.
        }

        public override async Task CleanupAsync()
        {
            // Individual test-level cleanup code.
            await base.CleanupAsync();
        }

        public override void Run(CancellationToken cancellationToken)
        {
            // Scenario execution using synchronous APIs
        }

        public override async Task RunAsync(CancellationToken cancellationToken)
        {
            // Scenario execution using asynchronous APIs
        }
    }

Structure of a sample performance test project

The below code map shows Track 1 and Track 2 performance test projects created for the Azure Storage File Shares SDK. The Track 1 project references Microsoft.Azure.Storage.File.dll while the Track 2 project references Azure.Storage.Files.Shares.dll. Both projects reference Azure.Test.Perf and the individual test classes like UploadFile and DownloadFile inherit from PerfTest<TOptions>.

Build a performance test project

dotnet build --configuration Release --framework <supported-framework> <path/to/project/file>

Run the executable output of a project

dotnet run --configuration Release --framework <supported-framework> --no-build -p <path/to/project/file> [parameters needed for the test]

<supported-framework> can be one of netcoreapp2.1, netcoreapp3.1, net461 or net5.0.

Examples of performance tests

1. Azure Storage File Shares
   1. Track 1 performance test files
      Note that the code for Track 1 of this SDK does not live in the Azure SDK for .NET repo, so -
      1. We add a PackageReference to the latest SDK library in the project file
      2. The Track 1 performance test project is located next to the Track 2 performance test project (each in their own separate directory under perf/).
   2. Track 2 performance test files
2. Azure Search documents tests
   1. Track 2 performance test files