Skip to content

Latest commit

 

History

History

WindowsStudio

Windows Studio Effects camera sample application - C# .Net WinUI & WinRT

additional documentation regarding Windows Studio Effect (WSE) and its Driver-Defined Interfaces (DDIs)

This sample will only run fully on a system equipped with a Windows Studio Effects (WSE) camera, which in itself requires:

  1. a compatible NPU
  2. the related Windows Studio Effects driver package installed or pulled-in via Windows Update
  3. a camera opted into WSE by the device manufacturer in its driver. Currently these camera must be front-facing

This folder contains a C# sample named WindowsStudioSample_WinUI which checks if a Windows Studio Effects camera is available on the system. It then proceeds using WinRT APIs to query support and toggle DDIs.

WSE v1 effects

The sample leverages extended camera controls standardized in the OS and defined in Windows SDK such as the following 3 implemented as Windows Studio Effects in version 1:

  • Standard Blur, Portrait Blur and Segmentation Mask Metadata : KSPROPERTY_CAMERACONTROL_EXTENDED_BACKGROUNDSEGMENTATION (DDI documentation)
  • Eye Contact Standard and Teleprompter: KSPROPERTY_CAMERACONTROL_EXTENDED_EYEGAZECORRECTION (DDI documentation)
  • Automatic Framing: KSPROPERTY_CAMERACONTROL_EXTENDED_DIGITALWINDOW (DDI documentation) and KSPROPERTY_CAMERACONTROL_EXTENDED_DIGITALWINDOW_CONFIGCAPS (DDI documentation)

WSE v2 effects

It also taps into newer effects available in version 2 that are exposed using a set of DDIs custom to Windows Studio Effects. Since these are exclusive to Windows Studio Effects and shipped outside the OS, their definition is not part of the Windows SDK and has to be copied into your code base (see DDI documentation):

WSE limits frame formats and profiles

The code sample allows to see all the MediaTypes (frame formats) exposed by the camera and change the camera profile it is provisioned with. When initialized with profile for processing effects, WSE limits:

  • The number of streams available from a source with effects applied to 1
  • Processing of only color stream (i.e. no infrared stream, etc.)
  • MediaTypes exposed with:
    • Resolutions of at most 2560x1440 and at least 640x360
    • Framerate of at most 30fps and at least 15fps
    • Subtype in NV12 format only (WSE may take as input other subtypes, but will currently only expose out NV12)

That said as alluded to above, these limits are only imposed when the camera profile used is one of the below known video profile:

  • the default (or not specified), also known as Legacy profile
  • VideoRecording
  • VideoConferencing

With other profiles however, while these limits are not imposed and instead rely on the capabilities defined by the original camera driver, none of the Windows Studio Effects DDIs are supported. This is why for example, the Windows Camera Application might not support any effects when entering photo capture mode but offer a different set of MediaTypes to exercise such as with higher resolution and other subtypes.

In recent release, WSE will define and expose a custom "passthrough" profile that exposes all MediaTypes available (see DDI doc). This is meant to enable application scenarios such as initializing the camera to record video at higher resolution than 1440p or higher framerate than 30fps when the camera supports these capabilities but is limited by WSE in order to apply effects. The below code sample shows how to query for all supported camera profiles, including this one if available in the WSE version on the system.

Code walkthrough

The app demonstrates the following:

  1. Looks for a Windows Studio camera on the system which must conform to the 2 below criteria:

    1. Checks if the system exposes the related Windows Studio dev prop key.

      private const string DEVPKEY_DeviceInterface_IsWindowsCameraEffectAvailable = "{6EDC630D-C2E3-43B7-B2D1-20525A1AF120} 4";
      //...
      DeviceInformationCollection deviceInfoCollection = await DeviceInformation.FindAllAsync(MediaDevice.GetVideoCaptureSelector(), new List<string>() { DEVPKEY_DeviceInterface_IsWindowsCameraEffectAvailable });
    2. Searches for a front facing camera by checking the panel enclosure location of the camera devices.

      DeviceInformation selectedDeviceInfo = deviceInfoCollection.FirstOrDefault(x => x.EnclosureLocation.Panel == Windows.Devices.Enumeration.Panel.Front);
  2. Checks if camera profiles are supported and initializes a MediaCapture instance with either the selected profile or the default.

    // List all camera profiles exposed by this device
    if (MediaCapture.IsVideoProfileSupported(selectedDeviceInfo.Id))
    {
        m_availableCameraProfiles = MediaCapture.FindAllVideoProfiles(selectedDeviceInfo.Id).ToList();
        // ... see the lookup table "CameraProfileIdLUT" in KsHelper.cs that correlates profile GUIDs with legible names and see the method IdentifyCompatibleWindowsStudioCamera() that correlates this lookup table with the profiles retrieved with the above call.
    }
    
    // ...
    
    // Initialize MediaCapture instance with a specific camera profile
     await m_mediaCapture.InitializeAsync(
     new MediaCaptureInitializationSettings()
     {
         VideoDeviceId = selectedDeviceInfo.Id,
         MemoryPreference = MediaCaptureMemoryPreference.Cpu,
         StreamingCaptureMode = StreamingCaptureMode.Video,
         SharingMode = MediaCaptureSharingMode.ExclusiveControl,
         // Either the default (null) or a specific camera profile
         VideoProfile = (m_availableCameraProfiles == null ? null : m_availableCameraProfiles[UIProfilesAvailable.SelectedIndex])
     });
  3. Check if the newer set of Windows Studio Effects in version 2 are supported. These new DDIs are defined in a new property set see DDI documentation.

    // New Windows Studio Effects custom KsProperties live under this property set
    public static readonly Guid KSPROPERTYSETID_WindowsStudioEffects = 
        Guid.Parse("1666d655-21A6-4982-9728-52c39E869F90");
    
    // Custom KsProperties exposed in version 2
    public enum KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS : uint
    {
        KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_SUPPORTED = 0,
        KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_STAGELIGHT = 1,
        KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_CREATIVEFILTER = 2,
    };
    
    // ...
    // in InitializeCameraAndUI()
    // query support for new effects in v2
    byte[] byteResultPayload = GetExtendedControlPayload(
                                m_mediaCapture.VideoDeviceController,
                                KSPROPERTYSETID_WindowsStudioEffects,
                                (uint)KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS.KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_SUPPORTED);
    
    // reinterpret the byte array as an extended property payload
    KSCAMERA_EXTENDEDPROP_HEADER payloadHeader = FromBytes<KSCAMERA_EXTENDEDPROP_HEADER>(byteResultPayload);
    int sizeofHeader = Marshal.SizeOf<KSCAMERA_EXTENDEDPROP_HEADER>();
    int sizeofKsProperty = Marshal.SizeOf<KsProperty>(); ;
    int supportedControls = ((int)payloadHeader.Size - sizeofHeader) / sizeofKsProperty;
    
    for (int i = 0; i < supportedControls; i++)
    {
        KsProperty payloadKsProperty = FromBytes<KsProperty>(byteResultPayload, sizeofHeader + i * sizeofKsProperty);
    
        if (new Guid(payloadKsProperty.Set) == KSPROPERTYSETID_WindowsStudioEffects)
        {
            // if we support StageLight (also known as PortraitLight)
            if (payloadKsProperty.Id == (uint)KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS.KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_STAGELIGHT)
            {
                RefreshStageLightUI();
            }
            // if we support CreativeFilter
            else if (payloadKsProperty.Id == (uint)KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS.KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_CREATIVEFILTER)
            {
                RefreshCreativeFilterUI();
            }
        }
    }
    // ...
  4. Expose a button that launches the Windows Settings page for the camera where the user can also toggle effects concurrently while the sample app is running.

    // launch Windows Settings page for the camera identified by the specified Id
    var uri = new Uri($"ms-settings:camera?cameraId={Uri.EscapeDataString(m_mediaCapture.MediaCaptureSettings.VideoDeviceId)}");
    var fireAndForget = Windows.System.Launcher.LaunchUriAsync(uri);
    
    // launch the Windows Studio quick setting panel using URI if it is supported
    var uri = new Uri($"ms-controlcenter:studioeffects");
    var supportStatus = await Windows.System.Launcher.QueryUriSupportAsync(uri, Windows.System.LaunchQuerySupportType.Uri);
    if (supportStatus == Windows.System.LaunchQuerySupportStatus.Available)
    {
        var fireAndForget = Windows.System.Launcher.LaunchUriAsync(uri);
    }
  5. Proceeds to then stream from this camera and interact with Windows Studio effects via Extended Property DDI using VideoDeviceController.GetDevicePropertyByExtendedId() and VideoDeviceController.SetDevicePropertyByExtendedId() :

    1. Send a GET command and deserialize the retrieved payload, see implementation and usage of

      byte[] KsHelper.GetExtendedControlPayload(
          VideoDeviceController controller,
          Guid propertySet,
          uint controlId)

      and

      T KsHelper.FromBytes<T>(byte[] bytes, int startIndex = 0) 

      such as in this example:

      // send a GET call for the eye gaze DDI and retrieve the result payload
      byte[] byteResultPayload = KsHelper.GetExtendedControlPayload(
        m_mediaCapture.VideoDeviceController,
        KsHelper.KSPROPERTYSETID_ExtendedCameraControl,
        (uint)KsHelper.ExtendedControlKind.KSPROPERTY_CAMERACONTROL_EXTENDED_EYEGAZECORRECTION);
      
      // reinterpret the byte array as an extended property payload
      KsHelper.KsBasicCameraExtendedPropPayload payload = KsHelper.FromBytes<KsHelper.KsBasicCameraExtendedPropPayload>(byteResultPayload); 
      1. Send a SET command using a serialized payload containing flags value to toggle effects, see implementation and usage of
      void KsHelper.SetExtendedControlFlags(
          VideoDeviceController controller,
          Guid propertySet,
          uint controlId,
          ulong flags)

      and

      byte[] KsHelper.ToBytes<T>(T item) 

      such as in this example:

      // set the flags value for the corresponding extended control
      KsHelper.SetExtendedControlFlags(
        m_mediaCapture.VideoDeviceController,
        KsHelper.KSPROPERTYSETID_ExtendedCameraControl,
        (uint)KsHelper.ExtendedControlKind.KSPROPERTY_CAMERACONTROL_EXTENDED_EYEGAZECORRECTION,
        flagToSet);

The .sln solution file is also pulling a dependency on the existing WinRT Component named "ControlMonitorHelperWinRT" that wraps and projects to WinRT the IMFCameraControlMonitor native APIs so that we can refresh the app UI whenever an external application such as Windows Settings changes concurrently the current value of one of the DDI that drives Windows Studio effects.

Requirements

This sample is built using Visual Studio 2022 and requires Windows SDK version 22621 at the very least.

Using the samples

The easiest way to use these samples without using Git is to download the zip file containing the current version (using the following link or by clicking the "Download ZIP" button on the repo page). You can then unzip the entire archive and use the samples in Visual Studio 2022.

Download the samples ZIP

Notes:

  • Before you unzip the archive, right-click it, select Properties, and then select Unblock.
  • Be sure to unzip the entire archive, and not just individual samples. The samples all depend on the SharedContent folder in the archive.
  • In Visual Studio 2022, the platform target defaults to ARM64, so be sure to change that to x64 if you want to test on a non-ARM64 device.

Reminder: If you unzip individual samples, they will not build due to references to other portions of the ZIP file that were not unzipped. You must unzip the entire archive if you intend to build the samples.

For more info about the programming models, platforms, languages, and APIs demonstrated in these samples, please refer to the guidance, tutorials, and reference topics provided in the Windows 10 documentation available in the Windows Developer Center. These samples are provided as-is in order to indicate or demonstrate the functionality of the programming models and feature APIs for Windows.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.