additional documentation regarding Windows Studio Effect (WSE) and its Driver-Defined Interfaces (DDIs)
This sample will only run fully on a system equipped with a Windows Studio Effects (WSE) camera, which in itself requires:
- a compatible NPU
- the related Windows Studio Effects driver package installed or pulled-in via Windows Update
- a camera opted into WSE by the device manufacturer in its driver. Currently these camera must be front-facing
This folder contains a C# sample named WindowsStudioSample_WinUI which checks if a Windows Studio Effects camera is available on the system. It then proceeds using WinRT APIs to query support and toggle DDIs.
The sample leverages extended camera controls standardized in the OS and defined in Windows SDK such as the following 3 implemented as Windows Studio Effects in version 1:
- Standard Blur, Portrait Blur and Segmentation Mask Metadata : KSPROPERTY_CAMERACONTROL_EXTENDED_BACKGROUNDSEGMENTATION (DDI documentation)
- Eye Contact Standard and Teleprompter: KSPROPERTY_CAMERACONTROL_EXTENDED_EYEGAZECORRECTION (DDI documentation)
- Automatic Framing: KSPROPERTY_CAMERACONTROL_EXTENDED_DIGITALWINDOW (DDI documentation) and KSPROPERTY_CAMERACONTROL_EXTENDED_DIGITALWINDOW_CONFIGCAPS (DDI documentation)
It also taps into newer effects available in version 2 that are exposed using a set of DDIs custom to Windows Studio Effects. Since these are exclusive to Windows Studio Effects and shipped outside the OS, their definition is not part of the Windows SDK and has to be copied into your code base (see DDI documentation):
- Portrait Light (DDI documentation)
- Creative Filters (Animated, Watercolor and Illustrated) (DDI documentation)
The code sample allows to see all the MediaTypes (frame formats) exposed by the camera and change the camera profile it is provisioned with. When initialized with profile for processing effects, WSE limits:
- The number of streams available from a source with effects applied to 1
- Processing of only color stream (i.e. no infrared stream, etc.)
- MediaTypes exposed with:
- Resolutions of at most 2560x1440 and at least 640x360
- Framerate of at most 30fps and at least 15fps
- Subtype in NV12 format only (WSE may take as input other subtypes, but will currently only expose out NV12)
That said as alluded to above, these limits are only imposed when the camera profile used is one of the below known video profile:
- the default (or not specified), also known as Legacy profile
- VideoRecording
- VideoConferencing
With other profiles however, while these limits are not imposed and instead rely on the capabilities defined by the original camera driver, none of the Windows Studio Effects DDIs are supported. This is why for example, the Windows Camera Application might not support any effects when entering photo capture mode but offer a different set of MediaTypes to exercise such as with higher resolution and other subtypes.
In recent release, WSE will define and expose a custom "passthrough" profile that exposes all MediaTypes available (see DDI doc). This is meant to enable application scenarios such as initializing the camera to record video at higher resolution than 1440p or higher framerate than 30fps when the camera supports these capabilities but is limited by WSE in order to apply effects. The below code sample shows how to query for all supported camera profiles, including this one if available in the WSE version on the system.
The app demonstrates the following:
-
Looks for a Windows Studio camera on the system which must conform to the 2 below criteria:
-
Checks if the system exposes the related Windows Studio dev prop key.
private const string DEVPKEY_DeviceInterface_IsWindowsCameraEffectAvailable = "{6EDC630D-C2E3-43B7-B2D1-20525A1AF120} 4"; //... DeviceInformationCollection deviceInfoCollection = await DeviceInformation.FindAllAsync(MediaDevice.GetVideoCaptureSelector(), new List<string>() { DEVPKEY_DeviceInterface_IsWindowsCameraEffectAvailable });
-
Searches for a front facing camera by checking the panel enclosure location of the camera devices.
DeviceInformation selectedDeviceInfo = deviceInfoCollection.FirstOrDefault(x => x.EnclosureLocation.Panel == Windows.Devices.Enumeration.Panel.Front);
-
-
Checks if camera profiles are supported and initializes a MediaCapture instance with either the selected profile or the default.
// List all camera profiles exposed by this device if (MediaCapture.IsVideoProfileSupported(selectedDeviceInfo.Id)) { m_availableCameraProfiles = MediaCapture.FindAllVideoProfiles(selectedDeviceInfo.Id).ToList(); // ... see the lookup table "CameraProfileIdLUT" in KsHelper.cs that correlates profile GUIDs with legible names and see the method IdentifyCompatibleWindowsStudioCamera() that correlates this lookup table with the profiles retrieved with the above call. } // ... // Initialize MediaCapture instance with a specific camera profile await m_mediaCapture.InitializeAsync( new MediaCaptureInitializationSettings() { VideoDeviceId = selectedDeviceInfo.Id, MemoryPreference = MediaCaptureMemoryPreference.Cpu, StreamingCaptureMode = StreamingCaptureMode.Video, SharingMode = MediaCaptureSharingMode.ExclusiveControl, // Either the default (null) or a specific camera profile VideoProfile = (m_availableCameraProfiles == null ? null : m_availableCameraProfiles[UIProfilesAvailable.SelectedIndex]) });
-
Check if the newer set of Windows Studio Effects in version 2 are supported. These new DDIs are defined in a new property set see DDI documentation.
// New Windows Studio Effects custom KsProperties live under this property set public static readonly Guid KSPROPERTYSETID_WindowsStudioEffects = Guid.Parse("1666d655-21A6-4982-9728-52c39E869F90"); // Custom KsProperties exposed in version 2 public enum KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS : uint { KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_SUPPORTED = 0, KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_STAGELIGHT = 1, KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_CREATIVEFILTER = 2, }; // ... // in InitializeCameraAndUI() // query support for new effects in v2 byte[] byteResultPayload = GetExtendedControlPayload( m_mediaCapture.VideoDeviceController, KSPROPERTYSETID_WindowsStudioEffects, (uint)KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS.KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_SUPPORTED); // reinterpret the byte array as an extended property payload KSCAMERA_EXTENDEDPROP_HEADER payloadHeader = FromBytes<KSCAMERA_EXTENDEDPROP_HEADER>(byteResultPayload); int sizeofHeader = Marshal.SizeOf<KSCAMERA_EXTENDEDPROP_HEADER>(); int sizeofKsProperty = Marshal.SizeOf<KsProperty>(); ; int supportedControls = ((int)payloadHeader.Size - sizeofHeader) / sizeofKsProperty; for (int i = 0; i < supportedControls; i++) { KsProperty payloadKsProperty = FromBytes<KsProperty>(byteResultPayload, sizeofHeader + i * sizeofKsProperty); if (new Guid(payloadKsProperty.Set) == KSPROPERTYSETID_WindowsStudioEffects) { // if we support StageLight (also known as PortraitLight) if (payloadKsProperty.Id == (uint)KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS.KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_STAGELIGHT) { RefreshStageLightUI(); } // if we support CreativeFilter else if (payloadKsProperty.Id == (uint)KSPROPERTY_CAMERACONTROL_WINDOWS_EFFECTS.KSPROPERTY_CAMERACONTROL_WINDOWSSTUDIO_CREATIVEFILTER) { RefreshCreativeFilterUI(); } } } // ...
-
Expose a button that launches the Windows Settings page for the camera where the user can also toggle effects concurrently while the sample app is running.
// launch Windows Settings page for the camera identified by the specified Id var uri = new Uri($"ms-settings:camera?cameraId={Uri.EscapeDataString(m_mediaCapture.MediaCaptureSettings.VideoDeviceId)}"); var fireAndForget = Windows.System.Launcher.LaunchUriAsync(uri); // launch the Windows Studio quick setting panel using URI if it is supported var uri = new Uri($"ms-controlcenter:studioeffects"); var supportStatus = await Windows.System.Launcher.QueryUriSupportAsync(uri, Windows.System.LaunchQuerySupportType.Uri); if (supportStatus == Windows.System.LaunchQuerySupportStatus.Available) { var fireAndForget = Windows.System.Launcher.LaunchUriAsync(uri); }
-
Proceeds to then stream from this camera and interact with Windows Studio effects via Extended Property DDI using
VideoDeviceController.GetDevicePropertyByExtendedId()andVideoDeviceController.SetDevicePropertyByExtendedId():-
Send a GET command and deserialize the retrieved payload, see implementation and usage of
byte[] KsHelper.GetExtendedControlPayload( VideoDeviceController controller, Guid propertySet, uint controlId)
and
T KsHelper.FromBytes<T>(byte[] bytes, int startIndex = 0)
such as in this example:
// send a GET call for the eye gaze DDI and retrieve the result payload byte[] byteResultPayload = KsHelper.GetExtendedControlPayload( m_mediaCapture.VideoDeviceController, KsHelper.KSPROPERTYSETID_ExtendedCameraControl, (uint)KsHelper.ExtendedControlKind.KSPROPERTY_CAMERACONTROL_EXTENDED_EYEGAZECORRECTION); // reinterpret the byte array as an extended property payload KsHelper.KsBasicCameraExtendedPropPayload payload = KsHelper.FromBytes<KsHelper.KsBasicCameraExtendedPropPayload>(byteResultPayload);
- Send a SET command using a serialized payload containing flags value to toggle effects, see implementation and usage of
void KsHelper.SetExtendedControlFlags( VideoDeviceController controller, Guid propertySet, uint controlId, ulong flags)
and
byte[] KsHelper.ToBytes<T>(T item)
such as in this example:
// set the flags value for the corresponding extended control KsHelper.SetExtendedControlFlags( m_mediaCapture.VideoDeviceController, KsHelper.KSPROPERTYSETID_ExtendedCameraControl, (uint)KsHelper.ExtendedControlKind.KSPROPERTY_CAMERACONTROL_EXTENDED_EYEGAZECORRECTION, flagToSet);
-
The .sln solution file is also pulling a dependency on the existing WinRT Component named "ControlMonitorHelperWinRT" that wraps and projects to WinRT the IMFCameraControlMonitor native APIs so that we can refresh the app UI whenever an external application such as Windows Settings changes concurrently the current value of one of the DDI that drives Windows Studio effects.
This sample is built using Visual Studio 2022 and requires Windows SDK version 22621 at the very least.
The easiest way to use these samples without using Git is to download the zip file containing the current version (using the following link or by clicking the "Download ZIP" button on the repo page). You can then unzip the entire archive and use the samples in Visual Studio 2022.
Notes:
- Before you unzip the archive, right-click it, select Properties, and then select Unblock.
- Be sure to unzip the entire archive, and not just individual samples. The samples all depend on the SharedContent folder in the archive.
- In Visual Studio 2022, the platform target defaults to ARM64, so be sure to change that to x64 if you want to test on a non-ARM64 device.
Reminder: If you unzip individual samples, they will not build due to references to other portions of the ZIP file that were not unzipped. You must unzip the entire archive if you intend to build the samples.
For more info about the programming models, platforms, languages, and APIs demonstrated in these samples, please refer to the guidance, tutorials, and reference topics provided in the Windows 10 documentation available in the Windows Developer Center. These samples are provided as-is in order to indicate or demonstrate the functionality of the programming models and feature APIs for Windows.
This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.
When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.