diff --git a/docs/host-guest-design.md b/docs/host-guest-design.md new file mode 100644 index 000000000..2a1a6fd15 --- /dev/null +++ b/docs/host-guest-design.md @@ -0,0 +1,361 @@ +# 1DS C/C++ Host-Guest API detailed design + +## Preface + +There are scenarios where hybrid applications (C#, C/C++, JavaScript) may need to propagate telemetry +to Common Shared Telemetry library (1DS). + +In those scenarios all parts of application need to discover the SDK instance. For example: + +- App Core C++ SDK initializes Telemetry Stack as Telemetry `Host`, letting other delay-loaded components +of application to reuse its telemetry stack. +- App Core C++ SDK could use `LogManager::SetContext(...)` and Semantic Context C++ APIs to populate some +common low-level knowledge, accessible only from C++ layer, ex. `ext.user.localId`. +- Extension SDKs (Plugins) could act as Telemetry "Guest". These would discover the existing telemetry +stack and could use `evt_set_logmanager_context` C API to append their context variables to shared +telemetry context. +- App Common C# layer could also act as Telemetry "Guest". It latches itself to the existing instance +of native telemetry stack, appending its own variables available from application store, ex. `ext.app.name`, +`ext.app.id`. Since packaged application also knows from its package what platform it is designed for, +it could populate `ext.os.name`. For example, set it to `Meta Quest 2` / `Meta Quest Pro` intead of +generic `Android` moniker. Presently 1DS C/C++ SDK itself cannot auto-discover those intricacies. +- App layer could also propagate additional compile-time constants, such as `app.build.env`, git tag +of app, etc. + +Consolidated Shared Telemetry Context contains a set of fields populated by various elements of +a system stack. From top C# / JS layers to the bottom C++ layer, spanning across extension plugins / +libraries loaded in app context. Those libraries could consume telemetry stack via C API. + +Shared context properties could get stamped on all events emitted by a product, irrespective of whether +these events originated from a high-level app written in C#, or a lower-level extension SDK written in C. + +## Ultimate user guide to 1DS C/C++ SDK Host-Guest API + +`Host`-`Guest` API has been designed for the following scenarios: + +### Sharing telemetry stack + +Main component - `Host` loads its accessory component (or SDK) - `Guest`. Both components use the +same shared dynamically loadable 1DS C++ SDK binary, e.g. `ClientTelemetry.dll`, `libmaesdk.so`, or +`libcat.so` - whatever is the "distro" used to package 1DS C++ SDK. + +`Guest` could dynamically discover and load 1DS C++ via C API. It could latch to currently initialized +instance of its `Host` component `LogManager`. `Guest` could also create its own totally separate sandboxed +instance of a `Guest` `LogManager`. `GetProcAddress` is supported on Windows. `dlsym` supported on Linux +and Android. Lazy-binding (automagic binding / auto-discovery of `Host` telemetry stack) is supported +on Linux, Mac and Android. `P/Invoke` for C# is also fully supported cross-platform for .NET and Mono. +1DS C API provides one unified struct layout, with packed structs approach that works on modern +Intel-x64 and ARM64 OS. One single C# assembly could interoperate with 1DS C++ SDK in a uniform way. + +### SDK-in-SDK scenario + +Native code SDK could load another extension/accessory SDK. Both parts must share the same telemetry stack. +Extension SDK could be written in C or C++. Main SDK is treated as `Host`, additional SDKs are treated +as `Guests`. `Host` could also facilitate the ingection of Diagnostic Data Viewer plugin, in order to +satisfy our Privacy and Compliance oblogations. Additional `Guest` modules could enrich the main `Host` +shared telemetry context with their properties. + +### Telemetry flows and `Telemetry Data Isolation` scenarios + +In some cases many different application modules (plugins) get loaded into the main app address space. +For example, `Azure Calling SDK` or `Microsoft Information Protection SDK` running in another product. +These plugin(s) may not necessarily need to share their telemetry flows with the main app. In that case +the modules must operate within their own trust and data collection boundary. Data uploads need to be +controlled separately, with Required Service Data flowing to Azure location of a service resource; +while Optional Customer Data may need to flow to its own regional EUDB-compliant collector. + +`Host`-`Guest` API solves this challenge by providing partitioning for different components using the +same telemetry SDK. If necessary, different modules telemetry collection processes run totally isolated +from one another. + +## Common Considerations + +`HostGuestTests.cpp` module in functional test contains several usage examples. + +See detailed explanation of configuration options and examples below. + +### Dissecting Host configuration + +`Host` configuration example: + +```json +{ + "cacheFilePath": "MyOfflineStorage.db", + "config": { + "host": "C-API-Host", + "scope": "*" + }, + "stats": { + "interval": 0 + }, + "name": "C-API-Host", + "version": "1.0.0", + "primaryToken": "ffffffffffffffffffffffffffffffff-ffffffff-ffff-ffff-ffff-ffffffffffff-0001" + "maxTeardownUploadTimeInSec": 5, + "hostMode": true, + "minimumTraceLevel": 0, + "sdkmode": 0 +} +``` + +`Host` could specify the two matching parameters: + +- `"host": "C-API-Host"` +- `"name": "C-API-Host"` + +If host parameter matches the name parameter, then it assumed that the `Host` module acts as the one and +only `Host` in the application. It will be creating its own data collection sandbox. It will not latch to +any other `Host` modules that could be running in the same app. Multiple `Host` modules supported. + +In some scenarios a `Host` would prefer to latch (join) an existing telemetry session. This is especially +helpful if multiple Hosts need to share one data collection domain and their startup/load order is not +clearly defined. In that case, a session initialized by first `Host` could be shared with other Hosts. +Data collection domain performs ref-counting of instances latched to it. + +Hosts could specify `"host": "*"` to attach to existing data collection session. If first `Host` leaves +(unloads or closes its handle), remaining entities in that session continue operating until the last +`Host` leaves the data collection domain. + +Both Guests and Hosts may utilize the `scope` parameter that controls if these would be sharing the +same common telemetry context shared within a sandbox: + +- `scope="*"` - SHARED or ALL, means that a component will contribute its context to shared context. +- `scope="-"` - NONE or RESTRICTED, means that a component will not contribute its context, and will +not receive any values from the shared context. This mechanism allows to satisfy data collection +and privacy obligations. Each entity acts within their own data collection and compliance boundary +without sharing any of their telemetry contexts with other modules in the process. + +## Dissecting Guest configuration + +Guest configuration example: + +```json +{ + "cacheFilePath": "MyOfflineStorage.db", + "config": { + "host": "*", + "scope": "*" + }, + "stats": { + "interval": 0 + }, + "name": "C-API-Guest", + "version": "1.0.0", + "primaryToken": "ffffffffffffffffffffffffffffffff-ffffffff-ffff-ffff-ffff-ffffffffffff-0002", + "maxTeardownUploadTimeInSec": 5, + "hostMode": false, + "minimumTraceLevel": 0, + "sdkmode": 0 +} +``` + +Guest entity: + +- specifies its own data storage file. This is helpful if Guest starts up prior to any other `Host`. +- `"host": "*"` parameter allows the Guest to latch to any host. +- `"scope": "*"` parameter allows the Guest to contribute and share its telemetry context with other modules (`Host` and Guests). +- Hosts and Guests to present themselves with unique name, ex. `"name": "C-API-Guest"` and unique version, ex. `1.0.0`. +- Guest must specify `"hostMode": false`. That is how SDK knows that a Guest is expected to join another `Host`'s sandbox. +- Guest may omit the scope parameter. In this case the Guest cannot capture the main `Host` telemetry contexts. +This is done intentionally as a security feature. Main application developers may ask their plugin developers +to never capture any telemetry contexts populated by the main application. For example, in some cases - main +application `ext.user.localId` or session `TraceId` cannot be shared with extension. There is no explicit +permission model. Since most components are expected to be assembled and tested by product development teams, +the team should audit the usage of Guest scope parameter by the plugins it is loading. There is runtime code +isolation provided by this mechanism. It is based on trust that all loadable modules exercise their due +diligence while setting up their telemetry configuration. + +### End-to-end example + +`Host` code: + +```cpp + // Host JSON configuration: + const char* hostConfig = JSON_CONFIG( + { + "cacheFilePath" : "/some/path/MyOfflineStorage.db", + "config" : { + "host" : "C-API-Host", + "scope" : "*" + }, + "name" : "C-API-Host", + "version" : "1.0.0", + "primaryToken" : "ffffffffffffffffffffffffffffffff-ffffffff-ffff-ffff-ffff-ffffffffffff-0001", + "hostMode" : true + }); + + // Host initializes in Host mode, waiting for Guest()s to register. + evt_handle_t hostHandle = evt_open(hostConfig); + + // evt_prop[] array that contains common context properties. + // Contexts between Hosts and Guests could be merged into one shared context. + evt_prop hostContext[] = TELEMETRY_EVENT( + _STR("ext.device.localId", "a:4318b22fbc11ca8f"), + _STR("ext.device.make", "Microsoft"), + _STR("ext.device.model", "Clippy"), + _STR("ext.os.name", "MS-DOS"), + _STR("ext.os.ver", "2100") + ); + + // Host appends common context properties at top-level LogManager. + // These variables will be shared with Guest(s). + evt_set_logmanager_context(hostHandle, hostContext); + + evt_prop hostEvent[] = TELEMETRY_EVENT( + // Part A/B fields + _STR(COMMONFIELDS_EVENT_NAME, "Event.Host"), + _INT(COMMONFIELDS_EVENT_PRIORITY, static_cast(EventPriority_Immediate)), + _INT(COMMONFIELDS_EVENT_LATENCY, static_cast(EventLatency_Max)), + _INT(COMMONFIELDS_EVENT_LEVEL, DIAG_LEVEL_REQUIRED), + _STR("strKey", "value1"), + _INT("intKey", 12345), + _DBL("dblKey", 3.14), + _BOOL("boolKey", true), + _GUID("guidKey", "{01020304-0506-0708-090a-0b0c0d0e0f00}"); + evt_log(hostHandle, hostEvent); + +``` + +In above example: + +- `Host` performs initialization. +- populates its top-level LogManager semantic context with known values. + +For example, the `Host` C++ layer could use native API to access the lower-level platform-specific +Device Id, Device Make, Model. `Host` may emit a telemetry event that would combine the event data +with its context data. + +Guest code: + +```cpp + + // Guest JSON configuration: + const char* guestConfig = JSON_CONFIG( + { + "config" : { + "host" : "*", + "scope" : "*" + }, + "name" : "C-API-Guest", + "version" : "1.0.0", + "primaryToken" : "ffffffffffffffffffffffffffffffff-ffffffff-ffff-ffff-ffff-ffffffffffff-0002", + "hostMode" : false + }); + + // Guest initializes in Guest mode and latches to previously running Host. + auto guestHandle = evt_open(guestConfig); + + // evt_prop[] array that contains context properties: + evt_prop guestContext[] = TELEMETRY_EVENT( + _STR("ext.app.id", "com.Microsoft.Clippy"), + _STR("ext.app.ver", "1.0.0"), + _STR("ext.app.locale", "en-US"), + _STR("ext.net.cost", "Unmetered"), + _STR("ext.net.type", "QuantumLeap")); + + // Guest could append some of its common context properties on top of shared context: + evt_set_logmanager_context(guestHandle, guestContext); + + evt_prop guestEvent[] = TELEMETRY_EVENT( + _STR("name", "Event.Guest"), + _INT(COMMONFIELDS_EVENT_PRIORITY, static_cast(EventPriority_Immediate)), + _INT(COMMONFIELDS_EVENT_LATENCY, static_cast(EventLatency_Max)), + _INT(COMMONFIELDS_EVENT_LEVEL, DIAG_LEVEL_REQUIRED), + _STR("strKey", "value2"), + _INT("intKey", 67890), + _DBL("dblKey", 3.14), + _BOOL("boolKey", false), + _GUID("guidKey", "{01020304-0506-0708-090a-0b0c0d0e0f01}"); + evt_log(guestHandle, guestEvent); + +``` + +In above example: + +- Guest registers and shares the scope with the `Host`. +- Guest entity could operate on a totally different abstraction layer, e.g. higher-level Unity C# or Android Java app. +It could obtain certain system parameters that are easily accessible only by the higher-level app. Such as, app store +application name and version. It could be a layer that performs User Authentication and Authorization, subsequently +sharing the User Identity as part of common telemetry context shared with lower-level code across the language boundary. + +Reference design showing how to use 1DS C API from .NET Core, Mono and Unity applications is provided. + +Above examples generate the following event payloads. + +`Host` Event payload in Common Schema notation: + +```json +{ + "data": { + "boolKey": true, + "dblKey": 3.14, + "guidKey": [[4,3,2,1,6,5,8,7,9,10,11,12,13,14,15,0]], + "intKey": 12345, + "strKey": "value1" + }, + "ext": { + "device": { + "localId": "a:4318b22fbc11ca8f", + "make": "Microsoft", + "model": "Clippy" + }, + "os": { + "name": "MS-DOS", + "ver": "2100" + } + }, + "iKey": "o:7c8b1796cbc44bd5a03803c01c2b9d61", + "name": "Event.Host", + "time": 1680074712000, + "ver": "3.0" +} +``` + +Guest Event payload in Common Schema notation. Note that Guest event emitted after `Host` initialization +contains the superset of all consolidated common properties: + +```json +{ + "data": { + "boolKey": true, + "dblKey": 3.14, + "guidKey": [[4,3,2,1,6,5,8,7,9,10,11,12,13,14,15,0]], + "intKey": 12345, + "strKey": "value1" + }, + "ext": { + "app": { + "id": "com.Microsoft.Clippy", + "locale": "en-US", + "name": "com.Microsoft.Clippy", + "ver": "1.0.0" + }, + "device": { + "localId": "a:4318b22fbc11ca8f", + "make": "Microsoft", + "model": "Clippy" + }, + "net": { + "cost": "Unmetered", + "provider": "", + "type": "QuantumLeap" + }, + "os": { + "name": "MS-DOS", + "ver": "2100" + } + }, + "iKey": "o:7c8b1796cbc44bd5a03803c01c2b9d61", + "name": "Event.Guest", + "time": 1680074712000, + "ver": "3.0" +} +``` + +`Host`-`Guest` approach allows us to share one common telemetry diagnostic context across the language +boundaries in a hybrid application designed with different programming languages: C/C++, C#, and +JavaScript. Other programming languages may easily leverage Foreign Function Interface and 1DS C API. + +`Host`-`Guest` interface plays a central role in aggregation of different module contexts into one +common shared telemetry context of application. C++ example is available in `SampleCppLogManagers` +project. diff --git a/examples/c/SampleC/deploy-dll.cmd b/examples/c/SampleC/deploy-dll.cmd index 8ba37cacd..8d42b51fc 100644 --- a/examples/c/SampleC/deploy-dll.cmd +++ b/examples/c/SampleC/deploy-dll.cmd @@ -2,15 +2,9 @@ set PROJECT_DIR=%~dp0 @mkdir %PROJECT_DIR%\include -copy %PROJECT_DIR%..\..\..\lib\include\public\mat.h %PROJECT_DIR%\include -copy %PROJECT_DIR%..\..\..\lib\include\public\Version.h %PROJECT_DIR%\include -copy %PROJECT_DIR%..\..\..\lib\include\public\ctmacros.hpp %PROJECT_DIR%\include +copy /Y %PROJECT_DIR%..\..\..\lib\include\public\mat.h %PROJECT_DIR%\include +copy /Y %PROJECT_DIR%..\..\..\lib\include\public\ctmacros.hpp %PROJECT_DIR%\include @mkdir %PROJECT_DIR%\lib\%1\%2 -copy %PROJECT_DIR%..\..\..\Solutions\out\%1\%2\win32-dll\*.lib %PROJECT_DIR%\lib\%1\%2 - -@mkdir -p %PROJECT_DIR%\%1\%2 -copy %PROJECT_DIR%..\..\ -copy %PROJECT_DIR%..\..\..\Solutions\out\%1\%2\win32-dll\*.* %PROJECT_DIR%\lib\%1\%2 -copy %PROJECT_DIR%..\..\..\Solutions\out\%1\%2\win32-dll\*.* %3 +robocopy %PROJECT_DIR%..\..\..\Solutions\out\%1\%2\win32-dll %3 *.dll /S exit /b 0 diff --git a/examples/cpp/SampleCppLogManagers/SampleCppLogManagers.vcxproj b/examples/cpp/SampleCppLogManagers/SampleCppLogManagers.vcxproj index 7f6b47434..a8cc24be1 100644 --- a/examples/cpp/SampleCppLogManagers/SampleCppLogManagers.vcxproj +++ b/examples/cpp/SampleCppLogManagers/SampleCppLogManagers.vcxproj @@ -1,196 +1,196 @@ - - - - - - Debug - Win32 - - - Release - Win32 - - - Debug - x64 - - - Release - x64 - - - - 15.0 - {77053F92-F003-4D1C-A489-1DEB7CFEA4EC} - Win32Proj - SampleCppLogManagers - - - - Application - true - Unicode - - - Application - false - true - Unicode - - - Application - true - Unicode - - - Application - false - true - Unicode - - - - - - - - - - - - - - - - - - - - true - $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public;$(SolutionDir)\..\lib\pal\ - $(ProjectDir) - $(Configuration)\ - $(LibraryPath) - - - true - $(ProjectDir) - $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public;$(SolutionDir)\..\lib\pal\ - $(LibraryPath) - - - false - $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public - - - false - $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public - - - - NotUsing - Level3 - Disabled - true - WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - pch.h - - - Console - true - - - $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) - - - Deploy DLLs - - - - - NotUsing - Level3 - Disabled - true - _DEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - pch.h - - - Console - true - - - $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) - - - Deploy DLLs - - - - - NotUsing - Level3 - MaxSpeed - true - true - true - WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - pch.h - - - Console - true - true - true - - - $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) - - - Deploy DLLs - - - - - NotUsing - Level3 - MaxSpeed - true - true - true - NDEBUG;_CONSOLE;%(PreprocessorDefinitions) - true - pch.h - - - Console - true - true - true - - - $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) - - - Deploy DLLs - - - - - - - - {216a8e97-21f7-4bef-9e52-7f772c177c32} - - - - - - - - - + + + + + + Debug + Win32 + + + Release + Win32 + + + Debug + x64 + + + Release + x64 + + + + 15.0 + {77053F92-F003-4D1C-A489-1DEB7CFEA4EC} + Win32Proj + SampleCppLogManagers + + + + Application + true + Unicode + + + Application + false + true + Unicode + + + Application + true + Unicode + + + Application + false + true + Unicode + + + + + + + + + + + + + + + + + + + + true + $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public;$(SolutionDir)\..\lib\pal\ + $(ProjectDir) + $(Configuration)\ + $(LibraryPath) + + + true + $(ProjectDir) + $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public;$(SolutionDir)\..\lib\pal\ + $(LibraryPath) + + + false + $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public + + + false + $(VC_IncludePath);$(WindowsSDK_IncludePath);$(SolutionDir)\..\lib\include\public + + + + NotUsing + Level3 + Disabled + true + WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + pch.h + + + Console + true + + + $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) + + + Deploy DLLs + + + + + NotUsing + Level3 + Disabled + true + _DEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + pch.h + + + Console + true + + + $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) + + + Deploy DLLs + + + + + NotUsing + Level3 + MaxSpeed + true + true + true + WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + pch.h + + + Console + true + true + true + + + $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) + + + Deploy DLLs + + + + + NotUsing + Level3 + MaxSpeed + true + true + true + NDEBUG;_CONSOLE;%(PreprocessorDefinitions) + true + pch.h + + + Console + true + true + true + + + $(MSBuildProjectDirectory)\deploy-dll.cmd $(Configuration) $(Platform) $(OutDir) + + + Deploy DLLs + + + + + + + + {216a8e97-21f7-4bef-9e52-7f772c177c32} + + + + + + + + + \ No newline at end of file diff --git a/examples/cpp/SampleCppMini/main.cpp b/examples/cpp/SampleCppMini/main.cpp index 3cddf79a2..1d371ced6 100644 --- a/examples/cpp/SampleCppMini/main.cpp +++ b/examples/cpp/SampleCppMini/main.cpp @@ -31,7 +31,7 @@ void test_cpp_api(const char * token, int ticketType, const char *ticket) if (ticket != nullptr) { - const char *ticketNames[7] = + const char *ticketNames[8] = { "TicketType_MSA_Device", "TicketType_MSA_User", @@ -40,7 +40,7 @@ void test_cpp_api(const char * token, int ticketType, const char *ticket) "TicketType_AAD", "TicketType_AAD_User", "TicketType_AAD_JWT", - "TicketType_AAD_Device", + "TicketType_AAD_Device" }; printf("\nSet ticket %s=%s\n", ticketNames[ticketType], ticket); auto tc = LogManager::GetAuthTokensController(); diff --git a/lib/api/ContextFieldsProvider.cpp b/lib/api/ContextFieldsProvider.cpp index e5792cc66..f3a922a26 100644 --- a/lib/api/ContextFieldsProvider.cpp +++ b/lib/api/ContextFieldsProvider.cpp @@ -8,8 +8,104 @@ #include "pal/PAL.hpp" #include "utils/StringUtils.hpp" +#include + namespace MAT_NS_BEGIN { + // clang-format off + /** + * This map allows to remap from canonical Common Schema JSON notation to "CommonFields" + * (ex. AppInfo.*) notation historically used by Aria v1/v2 and 1DS v3 SDKs. Field name + * reshaping is performed as follows: + * + * CS3.0/4.0 JSON notation -> Common Alias -> :CsProtocol::Record object -> CS on wire + * + * Common Alias (no reshaping) -> :CsProtocol::Record object -> CS on wire + * + * Lookup for the data transform is a hashtable-based. Performed only in case if + * customer-supplied Common Context property starts with "ext.": check 4 bytes match, + * then perform the hash map lookup: + * - If there is a match, promote to corresponding COMMONFIELDS_* name. + * - If there's no match, keep as is with its original context field name. + * This logic allows to respect both - "canonical" names and "legacy" entity names. + * + * Why do we have to support both naming conventions? It's an organizational choice whether + * to rely on 'legacy' AppInfo.*, EventInfo.*, naming (Aria-Kusto) or migrate to more modern, + * standard Common Schema notation (standalone Kusto). Organizational choice depends on + * final data storage and consumption model. From a developer perspective - the solution is + * to allow a developer to use the naming convention as they would use to query a dataset + * in Kusto or ADLS Gen2. Having 1-1 mapping between instrumentation properties and storage + * allows to resolve ambiguities and avoid confusion. + * + * For any extension that is not supported by the map below, a developer could implement + * their own custom IDecorator - to stamp a common field property at decorator-level. + */ + static const std::unordered_map kCommonSchemaToCommonFieldsMap= + { + // ext.app extension: + // https://1dsdocs.azurewebsites.net/schema/PartA/app.html + {"ext.app.id", COMMONFIELDS_APP_ID}, + {"ext.app.ver", COMMONFIELDS_APP_VERSION}, + {"ext.app.name", COMMONFIELDS_APP_NAME}, + {"ext.app.locale", COMMONFIELDS_APP_LANGUAGE}, + // {"ext.app.asId", NOT_SUPPORTED}, + // {"ext.app.sesId", NOT_SUPPORTED}, + // {"ext.app.userId", NOT_SUPPORTED}, // use "ext.user.*id" instead + {"ext.app.expId", COMMONFIELDS_APP_EXPERIMENTIDS}, + {"ext.app.env", COMMONFIELDS_APP_ENV}, + // ext.device extension: + // https://1dsdocs.azurewebsites.net/schema/PartA/device.html + // {"ext.device.id", NOT_SUPPORTED}, // Device g: ID can only be populated via MSAL ticket claim. + {"ext.device.deviceClass", COMMONFIELDS_DEVICE_CLASS}, + {"ext.device.make", COMMONFIELDS_DEVICE_MAKE}, + {"ext.device.model", COMMONFIELDS_DEVICE_MODEL}, + {"ext.device.localId", COMMONFIELDS_DEVICE_ID}, + // {"ext.device.auth*", NOT_SUPPORTED}, // Use IDecorator + // {"ext.device.org*", NOT_SUPPORTED}, // Use IDecorator + // ext.net extension: + // https://1dsdocs.azurewebsites.net/schema/PartA/net.html + {"ext.net.provider", COMMONFIELDS_NETWORK_PROVIDER}, + {"ext.net.cost", COMMONFIELDS_NETWORK_COST}, + {"ext.net.type", COMMONFIELDS_NETWORK_TYPE}, + // ext.os extension: + // https://1dsdocs.azurewebsites.net/schema/PartA/os.html + {"ext.os.name", COMMONFIELDS_OS_NAME}, + + // Special case for CS3.0 vs CS4.0: + // - ext.os.ver - exists in both - CS3.0 and CS4.0 + // - ext.os.build - exists only in CS4.0 +#if defined(HAVE_CS4) || defined(HAVE_CS4_FULL) + {"ext.os.ver", COMMONFIELDS_OS_VERSION}, + {"ext.os.build", COMMONFIELDS_OS_BUILD}, +#else + // For some historical reason, the code treated + // COMMONFIELDS_OS_BUILD as an alias for ext.os.ver. + // Keep it that way, so we don't break the contract + // for existing apps. COMMONFIELDS_OS_BUILD lands + // on extOs[0].ver anyways. Thus, we keep consistency + // for devs that use Common Schema notation. + {"ext.os.ver", COMMONFIELDS_OS_BUILD}, +#endif + //{"ext.os.locale", NOT_SUPPORTED}, // Use IDecorator + //{"ext.os.bootId", NOT_SUPPORTED}, // Use IDecorator + //{"ext.os.expId", NOT_SUPPORTED}, // Use IDecorator + // ext.user extension: + // https://1dsdocs.azurewebsites.net/schema/PartA/user.html + // {"ext.user.id", NOT_SUPPORTED}, // User g: ID can only be populated via MSAL ticket claim. + {"ext.user.localId", COMMONFIELDS_USER_ID}, + // {"ext.user.authId", NOT_SUPPORTED}, // Use IDecorator + {"ext.user.locale", COMMONFIELDS_USER_LANGUAGE}, + // ext.loc extension: + // https://1dsdocs.azurewebsites.net/schema/PartA/loc.html + // {"ext.loc.id", NOT_SUPPORTED}, // Use IDecorator + // {"ext.loc.country", NOT_SUPPORTED}, // Use IDecorator + {"ext.loc.tz", COMMONFIELDS_USER_TIMEZONE}, + {"ext.loc.timezone", COMMONFIELDS_USER_TIMEZONE}, // alias of 'tz' + // ext.m365 extension: + // https://1dsdocs.azurewebsites.net/schema/PartA/m365a.html + {"ext.m365a.enrolledTenantId", COMMONFIELDS_COMMERCIAL_ID } + }; + // clang-format on ContextFieldsProvider::ContextFieldsProvider() : ContextFieldsProvider(nullptr) @@ -256,12 +352,33 @@ namespace MAT_NS_BEGIN record.extOs[0].name = iter->second.as_string; } +#if defined(HAVE_CS4) || defined(HAVE_CS4_FULL) + // Straightforward implementation for CS4.0+. No quirks. + // - OS_VERSION maps to ext.os.ver field. + iter = m_commonContextFields.find(COMMONFIELDS_OS_VERSION); + if (iter != m_commonContextFields.end()) + { + record.extOs[0].ver = iter->second.as_string; + } + // - OS_BUILD maps to ext.os.build field. + iter = m_commonContextFields.find(COMMONFIELDS_OS_BUILD); + if (iter != m_commonContextFields.end()) + { + record.extOs[0].build = iter->second.as_string; + } +#else + // This is a historical quirk due to difference between CS3.0 and CS4.0 + // `ext.os.ver` exists in both schemas; but `ext.os.build` only in CS4.0 + // However, it appears like the preference in this code has been to use + // the newer Aria-style alias. Fixing this could break existing apps. + // Thus, we keep the legacy behavior untouched. iter = m_commonContextFields.find(COMMONFIELDS_OS_BUILD); if (iter != m_commonContextFields.end()) { //EventProperty prop = (*m_commonContextFieldsP)[COMMONFIELDS_OS_VERSION]; record.extOs[0].ver = iter->second.as_string; } +#endif iter = m_commonContextFields.find(COMMONFIELDS_USER_ID); if (iter != m_commonContextFields.end()) @@ -433,6 +550,19 @@ namespace MAT_NS_BEGIN void ContextFieldsProvider::SetCommonField(const std::string& name, const EventProperty& value) { LOCKGUARD(m_lock); + // Any common field that starts with "ext." prefix and exists in kCommonSchemaToCommonFieldsMap + // is considered to be a Part A extension property. Code below allows to remap from JSON CS + // notation to CommonFields, e.g. (AppInfo.*, DeviceInfo.*, etc.) notation. + if (name.rfind("ext.", 0)==0) + { + const auto it = kCommonSchemaToCommonFieldsMap.find(name); + if (it != kCommonSchemaToCommonFieldsMap.end()) + { + // Rename the key from Common Schema dotted notation to COMMONFIELDS_* alias + m_commonContextFields[it->second] = value; + return; + } + } m_commonContextFields[name] = value; } @@ -451,6 +581,20 @@ namespace MAT_NS_BEGIN } } + void ContextFieldsProvider::ClearParentContext() + { + // This method allows to disassociate from parent LogManager context due to isolation + // reasons. For example, when Guest attaches to Host LogManager, it could be configured + // to avoid capturing common context properties. Logger context on creation by default + // acquires the properties populated by its LogManager. Guest Logger context should be + // wiped clean if Guest scope has been set to none ("-"). + m_parent = nullptr; + m_commonContextFields.clear(); + m_customContextFields.clear(); + m_commonContextEventToConfigIds.clear(); + PAL::registerSemanticContext(this); + } + void ContextFieldsProvider::SetParentContext(ContextFieldsProvider* parent) { m_parent = parent; diff --git a/lib/api/ContextFieldsProvider.hpp b/lib/api/ContextFieldsProvider.hpp index 20c9bcdc9..ea9dbaa57 100644 --- a/lib/api/ContextFieldsProvider.hpp +++ b/lib/api/ContextFieldsProvider.hpp @@ -32,6 +32,8 @@ namespace MAT_NS_BEGIN virtual void SetCustomField(const std::string& name, const EventProperty& value) override; virtual void SetParentContext(ContextFieldsProvider* parent); + virtual void ClearParentContext(); + virtual void SetTicket(TicketType type, std::string const& ticketValue) override; virtual void SetEventExperimentIds(std::string const & eventName, std::string const & experimentIds) override; diff --git a/lib/api/LogManagerFactory.cpp b/lib/api/LogManagerFactory.cpp index 0f6a31171..ebed45df9 100644 --- a/lib/api/LogManagerFactory.cpp +++ b/lib/api/LogManagerFactory.cpp @@ -25,6 +25,30 @@ namespace MAT_NS_BEGIN std::recursive_mutex ILogManagerInternal::managers_lock; std::set ILogManagerInternal::managers; + // Internal utility function to validate if LogManager instance (handle) + // is still alive. Used in Host-Guest scenarios to determine if instance + // needs to be recreated. It will return `false` in case if ILogManager + // pointer does not refer to valid object, OR in case if instance has + // already called `FlushAndTeardown` method and destroyed its loggers. + static inline bool IsInstanceAlive(ILogManager* instance) + { + bool result = true; + // Quick peek at the list of LogManagers to check if this entity + // has been destroyed. + LOCKGUARD(ILogManagerInternal::managers_lock); + if (ILogManagerInternal::managers.count(instance)) + { + // This is a valid instance. One of instances created via: + // > ILogManager* LogManagerFactory::Create(ILogConfiguration& configuration) + // method. Instance type is (LogManagerImpl*) casted to ILogManager. + // Use static_cast to reconstruct back to its actual type: + const auto instance_internal = static_cast(instance); + // Call its internal method to check if FlushAndTeardown has been called: + result = instance_internal->IsAlive(); + } + return result; + } + /// /// Creates an instance of ILogManager using specified configuration. /// @@ -159,6 +183,14 @@ namespace MAT_NS_BEGIN // If there was no module configuration supplied explicitly, then do we treat the client as host or guest? c[CFG_BOOL_HOST_MODE] = (name == host); + if (!IsInstanceAlive(shared[host].instance)) + { + // "Reanimate" this instance by creating new instance using new config. + // This allows guests to reattach to the same host by name after its + // reinitialization, e.g. in EUDB scenarios where URL needs to change. + // Guests can keep holding on to the same instance handle. + shared[host].instance = Create(c); + } return shared[host].instance; } @@ -177,9 +209,21 @@ namespace MAT_NS_BEGIN if (kv.second.names.count(name)) { kv.second.names.erase(name); - if (kv.second.names.empty()) + auto instance = shared[host].instance; + const bool forceRelease = !IsInstanceAlive(instance); + const bool zeroGuestsRemaining = kv.second.names.empty(); + if (zeroGuestsRemaining || forceRelease) { - // Last owner is gone, destroy + if (!zeroGuestsRemaining) + { + // In this case the logs emitted by Guests attached to "stale" Host + // would be lost. Emit a warning when that happens. Typically it + // could happen when the main app is unloaded and shut down its + // telemetry, but Guest library is still running some processing. + LOG_WARN("Host released before Guests: %s", name.c_str()); + dump(); + } + // Destroy it. Destroy(shared[host].instance); shared.erase(host); } diff --git a/lib/api/LogManagerImpl.cpp b/lib/api/LogManagerImpl.cpp index 6a98e6406..f6e6750b4 100644 --- a/lib/api/LogManagerImpl.cpp +++ b/lib/api/LogManagerImpl.cpp @@ -623,7 +623,6 @@ namespace MAT_NS_BEGIN LOG_INFO("SetContext"); EventProperty prop(value, piiKind); m_context.SetCustomField(name, prop); - m_context.SetCustomField(name, prop); { LOCKGUARD(m_dataInspectorGuard); for(const auto& dataInspector : m_dataInspectors) diff --git a/lib/api/LogManagerImpl.hpp b/lib/api/LogManagerImpl.hpp index f48b7a847..f3f0b1bb1 100644 --- a/lib/api/LogManagerImpl.hpp +++ b/lib/api/LogManagerImpl.hpp @@ -308,6 +308,11 @@ namespace MAT_NS_BEGIN virtual bool StartActivity() override; virtual void EndActivity() override; + virtual bool IsAlive() + { + return m_alive; + } + protected: std::unique_ptr& GetSystem(); void InitializeModules() noexcept; diff --git a/lib/api/Logger.cpp b/lib/api/Logger.cpp index 8540d0bcf..aef35046c 100644 --- a/lib/api/Logger.cpp +++ b/lib/api/Logger.cpp @@ -230,7 +230,8 @@ namespace MAT_NS_BEGIN // Since common props would typically be populated by the root-level // LogManager instance and we are detaching from that one, we need // to populate this context with common props directly. - PAL::registerSemanticContext(&m_context); + m_context.ClearParentContext(); + return; } m_context.SetParentContext(static_cast(context)); } diff --git a/lib/api/capi.cpp b/lib/api/capi.cpp index 9656a7f2d..f64fa2b94 100644 --- a/lib/api/capi.cpp +++ b/lib/api/capi.cpp @@ -13,13 +13,17 @@ #include "pal/TaskDispatcher_CAPI.hpp" #include "utils/Utils.hpp" +#include "LogManager.hpp" + #include "pal/PAL.hpp" #include "CommonFields.h" +#include "config/RuntimeConfig_Default.hpp" #include #include #include +#include static const char * libSemver = TELEMETRY_EVENTS_VERSION; @@ -91,7 +95,8 @@ evt_status_t mat_open_core( { if (client->ctx_data == config) { - // Guest instance with the same config is already open + // Guest or Host instance with the same config is already open + ctx->handle = code; return EALREADY; } // hash code is assigned to another client, increment and retry @@ -102,11 +107,18 @@ evt_status_t mat_open_core( isHashFound = true; } while (!isHashFound); + // Make sure that we fully inherit the default configuration, then + // overlay custom configuration on top of default. + clients[code].config = ILogConfiguration(); + Variant::merge_map(*clients[code].config, *defaultRuntimeConfig); + // JSON configuration must start with { if (config[0] == '{') { // Create new configuration object from JSON - clients[code].config = MAT::FromJSON(config); + ILogConfiguration jsonConfig = MAT::FromJSON(config); + // Overwrite default values with custom configuration. + Variant::merge_map(*clients[code].config, *jsonConfig, true); } else { @@ -114,7 +126,7 @@ evt_status_t mat_open_core( // That approach allows to consume the lightweght C API without JSON parser compiled in. std::string moduleName = "CAPI-Client-"; moduleName += std::to_string(code); - clients[code].config = + VariantMap customConfig = { { CFG_STR_FACTORY_NAME, moduleName }, { "version", "1.0.0" }, @@ -126,6 +138,8 @@ evt_status_t mat_open_core( }, { CFG_STR_PRIMARY_TOKEN, config } }; + // Overwrite host-guest related settings using VariantMap above. + Variant::merge_map(*clients[code].config, customConfig, true); } // Remember the original config string. Needed to avoid hash code collisions @@ -231,9 +245,12 @@ evt_status_t mat_open_with_params(evt_context_t *ctx) } /** - * Marashal C struct to C++ API + * Marashal C struct to C++ API for the following methods: + * - ILogger->LogEvent(...) + * - ILogger->SetContext(...) - for each string key - prop value + * - ILogManager->SetContext(...) - for each string key - prop value */ -evt_status_t mat_log(evt_context_t *ctx) +evt_status_t mat_sendprops(evt_context_t* ctx, evt_call_t op) { VERIFY_CLIENT_HANDLE(client, ctx); @@ -242,10 +259,24 @@ evt_status_t mat_log(evt_context_t *ctx) EventProperties props; props.unpack(evt, ctx->size); + // Determine ingestion token to use for this record. + std::string token; + + // Use LogManager configuration primary token if available. + if (config.HasConfig(CFG_STR_PRIMARY_TOKEN)) + { + token = static_cast(config[CFG_STR_PRIMARY_TOKEN]); + } + + // Allow to override iKey per event via property. + // C API client maintains one handle for different tenants. auto m = props.GetProperties(); - EventProperty &prop = m[COMMONFIELDS_IKEY]; - std::string token = prop.as_string; - props.erase(COMMONFIELDS_IKEY); + if (m.count(COMMONFIELDS_IKEY)) + { + EventProperty& prop = m[COMMONFIELDS_IKEY]; + token = prop.as_string; + props.erase(COMMONFIELDS_IKEY); + } // Privacy feature for OTEL C API client: // @@ -275,12 +306,91 @@ evt_status_t mat_log(evt_context_t *ctx) if (logger == nullptr) { ctx->result = EFAULT; /* invalid address */ + return ctx->result; } - else + + // Enforce if scope sharing is not enabled (default). + // However, if a client / extension explicitly opts-in to append its + // context to host variables, then we respect their choice and append + // the corresponding host context values on guest events. This is + // required for the case where the app, SDK, and extension SDKs run + // within the same data silo / trust boundary, ensuring uniform and + // consistent symmetric data contract for all parties involved. + if (scope == CONTEXT_SCOPE_NONE) + { + if (op == EVT_OP_SET_LOGMANAGER_CONTEXT) + { + // We are running in isolation. It does not make sense to propagate + // the context at LogManager (parent) -level since we do not have + // ability to modify it. Downgrade to EVT_OP_SET_LOGGER_CONTEXT. + op = EVT_OP_SET_LOGGER_CONTEXT; + } + } + + // Allows to pass C API properties to ILogger or LogManager semantic context. + // Semantic context layer provides access to both - Common Fields (Part A), + // as well as Custom Fields (Part C). For C API due to performance reasons - + // the determination of what is considered Common versus Custom is done + // by checking the first 4 bytes of context property name: + // - This is in alignment with canonical Common Schema JSON notation. + // - This is much faster than validating the list of old-style COMMONFIELDS_ + // aliases. + // + // Developers can find the mapping on 1DS site or use ContextFieldProvider.cpp + // as a reference. + auto populateContext = [m](ISemanticContext& context) + { + for (auto prop : m) + { + if (prop.first.rfind("ext.", 0) == 0) + { + context.SetCommonField(prop.first, prop.second); + } + else + { + context.SetCustomField(prop.first, prop.second); + } + } + }; + + switch (op) { - logger->SetParentContext(nullptr); + // This path allows to implement common props stamping via C#->C->C++ API. + // Calls ILogger->SetContext(...) to populate the "local" current ILogger context. + case EVT_OP_SET_LOGGER_CONTEXT: + { + const auto loggerContext = logger->GetSemanticContext(); + populateContext(*loggerContext); + ctx->result = EOK; + } + break; + + // This path allows to implement common props stamping via C#->C->C++ API. + // Calls ILogManager->SetContext(...) to populate the "global" LogManager context. + case EVT_OP_SET_LOGMANAGER_CONTEXT: + { + auto& logManagerContext = client->logmanager->GetSemanticContext(); + populateContext(logManagerContext); + ctx->result = EOK; + } + break; + + // Original implementation of evt_log call remains unaltered. Note that the processing + // of Common Fields via C API could only done using: + // - set_logger_context(_s) + // - set_logmanager_context(_s) + // Practical reasons: + // - done intentionally to avoid altering behavior of legacy code. + // - Part A extension values typically remain constant for an app/user session. + case EVT_OP_LOG: logger->LogEvent(props); ctx->result = EOK; + break; + + default: + // Unsupported API. Call failed. + ctx->result = EFAULT; + break; } return ctx->result; } @@ -350,7 +460,7 @@ extern "C" { /** * Simple stable backwards- / forward- compatible ABI interface */ - evt_status_t EVTSDK_LIBABI_CDECL evt_api_call_default(evt_context_t *ctx) + MATSDK_LIBABI evt_status_t EVTSDK_LIBABI_CDECL evt_api_call_default(evt_context_t* ctx) { evt_status_t result = EFAIL; @@ -383,7 +493,7 @@ extern "C" { break; case EVT_OP_LOG: - result = mat_log(ctx); + result = mat_sendprops(ctx, EVT_OP_LOG); break; case EVT_OP_PAUSE: @@ -422,6 +532,23 @@ extern "C" { case EVT_OP_FLUSHANDTEARDOWN: result = mat_flushAndTeardown(ctx); break; + + // New API in v3.7.1. This does not break ABI compat from v3.7.0. + // If v3.7.0 client calls into v3.7.1 implementation, then the + // call is a noop - handled by safely returning ENOTSUP. + // No new structs, no struct layout changes. + case EVT_OP_SET_LOGGER_CONTEXT: + result = mat_sendprops(ctx, EVT_OP_SET_LOGGER_CONTEXT); + break; + + // New API in v3.7.1. This does not break ABI compat from v3.7.0. + // If v3.7.0 client calls into v3.7.1 implementation, then the + // call is a noop - handled by safely returning ENOTSUP. + // No new structs, no struct layout changes. + case EVT_OP_SET_LOGMANAGER_CONTEXT: + result = mat_sendprops(ctx, EVT_OP_SET_LOGMANAGER_CONTEXT); + break; + // Add more OPs here default: diff --git a/lib/include/public/ILogger.hpp b/lib/include/public/ILogger.hpp index 60cb6c1cd..82ef52cb9 100644 --- a/lib/include/public/ILogger.hpp +++ b/lib/include/public/ILogger.hpp @@ -46,21 +46,22 @@ Event tags that can be assigned to influence how the telemetry client handles ev Some tags require formal approval before they can be used. Refer to https://osgwiki.com/wiki/Common_Schema_Event_Overrides for details on the requirements and how to start the approval process. */ - + // CsFlags value #define MICROSOFT_EVENTTAG_COSTDEFERRED_LATENCY 0x00040000 #define MICROSOFT_EVENTTAG_CORE_DATA 0x00080000 #define MICROSOFT_EVENTTAG_INJECT_XTOKEN 0x00100000 -#define MICROSOFT_EVENTTAG_REALTIME_LATENCY 0x00200000 -#define MICROSOFT_EVENTTAG_NORMAL_LATENCY 0x00400000 +#define MICROSOFT_EVENTTAG_REALTIME_LATENCY 0x00200000 // EventLatencyRealTime = 0x0200L +#define MICROSOFT_EVENTTAG_NORMAL_LATENCY 0x00400000 // EventLatencyNormal = 0x0100L -#define MICROSOFT_EVENTTAG_CRITICAL_PERSISTENCE 0x00800000 -#define MICROSOFT_EVENTTAG_NORMAL_PERSISTENCE 0x01000000 +#define MICROSOFT_EVENTTAG_CRITICAL_PERSISTENCE 0x00800000 // EventPersistenceCritical = 0x02L +#define MICROSOFT_EVENTTAG_NORMAL_PERSISTENCE 0x01000000 // EventPersistenceNormal = 0x01L -#define MICROSOFT_EVENTTAG_DROP_PII 0x02000000 -#define MICROSOFT_EVENTTAG_HASH_PII 0x04000000 -#define MICROSOFT_EVENTTAG_MARK_PII 0x08000000 +#define MICROSOFT_EVENTTAG_DROP_PII 0x02000000 // DropIdentifiers = 0x200000L +#define MICROSOFT_EVENTTAG_HASH_PII 0x04000000 // HashIdentifiers = 0x100000L +#define MICROSOFT_EVENTTAG_MARK_PII 0x08000000 // IsPII = 0x080000L +#define MICROSOFT_EVENTTAG_SCRUB_IP 0x10000000 // ScrubIpIdentifiers = 0x400000L /// /// The PageActionData structure represents the data of a page action event. diff --git a/lib/include/public/ctmacros.hpp b/lib/include/public/ctmacros.hpp index 42547e41d..3188ec111 100644 --- a/lib/include/public/ctmacros.hpp +++ b/lib/include/public/ctmacros.hpp @@ -127,4 +127,48 @@ #define EVTSDK_LIBABI_CDECL MATSDK_LIBABI_CDECL #define EVTSDK_SPEC MATSDK_SPEC +/* Implement struct packing for stable FFI C API to allow for C# apps + * written in Mono and .NET Standard 2.x to call into 1DS C API. + */ +#ifdef HAVE_MAT_ABI_V3_1_0 +/* Legacy v3.1 struct ABI. Not compatible with cross-plat C# projection */ +#define MATSDK_PACKED_STRUCT +#define MATSDK_PACK_PUSH +#define MATSDK_PACK_POP +#define MATSDK_ALIGN64(x) x +/* Modern v3.7 struct ABI. Compatible with cross-plat C# callers on both + * 32-bit and 64-bit Intel and ARM architectures - on Windows, Android + * and Mac. This should ideally be the default going forward, as it + * ensures predictable, compiler optimization level-agnostic C API FFI. + */ +#elif __clang__ +# define MATSDK_PACKED_STRUCT __attribute__((packed)) +# define MATSDK_PACK_PUSH +# define MATSDK_PACK_POP +#define MATSDK_ALIGN64(x) union { x; uint64_t padding; } +#elif __GNUC__ +# define MATSDK_PACKED_STRUCT __attribute__((packed)) +# define MATSDK_PACK_PUSH +# define MATSDK_PACK_POP +#define MATSDK_ALIGN64(x) union { x; uint64_t padding; } +#elif _MSC_VER +# define MATSDK_PACKED_STRUCT +# define MATSDK_PACK_PUSH __pragma(pack(push, 1)) +# define MATSDK_PACK_POP __pragma(pack(pop)) +#define MATSDK_ALIGN64(x) union { x; uint64_t padding; } +#else +/* Fallback to HAVE_MAT_ABI_V3_1_0 : compatible with prebuilt shared libraries + * that used the old C API only within the same arch/compiler domain. Unfortunately + * the old layout is not usable if you'd like to invoke C API from Mono (e.g. Unity) + * or cross-platform .NET Standard apps. + */ +#ifndef HAVE_MAT_ABI_V3_1_0 +#define HAVE_MAT_ABI_V3_1_0 +#endif +# define MATSDK_PACKED_STRUCT +# define MATSDK_PACK_PUSH +# define MATSDK_PACK_POP +#define MATSDK_ALIGN64(x) x +#endif + #endif diff --git a/lib/include/public/mat.h b/lib/include/public/mat.h index 3ad8d36f7..c61a174bb 100644 --- a/lib/include/public/mat.h +++ b/lib/include/public/mat.h @@ -9,7 +9,13 @@ * For version handshake check there is no mandatory requirement to update the $PATCH level. * Ref. https://semver.org/ for Semantic Versioning documentation. */ -#define TELEMETRY_EVENTS_VERSION "3.1.0" +#ifdef HAVE_MAT_ABI_V3_1_0 +/* Allow to fallback to same C ABI interface as in old releases */ +#define TELEMETRY_EVENTS_VERSION "3.1.0" +#else +/* More modern "cross-arch" ABI interface with fixed padding. */ +#define TELEMETRY_EVENTS_VERSION "3.7.0" +#endif #include "ctmacros.hpp" @@ -60,7 +66,18 @@ extern "C" { EVT_OP_VERSION = 0x0000000B, EVT_OP_OPEN_WITH_PARAMS = 0x0000000C, EVT_OP_FLUSHANDTEARDOWN = 0x0000000D, - EVT_OP_MAX = EVT_OP_FLUSHANDTEARDOWN + 1, + /** + * Context operations allow to set ILogger or ILogManager semantic context values. + * In addition to custom Part C context values, Common Schema attributes, e.g. `ext.device.id` + * or `ext.app.name` - Part A values are respected and applied using corresponding + * ISemanticContext API call. This approach allows to express most Common Schema + * event fields to C API for extensions, SDK-in-SDK, and higher-level programming + * languages such as Unity C# and .NET Standard. + */ + EVT_OP_SET_LOGGER_CONTEXT = 0x0000000E, + EVT_OP_SET_LOGMANAGER_CONTEXT = 0x0000000F, + EVT_OP_MAX = EVT_OP_SET_LOGMANAGER_CONTEXT + 1, + EVT_OP_MAXINT = 0xFFFFFFFF } evt_call_t; typedef enum evt_prop_t @@ -80,11 +97,13 @@ extern "C" { TYPE_BOOL_ARRAY, TYPE_GUID_ARRAY, /* NULL-type */ - TYPE_NULL + TYPE_NULL, + TYPE_MAXINT = 0xFFFFFFFF } evt_prop_t; - typedef struct evt_guid_t + typedef struct MATSDK_PACKED_STRUCT evt_guid_t { +MATSDK_PACK_PUSH /** * * Specifies the first eight hexadecimal digits of the GUID. @@ -112,19 +131,22 @@ extern "C" { * */ uint8_t Data4[8]; +MATSDK_PACK_POP } evt_guid_t; typedef int64_t evt_handle_t; typedef int32_t evt_status_t; typedef struct evt_event evt_event; - typedef struct evt_context_t + typedef struct MATSDK_PACKED_STRUCT evt_context_t { +MATSDK_PACK_PUSH evt_call_t call; /* In */ evt_handle_t handle; /* In / Out */ - void* data; /* In / Out */ + MATSDK_ALIGN64(void* data); evt_status_t result; /* Out */ uint32_t size; /* In / Out */ +MATSDK_PACK_POP } evt_context_t; /** @@ -184,12 +206,14 @@ extern "C" { uint64_t** as_arr_time; } evt_prop_v; - typedef struct evt_prop + typedef struct MATSDK_PACKED_STRUCT evt_prop { - const char* name; +MATSDK_PACK_PUSH + MATSDK_ALIGN64(const char* name); evt_prop_t type; evt_prop_v value; uint32_t piiKind; +MATSDK_PACK_POP } evt_prop; /** @@ -484,17 +508,22 @@ extern "C" { /** * - * Logs a telemetry event (security-enhanced _s function) + * Sends a collection of telemetry event properties (security-enhanced _s function). + * This is internal API used by other functions: + * - evt_log_s - calls ILogger->LogEvent(props) + * - evt_set_logger_context_s - sets ILogger semantic context values + * - evt_set_logmanager_context_s - sets ILogManager semantic context values * * SDK handle. + * Code of event properties operation. * Number of event properties in array. * Event properties array. * */ - static inline evt_status_t evt_log_s(evt_handle_t handle, uint32_t size, evt_prop* evt) + static inline evt_status_t evt_sendprops_s(evt_handle_t handle, evt_call_t op, uint32_t size, evt_prop* evt) { evt_context_t ctx; - ctx.call = EVT_OP_LOG; + ctx.call = op; ctx.handle = handle; ctx.data = (void *)evt; ctx.size = size; @@ -503,24 +532,113 @@ extern "C" { /** * - * Logs a telemetry event. - * Last item in evt_prop array must be { .name = NULL, .type = TYPE_NULL } + * Sends a collection of telemetry event properties. + * This is internal function used by other functions: + * - evt_log - calls ILogger->LogEvent(props) + * - evt_set_logger_context - sets ILogger semantic context values + * - evt_set_logmanager_context - sets ILogManager semantic context values * * SDK handle. - * Number of event properties in array. + * Code of event properties operation. * Event properties array. * */ - static inline evt_status_t evt_log(evt_handle_t handle, evt_prop* evt) + static inline evt_status_t evt_sendprops(evt_handle_t handle, evt_call_t op, evt_prop* evt) { evt_context_t ctx; - ctx.call = EVT_OP_LOG; + ctx.call = op; ctx.handle = handle; ctx.data = (void *)evt; ctx.size = 0; return evt_api_call(&ctx); } + /** + * + * Logs a telemetry event (security-enhanced _s function) + * + * SDK handle. + * Number of event properties in array. + * Event properties array. + * + */ + static inline evt_status_t evt_log_s(evt_handle_t handle, uint32_t size, evt_prop* evt) + { + return evt_sendprops_s(handle, EVT_OP_LOG, size, evt); + } + + /** + * + * Logs a telemetry event. + * Last item in evt_prop array must be { .name = NULL, .type = TYPE_NULL } + * + * SDK handle. + * Event properties array. + * + */ + static inline evt_status_t evt_log(evt_handle_t handle, evt_prop* evt) + { + return evt_sendprops(handle, EVT_OP_LOG, evt); + } + + /** + * + * Sets ILogger semantic context using a collection of properties (security-enhanced _s function) + * + * SDK handle. + * Number of event properties in array. + * Event properties array. + * + */ + static inline evt_status_t evt_set_logger_context_s(evt_handle_t handle, uint32_t size, evt_prop* evt) + { + return evt_sendprops_s(handle, EVT_OP_SET_LOGGER_CONTEXT, size, evt); + } + + /** + * + * Sets ILogger semantic context using a collection of properties. + * Last item in evt_prop array must be { .name = NULL, .type = TYPE_NULL } + * + * SDK handle. + * Number of event properties in array. + * Event properties array. + * + */ + static inline evt_status_t evt_set_logger_context(evt_handle_t handle, evt_prop* evt) + { + return evt_sendprops(handle, EVT_OP_SET_LOGGER_CONTEXT, evt); + } + + /** + * + * Sets ILogManager semantic context using a collection of properties (security-enhanced _s function) + * + * SDK handle. + * Number of event properties in array. + * Event properties array. + * + */ + static inline evt_status_t evt_set_logmanager_context_s(evt_handle_t handle, uint32_t size, evt_prop* evt) + { + return evt_sendprops_s(handle, EVT_OP_SET_LOGMANAGER_CONTEXT, size, evt); + } + + /** + * + * Sets ILogManager semantic context using a collection of properties. + * Last item in evt_prop array must be { .name = NULL, .type = TYPE_NULL } + * + * SDK handle. + * Number of event properties in array. + * Event properties array. + * + */ + static inline evt_status_t evt_set_logmanager_context(evt_handle_t handle, evt_prop* evt) + { + return evt_sendprops(handle, EVT_OP_SET_LOGMANAGER_CONTEXT, evt); + } + /* This macro automagically calculates the array size and passes it down to evt_log_s. * Developers don't have to calculate the number of event properties passed down to *'Log Event' API call utilizing the concept of Secure Template Overloads: diff --git a/lib/modules b/lib/modules index 1f0cccb44..883a3e3b9 160000 --- a/lib/modules +++ b/lib/modules @@ -1 +1 @@ -Subproject commit 1f0cccb443b092003a7ec46d4954a634c60eea4b +Subproject commit 883a3e3b9c63fa713bdde31bf7417e63e416fe4e diff --git a/lib/stats/Statistics.cpp b/lib/stats/Statistics.cpp index 2f421714c..85f93d6b3 100644 --- a/lib/stats/Statistics.cpp +++ b/lib/stats/Statistics.cpp @@ -20,6 +20,7 @@ namespace MAT_NS_BEGIN { m_logManager(telemetrySystem.getLogManager()), m_baseDecorator(m_logManager), m_semanticContextDecorator(m_logManager), + m_isScheduled(false), m_isStarted(false) { } diff --git a/tests/functests/APITest.cpp b/tests/functests/APITest.cpp index 868c81f22..1c6293f30 100644 --- a/tests/functests/APITest.cpp +++ b/tests/functests/APITest.cpp @@ -19,6 +19,7 @@ #include #include +#include #include #include "PayloadDecoder.hpp" @@ -616,6 +617,53 @@ TEST(APITest, LogManager_Reinitialize_Test) #define EVENT_NAME_PURE_C "Event.Name.Pure.C" #define JSON_CONFIG(...) #__VA_ARGS__ + +#ifndef HAVE_MAT_ABI_V3_1_0 +/* NOTE: if your library needs to be built with legacy C API, make sure that + * the same build settings get applied to your Functional Tests build. + */ +template +void check_size() { + static_assert(ExpectedSize == RealSize, "Size is off!"); +} + +/* This test performs size-checks of cross-architecture C API structs. Same C# projection + * compiled as IL assembly could be used on Mono, .NET Standard across currently supported + * architectures: Windows, Linux, Android and Mac, 32-bit/64-bit, Intel and ARM. + */ +TEST(APITest, C_API_Test_CheckSize) +{ + check_size(); + check_size(); + check_size(); + check_size(); + check_size(); + + // C# should take care of context.data void* being 32-bit on x86 and 64-bit on x64. + // Presently we support all modern 64-bit OS with C# projection (Windows-x64, + // Linux-64, Mac Intel-x64 and ARM-x64, as well as Meta Quest 2+ ARM64). + evt_context_t context; + check_size< evt_context_t, + sizeof(context.call) + + sizeof(uint64_t) + // sizeof(void*) + 32-bit padding AFTER on 32-bit OS + sizeof(context.handle) + + sizeof(context.result) + + sizeof(context.size)>(); + + // C# should take care of prop.name char* being 32-bit on x86 and 64-bit on x64. + // Presently we support all modern 64-bit OS with C# projection (Windows-x64, + // Linux-64, Mac Intel-x64 and ARM-x64, as well as Meta Quest 2+ ARM64). + evt_prop prop; + check_size(); + + check_size(); +} + +#endif TEST(APITest, C_API_Test) { TestDebugEventListener debugListener; @@ -651,7 +699,7 @@ TEST(APITest, C_API_Test) ( // Part A/B fields _STR(COMMONFIELDS_EVENT_NAME, EVENT_NAME_PURE_C), // Event name - _INT(COMMONFIELDS_EVENT_TIME, static_cast(now * 1000L)), // Epoch time in millis, ms since Jan 01 1970. (UTC) + _INT(COMMONFIELDS_EVENT_TIME, static_cast(ticks.ticks)), // Epoch time in .NET ticks _DBL("popSample", 100.0), // Effective sample rate _STR(COMMONFIELDS_IKEY, TEST_TOKEN), // iKey to send this event to _INT(COMMONFIELDS_EVENT_POLICYFLAGS, 0xffffffff), // UTC policy bitflags (optional) @@ -725,7 +773,7 @@ TEST(APITest, C_API_Test) // Must remove event listener befor closing the handle! client->logmanager->RemoveEventListener(EVT_LOG_EVENT, debugListener); - evt_flushAndTeardown(handle); + // evt_flushAndTeardown(handle); // <-- This is redundant since evt_close ref-counts and performs FlushAndTeardown evt_close(handle); ASSERT_EQ(capi_get_client(handle), nullptr); diff --git a/tests/functests/BasicFuncTests.cpp b/tests/functests/BasicFuncTests.cpp index 51848d054..5dc90752c 100644 --- a/tests/functests/BasicFuncTests.cpp +++ b/tests/functests/BasicFuncTests.cpp @@ -181,7 +181,7 @@ class BasicFuncTests : public ::testing::Test, std::remove(fileName.c_str()); } - virtual void Initialize() + virtual void Initialize(bool statsOff = false) /* stats are flaky because event is exceeding ingestion capacity */ { receivedRequests.clear(); auto configuration = LogManager::GetLogConfiguration(); @@ -199,7 +199,8 @@ class BasicFuncTests : public ::testing::Test, configuration[CFG_INT_MAX_TEARDOWN_TIME] = 2; // 2 seconds wait on shutdown configuration[CFG_STR_COLLECTOR_URL] = serverAddress.c_str(); configuration[CFG_MAP_HTTP][CFG_BOOL_HTTP_COMPRESSION] = false; // disable compression for now - configuration[CFG_MAP_METASTATS_CONFIG][CFG_INT_METASTATS_INTERVAL] = 30 * 60; // 30 mins + int64_t statsInterval = (statsOff) ? 0 : 30 * 60; // 30 mins; + configuration[CFG_MAP_METASTATS_CONFIG][CFG_INT_METASTATS_INTERVAL] = statsInterval; configuration["name"] = __FILE__; configuration["version"] = "1.0.0"; @@ -763,7 +764,7 @@ TEST_F(BasicFuncTests, restartRecoversEventsFromStorage) { { CleanStorage(); - Initialize(); + Initialize(true); // This code is a bit racy because ResumeTransmission is done in Initialize LogManager::PauseTransmission(); EventProperties event1("first_event"); @@ -780,7 +781,7 @@ TEST_F(BasicFuncTests, restartRecoversEventsFromStorage) } { - Initialize(); + Initialize(true); EventProperties fooEvent("fooEvent"); fooEvent.SetLatency(EventLatency_RealTime); fooEvent.SetPersistence(EventPersistence_Critical); @@ -788,7 +789,7 @@ TEST_F(BasicFuncTests, restartRecoversEventsFromStorage) LogManager::UploadNow(); // 1st request for realtime event - waitForEvents(3, 5); // start, first_event, second_event, ongoing, stop, start, fooEvent + waitForEvents(3, 3); // first_event, second_event, fooEvent // we drop two of the events during pause, though. EXPECT_GE(receivedRequests.size(), (size_t)1); if (receivedRequests.size() != 0) diff --git a/tests/functests/CMakeLists.txt b/tests/functests/CMakeLists.txt index ba0e524e7..c6e9059a5 100644 --- a/tests/functests/CMakeLists.txt +++ b/tests/functests/CMakeLists.txt @@ -2,6 +2,7 @@ message("--- functests") set(SRCS APITest.cpp + HostGuestTests.cpp BasicFuncTests.cpp LogSessionDataFuncTests.cpp Main.cpp diff --git a/tests/functests/FuncTests.vcxproj b/tests/functests/FuncTests.vcxproj index 79d3f97c0..54a3403a0 100644 --- a/tests/functests/FuncTests.vcxproj +++ b/tests/functests/FuncTests.vcxproj @@ -418,6 +418,7 @@ + diff --git a/tests/functests/FuncTests.vcxproj.filters b/tests/functests/FuncTests.vcxproj.filters index a6aee552b..663e3d447 100644 --- a/tests/functests/FuncTests.vcxproj.filters +++ b/tests/functests/FuncTests.vcxproj.filters @@ -11,6 +11,7 @@ common + @@ -22,7 +23,7 @@ - + @@ -81,4 +82,4 @@ {5f02135a-4d1e-496a-900a-e5ea8cfb222d} - \ No newline at end of file + diff --git a/tests/functests/HostGuestTests.cpp b/tests/functests/HostGuestTests.cpp new file mode 100644 index 000000000..e2483846d --- /dev/null +++ b/tests/functests/HostGuestTests.cpp @@ -0,0 +1,619 @@ +// +// Copyright (c) Microsoft Corporation. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 +// +#ifndef _CRT_SECURE_NO_WARNINGS +#define _CRT_SECURE_NO_WARNINGS +#endif + +#include "mat/config.h" + +#ifdef _MSC_VER +#pragma warning(disable : 4389) +#endif + +#include "CsProtocol_types.hpp" +#include "common/Common.hpp" + +#include +#include +#include + +#include "PayloadDecoder.hpp" + +#include "IDecorator.hpp" +#include "mat.h" + +#ifdef HAVE_MAT_JSONHPP +#include "json.hpp" +#endif + +#include + +using namespace MAT; + +// Allows to dump test events in their perfect Common Schema shape. +// #define ALLOW_RECORD_DECODER + +// 1DSCppSdkTest sandbox key +#define TEST_TOKEN "7c8b1796cbc44bd5a03803c01c2b9d61-b6e370dd-28d9-4a52-9556-762543cf7aa7-6991" +#define DUMMY_TOKEN "ffffffffffffffffffffffffffffffff-ffffffff-ffff-ffff-ffff-ffffffffffff-ffff" + +#define EVENT_NAME_HOST "Event.Name.Host" +#define EVENT_NAME_GUEST "Event.Name.Guest" + +#define JSON_CONFIG(...) #__VA_ARGS__ + +class TestDebugEventListener : public DebugEventListener +{ + public: + std::atomic netChanged; + std::atomic eps; + std::atomic numLogged0; + std::atomic numLogged; + std::atomic numSent; + std::atomic numDropped; + std::atomic numReject; + std::atomic numHttpError; + std::atomic numHttpOK; + std::atomic numCached; + std::atomic numFiltered; + std::atomic logLatMin; + std::atomic logLatMax; + std::atomic storageFullPct; + std::atomic storageFailed; + + std::function OnLogX; + + TestDebugEventListener() : + netChanged(false), + eps(0), + numLogged0(0), + numLogged(0), + numSent(0), + numDropped(0), + numReject(0), + numHttpError(0), + numHttpOK(0), + numCached(0), + numFiltered(0), + logLatMin(100), + logLatMax(0), + storageFullPct(0), + storageFailed(false) + { + resetOnLogX(); + } + + void reset() + { + netChanged = false; + eps = 0; + numLogged0 = 0; + numLogged = 0; + numSent = 0; + numDropped = 0; + numReject = 0; + numHttpError = 0; + numHttpOK = 0; + numCached = 0; + numFiltered = 0; + logLatMin = 100; + logLatMax = 0; + storageFullPct = 0; + storageFailed = false; + resetOnLogX(); + } + + virtual void OnLogXDefault(::CsProtocol::Record&){ + + }; + + void resetOnLogX() + { + OnLogX = [this](::CsProtocol::Record& record) + { + OnLogXDefault(record); + }; + } + + virtual void OnDebugEvent(DebugEvent& evt) + { + switch (evt.type) + { + case EVT_LOG_EVENT: + case EVT_LOG_LIFECYCLE: + case EVT_LOG_FAILURE: + case EVT_LOG_PAGEVIEW: + case EVT_LOG_PAGEACTION: + case EVT_LOG_SAMPLEMETR: + case EVT_LOG_AGGRMETR: + case EVT_LOG_TRACE: + case EVT_LOG_USERSTATE: + case EVT_LOG_SESSION: + { + /* Test-only code */ + ::CsProtocol::Record& record = *static_cast<::CsProtocol::Record*>(evt.data); + numLogged++; + OnLogX(record); + } + break; + + case EVT_REJECTED: + numReject++; + break; + + case EVT_ADDED: + break; + + /* Event counts below would never overflow the size of unsigned int */ + case EVT_CACHED: + numCached += (unsigned int)evt.param1; + break; + + case EVT_DROPPED: + numDropped += (unsigned int)evt.param1; + break; + + case EVT_SENT: + numSent += (unsigned int)evt.param1; + break; + + case EVT_STORAGE_FULL: + storageFullPct = (unsigned int)evt.param1; + break; + + case EVT_STORAGE_FAILED: + storageFailed = true; + break; + + case EVT_CONN_FAILURE: + case EVT_HTTP_FAILURE: + case EVT_COMPRESS_FAILED: + case EVT_UNKNOWN_HOST: + case EVT_SEND_FAILED: + + case EVT_HTTP_ERROR: + numHttpError++; + break; + + case EVT_HTTP_OK: + numHttpOK++; + break; + case EVT_FILTERED: + numFiltered++; + break; + + case EVT_SEND_RETRY: + case EVT_SEND_RETRY_DROPPED: + break; + + case EVT_NET_CHANGED: + netChanged = true; + break; + + case EVT_UNKNOWN: + default: + break; + }; + }; + + void printStats() + { + std::cerr << "[ ] netChanged = " << netChanged << std::endl; + std::cerr << "[ ] numLogged0 = " << numLogged0 << std::endl; + std::cerr << "[ ] numLogged = " << numLogged << std::endl; + std::cerr << "[ ] numSent = " << numSent << std::endl; + std::cerr << "[ ] numDropped = " << numDropped << std::endl; + std::cerr << "[ ] numReject = " << numReject << std::endl; + std::cerr << "[ ] numCached = " << numCached << std::endl; + std::cerr << "[ ] numFiltered = " << numFiltered << std::endl; + } +}; + +//////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// + +TestDebugEventListener debugListener; + +const unsigned maxEventsCount = 1; + +unsigned totalEvents = 0; + +/* + +// +// EXAMPLE #1: configure host instance via C API. +// + +const char* hostConfig = JSON_CONFIG( + { + "cacheFilePath" : "MyOfflineStorage.db", + "config" : { + "host" : "Mesh-Core-C-API-Host", + "scope" : "*" + }, + "stats" : { + "interval" : 0 + }, + "name" : "Mesh-Core-C-API-Host", + "version" : "1.0.0", + "primaryToken" : "7c8b1796cbc44bd5a03803c01c2b9d61-b6e370dd-28d9-4a52-9556-762543cf7aa7-6991", + "maxTeardownUploadTimeInSec" : 1, + "hostMode" : true, + "minimumTraceLevel" : 0, + "sdkmode" : 0, + "disableZombieLoggers": true + }); + +// +// EXAMPLE #2: configure host instance in C++ via JSON configuration above. +// + +static ILogConfiguration hostConfiguration = MAT::FromJSON(hostConfig); +*/ + +const char* guestConfig = JSON_CONFIG( + { + "config" : { + "host" : "C-API-Host", + "scope" : "*" + }, + "stats" : { + "interval" : 0 + }, + "name" : "C-API-Guest", + "version" : "1.0.0", + "primaryToken" : "7c8b1796cbc44bd5a03803c01c2b9d61-b6e370dd-28d9-4a52-9556-762543cf7aa7-6991", + "maxTeardownUploadTimeInSec" : 0, + "hostMode" : false, + "minimumTraceLevel" : 0, + "sdkmode" : 0, + "disableZombieLoggers" : true + }); + +const char* guestConfigIsolation = JSON_CONFIG( + { + "config" : { + "host" : "C-API-Host", + "scope" : "-" + }, + "stats" : { + "interval" : 0 + }, + "name" : "C-API-Guest2", + "version" : "1.0.0", + "primaryToken" : "ffffffffffffffffffffffffffffffff-ffffffff-ffff-ffff-ffff-ffffffffffff-ffff", + "maxTeardownUploadTimeInSec" : 0, + "hostMode" : false, + "minimumTraceLevel" : 0, + "sdkmode" : 0, + "disableZombieLoggers" : true + }); + +std::time_t now = time(0); + +MAT::time_ticks_t ticks(&now); + +evt_prop guestContext[] = TELEMETRY_EVENT( + _STR("ext.app.id", "com.Microsoft.Clippy"), + _STR("ext.app.ver", "1.0.0"), + _STR("ext.app.locale", "en-US"), + _STR("ext.net.cost", "Unmetered"), + _STR("ext.net.type", "QuantumLeap")); + +// C-style definition of a guest event that contains iKey +evt_prop guestEvent[] = TELEMETRY_EVENT( + // Part A/B + _STR(COMMONFIELDS_EVENT_NAME, EVENT_NAME_GUEST), // Event name + _INT(COMMONFIELDS_EVENT_TIME, static_cast(now * 1000L)), // Epoch time + _DBL("popSample", 100.0), // Effective sample rate + _STR(COMMONFIELDS_IKEY, TEST_TOKEN), // iKey to send this event to +// _INT(COMMONFIELDS_EVENT_PRIORITY, static_cast(EventPriority_Immediate)), +// _INT(COMMONFIELDS_EVENT_LATENCY, static_cast(EventLatency_Max)), + _INT(COMMONFIELDS_EVENT_LEVEL, DIAG_LEVEL_REQUIRED), + // Part C + _STR("strKey", "value1"), + _INT("intKey", 12345), + _DBL("dblKey", 3.14), + _BOOL("boolKey", true), + _GUID("guidKey", "{01020304-0506-0708-090a-0b0c0d0e0f00}"), + _TIME("timeKey", ticks.ticks)); // .NET ticks + +// C-style definition of a guest event that uses its client instance iKey +evt_prop guestEventIsolated[] = TELEMETRY_EVENT( + // Part A/B + _STR(COMMONFIELDS_EVENT_NAME, EVENT_NAME_GUEST), // Event name + _INT(COMMONFIELDS_EVENT_TIME, static_cast(now * 1000L)), // Epoch time + _DBL("popSample", 100.0), // Effective sample rate + _STR(COMMONFIELDS_IKEY, DUMMY_TOKEN), // iKey to send this event to + // _INT(COMMONFIELDS_EVENT_PRIORITY, static_cast(EventPriority_Immediate)), // <-- Useful for realtime force-push + // _INT(COMMONFIELDS_EVENT_LATENCY, static_cast(EventLatency_Max)), + _INT(COMMONFIELDS_EVENT_LEVEL, DIAG_LEVEL_REQUIRED), + // Part C + _STR("strKey", "value1"), + _INT("intKey", 12345), + _DBL("dblKey", 3.14), + _BOOL("boolKey", true), + _GUID("guidKey", "{01020304-0506-0708-090a-0b0c0d0e0f00}"), + _TIME("timeKey", ticks.ticks)); // .NET ticks + +////////////////////////////////////////////////////////////////////////////////////////// +// HOST TEST +////////////////////////////////////////////////////////////////////////////////////////// +void createHostCpp() +{ + auto& cfg = LogManager::GetLogConfiguration(); + cfg["name"] = "C-API-Host"; + cfg["version"] = "1.0.0"; + cfg["config"]["host"] = "C-API-Host"; + cfg["hostMode"] = true; + cfg["primaryToken"] = TEST_TOKEN; + cfg[CFG_STR_COLLECTOR_URL] = COLLECTOR_URL_PROD; + cfg["stats"]["interval"] = 0; // no stats events + cfg["maxTeardownUploadTimeInSec"] = 0; // fast teardown + cfg["disableZombieLoggers"] = true; + + totalEvents = 0; + debugListener.OnLogX = [&](::CsProtocol::Record& record) + { + totalEvents++; + EXPECT_EQ(record.name, EVENT_NAME_HOST); // Verify event name + auto recordTimeTicks = MAT::time_ticks_t(record.time); // Verify event time + EXPECT_EQ(record.time, int64_t(recordTimeTicks.ticks)); + std::string iToken_o = "o:"; + iToken_o += TEST_TOKEN; + EXPECT_THAT(iToken_o, testing::HasSubstr(record.iKey)); // Verify event iKey + ASSERT_STREQ(record.data[0].properties["strKey"].stringValue.c_str(), "value1"); // Verify string + ASSERT_EQ(record.data[0].properties["intKey"].longValue, 12345); // Verify integer + ASSERT_EQ(record.data[0].properties["dblKey"].doubleValue, 3.14); // Verify double + ASSERT_EQ(record.data[0].properties["boolKey"].longValue, 1); // Verify boolean + + ASSERT_EQ(record.extDevice[0].localId, "a:4318b22fbc11ca8f"); // Verify ext.device.localId + ASSERT_EQ(record.extProtocol[0].devMake, "Microsoft"); // NOTE the schema quirk == ext.device.make + ASSERT_EQ(record.extProtocol[0].devModel, "Clippy"); // NOTE the schema quirk == ext.device.model + ASSERT_EQ(record.extOs[0].name, "MS-DOS"); // Verify ext.os.name + ASSERT_EQ(record.extOs[0].ver, "2100"); // Verify ext.os.ver +#ifdef ALLOW_RECORD_DECODER + // Transform to JSON and print + std::string s; + exporters::DecodeRecord(record, s); + printf( + "*************************************** Event %u ***************************************\n%s\n", + totalEvents, + s.c_str()); +#endif + }; + + // C++ syntax for populating Common Part A properties in context. + const auto logger = LogManager::Initialize(TEST_TOKEN, cfg); + EXPECT_NE(logger, nullptr); + + // Populate common Part A ext.* properties using `SetCommonField` API + // These properties would be inherited by Guest instances. + const auto context = LogManager::GetSemanticContext(); + context->SetCommonField("ext.device.localId", "a:4318b22fbc11ca8f"); + context->SetCommonField("ext.device.make", "Microsoft"); + context->SetCommonField("ext.device.model", "Clippy"); + context->SetCommonField("ext.os.name", "MS-DOS"); + context->SetCommonField("ext.os.ver", "2100"); + + const auto instance = LogManager::GetInstance(); + EXPECT_NE(instance, nullptr); + + LogManager::AddEventListener(EVT_LOG_EVENT, debugListener); + + for (size_t i = 0; i < maxEventsCount; i++) + { + EventProperties props{ + EVENT_NAME_HOST, + {{COMMONFIELDS_EVENT_TIME, static_cast(now * 1000L)}, // Epoch time + {"popSample", 100.0}, // Effective sample rate + {COMMONFIELDS_EVENT_LEVEL, DIAG_LEVEL_REQUIRED}, + {"strKey", "value1"}, + {"intKey", 12345}, + {"dblKey", 3.14}, + {"boolKey", static_cast(true)} + } + }; + logger->LogEvent(props); + } + + EXPECT_EQ(totalEvents, maxEventsCount); + + // Remove debug listener + LogManager::RemoveEventListener(EVT_LOG_EVENT, debugListener); +} + +////////////////////////////////////////////////////////////////////////////////////////// +// GUEST TEST +////////////////////////////////////////////////////////////////////////////////////////// +void createGuest() +{ + totalEvents = 0; + debugListener.OnLogX = [&](::CsProtocol::Record& record) + { + totalEvents++; + EXPECT_EQ(record.name, EVENT_NAME_GUEST); // Verify event name + auto recordTimeTicks = MAT::time_ticks_t(record.time); // Verify event time + EXPECT_EQ(record.time, int64_t(recordTimeTicks.ticks)); + + ASSERT_STREQ(record.data[0].properties["strKey"].stringValue.c_str(), "value1"); // Verify string + ASSERT_EQ(record.data[0].properties["intKey"].longValue, 12345); // Verify integer + ASSERT_EQ(record.data[0].properties["dblKey"].doubleValue, 3.14); // Verify double + ASSERT_EQ(record.data[0].properties["boolKey"].longValue, 1); // Verify boolean + auto guid = record.data[0].properties["guidKey"].guidValue[0].data(); + auto guidStr = GUID_t(guid).to_string(); + std::string guidStr2 = "01020304-0506-0708-090a-0b0c0d0e0f00"; + ASSERT_STRCASEEQ(guidStr.c_str(), guidStr2.c_str()); // Verify GUID + ASSERT_EQ(record.data[0].properties["timeKey"].longValue, (int64_t)ticks.ticks); // Verify time + + // Host context properties must remain the same + ASSERT_EQ(record.extDevice[0].localId, "a:4318b22fbc11ca8f"); // Verify ext.device.localId + ASSERT_EQ(record.extProtocol[0].devMake, "Microsoft"); // NOTE the schema quirk == ext.device.make + ASSERT_EQ(record.extProtocol[0].devModel, "Clippy"); // NOTE the schema quirk == ext.device.model + ASSERT_EQ(record.extOs[0].name, "MS-DOS"); // Verify ext.os.name + ASSERT_EQ(record.extOs[0].ver, "2100"); // Verify ext.os.ver + + // These new properties got appended by Guest + ASSERT_EQ(record.extApp[0].id, "com.Microsoft.Clippy"); // Verify ext.app.id + ASSERT_EQ(record.extApp[0].ver, "1.0.0"); // Verify ext.app.ver + ASSERT_EQ(record.extApp[0].locale, "en-US"); // Verify ext.app.locale + ASSERT_EQ(record.extNet[0].cost, "Unmetered"); // Verify ext.net.cost + ASSERT_EQ(record.extNet[0].type, "QuantumLeap"); // Verify ext.net.type +#ifdef ALLOW_RECORD_DECODER + // Transform to JSON and print + std::string s; + exporters::DecodeRecord(record, s); + printf( + "*************************************** Event %u ***************************************\n%s\n", + totalEvents, + s.c_str()); +#endif + }; + + // Keep host LogManager running and attach Guest to it. + const auto guestHandle = evt_open(guestConfig); + ASSERT_NE(guestHandle, 0); + evt_pause(guestHandle); + const auto client = capi_get_client(guestHandle); + ASSERT_NE(client, nullptr); + ASSERT_NE(client->logmanager, nullptr); + + // Use parent LogManager context to append additional context variables. + evt_set_logmanager_context(guestHandle, guestContext); + + client->logmanager->AddEventListener(EVT_LOG_EVENT, debugListener); + + for (size_t i = 0; i < maxEventsCount; i++) + { + evt_log(guestHandle, guestEvent); + } + + EXPECT_EQ(totalEvents, maxEventsCount); + evt_flush(guestHandle); + + // Remove debug listener + client->logmanager->RemoveEventListener(EVT_LOG_EVENT, debugListener); + + // Close guest + evt_close(guestHandle); + ASSERT_EQ(capi_get_client(guestHandle), nullptr); +} + +////////////////////////////////////////////////////////////////////////////////////////// +// GUEST TEST ISOLATED +////////////////////////////////////////////////////////////////////////////////////////// +void createGuestIsolated() +{ + // Reset total events + totalEvents = 0; + debugListener.OnLogX = [&](::CsProtocol::Record& record) + { + totalEvents++; + EXPECT_EQ(record.name, EVENT_NAME_GUEST); // Verify event name + auto recordTimeTicks = MAT::time_ticks_t(record.time); // Verify event time + EXPECT_EQ(record.time, int64_t(recordTimeTicks.ticks)); + std::string iToken_o = "o:"; + iToken_o += DUMMY_TOKEN; + EXPECT_THAT(iToken_o, testing::HasSubstr(record.iKey)); // Verify event iKey + ASSERT_STREQ(record.data[0].properties["strKey"].stringValue.c_str(), "value1"); // Verify string + ASSERT_EQ(record.data[0].properties["intKey"].longValue, 12345); // Verify integer + ASSERT_EQ(record.data[0].properties["dblKey"].doubleValue, 3.14); // Verify double + ASSERT_EQ(record.data[0].properties["boolKey"].longValue, 1); // Verify boolean + auto guid = record.data[0].properties["guidKey"].guidValue[0].data(); + auto guidStr = GUID_t(guid).to_string(); + std::string guidStr2 = "01020304-0506-0708-090a-0b0c0d0e0f00"; + ASSERT_STRCASEEQ(guidStr.c_str(), guidStr2.c_str()); // Verify GUID + ASSERT_EQ(record.data[0].properties["timeKey"].longValue, (int64_t)ticks.ticks); // Verify time + + // Host context properties are NOT shared this time. We run in isolation. + ASSERT_NE(record.extDevice[0].localId, "a:4318b22fbc11ca8f"); // Verify ext.device.localId + ASSERT_NE(record.extProtocol[0].devMake, "Microsoft"); // NOTE the schema quirk == ext.device.make + ASSERT_NE(record.extProtocol[0].devModel, "Clippy"); // NOTE the schema quirk == ext.device.model + ASSERT_NE(record.extOs[0].name, "MS-DOS"); // Verify ext.os.name + ASSERT_NE(record.extOs[0].ver, "2100"); // Verify ext.os.ver + + // These new properties got appended by Guest + ASSERT_EQ(record.extApp[0].id, "com.Microsoft.Clippy"); // Verify ext.app.id + ASSERT_EQ(record.extApp[0].ver, "1.0.0"); // Verify ext.app.ver + ASSERT_EQ(record.extApp[0].locale, "en-US"); // Verify ext.app.locale + ASSERT_EQ(record.extNet[0].cost, "Unmetered"); // Verify ext.net.cost + ASSERT_EQ(record.extNet[0].type, "QuantumLeap"); // Verify ext.net.type +#ifdef ALLOW_RECORD_DECODER + // Transform to JSON and print + std::string s; + exporters::DecodeRecord(record, s); + printf( + "*************************************** Event %u ***************************************\n%s\n", + totalEvents, + s.c_str()); +#endif + }; + + // Keep host LogManager running and attach Guest to it. + const auto guestHandle = evt_open(guestConfigIsolation); + ASSERT_NE(guestHandle, 0); + evt_pause(guestHandle); + const auto client = capi_get_client(guestHandle); + ASSERT_NE(client, nullptr); + ASSERT_NE(client->logmanager, nullptr); + + // Guest uses its own context and not the parent context. + evt_set_logger_context(guestHandle, guestContext); + + client->logmanager->AddEventListener(EVT_LOG_EVENT, debugListener); + + for (size_t i = 0; i < maxEventsCount; i++) + { + evt_log(guestHandle, guestEventIsolated); + } + + EXPECT_EQ(totalEvents, maxEventsCount); + evt_flush(guestHandle); + + // Remove debug listener + client->logmanager->RemoveEventListener(EVT_LOG_EVENT, debugListener); + + // Close guest + evt_close(guestHandle); + ASSERT_EQ(capi_get_client(guestHandle), nullptr); +} + +// Create and teardown host. +// Validate that events get emitted with right props. +TEST(HostGuestTest, C_API_CreateHost) +{ + createHostCpp(); + LogManager::FlushAndTeardown(); +} + +// Verify that we can create Host + Guest pair. +// Validate that events get emitted with right props. +TEST(HostGuestTest, C_API_CreateGuest) +{ + createHostCpp(); + createGuest(); + LogManager::FlushAndTeardown(); +} + +// Verify that we properly deallocated all resources: +// same Host + Guest pair can be recreated again. +// Validate that events get emitted with right props. +TEST(HostGuestTest, C_API_CreateGuestAgain) +{ + createHostCpp(); + createGuest(); + LogManager::FlushAndTeardown(); +} + +// Create Host with "sandboxed" Guest with limited scope +// that does not inherit its Host context. +// Validate that events get emitted with right props. +TEST(HostGuestTest, C_API_CreateGuestIsolated) +{ + createHostCpp(); + createGuestIsolated(); + LogManager::FlushAndTeardown(); +} + +// TEST_PULL_ME_IN(HostGuestTests)