Do we need the functionality to test notification specifications (schemas)? (April Release)

[we-ai]

Currently there's no way to test a notification schema after its creation/update. To see whether the spec works correctly, we either wait for the scheduled runs (at 3PM or 8 PM) or manually trigger/run it, but both ways of tests are not ideal because:

• Scheduled runs are too time-limited for testing schemas, and draft schemas are skipped. Creating/updating a schema may require adjusting conditions and checking results multiple times. Scheduled runs only generate results at most twice (3pm and 8pm) per day based on current settings, which is quite limiting. In addition, we may only want to check whether the schema works without sending out notifications (draft mode), but scheduled runs cannot give us such result without really running the schema.
• Manually triggering/running a schema to check the result is flexible in schedules, but it requires extensive communications between ops and dev teams. The communications may delay progress of a schema testing.

To speed up test iterations and minimize communication costs when creating/updating schemas, in SMDB, we may need an option (a button labeled with "Test Run" or "Fake Run" or "Dry Run" or "Trial Run"?) to test the schema under creation/update. The test may generate below results:

• If there's no error running the schema, display count of emails and SMS messages that will be sent. The test only does counts. No actual notifications (emails or SMS messages) should be sent during schema test.
• If an error occurs, display the error so that ops/dev team can quickly fix the error in schema or code.

Please give suggestions @brotzmanmj @anthonypetersen @HanaShiho

[anthonypetersen]

Something like this would definitely be useful, one suggestion I have is to have this point at Firestore as opposed to BQ so there isn't confusion caused by delays in the time it takes data to get from one source to the other.

[we-ai]

Good comment.
BigQuery (BQ) supports flexible and quick queries to allow us efficiently send out notifications. Firestore queries are less so, resulting in speed issues before. For our notification task, 2 main functionalities matter:

• BQ supports join clause which is important to search across multiple tables, but Firestore (NoSQL) doesn't. With the growth of our study, we may soon need a query like find participants ("participants" collection/table) who have lung cancer ("cancerOccurance" ? collection/table) and received notification generated from spec with id "abcd" (notifications collection/table), BQ can get the result quickly with only one query. Using Firestore to achieve the goal will be more complicated and slow.
• BQ supports querying rows with a missing value in a column, but Firestore doesn't search docs with missing values. In our daily notification, for each notification spec, we need result of find participants who meet all conditions in spec but haven't received a notification generated from the spec. This is easy and quick for BQ, but less so for Firestore.

Currently the Firestore data are synced to BQ every 1 (stage and prod) or 2 (dev) hours, and each of the scheduled notifications run once (3 pm or 8 pm) per day, so the data lagging in BQ is not a big issue for notifications. And we can further improve Firestore to BQ data syncs, as discussed here.

[anthonypetersen]

My main concern is that specification testing might want to be done in real time, so whatever option we provide I think it's important that there not need to be waiting between taking actions with test participants that would allow them to become eligible for a notification and when this testing functionality would show that a spec is working properly.

[we-ai]

Good point.
To me, our production requirements are more important than schema tests. So we may not need to change our production workflow to fit tests.
To be sure the related data are in BQ before testing a schema, currently we have 2 options: 1) get data ready in Firebase at least 2 hours before the schema tests; 2) manually force data syncs from Firestore to BigQuery.

If in the future we need a functionality like I need update data of participants A, B, C etc., and immediately run schemas X, Y, Z to test the changed data, it's doable and we can discuss the implementation details.

Just my thoughts. Your comments are welcome.

[anthonypetersen]

Sounds good, we can cross that bridge when we get there. To be fair, I only really foresaw that kind of scenario in dev when we are initially creating the specs.

[we-ai]

I agree the scenario is rare. A forced data sync or local code run should do it.

[we-ai]

The PR implementing dry run functionality is here. @bransteitterbr @HanaShiho

[we-ai]

Read for dev tests.

[HanaShiho]

Tested in dev, ready for stage

[HanaShiho]

Ready for prod

[we-ai]

PR#640 implemented the dry run functionality in April (2024) release.