Finding Root to High Firestore Costs

[anthonypetersen]

The main issue still in our monthly GCP costs is how many Firestore read operations are taking place each month (last month there were almost 3 Billion). I know that one main culprit is our backup processes, but I would like to start getting an understanding of how many documents are being read in each function within Firestore.js.

My initial idea is pretty simple: in every function in Firestore.js that reads documents from Firestore, we simply console log the size of the snap shot after it completes and include in the log which function read the documents. At the end of the month we can do a simple log export and generate a document count by function to get an idea of where we should focus our attention in the future.

To clarify, I'm not aiming to solve the issue right away, I just want to do something that takes minimal effort that can start to give us an idea of what the best approach might be in the future. I know my idea is pretty elementary so I wanted to start a quick discussion to see if anyone else had a different idea.

[anthonypetersen]

@we-ai @JoeArmani @Davinkjohnson

[Davinkjohnson]

Tony, is there a way to add to your log function which site called the API to perform the reads? (internal calls, vs calls from sites) I like the idea of trying to understand where the largest volume of calls is coming from, and it would be great if we could add a little extra context.

[anthonypetersen]

We might have to get creative with that if we don't want to add code everywhere passing around variables, but it's something we can figure out.

[JoeArmani]

We could look at logging the site code just after the auth object is built?

[anthonypetersen]

That would work, we would just need to tie the session IDs together when exporting the data.

[we-ai]

This is a good start to find root causes.
For logs, we'll need a unique string for filtering, and also info about called functions. Here is one possible format: Documents read in Firestore: 10; function called: updateParticipantRecord.

[JoeArmani]

It may be useful to maintain a Firestore doc for each tracked function and .increment(size) each time. It would be a big one month spike in reads/writes to track this though.

{
  funcName: <string> 
  totalReads: <number> // increment this with .size() each time a function returns
  totalWrites: <number> // increment...
  siteUsage: {
    // counter for each site by site code, also increment where applicable
    <siteCode>: {
      reads: <number>
      writes: <number>
    }
  } 
}

Other thoughts:

updateParticipantData:
updateParticipantData() -> checkDerivedVariables().
This fetches the participant once for each function call.
-I think we can merge the updated participant data, into the participant data obj, and pass that into checkDerivedVariables.
This would cut participant reads in half for anything using updateParticipantData (apps and API calls) and speed up the process.

Home collections:
This currently fetches all participants that have given samples since 4/1/24 every time the ‘Print Labels’ process runs, then filters the ones with existing mouthwash samples before returning the data. We could set an initializer property for participants who qualify and haven’t given a mouthwash sample when their baseline specimens are recorded…then only query those participants (a significantly narrower set) and only query them once (instead of over and over each time the process runs).

December 2023 release:
It looks like something caused a big spike in Jan. Is that at all related to a participant spike, or should we inspect the December release for clues?

[anthonypetersen]

Definitely like the idea to talk the additional call to fetch user data out of the checkDerivedVariables process.