Define Runtime Metrics

[brianlee731]

Not in any particular order:

Define data, metrics we want to capture

• Stage in, stage out, processing
• CPU, Memory usage, Network IO
• Job Inputs (stac + params)
• Job Outputs
• CWL run logs
  Storage Options for captured metrics
• S3? Upload as a part of the outputs?
• Kibana/EKS
• Needs to be "forever"
  Document how to capture these logs
  Are there nearterm 'wins' for this? Can we uplaod the CWL logs to S3 after a job based on the airflow Job/OGC process id?

Original Note:

1. Add runtime metrics to job output (e.g. the cwl logs for running algorithm vs 'job')
2. Job details/metrics should be stored for a long time.
3. Document what metrics we're capturing and how Users can access it

[mike-gangl]

How do these individual job/process metrics relate to #213

[rtapella]

What's the objective behind these measurements? What are we trying to monitor/learn? (e.g. some may just be performance/load tracking, cost tracking,etc. and some may be related to user-success)