Use a github action to centralize gin data preparation

[h-mayorquin]

While working on #1013 I was having trouble getting the data right from the caches (see the diff in that PR). Centralizing the logic should make this kind of problem easier to deal with.

This is a partial move in the direction of #993.

[pauladkisson]

Looks good, this PR should probably delete update-s3-testing-data.yml to avoid confusion.

[h-mayorquin]

Looks good, this PR should probably delete update-s3-testing-data.yml to avoid confusion.

What possible confusion do you in mind?

[pauladkisson]

Looks good, this PR should probably delete update-s3-testing-data.yml to avoid confusion.

What possible confusion do you in mind?

update-s3-testing-data.yml is disabled, and actions/load-data seems to be a replacement (is it not?), so we should delete update-s3-testing-data.yml to avoid maintaining vestigial workflows that aren't in use.

[h-mayorquin]

Looks good, this PR should probably delete update-s3-testing-data.yml to avoid confusion.

What possible confusion do you in mind?

update-s3-testing-data.yml is disabled, and actions/load-data seems to be a replacement (is it not?), so we should delete update-s3-testing-data.yml to avoid maintaining vestigial workflows that aren't in use.

No, it is not meant to replace it. This assumes the data is already on the s3 and loads it either from the cache or from the s3 if the former fails. At the moment we don't have a mechanism to automatically transfer the data from gin to s3. We should meet at brainstorm that.

[pauladkisson]

No, it is not meant to replace it. This assumes the data is already on the s3 and loads it either from the cache or from the s3 if the former fails.

I see. update-testing-data.yml on roiextractors is more like load-data/action.yml than update-s3-testing-data.yml, so that's why I misunderstood.

At the moment we don't have a mechanism to automatically transfer the data from gin to s3. We should meet at brainstorm that.

gin is very finnicky, so we might just be better off keeping our current (manual) system.

[h-mayorquin]

gin is very finnicky, so we might just be better off keeping our current (manual) system.

What troubles have you experienced? Not that I disagree.

[pauladkisson]

What troubles have you experienced?

Failed downloads mostly.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.58%. Comparing base (36464df) to head (0a82274).
Report is 15 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1095      +/-   ##
==========================================
+ Coverage   90.44%   90.58%   +0.13%     
==========================================
  Files         129      129              
  Lines        8055     8164     +109     
==========================================
+ Hits         7285     7395     +110     
+ Misses        770      769       -1     

Flag	Coverage Δ
unittests	90.58% <ø> (+0.13%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 2 files with indirect coverage changes