Ingest QOL tweaks

[fvankrieken]

All coming up as part of #1334 and the associated datasets moving over to ingest.

1. originally moved too much logic to init.py, now mainly just moves cli target and ingest function (the function that runs the whole ingest process) to init so that it's easily accessible
2. make use of the .lifecycle dir for "staging" the process
3. make use of the .lifecycle dir to create a local file system reminiscent of edm-recipes/datasets where the outputs are moved to. Just seemed to be nice for troubleshooting (slash running without a connection to s3)
4. ability to provide the path to a source dataset file when running via cli. Just overrides whatever source is found in the template
5. just make one "archive_dataset". @alexrichey based on our pairing yesterday though, maybe I'll scrap this - you could argue we want a different connector class for "raw" datasets and our parquet datasets, since we interact with them in fairly different ways

[codecov]

Codecov Report

Attention: Patch coverage is 69.76744% with 13 lines in your changes missing coverage. Please review.

Project coverage is 72.20%. Comparing base (32d4887) to head (4afeef4).
Report is 6 commits behind head on main.

Files with missing lines	Patch %	Lines
dcpy/lifecycle/scripts/validate_ingest.py	0.00%	9 Missing ⚠️
dcpy/lifecycle/_cli.py	0.00%	2 Missing ⚠️
.../lifecycle/scripts/ingest_with_library_fallback.py	0.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1454      +/-   ##
==========================================
+ Coverage   72.18%   72.20%   +0.02%     
==========================================
  Files         118      119       +1     
  Lines        6133     6145      +12     
  Branches      725      726       +1     
==========================================
+ Hits         4427     4437      +10     
- Misses       1550     1552       +2     
  Partials      156      156              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sf-dcp]

a bit of a nit: naming this var as INGEST_STAGING_DIR feels more intuitive to me

[sf-dcp]

same comment for OUTPUT_DIR on line 12 below

[sf-dcp]

Need a fix here: we don't need the line 113 sinc you are using the STAGING_DIR path. Also, I would rename ingest_parent_dir to ingest_staging_dir in the fn signuture

[fvankrieken]

I should clean this up, but am still going to provide a folder for this to run in that's separated from the "usual" ingest folders, just to keep it separate from anyone running an "official" job on their own comp

[fvankrieken]

See the latest commit for renaming - mainly, worked to distinguish any path that's set for ingest (i.e. a variable or override that's set for these sort of "parent" folders that contain many datasets) vs any folder that's specific to the dataset being run.

4afeef4

It's still a little more complicated than it should be but given how this operates now I think it makes sense

[fvankrieken]

This is how the 'validate ingest" script outputs stuff now

[sf-dcp]

replace "config.json" with CONFIG_FILENAME that you defined above?

[fvankrieken]

ty - this looks like a rebase mistake, the same line is right below it but fixed

[sf-dcp]

Curious, what is the need for 4th commit? In which case you wouldn't want to use the template for defining local source?

[fvankrieken]

Curious, what is the need for 4th commit? In which case you wouldn't want to use the template for defining local source?

Great question. Rationale is that for something that's a local csv (or emailed csv) that might have a different filename each time. I don't think that it makes sense to have to commit changes (or just modify the template) to archive it, it's something that should be valid to override at runtime