Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set correct path for uploaded file in sim config #1054

Merged
merged 9 commits into from
Dec 14, 2023

Conversation

mattdailis
Copy link
Collaborator

@mattdailis mattdailis commented Dec 11, 2023

Closes #1046

Description

Aerie allows the mission model to declare a sim config parameter of type Path. The Aerie UI displays this parameter type with a Browse button that allows the user to upload a file.

The aerie gateway changes the names of uploaded files to ensure uniqueness (e.g. if I upload banananation.jar multiple times, I do not typically want it to overwrite other files with the same name).

The Aerie UI sets the name of the Path parameter in the sim config to the name of the original file, not the modified name generated by the gateway. This path also omits the absolute path prefix of the location at which the aerie_file_store is mounted in the aerie_merlin and aerie_merlin_worker_* containers.

As a consequence of the above, there is no way for the mission model to take the provided Path parameter and resolve it to the correct file location in the file system. This negates the utility of using Path parameters in a sim config.

Problem Statement

A mission model should be able to use a sim config parameter of type Path to find a file that the planner uploaded using the UI's sim config form.

Approach

The most targeted fix (i.e. a fix that only touches one repo) is to update the UI to fill in the Path parameter with the correct absolute path to the uploaded file. It can do this with the following steps:

  1. After uploading the files, keep track of the uploaded file ids
  2. Issue a follow-up query to get the generated names of those files
  3. Traverse the sim config, finding all Path parameters and replace them with a hard-coded prefix followed by the generated filename, e.g:
/usr/src/app/merlin_file_store/my-file-name-1702092509728-CMvkjzriKBQvuc.json

A drawback of this approach is that it hard-codes the mount point /usr/src/app/merlin_file_store/. Perhaps an easy step we could take would be to set this in an environment variable.

  • TODO set /usr/src/app/merlin_file_store/ in an environment variable

Verification

As a manual test, upload a banananation mission model, and upload a text file to the initialDataPath sim config parameter. Run a simulation, and observe the /data/line_count resource - it should correctly report the number of lines in your uploaded file.

Future work

  • The fact that the UI needs to know the mount point of the merlin_file_store feels a bit strange - in theory, the simulation context should be able to add that prefix to every provided path parameter.
  • There is no way to choose an existing file, or even see the existing set of uploaded files. It may be worth looking into some UI to manage uploaded files

@mattdailis mattdailis added the fix A bug fix label Dec 11, 2023
@mattdailis mattdailis requested a review from duranb December 11, 2023 18:37
@mattdailis mattdailis self-assigned this Dec 11, 2023
@mattdailis mattdailis requested a review from a team as a code owner December 11, 2023 18:37
@mattdailis mattdailis force-pushed the fix/upload-files-sim-config branch from 48dd6af to 5a58388 Compare December 11, 2023 21:44
Copy link
Collaborator

@duranb duranb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good for the most part other than one comment that I left!

src/utilities/effects.ts Outdated Show resolved Hide resolved
@mattdailis mattdailis force-pushed the fix/upload-files-sim-config branch from 5a58388 to 284070b Compare December 13, 2023 17:28
src/utilities/effects.ts Outdated Show resolved Hide resolved
Copy link
Collaborator

@duranb duranb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯 🥇 🚀

@mattdailis mattdailis merged commit 3a86919 into develop Dec 14, 2023
4 checks passed
@mattdailis mattdailis deleted the fix/upload-files-sim-config branch December 14, 2023 16:02
JosephVolosin pushed a commit that referenced this pull request Aug 20, 2024
* Set correct path for uploaded file in sim config

* Make eslint happy

* Undo unintentional change

* Provide type for filenames

* Add unit test and fix bug found by unit test

* Alphabetically sort keys for eslint

* Add PUBLIC_AERIE_FILE_STORE_PREFIX env var

* Reduce use of continue where simple if statement suffices

* Rename filenames to generatedFilenames and pathsToReplace
JosephVolosin pushed a commit that referenced this pull request Oct 21, 2024
* Set correct path for uploaded file in sim config

* Make eslint happy

* Undo unintentional change

* Provide type for filenames

* Add unit test and fix bug found by unit test

* Alphabetically sort keys for eslint

* Add PUBLIC_AERIE_FILE_STORE_PREFIX env var

* Reduce use of continue where simple if statement suffices

* Rename filenames to generatedFilenames and pathsToReplace
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fix A bug fix
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Match filepath for uploaded files in simulation configuration
2 participants