Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to the UI for importing and enriching data [MAP-347] [MAP-78] [MAP-77] #154

Merged
merged 29 commits into from
Dec 16, 2024

Conversation

Moggach
Copy link

@Moggach Moggach commented Dec 10, 2024

Description

This PR adds:
if an import or update is in a queue behind others indicate this in the UI
a progress bar when importing and updating records
when an job is going to take longer than 5 minutes tell the user it is finished via email

Motivation and Context

Addresses issues MAP-347,, MAP-77, MAP-78

How Can It Be Tested?

Run the branch locally
Import and/or update a data source
Testing progress bar:

  • If working with smaller data sets update the IMPORT_UPDATE_ALL_BATCH_SIZE to a small size e.g. 2
  • Observe a progress bar during the import or update

Testing job queue:

  • Try requesting more than one import/update job at once and observe that the UI indicates that you are in a queue

Testing email notification:

  • set timedelta in get_scheduled_batch_job_progress to something low (e.g. 1 minute)
  • observe that the UI indicates you will be sent an email
  • observe the email printed to the console

… statuses of to do or doing and add it to graphql query for use in front end
…alDataSource model for use in updating progress bar in UI
…y and make progress bar component display even when job is not in progress
…_progress method and update BatchJobProgress types
…bjects in ExternalDataSourceInspectPage and display it in the UI
…a message in the UI when a job will take more than 5 minutes
@commonknowledge-bot commonknowledge-bot deployed to improvements-to-import-enrich-data-ui - meep-database PR #154 December 10, 2024 18:53 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-backend PR #154 December 10, 2024 18:54 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-backend PR #154 December 10, 2024 18:55 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-frontend PR #154 December 10, 2024 18:55 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-worker PR #154 December 10, 2024 18:55 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-worker PR #154 December 10, 2024 19:02 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-backend PR #154 December 10, 2024 19:02 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-backend PR #154 December 12, 2024 17:51 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-worker PR #154 December 12, 2024 17:51 — with Render Destroyed
@Moggach Moggach marked this pull request as ready for review December 12, 2024 17:51
@Moggach
Copy link
Author

Moggach commented Dec 12, 2024

I've refactored the timing check to use estimated time only as we need to make the decision before the job is finished

@janbaykara janbaykara changed the title Improvements to the UI for importing and enriching data Improvements to the UI for importing and enriching data [MAP-347] Dec 12, 2024
@janbaykara janbaykara changed the title Improvements to the UI for importing and enriching data [MAP-347] Improvements to the UI for importing and enriching data [MAP-347] [MAP-78] Dec 12, 2024
Copy link

linear bot commented Dec 12, 2024

@janbaykara
Copy link
Member

OK. I think early next week @joaquimds or I just need to test this robustly, fix whatever's required and then merge. It might be right as rain already though, looking good @Moggach!

@janbaykara janbaykara changed the title Improvements to the UI for importing and enriching data [MAP-347] [MAP-78] Improvements to the UI for importing and enriching data [MAP-347] [MAP-78] [MAP-77] Dec 12, 2024
Copy link

linear bot commented Dec 12, 2024

@commonknowledge-bot commonknowledge-bot deployed to improvements-to-import-enrich-data-ui - meep-database PR #154 December 16, 2024 13:02 — with Render Active
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-backend PR #154 December 16, 2024 13:02 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-backend PR #154 December 16, 2024 13:02 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-frontend PR #154 December 16, 2024 13:02 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot had a problem deploying to improvements-to-import-enrich-data-ui - meep-intelligence-hub-worker PR #154 December 16, 2024 13:02 — with Render Failure
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-frontend PR #154 December 16, 2024 13:02 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-backend PR #154 December 16, 2024 13:11 — with Render Destroyed
@commonknowledge-bot commonknowledge-bot temporarily deployed to improvements-to-import-enrich-data-ui - meep-intelligence-hub-worker PR #154 December 16, 2024 13:11 — with Render Destroyed
@janbaykara janbaykara merged commit 8d8b0a0 into main Dec 16, 2024
4 checks passed
@janbaykara janbaykara deleted the improvements-to-import-enrich-data-ui branch December 16, 2024 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants