[EPIC]: Track migration process and display it in a dashboard

[nfx]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

There's no easy way to know what still needs to be migrated to UC within a given workspace

Proposed Solution

• [FEATURE] Create migration-progress scheduled daily workflow #2578
• [FEATURE] Support rerunning crawlers #2576
  • [FEATURE] Crawlers should be able to write to another catalog in addition to the hive_metastore #2575
• [FEATURE] Update the inventory with the migration-progress crawled objects #2574
• [FEATURE] Persist the migration-progress crawled objects in the ucx.history table #2573
  • [FEATURE] Create a ucx catalog via a cli-command #2571
  • [FEATURE] Create a ucx.history table #2572
  • [FEATURE] Create a ucx.workflow_runs table #2600
  • [FEATURE] Create a ucx.errors table #2603
  • [BUG]: Dashboard and workflow refresh during the migration-progress-experimental job do not update the historical log #3237
  • [TODO] bring back self.tables_migrator.index() #3153
• [FEATURE] Skip the migration-progress run when the ucx catalog does not exists #2577
• [FEATURE] Skip the migration-progress run when the assessment job did not run yet #2816
• [FEATURE] Set RemoveAfter property on ucx catalogs in integration test #2594
  • Pre-requisite: Change in watchdog
• [FEATURE] Run the workflow static analysis as part of the migration progress workflow #2595
• [FEATURE] Visualize migration process in dashboard #2596
  • [FEATURE] Visualize dashboard migration progress in migration-progress dashboard #3045
  • [FEATURE] Visualize job migration progress in migration-progress dashboard #3517
• [FEATURE] Encode dataclasses to a history log entry #3064
  • [FEATURE]: History log encoder for clusters #3057
  • [FEATURE]: History log encoder for grants #3058
  • [FEATURE]: History log encoder for jobs #3059
  • [FEATURE]: History log encoder for pipelines #3060
  • [FEATURE]: History log encoder for tables #3061
  • [FEATURE]: History log encoder for udfs #3062
  • [FEATURE]: History log encoder for cluster policies #3063
• Dashboards
  • [FEATURE] Redash Dashboards Crawler #3366
  • [FEATURE] Lakeview Dashboards Crawler #3367
  • [FEATURE] History log encoder for Redash dashboards #3368
  • [FEATURE] History log encoder for Lakeview dashboards #3369
• UsedTables
  • Add used tables to table/views as failure when referencing non-migrated table

Potential challenges

• [FEATURE] Let crawlers support append to tables #2597
• but for migration progress purposes, we need to overwrite the tables. or add another column with a timestamp and modify "fetch latest" queries to fetch the latest timestamp of the snapshot. fetching the latest timestamp from the snapshot allows to build a bar-chart widget to see how fast migration progresses, but we don't really care about it it. if we do fetch-latest-timestamp, all our views and dashboards would become a bit more complicated. but that's fine. --> Decision is made to keep the current ucx inventory and store history in a separate table (see proposed solution above)
• let's keep the status of migration progress in HMS (for now), but we can change this decision in a few weeks. --> Decision is made to store the migration process in a ucx catalog.

Migration process crawlers

Assessment tasks that make sense to re-run on migration-progress workflow:

• crawl_tables
• assess_jobs - potentially harden the code there as well
• assess_clusters - potentially harden as well
• assess_pipelines - potentially harden as well
• crawl_cluster_policies
• assess_global_init_scripts

not to be re-run:

• crawl_mounts - we already pre-created external locations
• setup_tacl - we don't need to crawl grants
• crawl_grants - no need to, i think
• estimate_table_size_for_migration - most likely not necessary
• guess_external_locations - we already migrated external locations by this point
• assess_incompatible_submit_runs - not going to be necessary in september
• workspace_listing - we are going to analyse only those notebooks that are part of jobs in the scope of static analysis
• crawl_permissions - we expect permissions to already be migrated
• crawl_groups - we expect groups to already be migrated

Additional Context

• [FEATURE]: Tag migrated clusters/jobs/dlts and provide a dashboard of percentage of those that were migrated or remaining #2073

[asnare]

During team discussion we talked about this in the context of refreshing the inventory, which is slightly different to the proposed solution above but maybe represents an acceptable alternative.

A complication that we discussed is that in the current crawling code there's not always a clear separation between inventory and findings: sometimes findings are attached to inventory items during the crawling process and these are stored in the same table. Teasing out how to separate these is likely to be quite time-consuming.

[asnare]

After some more deliberation, we've narrowed the requirements for this a bit. The way this feature is going to work, in blunt terms:

• Essentially the assessment workflow will be re-run (by default) daily.
• All crawlers and so forth will refresh their state.

There's still an open question about the groups and permissions crawlers: these can be time-consuming and are intended to be both an early (and relatively simple) part of the migration process: once completed is there any point to re-running the crawlers as part of the assessment?

At the moment I'm leaning towards not re-running them by default, but I think there needs to be a configuration option to include them:

• Group mapping is planned as part of the crawler and persisted as part of the inventory. Changing the mapping strategy at the moment means re-running the crawler to ensure the mapping is updated.
• Permissions across a landscape can be fairly dynamic and need refreshing. (For security reasons I'd prefer that we not need to scan them at all because the crawl/migrate process is racy, but until the experimental workflow is the only way to do it I think we're stuck with the current approach.)
• When errors occur during migration, one of the simplest mitigations is to remove the problematic group or permission. In this case it's best if the inventory can be updated to reflect the new situation.

The idea of keeping a history so that progress can be tracked still needs to be refined.