[sled-agent] Integrate config-reconciler #8064

jgallagher · 2025-04-29T17:35:10Z

This PR integrates the new sled-agent-config-reconciler crate with sled-agent. It will not currently pass tests due to the reconciler not being completely implemented, but I'd like to get any feedback on this integration work itself (particularly as it pertains to the API of sled-agent-config-reconciler). See the description of #8063 for more context.

There are a couple serious warts with this PR:

The inventory system has not been updated with all the details we need to report for the reconciler. This is a bigger chunk of work because it involves a database migration and touches various bits of Nexus, so I'll do that in a separate PR.
This integration removes most uses of the StorageManager (because its functionality is being absorbed into sled-agent-config-reconciler); however, the storage manager also has a rich set of test support. This PR leaves a couple sled-agent submodules using that test support (support-bundle/storage and zone-bundle). In the long run I think it'd be better to rework these (if there are no remaining production uses of StorageManager), but for now I think this is... okay? Feedback welcome.

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. Fixes #7774.

smklein · 2025-04-29T20:09:52Z

dev-tools/omdb/src/bin/omdb/sled_agent.rs

@@ -34,14 +34,6 @@ enum SledAgentCommands {
    #[clap(subcommand)]
    Zones(ZoneCommands),

-    /// print information about zpools


Are you expecting that inventory will supplant this info? Or are you planning on replacing this access to the sled agent later?

I was expecting that inventory would supplant this. (I think maybe it already has, in practice? I definitely only look at inventory when I'm curious about zpools; I don't think I've ever used these omdb subcommands.)

nexus/reconfigurator/planning/src/system.rs

sled-agent/src/services.rs

sled-agent/src/artifact_store.rs

sled-agent/src/long_running_tasks.rs

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. We don't yet ledger `OmicronSledConfig`s to disk, so we're free to fiddle with the details of its fields without worrying about backwards compatibility. Fixes #7774.

…ig reconciler (#8188) The primary change here is replacing these inventory fields (a subset of `OmicronSledConfig`): ```rust pub omicron_zones: OmicronZonesConfig, pub omicron_physical_disks_generation: Generation, ``` with these: ```rust pub ledgered_sled_config: Option<OmicronSledConfig>, pub reconciler_status: ConfigReconcilerInventoryStatus, pub last_reconciliation: Option<ConfigReconcilerInventory>, ``` Once #8064 lands, all three of these will be filled in meaningfully; as of this PR, only `ledgered_sled_config` is populated. (`reconciler_status` is always `NotYetRun` and `last_reconciliation` is always `None`, since there is no reconciler yet.) The rest of the changes are all fallout from changing inventory: * Update `omdb` printing * Update sled-agent to report the new inventory fields * Update consumers of inventory (tests, reconfigurator planner, one Nexus RPW) - these all just look at `ledgered_sled_config` for now, but will need to be updated on #8064 once other fields are populated * Update database schema, model, and queries (the bulk of the diff). This requires dropping all preexisting collections, since there's no way to migrate from just `omicron_zones` to a full `OmicronSledConfig`. The first few schema migrations take care of this. Before merging I'll go through an upgrade on a racklette and confirm things come back up okay after the schema migration blows away all the pre-update inventory collections. (We think this is fine, but it'd be good to confirm.) But I think this is close enough that it's reviewable. Couple other minor changes that came along for the ride: * Closes #6770 (`inv_sled_omicron_zones` is gone now) * Fixes #8084 (added `image_source` columns to the inventory zone config table, so we don't lose `ImageSource::Artifact { hash }` values reported by sled-agent)

jgallagher · 2025-05-28T18:28:08Z

I'm putting racklette testing notes for this branch plus a few followups in comments on the last of those followups (#8220).

andrewjstone

This is a hard PR to review, given its broad scope. It was made somewhat easier by recognizing a few patterns such as replacing calls to the storage manager with rx channels for disk and datasets.

It all appears correct to me, but again, hard to really tell. I'm sure it was tedious to implement as well :)

Regardless, looks good enough to merge and continue with.

andrewjstone · 2025-05-28T17:58:13Z

sled-agent/api/src/lib.rs

-        method = GET,
-        path = "/datasets",
-    }]
-    async fn datasets_get(


Were these only used by the OMDB commands that got removed?

andrewjstone · 2025-05-28T18:05:22Z

sled-agent/src/support_bundle/storage.rs

    async fn dyn_datasets_config_list(&self) -> Result<DatasetsConfig, Error> {
-        self.datasets_config_list().await.map_err(|err| err.into())
+        // TODO-cleanup This is super gross; add a better API (maybe fetch a


Did you want to clean this up in this PR?

andrewjstone · 2025-05-28T18:32:09Z

nexus-sled-agent-shared/src/inventory.rs

+    /// Given a sled config, produce a reconciler result that sled-agent could
+    /// have emitted if reconciliation succeeded.
+    ///
+    /// This method should only be used by tests and dev tools; real code should


Should we mark this with #[test] and maybe #[cfg(any(test, feature = "testing"))] ?

andrewjstone · 2025-05-28T18:33:44Z

nexus/reconfigurator/planning/src/planner.rs

@@ -214,17 +214,18 @@ impl<'a> Planner<'a> {
        // The sled is not expunged. We have to see if the inventory
        // reflects the parent blueprint disk generation. If it does
        // then we mark any expunged disks decommissioned.
+        //
+        // TODO-correctness We inspect `last_reconciliation` here to confirm


On balance, this seems like the right choice to me. We should know the sled agent has acted before decommissioning.

jgallagher requested review from andrewjstone, davepacheco and smklein April 29, 2025 17:35

jgallagher added a commit that referenced this pull request Apr 29, 2025

OmicronSledConfig: Just one Generation

7cd6e49

This is somewhat extracted from #8064, but can be landed independently and will make some of the followup sled-agent-config-reconciler PRs a little cleaner. Fixes #7774.

jgallagher mentioned this pull request Apr 29, 2025

OmicronSledConfig: Just one Generation #8067

Merged

smklein reviewed Apr 29, 2025

View reviewed changes

andrewjstone reviewed Apr 30, 2025

View reviewed changes

sled-agent/src/artifact_store.rs Show resolved Hide resolved

sled-agent/src/long_running_tasks.rs Outdated Show resolved Hide resolved

jgallagher mentioned this pull request Apr 30, 2025

Introduce sled-agent-config-reconciler skeleton #8063

Merged

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from abd7542 to 2574c5c Compare April 30, 2025 19:17

Base automatically changed from john/sled-agent-config-reconciler-1 to main May 1, 2025 12:34

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from 2574c5c to a057195 Compare May 2, 2025 14:59

This was referenced May 15, 2025

[sled-agent-config-reconciler] Flesh out main reconicilation task #8162

Merged

[sled-agent] Don't implicitly create datasets when launching zones #8173

Open

[inventory] Add full OmicronSledConfig and fields for upcoming config reconciler #8188

Merged

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from a057195 to 0faddda Compare May 21, 2025 20:38

jgallagher added 9 commits May 22, 2025 11:14

[sled-agent] Integrate config-reconciler

501f74f

use new ConfigReconcilerSpawnToken interface

ce36cb7

catch up to changes on main since last rebase

385807f

fill in some TODOs that were waiting on reconciler

2abceb7

fix failing planner tests

92b4916

expectorate

78c60a3

flesh out RSS wait_for_config_reconciliation_on_sled()

6b543b8

cargo fmt

6bce900

clippy

8ff4ae3

jgallagher force-pushed the john/sled-agent-config-reconciler-2 branch from 0faddda to 8ff4ae3 Compare May 22, 2025 15:30

illumos test fix

3ab3e79

jgallagher marked this pull request as ready for review May 22, 2025 21:24

jgallagher requested a review from sunshowers May 22, 2025 21:24

jgallagher requested review from andrewjstone, papertigers and smklein May 22, 2025 21:24

Merge branch 'main' into john/sled-agent-config-reconciler-2

bae91e4

This was referenced May 27, 2025

[sled-agent-config-reconciler] Add separate OmicronDatasets type #8218

Open

[sled-agent-config-reconciler] Check zone dataset dependencies before starting zones #8219

Open

andrewjstone approved these changes May 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[sled-agent] Integrate config-reconciler #8064

[sled-agent] Integrate config-reconciler #8064

Uh oh!

jgallagher commented Apr 29, 2025

Uh oh!

smklein Apr 29, 2025

Uh oh!

jgallagher Apr 30, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgallagher commented May 28, 2025

Uh oh!

andrewjstone left a comment

Uh oh!

andrewjstone May 28, 2025

Uh oh!

andrewjstone May 28, 2025

Uh oh!

andrewjstone May 28, 2025

Uh oh!

andrewjstone May 28, 2025

Uh oh!

Uh oh!

[sled-agent] Integrate config-reconciler #8064

Are you sure you want to change the base?

[sled-agent] Integrate config-reconciler #8064

Uh oh!

Conversation

jgallagher commented Apr 29, 2025

Uh oh!

smklein Apr 29, 2025

Choose a reason for hiding this comment

Uh oh!

jgallagher Apr 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jgallagher commented May 28, 2025

Uh oh!

andrewjstone left a comment

Choose a reason for hiding this comment

Uh oh!

andrewjstone May 28, 2025

Choose a reason for hiding this comment

Uh oh!

andrewjstone May 28, 2025

Choose a reason for hiding this comment

Uh oh!

andrewjstone May 28, 2025

Choose a reason for hiding this comment

Uh oh!

andrewjstone May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!