Allow multiple management planes #1124

aphralG · 2025-06-12T15:43:18Z

Proposed changes

Add Auxiliary Command Server block to agent config.
Modified Command plugin to take a command server type so two can be created one for each Management plane
Added CommandServerType to context for management plane sepcific messages "command" or "auxiliary"

auxiliary_command:
    server: 
        host: "second.management.plane"
        port: 9999
    auth: 
        token: "1234"
        tokenpath: "path/to/my_token"
    tls: 
        cert: "some.cert"
        key: "some.key"
        ca: "some.ca"
        skip_verify: false
        server_name: "server-name"

Checklist

Before creating a PR, run through this checklist and mark each as complete.

I have read the CONTRIBUTING document
I have run make install-tools and have attached any dependency changes to this pull request
If applicable, I have added tests that prove my fix is effective or that my feature works
If applicable, I have checked that any relevant tests pass after adding my changes
If applicable, I have updated any relevant documentation (README.md)
If applicable, I have tested my cross-platform changes on Ubuntu 22, Redhat 8, SUSE 15 and FreeBSD 13

dhurley · 2025-06-16T10:47:56Z

internal/command/command_plugin.go

+		conn              grpc.GrpcConnectionInterface
+		commandService    commandService
+		subscribeChannel  chan *mpi.ManagementPlaneRequest
+		commandServerType string


Could this be an enum?

dhurley · 2025-06-16T10:49:12Z

internal/command/command_plugin.go


 	cp.messagePipe = messagePipe
 	cp.commandService = NewCommandService(cp.conn.CommandServiceClient(), cp.config, cp.subscribeChannel)

-	go cp.monitorSubscribeChannel(ctx)
+	newCtx := context.WithValue(


Can you create the new context at the start of the function? Just so it can be used for the debug message above as well.

dhurley · 2025-06-16T10:52:13Z

internal/logger/logger.go

@@ -33,6 +34,7 @@ var (
 	}

 	CorrelationIDContextKey = contextKey(CorrelationIDKey)
+	ServerTypeContextKey    = contextKey(ServerTypeKey)
 )

 type (


On line 77 can you add ServerTypeContextKey to the list pf handlers so that the server type gets automatically added to the log messages?

dhurley · 2025-06-16T10:53:50Z

internal/command/command_plugin.go

+			slog.DebugContext(ctx, "Command plugin received unknown topic", "topic", msg.Topic)
+		}
+	} else {
+		slog.Info("Sever type is not right ignoring message", "command_server_type",


Can you remove this info log as it will spam the log file?

Ya this was supposed to be removed was just using for testing

dhurley · 2025-06-16T11:05:06Z

internal/command/command_plugin.go

@@ -226,12 +248,24 @@ func (cp *CommandPlugin) monitorSubscribeChannel(ctx context.Context) {
 				slog.InfoContext(ctx, "Received management plane config upload request")
 				cp.handleConfigUploadRequest(newCtx, message)
 			case *mpi.ManagementPlaneRequest_ConfigApplyRequest:
+				if cp.commandServerType != "command" {
+					slog.WarnContext(newCtx, "Auxiliary command server can not perform config apply",
+						"command_server_type", cp.commandServerType)


Think you need to send a failure response back to the management plane here as well

dhurley · 2025-06-16T11:05:20Z

internal/command/command_plugin.go

 				slog.InfoContext(ctx, "Received management plane config apply request")
 				cp.handleConfigApplyRequest(newCtx, message)
 			case *mpi.ManagementPlaneRequest_HealthRequest:
 				slog.InfoContext(ctx, "Received management plane health request")
 				cp.handleHealthRequest(newCtx)
 			case *mpi.ManagementPlaneRequest_ActionRequest:
+				if cp.commandServerType != "command" {
+					slog.WarnContext(newCtx, "Auxiliary command server can not perform api action",


Same as comment above

dhurley · 2025-06-16T11:05:51Z

internal/command/command_service.go

@@ -261,6 +261,7 @@ func (cs *CommandService) UpdateClient(ctx context.Context, client mpi.CommandSe
 	cs.subscribeClientMutex.Unlock()

 	cs.isConnected.Store(false)
+	// Will this have the sever type ?


Can you remove comment?

dhurley · 2025-06-16T12:50:31Z

internal/plugin/plugin_manager.go

+			// Followup PR to add read plugin eventually
+		}
+	} else {
+		slog.InfoContext(ctx, "Agent is not connected to an auxiliary management plane. "+


I wonder if this log message will cause confusion to the end user. Maybe we should just have a debug message for when an auxiliary command server isn'r configured

sean-breen · 2025-06-19T11:25:40Z

internal/command/command_plugin.go

@@ -336,3 +395,7 @@ func (cp *CommandPlugin) createDataPlaneResponse(correlationID string, status mp
 		},
 	}
 }
+
+func (s ServerType) String() string {


Can you move this to the top near the ServerType declaration?

I've moved all the SeverType code into the model package in server.go

sean-breen · 2025-06-19T11:40:52Z

internal/plugin/plugin_manager.go

@@ -59,6 +60,26 @@ func addCommandAndFilePlugins(ctx context.Context, plugins []bus.Plugin, agentCo
 	return plugins
 }

+func addAuxiliaryCommandAndFilePlugins(ctx context.Context, plugins []bus.Plugin,


Should this be addAuxiliaryCommandPlugin?

No, the read only file plugin is added to this function in the next PR

…command-server-to-agent-config

dhurley · 2025-06-27T10:22:23Z

internal/command/command_plugin.go


 	return nil
 }

 func (cp *CommandPlugin) Close(ctx context.Context) error {
-	slog.InfoContext(ctx, "Closing command plugin")
+	slog.InfoContext(ctx, "Closing command plugin", "command_server_type", cp.commandServerType.String())


Can you update the context with the server type since the context is also being passed to other functions?

dhurley · 2025-06-27T10:26:57Z

internal/command/command_plugin.go

+		ctx,
+		logger.ServerTypeContextKey, slog.Any(logger.ServerTypeKey, cp.commandServerType.String()),
+	)
+	slog.DebugContext(newCtx, "Starting command plugin", "command_server_type", cp.commandServerType.String())


Suggested change

slog.DebugContext(newCtx, "Starting command plugin", "command_server_type", cp.commandServerType.String())

slog.DebugContext(newCtx, "Starting command plugin")

Server type is already being added by the logger

dhurley · 2025-06-27T10:27:43Z

internal/command/command_plugin.go

-		cp.processDataPlaneResponse(ctx, msg)
-	default:
-		slog.DebugContext(ctx, "Command plugin received unknown topic", "topic", msg.Topic)
+	slog.DebugContext(ctx, "Processing command", "command_server_type", logger.ServerType(ctx))


Suggested change

slog.DebugContext(ctx, "Processing command", "command_server_type", logger.ServerType(ctx))

slog.DebugContext(ctx, "Processing command")

dhurley · 2025-06-27T10:29:01Z

internal/command/command_plugin.go

+			slog.ErrorContext(ctx, "Unable to update data plane health", "error", err,
+				"command_server_type", cp.commandServerType.String())


Suggested change

slog.ErrorContext(ctx, "Unable to update data plane health", "error", err,

"command_server_type", cp.commandServerType.String())

slog.ErrorContext(ctx, "Unable to update data plane health", "error", err)

…command-server-to-agent-config

dhurley · 2025-06-27T12:44:26Z

internal/command/command_plugin.go

-		slog.DebugContext(ctx, "Command plugin received unknown topic", "topic", msg.Topic)
+	slog.DebugContext(ctx, "Processing command")
+
+	if logger.ServerType(ctx) == cp.commandServerType.String() || logger.ServerType(ctx) == "" {


If logger.ServerType(ctx) == "" then we should create a new context with the serverType.

dhurley · 2025-07-01T14:51:08Z

internal/command/command_plugin.go

+			Message: message,
+			Error:   "Can not perform write action as auxiliary command server",
+		},
+		InstanceId: request.GetActionRequest().GetInstanceId(),


The instance ID is got in a different way if its a config apply request

…command-server-to-agent-config

allow multiple management planes

e07bf64

aphralG self-assigned this Jun 12, 2025

aphralG requested a review from a team as a code owner June 12, 2025 15:43

github-actions bot added the chore Pull requests for routine tasks label Jun 12, 2025

dhurley reviewed Jun 16, 2025

View reviewed changes

PR feedback

650a0d9

sean-breen requested changes Jun 19, 2025

View reviewed changes

dhurley added the v3.x Issues and Pull Requests related to the major version v3 label Jun 19, 2025

aphralG added 2 commits June 23, 2025 10:54

Merge branch 'add-auxiliary-command-server-proto' into add-auxiliary-…

2487a4d

…command-server-to-agent-config

PR feedback

b710855

aphralG requested review from dhurley and sean-breen June 23, 2025 10:21

john-david3 approved these changes Jun 23, 2025

View reviewed changes

dhurley reviewed Jun 27, 2025

View reviewed changes

aphralG added 2 commits June 27, 2025 12:19

Merge branch 'add-auxiliary-command-server-proto' into add-auxiliary-…

1fa09b9

…command-server-to-agent-config

PR feedback

8e26d81

dhurley reviewed Jun 27, 2025

View reviewed changes

aphralG added 2 commits June 27, 2025 14:01

PR feedback

f2d1fb2

fix message response

3f39fbf

sean-breen approved these changes Jun 30, 2025

View reviewed changes

dhurley reviewed Jul 1, 2025

View reviewed changes

aphralG added 2 commits July 1, 2025 16:46

Merge branch 'add-auxiliary-command-server-proto' into add-auxiliary-…

16b08c0

…command-server-to-agent-config

fix instance ID

845f3dd

	slog.DebugContext(newCtx, "Starting command plugin", "command_server_type", cp.commandServerType.String())
	slog.DebugContext(newCtx, "Starting command plugin")

	slog.DebugContext(ctx, "Processing command", "command_server_type", logger.ServerType(ctx))
	slog.DebugContext(ctx, "Processing command")

		slog.ErrorContext(ctx, "Unable to update data plane health", "error", err,
		"command_server_type", cp.commandServerType.String())

Allow multiple management planes #1124

Are you sure you want to change the base?

Allow multiple management planes #1124

Uh oh!

Conversation

aphralG commented Jun 12, 2025

Proposed changes

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!