Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MULTIARCH-4252: namespace-scoped Pod Placement Config #453

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

MULTIARCH-4252: namespace-scoped Pod Placement Config #453

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0dc7122
Fix naming of CRD in MTO-0001.md
aleskandro Feb 5, 2025
2b1d640
MULTIARCH-4252: Introducing the namespace-scoped PodPlacementConfig
aleskandro Feb 5, 2025
4753570
MULTIARCH-5320: Priority handling
aleskandro Feb 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 23 additions & 23 deletions docs/enhancements/MTO-0001.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ responsible for from a high-level user perspective.
#### Operator installation
1. The cluster-admin submits a Subscription object to install the
multiarch-manager-operator, either via CLI or the OperatorHub
2. The cluster admin submits the default `PodPlacementConfig` Custom Resource
2. The cluster admin submits the default `ClusterPodPlacementConfig` Custom Resource
(CR) adding their specific configuration related to initializing the
multiarch-manager-operator's components; the cluster admin can customize the
CR by using a namespace selector to control pod filtering and logging levels
Expand Down Expand Up @@ -243,9 +243,9 @@ sequenceDiagram
performed by the users to remove the scheduling gate and resume scheduling

#### Variation
- *Pods ignored during scheduling*: the `PodPlacementConfig` CR allows a
- *Pods ignored during scheduling*: the `ClusterPodPlacementConfig` CR allows a
namespaceSelector to be specified. The mutating webhook is patched with this
selector during the PodPlacementConfig's reconciliation process. This gives
selector during the `ClusterPodPlacementConfig`'s reconciliation process. This gives
flexibility for the administrator on how the pods will be selected for
mutation.
- *NodeAffinity block already set in the gated pod*: when patching the node
Expand Down Expand Up @@ -297,7 +297,7 @@ permissions:
- Watch, List, Get
- ConfigMaps (MMO operator namespace and openshift-image-registry)
- Watch, List, Get
- PodPlacementConfig CRD
- ClusterPodPlacementConfig CRD
- Watch, List, Get, Update
- admissionregistration.k8s.io/MutatingWebhookConfiguration
- Get, Update
Expand All @@ -310,7 +310,7 @@ permissions:
- config.openshift.io/Image
- Watch, List, Get

#### PodPlacementConfig CR
#### ClusterPodPlacementConfig CR
The configuration CR of the operator allows the admins to set the following:
- a `NamespaceSelector` to filter the namespaces MMO should patch the pods of.
The `openshift-*`, `kube-*` and `hypershift-*` namespaces are implicitly excluded.
Expand All @@ -331,8 +331,8 @@ const (
)


// PodPlacementConfigSpec defines the desired state of PodPlacementConfig
type PodPlacementConfigSpec struct {
// ClusterPodPlacementConfigSpec defines the desired state of ClusterPodPlacementConfig
type ClusterPodPlacementConfigSpec struct {
// logVerbosity is the log level for the pod placement controller
// Valid values are: "Normal", "Debug", "Trace", "TraceAll".
// Defaults to "Normal".
Expand All @@ -349,31 +349,31 @@ type PodPlacementConfigSpec struct {

}

// PodPlacementConfigStatus defines the observed state of PodPlacementConfig
type PodPlacementConfigStatus struct {
// Conditions represents the latest available observations of a PodPlacementConfig's current state.
// ClusterPodPlacementConfigStatus defines the observed state of ClusterPodPlacementConfig
type ClusterPodPlacementConfigStatus struct {
// Conditions represents the latest available observations of a ClusterPodPlacementConfig's current state.
Conditions []metav1.Condition `json:"conditions,omitempty"`
}

//+kubebuilder:object:root=true
//+kubebuilder:subresource:status

// PodPlacementConfig is the Schema for the podplacementconfigs API
// +kubebuilder:validation:XValidation:rule="self.metadata.name == 'cluster'",message="podplacementconfig is a singleton, .metadata.name must be 'cluster'"
type PodPlacementConfig struct {
// ClusterPodPlacementConfig is the Schema for the ClusterPodPlacementConfigs API
// +kubebuilder:validation:XValidation:rule="self.metadata.name == 'cluster'",message="ClusterPodPlacementConfig is a singleton, .metadata.name must be 'cluster'"
type ClusterPodPlacementConfig struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`

Spec PodPlacementConfigSpec `json:"spec,omitempty"`
Status PodPlacementConfigStatus `json:"status,omitempty"`
Spec ClusterPodPlacementConfigSpec `json:"spec,omitempty"`
Status ClusterPodPlacementConfigStatus `json:"status,omitempty"`
}

```

```yaml=

apiVersion: multiarch.openshift.io/v1alpha1
kind: PodPlacementConfig
kind: ClusterPodPlacementConfig
metadata:
name: cluster
spec:
Expand All @@ -393,9 +393,9 @@ the following main components:
a pod having the scheduling gate, it builds a set of pod-compatible
architectures and patches the pod's node affinity so that it is scheduled on
nodes having a CPU architecture supported by all the images used by the pod
- `PodPlacementConfigController`: both the `SchedulingGateMutatingWebhook` and
- `ClusterPodPlacementConfigController`: both the `SchedulingGateMutatingWebhook` and
the `PodPlacementController` consume the configuration information provided by
admins via the `PodPlacementConfig` CR. The `PodPlacementConfigController`
admins via the `ClusterPodPlacementConfig` CR. The `ClusterPodPlacementConfigController`
is run by the operator to roll out the configuration of the business-logic components
- `Facade`: expose an interface hiding the location and nature of the image
metadata. The `getCompatibleArchitectureSet(image)` method will return the set
Expand Down Expand Up @@ -442,11 +442,11 @@ classDiagram

}

class PodPlacementConfig {
class ClusterPodPlacementConfig {
<<CRD>>
}

class PodPlacementConfigController {
class ClusterPodPlacementConfigController {
<<controller>>

}
Expand Down Expand Up @@ -520,7 +520,7 @@ classDiagram
Pod "1" --> "*" Secret : ImagePullSecrets
PodPlacementController --> Pod : watches and mutates<br>(Patches nodeAffinity)
SchedulingGateMutatingWebhook --> Pod : mutate<br>(Adds schedulingGate)
PodPlacementConfigController --> PodPlacementConfig : reconciles
ClusterPodPlacementConfigController --> ClusterPodPlacementConfig : reconciles

PodPlacementController --> Facade : uses
RegistryInspector --> globalPullSecret : watches and caches
Expand Down Expand Up @@ -551,7 +551,7 @@ In general, the `NodeSelectorTerms` are ORed while the `nodeSelectorTerms[].matc
#### Image inspection
The inspection of images relies upon several mechanisms, some of which depend on
the specific scenario. The operator aims to execute in any Kubernetes
distribution and will be user-config-driven, using the `PodPlacementConfig` CRD
distribution and will be user-config-driven, using the `ClusterPodPlacementConfig` CRD
described in the previous sections. However, some behaviors can be automated to
reduce the configuration effort for the user in Openshift specific scenarios.

Expand Down Expand Up @@ -620,7 +620,7 @@ to the settings of the `image.config.openshift.io/cluster` object's `registrySou
It's worth noting that using `LeaderElectionReleaseOnCancel` for the `PodPlacementController` is considered safe even due
to its inherently stateless nature and the Kubernetes optimistic concurrency principle. The active-passive replication
model is employed because assigning shards of pods to different active-active replicas is challenging.
- Pod filtering: namespaces for which the pods should undergo mutation is decided through the namespaceSelector in the PodPlacementConfig CR.
- Pod filtering: namespaces for which the pods should undergo mutation is decided through the namespaceSelector in the `PodPlacementConfig` CR.
- Pods in `SchedulingGated` state and the pod placement controller is down
- If the controller cannot be recovered, the scheduling gate must be manually removed from the pods.
- Pods in `SchedulingGated` state when the operator is uninstalled
Expand Down
Loading