KEP-3953: Node Resource Hot Plug

[Karthik-K-N]

• One-line PR description: Node Resource Hot Plug

• Issue link: Node Resource Hot Plug #3953

• Other comments:

[bart0sh]

/assign @mrunalp @SergeyKanzhelev @klueska

[kad]

/cc

[ffromani]

/cc

[fmuyassarov]

/cc

[Karthik-K-N]

Hi All, We have addressed comments and concerns, and have added more details to KEP

Note: One pending issue needs opinion is regarding updating oom_score_adj for a container which is currently being discussed here opencontainers/runc#4669

Please let us know your thoughts
Thanks

[towca]

Is Node autoscaling a consideration for this feature, or do you expect for it to only be used on Nodes that aren't expected to be horizontally autoscaled?

Cluster Autoscaler uses existing Nodes as templates for how a new Node from the same group would look like. How should it treat Nodes which were affected by hot plugging? Can we distinguish which Nodes were affected?

In any case, if the feature is meant to be used with Cluster Autoscaler and if it can result in Nodes from the same NodeGroup having different allocatable values, it will require making CA aware of the hotplugging. Otherwise CA will just pick a random Node from the group and assume that all new Nodes will have the same allocatable value - which will lead to bad decisions (repeatedly provisioning a Node that a pending Pod can't actually fit on, or not provisioning a Node that could actually help a pending Pod).

[j4ckstraw]

FYI, when resource decreased and with kubelet restart, it will check if kubelet can admit the pod, TheOutOfxxx error will be encountered. refer https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/lifecycle/predicate.go#L213

[Karthik-K-N]

FYI, when resource decreased and with kubelet restart, it will check if kubelet can admit the pod, TheOutOfxxx error will be encountered. refer https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/lifecycle/predicate.go#L213

Yeah, Thank you for bringing this point and though this KEP primarily focuses on resource hot plug and some level of attention is also given for hot unplug like setting the node as not ready incase of hot unplug.
This can be further refined in a separate KEP which focuses on hot unplug scenarios.

[Karthik-K-N]

Is Node autoscaling a consideration for this feature, or do you expect for it to only be used on Nodes that aren't expected to be horizontally autoscaled?

We intend to use this feature on all cluster irrespective of their autoscaling capability.

Cluster Autoscaler uses existing Nodes as templates for how a new Node from the same group would look like. How should it treat Nodes which were affected by hot plugging? Can we distinguish which Nodes were affected?

In any case, if the feature is meant to be used with Cluster Autoscaler and if it can result in Nodes from the same NodeGroup having different allocatable values, it will require making CA aware of the hotplugging. Otherwise CA will just pick a random Node from the group and assume that all new Nodes will have the same allocatable value - which will lead to bad decisions (repeatedly provisioning a Node that a pending Pod can't actually fit on, or not provisioning a Node that could actually help a pending Pod).

Can we make autoscaler aware of hotplug capabilities? If not any recommendation/opinion on the kind of API that can make the autoscaler aware of change in resource capacity.

[marquiz]

What plugins are you referring to? Device plugins or something else. Might consider clarification here

[Karthik-K-N]

oh yeah, I meant to say NRI plugins, Let me explicitly specify that.

[marquiz]

Do we capture the corner case where node capacity would increase but some of the resources go away, e.g. new cpuset is bigger but doesn't contain the old cpuset (dumb example 0-3 -> 2-9). CPU manager sync returns an error and node goes to not ready?

[Karthik-K-N]

So for now we thought to only record an event during resync failures. may be need to check how we can identify what was the assigned cpu set before hotplug.

[esotsal]

/cc

[esotsal]

Suggested change

	Hotplug: Dynamically add compute resources (CPU, Memory, Swap Capacity and HugePages) to the node, either via software (online offlined resources) or via hardware (physical additions while the system is running)
	Node Compute Resource Hot Plug: Dynamically add node compute resources (CPU, Memory, Swap Capacity and HugePages) to the node, either via software (online offlined resources) or via hardware (physical additions while the system is running)

[Karthik-K-N]

We inclined towards keeping hotplug in glossary.

[esotsal]

Suggested change

	Hotunplug: Dynamically remove compute resources (CPU, Memory, Swap Capacity and HugePages) to the node, either via software (make resources go offline) or via hardware (physical removal while the system is running)
	Node Compute Resource Hot Unplug: Dynamically remove compute resources (CPU, Memory, Swap Capacity and HugePages) from the node, either via software (make resources go offline) or via hardware (physical removal while the system is running)

[esotsal]

Since Swap is mentioned in this document and since Swap is supported only with cgroup v2. Does this mean intention for this KEP is not to be supported for cgroup v1 or ?

[haircommander]

technically speaking cgroupv1 is feature frozen, but I think we'll accidentally have support for cgroupv1 unless we explicitly skip the detection on v1

[esotsal]

Do we need to add something about huge pages ?

Any implications for Nodes using DPDK / SRIOV with relation to CNIs which we need to consider ?

[Karthik-K-N]

Is Node autoscaling a consideration for this feature, or do you expect for it to only be used on Nodes that aren't expected to be horizontally autoscaled?

Cluster Autoscaler uses existing Nodes as templates for how a new Node from the same group would look like. How should it treat Nodes which were affected by hot plugging? Can we distinguish which Nodes were affected?

In any case, if the feature is meant to be used with Cluster Autoscaler and if it can result in Nodes from the same NodeGroup having different allocatable values, it will require making CA aware of the hotplugging. Otherwise CA will just pick a random Node from the group and assume that all new Nodes will have the same allocatable value - which will lead to bad decisions (repeatedly provisioning a Node that a pending Pod can't actually fit on, or not provisioning a Node that could actually help a pending Pod).

Based on the conversation over slack and here, Added a section about compatability with Cluster Autoscaler: https://github.com/Karthik-K-N/enhancements/tree/node-resize/keps/sig-node/3953-node-resource-hot-plug#compatability-with-cluster-autoscaler.
Please take a look. Thanks.

[towca]

Based on the conversation over slack and here, Added a section about compatability with Cluster Autoscaler: https://github.com/Karthik-K-N/enhancements/tree/node-resize/keps/sig-node/3953-node-resource-hot-plug#compatability-with-cluster-autoscaler.
Please take a look. Thanks.

Thank you, the added section accurately captures the problem and possible solutions. Could you add some information on when you want to address this part? When going to beta with the feature?

[sanposhiho]

Revisiting here, as far as I remember at the previous discussion somewhere, no change is required in the scheduler, is it still true?
If not, or if you're not sure, please make sure adding someone from sig-scheduling to review it.

[Karthik-K-N]

Revisiting here, as far as I remember at the previous discussion somewhere, no change is required in the scheduler, is it still true? If not, or if you're not sure, please make sure adding someone from sig-scheduling to review it.

Yes, still it stands the same, No changes are required from scheduler, As you mentioned in previous review I have updated the KEP design details to include the phrase

    Scheduler will automatically schedule any pending pods.
    This is done as an expected behavior and does not require any changes in the existing design of the scheduler, as the scheduler watches the available capacity of the node and creates pods accordingly.

For more reference our previous discussion: Slack, Enhancement

[sanposhiho]

Sure, as long as that part isn't changed, I believe it's ok from sig-scheduling side.

[Karthik-K-N]

Sure, as long as that part isn't changed, I believe it's ok from sig-scheduling side.

yeah, I will reach out, if there are any changes in future, Thank you for taking a look.

[Karthik-K-N]

Based on the conversation over slack and here, Added a section about compatability with Cluster Autoscaler: https://github.com/Karthik-K-N/enhancements/tree/node-resize/keps/sig-node/3953-node-resource-hot-plug#compatability-with-cluster-autoscaler.
Please take a look. Thanks.

Thank you, the added section accurately captures the problem and possible solutions. Could you add some information on when you want to address this part? When going to beta with the feature?

Thanks for the review, We plan to provide compatibility with autoscaler by storing the initial node resource in node object and also plan to provide this feature during beta graduation so we can get started now. But we are open for community feedback. Thank you.

[towca]

Thanks for the review, We plan to provide compatibility with autoscaler by storing the initial node resource in node object and also plan to provide this feature during beta graduation so we can get started now. But we are open for community feedback. Thank you.

Sounds good to me, thanks!

[elmiko]

gave a read of the kep, no comments or suggestions currently, but happy to see this work progressing.