fix: [NPM] [Linux] race condition when editing NetPol with "except" CIDR #2841

huntergregory · 2024-07-10T23:55:45Z

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
relevant PR labels added

Testing

Verified that the nomatch error does not occur any more on several manual tests.

Performed testing by applying the following NetPol, then editing the except block. There was an indeterministic repro on prior NPM versions.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: policy-with-cidr-except
spec:
  egress:
  - to:
    - ipBlock:
        cidr: 10.0.1.0/24
    - ipBlock:
        cidr: 10.0.2.0/24
    - ipBlock:
        cidr: 10.0.10.0/24
    - ipBlock:
        cidr: 10.0.20.0/24
    - ipBlock:
        cidr: 10.0.30.0/24
    - ipBlock:
        cidr: 10.0.40.0/24
    - ipBlock:
        cidr: 10.0.60.0/24
    - ipBlock:
        cidr: 10.1.0.0/16
        except:
        - 10.1.1.0/24
  ingress:
  - from:
    - ipBlock:
        cidr: 10.0.1.0/24
    - ipBlock:
        cidr: 10.0.2.0/24
    - ipBlock:
        cidr: 10.0.10.0/24
    - ipBlock:
        cidr: 10.0.20.0/24
    - ipBlock:
        cidr: 10.0.30.0/24
    - ipBlock:
        cidr: 10.0.40.0/24
    - ipBlock:
        cidr: 10.0.60.0/24
    - ipBlock:
        cidr: 10.1.0.0/16
        except:
        - 10.1.1.0/24
  podSelector: {}
  policyTypes:
  - Egress
  - Ingress

Logs from repro:

I0710 23:28:34.710087       1 dataplane.go:612] [DataPlane] Update Policy called for default/policy-with-cidr-except
I0710 23:28:34.710116       1 dataplane.go:545] [DataPlane] Remove Policy called for default/policy-with-cidr-except
I0710 23:28:34.710201       1 chain-management_linux.go:366] Executing iptables command with args [-w 60 -D AZURE-NPM-INGRESS -j AZURE-NPM-INGRESS-3221666101 -m set --match-set azure-npm-784554818 dst -m comment --comment INGRESS-POLICY-default/policy-with-cidr-except-TO-ns-default-IN-ns-default]
I0710 23:28:34.716043       1 chain-management_linux.go:366] Executing iptables command with args [-w 60 -D AZURE-NPM-EGRESS -j AZURE-NPM-EGRESS-3221666101 -m set --match-set azure-npm-784554818 src -m comment --comment EGRESS-POLICY-default/policy-with-cidr-except-FROM-ns-default-IN-ns-default]
I0710 23:28:34.749957       1 restore.go:188] running this restore command: [iptables-nft-restore -w 60 -T filter --noflush]
I0710 23:28:34.754754       1 dataplane.go:330] [DataPlane] [ApplyDataPlane] [APPLY-DP] starting to apply ipsets
I0710 23:28:34.754793       1 ipsetmanager.go:455] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [], toDeleteCache: map[cidr-policy-with-cidr-except-in-ns-default-0-0IN:0xc0009b0b00 cidr-policy-with-cidr-except-in-ns-default-0-0OUT:0xc0009b0860 cidr-policy-with-cidr-except-in-ns-default-0-1IN:0xc0009b0b50 cidr-policy-with-cidr-except-in-ns-default-0-1OUT:0xc0009b08b0 cidr-policy-with-cidr-except-in-ns-default-0-2IN:0xc0009b0ba0 cidr-policy-with-cidr-except-in-ns-default-0-2OUT:0xc0009b0900 cidr-policy-with-cidr-except-in-ns-default-0-3IN:0xc0009b0bf0 cidr-policy-with-cidr-except-in-ns-default-0-3OUT:0xc0009b0950 cidr-policy-with-cidr-except-in-ns-default-0-4IN:0xc0009b0c40 cidr-policy-with-cidr-except-in-ns-default-0-4OUT:0xc0009b09a0 cidr-policy-with-cidr-except-in-ns-default-0-5IN:0xc0009b0c90 cidr-policy-with-cidr-except-in-ns-default-0-5OUT:0xc0009b09f0 cidr-policy-with-cidr-except-in-ns-default-0-6IN:0xc0009b0ce0 cidr-policy-with-cidr-except-in-ns-default-0-6OUT:0xc0009b0a40 cidr-policy-with-cidr-except-in-ns-default-0-7IN:0xc0009b0d30 cidr-policy-with-cidr-except-in-ns-default-0-7OUT:0xc0009b0a90]
I0710 23:28:34.754943       1 restore.go:188] running this restore command: [ipset restore]
I0710 23:28:34.761373       1 restore.go:299] continuing after line 17 for command [ipset restore]
I0710 23:28:34.761485       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:34 [1] skipping destroy line for set cidr-policy-with-cidr-except-in-ns-default-0-6OUT since the set is in use by a kernel component
2024/07/10 23:28:34 [1] error: on try number 1, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-2976224028]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 17: Set cannot be destroyed: it is in use by a kernel component
]]
I0710 23:28:34.763021       1 restore.go:299] continuing after line 1 for command [ipset restore]
I0710 23:28:34.763121       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:34 [1] skipping destroy line for set cidr-policy-with-cidr-except-in-ns-default-0-5OUT since the set is in use by a kernel component
2024/07/10 23:28:34 [1] error: on try number 2, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-1488736011]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component
]]
I0710 23:28:34.765087       1 restore.go:299] continuing after line 1 for command [ipset restore]
I0710 23:28:34.765186       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:34 [1] skipping destroy line for set cidr-policy-with-cidr-except-in-ns-default-0-2IN since the set is in use by a kernel component
2024/07/10 23:28:34 [1] error: on try number 3, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-1687449499]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component
]]
I0710 23:28:34.766865       1 restore.go:299] continuing after line 1 for command [ipset restore]
I0710 23:28:34.766945       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:34 [1] skipping destroy line for set cidr-policy-with-cidr-except-in-ns-default-0-4OUT since the set is in use by a kernel component
2024/07/10 23:28:34 [1] error: on try number 4, failed to run command [ipset restore]. Rerunning with updated file. err: [line-number error for line [-X azure-npm-3478126754]: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component
]]
2024/07/10 23:28:34 [1] error: failed to apply ipsets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component
]
2024/07/10 23:28:34 [1] syncNetPol error due to error syncing 'default/policy-with-cidr-except': [syncNetPol] error due to  [syncAddAndUpdateNetPol] Error: failed to update translated NPMNetworkPolicy into Dataplane due to [DataPlane] error while updating policy: [DataPlane] [APPLY-DP] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component
], requeuing
E0710 23:28:34.769811       1 networkPolicyController.go:195] error syncing 'default/policy-with-cidr-except': [syncNetPol] error due to  [syncAddAndUpdateNetPol] Error: failed to update translated NPMNetworkPolicy into Dataplane due to [DataPlane] error while updating policy: [DataPlane] [APPLY-DP] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 1] and stdErr [ipset v7.5: Error in line 1: Set cannot be destroyed: it is in use by a kernel component
], requeuing
I0710 23:28:34.775238       1 dataplane.go:612] [DataPlane] Update Policy called for default/policy-with-cidr-except
I0710 23:28:34.775257       1 dataplane.go:615] [DataPlane] Policy default/policy-with-cidr-except is not found.
I0710 23:28:34.775263       1 dataplane.go:394] [DataPlane] Add Policy called for default/policy-with-cidr-except
I0710 23:28:34.775267       1 types.go:214] [DataPlane] enqueuing policy default/policy-with-cidr-except in netPolQueue
I0710 23:28:34.775273       1 dataplane.go:408] [DataPlane] [ADD-NETPOL] new pending netpol count: 1
I0710 23:28:34.775288       1 networkPolicyController.go:191] Successfully synced 'default/policy-with-cidr-except'
I0710 23:28:35.148630       1 dataplane.go:421] [DataPlane] adding policies [0xc000338d10]
I0710 23:28:35.148828       1 dataplane.go:330] [DataPlane] [ApplyDataPlane] [ADD-NETPOL] starting to apply ipsets
I0710 23:28:35.148870       1 ipsetmanager.go:455] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [cidr-policy-with-cidr-except-in-ns-default-0-1IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-3IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-7OUT: &{membersToAdd:map[10.1.5.0/24 nomatch:{}] membersToDelete:map[10.1.4.0/24 nomatch:{}]},cidr-policy-with-cidr-except-in-ns-default-0-3OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-0IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-5IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-1OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-2OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-6IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-7IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-0OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-5OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-6OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-2IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-4IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-4OUT: &{membersToAdd:map[] membersToDelete:map[]}], toDeleteCache: map[]
I0710 23:28:35.148947       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 1, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
I0710 23:28:35.151875       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 2, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
I0710 23:28:35.154507       1 restore.go:188] running this restore command: [ipset restore]
I0710 23:28:35.157913       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 3, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 4, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
I0710 23:28:35.168315       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] error: failed to apply ipsets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
E0710 23:28:35.172634       1 dataplane.go:431] [DataPlane] [BACKGROUND] failed to add policies. will retry one policy at a time. err: [DataPlane] [ADD-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] [DataPlane] [BACKGROUND] failed to add policies. err: [DataPlane] [ADD-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
I0710 23:28:35.174607       1 dataplane.go:330] [DataPlane] [ApplyDataPlane] [ADD-NETPOL] starting to apply ipsets
I0710 23:28:35.174659       1 ipsetmanager.go:455] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [cidr-policy-with-cidr-except-in-ns-default-0-1OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-3OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-0IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-5IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-0OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-2OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-6IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-7IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-4OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-5OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-6OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-2IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-4IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-7OUT: &{membersToAdd:map[10.1.5.0/24 nomatch:{}] membersToDelete:map[10.1.4.0/24 nomatch:{}]},cidr-policy-with-cidr-except-in-ns-default-0-1IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-3IN: &{membersToAdd:map[] membersToDelete:map[]}], toDeleteCache: map[]
I0710 23:28:35.174868       1 restore.go:188] running this restore command: [ipset restore]
I0710 23:28:35.177621       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 1, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
I0710 23:28:35.180657       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 2, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 3, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
I0710 23:28:35.185743       1 restore.go:188] running this restore command: [ipset restore]
I0710 23:28:35.187578       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 4, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]
2024/07/10 23:28:35 [1] error: failed to apply ipsets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] [DataPlane] [BACKGROUND] failed to add policy one at a time. default/policy-with-cidr-except. err: [DataPlane] [ADD-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
E0710 23:28:35.190302       1 dataplane.go:443] [DataPlane] [BACKGROUND] failed to add policy one at a time. policyKey: default/policy-with-cidr-except. err: [DataPlane] [ADD-NETPOL] error while applying IPSets: ipset restore failed when applying ipsets: Operation [RunCommandWithFile] failed with error code [999], full cmd [], full error after 5 tries, failed to run command [ipset restore] with error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
I0710 23:28:35.648957       1 dataplane.go:421] [DataPlane] adding policies [0xc000338d10]
I0710 23:28:35.649086       1 dataplane.go:330] [DataPlane] [ApplyDataPlane] [ADD-NETPOL] starting to apply ipsets
I0710 23:28:35.649173       1 ipsetmanager.go:455] [IPSetManager] dirty caches. toAddUpdateCache: to create: [], to update: [cidr-policy-with-cidr-except-in-ns-default-0-2IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-4IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-4OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-5OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-6OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-7OUT: &{membersToAdd:map[10.1.5.0/24 nomatch:{}] membersToDelete:map[10.1.4.0/24 nomatch:{}]},cidr-policy-with-cidr-except-in-ns-default-0-1IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-3IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-5IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-1OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-3OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-0IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-7IN: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-0OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-2OUT: &{membersToAdd:map[] membersToDelete:map[]},cidr-policy-with-cidr-except-in-ns-default-0-6IN: &{membersToAdd:map[] membersToDelete:map[]}], toDeleteCache: map[]
I0710 23:28:35.649334       1 restore.go:188] running this restore command: [ipset restore]
2024/07/10 23:28:35 [1] expected error with line number, but couldn't detect one with error regex pattern [Error in line (\d+):] for command [ipset restore] with stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]
2024/07/10 23:28:35 [1] error: on try number 1, failed to run command [ipset restore]. Rerunning with same file. err: [unknown error: error running command [ipset restore] with err [exit status 2] and stdErr [ipset v7.5: Unknown argument: `nomatch'
Try `ipset help' for more information.
]]

...

Signed-off-by: Hunter Gregory <[email protected]>

npm/pkg/dataplane/ipsets/ipsetmanager_linux.go

Signed-off-by: Hunter Gregory <[email protected]>

huntergregory · 2024-07-24T16:38:53Z

/azp run Azure Container Networking PR

azure-pipelines · 2024-07-24T16:39:04Z

Azure Pipelines successfully started running 1 pipeline(s).

huntergregory · 2024-07-24T16:39:09Z

/azp run NPM Conformance Tests

azure-pipelines · 2024-07-24T16:39:20Z

Azure Pipelines successfully started running 1 pipeline(s).

huntergregory · 2024-07-24T16:39:30Z

/azp run NPM Scale Test

azure-pipelines · 2024-07-24T16:39:41Z

Azure Pipelines successfully started running 1 pipeline(s).

…IDR (#2841) * fix: syntax error when deleting nomatch CIDR ipset Signed-off-by: Hunter Gregory <[email protected]> * test: ut members with nomatch Signed-off-by: Hunter Gregory <[email protected]> --------- Signed-off-by: Hunter Gregory <[email protected]>

… "except" CIDR (#2841) * fix: syntax error when deleting nomatch CIDR ipset Signed-off-by: Hunter Gregory <[email protected]> * test: ut members with nomatch Signed-off-by: Hunter Gregory <[email protected]> --------- Signed-off-by: Hunter Gregory <[email protected]>

… NPM pipeline fixes (#2882) * [backport] [fix] ci: update cyclonus workflow (#2866) ci: update cyclonus workflow Signed-off-by: Hunter Gregory <[email protected]> * [backport] ci: Update triggers for NPM pipelines (#2867) ci: add triggers to npm pipelines Signed-off-by: Hunter Gregory <[email protected]> * [backport] ci: update Cyclonus workflow triggers (#2868) ci: update Cyclonus triggers Signed-off-by: Hunter Gregory <[email protected]> * [backport] ci: [NPM] speed up Windows conformance and disable Windows Cyclonus/Scale (#2874) * ci: comment out windows cyclonus and windows scale test Signed-off-by: Hunter Gregory <[email protected]> * ci: slim down windows conformance to 14 tests Signed-off-by: Hunter Gregory <[email protected]> * ci: make sure conformance skips "Linux Only" Signed-off-by: Hunter Gregory <[email protected]> * ci: conformance was not running test cases due to formatting Signed-off-by: Hunter Gregory <[email protected]> --------- Signed-off-by: Hunter Gregory <[email protected]> * [backport] ci: [NPM] NPM Conformance Test improvements (#2875) * ci: Always delete npm conformance clusters * test: pin k8s suite version * ci: comment out updated cyclonus * chore: cleanup yaml Signed-off-by: Hunter Gregory <[email protected]> * [backport] ci: [NPM] add bash directive to scale scripts (#2876) ci: add bash directive to scale scripts Signed-off-by: Hunter Gregory <[email protected]> * [backport] fix: [NPM] [Linux] race condition when editing NetPol with "except" CIDR (#2841) * fix: syntax error when deleting nomatch CIDR ipset Signed-off-by: Hunter Gregory <[email protected]> * test: ut members with nomatch Signed-off-by: Hunter Gregory <[email protected]> --------- Signed-off-by: Hunter Gregory <[email protected]> --------- Signed-off-by: Hunter Gregory <[email protected]> Co-authored-by: John Payne <[email protected]>

huntergregory added bug npm Related to NPM. linux labels Jul 10, 2024

huntergregory requested a review from a team as a code owner July 10, 2024 23:55

huntergregory requested a review from matmerr July 10, 2024 23:55

fix: syntax error when deleting nomatch CIDR ipset

c68974f

Signed-off-by: Hunter Gregory <[email protected]>

huntergregory force-pushed the huntergregory/npm-cidr-except branch from 3b81c45 to c68974f Compare July 11, 2024 00:02

timraymond previously approved these changes Jul 19, 2024

View reviewed changes

npm/pkg/dataplane/ipsets/ipsetmanager_linux.go Show resolved Hide resolved

huntergregory dismissed timraymond’s stale review via f9015b7 July 19, 2024 22:52

test: ut members with nomatch

7ca1707

Signed-off-by: Hunter Gregory <[email protected]>

huntergregory force-pushed the huntergregory/npm-cidr-except branch from f9015b7 to 7ca1707 Compare July 19, 2024 22:55

huntergregory added 2 commits July 19, 2024 15:55

Merge branch 'master' into huntergregory/npm-cidr-except

92b045c

Merge branch 'master' into huntergregory/npm-cidr-except

7c3eda0

timraymond approved these changes Jul 22, 2024

View reviewed changes

Merge branch 'master' into huntergregory/npm-cidr-except

6a4f64c

huntergregory enabled auto-merge July 24, 2024 19:19

matmerr approved these changes Jul 24, 2024

View reviewed changes

huntergregory added this pull request to the merge queue Jul 24, 2024

Merged via the queue into master with commit 2688c59 Jul 24, 2024
31 checks passed

huntergregory deleted the huntergregory/npm-cidr-except branch July 24, 2024 23:38

huntergregory mentioned this pull request Jul 26, 2024

[backport] fix/ci: Linux NPM race condition editing "except" CIDR and NPM pipeline fixes #2882

Merged

4 tasks

huntergregory mentioned this pull request Aug 28, 2024

fix: [NPM] [Linux] panic if applyIPSets continues to fail #2964

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: [NPM] [Linux] race condition when editing NetPol with "except" CIDR #2841

fix: [NPM] [Linux] race condition when editing NetPol with "except" CIDR #2841

huntergregory commented Jul 10, 2024 •

edited

Loading

huntergregory commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

huntergregory commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

huntergregory commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

fix: [NPM] [Linux] race condition when editing NetPol with "except" CIDR #2841

fix: [NPM] [Linux] race condition when editing NetPol with "except" CIDR #2841

Conversation

huntergregory commented Jul 10, 2024 • edited Loading

Testing

huntergregory commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

huntergregory commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

huntergregory commented Jul 24, 2024

azure-pipelines bot commented Jul 24, 2024

huntergregory commented Jul 10, 2024 •

edited

Loading