-
Notifications
You must be signed in to change notification settings - Fork 893
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Istio Ambient support #2676
Comments
What are the components that need to change for this? Would the profile controller need to create waypoint proxies for each user's namespace? |
Waypoint proxies are automatically created when the Gateway resource gets created (so is a bit simpler than crafting deployments/services). You can configure it in such a way that there's 1 instance per namespace that handles all L7 traffic for that namespace. |
@peterj one waypoint proxy per dynamic on demand namespace would be a critical change. |
That's how the ambient is designed to handle L7. So instead of running sidecar next to every workload, you run 1 waypoint (L7) proxy per namespace. |
Yeah I think the profile controller would be instantiating waypoint proxies for each user namespace. That shouldn't be so hard though (the profile controller already creates other per-namespace resources) This probably gets more complicated wrt to the Gateway API though. Do we need to migrate all VirtualServices and such to the Gateway API too? I'm not clear on what is interoperable |
The mixing of resources - VirtualService and Gateway API resources - is not supported in ambient. So, it's either Gateway API + HTTPRoute/TLSroute/TCPRoute or no Gateway API. Is there an inventory of VirtualServices (and features within) currently used by kubeflow? From the Istio docs, the following features are supported in HTTPRoutes (Gateway API):
|
Someone else will have to speak to it, but from what I can tell the features from |
The RequestAuthentication and AuthorizationPolicies can be used as-is. The only difference is that you have to use "targetRefs" instead of a selector label to point to the service. Here's an example from the docs: apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: view-only
namespace: default
spec:
# THIS IS DIFFERENT- you're targeting a Gateway, instead of using labels.
targetRefs:
- kind: Gateway
group: gateway.networking.k8s.io
name: default
action: ALLOW
rules:
- from:
- source:
namespaces: ["default", "istio-system"]
to:
- operation:
methods: ["GET"] And request auth: apiVersion: security.istio.io/v1
kind: RequestAuthentication
metadata:
name: "jwt-example"
namespace: foo
spec:
targetRef:
kind: Gateway
group: gateway.networking.k8s.io
name: httpbin-gateway
jwtRules:
- issuer: "[email protected]"
jwksUri: "https://raw.githubusercontent.com/istio/istio/release-1.22/security/tools/jwt/samples/jwks.json" But, of course, it would be great to test these things out beforehand :) Is there a good walkthrough and a collection scenarios one can run through that touch these features? (I am not so familiar with kubeflow, but if someone points me to the scenarios, I could probably test it out with Gateway API) |
In a PR you will just trigger many scenarios automatically, so that is the easiest way to test. Just check out all the GitHub workflows |
this is great work, thanks for moving this forward. |
ty @peterj this is really helpful clarifications! Do you know how the Gateway API works for setting up authentication at the gateway? I'm thinking of this |
I think there might be an easier way to define the ext-authz that doesn't use EnvoyFile -- https://istio.io/latest/docs/tasks/security/authorization/authz-custom/ I can see the oauth2-proxy is using the configuration above, but I am not sure why the oidc-service is using an EnvoyFilter... It should work with the same configuration I think |
Also, to answer the original question - based on the docs here the |
oidc-authservice will be eliminated soon :-) |
This is an amazing effort @peterj! I'd also like to help on the changes required. Another important component here though is going to be KServe and Knative, that create a lot of VirtualServices under the hood. Knative specifically sounds very scary since we have no influence over, from the Kubeflow side. cc @yuzisun EDIT: Maybe the only way to start with this would be with RawDeployment mode of KServe, which doesn't require Istio. But we need to try it out. My concern would be with the |
The Ingress resource still works with Istio (i.e. Istio implements the ingress class - ref), but it would make sense to move that over to Kubernetes Gateway API. |
I haven't tested anything in knative yet, but they have some discussion of ambient working |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
/lifecycle frozen |
Something that concerns me with ambient is that I can't figure out how to do deny-by-default. With sidecar, you can create an Does anyone have a solution? It feels like a pretty big hole so I assume I'm missing something here |
I'm trying to deploy kubeflow with istio ambient mode but I find that it might be difficult to migrate the ext-auth: istio/istio#51214 |
We will try istio-cni first for rootless istio, so please help there as well #2907 |
Istio Ambient mode is a different deployment model from the “traditional” (sidecar) mode of Istio. The ambient mode (sidecar-less) doesn’t require injecting sidecars into the deployments.
Here are the high-level differences between the two modes:
The concerns handled by the sidecar proxy in the sidecar Istio are split into two components in Istio ambient:
Ztunnel (handles L4 concerns, mTLS, authorization policies without any HTTP)
Waypoint proxy (handles L7 concerns, i.e. traffic splitting, matching, header manipulation, etc., more or less everything that gets defined in the VirtualService)
Ingress gateway isn’t installed by default anymore when using profile=ambient
Kubernetes Gateway API is used for ingress (and waypoint proxy) deployments
Any L7 VirtualServices or AuthorizationPolicies must have a “targetRef” section that specifies which waypoint proxy handles the L7 configuration
Waypoint proxies
Any VirtualService or AuthorizationPolicy that uses HTTP concepts will require a waypoint proxy. Given that there are 3 namespaces (that I identified so far), I’d suggest a per-namespace deployment of a waypoint proxy.
In addition to the waypoint proxy, the resources will have to be updated to use the waypoint proxies:
Work items
Migrate to Kubernetes Gateway API
Move to the latest Istio (Ambient will be beta in the next release (1.22))
Move to Ambient mode
Identify components that need waypoint proxy
Deploy waypoint proxies
Switch the Istio profile from default → ambient (or have an option of doing one or the other) - since Ambient will still be Beta, we shouldn’t make it a default option
The text was updated successfully, but these errors were encountered: