This reference implementation demonstrates the recommended starting (baseline) infrastructure architecture for a general purpose AKS cluster. This implementation and document is meant to guide an interdisciplinary team or multiple distinct teams like networking, security and development through the process of getting this general purpose baseline infrastructure deployed and understanding the components of it.
We walk through the deployment here in a rather verbose method to help you understand each component of this cluster, ideally teaching you about each layer and providing you with the knowledge necessary to apply it to your workload.
This project has a companion set of articles that describe challenges, design patterns, and best practices for a secure AKS cluster. You can find this article on the Azure Architecture Center at Azure Kubernetes Service (AKS) Baseline Cluster. If you haven't reviewed it, we suggest you read it as it will give added context to the considerations applied in this implementation. Ultimately, this is the direct implementation of that specific architectural guidance.
This architecture is infrastructure focused, more so than on workload. It concentrates on the AKS cluster itself, including concerns with identity, post-deployment configuration, secret management, and network topologies.
The implementation presented here is the minimum recommended baseline for most AKS clusters. This implementation integrates with Azure services that will deliver observability, provide a network topology that will support multi-regional growth, and keep the in-cluster traffic secure as well. This architecture should be considered your starting point for pre-production and production stages.
The material here is relatively dense. We strongly encourage you to dedicate time to walk through these instructions, with a mind to learning. We do NOT provide any "one click" deployment here. However, once you've understood the components involved and identified the shared responsibilities between your team and your great organization, it is encouraged that you build suitable, auditable deployment processes around your final infrastructure.
Throughout the reference implementation, you will see reference to Contoso Bicycle. They are a fictional small and fast-growing startup that provides online web services to its clientele on the west coast of North America. They have no on-premises data centers and all their containerized line of business applications are now about to be orchestrated by secure, enterprise-ready AKS clusters. You can read more about their requirements and their IT team composition. This narrative provides grounding for some implementation details, naming conventions, etc. You should adapt as you see fit.
Finally, this implementation uses the ASP.NET Core Docker sample web app as an example workload. This workload is purposefully uninteresting, as it is here exclusively to help you experience the baseline infrastructure.
- AKS v1.22
- System and User node pool separation
- AKS-managed Azure AD
- Azure AD-backed Kubernetes RBAC (local user accounts disabled)
- Managed Identities
- Azure CNI
- Azure Monitor for containers
- Azure Virtual Networks (hub-spoke)
- Azure Firewall managed egress
- Azure Application Gateway (WAF)
- AKS-managed Internal Load Balancers
- Flux GitOps Operator
- Traefik Ingress Controller
- Azure AD Pod Identity
- Secrets Store CSI Driver for Kubernetes
- Kured
A deployment of AKS-hosted workloads typically experiences a separation of duties and lifecycle management in the area of prerequisites, the host network, the cluster infrastructure, and finally the workload itself. This reference implementation is similar. Also, be aware our primary purpose is to illustrate the topology and decisions of a baseline cluster. We feel a "step-by-step" flow will help you learn the pieces of the solution and give you insight into the relationship between them. Ultimately, lifecycle/SDLC management of your cluster and its dependencies will depend on your situation (team roles, organizational standards, etc), and will be implemented as appropriate for your needs.
Please start this learning journey in the Preparing for the cluster section. If you follow this through to the end, you'll have our recommended baseline cluster installed, with an end-to-end sample workload running for you to reference in your own Azure subscription.
There are considerations that must be addressed before you start deploying your cluster. Do I have enough permissions in my subscription and AD tenant to do a deployment of this size? How much of this will be handled by my team directly vs having another team be responsible?
- Begin by ensuring you install and meet the prerequisites
- Procure client-facing and AKS Ingress Controller TLS certificates
- Plan your Azure Active Directory integration
Microsoft recommends AKS be deploy into a carefully planned network; sized appropriately for your needs and with proper network observability. Organizations typically favor a traditional hub-spoke model, which is reflected in this implementation. While this is a standard hub-spoke model, there are fundamental sizing and portioning considerations included that should be understood.
This is the heart of the guidance in this reference implementation; paired with prior network topology guidance. Here you will deploy the Azure resources for your cluster and the adjacent services such as Azure Application Gateway WAF, Azure Monitor, Azure Container Registry, and Azure Key Vault. This is also where you put the cluster under GitOps orchestration.
We perform the prior steps manually here for you to understand the involved components, but we advocate for an automated DevOps process. Therefore, incorporate the prior steps into your CI/CD pipeline, as you would any infrastructure as code (IaC). We have included a starter GitHub workflow that demonstrates this.
Without a workload deployed to the cluster it will be hard to see how these decisions come together to work as a reliable application platform for your business. The deployment of this workload would typically follow a CI/CD pattern and may involve even more advanced deployment strategies (blue/green, etc). The following steps represent a manual deployment, suitable for illustration purposes of this infrastructure.
- Just like the cluster, there are workload prerequisites to address
- Configure AKS Ingress Controller with Azure Key Vault integration
- Deploy the workload
Now that the cluster and the sample workload is deployed; it's time to look at how the cluster is functioning.
Most of the Azure resources deployed in the prior steps will incur ongoing charges unless removed.
We have provided some sample deployment scripts that you could adapt for your own purposes while doing a POC/spike on this. Those scripts are found in the inner-loop-scripts directory. They include some additional considerations and may include some additional narrative as well. Consider checking them out. They consolidate most of the walk-through performed above into combined execution steps.
While this reference implementation tends to avoid preview features of AKS to ensure you have the best customer support experience; there are some features you may wish to evaluate in pre-production clusters that augment your posture around security, manageability, etc. As these features come out of preview, this reference implementation may be updated to incorporate them.
Consider trying out and providing feedback on the following:
- Automatic Node Upgrade
- Host-based encryption - Leverages added data encryption on your VMs' temp and OS disks.
- Generation 2 VM support - Increased memory options, Intel SGX support, and UEFI-based boot architectures.
- Auto Upgrade Profile support
- Customizable Node & Kublet config
- GitOps as an add-on
- Azure AD Pod Identity as an add-on
The AKS Baseline was used as the foundation for the following additional reference implementations. These build on the learnins of the AKS Baseline and applies a specific lens to the cluster to align a specific topology, requirement, and/or workload type.
- AKS Baseline for Multi-Region Clusters
- AKS Baseline for Regulated Workloads
- AKS Baseline for Microservices
- Azure Landing Zones, Enterprise-Scale Reference Implementation using Terraform
This reference implementation intentionally does not cover more advanced scenarios. For example topics like the following are not addressed:
- Cluster lifecycle management with regard to SDLC and GitOps
- Workload SDLC integration (including concepts like Bridge to Kubernetes, advanced deployment techniques, etc)
- Container security
- Multiple (related or unrelated) workloads owned by the same team
- Multiple workloads owned by disparate teams (AKS as a shared platform in your organization)
- Cluster-contained state (PVC, etc)
- Windows node pools
- Scale-to-zero node pools and event-based scaling (KEDA)
- Terraform
- Bedrock
- dapr
Keep watching this space, as we build out reference implementation guidance on topics such as these. Further guidance delivered will use this baseline AKS implementation as their starting point. If you would like to contribute or suggest a pattern built on this baseline, please get in touch.
Kubernetes is a very flexible platform, giving infrastructure and application operators many choices to achieve their business and technology objectives. At points along your journey, you will need to consider when to take dependencies on Azure platform features, OSS solutions, support channels, regulatory compliance, and operational processes. We encourage this reference implementation to be the place for you to start architectural conversations within your own team; adapting to your specific requirements, and ultimately delivering a solution that delights your customers.
- Azure Kubernetes Service Documentation
- Microsoft Azure Well-Architected Framework
- Microservices architecture on AKS
Please see our contributor guide.
This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.
With ❤️ from Microsoft Patterns & Practices, Azure Architecture Center.