Skip to content

Commit daa0523

Browse files
abaguasabaguas
committed
Create test setup for Azure DNS integration
As stated in #1772, we would like to have test setup with the main DNS providers. This PR creates a complete guide documenting how to test the Azure integration, including terraform and helm configuration. According to the [docs](https://azure.microsoft.com/en-us/pricing/details/dns/) the infrastructure should cost $0.90 per month, but it can also be destroyed and re-provisioned anytime: ``` First 25 hosted DNS zones $0.50 per zone per month First billion DNS queries/month $0.40 per million ``` Signed-off-by: Andre Aguas <[email protected]>
1 parent fae2e41 commit daa0523

14 files changed

+337
-0
lines changed

‎.gitignore

+5
Original file line numberDiff line numberDiff line change
@@ -32,3 +32,8 @@ changes
3232
cosign.key
3333
*.sig
3434
*.att
35+
36+
# Terraform
37+
terraform.tfstate*
38+
.terraform.lock.hcl
39+
.terraform/

‎dns-providers/README.md

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# DNS provider tests
2+
3+
In this space we document setups where we connect a local cluster to upstream DNS providers with the goal of testing the integrations.

‎dns-providers/azure/README.md

+114
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# Test Azure DNS integration from a local cluster
2+
3+
This is a guide how to test the Azure DNS integration of K8GB
4+
5+
## Azure infrastructure
6+
7+
### Azure subscription
8+
9+
First you will need an Azure subscription, if you don't have one already you can get started with a [free account](https://azure.microsoft.com/en-us/pricing/purchase-options/azure-account).
10+
Afterwards, login to your subscription in your terminal using `az login`.
11+
12+
Afterwards store your Azure subcription ID in an environment variable. We will use it as a terraform variable.
13+
```
14+
export ARM_SUBSCRIPTION_ID="$(az account show --query id -o tsv)"
15+
```
16+
17+
### DNS Zone and service principal
18+
19+
The next step is to create a DNS zone and a service principal that allows K8GB to modify records in the zone.
20+
You can use the terraform code provided in the `terraform` folder to get started. You will be prompted with the name of the DNS zone. The name needs to be unique in Azure, but you don't need to own the zone for the purpose of this guide:
21+
```
22+
$ cd terraform
23+
$ terraform init
24+
$ terraform apply
25+
var.dns_zone_name
26+
Name of the DNS zone
27+
28+
Enter a value: k8gb.io
29+
```
30+
31+
You will now have a service principal. You can retrieve its ID and secret with the following commands
32+
```
33+
terraform output service_principal_client_id
34+
terraform output service_principal_client_secret
35+
```
36+
37+
### Create local clusters
38+
39+
We have everything we need from Azure, we can now create a local cluster.
40+
This command will create the clusters `k3d-test-gslb1` and `k3d-test-gslb2`, and install k8gb from the branch you are on:
41+
```
42+
K8GB_LOCAL_VERSION=test FULL_LOCAL_SETUP_WITH_APPS=false make deploy-full-local-setup
43+
```
44+
45+
### Connect K8GB to Azure
46+
47+
At this moment K8GB is using the upstream DNS server running on the local cluster `k3d-edgedns`. We want to point it to the DNS infrastructure we created in Azure.
48+
49+
To do that we will need to create a secret on both clusters, on the `k8gb` namespace with the name `external-dns-secret-azure`. The format of the secret is documented in the [external dns docs](https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/azure.md#creating-a-configuration-file-for-the-service-principal). If you are in your `terraform` folder you can create it using:
50+
```
51+
cat <<-EOF > azure.json
52+
{
53+
"tenantId": "$(az account show --query tenantId -o tsv)",
54+
"subscriptionId": "$(az account show --query id -o tsv)",
55+
"resourceGroup": "rg-k8gb",
56+
"aadClientId": "$(terraform output --raw service_principal_client_id)",
57+
"aadClientSecret": "$(terraform output --raw service_principal_client_secret)"
58+
}
59+
EOF
60+
```
61+
Now apply the secret to both of the clusters:
62+
```
63+
kubectl create secret generic external-dns-secret-azure -n k8gb --from-file azure.json --context k3d-test-gslb1
64+
kubectl create secret generic external-dns-secret-azure -n k8gb --from-file azure.json --context k3d-test-gslb2
65+
```
66+
67+
### Create application
68+
69+
Finally, we can create a GSLB resouce that will trigger a reconciliation loop of the controller and configure DNS name delegation on Azure.
70+
To do that we will need to configure the DNS zone we create on K8GB:
71+
```
72+
# replace with your zone
73+
EDGE_DNS_ZONE="k8gb.io"
74+
```
75+
```
76+
DNS_ZONE="cloud.${EDGE_DNS_ZONE}"
77+
EDGE_DNS_SERVER=$(az network dns record-set ns list --resource-group rg-k8gb --zone-name "$EDGE_DNS_ZONE" --query "[?name=='@'].NSRecords[0].nsdname" --output tsv | sed 's/\.$//')
78+
```
79+
80+
```
81+
cd helm
82+
helm package -u . > /dev/null && helm template k8gb k8gb-v0.1.0.tgz -n k8gb -f values.yaml -f values-eu.yaml --set "k8gb.k8gb.dnsZone=$DNS_ZONE" --set "k8gb.k8gb.edgeDNSZone=$EDGE_DNS_ZONE" --set "k8gb.k8gb.edgeDNSServers[0]=$EDGE_DNS_SERVER" > manifests-eu.yaml
83+
helm package -u . > /dev/null && helm template k8gb k8gb-v0.1.0.tgz -n k8gb -f values.yaml -f values-us.yaml --set "k8gb.k8gb.dnsZone=$DNS_ZONE" --set "k8gb.k8gb.edgeDNSZone=$EDGE_DNS_ZONE" --set "k8gb.k8gb.edgeDNSServers[0]=$EDGE_DNS_SERVER" > manifests-us.yaml
84+
85+
kubectl apply -f manifests-eu.yaml --context k3d-test-gslb1
86+
kubectl apply -f manifests-us.yaml --context k3d-test-gslb2
87+
```
88+
89+
### Verify zone delegation in Azure
90+
91+
And voila, our local clusters are now integrated with Azure. We can quickly verify everything is working.
92+
93+
In Azure we should find the following records (the IP addresses may be different depending on the allocation by docker):
94+
| Name | Type | Value |
95+
| -------- | ------- | ------- |
96+
| cloud | NS | gslb-ns-eu-cloud.k8gb.io gslb-ns-us-cloud.k8gb.io
97+
| gslb-ns-eu-cloud | A | 172.18.0.6 172.18.0.7
98+
| gslb-ns-us-cloud | A | 172.18.0.10 172.18.0.11
99+
```
100+
az network dns record-set a list --resource-group rg-k8gb --zone-name "$EDGE_DNS_ZONE" --output json
101+
az network dns record-set ns list --resource-group rg-k8gb --zone-name "$EDGE_DNS_ZONE" --output json
102+
```
103+
104+
You can also fetch the records using the following DNS query:
105+
```
106+
dig @${EDGE_DNS_SERVER} cloud.k8gb.io
107+
...
108+
;; AUTHORITY SECTION:
109+
cloud.k8gb.io. 5 IN NS gslb-ns-eu-cloud.k8gb.io.
110+
cloud.k8gb.io. 5 IN NS gslb-ns-us-cloud.k8gb.io.
111+
...
112+
```
113+
114+
Unfortunately the A records cannot be queried because they are private IP addresses and Azure does not return them in a public DNS zone, but this is enough for testing.

‎dns-providers/azure/helm/Chart.yaml

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
apiVersion: v2
2+
name: k8gb
3+
description: A Helm chart for Kubernetes Global Balancer
4+
icon: https://www.k8gb.io/assets/images/icon-192x192.png
5+
type: application
6+
version: v0.1.0
7+
dependencies:
8+
- name: k8gb
9+
repository: file://../../../chart/k8gb
10+
version: v0.14.0
11+
12+
home: https://www.k8gb.io/
13+
sources:
14+
- https://github.com/k8gb-io/k8gb

‎dns-providers/azure/helm/templates/gslb.yaml

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
---
2+
apiVersion: k8gb.absa.oss/v1beta1
3+
kind: Gslb
4+
metadata:
5+
name: failover-playground-istio
6+
namespace: test-azure
7+
spec:
8+
resourceRef:
9+
apiVersion: networking.istio.io/v1
10+
kind: VirtualService
11+
matchLabels:
12+
app: failover-playground-istio
13+
strategy:
14+
type: failover
15+
dnsTtlSeconds: 5
16+
primaryGeoTag: "eu"
17+
---
18+
apiVersion: networking.istio.io/v1
19+
kind: VirtualService
20+
metadata:
21+
name: failover-playground-istio
22+
namespace: test-azure
23+
labels:
24+
app: failover-playground-istio
25+
spec:
26+
gateways:
27+
- istio-ingress/failover-playground-istio
28+
hosts:
29+
- failover-playground-istio.{{ .Values.k8gb.k8gb.dnsZone }}
30+
http:
31+
- route:
32+
- destination:
33+
host: frontend-podinfo
34+
port:
35+
number: 9898
36+
---
37+
apiVersion: networking.istio.io/v1
38+
kind: Gateway
39+
metadata:
40+
name: failover-playground-istio
41+
namespace: istio-ingress
42+
spec:
43+
selector:
44+
app: istio-ingressgateway
45+
servers:
46+
- hosts:
47+
- failover-playground-istio.{{ .Values.k8gb.k8gb.dnsZone }}
48+
port:
49+
name: http
50+
number: 8080
51+
protocol: http

‎dns-providers/azure/helm/templates/ns.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: v1
2+
kind: Namespace
3+
metadata:
4+
name: test-azure
5+
labels:
6+
istio-injection: enabled

‎dns-providers/azure/helm/templates/podinfo.yaml

+43
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
apiVersion: v1
3+
kind: Service
4+
metadata:
5+
name: frontend-podinfo
6+
namespace: test-azure
7+
labels:
8+
app.kubernetes.io/name: frontend-podinfo
9+
spec:
10+
type: ClusterIP
11+
ports:
12+
- port: 9898
13+
targetPort: http
14+
protocol: TCP
15+
name: http
16+
selector:
17+
app.kubernetes.io/name: frontend-podinfo
18+
---
19+
apiVersion: v1
20+
kind: Pod
21+
metadata:
22+
name: frontend-podinfo
23+
namespace: test-azure
24+
labels:
25+
app.kubernetes.io/name: frontend-podinfo
26+
spec:
27+
containers:
28+
- name: podinfo
29+
image: "ghcr.io/stefanprodan/podinfo:5.1.1"
30+
command:
31+
- ./podinfo
32+
- --port=9898
33+
ports:
34+
- name: http
35+
containerPort: 9898
36+
protocol: TCP
37+
resources:
38+
requests:
39+
memory: 64Mi
40+
cpu: 250m
41+
limits:
42+
memory: 128Mi
43+
cpu: 500m

‎dns-providers/azure/helm/values-eu.yaml

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
k8gb:
2+
k8gb:
3+
clusterGeoTag: "eu"
4+
extGslbClustersGeoTags: "us"

‎dns-providers/azure/helm/values-us.yaml

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
k8gb:
2+
k8gb:
3+
clusterGeoTag: "us"
4+
extGslbClustersGeoTags: "eu"

‎dns-providers/azure/helm/values.yaml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
k8gb:
2+
k8gb:
3+
dnsZone: "<helm set>"
4+
edgeDNSZone: "<helm set>"
5+
edgeDNSServers:
6+
- "<helm set"
7+
8+
azuredns:
9+
enabled: true
10+
authSecretName: external-dns-secret-azure
11+
createAuthSecret:
12+
enabled: false

‎dns-providers/azure/terraform/main.tf

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# dns zone
2+
3+
resource "azurerm_resource_group" "this" {
4+
name = var.resource_group_name
5+
location = var.resource_group_location
6+
}
7+
8+
resource "azurerm_dns_zone" "this" {
9+
name = var.dns_zone_name
10+
resource_group_name = azurerm_resource_group.this.name
11+
}
12+
13+
# service principal
14+
15+
resource "azuread_application" "k8gb" {
16+
display_name = "k8gb"
17+
}
18+
19+
resource "azuread_service_principal" "k8gb" {
20+
client_id = azuread_application.k8gb.client_id
21+
}
22+
23+
resource "azuread_service_principal_password" "k8gb" {
24+
service_principal_id = azuread_service_principal.k8gb.id
25+
end_date = "2099-01-01T00:00:00Z"
26+
}
27+
28+
resource "azurerm_role_assignment" "dns_zone_contributor" {
29+
principal_id = azuread_service_principal.k8gb.object_id
30+
role_definition_name = "DNS Zone Contributor"
31+
scope = azurerm_dns_zone.this.id
32+
}
33+
34+
resource "azurerm_role_assignment" "resource_group_reader" {
35+
principal_id = azuread_service_principal.k8gb.object_id
36+
role_definition_name = "Reader"
37+
scope = azurerm_resource_group.this.id
38+
}

‎dns-providers/azure/terraform/output.tf

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
output "service_principal_client_id" {
2+
value = azuread_service_principal.k8gb.client_id
3+
description = "client id of the service principal"
4+
}
5+
6+
output "service_principal_client_secret" {
7+
value = azuread_service_principal_password.k8gb.value
8+
description = "client secret of the service principal"
9+
sensitive = true
10+
}

‎dns-providers/azure/terraform/providers.tf

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
terraform {
2+
required_version = ">=1.9"
3+
required_providers {
4+
azurerm = {
5+
source = "hashicorp/azurerm"
6+
version = "4.8.0"
7+
}
8+
azuread = {
9+
source = "hashicorp/azuread"
10+
version = "3.0.2"
11+
}
12+
}
13+
}
14+
provider "azurerm" {
15+
features {}
16+
}
17+
provider "azuread" {}

‎dns-providers/azure/terraform/variables.tf

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
variable "resource_group_location" {
2+
type = string
3+
default = "germanywestcentral"
4+
description = "Location for all resources"
5+
}
6+
7+
variable "resource_group_name" {
8+
type = string
9+
default = "rg-k8gb"
10+
description = "Resource group name to be created"
11+
}
12+
13+
variable "dns_zone_name" {
14+
type = string
15+
description = "Name of the DNS zone"
16+
}

0 commit comments

Comments
 (0)