k8s/AzureAKS

• Links https://docs.microsoft.com/en-us/azure/aks/kubernetes-service-principal?tabs=azure-cli

• 2024 AKS CNI + Cilium
• SP - Service principal - used by the k8s cluster

  • NOTE: only valid for 1 year - rotate credentials

    • Saved on azure node(vm) at /etc/kubernetes/azure.json
    • If created with az aks create, saved on laptop at ~/.azure/aksServicePrincipal.json
    • got to delete SP, on cluster deletion, not done for you. query for your clusters servicePrincipalProfile.clientId and then delete it using the az ad sp delete
  • Give it roles/access e.g.
    • Access to Azure Container registry to pull images
    • Access storage in different RG
    • Access IP in different RG

Create AKS

• Go

  RG_NAME=myResourceGroup-NP
  CLUSTER_NAME=myAKSCluster
  LOCATION=canadaeast
  
  Create the AKS cluster and specify *azure* for the network plugin and network policy.
  
  ```azurecli
  az aks create \
      --resource-group $RG_NAME \
      --name $CLUSTER_NAME \
      --node-count 1 \
      --network-plugin azure \
      --network-policy azure

AKS RBAC

• Assign role to AD group

  AKS_ID=$(az aks show \
      --resource-group myRGroup \
      --name myAKSCluster \
      --query id -o tsv)
  
  APPDEV_ID=$(az ad group create \
      --display-name appdev \
      --mail-nickname appdev \
      --query objectId -o tsv)
  
  az role assignment create \
    --assignee $APPDEV_ID \
    --role "Azure Kubernetes Service Cluster User Role" \
    --scope $AKS_ID

AKS Autoscaler and NodePools

• GitHub - https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/expander/priority/readme.md

• Explained - https://www.thorsten-hans.com/aks-cluster-auto-scaler-inside-out/#why-should-you-use-cluster-auto-scaler

• Azure doc - https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile

• e.g. ConfigMap - https://docs.giantswarm.io/advanced/spot-instances/azure/ondemand-fallback/

• Use priorities to pick preferd pool order, From https://github.com/Azure/AKS/issues/2359

  az aks update --subscription xxx --resource-group xxx --name xxx --cluster-autoscaler-profile expander=priority

AKS spot node pool

• k8s node run with Taints: kubernetes.azure.com/scalesetpriority=spot:NoSchedule