• k8s
• AutoScaler

k8s/AutoScaler

• k8s - builtin GA autoscaler - https://github.com/kubernetes/autoscaler/tree/master

View/Debug

• History and status

   kubectl get configmaps -n kube-system cluster-autoscaler-status -o yaml
  
  apiVersion: v1
  data:
    status: |+
      Cluster-autoscaler status at 2024-03-20 01:10:58.844440912 +0000 UTC:
      Cluster-wide:
        Health:      Healthy (ready=5 unready=0 (resourceUnready=0) notStarted=0 longNotStarted=0 registered=5 longUnregistered=0)
                     LastProbeTime:      2024-03-20 01:10:58.66229529 +0000 UTC m=+412459.398307389
                     LastTransitionTime: 2024-03-15 06:36:53.20437346 +0000 UTC m=+13.940385459
        ScaleUp:     NoActivity (ready=5 registered=5)
                     LastProbeTime:      2024-03-20 01:10:58.66229529 +0000 UTC m=+412459.398307389
                     LastTransitionTime: 2024-03-15 06:36:53.20437346 +0000 UTC m=+13.940385459
        ScaleDown:   NoCandidates (candidates=0)
                     LastProbeTime:      2024-03-20 01:10:58.66229529 +0000 UTC m=+412459.398307389
                     LastTransitionTime: 2024-03-15 06:36:53.20437346 +0000 UTC m=+13.940385459

Config / Priority Scaling

• The k8s cluster must be deployed/configured with expander=priority. e.g. terraform snippet

  resource "azurerm_kubernetes_cluster" "k8s" {
    name       = ...
    ...
    ...
   auto_scaler_profile {
      # https://learn.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile
      balance_similar_node_groups   = false # Pick Zone's 2022 only on scale up
      expander                      = "priority"
      max_graceful_termination_sec  = 60    # default 600 seconds
      max_node_provisioning_time    = "10m" # default 15m
      skip_nodes_with_local_storage = false
      skip_nodes_with_system_pods   = false
      #lots more settings see https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/resources/kubernetes_cluster#private_cluster_public_fqdn_enabled
    }
    ...
  }

• Deploy configmap into cluster

  #Config map used with AKS priority scaling - autoscaler - expander - priority.
  # regex match .* = anyting NodePool name. from high to low.
  # https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/expander/priority
  # Status:  kubectl get configmaps -n kube-system cluster-autoscaler-status -o yaml
  # See: Terrafrom - tm-infra/terraform/modules/aks-k8s-cluster/main-k8s.tf for priority.
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: cluster-autoscaler-priority-expander
    namespace: kube-system
  data:
    # Regex to match VMSS scaleset
    priorities: |-
          70:
            - "aks-spote4sv3-.*"
          60:
            - "aks-spotd4sv3-.*"
          40:
            - "aks-spotd8sv3-.*"
            - "aks-spote8sv3-.*"
          30:
            - ".*spot.*"
          10:
            - ".*"