Summary

Operational muscle-memory for the CKA: the setup workflows I don’t run daily, each as imperative-first → declarative-where-needed → verify. Covers RBAC, kubeadm cluster create/upgrade, workloads & scheduling, storage, and services/networking. Troubleshooting is deliberately excluded. Assumes the study-plan shell setup (k=kubectl, do="--dry-run=client -o yaml", now="--force --grace-period=0").

Rule of thumb: if a kubectl create generator exists, use it and pipe through $do to get editable YAML. Four things have no generator and must be hand-written — PV/PVC/StorageClass, NetworkPolicy, Gateway/HTTPRoute. Those are the ones to drill.

1. RBAC

Model: subjects (User, Group, ServiceAccount) get verbs on resources via a Role (namespaced) or ClusterRole (cluster-wide), connected by a binding. Four objects, two scopes:

Grant scope Permission object Binding object
One namespace Role RoleBinding
Cluster-wide ClusterRole ClusterRoleBinding

The subtlety that trips people: a ClusterRole bound with a RoleBinding grants its verbs only inside that RoleBinding’s namespace. That’s the intended way to reuse the built-in view / edit / admin ClusterRoles per-namespace without redefining them.

ServiceAccount → attach to a workload

k create sa app-sa                                  # imperative
k set serviceaccount deploy/web app-sa              # attach to existing deployment
# or in a pod/deployment spec:  spec.serviceAccountName: app-sa

Role + RoleBinding + SA (namespaced grant)

k create role pod-reader --verb=get,list,watch --resource=pods $do
k create rolebinding pr-b --role=pod-reader --serviceaccount=default:app-sa $do

Bind an SA to an existing ClusterRole (view)

# scoped to one namespace (ClusterRole + RoleBinding pattern):
k create rolebinding view-b --clusterrole=view --serviceaccount=default:app-sa -n default

# cluster-wide (all namespaces):
k create clusterrolebinding view-all --clusterrole=view --serviceaccount=default:app-sa

Users (client certs)

There is no User object in Kubernetes — a user is just the CN of a client cert the cluster CA trusts, and the cert’s O (organization) fields become the user’s groups. “Create a user” = get a cert signed by /etc/kubernetes/pki/ca.{crt,key}. Three ways on a kubeadm cluster, fastest first:

# 1. kubeadm does it all — signs with the cluster CA, prints a ready-to-use kubeconfig
kubeadm kubeconfig user --client-name=jane --org=dev > jane.kubeconfig   # --org repeatable

# 2. sign the cert yourself against the kubeadm CA
openssl genrsa -out jane.key 2048
openssl req -new -key jane.key -out jane.csr -subj "/CN=jane/O=dev"       # CN=user, O=group
openssl x509 -req -in jane.csr -days 365 -CAcreateserial -out jane.crt \
  -CA /etc/kubernetes/pki/ca.crt -CAkey /etc/kubernetes/pki/ca.key

# 3. CSR API — portable, needs no access to the CA key
cat <<EOF | k apply -f -
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata: { name: jane }
spec:
  request: $(base64 -w0 jane.csr)
  signerName: kubernetes.io/kube-apiserver-client
  expirationSeconds: 86400
  usages: [client auth]
EOF
k certificate approve jane
k get csr jane -o jsonpath='{.status.certificate}' | base64 -d > jane.crt

Wire the cert into a kubeconfig (skip if you used method 1 — it already did this):

k config set-credentials jane --client-key=jane.key --client-certificate=jane.crt --embed-certs
k config set-context jane --cluster=kubernetes --user=jane

Grant + verify — bind to the user (CN) or the whole group (O):

k create rolebinding jane-edit --clusterrole=edit --user=jane -n dev       # namespaced, one user
k create clusterrolebinding dev-view --clusterrole=view --group=dev        # everyone with O=dev
k auth can-i get pods --as=jane --as-group=dev -n dev
k --kubeconfig jane.kubeconfig get pods -n dev                             # the real end-to-end test

Verify as admin — the whole point

k auth can-i list pods --as=system:serviceaccount:default:app-sa -n default   # yes/no
k auth can-i --list --as=system:serviceaccount:default:app-sa                  # full matrix
k auth can-i '*' '*' --as=system:serviceaccount:default:app-sa                 # admin check
k auth whoami                                                                  # who am I now

Subject string for a ServiceAccount is always system:serviceaccount:<ns>:<name>. Useful flags on create role: --resource=pods/log (subresources), --resource-name=my-pod (restrict to a named object), --verb='*'.

2. kubeadm — create + upgrade

Full command rhythm below; see [[kubeadm]] for the why (static pods, PKI, phases).

Create a simple cluster

# --- every node: prereqs ---
sudo swapoff -a                                     # kubelet refuses to start with swap
sudo sed -i '/ swap / s/^/#/' /etc/fstab            # keep it off after reboot
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay && sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sudo sysctl --system
# containerd installed + running, cgroup driver = systemd

# --- control-plane node ---
sudo kubeadm init \
  --pod-network-cidr=10.244.0.0/16 \
  --control-plane-endpoint=10.0.0.10:6443 \
  --kubernetes-version=v1.35.0

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

k apply -f <CNI-manifest>                           # e.g. Calico/Flannel; nodes NotReady until CNI is up
sudo kubeadm token create --print-join-command      # copy the printed join line

# --- worker node ---
sudo kubeadm join 10.0.0.10:6443 --token <t> --discovery-token-ca-cert-hash sha256:<hash>

Verify: k get nodes -o wide → all Ready, k get pods -n kube-system.

Upgrade (control-plane first, then each worker)

# ===== first control-plane node =====
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.35.1-*
sudo apt-mark hold kubeadm

sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.35.1                  # apply = FIRST control-plane only

k drain <cp-node> --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get install -y kubelet=1.35.1-* kubectl=1.35.1-*
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
k uncordon <cp-node>

# ===== each other node (workers + extra CP nodes), SSH in =====
sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.35.1-* && sudo apt-mark hold kubeadm
sudo kubeadm upgrade node                           # node = every node except the first CP
# then drain (from where kubectl lives) → install kubelet/kubectl → restart → uncordon

Verify: k get nodes shows the new VERSION on each node after its turn.

3. Cluster config: Helm, Kustomize, CRDs

All three sit in the 25% Cluster Architecture domain — install and templating tooling.

Helm

helm repo add bitnami https://charts.bitnami.com/bitnami && helm repo update
helm search repo nginx
helm show values bitnami/nginx                 # discover what's configurable
helm install web bitnami/nginx -n web --create-namespace \
  -f values.yaml --set service.type=NodePort   # -f files first, then --set; --set wins conflicts
helm upgrade web bitnami/nginx --set replicaCount=3
helm rollback web 1
helm list -A
helm uninstall web -n web
helm template web bitnami/nginx                # render manifests locally, no cluster contact
helm install web bitnami/nginx --dry-run --debug   # preview against the live API
helm get values web                            # what a release was actually installed with

Kustomize

kubectl ships Kustomize built in (-k and kubectl kustomize). Base + overlay:

# base/kustomization.yaml
resources: [deployment.yaml, service.yaml]
---
# overlay/prod/kustomization.yaml
resources: [../../base]
namePrefix: prod-
namespace: prod
labels:                          # replaces the older commonLabels
  - pairs: { env: prod }
    includeSelectors: true
replicas:
  - { name: web, count: 5 }
images:
  - { name: nginx, newTag: "1.27" }
patches:
  - path: patch.yaml
    target: { kind: Deployment, name: web }
configMapGenerator:              # name gets a content hash → editing config forces a rollout
  - name: app
    literals: [COLOR=blue]

kubectl kustomize overlay/prod/       # preview the rendered YAML
kubectl apply -k overlay/prod/        # build + apply

CRDs & Operators

Apply order is the trap — the CRD must be registered before any custom resource of that kind:

k apply -f crd.yaml
k get crds && k api-resources | grep <group>
k apply -f cr-instance.yaml
k get <crd-shortname>

Operators usually ship as a bundle (CRDs + RBAC + a controller Deployment), frequently installed via a Helm chart.

4. Workloads & Scheduling

Deployment lifecycle (all imperative)

k create deploy web --image=nginx --replicas=3
k scale deploy web --replicas=5
k set image deploy/web nginx=nginx:1.27
k annotate deploy/web kubernetes.io/change-cause="bump to 1.27"  # shows in rollout history; --record is gone
k rollout status deploy/web
k rollout history deploy/web
k rollout undo deploy/web                            # roll back one revision
k rollout undo deploy/web --to-revision=2
k expose deploy web --port=80 --target-port=8080     # creates a ClusterIP Service

StatefulSets & DaemonSets

No imperative generator for either — hand-write, or k create deploy x --image=… $do then change kind: and fields.

# DaemonSet — one pod per matching node, no replicas
apiVersion: apps/v1
kind: DaemonSet
metadata: { name: node-agent }
spec:
  selector: { matchLabels: { app: agent } }
  template:
    metadata: { labels: { app: agent } }
    spec:
      tolerations:                     # to also land on tainted control-plane nodes
        - { key: node-role.kubernetes.io/control-plane, effect: NoSchedule }
      containers:
        - { name: agent, image: fluentd }

# StatefulSet — stable identity + per-pod storage; needs a headless Service
apiVersion: v1
kind: Service                # clusterIP: None → DNS web-0.web.<ns>.svc.cluster.local
metadata: { name: web }
spec:
  clusterIP: None
  selector: { app: web }
  ports: [{ port: 80 }]
---
apiVersion: apps/v1
kind: StatefulSet
metadata: { name: web }
spec:
  serviceName: web           # must reference the headless Service above
  replicas: 3
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      containers:
        - name: web
          image: nginx
          volumeMounts: [{ name: data, mountPath: /usr/share/nginx/html }]
  volumeClaimTemplates:      # one PVC per pod: data-web-0, data-web-1, data-web-2
    - metadata: { name: data }
      spec:
        accessModes: [ReadWriteOnce]
        resources: { requests: { storage: 1Gi } }

Pods get stable ordinals (web-0..web-2), created/deleted in order. drain --ignore-daemonsets exists precisely because DaemonSet pods are node-bound and would just be recreated.

ConfigMaps & Secrets

k create cm app --from-literal=COLOR=blue --from-literal=MODE=fast
k create cm app-file --from-file=app.properties
k create secret generic db --from-literal=pw=s3cr3t

Consume as env vars or as a mounted volume:

# env — all keys at once, or one key
envFrom:
  - configMapRef: { name: app }
env:
  - name: DB_PW
    valueFrom:
      secretKeyRef: { name: db, key: pw }
# volume
volumes:
  - name: cfg
    configMap: { name: app-file }
volumeMounts:
  - name: cfg
    mountPath: /etc/app

Scheduling controls

k label node n1 disk=ssd                             # then nodeSelector: {disk: ssd}
k taint nodes n1 tier=db:NoSchedule                  # repel pods without a matching toleration
k taint nodes n1 tier=db:NoSchedule-                 # trailing '-' removes the taint

# nodeSelector — simplest hard constraint
nodeSelector: { disk: ssd }

# nodeAffinity — required (hard) vs preferred (soft)
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - { key: disk, operator: In, values: [ssd] }
    preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 50
        preference:
          matchExpressions:
            - { key: zone, operator: In, values: [a] }

# toleration — lets a pod land on the tainted node above
tolerations:
  - { key: tier, operator: Equal, value: db, effect: NoSchedule }

# podAntiAffinity — spread replicas across hosts
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels: { app: web }
        topologyKey: kubernetes.io/hostname

Bypass the scheduler entirely with nodeName: n1 in the pod spec, or drop a manifest into /etc/kubernetes/manifests/ on a node for a static pod (kubelet runs it, a mirror pod named <pod>-<node> shows up in the API).

Node maintenance: k cordon n1 (stop new pods) → k drain n1 --ignore-daemonsets --delete-emptydir-data (evict) → k uncordon n1.

HPA

k autoscale deploy web --min=2 --max=5 --cpu-percent=80    # needs metrics-server
k get hpa                                                  # TARGETS shows current/target %

Jobs & CronJobs

k create job pi --image=perl -- perl -Mbignum -wle 'print bpi(200)'
k create cronjob report --image=busybox --schedule="*/5 * * * *" -- /bin/sh -c date
k create job manual --from=cronjob/report            # run a cronjob once, right now

Resource requests, limits & quotas

k set resources deploy/web --requests=cpu=100m,memory=128Mi --limits=cpu=500m,memory=256Mi
k set env deploy/web LOG_LEVEL=debug
k create ns team-a
k create quota q --hard=cpu=2,memory=2Gi,pods=10 -n team-a

# per-container, in the pod spec
resources:
  requests: { cpu: 100m, memory: 128Mi }   # what the scheduler reserves — drives placement
  limits:   { cpu: 500m, memory: 256Mi }   # runtime ceiling — CPU throttled, memory over = OOMKilled

requests are the number the scheduler bin-packs against; a pod requesting more than a node has free stays Pending. A LimitRange sets per-container defaults + min/max in a namespace; a ResourceQuota caps the namespace total. Both reject over-limit pods at admission, before the scheduler runs. Verify: k describe quota -n team-a.

5. Services & Networking

Services

k expose deploy web --port=80 --target-port=8080                 # ClusterIP (default)
k expose deploy web --port=80 --type=NodePort                    # adds a node port (30000-32767)
k create service clusterip web --tcp=80:8080 $do                 # without an existing deploy
k get endpoints web            # or: k get endpointslices -l kubernetes.io/service-name=web

A Service finds pods by label selector; if k get endpoints web is empty, the selector doesn’t match the pod labels — the #1 “service doesn’t work” cause.

Ingress

k create ingress web --class=nginx \
  --rule="ex.com/app*=web:80" \
  --rule="ex.com/api*=api:80" $do

Declarative when you need TLS or explicit pathType:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: { name: web }
spec:
  ingressClassName: nginx
  tls:
    - hosts: [ex.com]
      secretName: ex-tls          # kubectl create secret tls ex-tls --cert=… --key=…
  rules:
    - host: ex.com
      http:
        paths:
          - path: /app
            pathType: Prefix       # Prefix | Exact | ImplementationSpecific
            backend:
              service: { name: web, port: { number: 80 } }

Gateway API (no generator — hand-write all three)

The exam’s least-reflexive YAML. Three objects: GatewayClass (who implements it) → Gateway (listeners/ports) → HTTPRoute (routing rules, attached via parentRefs).

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata: { name: prod }
spec: { controllerName: example.com/gateway-controller }
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata: { name: web-gw }
spec:
  gatewayClassName: prod
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: web-route }
spec:
  parentRefs:
    - name: web-gw
  hostnames: ["ex.com"]
  rules:
    - matches:
        - path: { type: PathPrefix, value: /app }
      backendRefs:
        - name: web
          port: 80

Verify: k get gateway web-gw → PROGRAMMED=True; k describe httproute web-route.

NetworkPolicy (no generator — hand-write)

Key mechanics: a pod is unrestricted until some policy selects it; selecting it flips that pod to default-deny for the listed policyTypes, and you then allow traffic back in. Policies are additive (union of all that match). To scope to a deployment, target the deployment’s pod labels in podSelector.

# default-deny everything in a namespace (both directions)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: app }
spec:
  podSelector: {}                 # {} = every pod in the namespace
  policyTypes: [Ingress, Egress]
---
# allow: frontend pods → web:80, and DNS egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: web-allow, namespace: app }
spec:
  podSelector:
    matchLabels: { app: web }     # the deployment's pods
  policyTypes: [Ingress, Egress]
  ingress:
    - from:
        - podSelector: { matchLabels: { role: frontend } }
        - namespaceSelector: { matchLabels: { team: web } }
        - ipBlock: { cidr: 10.0.0.0/16, except: [10.0.5.0/24] }
      ports:
        - { protocol: TCP, port: 80 }
  egress:
    - to: []                      # DNS to anywhere
      ports:
        - { protocol: UDP, port: 53 }

Watch the YAML shape: entries under one from are OR’d; splitting a podSelector + namespaceSelector into two list items (as above) is OR, but nesting them under a single - from: item is AND (pods with that label in that namespace).

6. Storage

Model: PV is a cluster-scoped piece of storage; PVC is a namespaced request; Kubernetes binds a PVC to a PV that satisfies capacity + access mode + storageClassName (+ optional label selector). A pod mounts the PVC, not the PV.

Access modes — note RWO is per node, not per pod:

Mode Short Meaning
ReadWriteOnce RWO mounted read-write by a single node (multiple pods on that node can share)
ReadOnlyMany ROX read-only by many nodes
ReadWriteMany RWX read-write by many nodes (needs a networked backend, e.g. NFS)
ReadWriteOncePod RWOP read-write by exactly one pod (1.29+) — the true exclusive lock

Reclaim policy (what happens to the PV when its PVC is deleted): Retain → PV goes to Released and will not rebind until an admin clears spec.claimRef; Delete → backing storage is deleted with the PVC. (Recycle is deprecated.)

Static PV + PVC + pod (hand-written)

apiVersion: v1
kind: PersistentVolume
metadata: { name: pv-data }
spec:
  capacity: { storage: 5Gi }
  accessModes: [ReadWriteOnce]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: manual         # must match the PVC to bind
  hostPath: { path: /mnt/data }    # demo only; real clusters use nfs/csi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: { name: pvc-data }
spec:
  accessModes: [ReadWriteOnce]
  storageClassName: manual
  resources: { requests: { storage: 5Gi } }

# in the pod/deployment spec
volumes:
  - name: data
    persistentVolumeClaim: { claimName: pvc-data }
volumeMounts:
  - name: data
    mountPath: /var/lib/data

Verify: k get pv,pvc → both Bound to each other.

Dynamic provisioning (StorageClass)

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: fast
  annotations: { storageclass.kubernetes.io/is-default-class: "true" }
provisioner: kubernetes.io/aws-ebs        # your cluster's CSI driver
volumeBindingMode: WaitForFirstConsumer   # bind once a pod schedules (respects topology); vs Immediate
allowVolumeExpansion: true                # required for later resize
reclaimPolicy: Delete

A PVC that just names storageClassName: fast (or omits it, using the default class) triggers the provisioner to create a PV automatically.

Resize a PVC

k patch pvc pvc-data -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}'
k get pvc pvc-data          # CAPACITY updates once expansion completes

Only grows, never shrinks; the StorageClass must have allowVolumeExpansion: true; some drivers need the pod to restart before the filesystem picks up the new size.

Gotchas

  • ClusterRole + RoleBinding = namespace-scoped grant. Same ClusterRole + ClusterRoleBinding = cluster-wide. This is how you reuse view/edit/admin per namespace.
  • SA subject format in auth can-i --as is system:serviceaccount:<ns>:<name> — miss the prefix and the check silently answers for a user of that literal name.
  • auth can-i --as=<sa> doesn’t reconstruct the SA’s groups. Impersonation is literal: Kubernetes adds system:authenticated but not system:serviceaccounts / system:serviceaccounts:<ns>. If a binding targets an SA group rather than the SA itself, add --as-group=system:serviceaccounts:<ns> or the check falsely answers “no.” Same for user certs — the groups come from the cert’s O fields, so pass --as-group to mirror them.
  • A user is a cert CN, not an object — nothing to “get” (k get user doesn’t exist). k create user doesn’t exist either; you sign a cert (kubeadm kubeconfig user / CSR API) and bind to --user=<CN> / --group=<O>.
  • kubeadm upgrade apply is first-control-plane-node-only; every other node (workers and extra control-plane nodes) uses kubeadm upgrade node.
  • Drain before upgrading/restarting the kubelet, uncordon after — restarting kubelet disrupts that node’s pods.
  • Helm value precedence: -f files apply in order, then --set overrides them (--set wins). Use helm template or helm install --dry-run --debug to preview without touching the cluster.
  • kubectl apply -k uses kubectl’s embedded Kustomize (can lag the standalone kustomize binary); configMapGenerator/secretGenerator append a content hash to the name, so editing generated config forces a rolling update — a hand-edited ConfigMap does not.
  • Install a CRD before any custom resource of its kind — apply order matters, and k get <kind> errors until the CRD is registered.
  • ResourceQuota / LimitRange reject over-limit pods at admission — before the scheduler runs, so the pod never even reaches Pending for a node.
  • DaemonSets have no replicas — one pod per matching node, so scheduling is driven by nodeSelector/affinity/tolerations, not a count. This is why drain needs --ignore-daemonsets.
  • StatefulSet volumeClaimTemplates PVCs are never auto-deleted — scaling down or deleting the StatefulSet leaves data-web-N behind (retained on purpose); clean them up by hand. And a StatefulSet needs a headless Service (clusterIP: None) or the per-pod DNS names won’t resolve.
  • Empty Service endpoints = selector/label mismatch. Check k get endpoints <svc> before suspecting anything else.
  • NetworkPolicy default-deny is triggered by selecting a pod — an empty podSelector: {} with a policyTypes list denies that direction for the whole namespace. In a from/to block, separate list items are OR; selectors nested in one item are AND.
  • RWO is per node, not per pod — two pods on the same node can both mount an RWO volume. Use RWOP for a true single-pod lock.
  • Retain PVs don’t auto-rebind — after the PVC is gone the PV sits in Released; clear spec.claimRef to reuse it.
  • PVCs only grow, and only when the StorageClass allows expansion.
  • SA tokens aren’t long-lived mounted Secrets anymore (1.24+) — they’re short-lived projected tokens via TokenRequest; set automountServiceAccountToken: false to opt a pod out.

Open questions

  • Gateway API is GA and on the exam — is it tested instead of Ingress now, or alongside? Worth timing both cold.
  • Does any storage task actually exercise WaitForFirstConsumer binding behavior, or is Immediate enough for exam scenarios?
  • PVC resize: which common CSI drivers do online (no restart) vs offline expansion — matters for whether the “resize” task needs a pod bounce to show new capacity.

References