kubeadm is a setup tool, not a daemon. It generates config and hands off to the kubelet. On the CKA it shows up most in Cluster Architecture, Installation & Configuration (25%) — upgrade workflow and token regeneration are near-certain tasks.
The mental model: kubeadm writes files; the kubelet acts on them. Everything the control plane “runs” — apiserver, etcd, scheduler, controller-manager — the kubelet runs as static pods by watching /etc/kubernetes/manifests/. kubeadm writes those manifests and the PKI certs they reference, then exits. This explains most of kubeadm’s behavior: why bouncing static pods requires touching the manifests directory, why cert renewal doesn’t restart anything automatically, why kubeadm itself isn’t present after init.
Almost every subcommand supports phase-level execution (kubeadm <cmd> phase --help), which lets you run one step in isolation. Useful when something fails mid-init or mid-upgrade.
Bootstraps the first control-plane node. Under the hood: generates the CA and all component certs under /etc/kubernetes/pki/, writes static pod manifests to /etc/kubernetes/manifests/ (kubelet picks these up immediately), creates the kubelet-config ConfigMap in kube-system, and prints a join command.
sudo kubeadm init \
--pod-network-cidr=10.244.0.0/16 \
--apiserver-advertise-address=10.0.0.10 \
--control-plane-endpoint=10.0.0.10:6443 \
--kubernetes-version=v1.35.0 \
--upload-certs \
--cri-socket=unix:///var/run/containerd/containerd.sock
--control-plane-endpoint should point to a load balancer or stable DNS — not the node IP — if there’s any chance of adding more control-plane nodes later. It can’t be cleanly retrofitted after init.
--upload-certs stores the control-plane certs encrypted in a Secret in kube-system so additional control-plane nodes can pull them during join --control-plane. The encryption key expires in ~2h.
Post-init, copy the kubeconfig so kubectl works as your user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Joins a node to the cluster. The mechanism differs between worker and control-plane nodes. For a worker: the kubelet presents the bootstrap token to the apiserver, gets a CSR signed (TLS bootstrapping), and from that point authenticates with its own client cert — the token is scaffolding it no longer needs. For an additional control-plane node: same flow, plus it pulls the control-plane certs from the Secret --upload-certs created, so it can serve the same CA-signed apiserver cert.
# Worker
sudo kubeadm join 10.0.0.10:6443 \
--token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:<hash>
# Additional control-plane node
sudo kubeadm join 10.0.0.10:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane \
--certificate-key <cert-key>
The CA cert hash pins which cluster the joining node is allowed to trust — it prevents a rogue apiserver from hijacking the join. Compute it manually if you lost the init output:
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt \
| openssl rsa -pubin -outform der 2>/dev/null \
| openssl dgst -sha256 -hex | sed 's/^.* //'
Bootstrap tokens are Kubernetes Secrets in kube-system with names like bootstrap-token-<id>. The apiserver has a token authentication plugin that reads these Secrets — no separate token service. They exist only to get a new kubelet through TLS bootstrapping; once it has a client cert, the token is irrelevant. That’s why 24h TTL is the right default: long enough to bootstrap, short enough to minimize the exposure window.
kubeadm token list
sudo kubeadm token create --ttl 2h
sudo kubeadm token delete <token>
The one to memorize — prints a complete, ready-to-paste join command:
sudo kubeadm token create --print-join-command
The upgrade sequence is ordered by dependency: the apiserver must be upgraded before any node-level component can talk to the new API. So control-plane first, then workers. The kubeadm binary must be upgraded before running it on each node because kubeadm reads its own version to know what to install and what manifests to generate — it can’t upgrade to a version it doesn’t know.
sudo kubeadm upgrade plan # read-only; shows current vs available
sudo kubeadm upgrade apply v1.35.1 # first control-plane node only — upgrades static pod manifests
sudo kubeadm upgrade node # every other node (additional control-plane nodes + workers)
upgrade apply rewrites the static pod manifests in /etc/kubernetes/manifests/. The kubelet notices and restarts the control-plane pods. upgrade node does the same for secondary control-plane nodes, plus syncs the kubelet-config ConfigMap down to /var/lib/kubelet/config.yaml (which upgrade apply doesn’t do on the first node, because it knows it’s writing that ConfigMap as part of the upgrade and will sync it on the next upgrade node call on workers).
Full upgrade sequence (memorize this rhythm):
# === First control-plane node ===
# Upgrade the kubeadm binary first
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.35.1-*
sudo apt-mark hold kubeadm
sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.35.1
# Drain, upgrade kubelet, uncordon
kubectl drain <cp-node> --ignore-daemonsets
sudo apt-mark unhold kubelet kubectl
sudo apt-get update && sudo apt-get install -y kubelet=1.35.1-* kubectl=1.35.1-*
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
kubectl uncordon <cp-node>
# === Each worker node (SSH into each) ===
sudo apt-mark unhold kubeadm
sudo apt-get update && sudo apt-get install -y kubeadm=1.35.1-*
sudo apt-mark hold kubeadm
sudo kubeadm upgrade node # syncs kubelet config, upgrades local control-plane components if present
# Back on the control-plane node (or wherever kubectl is configured):
kubectl drain <worker> --ignore-daemonsets
# Back on the worker:
sudo apt-mark unhold kubelet kubectl
sudo apt-get update && sudo apt-get install -y kubelet=1.35.1-* kubectl=1.35.1-*
sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet
# Back on control-plane:
kubectl uncordon <worker>
kubeadm is the CA for the cluster — it generated /etc/kubernetes/pki/ca.key and signed all component certs from it. certs renew re-signs certs against the same CA (which has a 10-year lifetime; component certs default to 1 year). Auto-renewal happens on upgrade apply.
sudo kubeadm certs check-expiration # shows kubeadm-managed PKI only
sudo kubeadm certs renew all
sudo kubeadm certs renew apiserver # renew a specific cert
After renewing, you must restart the static-pod control plane. The running apiserver/etcd/etc. processes opened their cert files at startup and hold those file descriptors — the renewed cert files on disk don’t take effect until the processes restart. The cleanest way is to move manifests out and back:
cd /etc/kubernetes/manifests
sudo mkdir -p /tmp/m && sudo mv *.yaml /tmp/m/ && sleep 20 && sudo mv /tmp/m/*.yaml .
Note: check-expiration only shows the kubeadm-managed PKI. Kubelet client/serving certs rotate separately via the CSR mechanism and won’t appear here.
Mostly useful for templating and pre-pulling images before an air-gapped init:
kubeadm config print init-defaults # print a full InitConfiguration/ClusterConfiguration template
sudo kubeadm config images list # images kubeadm needs for this version
sudo kubeadm config images pull # pre-pull them
Undoes what init/join did — cleans up manifests, certs, etcd member, kubelet state. Use before re-joining a broken node. Note it doesn’t clean CNI state or iptables rules, so do that manually:
sudo kubeadm reset
sudo rm -rf /etc/cni/net.d $HOME/.kube/config
sudo iptables -F && sudo iptables -t nat -F && sudo iptables -t mangle -F && sudo iptables -X
Not a kubeadm subcommand, but kubeadm owns the etcd PKI so the certs are in a known place. etcdctl needs TLS creds to talk to etcd; these are always the same paths on a kubeadm cluster:
# Backup
sudo ETCDCTL_API=3 etcdctl snapshot save /opt/snapshot.db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# Verify
sudo ETCDCTL_API=3 etcdctl snapshot status /opt/snapshot.db --write-out=table
# Restore to a new data dir
sudo ETCDCTL_API=3 etcdctl snapshot restore /opt/snapshot.db \
--data-dir=/var/lib/etcd-restore
After restore, point the etcd static pod at the new data dir: edit /etc/kubernetes/manifests/etcd.yaml, update --data-dir and the hostPath volume to /var/lib/etcd-restore. The kubelet restarts etcd on the new data.
| Path | What it is |
|---|---|
/etc/kubernetes/manifests/ |
Static pod manifests — kubelet watches this |
/etc/kubernetes/pki/ |
Control-plane certs and keys (kubeadm’s CA lives here) |
/etc/kubernetes/pki/etcd/ |
etcd-specific certs |
/etc/kubernetes/admin.conf |
Cluster-admin kubeconfig |
/var/lib/kubelet/config.yaml |
kubelet runtime config (synced from kubelet-config ConfigMap) |
/var/lib/etcd/ |
etcd data directory (default) |
Upgrade kubeadm binary before running upgrade commands on each node. kubeadm bootstraps itself — it needs to know the target version to generate correct manifests.
upgrade apply is first-control-plane-node-only; every subsequent node uses upgrade node. apply writes the new kubelet-config ConfigMap; upgrade node reads it. Running apply on a second control-plane node targets the wrong entrypoint and will either fail or mismatch state.
Drain before upgrading the kubelet, uncordon after. Restarting kubelet disrupts pods running on that node — drain ensures they’ve already been rescheduled elsewhere.
After certs renew, you must bounce the static pods. Renewing writes new files to disk. The running processes still have old file descriptors. They don’t notice the new files until they restart. A kubelet restart alone doesn’t help — the static pods are child processes of the kubelet that need their own restart.
--control-plane-endpoint must be set at init time. There’s no clean way to add it after the fact because it’s baked into multiple kubeconfigs and the apiserver cert SAN list.
Local edits to /var/lib/kubelet/config.yaml don’t survive an upgrade. The kubelet-config phase overwrites it from the ConfigMap. The ConfigMap is the source of truth.
kubeadm config images list covers control-plane images. What about CNI and CoreDNS — are those included, or is that a separate pre-pull step?A tailored 7-day study plan for the CKA exam (Kubernetes v1.35, target date July 9 2026), plus an exam-day reference covering allowed resources, domain weights, and doc links you can open during the exam. Assumes solid prior knowledge — focused on exam mechanics and muscle memory, not concept introductions.
| Item | Detail |
|---|---|
| K8s version | v1.35 |
| Time | 120 minutes |
| Tasks | ~15–25, each weighted (do high-weight first) |
| Pass | 66% (partial credit) |
| Clusters | ~6 clusters, each with its own nodes; switch with kubectl config use-context |
| Node access | SSH to control-plane/worker nodes, with sudo |
| Cluster tooling | kubeadm, etcd, CoreDNS, a CNI plugin; Helm + Kustomize expected |
| Allowed docs | kubernetes.io/docs, kubernetes.io/blog, helm.sh/docs (one browser tab) |
| Simulator | 2x Killer.sh sessions (CKA-A, CKA-B), 17 questions each, 36h access per session |
Re-verify the K8s version at the LF candidate FAQ a few days before the exam — it realigns 4–8 weeks after a K8s release.
Critical exam habit: every task starts with a context-switch command. Run it. A correct answer in the wrong cluster scores zero.
This plan assumes solid knowledge of Kubernetes internals (kubeadm, etcd/PKI, CNI, networking). It is tuned for exam speed, imperative fluency, and the highest-weight domains — not for learning concepts from scratch.
Open-book, but restricted to these sites through the browser inside the exam VM — no Google, Stack Overflow, GitHub, or personal notes. The docs search bar is fine, but you must not open external search results.
Every per-domain link below resolves to one of these allowed domains. The LF candidate FAQ and the CNCF curriculum are pre-exam reference only — they will not open during the exam.
Five domains. Troubleshooting (30%) and Cluster Architecture (25%) are 55% of the exam — spend your time accordingly.
RBAC, kubeadm cluster create/upgrade, HA control plane, Helm, Kustomize, extension interfaces (CNI/CSI/CRI), CRDs, Operators, etcd backup/restore.
Deployments & rollouts, ConfigMaps/Secrets, HPA, scheduling (affinity, taints/tolerations, nodeSelector).
StorageClasses, dynamic provisioning, PV/PVC, access modes, reclaim policies.
Pod connectivity, NetworkPolicies, Service types & endpoints, Gateway API, Ingress, CoreDNS.
Cluster/node failures, control-plane components, kubelet, networking, app failures, resource monitoring, container logs.
Save one Killer.sh session for the final dry run — activate the first now, each session is only 36h. Set up a local 2-node kubeadm sandbox (1 control-plane + 2 workers on throwaway Ubuntu 24.04 VMs) so you have a cluster you can break and rebuild fast.
Lock in your exam shell setup so it’s automatic on exam day:
alias k=kubectl
export do="--dry-run=client -o yaml"
export now="--force --grace-period=0"
source <(kubectl completion bash)
complete -o default -F __start_kubectl k
# vim: set ts=2 sw=2 et, and 'set paste' awareness for YAML
Highest weight, so it goes first and gets two days.
/var/lib/kubelet/config.yaml and --kubeconfig, fix it. systemctl status kubelet, journalctl -u kubelet./etc/kubernetes/manifests/, watch the API server / scheduler / controller-manager go down, recover.kubectl get nodes NotReady scenarios: CNI down, kubelet down, cert expiry, disk pressure.crictl ps, crictl logs, crictl inspect on a node where kubectl can’t reach a pod.kubectl get events --sort-by=.metadata.creationTimestamp.kubectl run tmp --rm -it --image=busybox -- nslookup svc).etcd snapshot + restore end-to-end:
ETCDCTL_API=3 etcdctl snapshot save /tmp/snap.db \
--cacert=... --cert=... --key=... --endpoints=https://127.0.0.1:2379
ETCDCTL_API=3 etcdctl snapshot restore /tmp/snap.db --data-dir=/var/lib/etcd-restore
# then point the static pod manifest at the new data-dir and restart
kubeadm init from scratch + join a worker. Time yourself.kubeadm upgrade plan/apply, drain, apt the kubelet/kubectl, uncordon). Near-guaranteed exam task — make it boring.kubectl auth can-i --as=system:serviceaccount:ns:sa.auth can-i.repo add, search, install, upgrade, --set, template, uninstall, list releases. Know helm install --dry-run.kubectl apply -k, patches, images: and replicas: transforms.kubectl expose, endpoints inspection.set image, rollout undo, rollout status, scaling, revision history.nodeName, static pods.k ... $do > x.yaml && vim x.yaml && k apply -f x.yaml. No hand-writing YAML from a blank file.Use your second Killer.sh session as a timed 2-hour mock. Treat it as the real thing: context-switch every task, flag-and-skip hard ones, high-weight first. Review only the questions you missed and re-drill those exact mechanics. Re-read the LF candidate FAQ and confirm the K8s version hasn’t shifted. Keep it a light day otherwise — don’t cram new material the night before.
$do/$now, completion, vim config.kubectl config use-context line first. Confirm with k config current-context.$do, edit, apply. Hand-write only what you must (PV, NetworkPolicy, Gateway).k get, k describe, auth can-i, nslookup, curl from a temp pod.k run nginx --image=nginx $do
k create deploy web --image=nginx --replicas=3 $do
k expose deploy web --port=80 --target-port=8080 --type=NodePort $do
k create cm app --from-literal=KEY=val $do
k create secret generic db --from-literal=pw=s3cr3t $do
k create role r --verb=get,list --resource=pods $do
k create rolebinding rb --role=r --serviceaccount=ns:sa $do
k get events --sort-by=.metadata.creationTimestamp
k get pods -A -o wide --field-selector status.phase!=Running
At its core, problem solving is navigating toward a goal — but the goal itself isn’t always clear in advance. Sometimes it’s precisely defined; sometimes you’ll only recognize it when you see it. That second case turns out to be surprisingly common in real work, and it means that goal ambiguity is often the actual difficulty, not the path-finding.
The field has been studied from several distinct traditions that each illuminated a different piece. Behaviourists (Thorndike, Watson, Skinner) explained problem solving as trial-and-error learning shaped by reinforcement — useful for describing how organisms adapt, but silent on the internal structures that make complex reasoning possible. Gestalt psychologists (Wertheimer, Duncker, Koffka) shifted focus to perception and restructuring: insight — the sudden “Aha!” — wasn’t random but arose from a reorganization of how the problem was seen. Cognitive psychologists (Newell, Simon, Anderson) formalized this into the problem space framework: a problem is defined by an initial state, a goal state, and operators that move between them; solving it is search through that space. Educationalists (Polya, Schoenfeld) brought the question down to practice — what heuristics and metacognitive habits actually help people solve problems they’ve never seen before?
Robertson’s book draws on all four traditions. The cognitive framework is central, but the Gestalt insight into representation and the educationalist emphasis on deliberate strategy both run through the treatment.
Problem taxonomy — what kind of problem you have determines which approaches apply:
| Axis | Poles |
|---|---|
| Knowledge requirement | Knowledge-lean (domain-general strategies suffice) ↔ Knowledge-rich (requires domain-specific expertise) |
| Goal specification | Well-defined (all needed info is given or inferable) ↔ Ill-defined (goal or constraints are vague) |
| Solver experience | Semantically lean (little relevant experience) ↔ Semantically rich (deep experience of this problem type) |
| Solution character | Standard (deliberate search through operators) ↔ Insight (solution preceded by sudden restructuring) |
A knowledge-lean, well-defined problem — a logic puzzle, a river-crossing problem — calls for very different approaches than an ill-defined, knowledge-rich one like debugging a production incident or designing a system under unclear requirements. Identifying which kind of problem you’re facing before starting is itself a meta-skill.
Schemas are the memory structures that make this classification fast in practice: organized representations of problem types in long-term memory that let you recognize “this is a problem of type X” and retrieve the relevant approach without having to reason from scratch. Building them deliberately is most of what expertise development looks like.
Newell and Simon’s foundational work frames every problem as a problem space: an initial state, a goal state, a set of operators that move between states, and a search process that navigates the space. This formalism makes representation precise — it isn’t just “how you think about the problem,” it’s the structure that determines which states exist, which operators are legal, and therefore which solutions are reachable at all. A representation that omits a relevant variable or encodes a constraint wrong doesn’t make the problem harder; it makes the correct solution unreachable by definition, no matter how hard or cleverly you search.
The Hobbits and Orcs problem makes this concrete. Three hobbits and three orcs must cross a river using a boat that holds at most two; orcs can never outnumber hobbits on either bank. Greeno (1974) and Thomas (1974) both studied how solvers approach this. Many people represent the state informally — a vague mental image of who is where — and quickly run into what feel like dead ends or illegal moves. The difficulty isn’t the logic; it’s the representation. A precise tuple (hobbits_left, orcs_left, boat_side) makes every legal operator explicit: enumerate the moves that keep each component valid, and the solution path becomes a straightforward search. Solvers who represented the state space this way made far fewer errors and reached the solution faster. The problem didn’t change; the representation did.
The nine-dot problem is a starker example. Connect nine dots arranged in a 3×3 grid using four straight lines without lifting the pen. Most people fail because they self-impose a square boundary around the dots — a constraint the problem statement never stated. The search space they’re exploring doesn’t contain the solution. The boundary isn’t in the problem; it’s in the representation they constructed when they read it.
This last point is what makes representation especially tricky. Bransford, Barclay, and Franks (1972) showed that people don’t encode problem statements neutrally — they construct an interpretation, and that interpretation can include inferences and assumptions the statement never made. You read “nine dots in a grid” and your encoding includes an imaginary boundary. You read “the server is timing out” and your encoding includes an implicit assumption about which server. The construction happens automatically; catching it requires deliberate effort.
A practical version: if you represent a failing Kubernetes pod as “the image is broken,” your search focuses on the registry and build pipeline. Represent it as “the pod can’t reach a ready state” and you open the search to readiness probes, resource limits, scheduling constraints, and network policy. The operators available to you are determined by the representation. Choosing the frame first is the highest-leverage step before doing anything else.
Heuristics are mental shortcuts that make search tractable — without them, the space of possible moves in almost any real problem is too large to explore exhaustively. Robertson focuses on three: hill climbing (always take the step that moves closest to the goal), means-end analysis (identify the gap between current state and goal state, then select an operator that reduces it), and working backward (start from the goal and ask what would need to be true immediately before it).
Each is genuinely useful, and each has a characteristic failure mode. Hill climbing fails when you have to move away from the goal to ultimately reach it — a maze with dead ends is the simple version, but software dependency resolution is a real one where installing the right package requires temporarily removing a conflicting one. Means-end analysis can get locked onto a subgoal that isn’t actually necessary, spending effort closing a gap the solution doesn’t require; a classic example is a student who rewrites a proof step to get closer to a known lemma, not realizing the proof doesn’t need that lemma at all. Working backward is powerful for debugging (trace from the error back through the call stack to the source) but doesn’t generalize well to open-ended design problems where the goal state is fuzzy.
The Gestalt tradition identified two mechanisms that explain why heuristics break down even when the solver has all the information they need.
Functional fixedness (Duncker 1945): solvers encode objects in their conventional role and fail to see them as candidates for other uses. The thumbtack box in the candle problem never enters the search as a shelf because the initial encoding locked it out as a container. This isn’t a reasoning error — it’s a representational one. A real-world version: a developer who can’t see a config file as a substitute for a database entry because “config files don’t work that way” is imposing a constraint that exists only in the encoding, not in the problem.
Set effects / Einstellung (Luchins 1942): solving a class of problems with a particular method creates a persistent bias toward that method even when simpler alternatives exist. Luchins’s water jug problems show this directly: participants who learned a complex filling procedure applied it to problems with a trivial direct solution, often without noticing the shortcut was available. The set operates below awareness, biasing which operators the solver considers at all.
Wertheimer’s distinction between reproductive thinking (applying learned procedures regardless of fit) and productive thinking (reasoning from the actual structure of the current problem) frames both: functional fixedness and set effects are both forms of structural blindness. Knowing this is why strategy rotation works — it isn’t just a general fallback, it’s specifically a way of fighting the implicit exclusions that set and fixedness introduce.
Insight — the sudden shift that makes a previously intractable problem feel obvious — isn’t a random gift. The key theoretical reframe comes from Kaplan and Simon (1990): conventional problem solving is searching in a representation, navigating the problem space you already have. Insight is searching for a representation — finding a better problem space first. This isn’t a categorically different cognitive process; it’s search operating over the space of possible framings rather than the space of moves within a given framing. When you’re stuck, you aren’t following the wrong path — you’re on the wrong map.
Ohlsson’s Representational Change Theory (RCT) explains why impasse happens and how the brain escapes it. The initial representation is assembled from salient features and whatever long-term memory retrieves first. If the solution isn’t reachable from that representation, you hit impasse — forward progress stops regardless of effort. Four escape mechanisms:
These are largely unconscious — they happen through spreading activation and cued retrieval, not deliberate reasoning.
MacGregor et al.’s Criterion for Satisfactory Progress Theory (CSPT) adds the conscious side. Solvers use hill climbing while monitoring progress. When expected progress isn’t achieved — criterion failure — they look for other states to expand. Working memory capacity matters: solvers who can look further ahead before hitting criterion failure are more likely to recognize that a solution requires an initially unpromising move. In the nine-dot problem, people who could anticipate further consequences of each line were more likely to discover that lines must extend outside the imaginary boundary.
RCT and CSPT aren’t competing — they’re complementary. Gilhooly and Murphy (2005) found evidence for both unconscious (Type 1) and conscious (Type 2) processes in insight, with the balance depending on problem type. Perceptual and spatial insight problems draw more on unconscious chunk decomposition; verbal and logical ones involve more conscious monitoring and search. Incubation — stepping away from a stuck problem — works because it allows unconscious re-encoding to run without conscious hill climbing suppressing the process by reinforcing the failed frame.
Norman Maier’s two-string problem demonstrates the unconscious mechanism directly: participants must tie together two strings that are too far apart to reach simultaneously. Most keep trying variations on walking between them. The solution requires reframing one string as a pendulum. Solvers who were casually nudged toward pendulum-like motion by the experimenter solved it faster without realizing why — the nudge triggered re-encoding below awareness. A software equivalent: the developer who’s spent an hour convinced the bug is a logic error in a function, takes a five-minute walk, and immediately wonders whether it’s an encoding issue upstream — and it is. The information didn’t change; the representation it was organized into did.
Expert performance has three characteristic signatures. First, fast categorization: experts classify problems by deep structure, not surface features. Chi, Feltovich, and Glaser (1981) found that novice physics students grouped problems by surface appearance (“this is an inclined plane problem”) while experts grouped them by underlying principle (“this is a conservation of energy problem”) — a fundamentally different representation that immediately implies a solution strategy. Second, perceptual chunking: experts encode larger meaningful configurations as single units, letting them process more information per cognitive step. Third, long-term working memory: Ericsson and Kintsch (1995) documented that experts develop domain-specific encoding strategies that effectively extend working memory into long-term memory — they can retrieve and manipulate information that novices would need to hold in limited-capacity short-term memory.
The chess evidence established all three. De Groot (1965) first showed that grandmasters didn’t examine more moves than masters — they examined better moves, because they recognized meaningful board configurations immediately. Chase and Simon (1973) made the mechanism explicit: expert players reconstructed realistic mid-game positions from five-second exposures; when pieces were placed randomly, the advantage disappeared entirely. The skill was in pattern recognition applied to meaningful chess structure, not memory capacity or processing speed. The same signatures appear in radiology (Lesgold et al. 1988: experts see clinically meaningful regions; novices see individual features), programming (McKeithen et al. 1981: expert programmers chunk code into functional units), and physics problem solving.
How expertise develops. Three models from the chapter each illuminate a different dimension. Fitts and Posner (1967) describe three stages: a cognitive stage (resource-intensive, reliant on declarative knowledge, slow and error-prone), an associative stage (rules are compiled, performance becomes more automatic), and an autonomous stage (highly proficient, no longer reliant on conscious control). Dreyfus (1997) extends this to five stages — novice, advanced beginner, competent, proficient, expert — and makes a point often missed: rule-based systems cannot account for expert intuition. Experts don’t follow rules; they perceive situations holistically and respond based on pattern recognition accumulated over thousands of exposures. This is why experts are often poor at explaining what they do. Glaser (1996) focuses on the changing locus of control: from external support (dependent on teachers and coaches), through transitional (support gradually withdrawn), to self-regulatory (learning fully under the learner’s own direction).
The Power Law of Practice describes the shape of skill acquisition: rapid improvement early, dramatically diminishing returns as practice accumulates (Newell & Rosenbloom 1981). This is consistent across domains — it’s a structural feature of skill acquisition, not a motivation problem. It explains why early study of a domain has very high return and why pushing past intermediate competence requires disproportionate effort.
Deliberate practice (Ericsson, Krampe & Tesch-Römer 1993) is what determines how fast you move through those stages. Not all practice is equal. Deliberate practice is effortful, aimed at the edges of current ability, with immediate and accurate feedback, focused on specific components that need improvement. Comfortable practice within well-established skill produces much less improvement than shorter periods of targeted, difficult work at the boundary of competence. Ericsson and Charness (1994) extend this: expertise involves continuous effortful adaptation to domain demands, not consolidation of learned routines.
The intermediate effect. Boshuizen and Schmidt (1990, 1992) documented a striking pattern in medical expertise: people with intermediate knowledge can outperform both novices and experts on some tasks, and be outperformed by novices on others. Intermediates encode too much detail — they “process too much garbage” — and so recall more clinical propositions than experts, but make worse diagnostic decisions because the detail obscures the pattern. Experts have compiled their knowledge into higher-level scripts that suppress irrelevant detail. The implication: accumulating more knowledge at the same representational level isn’t always progress; what matters is whether knowledge is compiled into higher-level schemas.
Domain specificity. Expertise doesn’t transfer across domains. Content knowledge is domain-specific (deep physics knowledge doesn’t help with medical diagnosis). Task knowledge — procedural and strategic knowledge — transfers only to closely related domains that share a genuine skill overlap. There is no such thing as a general expert reasoner in the meaningful sense; there are domain experts and the domains may or may not share transferable skills.
The automaticity paradox. Automatisation ought to make expert performance brittle — if procedures run without conscious access, how do experts handle novel problems? Hatano and Inagaki (1986) resolve this with the distinction between routine expertise (fast, efficient, handles typical problems with compiled procedures) and adaptive expertise (flexible, handles novel and exceptional cases through schemas that explicitly represent exceptions and edge cases). Expert schemas aren’t just libraries of common cases — they include strategies and knowledge structures for when the common case doesn’t apply. This is what distinguishes a senior engineer who can debug an unfamiliar system from one who can only optimize systems they built.
Downsides of expertise. Two documented failure modes. Ottati et al. (2015) demonstrated the earned dogmatism effect: people who perceive themselves as expert become more closed-minded — expertise feels like a license to dismiss alternatives without engaging with them. Fisher and Keil (2015) documented the illusion of explanatory depth: experts may have forgotten how much they’ve forgotten; they report more explanatory competence than they actually possess when pressed to generate detailed mechanistic explanations. Both are calibration failures rather than competence failures — the expertise is real, but the metacognitive monitoring of its limits is broken.
A common mental model of memory is a recording — you store an experience, and later you play it back. The cognitive science is very different: memory is reconstructive. When you remember something, the brain assembles the recollection from fragments, filling gaps with inference, prior expectations, and current context. This makes memory fast and flexible but also systematically unreliable in specific ways.
Elizabeth Loftus’s research on eyewitness testimony is the most studied example. Her experiments showed that the phrasing of a post-event question — “How fast were the cars going when they smashed into each other?” vs. “contacted” — significantly changed what participants later remembered, including whether they recalled seeing broken glass that wasn’t there. The memory wasn’t retrieved; it was reconstructed around the implicit suggestion. A more technical analogue: write a complex module, set it aside for six months, then need to modify it. You’ll feel confident about how it works — and be wrong about specific details in ways that cause bugs. The confidence is real; it comes from the reconstruction feeling fluent, not from the reconstruction being accurate.
The consequence for schema building: a schema that gets reinforced through repeated reconstruction of a slightly wrong version becomes a well-practiced error. Schema quality — whether the pattern you’ve chunked is actually correct — matters as much as schema quantity.
Retrieval is surface-driven, not structure-driven. When you encounter a problem and search memory for a relevant analogue, retrieval is largely governed by surface similarity — whether the objects, vocabulary, and perceptual features of the new situation match something stored. Structural similarity, which is what actually determines whether the analogue will help, doesn’t drive retrieval as reliably. Gentner, Rattermann, and Forbus (1993) demonstrated this directly: people were more likely to spontaneously retrieve surface-similar analogues even when the structurally similar analogue was the one that would actually solve the problem. The two properties — retrievability and inferential soundness — come apart in ways that matter enormously.
This is the mechanism behind the Gick & Holyoak fortress/tumor finding. Participants had the relevant structural analogue in memory — they’d just read the fortress convergence story — but didn’t spontaneously apply it to the radiation problem because the surface domains are completely different. The analogue was there; the retrieval process didn’t surface it. Only when participants were explicitly prompted to use the story as an analogy did transfer occur at meaningful rates.
Chen, Mo, and Honomichl (2004) showed that even analogues learned long ago can be retrieved effectively when structural features were encoded strongly at learning time. The practical implication: encoding structural labels matters. Not just “I solved a rate problem” but “I solved a convergence problem” — a label that primes retrieval when the next convergence problem appears regardless of what domain it’s in. Encoding strategies like elaboration and dual coding work partly because they build richer reconstructive cues, not because they improve storage of raw content.
The hope that becoming good at hard problems in one domain makes you better at hard problems in general is mostly false. Transfer of skill happens when two problems share underlying abstract structure — not surface features, not domain vocabulary, not difficulty level.
Transfer types. Transfer can be positive (prior experience helps), negative (prior experience interferes — what the Einstellung effect produces when a learned procedure is applied where it no longer fits), near (similar domain, slightly different problem), or far (different domain, same underlying structure). Far positive transfer is what people usually mean when they talk about generalizable problem-solving skill, and it’s also the hardest to reliably produce.
Problems with identical underlying structure but different surface content are called isomorphs. Simon and Hayes (1976) demonstrated that isomorphic problems can vary dramatically in difficulty: two river-crossing problems with identical state-space structure differ in solution rate because the surface framing changes which operators feel natural to try. Homomorphs are structurally similar but not identical — the same template with slight variations in constraints. Both transfer when solvers notice the structural match; neither transfers reliably when they don’t.
Three kinds of similarity determine whether transfer occurs. Surface similarity means the objects or perceptual features of the two problems resemble each other. Relational similarity (also called structural similarity) means the relations between objects are the same — the underlying structure maps. Procedural similarity means the same solution method applies even if surface and structure differ. Surface similarity drives what gets retrieved from memory; relational similarity determines whether the analogue is actually useful. These two come apart: you can retrieve a surface-similar analogue that gives you nothing, or fail to retrieve a structurally identical one because it looks nothing like the current problem.
Successful analogical transfer requires three things in sequence: retrieval of a relevant analogue from long-term memory, mapping of corresponding roles across source and target (which objects in the source play the same role as which objects in the target), and adaptation of the source solution to the target context. Failure at any step blocks transfer. Most naturally occurring transfer failure is a retrieval failure — the right analogue is in memory but wasn’t retrieved because it doesn’t look like the current problem on the surface.
Gick and Holyoak’s (1983) radiation/fortress study is the canonical demonstration. A doctor must destroy a tumor using radiation, but a strong-enough ray damages surrounding tissue; the solution is multiple weak rays converging from different directions. Most participants fail. Participants who had previously read a story about a general who converged small army groups along separate roads to capture a fortress solved it at much higher rates — the underlying structure is identical. But participants who read the story without being prompted to use it as an analogy transferred poorly anyway. The structural mapping existed; retrieval didn’t surface it without explicit prompting.
Gentner’s Structure Mapping Theory (SMT) formalizes what makes an analogy work. The central claim: analogy is a mapping of relational structure from a source to a target, independent of the objects involved. What gets mapped isn’t attributes of objects (“both involve radiation” / “both involve armies”) but the relations between objects — the structural skeleton. The principle of systematicity says that mappings which preserve higher-order relations (relations between relations) are preferred over mappings that only match first-order features. “The atom is like a solar system” works because the relational structure maps: electrons orbit the nucleus as planets orbit the sun, and the force-governs-orbit relationship holds in both — even though the objects share no surface features. A mapping based only on “both are round” would score low on systematicity and license few useful inferences.
Gentner and Gentner (1983) showed that the choice of source analogy materially affects what inferences people draw. Participants given a “flowing water” analogy for electricity reasoned well about resistance; those given a “teeming crowds” analogy reasoned better about batteries. The metaphor wasn’t decorative — it determined which structural inferences were made.
Metaphors are compressed analogies. Novel metaphors require active structural comparison; as they become conventional, they become category assertions and lose their live analogical force. Bowdle and Gentner (2005) call this the “career of metaphor”: “electricity flows” was once a live structural analogy, now it’s nearly literal. The freshest analogies are often most useful, before the structural mapping has been compressed into a label that no longer prompts active reasoning.
In practice: the habit worth building is asking “what underlying pattern does this resemble?” — not “have I seen something like this before?” A DNS caching bug and a memory interference problem look nothing alike on the surface but share the same structure of stale state contaminating fresh queries. Recognizing that is transfer. Recognizing that “both involve the network” is not.
Metacognition is cognition about cognition — monitoring and regulating your own thinking processes as you work. It’s distinct from domain knowledge and distinct from raw ability. Two people with equal skill can have very different outcomes if one actively tracks what strategy they’re using, whether it’s working, and when to switch, while the other just pushes forward.
Alan Schoenfeld’s research on mathematical problem solving makes this concrete. He compared novice and expert mathematicians working through unfamiliar problems in think-aloud protocols. Novices almost never stopped to evaluate whether what they were doing was working — they’d invest ten or fifteen minutes in a fruitless approach without questioning the approach itself. Expert mathematicians regularly paused: “Is this going anywhere? What else could I try? Have I seen this structure before?” This meta-monitoring wasn’t slower; it was faster overall, because it cut off long detours early. The software equivalent is the developer who rubber-ducks early (forcing articulation of the problem often reveals the bug immediately) versus the one who keeps reading the same function expecting a different insight.
Glaser’s (1996) change-of-agency model frames this as a developmental arc. In the early stage of any domain, monitoring is externalized: teachers, coaches, and rubrics provide the feedback loop that the learner can’t yet run internally. In the transitional stage, the learner begins developing self-monitoring — identifying criteria for high performance and tracking their own progress against them. The final, self-regulatory stage is the metacognitive ideal: the learner designs their own practice environment, identifies their own gaps, and decides when external input is worth seeking. The stages are not just a progression of skill — they’re a progression of metacognitive internalization. Chi, Glaser, and Rees (1983) found a concrete instance of this in physics: experts were better than novices not only at solving problems but at assessing a problem’s difficulty in advance and knowing which schemas applied. Knowing what you don’t yet know — and being right about it — is a metacognitive skill as much as a domain knowledge skill.
Metacognition is also the mechanism that makes everything else in this post actually work. You can only rotate strategies deliberately if you’re monitoring which strategy you’re using. You can only trigger diffuse mode at the right moment if you’re tracking how long you’ve been stuck. You can only build schemas from experience if you reflect on what happened rather than just moving on. Externalizing the monitoring loop — logging representations, strategies, and outcomes — gives you data about your own thinking patterns that you can’t access from inside a stuck problem.
These are reference items — templates and habits that apply the mechanisms above.
For each constraint, explicitly label it: real (externally imposed), assumed (inferred but unverified), negotiable, or unnecessary. Most breakthroughs come from discovering a constraint was assumed, not real.
If progress has stopped, the strategy is probably the problem. Before rotating, ask: is the approach I’m reaching for there because the surface of this problem matches a prior one — not because the structure does? Surface similarity drives retrieval, and the Einstellung effect means the most available strategy may be the wrong one. Then rotate:
A schema entry needs: the structural label (the relational pattern, not the surface domain), conditions of applicability, failure modes, and a worked example. The structural label is what makes the schema retrievable across domains — not “Kubernetes networking problem” but “multi-hop dependency with invisible intermediary.”
Intermediate effect warning: accumulating more detail at the same representational level isn’t progress. The goal is compilation into higher-level patterns that suppress irrelevant detail. If you’re recalling more propositions but solving problems no faster, you’re at the intermediate plateau, not advancing toward expert.
Deliberate practice means drilling at the boundary of current competence — problems that feel effortful, not fluent. Comfortable repetition within established skill produces little schema improvement.
Domain examples:
The central question is “What relational structure does this problem have?” — before asking what else shares it. Not “have I seen something like this before?” (surface-driven) but “what structural pattern is this?” (structure-driven).
Three-step transfer checklist:
If step 1 fails, it’s a retrieval failure, not a knowledge failure — the analogue may be there but stored under the wrong label. Build a personal analogy library: for each solved problem, record the structural label. These become retrieval hooks for future problems across all domains.
Structural mappings worth building:
For each non-trivial problem, log:
Over time this becomes a personal structural pattern library — more useful than a log of domain-specific events.
Abstract practice rarely transfers. Attach new concepts to a real system, and when you do, explicitly name the structural mapping — not just “I applied X to Y” but “X and Y share structure Z, so the solution transferred.”
The anchor is transferable only if the relational structure is explicit — otherwise it stays a domain-specific fact.
This is the practice of building adaptive expertise — the schemas that cover exceptions, not just typical cases. Routine expertise handles familiar problems efficiently; adaptive expertise handles novel ones. The gap is trained by working on problems that force re-encoding, constraint relaxation, and chunk decomposition (Ohlsson’s RCT escape mechanisms).
Insight-heavy problems serve this function: math puzzles requiring non-obvious transformations, lateral thinking puzzles, spatial problems, short creative coding exercises (fractals, simulations, animations). Not a break from analytical work — targeted training for the parts of problem-solving that deliberate domain practice doesn’t reach.