A pod resolves service names inside the cluster without a hiccup, pings 8.8.8.8 fine, and then fails on ping google.com with three words: bad address. Your first instinct is CNI, NetworkPolicy, kube-dns health. The actual culprit in the classic case is smaller and stranger: musl libc, and one line in /etc/resolv.conf — options ndots:5.
I first hit this running Alpine 3.5/3.6/3.7 images, and it still shows up in 2026 whenever a musl-based image meets Kubernetes’ default DNS configuration. Here’s the full trail, the fix that works today, and why the popular 2018 workaround is the wrong tool.
The symptom
The failing pod could reach the API server, resolve any internal name, and ping raw IPs. Only external hostnames failed:
# kubectl run alpine-test -i -t --image=alpine:3.6 -- sh
/ # ping sudops.com
ping: bad address 'sudops.com'
/ # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=55 time=8.2 ms
The diagnostic grid ruled out everything above the pod in minutes:
| Test | Result | Rules out |
|---|---|---|
| Other images in same namespace resolve external names | OK | CNI, NetworkPolicy, node network |
| Same Alpine image in a different namespace | Fails | Namespace-specific config |
ping by IP from the failing pod |
OK | L3 connectivity, NAT |
Cluster-internal names (kubernetes.default) |
OK | CoreDNS reachability |
| External names on Alpine 3.5 / 3.6 / 3.7 | All fail | Alpine-version-specific behavior |
Same cluster, same node, same DNS server — Debian images curl fine, Alpine images don’t. That pattern points at one place: the resolver inside the container.
What kubelet actually hands the pod
Every pod gets a /etc/resolv.conf written by the kubelet at start:
nameserver 10.254.0.2
search devns.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
Read the ndots:5 line carefully, because it changes every lookup the container makes. The rule: a name containing fewer than 5 dots is first tried against each search domain, then as an absolute name. sudops.com has one dot. So the resolver’s first attempts are:
sudops.com.devns.svc.cluster.local → NXDOMAIN
sudops.com.svc.cluster.local → NXDOMAIN
sudops.com.cluster.local → NXDOMAIN
sudops.com. → the one that actually works
This looks wasteful on glibc (four queries for one name), but it works there: glibc walks the search list, collects the NXDOMAINs, and falls through to the absolute query. Alpine doesn’t use glibc. It uses musl, and the musl versions shipped in Alpine 3.5–3.7 mishandled exactly this scenario. Between the search-path expansion and the eventual absolute query, the lookup stalled or failed outright.
Two aggravating factors worth knowing, both still real today:
- Query name length. Appending search domains makes names longer. The DNS limit is 253 characters; long namespace names and deep search lists can blow past it on the intermediate queries even when the real name is short.
- Query volume.
ndots:5multiplies CoreDNS traffic. In clusters with thousands of pods, search-domain churn is a classic source of CoreDNS load that looks like “DNS is slow” but is really “DNS is asked four times too often.”
The container never sees any of this. From inside, it’s indistinguishable from the network being broken — hence the hours lost to tcpdump on the wrong problem.
The 2018 fix, and its sharp edges
The workaround that circulated back then was to comment the line out before the app starts:
sed -i 's/options ndots:5/#options ndots:5/g' /etc/resolv.conf
Which fails immediately, and instructively:
sed: can't move '/etc/resolv.confMdACPl' to '/etc/resolv.conf': Resource busy
/etc/resolv.conf in a container is a bind-mounted file, not a regular file. sed -i works by writing a temp file and renaming it over the target — and you cannot rename over a mount point. The rewrite-in-place variant gets around that:
echo "$(sed 's/options ndots:5/#options ndots:5/g' /etc/resolv.conf)" > /etc/resolv.conf
With the option gone, musl falls back to the default (ndots:1), tries sudops.com as an absolute name first, and suddenly everything works. The approach loosens up enough to confirm the diagnosis. Then throw it away, for three reasons:
- It’s per-container bootstrapping. Every image needs the hack in its entrypoint, and it only runs when the container starts, which makes failures order-dependent.
- It’s fragile. Anything that overwrites resolv.conf (a container restart recreates it from the pod spec) silently undoes it.
- It was already obsolete when it was written. Kubernetes shipped the proper fix,
dnsConfig, in 1.9, back in December 2017.
The fix for 2026: put it in the pod spec
dnsConfig lets the workload itself control resolv.conf options. No image rebuild, no entrypoint scripts, survives restarts by definition:
apiVersion: v1
kind: Pod
metadata:
name: alpine-app
spec:
dnsConfig:
options:
- name: ndots
value: "1"
containers:
- name: app
image: alpine:3.20
command: ["sh", "-c", "apk add --no-cache bind-tools && sleep 3600"]
With ndots:1, any name containing a dot is tried absolute-first. External lookups become single queries, internal names still resolve through the search list, and CoreDNS load drops proportionally. If you’re worried about short in-cluster names competing with dotted external names, ndots:2 is a reasonable middle ground; most clusters do fine with 1.
Fine-tuning searches is optional — but if your workloads only ever talk to their own namespace and absolute names, narrowing the list shrinks the blast radius of a broken search entry:
dnsConfig:
searches:
- my-app.svc.cluster.local
- svc.cluster.local
options:
- name: ndots
value: "1"
The alternatives, and when each one is right:
| Approach | When it fits | Cost |
|---|---|---|
dnsConfig with ndots:1 |
Default choice — any pod spec you control | None; one YAML block |
FQDN with trailing dot (api.example.com.) |
App config, client URLs | Escaping behavior varies by app/runtime |
| Upgrade Alpine (3.13+, musl 1.2.x) | You control the Dockerfile | musl fixed many resolver edge cases; ndots tuning still worthwhile |
| Switch to a glibc base (debian-slim, ubuntu) | Image size is not a hard constraint | Bigger image, revisited CVE surface |
| Edit resolv.conf at entrypoint (2018 style) | Never, on anything you can fix above | Fragile, per-image, reinvents dnsConfig |
If nobody on the team can edit the manifest (a vendor chart, a legacy Deployment under another team), a MutatingAdmissionPolicy (1.32+, replacing the mutation half of admission webhooks) can inject dnsConfig cluster-wide. Heavier, but it centralizes the fix instead of scattering entrypoint hacks.
The debugging toolkit that took me straight to the answer
# What does the resolver see?
kubectl exec -it alpine-app -- cat /etc/resolv.conf
# Query the internal DNS directly (FQDN so musl skips the search list)
kubectl exec -it alpine-app -- nslookup kubernetes.default.svc.cluster.local 10.96.0.10
# Watch actual queries when the pod does a lookup (run in a second terminal)
kubectl exec -it alpine-app -- sh -c 'apk add --no-cache tcpdump; tcpdump -n -i any port 53'
# CoreDNS is answering — what is it asked, and replying?
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
Better: kubectl debug -it alpine-app --image=nicolaka/netshoot gives a troubleshooting container with dig, drill, tcpdump and friends already installed — no apk add round trip, and no assumptions about what the app image contains.
Tip: When debugging any DNS complaint inside a pod, reproduce with
nslookupand check a lookup that works (an internal name) in the same shell. The contrast — “internal resolves, external doesn’t” — is what turns a vague “DNS is broken” report into a searchable problem in about 60 seconds.
Checklist
- Internal names resolve, external names fail → check
/etc/resolv.confand the libc, in that order options ndots:5present → setdnsConfig.ndots:1on the pod spec- Never
sed -ia bind-mounted resolv.conf; don’t ship entrypoint hacks whendnsConfigexists - Old Alpine (the 3.5–3.7 era) images → upgrade when possible; tuning ndots is still the first move
- Suspect a deep search list → count the queries with tcpdump before and after the fix
- Cluster-wide fix needed → MutatingAdmissionPolicy, not per-image scripts
If you want to watch this failure and its fix side by side, a single-node test cluster is enough. The smallest VPS on any mainstream provider ($5–6/month territory: DigitalOcean, Vultr, Hetzner) runs k3s in ten minutes, and reproducing a broken resolv.conf takes thirty seconds.
The takeaway
Kubernetes gives every pod an /etc/resolv.conf tuned for cluster-internal discovery. glibc absorbs the resulting extra queries; a decade-old musl did not, and that gap produced one of the most persistently misdiagnosed failures in Kubernetes troubleshooting. The lesson generalizes past DNS: when a container misbehaves and everything outside it looks healthy, the difference is usually inside the image — and the right fix lives in the pod spec, not in the app’s startup script.
Related reading on this site: Kubernetes Pods That Won’t Delete — same principle, different layer: the symptom misleads, the ownership chain explains. 中文原文：kubernetes 使用基于 alpine 镜像无法正常解析外网DNS。
