Kubernetes networking connects pods, services, and the outside world. Understanding how traffic flows through a cluster is essential for debugging issues and designing secure applications.
Networking Fundamentals
The Kubernetes Network Model
Kubernetes implements a flat network where every pod gets its own IP address. This design eliminates the need for port mapping and simplifies application communication.
The following diagram shows the fundamental networking model. Every pod can communicate directly with every other pod using its IP address, regardless of which node hosts them.
┌─────────────────────────────────────────────┐
│ Cluster │
│ │
│ ┌─────────────┐ ┌─────────────┐ │
│ │ Pod A │ │ Pod B │ │
│ │ 10.1.1.10 │ ←───→ │ 10.1.2.20 │ │
│ └─────────────┘ └─────────────┘ │
│ │
└─────────────────────────────────────────────┘
Key principles:
- Pods can communicate with any other pod without NAT
- Nodes can communicate with any pod without NAT
- A pod sees itself with the same IP others see it
Pod-to-Pod Communication
Pods can reach each other directly by IP address. However, relying on pod IPs directly creates problems because pods are ephemeral and their IPs change when they restart.
This configuration shows direct pod-to-pod communication, which works but is fragile. You should avoid hardcoding pod IPs in your application configuration.
# Pod A can reach Pod B directly by IP
apiVersion: v1
kind: Pod
metadata:
name: pod-a
spec:
containers:
- name: app
command: ["curl", "http://10.1.2.20:8080"]
But pod IPs are ephemeral;they change when pods restart.
Services
ClusterIP (Default)
Services provide stable endpoints for pod communication. A ClusterIP Service gives you a fixed IP address and DNS name that routes to healthy pods matching the selector. This abstraction handles pod lifecycle transparently.
The following Service definition creates a stable endpoint that load balances across all pods with the label app: users. Clients connect to port 80, and the service forwards to port 8080 on the pods.
apiVersion: v1
kind: Service
metadata:
name: user-service
spec:
type: ClusterIP
selector:
app: users
ports:
- port: 80
targetPort: 8080
Here is how traffic flows through a ClusterIP service to the underlying pods. The service IP remains constant even as pods come and go.
┌─────────────────┐
Client Pod ───→ │ user-service │
│ 10.96.0.100:80 │
└────────┬────────┘
│
┌───────────────┼───────────────┐
↓ ↓ ↓
┌──────────┐ ┌──────────┐ ┌──────────┐
│ Pod 1 │ │ Pod 2 │ │ Pod 3 │
│ :8080 │ │ :8080 │ │ :8080 │
└──────────┘ └──────────┘ └──────────┘
Services provide:
- Stable DNS name (
user-service.namespace.svc.cluster.local) - Stable IP address
- Load balancing across pods
NodePort
When you need to expose a service outside the cluster without a cloud load balancer, NodePort opens a port on every node. External traffic hitting any node on that port reaches your service.
This Service configuration exposes your application on port 30080 of every cluster node. External clients can connect to any node's IP on this port to reach your service.
apiVersion: v1
kind: Service
metadata:
name: web-service
spec:
type: NodePort
selector:
app: web
ports:
- port: 80
targetPort: 8080
nodePort: 30080 # 30000-32767
Traffic can enter through any node and reaches the same service backend. This diagram shows how external traffic flows through the NodePort to your pods.
External ───→ Node1:30080 ───┐
───→ Node2:30080 ───┼───→ Service ───→ Pods
───→ Node3:30080 ───┘
LoadBalancer
In cloud environments, LoadBalancer services provision an external load balancer automatically. This is the easiest way to expose services publicly, though it creates one load balancer per service which can become expensive.
The following Service tells your cloud provider to provision an external load balancer. The provider assigns a public IP that routes traffic into your cluster.
apiVersion: v1
kind: Service
metadata:
name: api-service
spec:
type: LoadBalancer
selector:
app: api
ports:
- port: 443
targetPort: 8443
The cloud provider assigns an external IP that routes through the cluster. This shows the complete traffic path from the internet to your pods.
Internet ───→ Cloud LB ───→ NodePort ───→ Service ───→ Pods
(1.2.3.4)
Headless Services
For stateful applications that need direct pod access, headless services return all pod IPs instead of load balancing. This is essential for databases and other stateful workloads where clients need to connect to specific instances.
Setting clusterIP: None creates a headless service. Instead of providing a single virtual IP, DNS returns the IP addresses of all matching pods.
apiVersion: v1
kind: Service
metadata:
name: database
spec:
clusterIP: None # Headless
selector:
app: postgres
ports:
- port: 5432
DNS returns all pod IPs instead of a single virtual IP. This allows clients to implement their own connection logic or connect to specific instances.
$ nslookup database.default.svc.cluster.local
10.1.1.10
10.1.1.11
10.1.1.12
Ingress
HTTP/HTTPS Routing
Ingress resources provide HTTP-level routing for multiple services behind a single external IP. This is more cost-effective than creating a LoadBalancer for each service and enables path-based and host-based routing.
This Ingress configuration routes traffic based on URL path. Requests to /users go to the user service, while /orders routes to the order service. Both share the same external IP and TLS certificate.
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: app-ingress
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
spec:
ingressClassName: nginx
tls:
- hosts:
- api.example.com
secretName: tls-secret
rules:
- host: api.example.com
http:
paths:
- path: /users
pathType: Prefix
backend:
service:
name: user-service
port:
number: 80
- path: /orders
pathType: Prefix
backend:
service:
name: order-service
port:
number: 80
The Ingress Controller inspects incoming requests and routes them based on the path. This diagram shows how a single entry point fans out to multiple backend services.
┌─────────────────────┐
Internet ───HTTPS──→│ Ingress Controller │
│ (nginx/traefik) │
└──────────┬──────────┘
│
┌────────────────────┼────────────────────┐
│ /users │ /orders │
↓ ↓
┌───────────┐ ┌───────────┐
│user-service│ │order-service│
└───────────┘ └───────────┘
Ingress Controllers
The Ingress resource is just configuration. You need an Ingress Controller to actually handle the routing. Nginx is the most common choice, but Traefik, HAProxy, and cloud-native options are also available.
The following command installs the Nginx Ingress Controller. Once running, it watches for Ingress resources and configures routing accordingly.
# Install nginx ingress controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.0/deploy/static/provider/cloud/deploy.yaml
DNS
CoreDNS
Kubernetes includes CoreDNS to provide service discovery through DNS. Every service gets a DNS entry, and pods are configured to use the cluster DNS by default.
Service DNS:
<service>.<namespace>.svc.cluster.local
user-service.production.svc.cluster.local
Pod DNS:
<pod-ip-dashed>.<namespace>.pod.cluster.local
10-1-1-10.production.pod.cluster.local
DNS Resolution in Pods
You can reference services by short names within the same namespace or use fully qualified names for cross-namespace communication. This makes service discovery simple and consistent.
This pod configuration demonstrates DNS-based service discovery. Services in the same namespace can be referenced by name alone, while cross-namespace communication requires the namespace suffix.
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
env:
- name: USER_SERVICE_URL
value: "http://user-service" # Same namespace
- name: ORDER_SERVICE_URL
value: "http://order-service.orders" # Different namespace
Custom DNS Config
For special cases, you can override the default DNS configuration. This is useful when you need to resolve external domains or use custom DNS servers.
This configuration overrides default DNS behavior. You might need this when integrating with external systems that require specific DNS resolution.
apiVersion: v1
kind: Pod
spec:
dnsPolicy: "None"
dnsConfig:
nameservers:
- 8.8.8.8
searches:
- default.svc.cluster.local
- svc.cluster.local
options:
- name: ndots
value: "2"
The ndots option controls when Kubernetes appends search domains. Setting it to 2 means names with fewer than 2 dots get search domains appended.
Network Policies
Default Deny All
By default, Kubernetes allows all pod-to-pod communication. Network Policies let you restrict traffic. Start with a default deny policy, then explicitly allow the traffic you need.
This policy blocks all traffic to and from pods in the namespace. It serves as a foundation for a zero-trust network model where you explicitly allow only necessary communication.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-all
namespace: production
spec:
podSelector: {} # All pods
policyTypes:
- Ingress
- Egress
Allow Specific Traffic
After establishing default deny, create policies that allow specific communication patterns. This example allows the API pods to receive traffic from the ingress controller and communicate with the database.
This policy demonstrates a typical microservice pattern. The API can receive traffic from the ingress controller and frontend pods, and can communicate with the database and DNS. All other traffic is blocked.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-policy
namespace: production
spec:
podSelector:
matchLabels:
app: api
policyTypes:
- Ingress
- Egress
ingress:
# Allow from ingress controller
- from:
- namespaceSelector:
matchLabels:
name: ingress-nginx
ports:
- port: 8080
# Allow from other apps in same namespace
- from:
- podSelector:
matchLabels:
role: frontend
ports:
- port: 8080
egress:
# Allow to database
- to:
- podSelector:
matchLabels:
app: postgres
ports:
- port: 5432
# Allow DNS
- to:
- namespaceSelector: {}
podSelector:
matchLabels:
k8s-app: kube-dns
ports:
- port: 53
protocol: UDP
Note that allowing DNS egress is essential. Without it, pods cannot resolve service names and most applications will fail.
Namespace Isolation
For simpler security requirements, you can isolate namespaces from each other. This policy allows traffic only between pods in the same namespace.
# Only allow traffic within namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: namespace-isolation
spec:
podSelector: {}
policyTypes:
- Ingress
ingress:
- from:
- podSelector: {} # Same namespace only
CNI Plugins
Popular Options
The Container Network Interface (CNI) plugin you choose determines what networking features are available. Different plugins offer different tradeoffs between simplicity, performance, and advanced features.
| Plugin | Features |
|---|---|
| Calico | Network policies, BGP, high performance |
| Cilium | eBPF-based, advanced policies, observability |
| Flannel | Simple, overlay network |
| Weave | Encryption, multicast |
Calico Configuration
Calico is popular for its robust network policy support and performance. It can operate in overlay or non-overlay mode depending on your infrastructure requirements.
The following shows Calico installation and an example of its extended network policy syntax. Calico policies offer more features than standard Kubernetes NetworkPolicies, including support for global policies and IP-based rules.
# Install Calico
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
# Calico network policy (extended features)
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
name: advanced-policy
spec:
selector: app == 'api'
ingress:
- action: Allow
protocol: TCP
source:
selector: role == 'frontend'
destination:
ports:
- 8080
egress:
- action: Allow
protocol: TCP
destination:
nets:
- 10.0.0.0/8
ports:
- 443
Calico policies use a different syntax than standard Kubernetes NetworkPolicies and offer additional features like application layer policies.
Debugging
Common Commands
When networking issues arise, these commands help you understand what is happening. Start by checking if services have endpoints, then verify DNS resolution and connectivity.
This collection of commands covers the most common debugging scenarios. You will use these frequently when troubleshooting connectivity issues in your cluster.
# Check service endpoints
kubectl get endpoints user-service
# Test DNS resolution
kubectl run -it --rm debug --image=busybox -- nslookup user-service
# Test connectivity
kubectl run -it --rm debug --image=curlimages/curl -- curl http://user-service:80
# Check network policies
kubectl get networkpolicies
# Describe service
kubectl describe service user-service
# Check pod IPs
kubectl get pods -o wide
Traffic Flow Issues
When traffic is not reaching your pods, the most common cause is a selector mismatch between the service and pods. Always verify that endpoints exist and pods are ready.
This debugging session demonstrates how to diagnose a service with no endpoints. The key insight is that empty endpoints usually mean the selector does not match any pods.
# Check if service has endpoints
$ kubectl get endpoints user-service
NAME ENDPOINTS AGE
user-service 10.1.1.10:8080,10.1.1.11:8080 5m
# No endpoints? Check selector matches
$ kubectl get pods -l app=users
No resources found # Selector doesn't match!
# Check pod readiness
$ kubectl describe pod user-pod
Conditions:
Ready: False
ContainersReady: False
If endpoints are empty, either no pods match the selector or the matching pods are not ready. Check labels and readiness probes.
Network Policy Debugging
Network policies only work if your CNI plugin supports them. Flannel does not enforce network policies, so if you are using it, policies will be ignored. Also remember that policies are additive;you need explicit allow rules once any policy applies to a pod.
This sequence shows how to determine if a network policy is blocking traffic. Temporarily removing policies helps isolate whether the issue is policy-related or something else.
# Check if CNI supports network policies
# (Flannel doesn't, Calico/Cilium do)
# Test blocked traffic
$ kubectl exec -it frontend-pod -- curl http://backend-service
curl: (7) Failed to connect # Blocked by policy
# Temporarily allow all to test
$ kubectl delete networkpolicy default-deny-all
# Test again
$ kubectl exec -it frontend-pod -- curl http://backend-service
{"status": "ok"} # Works without policy
If deleting the network policy fixes connectivity, your policy rules need adjustment. If it still fails, the issue is elsewhere.
Best Practices
- Use Services for communication - Never hardcode pod IPs
- Implement Network Policies - Default deny, explicit allow
- Use namespaces for isolation - Separate environments/teams
- Choose appropriate service types - ClusterIP for internal, LoadBalancer for external
- Monitor DNS resolution - CoreDNS issues cause cascading failures
- Use Ingress for HTTP - Don't expose NodePorts directly
Conclusion
Kubernetes networking abstracts away infrastructure complexity while providing powerful primitives for service discovery, load balancing, and security. Master Services for stable endpoints, Ingress for HTTP routing, and Network Policies for security. Debug systematically: check endpoints, DNS resolution, and policy rules. The right CNI plugin depends on your security and performance requirements.