Network Observability: Packet Loss, Retransmits, and Latency Spikes

Philip Rehberger Nov 20, 2026 7 min read

Find the network issues that show up as application errors. Covers VPC flow logs and eBPF tools.

Most application incidents that get blamed on "the network" are not really network problems. Most network problems that take down applications never get diagnosed as network problems. The asymmetry is structural: application metrics are well-instrumented, network metrics are not, and the same symptom (slow request, intermittent error) can have either cause.

Fixing this asymmetry is what network observability is for. With the right data, the question "is it the network or the application?" gets answered in seconds rather than being argued about for an hour.

This guide covers the data sources that produce useful network observability and the failure modes they catch.

What Counts as a Network Problem

Four categories of network failure show up in application incidents:

  • Packet loss. Some fraction of packets between two endpoints are dropped. TCP retransmits eventually deliver them; latency for affected requests spikes.
  • Latency spikes. Round-trip times jump, sometimes for specific paths, sometimes globally. The application's request latency tracks.
  • DNS resolution failures or slowness. Names fail to resolve, or take seconds. Requests fail to even start. Common after a DNS provider change or a misconfigured resolver.
  • Connection limits. TCP source ports, connection table entries, NAT mappings — all bounded. When exhausted, new connections fail.

Each is invisible to a service-level latency metric. The application sees "slow request," not "15% of the packets are being dropped to this destination."

VPC Flow Logs

In AWS, GCP, and Azure, the canonical network observability source is flow logs. Each record is a summary of a flow — source, destination, port, protocol, packet count, byte count, allow/deny — written to S3 or equivalent.

Flow logs are not realtime. They are summaries flushed periodically (every minute or so). They do not contain payloads. What they do show is connectivity patterns, anomalous traffic, and rough flow statistics — and most importantly, whether traffic was even reaching its destination.

A few patterns flow logs reveal:

  • Sudden drop in traffic between two services. Routing or security-group misconfiguration after a change.
  • High REJECT counts to a destination. Security group denied traffic that the application expected to succeed.
  • Volume anomalies — a service receiving 10x the usual traffic suddenly.
  • Outbound traffic to unexpected destinations. Useful for security investigation.

Flow logs land in S3, get queried via Athena or piped into Loki / OpenSearch for ad-hoc analysis. The volume is large; storage tiering matters.

eBPF for Realtime Network Observability

Flow logs are good for forensics but slow for live incidents. eBPF-based tools fill the gap. Cilium Hubble, Inspektor Gadget, and standalone tools like tcpconnect and tcpretrans (BCC tools) surface network events from the kernel in realtime.

The events worth surfacing as metrics:

  • TCP retransmits per destination. Above 0.5-1%, indicative of packet loss or congestion.
  • TCP connection establishment failures. Connection-refused, timeouts.
  • DNS resolution latency and failure rate. Per-resolver, per-name.
  • HTTP / gRPC error rate as observed at the kernel level. A different cut from application metrics — lets you confirm whether the application saw the error or it happened in transit.

A Cilium-instrumented cluster exposes most of these via Hubble's gRPC API or Prometheus exporter. The data overlaps with what individual services emit but adds the perspective "this is what the network saw," which is sometimes the only reliable source.

DNS

DNS deserves its own observability surface. It is a frequent cause of incidents, an underused source of debug data, and absent from most metric stacks.

The metrics worth capturing:

  • Resolution latency distribution. p95 and p99 for resolutions per resolver. Spikes here precede every DNS-related outage.
  • Resolution failure rate. Per-resolver, per-query-type (A, AAAA, SRV).
  • Resolver health. If your in-cluster DNS (CoreDNS, kube-dns) is sick, the symptom is application-level, but the root cause is DNS.
  • NXDOMAIN rate. Sudden spikes often mean a misconfigured client, a typo in a config push, or a dependency renaming.

CoreDNS exposes Prometheus metrics natively. Tools like dns-exporter and the node-local-dns exporter add per-node detail.

Wiring DNS metrics through your alerting stack catches a class of incident that would otherwise look like "some services are intermittently failing."

Connection Tracking

Linux maintains a connection tracking table (nf_conntrack) for stateful firewall rules. Each entry corresponds to a flow. The table has a size limit, set by net.netfilter.nf_conntrack_max. When the table fills, new connections fail.

Most teams discover this limit in production. A high-traffic gateway, an unusual workload, a fleet running short-lived connections — any of these can fill the table.

Metrics to expose:

  • nf_conntrack_count current — how many entries are in use.
  • nf_conntrack_max configured limit.
  • Ratio between them. Alert above 80%.
  • nf_conntrack_no_per_cpu_lock_buckets and related counters for drops.

Most host exporters (node_exporter) include these. They are usually disabled or unmonitored by default; flipping them on is a five-minute change with real value.

TCP Retransmits

TCP retransmissions are the canonical signal of packet loss. A small rate (0.01-0.1%) is normal background; spikes above 1% indicate real packet loss or congestion.

The data source on Linux is /proc/net/snmp and /proc/net/netstat:

awk '/^Tcp:/{print $13}' /proc/net/snmp

Gives a cumulative count of TCP segments retransmitted. Compute the rate, divide by total segments out, get a percentage. Wire it into Prometheus.

When retransmits spike on a specific host, the next question is "to which destination?" eBPF tools like tcpretrans capture the destination of each retransmit, which narrows the problem to a specific path.

Per-Pod and Per-Service Visibility

In Kubernetes, network observability must be slicable by pod, service, and namespace. Otherwise an aggregate metric tells you "there is packet loss somewhere" without saying which application is affected.

Labels that pay off:

  • source_pod, dest_pod — for east-west flows.
  • source_namespace, dest_namespace — broader scope.
  • service — when traffic is going through a Kubernetes Service or Ingress.
  • verdict (allow / drop) — for policy-related observability.

Most CNI plugins with observability features (Cilium, Calico Enterprise) attach these labels. Without them, network metrics are correlated with application incidents only by manual investigation.

Egress Visibility

Egress traffic — calls to external APIs, third-party SDKs, partner integrations — is often the least-instrumented part of the stack and the most likely to be intermittently unreliable.

Minimum observability for egress:

  • Per-destination latency and error rate. Annotate destinations with the integration name ("Stripe API," "S3 us-east-1").
  • Outbound timeout rate. Cleanly distinct from "backend slow" — the call never returned at all.
  • DNS resolution time for external names.

Most teams add this only after a third-party API has caused an incident. Pre-instrumenting saves a debugging session every time it happens.

A Minimum Viable Network Observability Stack

For a Kubernetes environment with none today:

  1. Enable VPC flow logs at INFO level on production VPCs. Ship to S3 with 30-day retention.
  2. Deploy node_exporter on every node with the conntrack and TCP retransmit collectors enabled.
  3. Deploy CoreDNS metrics (or DNS exporter) and dashboard latency and error rate.
  4. Add a Grafana dashboard with conntrack utilization, TCP retransmit rate, DNS error rate, and flow log query links per service.
  5. Alert on conntrack >80%, retransmit rate >1%, DNS errors >1% sustained.

Add eBPF-based per-flow observability (Cilium Hubble, Inspektor Gadget) once the basics are in use and a specific incident makes the case.

A Closing Heuristic

The network is the most-blamed and least-instrumented layer in most stacks. The default assumption — "if the application metric is bad, the application is broken" — works until it doesn't. The cost of being able to say "the network is fine, look at the database" or "the network has 2% packet loss, that is the cause" is a small amount of instrumentation. The cost of not having that data is an hour of guessing per incident.


Figuring out whether the network is what's hurting your application? We help teams stand up the observability layer that answers it in seconds. scopeforged.com

Share this article

Related Articles

Need help with your project?

Let's discuss how we can help you build reliable software.