Lesson 14 of 14 · 11 min · Troubleshooting networks

Latency, bandwidth and packet loss

Covers: Ubuntu 24.04 LTS, iproute2 6.1 (tc netem), curl 8.5. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • The difference between latency, bandwidth, jitter and loss
  • How to measure each one, and where the time goes in a web request
  • Why packet loss hurts TCP so much more than its percentage suggests
  • How to simulate slow and lossy networks to test your application
  • Practical ways to reduce latency

In simple words

A network can be slow in different ways. Think of a road: latency is how long the trip takes, bandwidth is how many lanes there are, and packet loss is vehicles that never arrive. They need different fixes. This lesson shows how to measure each and why a little loss slows things down a lot.

Words to know

  • Latency: the time one message takes to arrive.
  • Bandwidth: how much data can pass each second.
  • Jitter: how much the delay changes from one message to the next.
  • Packet loss: messages that never arrive and have to be sent again.

Four different things

Term Meaning Analogy (a road) Unit
Latency how long one packet takes (one way) or a round trip (RTT) how long one car takes to drive the road milliseconds
Bandwidth how much data can be sent per second at best how many lanes the road has Mbit/s
Throughput what you actually achieve cars actually passing per hour Mbit/s
Jitter variation in latency how unpredictable the journey time is ms
Packet loss share of packets that never arrive cars that vanish percent

A fat pipe is not a fast pipe. A 1 Gbit/s satellite link still has about 600 ms of latency, so a chatty application that makes 50 sequential round trips needs 30 seconds, no matter the bandwidth. Conversely, a low-latency link with low bandwidth is fast for small requests and slow for downloads. Most web slowness is latency multiplied by the number of round trips, not lack of bandwidth.

Measuring latency: where does a request spend its time?

ping measures the network RTT. For a whole HTTP request, break it into phases with curl:

curl -o /dev/null -sS -w 'dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} ttfb %{time_starttransfer} total %{time_total}\n' https://example.com

Three real runs from the same machine:

dns 0.134324 connect 0.140678 tls 0.168611 ttfb 0.178526 total 0.178576
dns 0.050032 connect 0.163844 tls 0.192380 ttfb 0.202410 total 0.202455
dns 0.349723 connect 0.357252 tls 0.384865 ttfb 0.395347 total 0.395392

These values are cumulative seconds since the start. From the first run: DNS took 134 ms, the TCP connection another 6 ms (140.7 - 134.3), TLS 28 ms, and the server needed 10 ms for the first byte after TLS. Look at the second run: DNS was quick (50 ms) but the connect took 114 ms. In the third, DNS alone took 350 ms. Every phase can be the slow one, and they vary run to run, so measure many times and look at percentiles, not one number:

  • Median (p50) is the typical user.
  • p95 / p99 is the experience of the unlucky 5% or 1%. Averages hide these, and tail latency is what users complain about.

Where the time goes in a new HTTPS request: DNS lookup (1 RTT or more) + TCP handshake (1 RTT) + TLS 1.3 handshake (1 RTT) + request/response (at least 1 RTT) + server processing. With 100 ms RTT, a cold request cannot finish in under about 400 ms plus processing. Keep-alive and connection reuse remove the handshakes for later requests.

Bandwidth

Measure with iperf3: run iperf3 -s on one machine and iperf3 -c server on the other for a test of raw TCP throughput. Remember that it tests the path between those two hosts at that moment, with no relation to your application. Also check that it is the network and not the disk, CPU or the application: copying a file from slow storage is not a network test.

Bandwidth-delay product (BDP) = bandwidth x RTT = how much data must be "in flight" to fill the pipe. At 100 Mbit/s and 100 ms RTT the BDP is 1.25 MB, so TCP needs a window of at least that size; small windows or buffers limit throughput on long, fat paths. This is why a single download between continents may use only a fraction of the available bandwidth.

Packet loss and why it hurts

TCP treats packet loss as a signal of congestion. When it detects loss it retransmits the missing data and cuts its sending rate. Therefore:

  • Even 1% loss can reduce throughput dramatically on long-RTT paths, not by 1% but often by well over half.
  • Every loss costs at least one extra RTT of delay for that data, and may stall the whole stream behind it (head-of-line blocking).
  • On a wireless or congested link loss is usually bursty, which is worse than evenly spread loss.

UDP-based real-time traffic (calls, games, streaming) is affected differently: lost packets are simply gone, producing glitches and robotic audio, unless the application adds forward error correction.

Finding loss:

ping -c 100 host               # loss percentage in the summary
mtr -n -r -c 100 host          # loss per hop (judge by the final hop)
ss -ti dst 10.0.1.5            # per-connection TCP details: rtt, retrans, cwnd
nstat -az | grep -i -E 'retrans'   # system-wide retransmission counters

In ss -ti output look for retrans: and rtt: values. A rising retransmission count with healthy CPU points to the network.

Simulating bad networks

Do not wait for production to find out how your app behaves on a slow mobile connection in a train. Linux can add delay, jitter and loss to an interface with tc netem (needs root; do it on a test machine or container, never on a production interface, and remove it afterwards):

sudo tc qdisc add dev lo root netem delay 50ms 10ms loss 20%
ping -c 20 -i 0.1 127.0.0.1
sudo tc qdisc del dev lo root

Our run on the loopback interface printed:

20 packets transmitted, 10 received, 50% packet loss, time 1954ms
rtt min/avg/max/mdev = 93.331/100.053/114.165/6.639 ms, pipe 2

Why 100 ms and not 50: the rule applies to packets in both directions (request and reply each get 50 ms), and the loss is applied to both directions too, so a 20% setting can lose more than 20% of round trips, and random loss varies from run to run. Use it to test timeouts, retries and user experience. For containers and Kubernetes, tools such as Toxiproxy, Chaos Mesh and Linkerd/Istio fault injection do the same at higher levels.

Reducing latency

  1. Move content closer: a CDN for static files, regional deployments, and DNS/Anycast that route users to the nearest region. Distance is the one thing you cannot optimise away.
  2. Avoid round trips: keep-alive and connection pools, HTTP/2 or HTTP/3 (multiplexing), TLS 1.3 and session resumption, fewer sequential API calls (batch them or fetch in parallel), avoid redirect chains.
  3. Lower DNS cost: reasonable TTLs, caching resolvers close to the clients, dns-prefetch hints in pages.
  4. Compress and cache: gzip/brotli, Cache-Control, conditional requests; smaller responses need fewer packets.
  5. Fix server-side time: slow queries, cold caches and thread starvation show up as high "time to first byte". Latency is not always the network.
  6. Right-size MTU: avoid fragmentation and blackholes with tunnels and VPNs (path MTU).
  7. Reduce queueing ("bufferbloat"): a saturated link with large buffers adds hundreds of milliseconds to everything. Modern queue disciplines (fq_codel, cake) and traffic shaping help.
  8. Measure continuously: synthetic probes from several regions, real-user monitoring and percentiles on dashboards, with alerts on p95/p99 rather than only averages.

Common mistakes

  • Equating bandwidth with speed.
  • Looking at averages and missing tail latency.
  • Testing only on the fast office network or on localhost.
  • Reporting "packet loss" from one hop in mtr without checking later hops.
  • Leaving tc netem rules in place or applying them to a production interface.
  • Blaming the network for time spent in the application or database.

Try it yourself

  1. Run the curl timing command ten times against a site and calculate the median and worst case of total. Which phase varies the most?
  2. Use tc netem in a lab container to add 100 ms of delay and 2% loss; time a file download with and without it.
  3. Compute the bandwidth-delay product for a 50 Mbit/s link with 80 ms RTT.
  4. Take a page you built and count the number of sequential requests it makes; propose three ways to reduce the round trips.

Check yourself

What is the difference between latency and bandwidth?

Latency is the time a packet takes; bandwidth is how much data per second the link can carry. A link can have high bandwidth and high latency (satellite) or the opposite.

Why is p99 latency more useful than the average?

Averages hide the slowest requests. p99 shows what the worst 1% of users experience, which drives complaints and timeouts, especially when one page makes many requests.

Why does 1% packet loss hurt TCP throughput so much?

TCP interprets loss as congestion, retransmits the lost data (costing at least an RTT) and reduces its sending rate, so the effect on throughput is much larger than the loss percentage, especially at high RTT.

How many round trips does a new HTTPS connection need before the first byte of the response?

About three or four: DNS, the TCP handshake, the TLS 1.3 handshake, and the request/response itself. Reusing connections removes the handshakes.

Interview questions

A page loads slowly only for users in another country. What do you check and what do you change?

Measure DNS/connect/TLS/TTFB from that region (synthetic probes, real-user data), check the path with mtr, then reduce distance and round trips: CDN or regional deployment, keep-alive, HTTP/2 or 3, caching, and fewer sequential calls.

How can you test an application's behaviour under packet loss and latency?

Simulate with tc netem on a test interface, or use fault-injection tools (Toxiproxy, chaos tools, service mesh faults), and observe timeouts, retries and user-facing behaviour.

What is jitter and which applications are sensitive to it?

The variation in packet delay. Real-time audio and video, games and VoIP need steady delay; bulk transfers care more about throughput and loss.

Summary

  • Latency, bandwidth, jitter and loss are different; most web slowness is RTT multiplied by round trips.
  • Break a request into DNS, connect, TLS and first-byte times with curl -w; watch p95/p99, not just averages.
  • Packet loss hurts TCP far more than its percentage; find it with ping, mtr, ss -ti and retransmission counters.
  • Simulate bad networks with tc netem (carefully); reduce latency by moving content closer and removing round trips.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Networking for DevOps for DevOps Engineer and scores every answer, with what to fix.