Lesson 11 of 14 · 10 min · Troubleshooting networks

ping, traceroute and mtr: finding where a path breaks

Covers: Ubuntu 24.04 LTS, iputils-ping 20240117, traceroute 2.1, mtr 0.95. Updated 13 Nov 2026. Part of the DevOps Engineer roadmap.

What you will learn

  • What ping measures and how to read it
  • How traceroute discovers the path with TTL, and what * * * means
  • Why mtr is the best tool for intermittent problems
  • How to interpret loss and latency along a path without false alarms
  • A short method for localising a fault

In simple words

When a connection is slow or broken, you need to know where on the road the problem is. ping asks "are you there, and how long did the answer take?". traceroute lists every stop on the way. mtr does both, again and again. This lesson teaches you to read them without raising false alarms.

Words to know

  • ping: sends a small test message and measures the time until the reply.
  • Latency: the delay, usually in milliseconds (ms).
  • Hop: one router on the path between you and the destination.
  • Packet loss: the share of messages that never arrive.

ping

ping sends ICMP Echo Request packets and waits for Echo Reply. It answers three questions: is the host reachable, how long does a round trip take, and how many packets are lost.

ping -c 3 1.1.1.1
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 14.398/40.601/90.956/35.615 ms
  • rtt (round-trip time) min/avg/max and mdev (standard deviation = jitter). Here the average is 40 ms but the range is 14 to 91 ms: the path is unstable, even though nothing was lost.
  • Useful options: -c N count, -i 0.2 interval, -s 1400 packet size (test MTU with -M do to forbid fragmentation), -W 1 timeout per packet.

Interpret carefully:

  • A reply proves the network path works in both directions at layer 3.
  • No reply does not prove the host is down. Firewalls and cloud security groups commonly drop ICMP. Test the actual service port instead (curl, nc -zv host port).
  • Test in order from near to far: your own gateway (ip route), then an IP on the internet (1.1.1.1), then a name (example.com). If the IP works but the name fails, it is DNS.
  • Typical latencies: same data centre under 1 ms; same city 2-10 ms; across India 20-60 ms; India to Europe 120-160 ms; India to the US west coast 200-250 ms. The speed of light in fibre sets a floor: you cannot beat physics by buying a faster server.

traceroute: the path

IP packets carry a TTL (time to live) counter, reduced by one at each router. When it reaches zero the router discards the packet and sends back an ICMP "time exceeded" message. traceroute exploits this: it sends probes with TTL 1, then 2, then 3..., and each router that answers reveals itself.

traceroute -n -m 4 -w 1 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 4 hops max, 60 byte packets
 1  172.17.0.1  0.044 ms  0.015 ms  0.010 ms
 2  192.168.0.1  2.748 ms  3.261 ms  3.345 ms
 3  10.125.0.1  4.536 ms  5.161 ms  5.234 ms
 4  * * *

Each line is a hop with three probe times. -n skips reverse DNS (faster), -m sets the maximum hops. Hop 1 is the Docker bridge, hop 2 the home router (a private address), hop 3 the ISP's network (private addressing inside the provider), and hop 4 shows * * *.

* * * does not mean a fault. Many routers are configured not to answer TTL-expired probes, or rate-limit them, while forwarding your real traffic perfectly. A fault is when the path stops and every hop after a point is stars until the end, and the destination really is unreachable.

Variants: traceroute -T -p 443 host probes with TCP SYN to port 443, which passes firewalls that block ICMP/UDP; tracepath needs no root and also reports MTU; -I uses ICMP.

mtr: ping and traceroute combined

mtr runs traceroute continuously and keeps statistics for every hop, which shows intermittent loss and latency over time:

mtr -n -r -c 100 1.1.1.1        # report mode: 100 packets per hop, then print

A real shortened run (3 probes):

HOST: dd23a794e2e4                Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 172.17.0.1                 0.0%     3    0.1   0.1   0.1   0.1   0.0
  2.|-- 192.168.0.1                0.0%     3  175.0  67.1   5.7 175.0  93.7
  3.|-- 10.125.0.1                 0.0%     3  142.8  53.2   7.9 142.8  77.6
  4.|-- ???                       100.0     3    0.0   0.0   0.0   0.0   0.0

Columns: loss percent, packets sent, last, average, best and worst round trip, standard deviation. Use at least 100 probes for meaningful numbers; 3 is only a demonstration. Note the huge worst values at hops 2 and 3 compared with their best: a busy home link, or router slowness answering probes.

Reading mtr correctly

  • Loss at one hop only, not at later hops: harmless. That router deprioritises its own replies (control-plane rate limiting).
  • Loss that begins at a hop and continues to the final hop: real loss starting there.
  • Latency that jumps at a hop and stays high afterwards: the delay starts between that hop and the previous one (a congested or long link, such as an intercontinental one).
  • A high number at one hop that returns to normal later: that router is slow to answer probes; it is not slowing your traffic.
  • Always judge by the final hop (the destination) first; intermediate numbers explain where a problem on the end-to-end path begins.

A method for localising a fault

  1. Is it the application or the network? Try another host and another service from the same place. Use curl -v to see whether DNS, connect, TLS or the server is slow.
  2. Near to far: loopback → gateway → internet IP → name → target. The first step that fails shows the layer.
  3. Both directions matter: paths can differ each way (asymmetric routing). Run the test from the other side too, if you have access.
  4. Run mtr from the client toward the server and from the server toward the client, for a few minutes during the problem, and save the output as evidence for your provider.
  5. Compare with a known good time or place: is it only one ISP, one region, one office?
  6. Rule out the end hosts: CPU saturation, full connection tables, or a full NIC buffer can look like network loss.
  7. Check what changed: a new route, firewall rule, MTU change, VPN, or provider maintenance notice.

Common mistakes

  • Concluding "network down" because ping fails. ICMP may be filtered.
  • Panicking about a * * * hop or loss at a middle hop that does not continue.
  • Too few probes (a 3-packet test says nothing about 1% loss).
  • Using only one direction.
  • Forgetting that VPNs, NAT and load balancers change what you see in a path.
  • Testing a name and an IP interchangeably and mixing DNS faults with network faults.

Try it yourself

  1. Ping your gateway, then 1.1.1.1, then example.com. Record the times and explain any difference.
  2. Run traceroute -n to three destinations (a nearby site, an Indian site, a site in another continent). Count the hops and plot the latency jump.
  3. Run mtr -n -r -c 100 to a destination and find the hop where latency first grows. Does the final hop agree?
  4. Find a host that does not answer ping, and prove it is up with curl or nc -zv.

Check yourself

Why might a server be up even though ping fails?

ICMP can be blocked by a firewall or security group. The service ports may still work, so test them with curl or nc -zv host port.

How does traceroute discover the routers on a path?

It sends packets with increasing TTL. Each router that decrements TTL to zero sends back an ICMP "time exceeded" message, revealing its address, so TTL 1 finds the first hop, TTL 2 the second, and so on.

You see * * * at hop 6 but hops 7 to 10 respond normally. Is hop 6 broken?

No. That router just does not answer probes (or rate-limits them) while forwarding traffic. A real break shows stars from some hop to the end with the destination unreachable.

In an mtr report loss is 40% at hop 5 and 0% at hops 6 to 10. What does that mean?

Hop 5 rate-limits ICMP replies (control-plane); traffic passing through is not lost, since later hops, including the destination, show no loss.

Interview questions

A user in Bengaluru says your site is slow. How do you investigate the network side?

Compare with other users and locations; have the user run curl -w timings or mtr toward the site; check DNS, connect and TLS time; look for loss and latency jumps in the path; and rule out server-side slowness with server metrics.

What is TTL in an IP packet?

A hop counter that each router decrements. At zero the packet is discarded, which prevents routing loops from circulating packets forever. Traceroute uses it to map the path.

What is the difference between latency, jitter and packet loss?

Latency is the delay of a packet's trip; jitter is how much that delay varies; packet loss is the percentage of packets that never arrive. Real-time apps are sensitive to jitter and loss; bulk transfers to loss and bandwidth.

Summary

  • ping tests reachability, round-trip time and loss; failure can be ICMP filtering rather than a dead host.
  • traceroute uses increasing TTLs to list the hops; * * * in the middle is usually harmless.
  • mtr gives per-hop loss and latency over time; judge by the destination, and loss that does not persist to the end is not real.
  • Localise methodically: near to far, both directions, enough samples, and keep evidence.

Going deeper

See how much of this you can explain

A free mock interview asks you new questions on Networking for DevOps for DevOps Engineer and scores every answer, with what to fix.