Lesson 11 of 12 · 11 min · Troubleshooting
Network checks with ip, ss, dig and curl
What you will learn
- A repeatable order for network debugging: address, route, name, port, application
ip addr,ip route,ss,ping,dig,getentandcurland what their output means- How to tell "refused", "timed out" and "name not resolved" apart, because each points to a different cause
- Testing a TCP port without extra tools
The networking course after this one explains the theory (IP, DNS, TCP, ports). Here you learn the checks themselves.
In simple words
When one computer can't reach another, check in order, like tracing why a letter didn't arrive. Is my own address right? Is there a road out? Is the name correct? Is anyone at that door? Each check has one command. This lesson gives you the order, and what each error message is really telling you.
Words to know
- IP address: the number that identifies a machine on a network, such as
10.0.0.5. - Port: a numbered door on a machine. Each service listens at its own door, such as 443 for HTTPS.
- DNS: the system that turns a name like
example.cominto an IP address. - Connection refused: the machine answered, but nothing is listening on that port.
- Timed out: no answer came at all. Often a firewall is silently blocking the way.
The method: work up from the bottom
When "the app cannot reach X" or "X cannot reach the app", test one layer at a time and stop at the first one that fails:
- Do I have an address and a link?
ip -br addr - Do I have a route out?
ip route - Can I reach the other machine by IP?
ping,curlwith an IP - Does the name resolve?
dig,getent hosts - Is something listening on the port?
ss -tlnp(on the server) and a connect test (from the client) - Does the application answer correctly?
curl -v
Jumping straight to the application log wastes time if the real problem is DNS.
1 and 2: address and route
ip -br addr
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0@if84 UP 172.17.0.2/16
-br is the brief form. You want UP and an address on the interface. lo is the loopback (the machine talking to itself, 127.0.0.1).
ip route
default via 172.17.0.1 dev eth0
172.17.0.0/16 dev eth0 proto kernel scope link src 172.17.0.2
The default via line is the gateway: where traffic for everything not on the local network goes. No default route means "Network is unreachable". To see which path a destination uses: ip route get 1.1.1.1.
3: reach by IP
ping -c 3 127.0.0.1
1 packets transmitted, 1 received, 0% packet loss, time 0ms
ping sends ICMP echo requests. Many networks and cloud firewalls block ICMP, so a failed ping does not prove the host is down; but a working ping proves routing works. For a TCP service, test the port itself (below).
4: names
Applications use names; networks use IPs. /etc/resolv.conf tells the machine which DNS server to ask (on systemd-resolved systems it points to a local stub like 127.0.0.53).
dig example.com A +noall +answer
example.com. 278 IN A 104.20.23.154
example.com. 278 IN A 172.66.147.243
Columns: name, TTL (seconds to keep it cached), class, record type, value. Useful variations:
dig @1.1.1.1 example.com +short # ask a specific DNS server: separates "my resolver is wrong" from "the record is wrong"
dig nosuch.invalid | grep status # status: NXDOMAIN means the name does not exist
dig -x 8.8.8.8 +short # reverse lookup
getent hosts example.com # what the system's resolver gives programs (includes /etc/hosts)
dig talks to DNS only. getent hosts follows the same path as applications (the order in /etc/nsswitch.conf, usually /etc/hosts first, then DNS). If dig works but the application fails, compare with getent. Typical DNS failures: wrong server in resolv.conf, a firewall blocking port 53, a stale cached answer, or a typo in /etc/hosts.
5: ports
On the server, what is listening?
ss -tlnp
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=89,fd=5))
LISTEN 0 4096 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=72,fd=15))
Options: -t TCP, -l listening, -n numbers (no name lookups, faster), -p process (needs root to show other users' processes), -u for UDP. The key column is Local Address:
0.0.0.0:80or[::]:80: listening on all interfaces; reachable from outside (if firewalls allow).127.0.0.1:5432: listening on loopback only: only this machine can connect. A very common reason why "the database works locally but not from the app server".10.0.1.5:8080: one specific interface.
Nothing listening on the port means the service is down or bound to another port. Connection states: ss -tan | awk 'NR>1{print $1}' | sort | uniq -c counts established, time-wait and other states.
From the client, can I connect to that port? Without installing nc, use Bash's built-in /dev/tcp:
timeout 3 bash -c '</dev/tcp/127.0.0.1/80' && echo "port 80 open"
timeout 3 bash -c '</dev/tcp/127.0.0.1/81'
port 80 open
bash: connect: Connection refused
Three different failures, three different meanings
| What you see | What happened | Look at |
|---|---|---|
| Connection refused | The host answered "nothing listens here" | Service down, wrong port, bound to loopback |
| Timed out (no answer) | Packets disappear: nobody replies | Firewall or security group dropping traffic, wrong route, host down |
| Name or service not known / Could not resolve host | DNS failed before any connection | resolv.conf, DNS server, typo |
6: the application
curl -sS -m 3 http://localhost:9999/
curl: (7) Failed to connect to localhost port 9999 after 0 ms: Couldn't connect to server
Exit code 7 is "failed to connect". Useful curl forms:
curl -I http://localhost/ # headers only
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' http://localhost/
curl -v https://example.com/ 2>&1 | head -20 # whole conversation: DNS, connect, TLS, request, response
curl --resolve example.com:443:1.2.3.4 https://example.com/ # force an IP for a name (test a new server before DNS changes)
curl -H 'Host: myapp.test' http://127.0.0.1/ # test a virtual host without DNS
Real output of -w:
200 0.000648s
and of curl -sv:
* Trying [::1]:80...
* Connected to localhost (::1) port 80
> GET / HTTP/1.1
< HTTP/1.1 200 OK
Lines beginning with > are what you sent, < what came back, * what curl did. Read the status code: 2xx fine, 3xx redirect, 4xx the request is wrong (401/403 auth, 404 not found), 5xx the server failed (502/503/504 usually mean a proxy cannot reach the backend).
Other tools
tracerouteormtrshow the hops to a destination (where a path breaks or slows).tcpdump -i eth0 port 80captures packets when you need proof of what is on the wire (the Networking course explains it).nmapscans ports on hosts you are allowed to test.- On firewalls:
sudo iptables -L -n,sudo nft list ruleset,sudo ufw status. In the cloud, remember security groups and network ACLs sit outside the machine.
Common mistakes
- Testing from the wrong place. "Works for me" on the server and fails for users: test from the client's network.
- Assuming ping failure means down. ICMP is often blocked.
- Ignoring the bind address. A service on
127.0.0.1is invisible from other hosts. - Forgetting the machine firewall and the cloud firewall are two separate layers.
- Debugging DNS with only
dig. Also test withgetent hosts, which is what programs use.
Try it yourself
- Run
ip -br addr,ip routeandip route get 8.8.8.8and explain each line. - Start
python3 -m http.server 8000 --bind 127.0.0.1, find it withss -tlnp, and fetch it withcurl. Restart it with--bind 0.0.0.0and compare thessoutput. - Check a port that is closed and a port that is filtered (use a host you own). How does each failure look?
- Resolve a name with
dig, withdig @8.8.8.8and withgetent hosts. Make/etc/hostsoverride it temporarily and see which tools notice.
Check yourself
What is the difference between "connection refused" and "connection timed out"?
Refused means the target host replied that no process listens on that port (an active reject, so the network path works). Timed out means no reply arrived, which points to a firewall silently dropping packets, a wrong route or a down host.
ss -tlnp shows 127.0.0.1:5432 for PostgreSQL. Why can another server not connect?
It listens only on the loopback interface, so connections from outside the machine are impossible. It needs to bind to the machine's network address (and the firewall and pg_hba.conf must allow it).
dig returns the right IP but your application still cannot resolve the name. What next?
Applications use the system resolver (nsswitch.conf, /etc/hosts, the local stub). Test with getent hosts name, and check /etc/hosts and the container's own resolv.conf.
How do you test whether a TCP port is open when nc and telnet are not installed?
timeout 3 bash -c '</dev/tcp/HOST/PORT' && echo open, or curl -v telnet://HOST:PORT.
Interview questions
A website is unreachable from your laptop. What are your first five checks?
Does the name resolve (dig)? Can I reach the IP (ping/curl to the IP)? Is the port open from my side (connect test)? Is the service listening on the server (ss -tlnp) and healthy (systemctl status, logs)? Is a firewall or security group blocking it?
What does a 502 or 504 from nginx mean?
nginx works but could not get a good answer from the backend it proxies to: 502 Bad Gateway is an invalid or no response (often the backend is down), 504 Gateway Timeout is no response in time. Check the backend service and the connectivity between the two.
How can you tell whether a slow request is DNS, connect, TLS or the server?
Use curl's timing variables: curl -o /dev/null -sS -w 'dns %{time_namelookup} connect %{time_connect} tls %{time_appconnect} first byte %{time_starttransfer} total %{time_total}\n' URL. The jump between values shows the slow phase.
Summary
- Debug bottom-up: address, route, IP reachability, name, port, application. Stop at the first failure.
ip addr/route,ss -tlnp,dig+getent, andcurl -vanswer 90% of questions.- Refused, timed out and cannot-resolve are different failures with different causes.
- Mind the bind address (
127.0.0.1vs0.0.0.0) and the two firewall layers.
Going deeper
- ip(8): addresses, routes and links.
- ss(8): socket statistics and filters.
- curl documentation: every option and the exit codes.
- BIND 9 dig documentation: the dig manual.
See how much of this you can explain
A free mock interview asks you new questions on Linux for DevOps for DevOps Engineer and scores every answer, with what to fix.