Network Debugging Cheat-Sheet: ss, lsof, dig and tcpdump for ‘Why Can’t It Connect?’

“It can’t connect” is not a diagnosis. It’s four different faults wearing the same error message, and the only way to tell them apart is to test each layer in turn rather than guess.

A service can’t reach its database. A webhook times out. A health check flaps. Every one of these arrives as the same vague complaint, and the instinct is to restart something and hope. The disciplined alternative is to walk the stack: is the thing listening, is the socket forming, does the name resolve, do the packets arrive? Four questions, four tools, and you stop guessing. Everything below is on a stock Ubuntu 24.04 box; netstat is gone from modern installs anyway, and ss replaced it.

Is the thing even listening, and who owns the socket?

Start at the destination, not the client. Before you blame the network, confirm something is actually bound to the port you’re aiming at. This is the single command worth committing to muscle memory:

# What is listening on TCP, numeric, with the owning process
ss -ltnp

# l = listening, t = TCP, n = numeric (no DNS/port-name lookups),
# p = show process (needs root/sudo to see PIDs you don't own)

# Same for UDP listeners (DNS, syslog, some agents)
ss -lunp

# Just one port, e.g. is anything on 5432?
ss -ltnp 'sport = :5432'

The output tells you two things people routinely miss. First, the local address: a process bound to 127.0.0.1:5432 is reachable only from the box itself, so a remote client will never connect no matter how open the firewall is. You want 0.0.0.0:5432 (all IPv4) or [::]:5432 (all IPv6) for external reach. Second, the owning process in the users:(("...",pid=...)) column, so you know whether the thing you think is serving the port is the thing that actually holds it.

When two processes fight over a port, or you’re chasing a stray file descriptor, lsof answers from the process side:

# Every open network socket, with command and PID
sudo lsof -i -P -n

# -P = don't translate port numbers to names, -n = don't resolve hosts
# (both keep it fast and unambiguous)

# Who holds TCP port 8080 right now?
sudo lsof -i TCP:8080

# Everything a specific process has open on the network
sudo lsof -i -a -p 4821

If ss -ltnp shows nothing on the port, stop. The problem is the service, not the network. Check that it started, check its bind address in config, check the logs. No amount of firewall archaeology will help you here.

Is the connection forming, or dying in a state?

If something is listening but clients still struggle, look at the state of live connections. ss takes state filters directly, which turns a wall of output into a targeted question:

# Established connections to a service (are clients actually attached?)
ss -tnp state established '( sport = :443 )'

# Connections stuck half-open — a classic firewall/SYN-drop signature
ss -tnp state syn-sent

# Piling up in TIME-WAIT? Count them — churn, not a leak
ss -tan state time-wait | wc -l

# Anything the kernel is holding in CLOSE-WAIT is your app not
# closing sockets — that IS a leak, and it's your bug
ss -tanp state close-wait

The states tell a story. A wall of SYN-SENT means your outbound packets are leaving but nothing is coming back — a dropped SYN, almost always a firewall or security group. Thousands of TIME-WAIT entries are usually harmless connection churn, not a fault. But CLOSE-WAIT accumulating is the application failing to close its end of sockets, and that one is code you own. This is exactly the kind of signal worth wiring into your operational metrics rather than only discovering during an incident; it feeds directly into the MTTR you’re trying to keep low.

Is it name resolution, quietly?

A surprising share of “network” outages are DNS. The tell is a connection that works by IP but fails by hostname. dig is the precise instrument; resolvectl tells you what the system resolver is actually doing, which on Ubuntu 24.04 is systemd-resolved, not /etc/resolv.conf as you might assume.

# The answer only — clean and scriptable
dig +short api.example.com

# Full record with timing; check the ANSWER SECTION and Query time
dig api.example.com A

# Ask a specific resolver, bypassing the system one
dig @1.1.1.1 api.example.com

# Follow the delegation chain when a record looks stale or wrong
dig +trace api.example.com

# What is the SYSTEM actually resolving, and via which server?
resolvectl query api.example.com
resolvectl status

If dig @1.1.1.1 resolves but resolvectl query does not, the public record is fine and your local resolver configuration is the fault — a bad search domain, a stale cache, or split-horizon DNS pointing at the wrong server. Fix the resolver, not the application.

Do the packets actually arrive?

When the first three layers check out and it still fails, prove it at the wire. tcpdump settles the argument about whether traffic is even reaching the host; mtr shows where along the path loss appears; curl -v narrates a single request end to end.

# Are packets for port 5432 arriving on this interface at all?
# -n no name lookup, -nn no port names either, -i interface
sudo tcpdump -nn -i any 'tcp port 5432'

# Watch only the handshake for one peer — SYN, SYN-ACK, or silence
sudo tcpdump -nn 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0 and host 10.0.4.12'

# Where does the path break, and where does loss start? (TCP mode, port 443)
mtr -rwzbc 100 -T -P 443 api.example.com

# One request, fully narrated: DNS, TCP, TLS, then HTTP
curl -v --connect-timeout 5 https://api.example.com/health

Read tcpdump for the handshake. You send a SYN and see a SYN-ACK back: the path and firewall are fine, look higher up the stack. You send a SYN and see nothing: something between you and the host is dropping it silently, which is a firewall, security group or network ACL. You see the SYN-ACK but your side sends a RST: the local application refused it. curl -v localises the same fault to a phase — it prints when DNS resolved, when the TCP connection was established, and when the TLS handshake completed, so a hang after “Connected” but before “SSL connection” is a TLS problem, not a routing one.

Diagnose the layer, then act

The whole point of this sequence is to stop the retry-and-pray reflex. Confirm the listener with ss -ltnp, read the connection state, rule DNS in or out with dig against a known resolver, and only then reach for tcpdump to prove the wire. Each step tells you which layer owns the fault, and once you know the layer you know whose problem it is — the service, the resolver, the firewall or your own code.

And when the fault turns out to be a transient one — a dropped SYN, a resolver blip, a timeout — the fix is rarely “try again harder” at the network layer. It’s making the caller safe to retry in the first place, which is a design decision you make before the outage, not during it. The commands above tell you what broke; that tells you whether retrying is even safe.