A misleading 404 top-talkers pipeline on a 4 GB nginx log

At least four independent defects:

• grep 404 is a substring match. It selects lines where 404 appears anywhere: the status field, a path such as /404.html, a byte count such as 1404, or even a user agent. Those lines are counted as 404 responses even when the status is 200.
• sort -n is ascending. head -10 therefore prints the ten smallest counts, not the largest. The correct order is sort -rn, or sort -n | tail.
• head after ascending sort hides the heavy hitters. 10.0.0.5 with thousands of 404s is at the bottom and is cut off. That is why it is absent, not because the log is truncated.
• cut -d' ' -f1 assumes a single-space separator and a fixed first field. It happens to work for these IPv4 lines, but it will break on malformed lines or if the log format changes, and it does nothing to validate the status.
• uniq -c only counts adjacent duplicates, so the input must be sorted on the same key. Here it is sorted by the whole line, which is fine, but the pattern is fragile if you later add fields.

A quick check is to compare grep -c ' 404 ' access.log with awk '$9==404' access.log | wc -l; the counts will differ.

Use awk to test the status field exactly, then sort and count:

awk '$9 == 404 {print $1}' /var/log/nginx/access.log \
  | LC_ALL=C sort \
  | uniq -c \
  | sort -rn \
  | head -10

Why it works:

• In combined log format the fields are: 1 client address, 2 ident, 3 auth user, 4 date, 5 timezone, 6 method, 7 path, 8 protocol, 9 status, 10 bytes. $9 is the status, so $9 == 404 is an exact numeric comparison and does not match a 404 in the path or in the byte count.
• IPv6 addresses contain colons but no spaces, so they remain a single $1 field and are counted correctly. No special handling is needed.
• awk streams line by line, so the 4.1 GB file is never loaded into memory; the sort's working set is bounded by the number of distinct addresses, not lines.
• LC_ALL=C sort gives a stable byte order, and uniq -c then counts adjacent identical addresses. sort -rn puts the largest count first.
• If the log can contain leading spaces or malformed lines, add NF >= 9 to the awk condition.

The structure stays the same; only the filter and the extracted field change.

For 500s: awk '$9 == 500 {print $1}' ... | sort | uniq -c | sort -rn | head.

For bytes sent, you usually want a sum rather than a count: awk '$9 == 200 && $10 ~ /^[0-9]+$/ {sum += $10} END {print sum}'. The $10 ~ /^[0-9]+$/ guard matters because a - in the byte field would otherwise be treated as zero silently, understating traffic.

The pitfall is that field numbers are positional. They are reliable only while the line matches the expected combined format. A request path containing an unencoded space, a different log format such as JSON, or an extra vhost field shifts everything after it, and $9 may no longer be the status. For production, validate with awk 'NF != 10 {print NR": "$0}' | head or parse with a format-aware tool. Do not assume $9 is universal across nginx configurations.