Data Processing
Data Processing covers turning text streams into answers: redirecting input and output, piping commands, filtering with regular expressions, and archiving files. Questions often present a small pipeline and ask what it prints or which command produces a required result.
Streams and redirection
Every process has standard input (fd 0), standard output (fd 1), and standard error (fd 2). > overwrites stdout, >> appends, < feeds a file as stdin, 2> redirects errors, and 2>&1 merges errors into stdout. /dev/null discards output. A pipe | connects one command’s stdout to the next command’s stdin; stderr keeps going to the terminal unless redirected.
Filters and archives
grep PATTERN FILE searches text; -i ignores case, -v inverts, -n prints line numbers, -r recurses. cut -d: -f1 takes fields, sort orders lines (-n numeric, -r reverse, -u unique), uniq collapses adjacent duplicates (-c counts), wc -l counts lines, head/tail show ends, sed 's/old/new/g' substitutes, and awk '{print $1}' prints fields. tar bundles files: tar -czf archive.tar.gz dir creates a gzip-compressed archive; tar -xzf extracts.
Worked example
To count the most frequent first field in a log: cut -d' ' -f1 access.log | sort | uniq -c | sort -nr | head cut extracts addresses, sort groups identical lines, uniq -c counts adjacent duplicates, the second sort ranks counts, and head shows the top.
Traps
uniq does not count non-adjacent duplicates without sort; > destroys existing content; grep uses regular expressions, so . and * are metacharacters; tar’s f must be followed by the archive name; gzip normally replaces the original file.