Word Frequency

Word Frequency

What is this problem about?

"Word Frequency" is a unique challenge typically solved in a Unix Shell environment. You are given a text file and asked to count the frequency of each word and output the results sorted by frequency in descending order.

Why is this asked in interviews?

This is a popular question for DevOps, SRE, and Backend roles at companies like Google and Bloomberg. It tests your familiarity with command-line tools and the ability to pipe multiple small utilities together to solve a data processing task. It evaluates how you handle text formatting, sorting, and counting without writing a full-blown program in a language like Python or Java.

Algorithmic pattern used

The pattern involves a chain of shell commands:

1. cat to read the file.
2. tr or sed to replace spaces/newlines and normalize the text.
3. sort to group identical words together.
4. uniq -c to count the occurrences of each grouped word.
5. sort -nr to sort the counts numerically in descending order.
6. awk to format the final output.

Example explanation

File content:

sky is blue
blue is nice

1. Normalize to one word per line: sky, is, blue, blue, is, nice.
2. Sort: blue, blue, is, is, nice, sky.
3. Count: 2 blue, 2 is, 1 nice, 1 sky.
4. Sort by count: 2 is, 2 blue, 1 sky, 1 nice. (Note: Tie-breaking rules may apply).

Common mistakes candidates make

• Ignoring Whitespace: Not handling multiple spaces or tabs between words correctly.
• Case Sensitivity: Forgetting to handle "Word" and "word" if the problem requires case-insensitive counting.
• Piping Order: Getting the order of sort and uniq wrong. uniq only works on adjacent duplicate lines, so you must sort before counting.

Interview preparation tip

Practice your basic Unix commands: grep, sed, awk, sort, and uniq. Being able to perform complex text processing in a single line of shell script is a highly valued skill in production environments.