Back to Commands

uniq

text processingLinux/Unix

The uniq command is one of the most frequently used commands in Linux/Unix-like operating systems. uniq Report or omit repeated lines

See Examples View Syntax

Quick Reference

Command Name:

uniq

Category:

text processing

Platform:

Linux/Unix

Basic Usage:

uniq [options] [arguments]

Common Use Cases

Syntax

uniq [OPTION]... [INPUT [OUTPUT]]

Options

Option	Description
`-c, --count`	Prefix lines with the number of occurrences
`-d, --repeated`	Only print duplicate lines, one for each group
`-D, --all-repeated[=METHOD]`	Print all duplicate lines. METHOD can be 'none' (default), 'prepend', or 'separate'
`-f, --skip-fields=N`	Avoid comparing the first N fields
`--group[=METHOD]`	Show all items, separating groups with an empty line. METHOD can be 'separate' or 'prepend'
`-i, --ignore-case`	Ignore differences in case when comparing
`-s, --skip-chars=N`	Avoid comparing the first N characters
`-u, --unique`	Only print unique lines
`-z, --zero-terminated`	Line delimiter is NUL, not newline
`-w, --check-chars=N`	Compare no more than N characters in lines
`--help`	Display help and exit
`--version`	Output version information and exit

Examples

How to Use These Examples

The examples below show common ways to use the uniq command. Try them in your terminal to see the results. You can copy any example by clicking on the code block.

Basic Examples:

Remove duplicate consecutive lines from a file

uniq input.txt

Remove duplicate consecutive lines and save to a new file

uniq input.txt output.txt

Count occurrences of each line (-c)

uniq -c input.txt

Display only duplicate lines (-d)

uniq -d input.txt

Display only lines that appear exactly once (-u)

uniq -u input.txt

Advanced Examples:

Case-insensitive comparison (-i)

uniq -i mixed_case.txt

Skip first 3 fields when comparing lines (-f 3)

uniq -f 3 data.txt

Skip first 5 characters when comparing lines (-s 5)

uniq -s 5 data.txt

Compare only the first 10 characters of each line (-w 10)

uniq -w 10 data.txt

Count occurrences of each line, sorted by frequency

sort file.txt | uniq -c | sort -nr

Find all unique values in a specific column of CSV data

cut -d',' -f2 data.csv | sort | uniq

Find all duplicate values in a column

cut -d',' -f3 data.csv | sort | uniq -d

Count all unique values in a column

cut -d',' -f2 data.csv | sort | uniq | wc -l

Display lines that appear more than once (with count)

sort data.txt | uniq -c | awk '$1 > 1 {print $0}'

Find lines that appear in both file1 and file2

sort file1 file2 | uniq -d

Try It Yourself

Practice makes perfect! The best way to learn is by trying these examples on your own system with real files.

Understanding Syntax

Pay attention to the syntax coloring: commands, options, and file paths are highlighted differently.

Notes

The `uniq` command is a powerful text processing utility in Unix and Linux systems that filters out repeated lines from input text. Its primary function is to detect and handle duplicate lines, but it has an important limitation: it only works on adjacent duplicate lines. This means that the input typically needs to be sorted first with the `sort` command to ensure that duplicate lines are adjacent to each other. Here's a deeper look at how `uniq` works and its key features: 1. **Basic Functionality**: By default, `uniq` reads input (either from a file or standard input) and writes output with consecutive duplicate lines collapsed to just a single line. This provides a quick way to eliminate redundancy in data. 2. **Counting Occurrences**: With the `-c` option, `uniq` prefixes each output line with a count of how many times that line appeared consecutively in the input. This is extremely useful for frequency analysis and finding the most common items in a dataset. 3. **Filtering by Uniqueness**: The command can filter lines based on how many times they appear: - `-d` shows only lines that appear multiple times (duplicates) - `-u` shows only lines that appear exactly once (unique lines) - `-D` shows all duplicate lines (not just one representative from each group) 4. **Flexible Comparison**: `uniq` offers several ways to customize how lines are compared: - `-i` performs case-insensitive comparison - `-f N` ignores the first N fields of each line when comparing - `-s N` ignores the first N characters of each line - `-w N` only compares the first N characters of each line 5. **Field-Based Processing**: The ability to ignore fields makes `uniq` particularly useful for processing structured data like CSV files or log entries, where you might want to check uniqueness based on specific columns or data fields. In practice, `uniq` is frequently used in data processing pipelines, often in combination with other text processing tools like `sort`, `cut`, `grep`, and `awk`. A common pattern is `sort file.txt | uniq` to remove all duplicate lines from a file, or `sort file.txt | uniq -c | sort -nr` to count occurrences of each line and sort by frequency. Data analysts and system administrators regularly use `uniq` for tasks such as: - Analyzing log files to find common events or errors - Generating reports showing unique values in datasets - Finding duplicate data in configuration files - Creating frequency distributions of values - Removing duplicate entries from data files It's worth noting that while `uniq` is efficient for processing text files, it has limitations. For very large datasets, more specialized tools or databases might be more appropriate. Additionally, the requirement that duplicate lines must be adjacent means that `uniq` is almost always used after `sort` in a pipeline, which can impact performance for extremely large datasets. Despite these limitations, `uniq` remains one of the core text-processing utilities in the Unix/Linux toolbox, valued for its simplicity, efficiency, and integration with other command-line tools.

Related Commands

These commands are frequently used alongside uniq or serve similar purposes:

Explore All Commands

Use Cases

Learn By Doing

The best way to learn Linux commands is by practicing. Try out these examples in your terminal to build muscle memory and understand how the uniq command works in different scenarios.

See Examples Try Lab Exercises

$ uniq

View All Commands