Remove Duplicate Lines: Why Trimming Changes the Result

Whitespace is trimmed before duplicates are removed

The tool splits LF or CRLF lines, applies optional trimming, removes blank lines if selected, removes duplicates, and then optionally sorts. Trimming and blank-line removal are on by default; sorting is off. JavaScript trim() removes whitespace defined by the specification from both ends, preserving spaces inside a line.

The first input line has a space on each side of A. With trimming and blank-line removal on, the output is two lines: A and B. Turning off trimming alone leaves three: the spaced A, the unspaced A, and B. The JSON notation below puts quotes around line contents to make spaces visible; the tool itself separates output lines with line breaks.

[" A ", "A", "A", "", "B"]
trim = true:  ["A", "B"]
trim = false: [" A ", "A", "B"]

ECMAScript: String.prototype.trim

Exact strings, not numeric values or meaning

The tool uses Set to keep one copy of each identical string. Apple and apple remain separate, as do 01 and 1. A B and A B have different internal spacing and are also kept separate. It does not compare URL meaning, fold case, or normalize Unicode. With sorting off, the first-occurrence order is retained.

["Apple", "apple", "01", "1", "A B", "A  B"]

For keyword lists, try trimming before comparison. Turn it off when indentation or edge spaces matter. Blank-line removal still excludes whitespace-only lines when trimming is off. Keep the original, check a few representative lines, then process the full list. After changing input or options, click Remove duplicates again to generate a new result.

ECMAScript: Set.prototype.add

Frequently Asked Questions