Text Utilities

Line Sort and Deduplicate

Clean, deduplicate, and sort text line by line with processing statistics.

Line sort and deduplication
Processing order: clean, deduplicate, sort
0Input lines
0Output lines
0Duplicates removed
0Empty lines
Processed output
—

Line Sort and Deduplication

Process one value per line in a defined order: trim, remove blanks, deduplicate, then sort. Input, output, and duplicate counts are shown.

Use Cases

  • Clean domain, IP, dependency, or configuration lists
  • Create stable sorted input for review
  • Case-insensitive deduplication preserves the first spelling encountered

A line-oriented list cleaner: it splits pasted text on line breaks, trims each line, drops blank lines, removes duplicate lines and re-sorts the remainder, then reports input, output, removed-duplicate and blank counts. Use it to normalize domain lists, IP allowlists, dependency names or log exports into one deterministic, review-ready order.

The pipeline order is fixed: trim, drop blanks, dedupe, sort

All four stages run inside one computed pass and cannot be reordered. Trimming happens before deduplication, so a padded entry such as staging collapses into its unpadded twin and counts as a duplicate rather than a second item. Deduplication walks the list with a Set keyed by the line itself, or by toLocaleLowerCase() of the line when case sensitivity is switched off; the first surviving occurrence is kept and every later match increments the duplicate counter. Sorting runs last, after the list has already been shortened, which means position information is gone: with any sort option other than keeping the original order, you cannot tell from the output which duplicate originally came first. The replace-input button writes the result back into the editor, and a second pass over it changes nothing because trimming, deduplication and stable sorting are all idempotent.

What counts as a line break, and what does not

Splitting uses the pattern /\r?\n/, so only LF and CRLF delimit lines. A file that separates records with a lone CR, as classic Mac text and some embedded device exports do, arrives as one single line that deduplication and sorting treat as one entry. A trailing newline produces a final empty field, which is why a paste that looks like 7 lines reports 8 input lines and an extra blank. Trimming strips more than the ASCII space: trim() also removes U+00A0, the ideographic space U+3000 and a leading U+FEFF byte-order mark, so a first line copied from an Excel or Notepad export cleans up without a separate step. Output is always rejoined with a single LF, never with the delimiter style of the input.

Sorting is locale collation, not byte order

Ascending and descending both call localeCompare against the ICU collation of the browser default locale, so the order is not codepoint order: in en-US, ab sorts before aB because lowercase wins at the tertiary level, and hyphens and spaces are weighted weakly. Digits compare as characters, which puts 10 and 100 ahead of 9. The by-length mode compares a.length - b.length, and that length is counted in UTF-16 code units, so one emoji counts as 2; equal lengths fall back to localeCompare. Since ES2019 Array.prototype.sort is stable, so ties keep their pre-sort order. Because the collation comes from the browser locale, the same list can order differently on a ja-JP or tr-TR machine, and the command-line sort utility obeys LC_COLLATE instead, so the two rarely agree byte for byte.

Worked example

Cleaning a service list with every option left at its default

Input 
production
 staging 
Development
production

staging
API
api
Output 
api
API
Development
production
staging

The paste is split into 8 fields, one of them the empty line between production and staging. Trimming first makes the padded staging and production lines exact duplicates, so the duplicate counter reads 2 and 8 minus 1 blank minus 2 duplicates leaves 5 output lines. The sort places api before API, which is the sign that ICU collation ran instead of a codepoint comparison, where A (0x41) would precede a (0x61).

Limits and pitfalls

  • Sorting has no numeric branch: every comparison goes through localeCompare, so 9 sorts after 10 and after 100. Zero-pad to a fixed width first if you need numeric order.
  • Case-insensitive deduplication lowercases with toLocaleLowerCase() in the browser default locale, so under tr-TR the letter I becomes ı and I stays a separate entry from i.
  • Only LF and CRLF are treated as line breaks; a lone-CR file stays one long line, and the output is always written with LF, so round-tripping a CRLF file silently changes its line endings.
  • Whitespace-only lines survive unless trim is enabled as well, because the blank filter tests Boolean and a string of spaces is truthy.
  • The component sets no cap on line count or input size and holds the whole list plus a Set in memory; lines are never parsed or escaped, so tabs, quotes and CSV field boundaries pass through as literal text.

Related tools

  • Text Statistics — Count characters, words, lines and more from text.
  • Text Diff — Compare two texts and highlight added, removed and unchanged parts.
  • JSON Lines Validator — Validate and normalize JSONL/NDJSON line by line or convert it to a JSON array.
  • CSV / JSON Converter — Convert between CSV and JSON arrays with quoted fields, newlines, and custom delimiters.

When it goes wrong

The "Case sensitive" box is greyed out and clicking it does nothing.

CauseThat checkbox is bound to :disabled="!unique", because the flag only decides how the deduplication key is built: with case sensitivity on the key is the raw line, with it off the key is toLocaleLowerCase() of the line. No deduplication means no key, so the control is locked rather than merely pointless.

FixRe-tick "Remove duplicates" first; then clear the case box only if API and api genuinely have to collapse into one entry. If both must survive, keep case sensitivity on and accept that they occupy two output lines.

Copy and Replace input are greyed out, and the result panel holds nothing but a dash.

CauseBoth buttons carry :disabled="!processed.output", so an empty result disables them. The dash is a template placeholder rendered for a falsy output — it is not a character you can copy. Empty happens easily: trim plus blank removal turns a paste of blank and whitespace-only lines into an empty list.

FixUncheck "Remove empty lines" to see whether anything was left after trimming, and confirm the textarea is not empty. If the paste really was all whitespace, the empty result is correct and there is nothing to copy.

"Empty lines" reads 2 when I can find only one blank row in what I pasted.

CauseThe counter is computed on the pre-trim split array with line.trim() as the test, so rows holding only spaces or only tabs count as empty. A trailing newline adds one more empty field of its own, and the figure is reported whether or not "Remove empty lines" is ticked.

FixRead it as "blank rows in the paste, whitespace ignored". With trim and blank removal both on, cross-check it against input lines minus output lines minus duplicates removed; the three numbers should reconcile exactly.

I pressed Replace input and every counter changed, so I lost track of what was actually cleaned.

CausereplaceInput() writes processed.output back into the textarea, and the computed pass then rebuilds the stats from that shortened list. Input lines collapses onto output lines by definition, duplicates removed drops to 0, and a second run genuinely changes nothing.

FixCopy the four figures, or copy the output, before pressing it. Replace input is the final commit of an already-reviewed result, not a preview or an undo point — there is no history to step back through.

A Chinese name list sorted ascending looks pinyin-ordered rather than the byte order my database expects.

CauseAscending and descending both compare with localeCompare against the browser ICU collation. For Chinese the default collation is pinyin-based, and spaces and punctuation carry weak weight, so the result matches neither UTF-8 byte order nor GBK code order.

FixWhen a database or firewall will consume the list, sort it there instead, or pick "Keep original order" here so the file order survives the trip. Ordering is only this page's business when the list is read by a human.

One concrete scenario

Three exports of sixty hostnames, merged before a firewall review

Each of three load balancers exports 60 hostnames, and the operator pastes all three files into one box. That is 180 entries but 183 input lines: one blank row was left between the files twice, and the trailing newline of the last file cuts one more empty field. Fourteen hostnames occur in more than one export, and five of them match only after trimming, because they were copied out of a Markdown table with trailing spaces. Defaults — trim, blank removal, deduplication, case sensitivity, ascending — land on 166 lines. Then clear the case box: Example.COM and example.com are one DNS name, so duplicates removed rises and the list shrinks again. Compare the result against the firewall rules, not against the three exports.

Frequently Asked Questions

Which duplicate is kept? Are different cases duplicates?
Comparison is case-sensitive by default (Example and example are two entries); enabling case-insensitive mode treats them as duplicates and keeps the first spelling. Order stays stable — the original input order is preserved.
What sorting rule is used? Will numbers sort wrong?
Default is lexicographic, so 10 sorts before 9. For numeric or version lists, switch on natural sort (numeric comparison); for mixed content, normalize formats first.
What is the typical workflow?
The standard list-cleanup pipeline: trim edges → drop empty lines → deduplicate → sort. Perfect for domain lists, IP allowlists, dependency lists, and subscriber lists. All local — sensitive lists never leave the browser.

Content reviewed: