What Is a Useful Command-Line Check for Duplicate IDs?
From the earlier reference collection. Read the current software blog.
Parse the identifier column by name and report every repeated value with its source rows, rather than relying on fragile column positions.
Identify the field, not a position
A duplicate check should target the field that is supposed to be unique, such as stop_id or route_id, by its header name. A command that assumes the identifier is always the first column can silently check the wrong data after a file's columns are reordered. Read the header and fail clearly if the expected field is absent.
For CSV, use a CSV parser rather than splitting each line on commas. Quoted fields can contain commas, and line endings or escaped quotes can confuse a basic text pipeline. A small script is often clearer and safer than a long chain of shell filters.
Keep row provenance
For each identifier, record the source row number or another stable location as it is read. When an ID appears again, report both locations: duplicate stop_id 'A17' at rows 4 and 29. This gives the person fixing the data a direct path to the records involved.
Decide whether blank identifiers count as duplicates, invalid values, or a separate missing-field error. Repeated blanks are usually more helpful to report as missing IDs than as a collision, but the rule depends on the schema. State the choice in the diagnostic so an operator can interpret the result.
Choose normalization deliberately
Identifiers may be case-sensitive, whitespace-sensitive, or governed by another exact rule. Do not silently trim or lowercase them just to make a duplicate check easier. If the format defines normalization, apply it and report both the normalized key and original values when useful. Otherwise, compare values exactly and flag suspicious whitespace separately.
A practical check can maintain a map from ID to its first row. On a later occurrence, emit a duplicate message and append the new row. Keeping all occurrences can reveal an ID used three or more times, rather than stopping after the first collision.
Return a useful result
If duplicates violate the input contract, return a nonzero status and summarize the number of repeated IDs. Keep detailed findings on standard error or in a designated report file, especially if standard output is expected to carry transformed data. For a large file, cap displayed examples but preserve a full machine-readable report when requested.
A good duplicate check answers three questions: which field was checked, which values repeat, and where each occurrence came from. That makes it useful during both manual cleanup and automated feed validation.