Validate CSV headers before building dictionaries
Check CSV header uniqueness, required names and row widths before mapping fields, preserving IDs such as 007 as strings.
Validate before mapping by name
Dictionary-based CSV reading is convenient, but duplicate headers make field identity ambiguous. When two columns share a name, constructing a dictionary cannot preserve both values under that key. The supplied function reads the header first, checks it, and only then turns rows into dictionaries. Python’s CSV reader handles quoted fields and record boundaries [1].
The header must exist, contain at least one name, and have no empty names. It must also contain every requested required field. The code allows extra columns and reordered columns: required names are checked as a subset, and each row is zipped against the actual header order. An application that needs an exact schema should add an explicit extra-column policy.
Catch duplicates and wrong row widths
Duplicate names are rejected before dictionary construction, preventing silent loss of one of the same-named fields. Each later row must contain exactly as many fields as the header. A short or long row raises ValueError; the message includes the reader’s physical line number near the record where the width error was observed.
Physical line number is not the same as record number. A quoted field can contain an embedded newline, so one logical CSV record may span multiple physical lines. The csv reader tracks physical input lines, while applications often want a count of data records. If reporting user-facing row positions, track record counts separately and describe the distinction.
"""Python 3.10+; validate a CSV header before creating dictionaries."""
import csv
import io
import unittest
def checked_rows(text, required):
reader = csv.reader(io.StringIO(text, newline=""), strict=True)
header = next(reader, None)
if header is None or not header or any(not name for name in header):
raise ValueError("missing or empty header")
if len(header) != len(set(header)):
raise ValueError("duplicate header")
if not set(required) <= set(header):
raise ValueError("required columns missing")
result = []
for row in reader:
if len(row) != len(header):
raise ValueError(f"wrong field count near physical line {reader.line_num}")
result.append(dict(zip(header, row)))
return result
class HeaderTests(unittest.TestCase):
def test_reordered_columns_and_quoted_data(self):
text = 'note,id\n"a,b",007\n'
self.assertEqual(checked_rows(text, {"id"}), [{"note": "a,b", "id": "007"}])
def test_bad_headers_and_widths(self):
for text in ("", "id,id\n1,2\n", "id,\n1,2\n", "name\nAda\n", "id,name\n1\n", "id,name\n1,Ada,extra\n"):
with self.subTest(text=text), self.assertRaises(ValueError):
checked_rows(text, {"id"})
if __name__ == "__main__":
unittest.main()
Preserve textual identifiers
The parser returns strings. In the test, the identifier 007 remains exactly "007", preserving leading zeros that might be meaningful in an ID. Type conversion should be explicit and based on the column’s meaning rather than inferred from how its value looks. The reordered header and quoted comma in "a,b" also show that the code uses CSV parsing rules instead of splitting text manually.
The function accepts text and wraps it in StringIO with newline="". For an actual UTF-8 file, open it with newline="" as well; if it may begin with a UTF-8 byte order mark, use encoding="utf-8-sig" so the marker does not become part of the first header name [1]. This code does not open files or detect encodings itself.
Memory and schema choices
checked_rows appends every parsed dictionary to a list before returning. Its memory use therefore grows with the number and size of rows. That is appropriate for a compact example, but a large import may need a streaming interface that yields each validated row as it is read. Streaming can bound retained data, though a consumer may still choose to accumulate results.
The function requires named columns and exact row widths but does not impose an exact set of columns, semantic validation, or field-size policy. Those decisions depend on the application. Header and width checks are a first structural gate, not a complete validation of the data’s meaning.
Run the local checks
Download the example as how-can-you-validate-a-csv-header-quickly.py and run python3 how-can-you-validate-a-csv-header-quickly.py. It uses Python’s standard library only and requires Python 3.10 or newer. A successful run reports Ran 2 tests and OK.
The two test methods cover reordered columns, quoted content, a leading-zero identifier, invalid headers and short or long rows. They are local unittest observations for these inputs. They do not guarantee behavior for every dialect, file encoding, schema policy or large-file workload.