How Do You Document a Data Transformation?
From the earlier reference collection. Read the current software blog.
Make every conversion traceable by naming inputs, steps, assumptions, and the fields that appear in the output.
Inventory the source fields
Start by identifying the source dataset, file, and fields used. Include the version or retrieval context when available, and say whether the transformation consumes every row or filters some out. A reader should be able to locate the values that enter the process without guessing from a column name alone.
For example, a service that derives a local display time from a source timestamp should identify the original timestamp field and its stated time zone. If it combines several fields, list each one and explain their role. Note fields that look relevant but are ignored, especially when their omission could change interpretation.
Describe each processing step
Write the transformation as an ordered sequence: parse, validate, filter, convert, aggregate, and serialize, as applicable. Explain units and formats at each boundary. If a temperature value moves from tenths of a degree to whole degrees, show the conversion and rounding rule. If missing rows are discarded, state the condition that causes a row to be dropped.
A small worked record helps. Suppose an input row contains value=215 in tenths of a unit. A conversion might divide by ten and emit 21.5. If the output rounds to an integer, explain whether that produces 21, 22, or a language-specific result. Avoid vague phrases like “normalized” unless the exact operation is also described.
Record assumptions and exceptions
Transformations often depend on assumptions that are not encoded in the input: units, coordinate systems, local time zones, or sentinel values. State each assumption and how the system behaves when evidence is missing or contradictory. Distinguish an assumed default from a verified property of the source.
Explain errors and edge cases, including malformed values, duplicates, nulls, and out-of-range records. For example, a duplicate identifier might be rejected, retained as two rows, or resolved by choosing a record according to a timestamp. Document the actual policy. If the behavior differs by source, say so and link the corresponding mapping or parser configuration.
Specify the output contract
List the output fields, their types, units, and meanings. Describe whether records retain source identifiers, whether timestamps are shifted, and whether the output order is meaningful. If a field is derived, identify its source fields and formula. Include a compact before-and-after example that can be checked by hand.
Finish with traceability: record the transformation version or code release and keep enough provenance to connect an output to its inputs. When a source corrects prior data, explain whether the project recomputes historical outputs or preserves old snapshots. This lets downstream users decide whether a change reflects new source data, a changed algorithm, or both.