What Should a Schema Compatibility Test Check?
From the earlier reference collection. Read the current software blog.
Load representative old and new messages and verify how clients handle added fields and preserved data.
Keep versioned sample messages
A schema compatibility test begins with representative messages from the versions the application claims to support. Store small samples as fixtures and load them through the actual client decoder. A hand-constructed object that bypasses decoding may miss wire-format changes such as field numbering, optionality, or default behavior.
Include an old message with no newly added field and a new message with that field populated. Assert that the old client follows its documented behavior when the field is absent, and that a newer client reads the value when present.
Test the promised direction
Compatibility can mean different things: a new reader accepts old data, an old reader tolerates new data, or both. Name the direction in each test. For an additive field, verify exactly what happens when an older client encounters the message: it may ignore the field, preserve unknown data, or reject the message according to the schema and implementation.
Do not treat “the decoder did not crash” as sufficient. Check the resulting application behavior. If a new field is optional, confirm the old message receives the intended default or unknown state rather than a misleading concrete value.
Protect existing fields and meaning
Compare established fields before and after the schema change. Their values should retain their meaning, and required fields should not silently become absent. If a field's units, enumeration values, or interpretation changes, that is a semantic compatibility concern even when the wire structure still parses.
Include unknown enum values or unrecognized optional data if the format permits them. The client should have a documented fallback, such as an “other” state, rather than assuming the set of values never grows.
Make fixtures understandable
Record the schema version and the behavior each fixture is meant to protect. Keep samples minimal so a changed assertion has a clear cause. When a compatibility test fails, report which version pair and client behavior were involved.
These tests turn schema evolution into a concrete promise to consumers. They also help reviewers assess whether adding a field is truly additive for the readers that matter.