Programming · free software · the open webSubscribe via RSS

Endianness belongs in the format, not in a lucky host assumption

Binary formats must specify byte order and layout instead of inheriting a machine’s memory conventions. A two-byte example shows why explicit endianness is more portable than native alignment assumptions.

A number has a value; bytes need an order

The integer 0x0102 has a clear numeric value, but two bytes do not encode it unambiguously until a format assigns significance to positions. In big-endian order, the high byte comes first: 01 02. In little-endian order, the low byte comes first: 02 01. A reader that guesses the wrong order still receives two valid bytes, but reconstructs a different number.

That is why endianness is a property of a file or wire format. It should not be left to whichever CPU happens to produce the data. Python’s struct format prefixes make the choice explicit: < selects little-endian, > selects big-endian, and ! denotes network byte order, which is big-endian. [1]

The example fixes the two-byte field

The example packs the same unsigned two-byte value with <H and >H. The little-endian representation is 02 01; the big-endian representation is 01 02. Unpacking the little-endian bytes with <H reconstructs 0x0102, demonstrating that decoding must use the same format decision as encoding. The code’s hexadecimal output is a view of the bytes in storage order, not a different numeric value.

The H field specifies an unsigned short with a standard size of two bytes under these explicit byte-order prefixes. A format is a compact statement about representation: both the width and byte order are fixed for this field. Without that prefix, struct uses native mode, which follows the platform and compiler’s conventions and can include alignment padding. That default is useful when matching a local C struct, but it is a poor implicit contract between independently built programs.

import struct

little = struct.pack("<H", 0x0102)
big = struct.pack(">H", 0x0102)
print(little.hex())  # 0201
print(big.hex())     # 0102
assert struct.unpack("<H", little)[0] == 0x0102
Locally checked Python example · Download example

Byte order is not the same as alignment

A binary layout has more than one portability decision. Byte order determines the order of bytes within a multi-byte scalar. Alignment determines where fields start and whether padding bytes appear between them. In struct, the native @ mode uses native byte order, native sizes, and native alignment. The = mode keeps native byte order but uses standard sizes and no alignment. The < and > modes use standard sizes and no automatic alignment padding. [1]

Imagine a record containing a one-byte tag followed by a four-byte integer. A native C layout may insert padding after the tag so that the integer starts at an aligned address. A standard external format must decide whether that gap exists; the struct module does not invent padding for explicit standard layouts. If an implementation assumes a native struct layout while another serializes fields consecutively, they can disagree even when both machines use the same endian order. A protocol should specify field widths, ordering, padding, and any required alignment as separate rules.

Convert fields, not an imagined whole structure

Endianness concerns the representation of multi-byte values. It does not mean reversing every byte of a record. Suppose a record contains a one-byte flag, a two-byte count, and a four-byte timestamp. Changing endian convention affects the byte order inside the count and timestamp; the flag has no internal order to reverse. The record’s field sequence remains as its schema defines it, and padding is handled according to the format’s layout rules.

Cap’n Proto makes this explicit in its encoding specification: integers are encoded little-endian, primitive values are aligned to multiples of their sizes, and structures have defined data and pointer sections. Those are format rules, not assumptions about the host’s in-memory struct representation. [2] When translating between a host representation and a protocol, decode each field according to the protocol and then use the host-language value. A blanket byte reversal across an arbitrary structure can reorder unrelated fields, corrupt strings or byte arrays, and mishandle padding.

Choose the contract before writing bytes

For data that stays inside one process and must match a C ABI, native packing can be appropriate because the compiler’s layout is the intended contract. For files and network messages, explicit standard formats make the representation reproducible across platforms. The choice also affects evolution: fixed widths and deliberate padding let a reader locate the next field without relying on host-specific details.

The useful review question is not simply “Which endian is this machine?” It is “What representation does this format promise?” Once that answer is written down, encoder and decoder can share the same field widths, byte order, and alignment rules. The two-byte example is small, but it captures the whole discipline: make interpretation part of the format so the same bytes have the same meaning everywhere.

References

  1. Python struct: binary data
  2. Cap’n Proto encoding

Suggest a correction · All posts