JSON vs Protocol Buffers vs XML

Data serialization is the process of converting in-memory data structures (like objects, structs, maps, or trees) into a standardized sequence of bytes. This byte array can then be transmitted over a network socket, saved to a file on a disk, or cached in an in-memory database like Redis.

The reverse process, deserialization, reads the raw byte array and reconstructs the original in-memory data structure in the target application.

Real-World Analogy

Imagine packing furniture to move to another city. You cannot transport a fully assembled wardrobe easily because of its shape and size. Serialization is like disassembling the wardrobe into flat wooden panels, sorting them, and packing them into boxes (the byte sequence) with assembly instructions. Deserialization is the recipient unboxing and rebuilding the wardrobe in their new room according to the instructions.

JSON is a text-based format designed to represent structured data using key-value pairs and arrays. Over the last two decades, JSON has become the dominant standard for public-facing APIs, web pages, and config configurations.

Characteristics of JSON

• Self-Describing: JSON payloads carry both their structure and values. The keys (e.g. "name", "age") are repeated inside every single payload message.
• Human-Readable: Because JSON is standard ASCII/UTF-8 text, it is easy for developers to debug, print, and write manually.
• Schema-Optional: JSON does not enforce strict typing at the transport level. An API can append or change field types dynamically, though this can lead to runtime errors if validation is not handled in application code.
• High CPU Overhead: Parsing JSON requires scanning text strings character-by-character, evaluating tokens, allocating memory dynamically, and escaping characters. This makes serialization and deserialization CPU-bound and slow.

XML is a tags-based hierarchical text format that predates JSON. It is widely used in legacy systems, enterprise application integrations, and communication protocols like SOAP.

Characteristics of XML

• Schema Enforcement: XML supports strict data verification schemas, such as Document Type Definitions (DTDs) or XML Schema Definitions (XSDs), ensuring payloads strictly comply with specific standards.
• Verbose Metadata Overhead: Because XML requires opening and closing tags (e.g. <age>30</age>), the wire size contains substantial repetitive metadata.
• Complex Parsing Libraries: XML parsers are generally heavier than JSON parsers and require significant memory and CPU to parse Document Object Model (DOM) trees or navigate documents using XPath.

Protocol Buffers, developed by Google, is a binary-based, schema-first serialization format. Instead of repeating keys inside the payload, Protobuf relies on a pre-defined schema file (.proto) compiling to language-specific parser code.

The Protobuf Binary Wire Format

Unlike JSON and XML, which store keys as readable text, Protobuf omits text keys entirely. Instead, each field in a .proto schema is assigned a unique integer field number (tag):

message User {
  string name = 1;
  int32 age = 2;
}

When serialized, Protobuf writes only the field number, the wire type (describing how the data should be read), and the value payload.

For example, to represent {"age": 30}, Protobuf outputs two bytes:

1. 0x10: The key byte, calculated by bit-shifting the field number left by 3 and applying a bitwise OR with the wire type: (2 << 3) | 0 = 16 (which is 0x10 in hex).
2. 0x1E: The varint value of 30 (0x1E in hex).

This eliminates all field name text string overhead, drastically reducing payload sizes.

To maximize space savings, Protobuf uses Varints (variable-width integers) to serialize numeric types.

In standard memory representations, an int32 always occupies 4 bytes, even if the value is a small number like 5. Protobuf Varints represent integers using only the number of bytes necessary:

• Each byte contains 8 bits, but only the lower 7 bits are used to store the actual numeric value.
• The Most Significant Bit (MSB), the 8th bit, acts as a continuation bit.
• If the MSB is set to 1, it indicates that another byte follows to complete the integer.
• If the MSB is set to 0, it indicates that this byte is the final byte of the integer.

This variable-width scheme allows numbers between 0 and 127 to be serialized in a single byte, whereas larger numbers scale up to 5 bytes, saving substantial network bandwidth in systems dominated by low ID numbers or small counters.

Managing schema evolutions is critical for distributed systems where different versions of applications are deployed concurrently:

• Forward Compatibility: Older application binaries can parse incoming payloads generated by newer service binaries. In Protobuf, the parser achieves this by identifying unknown field numbers and skipping their values without throwing validation errors.
• Backward Compatibility: Newer application binaries can parse payloads generated by older service binaries. This requires new fields to be optional or set to sensible default values.
• JSON/XML Compatibility: Text formats handle schema changes easily because fields are resolved by name. If a client receives a JSON key it does not recognize, it simply ignores the property. However, renaming fields or changing data types (e.g. converting a string to a list) breaks compatibility across all formats, requiring coordination.

When designing backend architectures, select formats based on the specific performance profile of your system:

For high-throughput internal microservice communication where CPU performance and network interface bandwidth are bottlenecks, choose Protocol Buffers. For public-facing endpoints where ease of client integration and manual debugging are primary, JSON remains the industry standard.

• Protocol Buffers Encoding Specification — Google's official guide to Varint encoding, wire types, and message packing structures.
• RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format — Standard specification outlining grammar, types, and text validation rules.
• XML Schema Definition (XSD) reference — W3C official reference guidelines on defining XML structural schemas.