Protobufs 101 (Part 2)

In Part 1 we explored the basics of Protobuf wire formats. Now let’s move on to some slightly more advanced concepts:

• Repeated Fields
• Packed Repeated Fields
• Maps
• (Next: Negative Numbers)

Repeated Fields

Proto Definition

message FruitBasket {
  repeated string fruits = 1;
}

How It Works

In repeated fields, each value is encoded as a separate tag-value pair, even though they share the same field number.

Example

fruits: ["Apple", "Banana"]

This results in two tag-value pairs, both with field number 1.

Encoded Output (Hex)

0a 05 41 70 70 6c 65 0a 06 42 61 6e 61 6e 61

Breakdown

• 0a → Tag (field 1, length-delimited)

• 05 → Length of "Apple"

• 41 70 70 6c 65 → "Apple"

• 0a → Tag again (same field)

• 06 → Length of "Banana"

• 42 61 6e 61 6e 61 → "Banana"

First 7 bytes → Apple
Next 8 bytes → Banana

Packed Repeated Fields

Packed repeated fields are an optimization used for numeric types.

Instead of repeating the tag for every value, all values are grouped together.

Proto Definition

message FruitCounts {
  repeated int32 values = 1 [packed = true];
}

Encoding Structure

A packed repeated field is encoded as:

1. Tag
2. Length (total bytes of all values combined)
3. Concatenated varint-encoded values

Example

values: [3, 270, 86942]

Encoded Output (Hex)

0a 06 03 8e 02 9e a7 05

Breakdown

• 0a → Tag (field 1, length-delimited)
• 06 → Total byte length
• 03 → Varint(3)
• 8e 02 → Varint(270)
• 9e a7 05 → Varint(86942)

Key Insight

• Tag appears once
• Values are packed together

This reduces size significantly for large numeric arrays

Maps

Maps in Protobuf are actually syntactic sugar. Internally, they are encoded as repeated key-value message entries.

Proto Definition

message FruitInventory {
  map<string, int32> stock = 1;
}

What It Becomes Internally

Protobuf converts it into something like:

message FruitInventory {
  repeated Entry stock = 1;
}

message Entry {
  string key = 1;
  int32 value = 2;
}

Example

stock: {
  "apple": 3,
  "banana": 5
}

Encoded Output (Conceptual)

Each map entry is encoded as a separate embedded message:

Entry 1:
  key = "apple"
  value = 3

Entry 2:
  key = "banana"
  value = 5

So it behaves like:

stock: [
  { key: "apple", value: 3 },
  { key: "banana", value: 5 }
]

Key Insight

• Maps = repeated embedded messages
• Each entry has:
  • field 1 → key
  • field 2 → value
• Order is not guaranteed

Next up: Negative Numbers