How to Use Maps in Protobuf

1. Introduction

Protocol Buffers (protobuf) offer a fast and efficient way to serialize structured data. They’re a compact, high-performance alternative to JSON.

Unlike JSON, which is text-based and needs parsing, protobuf generates optimized code for multiple languages. This makes it easier to send structured data between different systems.

With protobuf, we define the data structure once in a .proto file. Then, we use the generated code to handle data transmission across streams and platforms. They’re ideal when dealing with typed, structured data—especially if the payload is just a few megabytes.

Protobuf supports common types like strings, integers, booleans, and floats. They also work well with lists and maps, making complex data easy to manage. In this tutorial, we’ll learn how to use maps in protobuf.

2. Understanding Maps in Protobuf

Let’s explore how we can define and use maps as part of a protobuf message.

2.1. What Are Maps?

A map is a key-value data structure, similar to a dictionary.

Each key links to a specific value, which makes lookups fast and efficient. We can think of a DNS system as an analogy: each domain name points to an IP address. Maps work in a similar way.

2.2. Syntax for Defining Maps

Protobuf 3 supports maps out of the box.

Here’s a simple example:

message Dictionary {
    map<string, string> pairs = 1;
}

map<key_type, value_type> defines the field. The key must be a scalar type like string, int32, or bool. The value can be any valid protobuf type – scalar, enum, or even another message.

3. Implementing Maps in Protobuf

Now that we’ve explored the benefits of using protobuf, let’s put theory into practice by building a food delivery system where each restaurant has its own menu.

3.1. Setting up Protobuf in Our Codebase

Before defining the message structure, the Protoc compiler must be integrated into the build lifecycle. This can be achieved by configuring the protobuf-maven-plugin in the project’s pom.xml file. By doing so, the protocol buffer definitions are automatically compiled into Java classes during the Maven build process.

Let’s add the plugin configuration to the build section of the pom.xml file:

<build>
    <plugins>
        <plugin>
              <groupId>org.xolstice.maven.plugins</groupId>
              <artifactId>protobuf-maven-plugin</artifactId>
              <version>0.6.1</version>
            <configuration>
                <protoSourceRoot>${project.basedir}/src/main/proto</protoSourceRoot>
                <protocArtifact>com.google.protobuf:protoc:4.30.2:exe:${os.detected.classifier}</protocArtifact>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>compile</goal>
                        <goal>test-compile</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

In addition to the compiler, the protocol buffers runtime is also required. Let’s add its dependency to the Maven POM file:

<dependency>
    <groupId>com.google.protobuf</groupId>
    <artifactId>protobuf-java</artifactId>
    <version>4.30.2</version>
</dependency>

We may use another version of the runtime, provided that it’s the same as the compiler’s version.

3.2. Defining a Message With Map Field

Let’s start by defining a simple protobuf message that includes a map. Here, we create a protobuf schema where the restaurants map stores a restaurant name as a key and its menu as a value. The menu itself is another map, mapping food items to their price:

syntax = "proto3"
message Menu {
    map<string, float> items = 1;
}

message FoodDelivery {
    map<string, Menu> restaurants = 1;
}

3.3. Populating the Map

Now that we’ve defined a map in our schema, we need to populate it with data in our code.

The map<k, v> structure in protobuf behaves like a Java HashMap, allowing us to store key-value pairs efficiently.

In our case, we use a map<string, Menu> to store restaurant names as keys and their corresponding menu items as values:

Food.Menu pizzaMenu = Food.Menu.newBuilder()
  .putItems("Margherita", 12.99f)
  .putItems("Pepperoni", 14.99f)
  .build();

Food.Menu sushiMenu = Food.Menu.newBuilder()
  .putItems("Salmon Roll", 10.50f)
  .putItems("Tuna Roll", 12.33f)
  .build();

Initially, we define the menu for the restaurants. Then we populate our map with the restaurant names and their respective menus too:

Food.FoodDelivery.Builder foodData = Food.FoodDelivery.newBuilder();

We start by creating an instance of the map. Next, we’ll simply put the restaurants in place and build the map:

foodData.putRestaurants("Pizza Place", pizzaMenu);
foodData.putRestaurants("Sushi Place", sushiMenu);

return foodData.build();

4. Storing and Retrieving Data From a Binary File

Next, we’ll write our protobuf map data to a binary file – a process known as serialization. This ensures efficient storage and easy transmission. And, of course, we’ll also read it back by deserializing the field.

4.1. Serializing the Protobuf Map to a Binary File

Serialization converts our structured data into a compact binary format, making it lightweight and fast to store or send over a network. Let’s see how we can implement this.

We start by defining a file path for the file where we’ll be writing this data:

private final String FILE_PATH = "src/main/resources/foodfile.bin";

Then, we’ll write the logic to serialize the file:

public void serializeToFile(Food.FoodDelivery delivery) {
    try (FileOutputStream fos = new FileOutputStream(FILE_PATH)) {
        delivery.writeTo(fos);
        logger.info("Successfully wrote to the file.");
    } catch (IOException ioe) {
        logger.warning("Error serializing the Map or writing the file");
    }
}

The generated source files allow direct writing to an output stream.

4.2. Deserializing the Binary File to a Protobuf Map

Now, let’s deserialize the binary file back into a Protobuf map. We’ll start by opening an input stream and use the methods generated by the Protobuf compiler to parse the stored data:

public Food.FoodDelivery deserializeFromFile(Food.FoodDelivery delivery) {
    try (FileInputStream fis = new FileInputStream(FILE_PATH)) {
        return Food.FoodDelivery.parseFrom(fis);
    } catch (FileNotFoundException e) {
        logger.severe(String.format("File not found: %s location", FILE_PATH));
        return Food.FoodDelivery.newBuilder().build();
    } catch (IOException e) {
        logger.warning(String.format("Error reading file: %s location", FILE_PATH));
        return Food.FoodDelivery.newBuilder().build();
    }
}

We open a file input stream and pass it on to the parseFrom() method, which reconstructs the protobuf object. If the file is missing or is empty, then we log the issue.

4.3. Displaying the Results After Deserialization

Now that we’ve deserialized the data, we can proceed with displaying the deserialized results:

public void displayRestaurants(Food.FoodDelivery delivery) {
    Map<String, Food.Menu> restaurants = delivery.getRestaurantsMap();
    for (Map.Entry<String, Food.Menu> restaurant : restaurants.entrySet()) {
        logger.info(String.format("Restaurant: %s", restaurant.getKey()));
        restaurant.getValue()
          .getItemsMap()
          .forEach((menuItem, price) -> logger.info(String.format(" - %s costs $ %.2f", menuItem, price)));
    }
}

Here, we display the stored data. Since our map holds all the restaurant names as keys and their respective menus as values, we can simply iterate through the data and log each restaurant along with its menu items and prices.

5. Testing Our Implementation

To ensure our Protobuf map operations work correctly, we verify that the serialization correctly writes data to the file and confirm that the deserialization restores the original data correctly.

We also need to capture the log output and check if the expected data is logged. So we begin with verifying that the serialization happens correctly:

@Test
void givenProtobufObject_whenSerializeToFile_thenFileShouldExist() {
    foodDelivery.serializeToFile(testData);
    File file = new File(FILE_PATH);
    assertTrue(file.exists(), "Serialized file should exist");
}

Once we’re done with verifying the serialization, let’s see how we can test whether deserialization happens:

@Test
void givenSerializedFile_whenDeserialize_thenShouldMatchOriginalData() {
    foodDelivery.serializeToFile(testData);
    Food.FoodDelivery deserializedData = foodDelivery.deserializeFromFile(testData);
    assertEquals(testData.getRestaurantsMap(), deserializedData.getRestaurantsMap(), "Deserialized data should match the original data");
}

Here we first serialize the file and then check if the testData map is equal to the deserializedData‘s map. After this, let’s verify whether we get the data logged correctly:

@Test
void givenDeserializedObject_whenDisplayRestaurants_thenShouldLogCorrectOutput() {
    foodDelivery.serializeToFile(testData);
    Food.FoodDelivery deserializedData = foodDelivery.deserializeFromFile(testData);
    Logger logger = Logger.getLogger(FoodDelivery.class.getName());
    TestLogHandler testHandler = new TestLogHandler();
    logger.addHandler(testHandler);
    logger.setUseParentHandlers(false);
    foodDelivery.displayRestaurants(deserializedData);
    List<String> logs = testHandler.getLogs();
    assertTrue(logs.stream().anyMatch(log -> log.contains("Restaurant: Pizza Place")),
      "Log should contain 'Restaurant: Pizza Place'");
    assertTrue(logs.stream().anyMatch(log -> log.contains("Margherita costs $ 12.99")),
      "Log should contain 'Margherita costs $ 12.99'");
}

To verify whether our application logs expected messages during execution, we need a way to capture and inspect log output programmatically. The TestLogHandler helps us do exactly that by extending Java’s Handler:

static class TestLogHandler extends Handler {
    private final List<String> logMessages = new ArrayList<>();

    @Override
    public void publish(LogRecord record) {
        if (record.getLevel().intValue() >= Level.INFO.intValue()) {
            logMessages.add(record.getMessage());
        }
    }

    @Override
    public void flush() {
    }

    @Override
    public void close() throws SecurityException {
    }

    public List<String> getLogs() {
        return logMessages;
    }
}

It has a list of log messages where we push in each and every LogRecord that has a level greater than or equal to the level of INFO log. We store it in a list since it helps keep the order of the logs as they appear in the console.

6. Conclusion

Using maps in Protobuf provides a structured and efficient way to manage key-value relationships in our data models. In this article, we explored how to define, serialize, and deserialize Protobuf maps in Java, ensuring that our data remains compact, readable, and easily transferable. By implementing robust unit tests, we verified that our serialization and deserialization processes function correctly, maintaining data integrity.

With the right Maven setup and best practices in place, we can now confidently integrate Protobuf maps into our applications, leveraging their performance benefits while keeping our codebase clean and maintainable.

The code associated with this article is available over on GitHub.