ESPHome protocol

I'm a huge fan of Homeassistant and home automation in general. The whole ecosystem with offline capabilities and privacy works great. Having multiple devices flashed with the aftermarket ESPHome firmware I got curious how these devices communicate with Homeassistant.

This blog post is about my journey discovering the ESPHome protocol and implementing a rust library for implementing native ESPHome devices in no_std environments.

The beginning

I tried to find documentation about how the whole process works, but since ESPHome is developed in close coordination with Homeassistant the documentation is quite sparse 🏜️. Luckily everything in the stack is open source, and only a few git repositories away.

First Observations

Homeassistant is mostly written in python and uses the ESPhome provided aioesphomeapi library to do it's thing.

While ESPHome uses C++ for the api implementation

• The protocol uses protobuf for message encoding (Great!) and seems to be RPC Style
• It uses TCP and optionally encrypts the stream using the noise protocol
• Connections are initiated from the Homeassistant side
• Devices are auto discovered using mDNS SRV lookups

Message Layout

All messages are built the same:

• Start with a Zero byte
• protobuf VarInt size of the message
• protobuf VarInt type of the message
• protobuf encoded message object matching type

For example the hello message definition looks like this:

message HelloRequest {
  option (id) = 1;
  option (source) = SOURCE_CLIENT;
  option (no_delay) = true;

  // Description of client (like User Agent)
  // For example "Home Assistant"
  // Not strictly necessary to send but nice for debugging
  // purposes.
  string client_info = 1;
  uint32 api_version_major = 2;
  uint32 api_version_minor = 3;
}

option (id) = 1; specifies that the message type is 1. option (source) = SOURCE_CLIENT; tells us that this message can only be sent by the client (Homeassistant). option (no_delay) = true; is true for almost all messages, I'm ignoring it for now. The rest is the definition for the message object.

Creating a rust library

Key goals with the rust library are:

• no_std compatible (I want to create my own ESPHome lookalike device)
• as fully typed as possible
• ergonomic API

First step is selecting a protobuf framework from the ones available on crates.io

I chose prost, because it is the most popular, has no_std support and a very nice build integration.

I created a new library cargo new --lib esphome-proto and added the proto files from ESPhome to the protos subfolder.

the build integration in build.rs is quite minimal:

let protos = ["protos/api.proto", "protos/api_options.proto"];
let includes = ["protos"];

prost_build::Config::new()
    .default_package_filename("api")
    .compile_protos(&protos, &includes)
    .unwrap();

The manual way

With the object decoder in place I started on decoding my first messages. The implementation would work like this:

1. Check if there is a Zero byte at the start of the buffer
2. Decode the size field
3. Decode the type field
4. Decode the payload using the correct object type based on the type field

Something like this:

pub fn decode(buffer: &mut &[u8]) -> Result<Header> {
    if buffer.is_empty() {
        return Err(Error::ShortBuffer);
    }

    if buffer.get_u8() != 0 {
        return Err(Error::InvalidStartByte);
    }

    let size = prost::encoding::decode_varint(buffer)?;
    let kind = prost::encoding::decode_varint(buffer)?;

    match kind {
        1 => todo!("Decode hello request into Header")
        3 => todo!("Decode connect request into Header")
        _ => todo!("Error unknown request")
    }
}

This manual process of matching Message types and objects is super boring and error prone, so I had to come up with a solution.

Parsing harder

Luckily for me the ESPHome developers included the type into option fields in the proto definition. But how can I access those? Prost does not support this directly...

Looking around crates.io I found the wonderful protobuf_parse which allowed me to parse the protobuf file in build.rs and in combination with some templating allowed me to generate additional code to match the message type to the message object.

If you want to see the details take a look here.

With the auto generated matching working after some iterations we arrived at the quite nice decode function:

pub fn decode(buffer: &mut &[u8]) -> Result<Self> {
    if buffer.is_empty() {
        return Err(Error::ShortBuffer);
    }

    if buffer.get_u8() != 0 {
        return Err(Error::InvalidStartByte);
    }

    let size = prost::encoding::decode_varint(buffer)?;
    let kind = prost::encoding::decode_varint(buffer)?;

    Ok(Header {
        _type: api::MessageType::from_u64(kind).ok_or(Error::UnknownMessageType(kind))?,
        size,
    })
}

No manual mapping of types! 100% Implementation for all messages in the proto 🎉

Wrapping up

With this in place I created some higher level methods to decode and encode whole messages and created a std based sample acting as a binary sensor. You can have a look at the whole crate on codeberg