Home Assistant with Multicast in Kubernetes with Multus

I recently changed my home server from TrueNAS Core to TrueNAS Scale, which means I've been rebuilding/migrating my home infrastructure from the FreeBSD jails based system used in Core to Kubernetes on Scale. One of the trickier services to change over was Home Assistant (a fantastic piece of home automation software) as I wanted to use multicast networking to add HomeKit and Apple TV integration.

The Home Assistant container installation instructions recommend using host networking, as some of the integrations (including some that I use a lot) use multicast networking, and configuring the container to properly expose the necessary ports can be difficult. While it's possible to use host networking with Kubernetes, and the Helm chart I'm using makes it easier, I noticed that iXSystems has also bundled in Multus, a way to "[attach] multiple network interfaces to pods in Kubernetes". I'd already ran across a few references to Multus when searching for a way to get multicast working on Kubernetes, so I tried it out, and it works great. Here's a brief overview of the configuration I'm using.

First there's a NetworkAttachmentDefinition which defines the network interface the pod will use. vlan60 is the name of the parent interface (in my case, a VLAN virtual interface) that exists on the host node. In my case vlan60 also has the k8s node IP address as well, 192.0.2.4. I've also statically allocated an IP address for Home Assistant (192.0.2.10 in the example). Now apply it with kubectl -n home-assistant apply -f home-assistant-multus.yaml (changing -n home-assistant to whatever namespace you're using).

# filename: home-assistant-multus.yaml
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: home-assistant-netif
spec:
  config: '{
            "cniVersion": "0.3.0",
            "type": "macvlan",
            "master": "vlan60",
            "mode": "bridge",
            "ipam": {
                "type": "static",
                "addresses": [
                    {
                        "address": "192.0.2.10/24",
                        "gateway": "192.0.2.1"
                    }
                ],
                "dns": {
                    "nameservers": ["198.51.100.2"],
                    "domain": "svc.paxswill.example.com",
                    "search": []
                }
            }
        }'

Using a host-local IPAM configuration would also probably work, and if I was exposing multiple containers to the network I would probably use that instead of the static configuration above. For more details see the CNI IPAM documentation.

The second piece of configuration is an annotation on the Home Assistant pod, k8s.v1.cni.cncf.io/networks: home-assistant-netif. When using the k8s-at-home Helm chart from above, that annotation is added to the podAnnotations object.

After applying both Kubernetes manifests above, wait for Home Assistant to come back up. Assuming you already had access to the web interface, navigate to Configuration -> General and scroll down to the Network section (the network section will only be visible if your user has "Advanced Mode" enabled). Change the settings so that only the new network interface is checked and restart Home Assistant, and that's it, my Home Assistant installation now has direct network access so it can use the multicast-based integrations withi minimal fuss.