Replies: 6 comments 22 replies
|
This is my current spec draft: Homie 5 Metadata Overlay ConventionDraft SpecificationPurpose: Add deployment and semantic metadata to Homie v5 devices in a structured, non-intrusive way. Table of Contents
1. IntroductionThe Homie convention defines a standardized MQTT topic and payload structure that enables IoT devices to publish their presence and capabilities for automatic discovery and interaction. As defined in the Homie core specification, topics consist of levels with lowercase identifiers and reserved This document defines a metadata overlay convention (the “overlay”) that runs alongside Homie v5 without modifying or interfering with device-owned topic trees. It enables logically separate metadata (such as room assignment, user-friendly names, and grouping) to be published independently of the device publisher, while retaining compatibility with standard Homie controllers. This specification makes no changes to Homie’s core topology rules or payload encoding (UTF-8 strings, retained messages, reserved topic characters). ([homieiot.github.io][2]) 2. Terminology
3. Topic HierarchyMetadata overlay topics live under the reserved segment: Where:
No overlay provider may write into 4. Retained Message SemanticsAll overlay documents MUST be published as retained messages so that controllers can discover metadata consistently after reconnects, similar to how Homie devices use retained messaging. ([homieiot.github.io][1]) To delete an overlay document, a provider MUST publish a retained message with an empty payload to the corresponding topic. 5. Provider Descriptor FormatProviders MUST also publish a descriptor message at: 5.1 Descriptor Message Encoding
5.2 Descriptor ContentEach descriptor MUST contain at least:
The descriptor MAY include:
6. Device Metadata Document FormatEach provider may publish metadata for a specific Homie device: 6.1 Document Encoding
6.2 Required Field
6.3 Optional Top-Level MembersThe overlay JSON MAY include:
6.4 Device-Level MetadataThe
6.5 Node-Level MetadataThe
6.6 Property MetadataWithin a node’s
The
Implementations MUST NOT assume any semantics for keys not defined in this specification. 6.7 Example{
"schema": 1,
"device": {
"name": "Bedroom Window Left",
"room": "bedroom",
"groups": ["windows"],
"tags": ["zigbee"]
},
"nodes": {
"contact": {
"name": "Window Contact",
"properties": {
"state": {
"name": "State",
"ui": { "icon": "mdi:window-closed" }
}
}
}
}
}7. Consumer Behavior7.1 Provider DiscoveryControllers that wish to discover available metadata providers SHOULD subscribe to: The 7.2 Provider SelectionBecause this convention does not define merging across providers, a controller MUST treat providers’ metadata independently. Controllers MAY allow users to configure which provider(s) they use. Controllers SHOULD ignore keys they do not understand. 8. Access ControlIn broker configurations, access control rules SHOULD ensure:
9. VersioningOverlay documents include a Alignment With Homie ConventionsThis overlay convention:
|
|
I think that would require a list of predefined topics, similar to the units in the Homie spec. Could be prefixed with a $ to make it a "homie assigned" one. Any other keys are free to use, but should be namespaced then to be safe (similar to other Homie concepts). |
|
I gave it a go (with some Claude help); see #343 |
Well besides beeing named $meta, this has nothing to do with what I have posted - so I see no point in discussing this then... |
|
That was actually on purpose. Apologies if that wasn't clear. To facilitate the discussion and see the trade offs;
I think these tradeoffs are what we need to discuss. Maybe @dcj or @jacoscaz have other things to consider? |
|
It would be unwise to express a preference without at least attempting to implemented both of these proposals. Instinctively, however, I prefer the approach of keeping metadata
Yes, this could lead to conflicts due to multiple metadata providers using the same keys. However, I find this preferable over forcing every metadata use case to rely on a persistent service whose reliability requirements match those of the MQTT broker. I'd much prefer implementing conflict-resolution within metadata consumers (the dumb servers, smart clients paradigm). |
|
@schaze
|
|
no experience in this modelling (thought about it but no ideas yet). I do know that OpenEMS avoids hierarchy explicitly because the things never add up anyway. |
|
It is possible, and even likely, that any discussion about hierarchy is/should-be a seperate discussion topic, and if so, happy to move it elsewhere, but I currently think there is some relationship between hierarchy and metadata (IMO hierarchy is something that likely will be specified as metadata), so I wanted to mention the overall concept here. But working out the details of how hiearchy is specified is a task onto itself! |
|
@dcj : Sorry, for the late reply. I tried to answer all your question as best as I can:
I did not define that yet - the above is still a "work in progress" and represents a properly worded form of my current thoughts around the topic.
fixed - interesting though as I had it written up by AI - I guess they can make such mistakes as well
See my answer further down in Tieskes post (#342 (comment)). Regarding device published metadata: using the device id would lead to a topic like this: homie/5/$meta/device-id/device-id. Looks a bit strange. I would prefer
Yes this is basically what the old v4 metadata extension does. see here: https://github.com/homieiot/convention/blob/develop/extensions/documents/homie_meta_extension.md
Here I am completely open. These were just first examples. Question would also be if we even include these in the extension spec. IMHO the extension should first define the structure and design on the mqtt level. What metadata we publish with that could be something separate maybe even part of the node/device profile spec discussion - it would be pure semantics. @Tieske / @jacoscaz what do you think about this?
Do you mean what service or how we define the semantics? For the later see previous paragraph for the first I would say there are various solutions.
Thanks for the support! :)
I am not sure I understand what you mean by that. |
I agree with @schaze here. The list of (well defined) keys for meta data can be an annex/appendix that is more dynamic than the extension itself. Profiles could extend that list on a case-by-case basis. |
|
I doubt the following has much to do with this metadata proposal: Yes, most of us love JSON, it is easy and useful. But when I look at a JSON object, the semantics and/or true data type of many strings/numbers is not defined. Examples include:
But none of this is explict, I shouldn't have to guess, and why should I have to manally write code that converts/parses a JSON blob into a dictionary/hash-map in my language of choice, converting string UUIDs to actual UUID objects, and timestamps to DateTime objects? One interesting approach to contemplate is tagged literals, one example is how EDN does it. Just saying.... |
|
@schaze Nit: the first of your two documents above lacks a "title" |
Some questions on single or multiple documents/providers:What is the use case for multiple documents, how would this work in practice? A web-ui needs metadata and hence implements a metadata-provider. So does a mobile-app. So there are two documents, does the user have to choose where he/she enters metadata? (I assume metadata is mostly human provided/driven, is that a fair assumption?) Is this use case in line with your thoughts @schaze @dcj ? I think there should be a shared understanding of what the data means. If the meaning is shared then why put it in multiple documents with duplication/inconsistency risks? Is the metadata only targeted at controllers, or also devices? because this choice has an impact on resource usage for small devices. PS. @dcj not responding to your more detailed questions/comments, as I think we need to decide on this fundamental item first |
It is not obvious to me that a metadata-consumer will ALWAYS be the metadata-provider, in other words, that should not be a requirement. I can imagine seperate providers for all kinds of things. @Tieske raises a valid concern, how does a metadata consumer establish/obtain the unified, and conflict-resolved, complete metadata of a device? I don't have an immediate answer to that, maybe the consumer would need to implement is own (multiple?) inheritance scheme, one historic example of this is CLOS
My reference to CLOS is in the vein of: joking, but maybe not joking... :-) |
|
My thoughts were the following: In my mind then there could be various sources for metadata:
Metadata is relevant/used on a controller side (I anyways haven't seen any device <-> device use case yet anywhere). A controller can merge these documents in one or address them separately. That is a point I haven't figured out yet. (two room definitions from 2 providers could be merge into an array) |
|
@dcj wrote:
Can you elaborate on that? what providers for what data? (not trying to be a pita just making sure we have the use cases) @schaze wrote:
Is this still user provided data? (so it describes a transport mechanism, but it is user provided?) As for the json-blob topics; I think we all agree that any such topic must have a single owner, since anything else will lead to data-loss/overwrites etc. |
|
@Tieske wrote:
In my specific example here yes. Background: |
|
When evaluating whether to introduce an additional persistent service into the Homie architecture, we should consider the reliability requirements that this service would have to match. The proposal of having a single metadata provider/manager would effectively introduce a new service within the Homie architecture that would have to match the MQTT broker in terms of both its availability and its ability to scale. IMHO, this is not a good idea.
|
|
can you elaborate on the nr. 1 above; creating a secondary source of truth? I don't understand how a single instance could cause multiple sources of truth, whereas multiple overlapping documents would not? If anything, then nr. 4 and nr.1 seem contradicting to me. The nr. 1. says there needs to be a single source of truth, whereas 4 means that interpretation of the result of multiple documents might differ based on clients, which results in multiple sources of truth (client-side) depending on the strategy used. |
|
Let me try to clarify my thoughts; apologies for the confusion. I am speaking primarily from an operational standpoint. Right now, the sole source of truth is the MQTT broker. Therefore, the MQTT broker is the only component of a Homie deployment subject to stringent high-availability requirements. Or as stringent as reasonable in the given environment. If we were to introduce a service tasked with metadata reconciliation for other services to "call" over MQTT using Obviously this is already the case with the MQTT broker, without which nothing can work. My argument is that adding a second service with basically the same availability requirements as the broker itself imposes too great an operational onus on implementors and users, discouraging adoption. Given the above, I do not see point 4. as a contradiction. The tradeoff of maintaining a single source of truth requires that "truth" be allowed to contain conflicting metadata. However, as the collection of everything published by metadata providers, it would still be "true". Not only that but the dynamic of how metadata is published and consumed would be much more similar to those of every other bit of data handled by Homie, making the spec more internally consistent and easier to grasp. |
|
I had another go at it, did some PoC implementations and tested them with my bridges to see what feels right. In practice this prooved quite flexible and easy to use. I also added handling of key collisions over multiple providers. Maybe I also move it to a PR like Tieske did. @dcj , @Tieske , @jacoscaz : I know it's a length read but I would really appreciate your input. Homie 5 Metadata Overlay ConventionDraft Specification — Schema Version 2Purpose: Add deployment and semantic metadata to Homie v5 devices in a structured, non-intrusive way using flexible key-value annotations. Table of Contents
1. IntroductionThe Homie convention defines a standardized MQTT topic and payload structure that enables IoT devices to publish their presence and capabilities for automatic discovery and interaction. As defined in the Homie core specification, topics consist of levels with lowercase identifiers and reserved This document defines a metadata overlay convention (the "overlay") that runs alongside Homie v5 without modifying or interfering with device-owned topic trees. It enables logically separate metadata — such as room assignment, user-friendly names, grouping, UI hints, and arbitrary deployment-specific annotations — to be published independently of the device publisher, while retaining compatibility with standard Homie controllers. Metadata is represented as annotations: free-form key-value pairs where values are either strings or lists of strings. This design avoids imposing a fixed schema and allows providers and consumers to define any annotation keys they require. This specification makes no changes to Homie's core topology rules or payload encoding (UTF-8 strings, retained messages, reserved topic characters). 2. TerminologyThe key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.
3. Topic HierarchyMetadata overlay topics live under the reserved Where:
No overlay provider SHALL write into 3.1 Relationship to Device ExtensionsThe metadata overlay convention is a namespace-level mechanism, not a device extension. Providers MUST NOT declare A single MQTT client MAY act as both a device publisher and a metadata provider (e.g., a bridge that publishes devices and also publishes metadata about them). These are separate roles: the device publisher owns 4. Publishing Requirements4.1 Quality of ServiceAll overlay documents and provider descriptors MUST be published with MQTT QoS level 1 (at least once) or higher. This ensures delivery even in the presence of transient network issues. 4.2 Retained MessagesAll overlay documents and provider descriptors MUST be published with the MQTT retained flag set to 4.3 RemovalTo remove an overlay document, a provider MUST publish a retained message with an empty payload to the corresponding topic. To remove a provider descriptor, a provider MUST publish a retained message with an empty payload to its Removal messages MUST also be published with QoS level 1 or higher. 5. Provider Descriptor FormatEach provider MUST publish a descriptor message at: 5.1 Descriptor Message EncodingThe payload MUST be UTF-8 encoded JSON. 5.2 Descriptor FieldsThe descriptor MUST contain:
5.3 Example{
"schema": 1,
"title": "deCONZ Bridge",
"description": "Metadata from the deCONZ Zigbee gateway"
}6. Device Metadata Document FormatEach provider MAY publish a metadata overlay document for a specific Homie device at: 6.1 Document EncodingThe payload MUST be UTF-8 encoded JSON. 6.2 Top-Level StructureThe overlay document is a JSON object with the following members:
No other top-level members are defined by this specification. Implementations SHOULD ignore unrecognized top-level members. 6.3 Annotation ValuesAn annotation is a key-value pair within an annotation object. Annotation keys are free-form strings. This specification does not restrict which keys providers MAY use. Annotation values MUST be one of:
No other JSON value types (numbers, booleans, objects, 6.4 Device-Level OverlayThe
No other members are defined at the device level. Implementations SHOULD ignore unrecognized members. 6.5 Node-Level OverlayWithin the
No other members are defined at the node level. Implementations SHOULD ignore unrecognized members. 6.6 Property-Level OverlayWithin a node's
No other members are defined at the property level. Implementations SHOULD ignore unrecognized members. 6.7 Structure SummaryEvery level of the metadata hierarchy follows a consistent pattern:
This uniform structure ensures that annotations and structural navigation are always clearly separated at every level. 6.8 Example{
"schema": 2,
"device": {
"annotations": {
"name": "Bedroom Window Left",
"room": "bedroom",
"groups": ["windows"],
"tags": ["zigbee"],
"icon": "mdi:window",
"manufacturer": "Aqara",
"firmware": "3.2.1"
},
"nodes": {
"contact": {
"annotations": {
"name": "Window Contact"
},
"properties": {
"state": {
"annotations": {
"name": "State",
"icon": "mdi:window-closed"
}
}
}
}
}
}
}6.9 Well-Known Annotation KeysThe following annotation keys are defined by this specification. They are not mandatory, but implementations SHOULD use them where applicable to promote interoperability across providers and consumers.
Providers MAY define additional keys beyond those listed above. Implementations MUST NOT assume any semantics for annotation keys not defined in this specification or agreed upon out-of-band. 7. Consumer Behavior7.1 Provider DiscoveryControllers that wish to discover available metadata providers SHOULD subscribe to: The single-level wildcard 7.2 Overlay DiscoveryTo receive all overlay documents from all providers, controllers SHOULD subscribe to: This subscription matches both 7.3 Provider IndependenceControllers MUST treat each provider's overlay documents independently. This convention does not define a mandatory merging strategy across providers. Controllers MAY implement a merge strategy to present a consolidated view (see §8 for a recommended approach). Controllers MAY allow users to configure which provider(s) to use. 7.4 Unknown KeysControllers SHOULD ignore annotation keys they do not recognize. Controllers MUST NOT reject an overlay document because it contains unrecognized keys. 8. Cross-Provider Annotation Merge (Informative)This section describes a recommended merge strategy for controllers that wish to present a consolidated view of annotations from multiple providers. This section is informative, not normative — implementations are free to use any merge strategy that suits their needs. 8.1 Merge RulesWhen merging annotations from multiple providers for the same entity (device, node, or property), the recommended strategy is:
8.2 ExamplesTwo string values for the same key:
Two array values for the same key:
Mixed string and array for the same key:
8.3 ScopeThe merge applies identically at every metadata level (device, node, property). For nodes and properties, merging occurs per matching identifier — only annotations for the same node ID or property ID are merged across providers. 9. Access ControlIn broker configurations, access control rules SHOULD ensure:
10. VersioningOverlay documents and provider descriptors include a The current overlay document schema version is Alignment With Homie ConventionsThis overlay convention:
|
|
I think the merge strategy is problematic because the datatype can change. 2 providers posting a string value for the same key, might end up generating an array. So where a string is expected, an array is delivered. How would a UI for example handle this if 2 providers each specify a different "name", it expects a string, how will it display an array? That said; I also do not have a solution for this, it's a hard problem. More general; aren't we over-complicating things? from the questions I asked earlier, all metadata so far is originating from the user. If that is the case, do we really expect a user to go and edit data in multiple places, if they pertain to the same Homie device? I see the theoretical problem, but have doubts on whether this is a problem in reality. @schaze iirc you had a yaml file somewhere that contains the user input for the metadata, which is then posted to the $meta topic. Do you plan to have multiple files via different services exposed, each carrying data for the same devices? Sounds like a headache to me. Can't we simplify to a single provider, with options to add multiple later? or maybe not, just accepting that we might need to do a next version if we really need multiple providers. |
|
It's not a perfect solution, but a pragmatic one. In the merged view every field can be a list. A consumer needs to be aware of that. In case of a name appearing twice you can always opt for a specific provider or merge them or pick the first. Not perfect but good enough and not complicated, in practice this will rarely be a problem. I will have multiple providers for meta data but not for the same metadata. Examples - all will be provided by a different service: Bridge Metadata - with firmware, device model and so on.../homie/5/$meta/deconzbridge/sensor-1:
{
"schema": 2,
"device": {
"annotations": {
"hc-deconz/manufacturer": "LUMI",
"hc-deconz/fw": "lumi.sensor_motion.aq2",
"hc-deconz/type": "ZHAPresence",
"hc-deconz/model-id": "lumi.sensor_motion.aq2"
}
}
}automation controller/homie/5/$meta/deconzbridge/sensor-1:
{
"schema": 2,
"device": {
"annotations": {
"hc-ctrl/part-of-group": ["alarm-interior-motion-sensors","motion-livingroom"],
}
}
}user-input metadata/homie/5/$meta/hc-meta/sensor-1:
{
"schema": 2,
"device": {
"annotations": {
"name": "Livingroom Motion Sensor Couch",
"room": "livingroom",
"groups": ["security", "motion"],
}
}
}None of that would collide in the merged view. And the described extension spec handles this easily. |
Uh oh!
There was an error while loading. Please reload this page.
A.1 Problem Statement
The Homie v5 core specification defines how devices publish:
However, many real-world deployments require additional information that:
Examples include:
These attributes represent deployment-specific metadata, not device facts.
Core Tension
There are three architectural constraints:
Writing metadata directly into
homie/5/<device-id>/...violates ownership boundaries and may introduce race conditions with device publishers.Therefore, a separate, namespaced overlay mechanism is required.
A.2 Design Goals
The metadata overlay convention is designed with the following goals:
Non-intrusive
It MUST NOT modify or interfere with device-owned topic trees.
Provider independence
Multiple metadata sources MUST be able to publish independently without coordination.
Atomicity
Metadata for a device SHOULD be representable as a single JSON document.
Discoverability
Metadata providers SHOULD be discoverable via MQTT alone.
Low topic count
The solution SHOULD avoid recreating topic explosion patterns similar to pre-v5 Homie metadata structures.
Controller simplicity
Consumers SHOULD be able to ingest metadata deterministically with minimal subscription complexity.
Semantic neutrality
The convention MUST NOT impose semantics beyond those explicitly defined.
A.3 Use Cases
A.3.1 Room Assignment
Assigning devices, nodes, or properties to a room or area.
Example:
The device firmware cannot know this; overlay metadata provides it.
A.3.2 User-Friendly Naming
Device description names may be technical or manufacturer-defined.
Overlay metadata enables:
Without altering the device publisher.
A.3.3 Logical Grouping
Controllers may want:
groups: ["windows", "security"]groups: ["downstairs", "lights"]Groups are deployment-specific and may overlap.
Overlay metadata supports such classification without modifying device structure.
A.3.4 UI Presentation Hints
UI systems may require:
These are controller concerns, not device concerns.
The overlay allows UI hints to be defined externally.
A.3.5 Automation and Rule Tagging
Rule engines may operate on tags such as:
Overlay metadata allows attaching such tags without modifying firmware or bridge logic.
A.3.6 LLM / Semantic Agent Context
Advanced controllers (including LLM-based agents) require contextual information such as:
By subscribing to overlay topics, an agent can construct a richer model of the environment without scraping multiple heterogeneous configuration sources.
A.3.7 External Imports
A provider may import metadata from:
The overlay namespace allows such imports without rewriting device publishers.
A.3.8 Manual Annotation
A user-operated CLI or dashboard may publish metadata via a
manualprovider namespace.Because providers are independent, manual annotations do not overwrite or interfere with automated providers.
A.4 Explicit Non-Goals
This convention intentionally does NOT:
These concerns are left to higher-level systems.
A.5 Architectural Rationale
The metadata overlay exists because:
The overlay mechanism therefore preserves Homie’s architectural principles while enabling practical deployment-level metadata.
All reactions