Canonical Format¶
Every SDMX structure in sdmx-core can write itself as a canonical YAML document. The format is called
fusion-yaml, and every document names its version on the first line:
format: "fusion-yaml(1.0.0)"
ref: "Codelist=SDMX:CL_FREQ(2.1)"
name: "Code list for Frequency (FREQ)"
items:
- A: "Annual"
- M: "Monthly"
- Q: "Quarterly"
The canonical format covers every maintainable artefact: codelists, concept schemes, data structures, dataflows, constraints, maps, VTL schemes, custom structures, and reference metadata sets. Each artefact has its own page in this section, listed alphabetically in the sidebar. The General Rules page describes the conventions that all of them share.
Why a canonical format?¶
SDMX already has several exchange formats (SDMX-ML, SDMX-JSON, Fusion-JSON). They are designed for exchange, and each one allows the same structure to be written in more than one way: elements can come in a different order, optional defaults may or may not be present, and whitespace and namespaces vary. Two files can therefore describe exactly the same structure and still differ byte for byte.
The canonical format has a different goal. It gives each structure exactly one textual form:
- Deterministic. The same structure always produces the same bytes. Fields are written in a fixed order, collections are sorted where SDMX gives them no order, and empty or default values are left out.
- Lossless. Every property of the structure is written. Reading the document back produces a structure that is
equal to the original (
deepEquals), and writing that structure again produces the same document. - Self-contained. A structure writes itself. No module needs to be registered, and nothing is looked up in the
SingletonStoreorFusionBeanStore. A structure can always produce its canonical form and its checksum. - Independent of the SDMX version. The format holds the properties of every SDMX version, both the current ones
and those from previous versions, in one model. A structure read from an SDMX 2.0 or 2.1 file keeps its version
specific properties (for example the measure dimension of a data structure, or a 2.1
structure set), and a structure
read from an SDMX 3.x file keeps its own (for example links and annotation values). The version in
fusion-yaml(1.0.0)is the version of the canonical format, not of SDMX, so a 1.0.0 document is not pinned to a specific SDMX version. This is the same approach as the Fusion-JSON format. - Readable. YAML is easy for people to read and compare. A value is written in its simplest form and only becomes an object when it carries more information (see Simple and complex values). A canonical document is therefore also a convenient way to review or diff a structure.
Caching and change detection¶
Because the canonical document is deterministic and lossless, its checksum is a fingerprint of the structure's content: the same content always has the same checksum, and any change gives a new one. This makes the checksum a natural key for caching and for detecting change.
- Content-addressed storage. A structure can be stored under its checksum instead of its URN. Deferred beans do this: when a structure is offloaded from memory it is written to the file system using its checksum as the file name. Identical content is only stored once, and a file never needs to be overwritten, because changed content is a different file.
- Versioning content behind a stable URN. In SDMX a structure can change while its URN stays the same, for example
a non-final structure, a
1.0version that is edited in place, or a wildcarded reference such as1.0+that resolves to a newer version over time. The URN alone cannot tell a cache whether what it holds is still current. A pointer that records the URN and the checksum can: if the checksum of the current structure differs, the content has changed and anything derived from it (a cached response, a super bean, a search index entry, a validation result) must be refreshed. If the checksum is the same, the cached result is still valid, however many times the structure was re-submitted. - Knowing the state of a whole system. Checksums can be combined into a Merkle tree. For example, hash the sorted checksums of all the structures of one type and agency, then hash those hashes per type, then once more for the whole system. The root hash describes the state of every structure in the system in a single value:
- two systems (for example a registry and its replica, or a server and a client cache) are in sync if their root hashes are equal;
- if they differ, comparing the hashes level by level finds the changed structures without transferring or comparing any structure that has not changed;
- recording the root hash over time gives a cheap way to tell whether anything in the system has changed since a given point.
In this section¶
- General Rules: the conventions shared by all artefacts. Read this first.
- One page per artefact, in alphabetical order. Each page shows the artefact in its simplest complete form, then a more complex form where simple values become objects, followed by a reference of every field.