Skip to content

Data Structure Definition

A Data Structure Definition (DSD) describes the shape of a dataset: its dimensions, groups, attributes, measures and any metadata attributes it uses. The document starts with ref: "DataStructure=AGENCY:ID(1.0)".

The writer is DataStructureCanonicalWriter.

Simple form

The simplest complete DSD has one concept scheme, dimensions and measures. Every component takes its id from its concept and has no representation, so each one is written as a plain string.

format: "fusion-yaml(1.0.0)"
ref: "DataStructure=TEST:TEST(1.0)"
name: "TEST"
conceptscheme: "TEST:CS(1.0)"
dimensions:
- "INTERNAL_CODE"
- "FREQ"
- "FOF_PERIOD"
measures:
- "OBS_VALUE"
Source: datastructure/testNoTimeSeries.yaml

The document starts with the common header. Then come these body fields:

  • conceptscheme: the short URN of the concept scheme that every component's concept belongs to. If the components use more than one concept scheme, conceptschemes is written instead (see Multiple concept schemes).
  • dimensions: the dimensions in key order. Each one is keyed by its concept id (components).
  • measures: the measures, in definition order.

Complex form

Dimensions and measures are written as a string when they have nothing to record except their concept. They become a single-key object, keyed by the concept, as soon as they have any of the following:

Component It becomes an object when it has
Dimension an id that is not its concept id, a uri, annotations, links, a representation (not for the time dimension), concept roles, or a measure dimension concept scheme
Measure an id that is not its concept id, a uri, annotations, links, a representation, or a mandatory usage (required)

A text format counts as a representation even when it has no fields set.

Attributes and metadata attributes are always written as objects. Every attribute has a relationship to record (see Attribute relationship). The relationship of a metadata attribute usage is optional, and a usage without one is written as an empty object (see Metadata attributes).

Representation

A dimension, attribute or measure with an enumeration or a text format is written as an object holding its representation:

dimensions:
- FREQ:
    codelist: "TEST:CL_FREQ(1.0)"
- REF_AREA:
    codelist: "TEST:CL_REF_AREA(1.0)"
- INDICATOR:
    codelist: "TEST:CL_INDICATOR(1.0)"
- T:
    id: "TIME_PERIOD"
attributes:
- TITLE:
    textType: "String"
    maxLength: 50
    dimensions:
    - "FREQ"
    - "REF_AREA"
    - "INDICATOR"
- SOURCE:
    textType: "AlphaNumeric"
    maxLength: 5
    minOccurs: 1
    maxOccurs: 2
    measures: "OBS_VALUE"
    level: "observation"
    required: true
- UNIT:
    valuelist: "ACY:VL_UNIT(1.0)"
    measures:
    - "OBS_VALUE"
    - "BIRTHS"
    level: "observation"
measures:
- "OBS_VALUE"
- "BIRTHS"
Source: datastructure/testAttributes.yaml (trimmed)

The enumeration key is the lower-case class of the referenced structure, for example codelist or valuelist (references). maxLength, minOccurs and maxOccurs are numbers. A maxOccurs with no upper bound is written as "unbounded".

Time dimension

The writer never writes the representation of the time dimension, because it is fixed. The time dimension always has the id TIME_PERIOD, so its form depends only on its concept:

  • If the concept is TIME_PERIOD and there is no uri, annotation, link or concept role, it is written as the string "TIME_PERIOD".
  • If the concept has a different id, for example T, it is written as an object that holds the id:

dimensions:
- "FREQ"
- "REF_AREA"
- "INDICATOR"
- T:
    id: "TIME_PERIOD"
Source: datastructure/testWithMSD.yaml

The id field is written for any component whose id differs from its concept id. On the time dimension it is simply the most common case.

Concept roles

A dimension or attribute with concept roles has a conceptRole field, so a dimension with concept roles is always written as an object. Each entry is the role concept as a canonical value (Concept=AGENCY:ID(VERSION).CONCEPT_ID), because the field name does not say what type of structure is referenced. The field follows the value-or-array rule: one role is written as a string, several as an array.

dimensions:
- FREQ:
    codelist: "TEST:CL_FREQ(1.0)"
    conceptRole: "Concept=SDMX:CROSS_DOMAIN_CONCEPTS(2.0).FREQ"
- REF_AREA:
    codelist: "TEST:CL_REF_AREA(1.0)"
    conceptRole:
    - "Concept=SDMX:CROSS_DOMAIN_CONCEPTS(2.0).REF_AREA"
    - "Concept=SDMX:CROSS_DOMAIN_CONCEPTS(2.0).COUNTERPART_AREA"
- INDICATOR:
    codelist: "TEST:CL_INDICATOR(1.0)"
- T:
    id: "TIME_PERIOD"
Source: datastructure/testConceptRoles.yaml (trimmed)

On an attribute, conceptRole is written before the relationship fields:

attributes:
- TITLE:
    conceptRole: "Concept=SDMX:CROSS_DOMAIN_CONCEPTS(2.0).TITLE"
    level: "dataflow"
Source: datastructure/testAttributeConceptRoles.yaml (trimmed)

Measure dimension

An SDMX 2.x measure dimension is represented by a concept scheme. It is written as an object with a conceptscheme field that holds the short URN of that scheme:

conceptscheme: "TEST:CS(1.0)"
dimensions:
- "FREQ"
- "REF_AREA"
- INDICATOR:
    conceptscheme: "TEST:CS_INDICATORS(1.0)"
- T:
    id: "TIME_PERIOD"
measures:
- "OBS_VALUE"
Source: datastructure/testMeasureDimension.yaml

This field has the same name as the top-level conceptscheme but means something different. At the top level it says where the concepts come from. Inside a dimension it is the dimension's representation.

Groups

Groups are written in one of two forms, chosen for all groups together.

In the map form, every group is written as GROUP_ID: dimension id(s). The dimensions follow the value-or-array rule, so a group with one dimension is written as a string. This form is used when no group has a uri, annotations or links:

groups:
  GROUP_1:
  - "EXR_TYPE"
  - "EXR_SUFFIX"
  GROUP_2:
  - "CURRENCY"
  - "CURRENCY_DENOM"
Source: datastructure/testGroups.yaml

In the array form, groups becomes an array of identifiables. This form is used when any group has a uri, annotations or links. Each group is an object keyed by its id, holding uri, dimensions (a value or array), annotations and links:

groups:
- GROUP_1:
    dimensions:
    - "EXR_TYPE"
    - "EXR_SUFFIX"
    annotations:
    - id: "GRP_ANN"
- GROUP_2:
    dimensions:
    - "CURRENCY"
    - "CURRENCY_DENOM"
Constructed from the writer.

Attribute relationship

An attribute records two independent things:

  1. Where it is reported (its attachment): against the whole dataflow, against a group, against a set of dimensions (some or all of them), or against each observation.
  2. Which measures it relates to (optional): the measures the attribute value describes. This is independent of where it is reported, so an attribute can be reported once for the dataflow and still relate to only one measure.

Each attribute is written as an object with these relationship fields, in this order:

Field Value Meaning
group group id the attribute is reported against the group (attachment level Group)
dimensions dimension id(s), value or array the attribute is reported against this subset (or the full set) of dimensions (attachment level DimensionGroup)
measures measure id(s), value or array the measures the attribute relates to
level "dataflow" or "observation" the attribute is reported against the dataflow (attachment level DataSet) or each observation (attachment level Observation)

level is only written for the two levels that group and dimensions cannot express. The attachment level is therefore always explicit, and is never inferred from measures:

Fields written Attachment level
group Group
dimensions DimensionGroup
level: "dataflow" DataSet
level: "observation" (with or without dimensions) Observation

Attributes reported at each level:

attributes:
- CS2.REP_YEAR_START:
    level: "dataflow"
- CS.COMMENT_DSET:
    level: "dataflow"
    required: true
- CS2.COLLECTION:
    dimensions:
    - "FREQ"
    - "REF_AREA"
    - "INDICATOR"
- CS2.OBS_STATUS:
    measures: "OBS_VALUE"
    level: "observation"
Source: datastructure/testDSDWithMultipleConceptSchemes.yaml (trimmed)

An attribute reported against a group:

attributes:
- OBS_CONF:
    codelist: "ECB:CL_OBS_CONF(1.0)"
    measures: "OBS_VALUE"
    level: "observation"
- G1_A1:
    group: "GROUP_1"
- G1_A2:
    group: "GROUP_1"
    required: true
Source: datastructure/testGroups.yaml (trimmed)

An attribute reported once for the dataflow, but relating only to the OBS_VALUE measure:

attributes:
- DS_UNIT:
    measures: "OBS_VALUE"
    level: "dataflow"
measures:
- "OBS_VALUE"
Source: datastructure/testDatasetAttributeMeasureRelationship.yaml (trimmed)

Required

A mandatory attribute or measure writes required: true inside its own object. A conditional one omits the field (false is the default). Dimensions never write required.

Because an attribute is always an object, required never changes its form. A measure that is otherwise simple becomes an object when it is mandatory:

measures:
- OBS_VALUE:
    required: true
- "BIRTHS"
Source: datastructure/testMeasureUsage.yaml (trimmed)

Metadata attributes

Metadata attribute usages from the attribute list are written as metadataAttributes, between attributes and measureList. Each entry is an object keyed by the metadata attribute reference (an id from the DSD's metadata structure). It holds the same relationship fields as an attribute (group, dimensions, measures, level) and any annotations. Usages have no uri, links, representation or required. The metadata structure itself is referenced by metadatastructure:

metadataAttributes:
- CONTACT:
    level: "dataflow"
- SOURCE:
    dimensions:
    - "FREQ"
    - "REF_AREA"
measures:
- "OBS_VALUE"
metadatastructure: "TEST:METADATA(1.0)"
Source: datastructure/testMetadataAttributeUsage.yaml (trimmed)

The relationship of a usage is optional. A usage without one can be reported at any level of the data, and is written as an empty object, for example - CONTACT: {}.

metadatastructure holds a short URN and follows the value-or-array rule. Only one reference is possible today. It is handled as a list so that the format is ready for SDMX 3.2, which allows more than one.

Multiple concept schemes

When the components use concepts from more than one concept scheme, conceptscheme is replaced by a conceptschemes map from alias to short URN. Each component key is then written as ALIAS.CONCEPT_ID:

conceptschemes:
  CS: "ECB:CS(1.0)"
  CS1: "SDMX:CS(1.0)"
  CONCEPTS: "TEST:CONCEPTS(1.0)"
  CS2: "TEST:CS(1.0)"
dimensions:
- "CS2.FREQ"
- "CS2.REF_AREA"
- "CS1.INDICATOR"
- CONCEPTS.T:
    id: "TIME_PERIOD"
attributes:
- CS2.REP_YEAR_START:
    level: "dataflow"
- CS.COMMENT_DSET:
    level: "dataflow"
    required: true
- CS2.OBS_STATUS:
    measures: "OBS_VALUE"
    level: "observation"
measures:
- "CONCEPTS.OBS_VALUE"
Source: datastructure/testDSDWithMultipleConceptSchemes.yaml (trimmed)

The aliases are assigned like this:

  • The concept schemes are sorted by URN (agency, then id, then version), and that is the order in which the map is written.
  • Each alias is the concept scheme's id. If that alias is already taken by an earlier scheme, a number is appended: 1, 2 and so on. In the example, three schemes have the id CS. They become CS, CS1 and CS2 in sort order.

The alias appears only in component keys. id, group dimensions and the attribute relationship fields (group, dimensions, measures) all use plain component ids.

Dimension list, attribute list and measure list

The descriptors dimensionList, attributeList and measureList are written only when they have a uri, annotations or links. Each one is written just before its components. Their ids are fixed, so no id is written.

conceptscheme: "TEST:CS(1.0)"
dimensionList:
  annotations:
  - id: "DIM_LIST_ANN"
    title: "Dimension List Annotation"
dimensions:
- "FREQ"
- "REF_AREA"
- "INDICATOR"
- T:
    id: "TIME_PERIOD"
attributeList:
  annotations:
  - id: "ATTR_LIST_ANN"
    title: "Attribute List Annotation"
attributes:
- TITLE:
    dimensions:
    - "FREQ"
    - "REF_AREA"
    - "INDICATOR"
measureList:
  annotations:
  - id: "MEASURE_LIST_ANN"
    title: "Measure List Annotation"
measures:
- "OBS_VALUE"
Source: datastructure/testDescriptorAnnotations.yaml

Field reference

Top-level body fields, in written order:

Field Value Written when
conceptscheme short URN of the single concept scheme the components use exactly one concept scheme
conceptschemes map of alias to concept scheme short URN, sorted by URN the components use more than one concept scheme
dimensionList object: uri, annotations, links the dimension list has a uri, annotations or links
dimensions array of dimensions (string or object) always
groups map of group id to dimension id(s), or array of group objects the DSD has groups
attributeList object: uri, annotations, links the attribute list has a uri, annotations or links
attributes array of attribute objects the DSD has attributes
metadataAttributes array of metadata attribute usage objects the attribute list has metadata attribute usages
measureList object: uri, annotations, links the measure list has a uri, annotations or links
measures array of measures (string or object) the DSD has measures
metadatastructure MSD short URN, value or array the DSD references a metadata structure

Fields of a dimension, attribute or measure object (the object key is the concept id, ALIAS.CONCEPT_ID if there are several concept schemes):

Field Value Written when
id component id the id differs from the concept id
uri URI the component has a uri
enumeration (codelist, valuelist, …) short URN the representation is enumerated (never for the time dimension)
text format fields (textType, maxLength, …) see representation the representation has a text format (never for the time dimension)
minOccurs number set on the representation
maxOccurs number, or "unbounded" set on the representation
conceptRole concept canonical value(s) (Concept=…), value or array dimension or attribute only: it has concept roles
conceptscheme short URN dimension only: SDMX 2.x measure dimension
group group id attribute only: it is reported against a group
dimensions dimension id(s), value or array attribute only: it is reported against dimensions
measures measure id(s), value or array attribute only: it relates to specific measures
level "dataflow" or "observation" attribute only: it is reported against the dataflow or each observation
required true attribute or measure only: its usage is mandatory
annotations see annotations the component has annotations
links see links the component has links

Fields of a group object (array form only, keyed by group id):

Field Value Written when
uri URI the group has a uri
dimensions dimension id(s), value or array always
annotations annotations array the group has annotations
links links array the group has links

Fields of a metadata attribute usage object (keyed by the metadata attribute reference):

Field Value Written when
group group id the usage is reported against a group
dimensions dimension id(s), value or array the usage is reported against dimensions
measures measure id(s), value or array the usage relates to specific measures
level "dataflow" or "observation" the usage is reported against the dataflow or each observation
annotations annotations array the usage has annotations

Reading notes

  • The concept scheme fields (conceptscheme / conceptschemes) must come before the components. A component key with an unknown alias, or with no concept scheme, is an error.
  • If a component object has no id, the component id is taken from the concept id.
  • The reader identifies the time dimension by its id, TIME_PERIOD.
  • Each attribute relationship is read as it is met: group, dimensions and level set the attachment level as shown in Attribute relationship, and measures sets the related measures. There is no default attachment level, so an attribute with no group, dimensions or level is an error, even if it has measures. Any level other than dataflow or observation is an error.
  • The attribute validation of the bean still applies, so a document the writer would not produce can be normalised. level: "observation" with no dimensions or measures relates the attribute to all the measures, and dimensions that include TIME_PERIOD with no level give attachment level Observation. In a DSD with a time dimension, level: "observation" with dimensions that do not include TIME_PERIOD gives attachment level DimensionGroup.
  • A conceptscheme field inside a dimension is read as the dimension's representation. A concept scheme representation makes the dimension a measure dimension.
  • An attribute or measure without required: true is read as conditional.