Data Structure Definition¶
A Data Structure Definition (DSD) describes the shape of a dataset: its dimensions, groups, attributes, measures and
any metadata attributes it uses. The document starts with ref: "DataStructure=AGENCY:ID(1.0)".
The writer is DataStructureCanonicalWriter.
Simple form¶
The simplest complete DSD has one concept scheme, dimensions and measures. Every component takes its id from its concept and has no representation, so each one is written as a plain string.
format: "fusion-yaml(1.0.0)"
ref: "DataStructure=TEST:TEST(1.0)"
name: "TEST"
conceptscheme: "TEST:CS(1.0)"
dimensions:
- "INTERNAL_CODE"
- "FREQ"
- "FOF_PERIOD"
measures:
- "OBS_VALUE"
The document starts with the common header. Then come these body fields:
conceptscheme: the short URN of the concept scheme that every component's concept belongs to. If the components use more than one concept scheme,conceptschemesis written instead (see Multiple concept schemes).dimensions: the dimensions in key order. Each one is keyed by its concept id (components).measures: the measures, in definition order.
Complex form¶
Dimensions and measures are written as a string when they have nothing to record except their concept. They become a single-key object, keyed by the concept, as soon as they have any of the following:
| Component | It becomes an object when it has |
|---|---|
| Dimension | an id that is not its concept id, a uri, annotations, links, a representation (not for the time dimension), concept roles, or a measure dimension concept scheme |
| Measure | an id that is not its concept id, a uri, annotations, links, a representation, or a mandatory usage (required) |
A text format counts as a representation even when it has no fields set.
Attributes and metadata attributes are always written as objects. Every attribute has a relationship to record (see Attribute relationship). The relationship of a metadata attribute usage is optional, and a usage without one is written as an empty object (see Metadata attributes).
Representation¶
A dimension, attribute or measure with an enumeration or a text format is written as an object holding its representation:
dimensions:
- FREQ:
codelist: "TEST:CL_FREQ(1.0)"
- REF_AREA:
codelist: "TEST:CL_REF_AREA(1.0)"
- INDICATOR:
codelist: "TEST:CL_INDICATOR(1.0)"
- T:
id: "TIME_PERIOD"
attributes:
- TITLE:
textType: "String"
maxLength: 50
dimensions:
- "FREQ"
- "REF_AREA"
- "INDICATOR"
- SOURCE:
textType: "AlphaNumeric"
maxLength: 5
minOccurs: 1
maxOccurs: 2
measures: "OBS_VALUE"
level: "observation"
required: true
- UNIT:
valuelist: "ACY:VL_UNIT(1.0)"
measures:
- "OBS_VALUE"
- "BIRTHS"
level: "observation"
measures:
- "OBS_VALUE"
- "BIRTHS"
The enumeration key is the lower-case class of the referenced structure, for example codelist or valuelist
(references). maxLength, minOccurs and maxOccurs are numbers. A maxOccurs with
no upper bound is written as "unbounded".
Time dimension¶
The writer never writes the representation of the time dimension, because it is fixed. The time dimension always
has the id TIME_PERIOD, so its form depends only on its concept:
- If the concept is
TIME_PERIODand there is no uri, annotation, link or concept role, it is written as the string"TIME_PERIOD". - If the concept has a different id, for example
T, it is written as an object that holds theid:
The id field is written for any component whose id differs from its concept id. On the time dimension it is
simply the most common case.
Concept roles¶
A dimension or attribute with concept roles has a conceptRole field, so a dimension with concept roles is always
written as an object. Each entry is the role concept as a canonical value
(Concept=AGENCY:ID(VERSION).CONCEPT_ID), because the field name does not say what type of structure is referenced.
The field follows the value-or-array rule: one role is written as a string, several
as an array.
dimensions:
- FREQ:
codelist: "TEST:CL_FREQ(1.0)"
conceptRole: "Concept=SDMX:CROSS_DOMAIN_CONCEPTS(2.0).FREQ"
- REF_AREA:
codelist: "TEST:CL_REF_AREA(1.0)"
conceptRole:
- "Concept=SDMX:CROSS_DOMAIN_CONCEPTS(2.0).REF_AREA"
- "Concept=SDMX:CROSS_DOMAIN_CONCEPTS(2.0).COUNTERPART_AREA"
- INDICATOR:
codelist: "TEST:CL_INDICATOR(1.0)"
- T:
id: "TIME_PERIOD"
On an attribute, conceptRole is written before the relationship fields:
Measure dimension¶
An SDMX 2.x measure dimension is represented by a concept scheme. It is written as an object with a conceptscheme
field that holds the short URN of that scheme:
conceptscheme: "TEST:CS(1.0)"
dimensions:
- "FREQ"
- "REF_AREA"
- INDICATOR:
conceptscheme: "TEST:CS_INDICATORS(1.0)"
- T:
id: "TIME_PERIOD"
measures:
- "OBS_VALUE"
This field has the same name as the top-level conceptscheme but means something different. At the top level it
says where the concepts come from. Inside a dimension it is the dimension's representation.
Groups¶
Groups are written in one of two forms, chosen for all groups together.
In the map form, every group is written as GROUP_ID: dimension id(s). The dimensions follow the
value-or-array rule, so a group with one dimension is written as a string. This
form is used when no group has a uri, annotations or links:
In the array form, groups becomes an array of identifiables. This form is used
when any group has a uri, annotations or links. Each group is an object keyed by its id, holding uri,
dimensions (a value or array), annotations and links:
groups:
- GROUP_1:
dimensions:
- "EXR_TYPE"
- "EXR_SUFFIX"
annotations:
- id: "GRP_ANN"
- GROUP_2:
dimensions:
- "CURRENCY"
- "CURRENCY_DENOM"
Attribute relationship¶
An attribute records two independent things:
- Where it is reported (its attachment): against the whole dataflow, against a group, against a set of dimensions (some or all of them), or against each observation.
- Which measures it relates to (optional): the measures the attribute value describes. This is independent of where it is reported, so an attribute can be reported once for the dataflow and still relate to only one measure.
Each attribute is written as an object with these relationship fields, in this order:
| Field | Value | Meaning |
|---|---|---|
group |
group id | the attribute is reported against the group (attachment level Group) |
dimensions |
dimension id(s), value or array | the attribute is reported against this subset (or the full set) of dimensions (attachment level DimensionGroup) |
measures |
measure id(s), value or array | the measures the attribute relates to |
level |
"dataflow" or "observation" |
the attribute is reported against the dataflow (attachment level DataSet) or each observation (attachment level Observation) |
level is only written for the two levels that group and dimensions cannot express. The attachment level is
therefore always explicit, and is never inferred from measures:
| Fields written | Attachment level |
|---|---|
group |
Group |
dimensions |
DimensionGroup |
level: "dataflow" |
DataSet |
level: "observation" (with or without dimensions) |
Observation |
Attributes reported at each level:
attributes:
- CS2.REP_YEAR_START:
level: "dataflow"
- CS.COMMENT_DSET:
level: "dataflow"
required: true
- CS2.COLLECTION:
dimensions:
- "FREQ"
- "REF_AREA"
- "INDICATOR"
- CS2.OBS_STATUS:
measures: "OBS_VALUE"
level: "observation"
An attribute reported against a group:
attributes:
- OBS_CONF:
codelist: "ECB:CL_OBS_CONF(1.0)"
measures: "OBS_VALUE"
level: "observation"
- G1_A1:
group: "GROUP_1"
- G1_A2:
group: "GROUP_1"
required: true
An attribute reported once for the dataflow, but relating only to the OBS_VALUE measure:
Required¶
A mandatory attribute or measure writes required: true inside its own object. A conditional one omits the field
(false is the default). Dimensions never write required.
Because an attribute is always an object, required never changes its form. A measure that is otherwise simple
becomes an object when it is mandatory:
Metadata attributes¶
Metadata attribute usages from the attribute list are written as metadataAttributes, between attributes and
measureList. Each entry is an object keyed by the metadata attribute reference (an id from the DSD's metadata
structure). It holds the same relationship fields as an attribute (group, dimensions, measures, level) and
any annotations. Usages have no uri, links, representation or required. The metadata structure itself is referenced
by metadatastructure:
metadataAttributes:
- CONTACT:
level: "dataflow"
- SOURCE:
dimensions:
- "FREQ"
- "REF_AREA"
measures:
- "OBS_VALUE"
metadatastructure: "TEST:METADATA(1.0)"
The relationship of a usage is optional. A usage without one can be reported at any level of the data, and is written
as an empty object, for example - CONTACT: {}.
metadatastructure holds a short URN and follows the value-or-array rule. Only one reference is possible today. It
is handled as a list so that the format is ready for SDMX 3.2, which allows more than one.
Multiple concept schemes¶
When the components use concepts from more than one concept scheme, conceptscheme is replaced by a
conceptschemes map from alias to short URN. Each component key is then written as ALIAS.CONCEPT_ID:
conceptschemes:
CS: "ECB:CS(1.0)"
CS1: "SDMX:CS(1.0)"
CONCEPTS: "TEST:CONCEPTS(1.0)"
CS2: "TEST:CS(1.0)"
dimensions:
- "CS2.FREQ"
- "CS2.REF_AREA"
- "CS1.INDICATOR"
- CONCEPTS.T:
id: "TIME_PERIOD"
attributes:
- CS2.REP_YEAR_START:
level: "dataflow"
- CS.COMMENT_DSET:
level: "dataflow"
required: true
- CS2.OBS_STATUS:
measures: "OBS_VALUE"
level: "observation"
measures:
- "CONCEPTS.OBS_VALUE"
The aliases are assigned like this:
- The concept schemes are sorted by URN (agency, then id, then version), and that is the order in which the map is written.
- Each alias is the concept scheme's id. If that alias is already taken by an earlier scheme, a number is appended:
1, 2 and so on. In the example, three schemes have the id
CS. They becomeCS,CS1andCS2in sort order.
The alias appears only in component keys. id, group dimensions and the attribute relationship fields (group,
dimensions, measures) all use plain component ids.
Dimension list, attribute list and measure list¶
The descriptors dimensionList, attributeList and measureList are written only when they have a uri,
annotations or links. Each one is written just before its components. Their ids are fixed, so no id is written.
conceptscheme: "TEST:CS(1.0)"
dimensionList:
annotations:
- id: "DIM_LIST_ANN"
title: "Dimension List Annotation"
dimensions:
- "FREQ"
- "REF_AREA"
- "INDICATOR"
- T:
id: "TIME_PERIOD"
attributeList:
annotations:
- id: "ATTR_LIST_ANN"
title: "Attribute List Annotation"
attributes:
- TITLE:
dimensions:
- "FREQ"
- "REF_AREA"
- "INDICATOR"
measureList:
annotations:
- id: "MEASURE_LIST_ANN"
title: "Measure List Annotation"
measures:
- "OBS_VALUE"
Field reference¶
Top-level body fields, in written order:
| Field | Value | Written when |
|---|---|---|
conceptscheme |
short URN of the single concept scheme | the components use exactly one concept scheme |
conceptschemes |
map of alias to concept scheme short URN, sorted by URN | the components use more than one concept scheme |
dimensionList |
object: uri, annotations, links |
the dimension list has a uri, annotations or links |
dimensions |
array of dimensions (string or object) | always |
groups |
map of group id to dimension id(s), or array of group objects | the DSD has groups |
attributeList |
object: uri, annotations, links |
the attribute list has a uri, annotations or links |
attributes |
array of attribute objects | the DSD has attributes |
metadataAttributes |
array of metadata attribute usage objects | the attribute list has metadata attribute usages |
measureList |
object: uri, annotations, links |
the measure list has a uri, annotations or links |
measures |
array of measures (string or object) | the DSD has measures |
metadatastructure |
MSD short URN, value or array | the DSD references a metadata structure |
Fields of a dimension, attribute or measure object (the object key is the concept id, ALIAS.CONCEPT_ID if there are
several concept schemes):
| Field | Value | Written when |
|---|---|---|
id |
component id | the id differs from the concept id |
uri |
URI | the component has a uri |
enumeration (codelist, valuelist, …) |
short URN | the representation is enumerated (never for the time dimension) |
text format fields (textType, maxLength, …) |
see representation | the representation has a text format (never for the time dimension) |
minOccurs |
number | set on the representation |
maxOccurs |
number, or "unbounded" |
set on the representation |
conceptRole |
concept canonical value(s) (Concept=…), value or array |
dimension or attribute only: it has concept roles |
conceptscheme |
short URN | dimension only: SDMX 2.x measure dimension |
group |
group id | attribute only: it is reported against a group |
dimensions |
dimension id(s), value or array | attribute only: it is reported against dimensions |
measures |
measure id(s), value or array | attribute only: it relates to specific measures |
level |
"dataflow" or "observation" |
attribute only: it is reported against the dataflow or each observation |
required |
true |
attribute or measure only: its usage is mandatory |
annotations |
see annotations | the component has annotations |
links |
see links | the component has links |
Fields of a group object (array form only, keyed by group id):
| Field | Value | Written when |
|---|---|---|
uri |
URI | the group has a uri |
dimensions |
dimension id(s), value or array | always |
annotations |
annotations array | the group has annotations |
links |
links array | the group has links |
Fields of a metadata attribute usage object (keyed by the metadata attribute reference):
| Field | Value | Written when |
|---|---|---|
group |
group id | the usage is reported against a group |
dimensions |
dimension id(s), value or array | the usage is reported against dimensions |
measures |
measure id(s), value or array | the usage relates to specific measures |
level |
"dataflow" or "observation" |
the usage is reported against the dataflow or each observation |
annotations |
annotations array | the usage has annotations |
Reading notes¶
- The concept scheme fields (
conceptscheme/conceptschemes) must come before the components. A component key with an unknown alias, or with no concept scheme, is an error. - If a component object has no
id, the component id is taken from the concept id. - The reader identifies the time dimension by its id,
TIME_PERIOD. - Each attribute relationship is read as it is met:
group,dimensionsandlevelset the attachment level as shown in Attribute relationship, andmeasuressets the related measures. There is no default attachment level, so an attribute with nogroup,dimensionsorlevelis an error, even if it hasmeasures. Anylevelother thandatafloworobservationis an error. - The attribute validation of the bean still applies, so a document the writer would not produce can be normalised.
level: "observation"with nodimensionsormeasuresrelates the attribute to all the measures, anddimensionsthat includeTIME_PERIODwith nolevelgive attachment level Observation. In a DSD with a time dimension,level: "observation"withdimensionsthat do not includeTIME_PERIODgives attachment level DimensionGroup. - A
conceptschemefield inside a dimension is read as the dimension's representation. A concept scheme representation makes the dimension a measure dimension. - An attribute or measure without
required: trueis read as conditional.