Writing Data¶

Overview¶

Data is written as a stream in sdmx-core by using one of two interfaces:

DataWriterEngine
ISeriesObsDataWriterEngine

The two interfaces provide the same functionality, with the only difference being how the information is packaged on the Interface API, the DataWriterEngine expects key-value pairs, while the ISeriesObsDataWriterEngine expects pre-packaged objects to be provided, the same objects that are read from the DataReaderEngine.

Like reading data, the DataWriterEngine is typically obtained via the higher-level manager interface: DataWriterManager

When writing data the DataStructureBean is always required, higher-level structural metadata can be provided such as the DataflowBean and ProvisionAgreementBean.

The model of what is being written by both Interfaces follows the SDMX Dataset model, and is represented below.

classDiagram
  Dataset "1" *-- "0..*" Series : series
  Series "1" *-- "0..*" Observation : observations
  Observation "1" *-- "0..*" Measure : measures
  Measure "1" *-- "1" ComponentValue : measureValue
  Observation "1" *-- "0..*" ComponentValue : attributeValues
  Series "1" *-- "0..*" ComponentValue : attributeValues
  Series "1" *-- "0..*" ComponentValue : dimenisonValues
  Dataset "1" *-- "0..*" ComponentValue : attributeValues

  class Dataset {
    +List~ComponentValue~ attributeValues
  }

  class Series {
    +List~ComponentValue~ dimenisonValues
    +List~ComponentValue~ attributeValues
  }

  class Observation {
    +ComponentValue dimensionAtObservation
    +List~ComponentValue~ attributeValues
  }


  class Measure {
    +ComponentValue value
  }

  class ComponentValue {
     +String conceptId
     +String value
  }

Note: It is not the responsibility of the DataWriterEngine to validate data, and as such the DataStructureBean and other structural metadata provided is just for information, not as a means to run validation checks; this is by design as the framework allows invalid data to be written, as the source may contain errors. Data validation is a separate part of the sdmx-core framework.

DataWriterEngine¶

The DataWriterEngine interface is a low level API providing methods to write various parts of the dataset, with each part method being in the context of the previous; for example 'writeAttributeValue' will write a dataset attribute, series attribute, or observation attribute depending on the current context.

Each write method typically takes the ID of the Component in the DSD, and the reported value, e.g. 'COUNTRY' is the Component ID, and 'USA' is the reported value.

public static void main(String[] args) {
    DataStructureBean dsd = TestDSDUtil.generateDSD(true, true, "FREQ", "COUNTRY", "INDICATOR");

    //Create a DataWriterEngine
    SDMX_SCHEMA sdmxMLVersion = SDMX_SCHEMA.VERSION_THREE;
    boolean prettyPrint = true;
    DataWriterEngine dwe = new CompactDataWriterEngine(sdmxMLVersion, System.out, prettyPrint);

    //Start the dataset with a dataset attribute UNIT
    dwe.startDataset(null, null, dsd, null);
    dwe.writeAttributeValue("UNIT", "Person");

    dwe.startSeries();

    //Series Dimension Values
    dwe.writeSeriesKeyValue("FREQ", "A");
    dwe.writeSeriesKeyValue("COUNTRY", "USA");
    dwe.writeSeriesKeyValue("INDICATOR", "BIRTHS");

    //Series Attribute Values
    dwe.writeAttributeValue("TITLE", "Number of Births US");
    dwe.writeAttributeValue("UNIT_MULT", "Millions");

    //Observation for 2024=14, with OBS_STATUS=E
    dwe.writeTimeSeries("2024", "14");
    dwe.writeAttributeValue("OBS_STATUS", "E");

    //Observation for 2025=14, with OBS_STATUS=A
    dwe.writeTimeSeries("2025", "13");
    dwe.writeAttributeValue("OBS_STATUS", "A");

    //Start a new series, and write key + obs
    dwe.startSeries();
    dwe.writeSeriesKeyValue("FREQ", "A");
    dwe.writeSeriesKeyValue("COUNTRY", "FRA");
    dwe.writeSeriesKeyValue("INDICATOR", "BIRTHS");
    dwe.writeTimeSeries("2024", "11");
    dwe.writeTimeSeries("2025", "12");

    //Close off all resources, flush any output to the stream
    dwe.close();
}

Time Series Data¶

The DataWriterEngine API natively supports time series data with only one measure (OBS_VALUE) through the method 'writeTimeSeries' which takes the time period and the observation value.

    dwe.startSeries();
    dwe.writeSeriesKeyValue("FREQ", "Q");
    dwe.writeSeriesKeyValue("COUNTRY", "FRA");
    dwe.writeSeriesKeyValue("INDICATOR", "BIRTHS");

    //Time Series for A:BIRTHS:FRA
    dwe.writeTimeSeries("2025-Q1", "11.1");
    dwe.writeTimeSeries("2025-Q2", "12.2");
    dwe.writeTimeSeries("2025-Q3", "13.3");
    dwe.writeTimeSeries("2025-Q4", "14.1");

The reported time period must be in one of the supported formats, and should correspond to the reported Frequency if there is a FREQ Dimension.

Frequency ID	Name	Format	Example
A	Annual	YYYY	2025
S	Semester	YYYY-Sn	2025-S1
T	Trimester	YYYY-Tn	2025-T1
Q	Quarterly	YYYY-Qn	2025-Q1
M	Monthly	YYYY-MM	2025-12
W	Weekly	YYYY-WW	2025-W22
D	Daily	YYYY-MM-DD	2025-10-30
H	Hourly	YYYY-MM-DD-Thh	2010-01T20
I	Date Time	YYYY-MM-DD-Thhss	2010-01T20:22:00

Note: The Frequency 'H' can be used to represent Half Yearly by setting TIME_FORMAT.setUseAlternateHalfYear(true);

Multiple Measures¶

Multiple Measures are supported through the method 'writeObservation' which takes an option 'dimension at observation' which would typically be "TIME_PERIOD" for data that has a time dimension, and additionally takes a list of Measures.

String dimensionId = "TIME_PERIOD";
String dimensionValue = "2025";

List<KeyValue> measures = new ArrayList<>();
measures.add(KeyValueImpl.getInstance("BIRTHS", "15"));
measures.add(KeyValueImpl.getInstance("DEATHS", "12"));
measures.add(KeyValueImpl.getInstance("MARRIAGES", "2"));

//Observation with 3 measures
dwe.writeObservation(dimensionId, dimensionValue, measures);

Non-Numerical Measures¶

The measures value does not have to be numerical, it can be in any format as defined by the DSD, including free text, or a Code ID.

String dimensionId = "TIME_PERIOD";
String dimensionValue = "2025";

List<KeyValue> measures = new ArrayList<>();
measures.add(KeyValueImpl.getInstance("TEMPERATURE", "HOT"));
measures.add(KeyValueImpl.getInstance("SPEED", "LOW"));
measures.add(KeyValueImpl.getInstance("POWER", "HIGH"));

//Observation with 3 measures
dwe.writeObservation(dimensionId, dimensionValue, measures);


### Multivalue Attributes

In SDMX v3.0 and above Data Attributes can have multiple values, for example DATASOURCE could be both IMF and OECD for an Observation value. The DataWriterEngine supports this by providing a varargs for the writeAttributeValue method.

```java
//Series Dimension Values
dwe.writeSeriesKeyValue("FREQ", "A");
dwe.writeSeriesKeyValue("COUNTRY", "USA");
dwe.writeSeriesKeyValue("INDICATOR", "BIRTHS");

//Multi-Value Attribute
dwe.writeAttributeValue("DATASOURCE", "IMF", "OECD");

Multilingual Attributes¶

In SDMX v3.0 and above Data Attributes which are free text with a language component, can report a value for different languages.

List<TextTypeWrapper> titles = new ArrayList<>();
titles.add(new TextTypeWrapperImpl("en", "English Title"));
titles.add(new TextTypeWrapperImpl("fr", "French Title"));
MultilingualNodeValue titleNode = new MultilingualNodeValue(titles);

//technically there can also be multiple values reported in multiple languges
//this is why the value is a list
IMultilingualKeyValue kv = new MultilingualKeyValue(dimensionValue, List.of(titleNode));

//Write the multilingual title
dwe.writeMultilingualAttributeValue(kv);

ISeriesObsDataWriterEngine¶

The ISeriesObsDataWriterEngine has a higher-level interface than the DataWriterEngine, it expects the Series and Observations to be prebuilt and provided as Keyable and Observation objects, corresponding 1:1 with the DataReaderEngine data model; in this regard the ISeriesObsDataWriterEngine is more compatible with the read part of the framework, but it is more effort to construct a Keyable than to use the DataWriterEngine API directly as shown in the example below, which writes one Series with one Observation.

DataStructureBean dsd = ...;
ISeriesObsDataWriterEngine soDwe = ...;

//Start the Dataset
IDatasetStructures structures = new DatasetStructures(dsd);
DatasetHeaderBean header = null;
IDatasetAttributes datasetAttributes = null;
soDwe.startDataset(structures, header, datasetAttributes);

//Build the Series Key
Keyable series = KeyableImpl.builder(dsd)
                            .key(KeyValueImpl.getInstance("FREQ", "A"))
                            .key(KeyValueImpl.getInstance("COUNTRY", "USA"))
                            .addAttribute(KeyValueImpl.getInstance("TITLE", "Number of Births US"))
                            .build();

//Write the Series Key
soDwe.writeKey(series);

//Build the Observation
Observation obs = ObservationImpl.builder(series)
                                 .dim("TIME_PERIOD", "2024")
                                 .measure(KeyValueImpl.getInstance("BIRTHS", "12"))
                                 .measure(KeyValueImpl.getInstance("DEATHS", "13"))
                                .addAttribute(KeyValueImpl.getInstance("OBS_STATUS", "E"))
                                .build();

//Write the Observation for the Series
soDwe.writeObservation(obs);

soDwe.close();

The output snippet for the above code, showing the Series written with one Observation

<message:DataSet xsi:type="ns1:DataSetType" ss:structureRef="TEST_TEST_1_0">
  <Series FREQ="A" COUNTRY="USA" TITLE="Number of Births US">
    <Obs TIME_PERIOD="2024" BIRTHS="12" DEATHS="13" OBS_STATUS="E"/>
  </Series>
</message:DataSet>

Where the ISeriesObsDataWriterEngine really shines is for quickly building Data Writer implementations, where the pre-packaged information is easier to work with, writing higher-level wrappers such as data writers that aggregate the observation values, and for data transformation, which reads from the DataReaderEngine and writes the keys and observations directly to the ISeriesObsDataWriterEngine without having to modify the objects or unpack them.

Data Writer Interoperability¶

RolldownDataWriterEngine¶

The framework provides the RolldownDataWriterEngine proxy enalbing a DataWriterEngine interface to conform to the ISeriesObsDataWriterEngine interface

SDMX_SCHEMA sdmxMLVersion = SDMX_SCHEMA.VERSION_THREE;
boolean prettyPrint = true;
DataWriterEngine dwe = new CompactDataWriterEngine(sdmxMLVersion, System.out, prettyPrint);

//Wrap the dwe to allow us to use the ISeriesObsDataWriterEngine API
ISeriesObsDataWriterEngine soDwe = new RolldownDataWriterEngine(dwe);

//Build the Series Key
Keyable series = KeyableImpl.builder(dsd)
                            .key(KeyValueImpl.getInstance("FREQ", "A"))
                            .key(KeyValueImpl.getInstance("COUNTRY", "USA"))
                            .addAttribute(KeyValueImpl.getInstance("TITLE", "Number of Births US"))
                            .build();

//Write the Series Key - this is unpacked and sent to the underlying DataWriterEngine
soDwe.writeKey(series);

RollupDataWriterEngine¶

Conversely the RollupDataWriterEngine proxy enables the ISeriesObsDataWriterEngine to act as a DataWriterEngine

ISeriesObsDataWriterEngine soDwe = new InMemoryDataWriterEngine();
DataWriterEngine dwe = new RollupDataWriterEngine(soDwe);

//Calls the start dataset on the ISeriesObsDataWriterEngine
dwe.startDataset(null, null, dsd, null);

//The RollupDataWriterEngine will create a Keyable and pass this to the ISeriesObsDataWriterEngine
dwe.startSeries();
dwe.writeSeriesKeyValue("FREQ", "A");
dwe.writeSeriesKeyValue("COUNTRY", "US");
dwe.writeSeriesKeyValue("INDICATOR", "BIRTHS");

//The RollupDataWriterEngine will create a Observation and pass this to the ISeriesObsDataWriterEngine
dwe.writeTimeSeries("2024", "12");

dwe.close();

Implementations¶

There are a large number of data writers in the sdmx-core framework which fall into the following categories:

Output - the writer generates an output in a specific format; e.g. SDMX-CSV
Proxy - the writer is a proxy on another writer, typically to perform a task before passing the message onto the proxy
Other - these include in-memory writers, empty writers, or validation writers

The following tables summarise the writers, grouping into Output, Proxy, and '-' to denote Other.

DataWriterEngine¶

Implementation	Type	Description
CompactDataWriterEngine	Output	Writes SDMX Compact / Structure Specific Data data format (XML)
EDIDataWriterEngineImpl	Output	Writes SDMX EDI data format
GenericDataWriterEngine	Output	Writes SDMX Generic data format (XML)
PreventEmptySeriesDataWriterProxy	Proxy	Proxies a DataWriterEngine - does not pass one Series with no attributes or Observations
RollupDataWriterEngine	Proxy	Proxies a ISeriesObsDataWriterEngine - passes all data to the proxy API
EmptyDataWriterEngine	-	Writes nothing

ISeriesObsDataWriterEngine¶

Implementation	Type	Description
KryoDataWriterEngine	Output	Serialises data to file using Kryo (used as a temporary store)
MVStoreDataWriterEngine	Output	Serialises data to file using MV Store (used as a temporary store)
ExcelWorkbookWriterEngine	Output	Writes to Excel
SdmxCsvDataWriterEngineV1	Output	Writes SDMX-CSV v1
SdmxJsonDataWriterEngine	Output	SDMX Json v1
SdmxJsonDataWriterEngineV2	Output	SDMX Json v2
DatasetInfoDataWriterEngine	Proxy	Captures distinct values written for each Component and the order in which they were written
DatasetValuesDataWriter	Proxy	Captures distinct values written for each Component
DataWriterEngineProxy	Proxy	Abstract proxy with hooks to modify series and observations
DatasetInfoWriterEngine	Proxy	Captures a count of datasets, series and observations
DataWriterAssertionProxy	Proxy	Proxies a ISeriesObsDataWriterEngine and keeps an in-memory record of what was written
GroupDataWriterEngine	Proxy	Converts all Group Attributes and converts to Series Attributes and passes to proxy
MappingDataWriterEngine	Proxy	Maps a dataset using a StructureMap and RepresentationMap passing the mapped dataset to the proxy
PeriodInfoDataWriterEngine	Proxy	Capture the start and end period details for the dataset
RolldownDataWriterEngine	Proxy	Proxies a DataWriterEngine - passes all data to the proxy API
ScalingDataWriterEngine	Proxy	Modifies measure values by applying a scaling factor
SubConstraintDataWriterEngine	Proxy	Captures a subset of distinct values
ComparableDataWriterEngine	-	Used to compare two datasets, irrespective of Series or Observation order
EmptySeriesObsDataWriterEngine	-	Writes nothing
InMemoryDataWriterEngine	-	Stores dataset in memory
ValidatingDataWriterEngine	-	Validates the dataset written according to the DSD and related metadata

The proxy framework supports the paradigm of chaining writers, which enables the output behaviour to be controlled based on the how the proxies are chained.

The following example collapses group attributes into their respective Series, and then scales the measure values, before writing to an SDMX-CSV output


flowchart LR
   GroupDataWriterEngine--> ScalingDataWriterEngine  --> SdmxCsvDataWriterEngineV1

Data Writer Manager¶

The DataWriterManager interface is used to generate a DataWriterEngine of a given format, this decouples the application from needing to know about specific implementations.

Like all other Manager Interfaces, the implementation should be configured as a Singleton and registered with the framework.

Note The DataWriterManager can only generate a ISeriesObsDataWriterEngine, this can be converted to the DataWriterEngine by wrapping it in a RollupDataWriterEngine. The DataWriterManager can only generate writers that are tied to a specific format, it does not create proxy writers, or 'other' writers.

DataWriterManager dwm = new DataWriterManagerImpl(SdmxCsvDataFormat.BASIC_V2);

DataFormat df = new SdmxJsonDataFormat(DATA_TYPE.SDMXJSON_2_0_0, null);
ISeriesObsDataWriterEngine dwe = dwm.getDataWriterEngine(df, System.out);