SDMX Beans¶
Overview¶
Structural metadata defined in the SDMX Information model (Codelists, Concept Schemes, Dataflows, etc) are collectively referred to as an SDMXBean in the sdmx-core framework, or simply as an SDMX Structure or just a structure. Each SDMX Bean is named after the SDMX class name combined with the 'Bean' postfix; for example CodelistBean.
The SDMXBean interfaces are found in the fusion-api-sdmx module under the model.beans package.
The implementations live in the fusion-sdmx-im Module in the beans package.
Validation and Immutability¶
All SDMXBeans are validated on construction, to ensure all required properties are set and have a valid value. SDMXBean are immutable, once constructed they can never be modified, all lists are immutable and there are no public setters. The only way to create an SDMXBean is to first create the mutable counterpart called the MutableBean. The MutableBean has no rules or checks for validity, as such it should only be used as a means to create an SDMXBean. An SDMXBean can only be modified if a MutableBean copy is created, modified and then converted back into a new SDMXBean.
//Create a Codelist with Frequency Codes (Annual, Daily, Monthly)
CodelistBean clFreq = TestCodelistInstances.getFREQ("SDMX", "1.0.0");
//Create a mutable copy of the Frequency Codelist
CodelistMutableBean clMutable = clFreq.getMutableInstance();
//Change the AgencyID
clMutable.setAgencyId("NEW_AGENCY");
//Overwrite old copy with the new copy
clFreq = clMutable.getImmutableInstance();
SDMX Beans are immutable: the content of an SDMX Bean cannot be modified, all collections are made unmodifiable on construction, and the SDMX Bean is fully validated to ensure all mandatory information is present and all business rules pass, such as prevention of a recursive hierarchy in a Codelist. This ensures any method that requires an SDMX Bean can be sure that it will be correct, and will not be modified by an external process.
Inheritance model¶
In keeping with the SDMX-IM, the sdmx-core models SDMX structures as a hierarchical model, taking advantage of the inheritance relationship and common relationships between the structures.
All structures in sdmx-core are built at the level of MaintainableBean; this is the most important interface as every SDMX structure is a Maintainable structure; it has a Structure Type, an Agency (owner), Identity, and Version. From these four properties, the unique URN can be built, which gives the structure a globally unique identity.
classDiagram
Serializable <|-- SDMXBean
SDMXBean <|-- SdmxStructureBean
SdmxStructureBean <|-- AnnotableBean
AnnotableBean <|-- IdentifiableBean
IdentifiableBean <|-- NameableBean
NameableBean <|-- VersionableBean
VersionableBean <|-- MaintainableBean
class Serializable {
}
class SDMXBean {
+SDMX_STRUCTURE_TYPE structureType
+SDMXBean parent
}
class SdmxStructureBean {
+MaintainableBean maintainableParent
}
class AnnotableBean {
+List<AnnotationBean> getAnnotations()
}
class IdentifiableBean {
+String getId()
+IURNAbsolute getUrn()
+List<ILinkBean> getLinks()
}
class NameableBean {
+List<TextTypeWrapper> getNames()
-List<TextTypeWrapper> getDescriptions()
}
class VersionableBean {
+ISdmxVersion getVersion()
}
class MaintainableBean {
+String getAgencyId()
}
Moving from the abstract to the concrete interfaces, all container structures extend MaintainableBean, either directly or indirectly, for example a SDMX Codelist and Concept Scheme are both Item Schemes with items, Item Scheme is a Maintainable structure.
Any structure that has an identity and globally unique identifier (URN) is an IdentifiableBean, this includes all Maintainable structures, and any composite structures that have identity, for example Codes, Concepts, Dimensions, Attributes, Measures.
The URN is a key property in both SDMX as a standard, and sdmx-core as a framework, as it is the globally unique, location independent identifier of a structure. sdmx-core is able to resolve a URN to a structure in the system, at a global level the URN resolver is used to resolve a URN to a structure.
classDiagram
IdentifiableBean <|-- NameableBean
NameableBean <|-- VersionableBean
NameableBean <|-- ItemBean
VersionableBean <|-- MaintainableBean
MaintainableBean <|-- ItemSchemeBean
ItemSchemeBean <|-- CodelistBean
ItemSchemeBean <|-- ConceptSchemeBean
MaintainableBean <|-- DataStructureBean
MaintainableBean <|-- DataflowBean
ItemBean <|-- CodeBean
ItemBean <|-- ConceptBean
class IdentifiableBean {
+id
+urn
}
class NameableBean {
+names
-descriptions
}
class VersionableBean {
+version
}
class MaintainableBean {
+agencyId
}
class ItemSchemeBean {
+items
}
class DataStructureBean {
+dimensions
+attributes
+measures
}
class DataflowBean {
+dataStructureReference
}
SdmxBeans Container¶
Collections of SDMX structures are passed in a system in a SdmxBeans container.
SdmxBeans beans = new SdmxBeansImpl();
beans.addIdentifiable(codelistCountry);
beans.addIdentifiable(codelistFrequency);
beans.addIdentifiable(conceptScheme);
Set<CodelistBean> clSet = beans.getCodelists(); //2 Codelists
Set<ConceptSchemeBean> csSet = beans.getConceptSchemes(); //1 Concept Scheme
The SdmxBeans container provides interfaces to add, find, and remove structures. The container can be used as a source for searching for content.
private Set<CodelistBean> getV1Codelists(SdmxBeans beans) {
IURN<CodelistBean> codelistFilter = URN.builder()
.version("1.0.0")
.build(CodelistBean.class);
return beans.getMaintainableBeans(codelistFilter);
}
To protect a SdmxBeans from change, a readonly copy can be created.
Deferred Beans¶
Every MaintainableBean has a toDeferred method which offloads the objects contents from heap memory. The toDeferred returns a copy of the structure which implements the same interface as the 'standard' structure, enabling the lightweight copy to be passed around a system as if it is the full structure.
The deferred structure automatically loads itself back into memory when required, for example, iterating all the Codes in a codelist will result in a load from the file system, and when the JVM requires additional memory, will automatically unload itself. The deferred beans framework is perfect for a system which has to manage a large amount of structures, without the need to provision a large amount of memory.
Building Deferred Beans¶
There are two options for creating a deferred bean, in both cases a lightweight copy of the full structure is created, with a copy of the full structure stored off heap, and only loaded in when required.
Offload to Filesystem
The default deferred bean location is on the file system, as a java serialisation. The file is written to Java temporary folder. The file is automatically deleted when the deferred bean is garbage collected. It is important to note that if the JVM is shut down before these deferred beans are garbage collected, the temporary file will not be deleted; as such it is recommended to set the java temp directory to a folder that can be deleted after JVM shutdown.
CodelistBean largeCodelist = ...
//offload all the codes out of memory onto the file system
largeCodelist = (CodelistBean) largeCodelist.toDeferred();
//iterate codes (loads back into memory)
for(CodeBean code : largeCodelist.getItems()) {
System.out.println(code.getId());
}
Custom Offload
A custom storage location can be used to store the serialised structure, e.g. a database. A custom location is provided by using a custom implementation of the IDeferredLoader interface.
CodelistBean largeCodelist = ...
IDeferredLoader<MaintainableBean> customLoader = ....;
//the customLoader is asked to store the full structure
//and must return the full structure when requested
largeCodelist = (CodelistBean) largeCodelist.toDeferred();
Super Beans¶
Many use cases for data processing require the Data Structure Definition and related Concepts and Codelists. SDMX is also nuanced in that the link between a Data Structure Definition and a Codelist is either direct through the Dimension, Attribute, or Measure, or indirect, through the Concept. Any method that requires a full package of Concepts, Codelists, and DataStructure would have to reconstruct the relationships. This is where the sdmx-core SuperBean helps. A SuperBean contains the cross referenced structures by composition, as opposed to by reference, this enables a single structure i.e. DataStructureSuperBean to be passed into a method, where all the referenced structures have been resolved and linked.
DataStructureSuperBean dsdSb = null;
//For each Dimension, print the enumerated values, if it uses one
for(DimensionSuperBean dim : dsdSb.getDimensions()) {
ConceptSuperBean concept = dim.getConcept();
//Print Dimension Name
System.out.println(concept.getName());
EnumeratedListBean<?> cl = dim.getCodelist(true);
if(cl != null) {
//Print ID -> Name of all Codes/Items
cl.getItems().forEach(item-> {
System.out.println(item.getId() +" -> "+ item.getName());
});
}
}
Building SuperBeans¶
As a SuperBean combines all referenced structures a container of the structures is required when building, or the means to retrieve structures on request.
There are several ways to build SuperBeans.
Direct
Construct directly for the specific structure type
DataStructureBean dsd = ... //to build from
SdmxBeanRetrievalManager retrievalManager = ...; //access to structures
SuperBeans existingBeans = ...; //optional existing beans
CrossReferenceExceptionHandler xsExHandler = ...//optional exception handler
DataStructureSuperBeanBuilder.getInstance().build(dsd, retrievalManager, existingBeans, xsExHandler)
Container
Create a SuperBeans container, this is equivalent of a SdmxBeans container for superbeans.
SdmxBeans buildFrom = ... //the input collection of beans
SuperBeans superBeans = SuperBeansBuilderImpl.getInstance().build(buildFrom);
//alternaively pass in additional arguments
SdmxBeans buildFrom = ... //the input collection of beans
SdmxBeanRetrievalManager retrievalManager = ...; //access to structures
SuperBeans existingBeans = ...; //optional existing beans
SuperBeansBuildErrorHander exHandler = ...//optional exception handler
CrossReferenceExceptionHandler xsExHandler = ...//optional exception handler
SuperBeans superBeans = SuperBeansBuilderImpl.getInstance().build(buildFrom,
existingBeans,
retrievalManager,
exHandler,
xsExHandler);
Retreival Manager
Build on demand by wrapping a SdmxBeanRetrievalManager.