Foundations of sdmx-core¶

SDMX Core has some foundational classes and configurations which underpin the framework. These must be understood before using the library.

Reading and Writing Streams¶

Problem Statement¶

SDMX Core streams information where possible to avoid keeping information in memory; this is especially true for reading and writing data, where datasets can be extremely large.

SDMX Core does not rely on file extensions or user-supplied metadata to know what type of SDMX message it is reading (format, version, type); instead it peeks into the stream and routes it to the correct reader. This enables the programmer to remain agnostic to the underlying exchange format, as SDMX Core takes care of interpreting the format and building it into a common Java model.

In order for SDMX Core to reliably work in this way, there must be a guarantee that the InputStream can be reset to the start of the stream after peeking into the stream. This is not a guarantee in Java as some implementations of InputStream do not support the reset method.

In addition, as Streams are passed between managers and services, there is not always a clear owner of the stream, which is problematic when it comes to knowing when a stream can be closed. If a stream is not closed, this can cause issues with deletion of temporary files, as the stream holds a lock on the file. In some cases a manager may think it should close the stream, but in some corner cases, it should not.

Readable Data Location¶

The solution to the problem is the SDMX Core ReadableDataLocation interface, which can wrap InputStreams, files, URLs, and other sources of information.

The ReadableDataLocation is an interface which allows the client to request the Java InputStream, and on each request the ReadableDataLocation passes a new InputStream, thus guaranteeing the stream is set to the beginning and allows multiple processes to read from the same source without creating a conflict.

The ReadableDataLocation can provide information on the base format of the stream, i.e. XML, CSV, JSON. It is able to do this by reading the first x-bytes of the stream to determine what it is reading. This enables the stream to be routed at a high level to the correct reader.

The ReadableDataLocation can be a compressed file or set of files, e.g. a zip. In this instance it offers methods to split the ReadableDataLocation into one or more uncompressed ReadableDataLocation, and automatically close the parent.

ReadableDataLocation tracks any InputStream that has been requested through its API, and on ReadableDataLocation.close all resources are removed, any InputStreams created from the ReadableDataLocation are closed, ensuring that any clients that forget to close the stream are handled in one location.

ReadableDataLocation can be protected, meaning that a higher-level process can protect it from being closed by a lower-level process that would ordinarily close the ReadableDataLocation.

ReadableDataLocation handles large streams by holding the stream off heap, e.g. by creating a temporary file; the standard setup for SDMX Core is to use in-memory streams until a threshold is hit, and then overflow to the file system; see SdmxSourceReadableDataLocationFactory.

The ReadableDataLocationFactory should be used to create a ReadableDataLocation as it decouples the code from choosing which type of ReadableDataLocation is created, however the concrete types can be used directly if required.

The FusionSystem.getRdlFactory() can be used to obtain a ReadableDataLocationFactory.

Writeable Data Location¶

The WriteableDataLocation supports writing data to a stream, it provides an output stream on request. As the WriteableDataLocation is a ReadableDataLocation, once the output is written it can be passed into a service as a source of information.

The default WriteableDataLocation handles streaming by writing to memory and overflowing to the file system once the memory buffer is full; the memory buffer size is global, which prevents multiple small buffers from exhausting memory.

The WriteableDataLocationFactory should be used to create a WriteableDataLocation as it decouples the code from choosing which type of WriteableDataLocation is created, however the concrete types can be used directly if required.

The FusionSystem.getWdlFactory() can be used to obtain a WriteableDataLocationFactory.

Inversion of Control¶

SDMX Core uses the Inversion of Control (IOC) design principle where the control of object creation, configuration, and flow is transferred from application code to a framework or container.

The IOC can be provided by the Spring Framework, or using the SDMX Core framework, or a combination of both.

SingletonStore¶

SDMX Core introduced a SingletonStore class to decouple the framework from Spring, whilst maintaining the ability to use the Spring framework if required.

The SingletonStore provides access to Singletons in the system, and the ability to register Singleton instances, typically as part of the application configuration process.

Register a Singleton

A singleton is registered by calling the registerInstance method. The instance is registered in the SingletonStore.

   SingletonStore.registerInstance(new SdmxSourceReadableDataLocationFactory());

Retrieve a Singleton

A singleton can be retrieved from its type, or any Interface / supertype.

   ReadableDataLocationFactory rdlFactory = SingletonStore.getSingleton(ReadableDataLocationFactory.class);

In some cases the singleton may not be registered yet, but is expected to be registered in the future; this can be the case when a singleton is constructed as part of the application startup process. In these cases the manager can register its interest in the class, and is provided the instance as soon as it is registered.

  SingletonStore.registerInterest(StructureReaderManager.class, (srm) -> {
    this.structureParsingManager = srm;
  });

The SingletonStore stores instances in its local maps, however an additional source of singletons can be provided by setting a ISingletonContainer on the SingletonStore. This enables the SingletonStore to search both its local maps, followed by the ISingletonContainer if the local maps did not contain the requested singleton.

  @Override
    public void contextInitialized(ServletContextEvent event) {
        try {
            super.contextInitialized(event);
            ApplicationContext appContext = getCurrentWebApplicationContext();
            //Register spring bean container with the
      //SingletonStore and FusionBeanStore
            container = new SpringBeansContainer(appContext);

            ....
            }
    }

FusionBeanStore¶

The FusionBeanStore is very similar to the SingletonStore in its ability to register and retrieve stateless instances from a global application container; the only difference is the FusionBeanStore can store multiple instances of the same interface — for example, a StructureReaderEngine can have multiple implementations, and each one will be stored in the FusionBeanStore. As the FusionBeanStore can store multiple instances of the same interface, a request to retrieve implementations of an interface returns a set of instances, not a single instance.

Note, the SingletonStore and FusionBeanStore are independent stores that serve different roles: the SingletonStore holds unique singleton instances, while the FusionBeanStore can hold multiple implementations of the same interface.

Register an Instance

A new instance is registered by calling registerInstance. The implementation class type can only be registered once, however different implementations of the same interface can be registered. When an instance is registered it is stored against its implementation type, and any supertypes or interfaces it extends or implements.

     FusionBeanStore.registerInstance(new CustomStrutureReaderFactory());

Retrieve a Singleton

A collection of any instances that are of a given type can be returned by passing the class type in. This enables instances to be retrieved at the level of the interface they implement, rather than requesting the concrete implementation class.

     Collection<StructureReaderFactory> stuctureReaderFactories = FusionBeanStore.getBeans(StructureReaderFactory.class);

SDMX Core & the Spring Framework¶

Both the SingletonStore and the FusionBeanStore have an optional sub-container, typed as ISingletonContainer and IFusionContainer respectively. The SpringBeansContainer implements both ISingletonContainer and IFusionContainer and on construction of the SpringBeansContainer it self registers on both the SingletonStore and FusionBeanStore.

The following example shows how the spring framework can be integrated seamlessly into the SDMX Core libraries.

@SpringBootApplication
public class MySpringBootApplication {

    @Autowired
    private ApplicationContext ctx;

    public static void main(String[] args) {
        SpringApplication.run(MySpringBootApplication.class, args);
    }

    @Bean
    ApplicationRunner startupInitialisation() {
        return args ->
            //Set the spring container on the SingletonStore
            SingletonStore.setContainer(new SpringBeansContainer(ctx));

            //Register some additional instances
            SdmxJsonModule.register();
        }
    }
}

Plugin Framework¶

SdmxCore is a plugin framework, enabling managers to dynamically build the capabilities based on the registered engines and factories. A concrete example is reading data; the DataReaderManager is provided a stream of data in a ReadableDataLocation, it is responsible for finding an appropriate implementation of the DataReaderEngine interface based on the format of the dataset. The DataReaderManager does not know about dataset formats, instead it delegates the task to all the registered instances of the DataReaderFactory interface that are in the FusionBeanStore.

     @Override
    public Collection<DataReaderFactory> getDataReaderFactories() {
        if(factories == null) {
            synchronized(this) {
                factories = new TreeSet<>(new FactoryComparator());
                factories.addAll(FusionBeanStore.getBeans(DataReaderFactory.class));
            }
        }
        return factories;
    }

Each DataReaderFactory is asked in turn to analyse the stream of data, and determine if the factory is able to read the data, if it can it returns a DataReaderEngine, if it can't it returns null.

    for(DataReaderFactory currentFactory : getDataReaderFactories()) {
          DataReaderEngine dre = currentFactory.getDataReaderEngine(sourceData, ...);
          if(dre != null) {
            return dre;
        }
    }

The plugin paradigm enables high-level manager classes to delegate work to any registered implementation of a specific interface. This is a commonly used design pattern in SDMX Core, and is used for managers that read and write data, structures, and reference metadata.

Fusion Modules¶

Fusion Modules are provided as a convenience mechanism for quickly setting up a system, by registering stateless instances to the SingletonStore and FusionBeanStore. Modules are called in the global configuration of an application.

All modules implement the IFusionModule marker interface, which enables modules to be easily discovered. The implementations of the IFusionModule interface are designed to set up a specific capability in the system, such as the ability to read and write SDMX content in JSON format.

Implementations of the IFusionModule are found in the java module's application package.

The following modules exist in SDMX Core.

Java Module	FusionModule	Description
fusion-core-sdmx	CoreSDMXModule	Register core singletons for SDMX work, includes registration of the DeferredModule, and SemverModule
fusion-core-sdmx	DeferredModule	??
fusion-core-sdmx	SemverModule	???
fusion-core-data	FusionCoreDataModule	is this required?
fusion-sdmx-json	FusionJsonModule	???
fusion-sdmx-json	SdmxJsonModule	???
fusion-sdmx-ml	SdmxMLModule	Register readers and writers for SDMX-ML (XML)
fusion-sdmx-csv	SdmxCsvModule	Register readers and writers for SDMX-CSV
fusion-sdmx-edi	SdmxEDIModule	Register readers and writers for SDMX-EDI
fusion-sdmx-publicationtable	PublicationTableModule	Register readers and writers for Publication Tables
fusion-sdmx-reporttemplate	ReportingTemplateModule	Register readers and writers for Report Templates
fusion-xl	FusionXLModule	Register readers and writers for FusionXL (excel plugin)

Example usage of a module to provide support for reading and writing SDMX-JSON.

    /**
     * read JSON structure file and print URN of all maintainable structures
     */
    public static void main(String[] args) {
        //Support JSON reading and writing
        SdmxJsonModule.register();
        String file = args[0];

        //The reader manager will automatically plugin the JSON
        //reader factory, required for reading SDMX-JSON
        StructureReaderManager srm = new StructureReaderManagerImpl();

        //Create a ReadableDataLocation on the file, autoclose
        try(ReadableDataLocation rdl = new ReadableDataLocationTmp(file)) {

          //Parse SDMX-JSON file into SdmxBeans container
            SdmxBeans beans = srm.parseStructures(rdl);

            //loop each maintainable structure and print URN
            beans.getAllMaintainables().forEach(m-> {
               System.out.println(m.getUrn());
            });
        }
    }

ApplicationStartupAware and Cache¶

The SDMX Core framework supports application startup aware classes through the interface ApplicationStartupAware and ApplicationStarter.

Any class which implements ApplicationStartupAware which is registered in the SingletonContainer or FusionBeanContainer (either directly or via Spring) will have the applicationStarted method called when the application has started. In order to make use of this feature an ApplicationStarter implementation must be used, as this is responsible for calling the applicationStarted methods on all ApplicationStartupAware implementations.

The SDMX Core framework provides an implementation of the ApplicationStartupAware interface. The ApplicationStartupAware implementation also calls the clearCache method of any registered class which implements the Cache interface; this is a useful feature if the application has a concept of restarting or resetting itself without the need to restart the JVM.

Exceptions¶

The SDMX Core framework has a common Exception class SdmxException all specific exception types extend this base class, for example SdmxNoResultsException. Each exception type contains a SDMX_ERROR_CODE which provides the SDMX exception Code expected in the response message.

Fusion System¶

FusionSystem provides static access to a global ReadableDataLocationFactory and WriteableDataLocationFactory, it also configures the system to use a MessageDecoder for decoding error codes into error messages from a properties file.

To use FusionSystem in an application call: FusionSystem.register();