Retrieving Structures¶

Overview¶

sdmx-core has been designed to decouple the physical location of the structure from the business logic of the application. This enables functional units to be designed which are not tightly coupled to a single application; for example a data validation function, which requires an input dataset combined with the structural metadata used to define the dataset - this type of function should be designed such that it could be added into any system easily, by simply plugging in the structure retrieval function into the system's own architecture.

The structure retrieval feature relies on two concepts:

Having a standardised reference mechanism for structures
Having a small set of simple interfaces for retrieving structures

The first concept is satisfied by leveraging the SDMX Uniform Resource Name.

The second concept is satisfied through simple Java interfaces for structure retrieval, principally the SdmxBeanRetrievalManager.

Structure Reference¶

The Uniform Resource Name (URN) is central to SDMX for referencing structural metadata.

Every IdentifiableBean has a URN, it is always generated on construction based on the Structure Type, Agency, Maintainable ID, Version, and Identifiable ID (if applicable)

    urn:sdmx:org.sdmx.infomodel.codelist.Code=SDMX:CL_FREQ(1.0).A

A URN uniquely identifies a structure in a system, and is used as a means to reference and query for structures. In official SDMX services, the URN is a globally unique identifier resolved through the URN Resolver Service.

---
title: URN Class Diagram
---
classDiagram
  IMaintainableRef <|-- IURN
  IMaintainableRef *-- IVersionRef
  IVersionRef *-- ISdmxVersion

  class IURN~IdentifiableBean~ {
     +SDMX_STRUCTURE_TYPE structureType
  }

  class IMaintainableRef {
     +String id
     +String agencyId
     +IVersionRef version
  }


  class IVersionRef {
     +ISdmxVersion minVersion
     +ISdmxVersion maxVersion
     +boolean isLatest
     +boolean isAbsolute
     +boolean isAll
  }

  class ISdmxVersion {
     +int major
     +int minor
     +int patch
     +String suffix
  }

Creating a URN¶

A URN is a property of a structure,

URNs can be created programmatically, using the URN builder

    IURN<CodeBean> codeUrn = URN.builder().agencyId("SDMX")
                                          .maintId("CL_FREQ")
                                          .version("1.0")
                                          .identifableId("A")
                                          .build(CodeBean.class);

Specialised Subtypes¶

IURN is a generic super type, it refers to any URN which could be either a URN that has all properties set, or a URN that contains wildcard parts and is used for querying multiple structures.

For methods that need to be more precise in what they expect, there are specialisations of the URN which enforce certain properties to be present, or set to certain values. The specialised types fall into two broad categories:

URNs that resolve to a a single structure
URNs that can resolve to multiple structures

URN resolving to a Single Structure¶

A URN that resolves to a single structure is an IURNSingle. The system ensures at build time that the URN does not contain wildcard references, to ensure it can only result in a single artefact being returned.

The IURNSingle has 3 specialised type:

IURNAbsolute has an absolute version number, e.g 1.0.0
IURNSemantic has a version number with a semantic reference, e.g. 1.0+.0
IURNMaintainable is an IURNAbsolute URN which resolves to a MaintainableBean

---
title: URN Single
---
classDiagram
  IURN <|-- IURNSingle

  IURNSingle <|-- IURNAbsolute
  IURNAbsolute *-- IURNMaintainable
  IURNAbsolute <|-- IURNMaintainable
  IURNSingle <|-- IURNSemantic

  class IURN~? extends IdentifiableBean~ {
     +SDMX_STRUCTURE_TYPE structureType
  }

  class IURNSingle~? extends IdentifiableBean~ {
     +IURNMaintainable~? extends MaintainableBean~ maintainableUrn
  }

  class IURNAbsolute~? extends IdentifiableBean~ {
     +ISdmxVersion absoluteVersion
     +IURNMaintainable maintainableUrn
  }

  class IURNSemantic~? extends IdentifiableBean~ {
     +ISdmxVersion absoluteVersion
  }

 class IURNMaintainable~? extends MaintainableBean~ {
 }

Note that an IURNAbsolute applies to any IdentifiableBean, which describes any structure that has a unique identity, including Codes and Concepts as well as Codelists and Concept Schemes. A MaintainableBean can always be derived from a URN, for example a Codelist URN can be derived from a Code URN; this is why the IURNAbsolute has the maintainableUrn property even if it is itself a MaintainableURN.

The IURNSingle is used when resolving a single structure of any type without knowing if it is a semantically versioned structure, an absolute reference, or a reference to a maintainable structure

private IdentifiableBean getStructure(SdmxBeanRetrievalManager brm, String urn) {
    IURNSingle<?> anyIdentifiableURN = URN.builder(urn)
                                          .buildSingle();

    return brm.getBean(anyIdentifiableURN);
}

The IURNSingle can also be a reference to a specific type of structure, by passing the structure class into the build method. The sdmx-core library ensures the URN package and class are the correct package and class for the type being built, and all the identifiers are present, for example a Code reference must include the ID of both the SDMX Codelist and the Code.

private static CodeBean getCode(SdmxBeanRetrievalManager brm, String urn) {
    IURNSingle<CodeBean> codeURN = URN.builder(urn)
                                      .buildSingle(CodeBean.class);

    return brm.getBean(codeURN);
}

Abstract class types can be requested; in this instance the sdmx-core framework builds the target URN and ensures the target class implements the abstract interface

private static ItemSchemeBean<?> getItemScheme(SdmxBeanRetrievalManager brm, String urn) {
    IURNSingle<ItemSchemeBean> itemSchemeURN = URN.builder(urn)
                                                  .buildSingle(ItemSchemeBean.class);

    return brm.getBean(itemSchemeURN);
}

Specialised subtypes can be built if more certainty over the reference type is required

private static IURNAbsolute<CodeBean> findAbsolute(SdmxBeanRetrievalManager brm, String urn) {
    IURNSingle<CodeBean> codeURN = URN.builder(urn)
                                      .buildSingle(CodeBean.class);
    CodeBean code = brm.getBean(codeURN);
    if(code == null) {
        throw new SdmxNoResultsException("No results for URN:" +urn);
    }
    return code.getURN();
}

The URN can also be constructed by individual properties

    IURNAbsolute<CodelistBean> clURN = URN.builder().agencyId("SDMX")
                                                    .maintId("CL_FREQ")
                                                    .version("1.0")
                                                    .buildAbsolute(CodelistBean.class)

The sdmx-core framework verifies the URN on build, if an absolute URN is built which is missing parts, for example no version is supplied, the build will fail with an SdmxSemmanticException

    //Error no version supplied
    IURNAbsolute<CodelistBean> error = URN.builder().agencyId("SDMX")
                                                    .maintId("CL_FREQ")
                                                    .buildAbsolute(CodelistBean.class)

URN resolving to a Multiple Structures¶

A URN which resolves to more than one structure is a URN that either has missing (wildcard) properties, for example the Agency ID is left empty, or where the URN contains multiple options, for example the Agency ID is BIS or IMF.

The two specialisations for these behaviours are:

IURNWildcard contains at least one wildcard property
IURNMultiple contains at least one property with multiple values, and may contain wildcard properties also

---
title: URN Single
---
classDiagram
  IURN <|-- IURNWildcard
  IURN <|-- IURNMultiple
  IURNMultiple*--IMultiMaintainableRef
  IMultiMaintainableRef*--IVersionRef

  class IURN {
     +SDMX_STRUCTURE_TYPE structureType
  }

  class IURNWildcard {

  }

  class IURNMultiple {
     +List~String~ identifiableIds
     +IMultiMaintainableRef multiReference
  }


 class IMultiMaintainableRef {
   +List~String~ agencyId
   +List~String~ id
   +IVersionRef version
 }

class IVersionRef {
   +ISdmxVersion minVersion
   +ISdmxVersion maxVersion
   +boolean isLatest
   +boolean isAbsolute
   +boolean isAll
}

Structure Retrieval¶

The SdmxBeanRetrievalManager is the cornerstone of sdmx-core for structure retrieval. The API takes a IURN, which defines the lookup criteria, and returns a structure or structures that match the request.

private static CodeBean getCode(SdmxBeanRetrievalManager brm, String urn) {
    IURNSingle<CodeBean> codeURN = URN.builder(urn)
                                      .buildSingle(CodeBean.class);

    return brm.getBean(codeURN);
}

There are a number of implementations of this interface, as structures tend to come from many different sources.

In-Memory¶

The simplest implmentation of the SdmxBeanRetrievalManager is the SdmxBeans container.

public static void main(String[] args) {
  //required to read Global Registry structures
    SdmxMLModule.register();

    printURN();
}

private static void printURN() {
    String url = "https://registry.sdmx.org/sdmx/v2/structure/dataflow/";
    SdmxBeans beans = getStructures(url);
    printURN(beans);
}

private static SdmxBeans getStructures(String url) {
    StructureReaderManager srm = new StructureReaderManagerImpl();
    try (ReadableDataLocation rdl = new ReadableDataLocationTmp(url) ){
        return srm.parseStructures(rdl);
    }
}

private static void printURN(SdmxBeanRetrievalManager brm) {
     IURNSingle<DataflowBean> urn = URN.builder()
                                      .agencyId("ESTAT")
                                      .maintId("BPM6_BOP_M")
                                      .version("2.0+.0")
                                      .buildSingle(DataflowBean.class);

    DataflowBean flow = beans.getBean(urn);
    if(flow == null) {
        throw new SdmxNoResultsException("No results for URN:" +urn);
    }
    //Prints URN of latest v2 dataflow
    System.out.println(flow.getUrn());
}

REST API¶

The RESTSdmxBeanRetrievalManager converts a URN to an SDMX web service call, the web service is queried with the web service response converted into the requested structure(s)

public static void main(String[] args) {
  //required to read Global Registry structures
    SdmxMLModule.register();

    printURN();
}

private static void printURN() {
   String url = "https://registry.sdmx.org/sdmx/v2";
   REST_API_VERSION apiVersion = REST_API_VERSION.v2_0_0;
   RESTSdmxBeanRetrievalManager api = new RESTSdmxBeanRetrievalManager(url, apiVersion);

   printURN(beans);
}

private static void printURN(SdmxBeanRetrievalManager brm) {
     IURNSingle<DataflowBean> urn = URN.builder()
                                      .agencyId("ESTAT")
                                      .maintId("BPM6_BOP_M")
                                      .version("2.0+.0")
                                      .buildSingle(DataflowBean.class);

    DataflowBean flow = beans.getBean(urn);
    if(flow == null) {
        throw new SdmxNoResultsException("No results for URN:" +urn);
    }
    //Prints URN of latest v2 dataflow
    System.out.println(flow.getUrn());
}

Other Implementations

Other SdmxBeanRetrievalManager implementations include

MaintainableDao get structures from database table, used by the FMR as its source of structures
MultipleBeanRetrievalManager proxy on more than one SdmxBeanRetrievalManager for lookup across a group of sources
CachingSdmxBeanRetrievalManager proxy which caches results
OverridingBeanRetreivalManager proxy where the proxy target can be swapped out
UnpersistedBeanRetreivalManager proxy which allows local beans to be stored on the thread
SecureSdmxBeanRetrievalManager proxy which adds a security layer on the proxy target
EmptySdmxBeanRetrievalManager always empty