Metadata And Reverse Engineering

Semarchy xDI uses Metadata to design, generate and run the data integration processes.

What is a Metadata?

A Metadata in Semarchy xDI is a design-time object used in mappings and processes. Depending on its type, a metadata:

  • Defines connection parameters to a datastore or a remote server.

  • Describes the data structure and format of datastores (for example, tables in a relational database).

  • Stores information used in mappings or processes.

A metadata can be used in mappings or processes to represent an inbound or outbound data store and map data according to its data model.

Metadata Types

Several types of metadata are available in Semarchy xDI. The main types are:

  • Server with Data structure: Defines the connection parameters to a data store and the format of the data it manages. E.g. RDBMS (relational database), File Server, REST API…​

  • Data structure: Defines the format of structured data. E.g. XML, JSON, Avro, Parquet…​

  • Server: Defines the connection parameters to a server. E.g. Mail Server, LDAP Server, SSH Server…​

  • Security: Defines security parameters (used by other metadata). E.g. Kerberos, Proxy, HTTP Security…​

Other types of metadata offer specific features:

  • Runtime: Allows creating Runtimes that are then available for selection:

    • In the Runtime and Navigator views.

    • In processes (case of the Execute Delivery action).

  • Variable: Defines a Variable Set. Variables from a Variable Set can be used in mappings and can optionally be persisted in a database. For more details, refer to the Variables section.

  • Function: Defines a library of user-defined functions that can be called in mappings.

  • Generic: Allows designers to define a structured element from which data can be retrieved by processes.

  • Object Selector: References elements from other Metadata in order to facilitate the design-time experience.

Shapes of Metadata

Depending on the metadata that is added to a mapping, the following types of data stores can be managed:

  • Databases: Semarchy xDI supports numerous relational and NoSQL database technologies.

  • Files: Flat and Hierarchical (E.g. JSON, XML)

  • API: REST and SOAP

  • Applications such as SAP, Salesforce, etc.

Create a Metadata

To create a new metadata:

  1. Click the metadata button button in the Project Explorer’s toolbar. Alternatively, right-click the parent project or folder where to create the new metadata and select New > Metadata.

  2. Select the metadata’s technology and press Next. metadata wizard

  3. Confirm the parent folder for your new metadata, enter a Metadata Model name, and then press Next.

  4. If the selected type of metadata requires to select a module, the Module Selection/Creation step opens:
    module selection

    The Type pane contains all the categories of modules, which represent all types of technologies provided in the current installation. The list is filtered to only show the relevant categories for the selected technology.

    This list contains the following categories:

    • Default: This category contains only one empty module called default. This module is available for any technology and is used as a fallback module at execution when no module is specified for a metadata.

    • Generic: This category contains modules that are not linked to any particular technology and can thus be used by any metadata.

    • <Technology name>: This category contains all the modules for a given technology.

      To manage modules in a workspace, refer to the Modules section.
      1. Select a category in the Type pane. The Module pane is filtered based on the selected category.

      2. Select a module in the Module pane. The content of the module is displayed in the Details pane.

      3. If needed, add files to the selected module by clicking the add button.

      4. Click Finish.

  5. If the created metadata is of type Server or Server With Data Structure, a dialog opens to enter the connection parameters such as URL and credentials. Enter those parameters and then press:

    • Next to configure and launch the Reverse-Engineering of the new metadata (see Reverse-Engineering).

    • Finish to complete the creation of the metadata without reverse-engineering its model. metadata wizard

You can only select in the New Metadata wizard the metadata types corresponding to components allowed by your license. The types not allowed appear disabled. The Show only available option hides the metadata types that are not allowed.
You can configure External Value Resolvers to retrieve sensitive information (usernames, passwords) from an external secret store. With this feature, instead of storing sensitive information in the metadata, you store references to secrets stored for example in HashiCorp Vault, AWS Secrets Manager Azure Key Vault, Google Cloud Secret Manager, etc.

Define The Data Model

Reverse-Engineering

Reverse-engineering is the most common way in Semarchy xDI to build the data model of a metadata. This mechanism connects to the data point (database server, file system, etc.) to retrieve its data structure (tables, files, etc.).

Reverse-Engineering can be run:

  • By the metadata creation wizard (see Create a Metadata).

  • Manually, by selecting Actions > Launch Dataschema Wizard from the contextual menu in the metadata editor. dataschema wizard

The following sections explain how to create metadata with examples for the most common types of metadata:

Define a Generic Model

A Generic Model is used to describe a specific data structure that can be used in mappings and processes.

To create a Generic Model:

  1. Click on the metadata New Metadata button in the Project Explorer toolbar. The New Model wizard opens.

  2. Select the Advanced option to display all types of metadata.

  3. In the Choose the type of Metadata tree, select Generic > Element.

    generic element

  4. Click Next.

  5. Select the parent folder or project for the new resource.

  6. Enter a Name for your new metadata file and then click Finish. The metadata editor opens.

  7. Select the Element node and enter the Name for this element in the Properties view.

A Generic Model is a hierarchy of elements and attributes. Attribute values can be retrieved for an element using the Semarchy xDI usual Xpath syntax.

To create a new element:

  1. Right-Click on the parent element.

  2. Select New > Element.

  3. In the Properties view, enter the name of the new element.

To add an attribute to an element:

  1. Right-Click on the parent element.

  2. Select New > Attribute.

  3. In the Properties view enter the Name and the Value of the attribute.

Update a Metadata

Designers can update the properties as well as the data structure (if applicable) of metadata objects. To open a metadata’s editor, double-click this metadata resource in the Project Explorer, or select Open in its contextual menu.

Modify Metadata Properties

To modify the properties of a metadata:

  • Edit properties from the metadata’s editor: all properties are shown in finger tabs. The list of available properties depends on the type of metadata. metadata properties

  • Alternatively, you can launch the configuration wizard: right-click the root node of the metadata in the editor and then select Actions > Launch Server Wizard.

Modify the Data Model of a Metadata

The data model of a metadata can be modified:

  • Manually: Double-click a node (like a table or a column in the case of a database metadata) in the metadata editor. All the editable properties for this node are shown in finger tabs that depend on the type of metadata. metadata properties

  • By reverse-engineering: In the metadata’s editor, right-click the root node and then select Actions > Launch DataSchema wizard.

Change the Module used by a Metadata

To change the module used by a single metadata, proceed as follows:

  1. Double-click this metadata from the Project Explorer to open its editor.

  2. Select the root node in the editor.

  3. In the Properties view, modify the Module property.

  4. Save the editor.

The Metadata Module Manager

The Metadata Module Manager allows changing the module used by several metadata at once.

To assign a module used to a set of metadata:

  1. Go to Utilities > Metadata Module Manager, the Metadata Module Manager opens.

  2. Select a category in the left panel of the Metadata Module Manager (for instance, PostgreSQL).

  3. The list of metadata from the current workspace that corresponds to the selected category is displayed in the right panel. The table view also shows the module used by each metadata.

  4. Click the browse button to select the module you want to assign.

  5. Tick the checkbox of each metadata you want to apply this module to.

  6. Click Perform Update.

If a module already exists for a given category, it is preselected in the Select a Module to apply field.
You can select another module using the browse button.
The categories in the left panel are filtered based on the metadata available in the current workspace.
This wizard is used only to assign modules to metadata. If you want to prepare modules for technologies that are not used by any metadata yet, use the Module Manager.