Metadata And Reverse Engineering

Semarchy xDI uses Metadata to design, generate and run the data integration processes.

What is A Metadata?

A Metadata in Semarchy xDI is a design-time object used in mappings and processes. Depending on its type, metadata:

  • Defines connection parameters to a datastore or a remote server.

  • Describes the data structure and format of datastores (for example, tables in a relational database).

  • Stores information used in mappings or processes.

A metadata can be used in mappings or processes to represent an inbound or outbound data store and map data according to its data model.

Metadata Types

Several types of metadata are available in Semarchy xDI. The main types are:

  • Server with Data structure: Defines the connection parameters to a data store and the format of the data it manages. For example, RDBMS (relational database), File Server, REST API, etc.

  • Data structure: Defines the format of structured data. For example, XML, JSON, Avro, Parquet, etc.

  • Server: Defines the connection parameters to a server.For example, Mail Server, LDAP Server, SSH Server, etc.

  • Security: Defines security parameters (used by other metadata). For example, Kerberos, Proxy, HTTP Security, etc.

Other types of metadata offer specific features:

  • Runtime: Allows creating Runtimes that are available for selection:

    • In the Runtime and Navigator views.

    • In processes (case of the Execute Delivery action).

  • Variable: Defines a Variable Set. Variables from a Variable Set can be used in mappings and can optionally be persisted in a database. Refer to the Variables for more information.

  • Function: Defines a library of user-defined functions that can be called in mappings.

  • Generic: Allows designers to define a structured element from which data can be retrieved by processes.

  • Object Selector: References elements from other metadata to facilitate the design-time experience.

Shapes of Metadata

Depending on the metadata that is added to a mapping, the following types of data stores can be managed:

  • Databases: Semarchy xDI supports numerous relational and NoSQL database technologies.

  • Files: Flat and Hierarchical (E.g. JSON, XML)

  • API: REST and SOAP

  • Applications such as SAP, Salesforce, etc.

Create A Metadata

To create a metadata:

  1. Click the metadata button button in the Project Explorer's toolbar. Alternatively, right-click the parent project or folder in which you wish to create the new metadata and click New > Metadata.

  2. Select the metadata technology and click Next. metadata wizard

  3. Confirm the parent folder for your new metadata, enter a Metadata Model name, and then click Next.

  4. If the selected type of metadata requires you to select a module, the Module Selection/Creation step opens:

    module selection

    The Type pane contains all the categories of modules that represent all types of technologies provided in the current installation. The list is filtered to only show the relevant categories for the selected technology.

    This list contains the following categories:

    • Default: This category only contains one empty module called default. This module is available for any technology and is used as a fallback module at execution when no module is specified for metadata.

    • Generic: This category contains modules that are not linked to any particular technology and can thus be used by any metadata.

    • <Technology name>: This category contains all the modules for a given technology.

      Refer to the Modules for more information on managing modules in a workspace.

      1. Select a category in the Type pane. The Module pane is filtered based on the selected category.

      2. Select a module in the Module pane. The content of the module is displayed in the Details pane.

      3. If needed, add files to the selected module by clicking the add button.

      4. Click Finish.

  5. If the created metadata is of type Server or Server With Data Structure, a dialog opens to enter the connection parameters such as URL and credentials. Enter those parameters and click:

    • Next to configure and launch the Reverse-Engineering of the new metadata. Refer to Reverse-Engineering for more information.

    • Finish to complete the creation of the metadata without reverse-engineering its model. metadata wizard

In the New Metadata wizard, you can only select the metadata types corresponding to components allowed by your license. The types that are not allowed appear disabled. The Show only available option hides the metadata types that are not allowed.
You can configure External Value Resolvers to retrieve sensitive information (usernames, passwords) from an external secret store. With this feature, instead of storing sensitive information in the metadata, you store references to secrets stored in an external secret store. For example in HashiCorp Vault, AWS Secrets Manager Azure Key Vault, Google Cloud Secret Manager, etc.

Define The Data Model

Reverse-Engineering

Reverse-engineering is the most common way to build the data model of metadata in Semarchy xDI. This mechanism connects to the data point (database server, file system, etc.) to retrieve its data structure (tables, files, etc.).

Reverse-engineering can be run:

  • By the metadata creation wizard. Refer to [Create Metadata] for more information.

  • Manually, by clicking Actions > Launch Dataschema Wizard from the contextual menu in the metadata editor. dataschema wizard

The following sections explain with examples how to create metadata for the most common types of metadata:

Define a Generic Model

A Generic Model is used to describe a specific data structure that can be used in mappings and processes.

To create a Generic Model:

  1. Click the metadata New Metadata button in the Project Explorer toolbar. The New Model wizard opens.

  2. Select the Advanced option to display all types of metadata.

  3. In the Choose the type of Metadata tree, select Generic > Element.

    generic element

  4. Click Next.

  5. Select the parent folder or project for the new resource.

  6. Enter a Name for your new metadata file and click Finish. The metadata editor opens.

  7. Select the Element node and enter the Name for this element in the Properties view.

A Generic Model is a hierarchy of elements and attributes. Attribute values can be retrieved for an element using the Semarchy xDI usual Xpath syntax.

To create a new element:

  1. Right-click the parent element.

  2. Click New > Element.

  3. In the Properties view, enter the name of the new element.

To add an attribute to an element:

  1. Right-click the parent element.

  2. Select New > Attribute.

  3. In the Properties view enter the Name and the Value of the attribute.

Update a Metadata

Designers can update the properties as well as the data structure (if applicable) of metadata objects. To open the metadata editor, double-click the metadata resource in Project Explorer, or click Open in its contextual menu.

Modify Metadata Properties

To modify the properties of a metadata:

  • Edit properties from the metadata editor: all properties are shown in finger tabs. The list of available properties depends on the type of metadata.

    • Properties with custom user values are enabled in the interface and can be modified directly.

    • Properties with default or undefined values are disabled. Click the property’s label to enable and then define a value (Leave it empty to define an empty value).

    metadata properties

  • Alternatively, you can launch the configuration wizard if available: right-click the root node of the metadata in the editor and then select Actions > Launch Server Wizard. The list of available wizards vary depending on the type of metadata.

Modify the Data Model of Metadata

The data model of metadata can be modified:

  • By reverse-engineering: In the metadata editor, right-click the root node and click Actions > Launch DataSchema wizard.

  • Manually: Double-click a node (a table or a column in the case of database metadata) in the metadata editor. All the editable properties for this node are shown in finger tabs that depend on the type of metadata. metadata properties

Change the Module used by Metadata

To change the module used by a single metadata:

  1. Double-click the metadata from Project Explorer to open its editor.

  2. Select the root node in the editor.

  3. In the Properties view, modify the Module property.

  4. Save your changes.

Metadata Module Manager

The Metadata Module Manager allows changing the module used by several metadata at once.

This wizard only allows assigning modules to metadata. If you want to prepare modules for technologies that are not used by any metadata yet, use the Module Manager.

To assign a module used to a set of metadata:

  1. Navigate to Utilities > Metadata Module Manager. The Metadata Module Manager opens.

  2. In the left panel of the Metadata Module Manager, select a category (for example, PostgreSQL).

    The categories are filtered based on the metadata available in the current workspace.

    The list of metadata from the current workspace that corresponds to the selected category is displayed in the right panel. The table view also shows the module used by each metadata.

  3. Click the browse button to select the module you want to assign.

    If a module already exists for a given category, it is preselected in the Select a Module to apply field.

  4. Select the checkbox of each metadata to which you want to apply this module.

  5. Click Perform Update.