Getting Started with Semarchy xDM Component

Overview

This article explains how to exchange data with Semarchy xDM.

The Semarchy xDM component is installed by default in Semarchy xDI but does not come with any predefined metadata or technology.
However, the component provides actions that can be added to processes to publish data to Semarchy xDM.

With this component:

  • You create the metadata by reverse-engineering the tables from the database schema hosting the data location, using that database’s technology (Oracle, SQL Server, or PostgreSQL).

  • You integrate, using mappings, data from and to the data location tables using the templates provided for that technology.

  • The Semarchy xDM component provides specific tools to handle load management operations required when publishing data in a Semarchy xDM data location.

Connect to your data and reverse-engineer the data structure

To create a Semarchy xDM metadata for a given data location:

  1. Launch the metadata creation wizard and select the database technology hosting the Semarchy xDM data location.

  2. Enter the database details and credentials, and then click Connect. getting started xdm server connection

  3. On the next page, on the Catalog Name, click Refresh, then select the name of your database from the list.

  4. On Schema Name, click Refresh and select from the list the schema containing the Semarchy xDM data location. getting started xdm schemas

  5. Click Next, refresh the list of tables, and select the ones to reverse-engineer.

  6. Click Finish. The tables are reverse-engineered in the metadata.

Create your mappings

This section explains how to create mappings with Semarchy xDM tables. These mappings are used in processes for extracting data from and loading data to Semarchy xDM.

Extract data from MD and GD tables

The data location’s golden data (GD) and master data (MD) tables are commonly used to consume records from the data location. To extract data from Semarchy xDM, use these tables as sources in your mappings.

Refer to Consume Data Using SQL in the Semarchy xDM documentation for more information about the patterns to consume master and golden data from Semarchy xDM.
Example of mapping to extract golden data from Semarchy xDM golden data table and load it into a table:

getting started xdm extract gd table

Load data into SD and SA tables

The Source Authoring (SA) and Source Data (SD) tables are used to load data into the data location.

Refer to Publish Data Using SQL for more information about loading data into Semarchy xDM using the SD and SA tables.

When loading data into the SD or SA tables of a data location, you must populate the business data columns according to your data flow requirements. You must also map the following columns:

  • B_CLASSNAME: Map this column with a literal value or a column containing the name of the entity that you want to load.

  • B_PUBID: For fuzzy and ID-matched entities, map this column to a literal value or a column containing the code of the publisher on behalf of whom you are publishing the data.

  • B_LOADID: Map this column with the Load ID, which is typically handled by the SEM_LOAD_ID variable. Refer to Use the load ID variable in your mappings for more information.

  • ID column: The ID column to map depends on the type of entity:

    • Basic entities: Column representing the primary key attribute for the entity.

    • ID matched entities: Column representing the primary key attribute for the entity.

    • Fuzzy matched entities: If the entity uses fuzzy matching, then load the value of the primary key coming from the source system into the B_SOURCEID column. If this primary key is composite, then concatenate the values of the composite primary key and load them in the B_SOURCEID column.

  • Reference columns: When loading data for entities that are related by a reference relationship, load the referencing entity with the value of the referenced primary key. The columns to load (F_, FS_, and FP_ columns) depend on the entity type of the referenced entity: for more details, refer to the References section of the Semarchy xDM SQL integration guide.

Use the load ID variable in your mappings

The SEM_LOAD_ID variable contains the Load ID that is used to load data into the Semarchy xDM data location. This variable is set when you call the Get LoadID action in a process, before the mappings.

To use the load ID in your mapping:

  1. In Project Explorer, select your project.

  2. Click Windows > Show View > Internal Resources.

  3. Click templates > templates.xdm > Global Variables.variable.md > Semarchy Variables.

  4. Drag and drop the SEM_LOAD_ID variable in the mapping.

  5. In the SA or SD target table, select the b_loadid column.

  6. In the Expression Editor view, add the %{SEM_LOAD_ID}% expression.

Example of mapping to load data from a database table into a source authoring (SA) table:

getting started xdm load sa table

Mappings loading the SA and SD tables must provide a Load ID. You must include these mappings in a process into which you will initialize and submit the load identified by this Load ID.

Create processes to publish data

Summary

The process to publish data to Semarchy xDM SD or SA tables is the following:

  1. Initialize a load identifying the set of records to publish. These records may span across multiple tables. The load is identified by a Load ID.

  2. Run mappings to Load data into SD and SA tables, mapping the Load ID to the B_LOADID column.

  3. Submit the load to Semarchy xDM. After submitting the load, you can follow the processing of the published data in the Semarchy xDM platform.
    You can also cancel the load if one of the mappings has failed.

The following image illustrates a process for loading data into a source data table:

getting started xdm process

Refer to Publish Data Using SQL for more information on publishing data into Semarchy xDM.

Definition

The Semarchy xDM component provides dedicated tools to handle the load management operations required when publishing data in a Semarchy xDM data location.

The following tools are available:

  • Get LoadID initializes a load and get a Load ID. This Load ID is stored in the SEM_LOAD_ID variable.

  • Submit Load submits a load and publish the data in the data location.

  • Cancel Load cancels a load.

  • Get Load Status gets the progress of a load after a submit.

Create a process and combine these tools to manage all the steps of your load:

  1. Select the appropriate tool from Palette > xDM Tools.

  2. Click the process background to add it.

  3. Drag and drop the database containing the Semarchy xDM repository onto the SOURCE Metadata Link placeholder.

  4. Fill in the parameters of the tool.

Example:

getting started xdm process tool overview

In the tools, the SOURCE Metadata Link refers to the database containing the Semarchy xDM repository.
Hence, when designing a process, ensure that you only drag and drop on the actions, the database containing the Semarchy xDM repository.
Depending on the implementation of your Semarchy xDM, this database may or may not be the same database as the one containing the data location.

Parameters

The following tables describe the most important parameters for each action.

Refer to the reference documentation of each action for the complete list of parameters.

Get LoadID

Parameter

Description

SemRepoSchema

Schema containing the Semarchy xDM repository tables.

DataLocationName

Name of the data location hosting the data hub into which the external load is to be performed.

UserName

Name of the user initializing the action. This user may or may not be a user-defined in the security realm of the application server.

Submit Load

Parameter

Description

SemRepoSchema

Schema containing the Semarchy xDM repository tables.

IntegrationJobName

Name of the integration job used to process data in this load.

UserName

Name of the user initializing the action. This user may or may not be a user-defined in the security realm of the application server.

Cancel Load

Parameter

Description

SemRepoSchema

Schema containing the Semarchy xDM repository tables.

UserName

Name of the user initializing the action. This user may or may not be a user-defined in the security realm of the application server.

MDMSchema

Name of the schema hosting the Semarchy xDM data location.