This tutorial teaches you how to build your first data consolidation application.

You will load data from multiple operational systems into the data hub where it will be matched and merged. In this first unit, you will create a very simple model with a simple match rule. Later, you will build on this foundation.

What you'll learn

Before you start

This tutorial is the first unit of the Data Consolidation track.

In this tutorial, we assume that you already set up a working Semarchy xDM environment and that you have design-time privileges to connect to the Semarchy Application Builder.

If you have not yet set up Semarchy xDM, follow first the Quick Install tutorials from the main tutorials menu.

GO TO TUTORIALS

Additional setup information can also be found in the installation guide.

Click on the button DOWNLOAD TUTORIAL RESOURCES to get the required resources, including the images and datasets used within this tutorial.

DOWNLOAD TUTORIAL RESOURCES

The estimated duration of this unit is about 1 hour and 40 minutes.

Semarchy xDM uses an entity-relationship diagram to help you create the logical model of your application.

What you'll learn

Create a model

To get started, you need to create a new logical model. The logical model will hold all of the metadata for your entities, attributes, rules, forms, collections, and applications.

  1. On the Application Builder main page, click on "New Model...":

  1. Enter B2BTutorial in the Name field and click Finish.

  1. Right-click on Diagrams and select Add Diagram:

  1. Enter MainDiagram in the Name field and click Finish.


Add the Company entity

You are now ready to create the Company entity. At the end of the creation process, it will be configured as follows:

Attribute

Datatype

Mandatory?

Notes

ID

Sequence

Yes

Auto-numbered ID

CompanyName

String(128)

Yes

None

Website

String(128)

No

None

Region

List of Values

No

Refers to RegionLOV

Address1

String(128)

No

None

PostalCode

String(128)

No

None

City

String(128)

No

None

State

String(128)

No

None

Country

List of Values

No

Refers to CountryLOV

To add a new entity to your model, the fastest way is to add it to your diagram.

  1. From the Diagram editor, click Add Entity, and then click on the diagram itself:

  1. Enter the following values in the first page of the wizard, and then click Next:

  1. On the next page, enter the following values to configure the primary key of your entity, and then click Finish:

Add a simple attribute

  1. On the Diagram editor, right-click on the Company entity, and then select Add Simple Attribute:

  1. Enter the following values, and then click Finish:

  1. Save all your work: click the Save all current contents button on the top of the Model Design View.

Add Lists of Values

You are now going to add 2 new Lists of Values (LOV) that you will use for the Region and Country attributes.

Create the Region LOV

  1. Right-click on List of Values in the Model Design View and select Add List of Values.

  1. Enter RegionLOV as Name, then click the Add Value button to add the first value of your LOV:

  1. Enter APAC for the Code and keep the auto-generated Label unmodified:

  1. Do the same for the other values:

Code

Label

EMEA

EMEA

NORTH_AMERICA

North America

SOUTH_AMERICA

South America

  1. Click Save.

Add the Region attribute to the Company entity

Let's now add to the Company entity the attribute that will use the Region LOV.

  1. Go back to the Main Diagram: select the Main Diagram editor tab or double-click on Diagrams>Main Diagram in the Model Design View:

  1. Right-click on the Company entity and select Add Simple Attribute.

  1. Enter the following values and click Finish:

Create the Country LOV

We will now create the CountryLOV list of values using an Excel file.

  1. Proceed as before to create this second list of values.

  1. Back to the List of Values editor, click the Import values button:
  1. Click the Browse button and select (or drag and drop) the data-consolidation\datasets\countries-lov.xlsx file from the Tutorial resources you downloaded earlier.

  1. Click Finish.
  2. Review the imported values:

  1. Save your work.

Add the Country attribute to the Company entity

Add the Country simple attribute to your entity the same way you did for Region, using the following values:

Add the other attributes

It is now time to add the other attributes to your entity.

  1. Following the same instructions you used for CompanyName, add the following attributes to the Company entity:

Attribute

Datatype

Mandatory?

Website

String(128)

No

Address1

String(128)

No

PostalCode

String(128)

No

City

String(128)

No

State

String(128)

No

  1. Return to the MainDiagram and click Show data types in the button bar to display the data types next to attribute names. Review your entity and check that all the required attributes have been added:

  1. Save your work.

Assign an icon to the entity

It is a best practice to assign icons to your entities. These icons will visually guide users in several places in your application.

  1. From the MainDiagram, click on the Company entity.
  2. In the Properties view below the editor, identify the Icon property within the Name and Description tab, and then click Select Image:

  1. Type "domain" in the search bar and press Enter to filter the images.
  2. Select the mdi/domain.svg image:

  1. Save your work.

Congratulations

You have successfully completed the first step of this tutorial by creating the main entity.

What we've covered

In the next step, you will add a second entity and link it with the Company entity.

You are now ready to create the Industry entity as described below:

Attribute

Datatype

Mandatory?

Notes

ID

String(20)

Yes

ID is set manually

Title

String(128)

Yes

Name of the Industry

ShortDescription

String(500)

No

Description of this industry in paragraph form

Industries in this model are reference data used to classify Companies. Here you will use the top level of the NAICS industry classification standard.

In the next steps, you will:

Create the Industry Entity

To create the Industry entity, follow the same process as for the Company entity:

  1. Add the entity to the diagram with the following values and click Next:

  1. Configure the entity's primary key attribute on the next page:

  1. Add the Title and ShortDescription attributes as you did previously for Company.

Attribute

Mandatory?

Datatype

Notes

Title

Yes

String(128)

Name of the Industry

ShortDescription

No

String(500)

Description of this industry in paragraph form

Assign an icon

Proceed the same way you did for Company to assign an icon for the Industry entity:

  1. From the MainDiagram, click on the Industry entity.
  2. In the Properties view, identify the Icon property within the Name and Description tab, and click Select Image:

  1. Type "format-list-bulleted-type" in the search bar to find and select mdi/format-list-bulleted-type.svg:

Create the Company-Industry reference

From the Main Diagram, create the reference relationship named CompaniesIndustry, from Company to Industry:

  1. Click Add Reference in the Palette.
  2. Click on the Company entity and drag the line to the Industry entity:

  1. Leave all values unmodified on the Create New Reference Relationship dialog, and then click Finish.

Congratulations

You have successfully completed the second step of this tutorial by adding another entity to your model and creating a reference relationship.

What we've covered

In the next step, you will configure record matching and survivorship.

In this step, you will configure the match and survivorship rules that will be used by your application.

What you'll learn

Matching data in Semarchy xDM

Matching in Semarchy xDM is performed on master records coming from source systems. It is a fundamental step of the Certification Process that aims at consolidating these master records into unique golden Records.

In the case of Fuzzy-Matched entities (such as the Company entity you created in the first step), Master and Golden records use different identifiers:

Match Rules define how incoming source records match together.

You control the rule details and the thresholds defining when matches merge and when merged records should be confirmed.

Once master records have been matched, Survivorship Rules define how their attributes will consolidate into the resulting golden record.

Create a match rule

It is now time to create your first match rule.

  1. In the Application Builder Model Design View, go to Entities>Company>Matcher.
  2. Right-click on Matcher and select Define SemQL Matcher:

  1. The description is optional. Click Finish.

  1. On the SemQL Matcher editor, click Add Match Rule:

  1. Set the Name of your first match rule to PerfectMatch.

  1. Scroll down the editor and identify the Match Condition field. Click Edit Expression:

  1. Copy and paste the following SemQL Condition in the SemQL editor, and then click OK.
Record1.CompanyName = Record2.CompanyName 
AND Record1.Address1 = Record2.Address1 
AND Record1.PostalCode = Record2.PostalCode 
AND Record1.Country = Record2.Country
  1. Click on SemQLMatcher - Company in the breadcrumb above the MatchRule editor to return to the SemQL Matcher editor:

  1. Scroll down to the Merge Policy section of the editor. Keep the default value for all thresholds. With this configuration, all matches will merge.

  1. Save your work.

Later you will add some nuance to handle fuzzy matches and suggestions.

Add publishers

Publishers are applications that provide source data to the data hub. They are commonly used in survivorship rules, but can also be used in match rules, enrichers, and in other locations in your model.

In this section, you will define 4 publishers, one for each source system that will submit data to xDM.

  1. Right-click on Publishers in the Model Design View and select Add Publisher.

  1. Enter CRM for the Name. Keep the Publisher Code and Label fields unmodified, and then click Finish.

  1. Repeat the previous steps to create 3 other publishers:
  1. Save your work.

Configure a survivorship rule

It is time to set up your first survivorship rule. A default rule was created automatically with the entity. You will edit this rule to define the level of trust you place in each of your publishers.

  1. To configure the default survivorship rule, go to Entities>Company>Survivorship Rules in the Model Design View and double-click DefaultRule.

  1. Scroll down to the Consolidation Rule section.
  2. Set Consolidation Strategy to Preferred Publisher to select and order your publishers manually.

  1. Click the Select Publishers button next to the Prefered Publishers property:

  1. Click the Edit Expression button next to the Ranking Expression property:

  1. Enter the following SemQL expression, to consolidate the value from the most recent record in case of a tie, and then click OK.
UpdateDate DESC

  1. Finally, set the Override Strategy property to Override - until next user change. With this configuration, changes by the data stewards are allowed and take precedence over the source systems:

  1. Save your work.

Later you will extend these to handle more complex requirements.

Congratulations

You have successfully completed the third step of this tutorial by configuring matching and survivorship rules.

What we've covered

In the next step, you will start to define the user experience for your application. You will learn how to add forms, collections, and more.

Semarchy xDM uses Material Design as the visual language for defining a business user interface. The user interface uses several application components that you will define in the next sections. These include display cards, forms, collections, steppers, search forms, action sets, business views, application actions, and more.

What you'll learn

Application components

Data Applications in Semarchy xDM rely on several components that you can directly generate and configure in the Application Builder. The following examples are taken from the Product Retail demo application.

Focus on Display Cards

Display Cards define a visual representation of a record in a nicely formatted way. They can be used in collections or in fields to offer a compact view of this entity.

When generating application components, you will be prompted to define how you want your records to be displayed in Display Cards. You will define SemQL expressions that will translate into formatted text for your primary text and secondary text. Here is how Display Cards will be configured for the Company and Industry entities:


Generate components

You will now use the Generate Application Components wizard to automatically generate most of the components you will need to enable user interaction with your data.

  1. In the Model Design View of the Application Builder, right-click on the Company entity under Entities and select Create Application Components.

  1. Keep all default settings and press Next.

  1. Observe on the next page that a default Primary Text Expression was set: SEM_TO_CHAR2(ID). This expression returns the ID converted to a string.

  1. Set the Primary Text Expression:

  1. Set the Secondary Text Expression: follow the same procedure as for the Primary Text Expression and select the FDN_Industry attribute, or paste FDN_Industry into the text area.
  2. Click Finish.

  1. Repeat the instructions from step 1 for the Industry entity, using the following expressions for the Display Card:
  1. Save your work.

Explore what was generated

The wizard has generated all the required objects for the Company entity:

Expand the Company entity to see all generated components:

Scroll down in the Model Design View and go to Applications>DefaultApplication.

Double-click on Folders and Actions and observe that a default list of menu items was initialized, including a BrowseCompany action:

Improve your application

In this section, you will improve your application before deploying it. You will:

Enable master data import

You are now going to configure the Import action for the Company entity so that it supports importing master records instead of golden records, which is the default setting.

  1. Navigate to Entity>Company>Actions Sets and double-click on CompanyActionSet to open the editor for the generated Action Set.

  1. Select the ImportAuthorCompanies action in the Actions list. This opens the Properties view Inspector below the editor:

  1. In the Name and Definition tab of the Properties view, set the Label property for this action to Import Master Records

  1. Select the Action Configuration tab, and then set Imported Record Type to Master Records

  1. Save the Action Set.

Customize golden data browsing

By default, an action was generated to browse golden records of the Company entity. You will customize the label and color of this action, and add a similar action for the master records.

  1. Navigate to Applications>DefaultApplication in the Model Design View and double-click on Folders and Actions:

  1. Select the BrowseCompany action.
  2. In the Name and Definition tab of the Properties view, set the Name of the action to BrowseCompanyGolden.

  1. Select the Display Properties tab and then set the following properties:

Enable master data browsing

To allow users to browse master records, you need to create a new action:

  1. In the Folders and Actions editor, click Add Action:

  1. Enter BrowseCompanyMaster for the Name of your new action, and then click Next.

  1. Configure the Business View to browse as follows, and then click Next:

  1. Configure Display Properties as follows, and then click Finish:

  1. Click on the new BrowseCompanyMaster action and set the Icon Color property to md:blue in the Display Properties tab of the Properties view:

  1. Customize the order of the actions in your menu: Select the BrowseCompanyMaster action, and then click Move Up to move it above BrowseIndustry:

Configure Global Search

The Global Search Configuration defines which business views are searched from the Global Search page, and how they are searched.

You will now configure the Global Search search for both Companies and Industries.

  1. Click on Global Search Configuration.
  2. In the Global Search Configuration editor, click Add Business View to Search above the Searched Business Views list:

  1. Enter the following values, and then click OK:

  1. Repeat the two previous steps to add Industry to the Global Search, using the following values:

  1. Save your work.

Brand your application

Now you will set some application properties to make things clear for your data stewards:

  1. Double-click Applications>DefaultApplication in the Model Design View.

  1. Configure the B2B Tutorial Application by setting the following properties:

  1. Save your model.
  2. Right-click on the root node in the Model Design View - element B2BTutorial [0.0] - and select Validate. No error or warning should be reported.

Congratulations

You have successfully completed the fourth step of this tutorial by configuring your first application!

What we've covered

In the next step, you will learn how to deploy this application and access it for the first time.

Semarchy xDM allows you to deploy your model to a Data Location. It will create all the physical tables and columns in a relational database (PostgreSQL, Oracle, or SQL Server), and generate the user interface of your application.

What you'll learn

Create a Datasource

Before creating a Data Location and deploying your model, you must create a Platform Datasource to connect to the SEMARCHY_EMP_TUTORIAL database schema that was created in the Quick Install tutorials.

  1. In the Application Builder toolbar, expand the user menu, and then select Welcome Page.

  1. Select Configuration.

  1. Scroll down to the Connections section and click Datasources.

  1. Click the Add datasource floating action button.

  1. Enter the following values and then click CREATE:

  1. Enter the following connection parameters in the editor:

  1. Enter the password of the database schema:

  1. Click Test this Datasource in the button bar above the editor:

  1. The Datasource test successful message should be displayed. Click OK.

If another message is displayed, identify the error and fix the connection parameters.

  1. Click the Save and apply datasource modification button on the datasource editor.

  1. Open the user menu on the top-right corner and then select All applications to go back to the Welcome Page.

Create the Data Location and deploy the application

In this section, you will create the B2BTutorial Data Location and deploy your model to this data location:

  1. Click the Management button in the Application Builder bar to access the Management perspective.

  1. Right-click on Data Locations and select New Data Location.

  1. Enter the following values, and then click Finish:

  1. Wait until deployment completes.

Connect to the application

In this section, you will connect to the application that you just deployed:

  1. In the Application Builder toolbar, expand the user menu, and then select Welcome Page.

  1. From the Welcome Page, click the B2B Tutorial application icon.

  1. If you have not yet completed the profile for the semadmin user in a previous tutorial, the following popup appears. Click CONTINUE.

  1. Enter the following information for semadmin and click CONTINUE:

  1. On the next screen, press GET STARTED.
  2. Discover the user interface which was built for your application:

  1. Click Companies (Golden) in the Navigation Drawer: Your application contains no customers yet. You will fix that soon.


Congratulations

You have successfully deployed and tested your first application.

What we've covered

In the next part, you will learn how to use this application to start loading data and browsing the results.

Now that you have successfully deployed your model and application, it is time to load some data.

In a data consolidation hub, data would be loaded via data integration technologies like REST APIs or SQL. In this tutorial, we load data via spreadsheets. But you will have the chance to do more realistic data integration in later tutorials!

Semarchy xDM generated all the code for you to import data from spreadsheets, match and merge records, and perform value survivorship.

For more information about using applications, you can refer to the Customer B2C Demo that showcases a fully-functional customer data application.


User profile

As a first time user, you should update your profile information and preferences.

Please follow these steps:

  1. In the B2B Tutorial application, open the user menu on the upper-right corner, and select Profile from the menu.

  1. Basic information about yourself is already set. Move your mouse over the default profile picture and click on the Change profile picture button:

  1. Select on your hard drive the semadmin.jpg picture which is available under the data-consolidation/pictures folder of the tutorial resources you downloaded previously. Click UPLOAD.

  1. Optionally, customize your user preferences like date format, language, etc (Settings tab) and your login or role (Advanced tab) and save.


Import industries

In this section, you will import Industry reference data values from an excel spreadsheet:

  1. In the B2B Tutorial application, click on Industries in the Navigation Drawer, and then select Import from the actions menu.

  1. Select the data-consolidation\datasets\industries-data.xlsx file from the tutorials resources that you downloaded.
  2. Review the import contents and then click CONTINUE:

  1. On the Define mappings screen, keep the default mappings and click CONTINUE:

  1. Review the Import Summary and then click FINISH to import the data:

  1. Review the imported data and click FINISH. This submits the data to the certification process so that all users will have access to it:

  1. Wait until the toaster at the bottom-left corner of the screen indicates "Changes successfully applied", and then click on CLICK TO REFRESH to see the imported data:

Import companies

Repeat similar steps to import companies:

  1. Click on Companies (Master) in the Navigation Drawer and select Import Master Records from the actions menu:

  1. Select the data-consolidation\datasets\companies-data.xlsx file from the tutorials resources you downloaded.
  2. Review the import contents and then click CONTINUE:

  1. Click CONTINUE on the next screen to confirm the mapping.
  2. Click FINISH on the Summary page:

  1. Review the imported data and then click FINISH.

  1. When the toaster indicates "Changes successfully applied", click on CLICK TO REFRESH to see the imported Master data. Note that 95 master records have been imported:

  1. Click on Companies (Golden) in the Navigation Drawer to see the golden data created from the master records. Observe that 74 Golden records were created (compared to the 95 Master records), which means that some records matched and merged. In the next tutorial unit, you will refine the match rules.

Use Global Search

Global search allows you to search for records across multiple entities. In this tutorial, we configured the Global Search to search in both Industries and Companies.

  1. Click on Global Search in the Navigation Drawer to access the feature.
  2. Type health in the Search text field and click the Search button:

  1. The application finds companies and industries which use the word "health" in their names:

  1. Search for chevron. Currently, xDM finds many Chevron records because the source records fail to match with your PerfectMatch rule. You will improve this soon!

Congratulations!

You successfully used the application, loaded, and searched data.

You successfully built your first data consolidation application.

What we've covered

What's next?

In the next unit of the Data consolidation tutorial track, Standardize, Match, and Merge you will learn all the mechanics to write your own match rules for your own data requirements.