This tutorial teaches you how to build your first data consolidation application.
You will load data from multiple operational systems into the data hub where it will be matched and merged. In this first unit, you will create a very simple model with a simple match rule. Later, you will build on this foundation.
What you'll learn
Creating the logical model
Creating the user experience
Deploying the application
Using the application
Before you start
This tutorial is the first unit of the Data Consolidation track.
In this tutorial, we assume that you already set up a working Semarchy xDM environment and that you have design-time privileges to connect to the Semarchy Application Builder.
If you have not yet set up Semarchy xDM, follow first the Quick Install tutorials from the main tutorials menu.
The estimated duration of this unit is about 1 hour and 40 minutes.
Semarchy xDM uses an entity-relationship diagram to help you create the logical model of your application.
What you'll learn
Creating the model
Creating the entities and add their attributes
Creating lists of values (LOVs)
Create a model
To get started, you need to create a new logical model. The logical model will hold all of the metadata for your entities, attributes, rules, forms, collections, and applications.
On the Application Builder main page, click on "New Model...":
Enter B2BTutorial in the Name field and click Finish.
Right-click on Diagrams and select Add Diagram:
Enter MainDiagram in the Name field and click Finish.
Add the Company entity
You are now ready to create the Company entity. At the end of the creation process, it will be configured as follows:
Attribute
Datatype
Mandatory?
Notes
ID
Sequence
Yes
Auto-numbered ID
CompanyName
String(128)
Yes
None
Website
String(128)
No
None
Region
List of Values
No
Refers to RegionLOV
Address1
String(128)
No
None
PostalCode
String(128)
No
None
City
String(128)
No
None
State
String(128)
No
None
Country
List of Values
No
Refers to CountryLOV
To add a new entity to your model, the fastest way is to add it to your diagram.
From the Diagram editor, click Add Entity, and then click on the diagram itself:
Enter the following values in the first page of the wizard, and then click Next:
Name: Company
Entity type: Fuzzy Matched (which is best suited for data consolidation use cases)
Select Historize Golden Records and Historize Master Records
On the next page, enter the following values to configure the primary key of your entity, and then click Finish:
Name: ID
Golden ID Generation: Sequence
Starts With: 1000001 (Golden IDs will be auto-incremented starting from this value)
ID generation: Manual (incoming companies will use their own ID generated by the source systems)
Add a simple attribute
On the Diagram editor, right-click on the Company entity, and then select Add Simple Attribute:
Enter the following values, and then click Finish:
Name: CompanyName
Type: String (Built-in Type)
Length: 128
Select Mandatory
Mandatory Validation Scope: Pre-consolidation only
Save all your work: click the Save all current contents button on the top of the Model Design View.
Add Lists of Values
You are now going to add 2 new Lists of Values (LOV) that you will use for the Region and Country attributes.
Create the Region LOV
Right-click on List of Values in the Model Design View and select Add List of Values.
Enter RegionLOV as Name, then click the Add Value button to add the first value of your LOV:
Enter APAC for the Code and keep the auto-generated Label unmodified:
Do the same for the other values:
Code
Label
EMEA
EMEA
NORTH_AMERICA
North America
SOUTH_AMERICA
South America
Click Save.
Add the Region attribute to the Company entity
Let's now add to the Company entity the attribute that will use the Region LOV.
Go back to the Main Diagram: select the Main Diagram editor tab or double-click on Diagrams>Main Diagram in the Model Design View:
Right-click on the Company entity and select Add Simple Attribute.
Enter the following values and click Finish:
Name: Region
Type: RegionLOV [List of values]
Keep Mandatory unselected.
Create the Country LOV
We will now create the CountryLOV list of values using an Excel file.
Proceed as before to create this second list of values.
Back to the List of Values editor, click the Import values button:
Click the Browse button and select (or drag and drop) the data-consolidation\datasets\countries-lov.xlsx file from the Tutorial resources you downloaded earlier.
Click Finish.
Review the imported values:
Save your work.
Add the Country attribute to the Company entity
Add the Country simple attribute to your entity the same way you did for Region, using the following values:
Name: Country
Type: CountryLOV [List of values]
Unselect Mandatory
Add the other attributes
It is now time to add the other attributes to your entity.
Following the same instructions you used for CompanyName, add the following attributes to the Company entity:
Attribute
Datatype
Mandatory?
Website
String(128)
No
Address1
String(128)
No
PostalCode
String(128)
No
City
String(128)
No
State
String(128)
No
Return to the MainDiagram and click Show data types in the button bar to display the data types next to attribute names. Review your entity and check that all the required attributes have been added:
Save your work.
Assign an icon to the entity
It is a best practice to assign icons to your entities. These icons will visually guide users in several places in your application.
From the MainDiagram, click on the Company entity.
In the Properties view below the editor, identify the Icon property within the Name and Description tab, and then click Select Image:
Type "domain" in the search bar and press Enter to filter the images.
Select the mdi/domain.svg image:
Save your work.
Congratulations
You have successfully completed the first step of this tutorial by creating the main entity.
What we've covered
Creating a model and a diagram
Creating an entity and adding attributes
Setting an icon on your entity
In the next step, you will add a second entity and link it with the Company entity.
You are now ready to create the Industry entity as described below:
Attribute
Datatype
Mandatory?
Notes
ID
String(20)
Yes
ID is set manually
Title
String(128)
Yes
Name of the Industry
ShortDescription
String(500)
No
Description of this industry in paragraph form
Industries in this model are reference data used to classify Companies. Here you will use the top level of the NAICS industry classification standard.
In the next steps, you will:
Create a Basic entity and add its attributes.
Add a reference to describe the one-to-many relationship between a Company and an Industry.
Create the Industry Entity
To create the Industry entity, follow the same process as for the Company entity:
Add the entity to the diagram with the following values and click Next:
Name: Industry
Entity Type: Basic
Configure the entity's primary key attribute on the next page:
Name: ID
ID Generation: Manual
Type: String [Built-in Type]
Length: 20
Add the Title and ShortDescription attributes as you did previously for Company.
Attribute
Mandatory?
Datatype
Notes
Title
Yes
String(128)
Name of the Industry
ShortDescription
No
String(500)
Description of this industry in paragraph form
Assign an icon
Proceed the same way you did for Company to assign an icon for the Industry entity:
From the MainDiagram, click on the Industry entity.
In the Properties view, identify the Icon property within the Name and Description tab, and click Select Image:
Type "format-list-bulleted-type" in the search bar to find and select mdi/format-list-bulleted-type.svg:
Create the Company-Industry reference
From the Main Diagram, create the reference relationship named CompaniesIndustry, from Company to Industry:
Click Add Reference in the Palette.
Click on the Company entity and drag the line to the Industry entity:
Leave all values unmodified on the Create New Reference Relationship dialog, and then click Finish.
Congratulations
You have successfully completed the second step of this tutorial by adding another entity to your model and creating a reference relationship.
What we've covered
Creating a basic entity and adding attributes
Setting an icon on the entity
Creating relationships between entities
In the next step, you will configure record matching and survivorship.
In this step, you will configure the match and survivorship rules that will be used by your application.
What you'll learn
The basics of matching
Creating a matcher which defines the merge thresholds
Creating your first match rule
Defining the survivorship rules
Matching data in Semarchy xDM
Matching in Semarchy xDM is performed on master records coming from source systems. It is a fundamental step of the Certification Process that aims at consolidating these master records into unique golden Records.
In the case of Fuzzy-Matched entities (such as the Company entity you created in the first step), Master and Golden records use different identifiers:
Master records are referred to by their IDs in source systems
Golden records use IDs generated as defined in your entity.
Match Rules define how incoming source records match together.
You control the rule details and the thresholds defining when matches merge and when merged records should be confirmed.
Once master records have been matched, Survivorship Rules define how their attributes will consolidate into the resulting golden record.
Create a match rule
It is now time to create your first match rule.
In the Application Builder Model Design View, go to Entities>Company>Matcher.
Right-click on Matcher and select Define SemQL Matcher:
The description is optional. Click Finish.
On the SemQL Matcher editor, click Add Match Rule:
Set the Name of your first match rule to PerfectMatch.
Scroll down the editor and identify the Match Condition field. Click Edit Expression:
Copy and paste the following SemQL Condition in the SemQL editor, and then click OK.
Record1.CompanyName = Record2.CompanyName
AND Record1.Address1 = Record2.Address1
AND Record1.PostalCode = Record2.PostalCode
AND Record1.Country = Record2.Country
Click on SemQLMatcher - Company in the breadcrumb above the MatchRule editor to return to the SemQL Matcher editor:
Scroll down to the Merge Policy section of the editor. Keep the default value for all thresholds. With this configuration, all matches will merge.
Save your work.
Later you will add some nuance to handle fuzzy matches and suggestions.
Add publishers
Publishers are applications that provide source data to the data hub. They are commonly used in survivorship rules, but can also be used in match rules, enrichers, and in other locations in your model.
In this section, you will define 4 publishers, one for each source system that will submit data to xDM.
Right-click on Publishers in the Model Design View and select Add Publisher.
Enter CRM for the Name. Keep the Publisher Code and Label fields unmodified, and then click Finish.
Repeat the previous steps to create 3 other publishers:
ERP
FINANCE
WEB
Save your work.
Configure a survivorship rule
It is time to set up your first survivorship rule. A default rule was created automatically with the entity. You will edit this rule to define the level of trust you place in each of your publishers.
To configure the default survivorship rule, go to Entities>Company>Survivorship Rules in the Model Design View and double-click DefaultRule.
Scroll down to the Consolidation Rule section.
Set Consolidation Strategy to Preferred Publisher to select and order your publishers manually.
Click the Select Publishers button next to the Prefered Publishers property:
Add publishers in the following order: FINANCE, ERP, WEB, CRM. You can use the Move Up/Move Down buttons to reorder the publishers.
Click Finish.
Click the Edit Expression button next to the Ranking Expression property:
Enter the following SemQL expression, to consolidate the value from the most recent record in case of a tie, and then click OK.
UpdateDate DESC
Finally, set the Override Strategy property to Override - until next user change. With this configuration, changes by the data stewards are allowed and take precedence over the source systems:
Save your work.
Later you will extend these to handle more complex requirements.
Congratulations
You have successfully completed the third step of this tutorial by configuring matching and survivorship rules.
What we've covered
Creating simple matching rules
Declaring publishers
Configuring simple survivorship rules
In the next step, you will start to define the user experience for your application. You will learn how to add forms, collections, and more.
Semarchy xDM uses Material Design as the visual language for defining a business user interface. The user interface uses several application components that you will define in the next sections. These include display cards, forms, collections, steppers, search forms, action sets, business views, application actions, and more.
What you'll learn
The different application components you can define in xDM
Generating default application components
Editing what was generated
Application components
Data Applications in Semarchy xDM rely on several components that you can directly generate and configure in the Application Builder. The following examples are taken from the Product Retail demo application.
Display cards provide a compact and visual representation of a record.
Collections configure how multiple records are displayed in a table or list or grid.
Forms define how one record is displayed in a form to browse or author the data.
Steppers determine the guided data authoring experience.
Action Sets are data actions available for one or multiple records.
Business Views tie all of the above together to form a consumable view.
Display Cards define a visual representation of a record in a nicely formatted way. They can be used in collections or in fields to offer a compact view of this entity.
When generating application components, you will be prompted to define how you want your records to be displayed in Display Cards. You will define SemQL expressions that will translate into formatted text for your primary text and secondary text. Here is how Display Cards will be configured for the Company and Industry entities:
Company Display Card:
Industry Display Card:
Generate components
You will now use the Generate Application Components wizard to automatically generate most of the components you will need to enable user interaction with your data.
In the Model Design View of the Application Builder, right-click on the Company entity under Entities and select Create Application Components.
Keep all default settings and press Next.
Observe on the next page that a default Primary Text Expression was set: SEM_TO_CHAR2(ID). This expression returns the ID converted to a string.
Set the Primary Text Expression:
Click the Edit Expression button at the upper-right corner of the text area
Remove the default expression.
Double-click the CompanyName attribute in the list of attributes on the left to add it to the expression.
Press OK.
Set the Secondary Text Expression: follow the same procedure as for the Primary Text Expression and select the FDN_Industry attribute, or paste FDN_Industry into the text area.
Click Finish.
Repeat the instructions from step 1 for the Industry entity, using the following expressions for the Display Card:
Primary Text Expression: Title
Secondary Text Expression: ID
Save your work.
Explore what was generated
The wizard has generated all the required objects for the Company entity:
Display Card
Collection
Form
Stepper
Action Set
Business View
Navigation Drawer
Expand the Company entity to see all generated components:
Scroll down in the Model Design View and go to Applications>DefaultApplication.
Double-click on Folders and Actions and observe that a default list of menu items was initialized, including a BrowseCompany action:
Improve your application
In this section, you will improve your application before deploying it. You will:
Enable the import of master records instead of golden records
Enable easy browsing of master records as well as golden records
Configure the Global Search feature
Set icons and labels for your application and its entities.
Enable master data import
You are now going to configure the Import action for the Company entity so that it supports importing master records instead of golden records, which is the default setting.
Navigate to Entity>Company>Actions Sets and double-click on CompanyActionSet to open the editor for the generated Action Set.
Select the ImportAuthorCompanies action in the Actions list. This opens the Properties view Inspector below the editor:
In the Name and Definition tab of the Properties view, set the Label property for this action to Import Master Records
Select the Action Configuration tab, and then set Imported Record Type to Master Records
Save the Action Set.
Customize golden data browsing
By default, an action was generated to browse golden records of the Company entity. You will customize the label and color of this action, and add a similar action for the master records.
Navigate to Applications>DefaultApplication in the Model Design View and double-click on Folders and Actions:
Select the BrowseCompany action.
In the Name and Definition tab of the Properties view, set the Name of the action to BrowseCompanyGolden.
Select the Display Properties tab and then set the following properties:
Label: Companies (Golden)
Icon Color: md:amber
Enable master data browsing
To allow users to browse master records, you need to create a new action:
In the Folders and Actions editor, click Add Action:
Enter BrowseCompanyMaster for the Name of your new action, and then click Next.
Configure the Business View to browse as follows, and then click Next:
Business View: select Company [Business View]
View Type: select Master Data
Configure Display Properties as follows, and then click Finish:
Select Use Custom Label
Label: Companies (Master)
Click on the new BrowseCompanyMaster action and set the Icon Color property to md:blue in the Display Properties tab of the Properties view:
Customize the order of the actions in your menu: Select the BrowseCompanyMaster action, and then click Move Up to move it above BrowseIndustry:
Configure Global Search
The Global Search Configuration defines which business views are searched from the Global Search page, and how they are searched.
You will now configure the Global Search search for both Companies and Industries.
Click on Global Search Configuration.
In the Global Search Configuration editor, click Add Business View to Search above the Searched Business Views list:
Enter the following values, and then click OK:
Name: Company
Business View: Company [Business View]
Repeat the two previous steps to add Industry to the Global Search, using the following values:
Name: Industry
Business View: Industry [Business View]
Save your work.
Brand your application
Now you will set some application properties to make things clear for your data stewards:
Double-click Applications>DefaultApplication in the Model Design View.
Configure the B2B Tutorial Application by setting the following properties:
Name: B2BTutorial
Label: B2B Tutorial
Title: B2B Tutorial
Avatar: images://mdi/domain.svg (copy and paste the value, or click on the Select Image button and search for "domain" to find that image)
Sort Method: Position. This option allows manual ordering of the navigation items.
Save your model.
Right-click on the root node in the Model Design View - element B2BTutorial [0.0] - and select Validate. No error or warning should be reported.
Congratulations
You have successfully completed the fourth step of this tutorial by configuring your first application!
What we've covered
Generating application components
Configuring entities' display cards
Branding your application
Configuring Global Search
Validating your model
In the next step, you will learn how to deploy this application and access it for the first time.
Semarchy xDM allows you to deploy your model to a Data Location. It will create all the physical tables and columns in a relational database (PostgreSQL, Oracle, or SQL Server), and generate the user interface of your application.
What you'll learn
Create a Datasource
Create a Data Location
Deploy the model to this Data Location
Connect to the application
Create a Datasource
Before creating a Data Location and deploying your model, you must create a Platform Datasource to connect to the SEMARCHY_EMP_TUTORIAL database schema that was created in the Quick Install tutorials.
In the Application Builder toolbar, expand the user menu, and then select Welcome Page.
Select Configuration.
Scroll down to the Connections section and click Datasources.
Click the Add datasource floating action button.
Enter the following values and then click CREATE:
Name: EMPLOYEE_TUTORIAL
Database Type: select the value that corresponds to your database.
Enter the following connection parameters in the editor:
The URL is populated with a default value. Check that the default host, port and database name in the connection URL are correct.
Driver: keep the default value.
Username: semarchy_b2b_tutorial.
Enter the password of the database schema:
Click the Edit Secret button next to the Password field.
Keep the current value of the Secure with field (or select another encryption method if you have configured any), enter semarchy_b2b_tutorial in the Value of "Password" field, and then click OK.
Click Test this Datasource in the button bar above the editor:
The Datasource test successful message should be displayed. Click OK.
If another message is displayed, identify the error and fix the connection parameters.
Click the Save and apply datasource modification button on the datasource editor.
Open the user menu on the top-right corner and then select All applications to go back to the Welcome Page.
Create the Data Location and deploy the application
In this section, you will create the B2BTutorial Data Location and deploy your model to this data location:
Click the Management button in the Application Builder bar to access the Management perspective.
Right-click on Data Locations and select New Data Location.
Enter the following values, and then click Finish:
Name: B2BTutorial
Data Source Name: select java:comp/env/jdbc/B2B_TUTORIAL
Deployed Model Edition: select B2BTutorial [0.0] [Model Edition]
Wait until deployment completes.
Connect to the application
In this section, you will connect to the application that you just deployed:
In the Application Builder toolbar, expand the user menu, and then select Welcome Page.
From the Welcome Page, click the B2B Tutorial application icon.
If you have not yet completed the profile for the semadmin user in a previous tutorial, the following popup appears. Click CONTINUE.
Enter the following information for semadmin and click CONTINUE:
First name: Semarchy
Last name: Admin
Email: semadmin@semarchy.com
On the next screen, press GET STARTED.
Discover the user interface which was built for your application:
The Navigation Drawer contains the actions that you configured in the Folders and Actions.
The Global Search form is displayed by default.
Click Companies (Golden) in the Navigation Drawer: Your application contains no customers yet. You will fix that soon.
Congratulations
You have successfully deployed and tested your first application.
What we've covered
Deploy a model into a Data Location.
Connect to the generated application.
In the next part, you will learn how to use this application to start loading data and browsing the results.
Now that you have successfully deployed your model and application, it is time to load some data.
In a data consolidation hub, data would be loaded via data integration technologies like REST APIs or SQL. In this tutorial, we load data via spreadsheets. But you will have the chance to do more realistic data integration in later tutorials!
Semarchy xDM generated all the code for you to import data from spreadsheets, match and merge records, and perform value survivorship.
For more information about using applications, you can refer to the Customer B2C Demo that showcases a fully-functional customer data application.
User profile
As a first time user, you should update your profile information and preferences.
Please follow these steps:
In the B2B Tutorial application, open the user menu on the upper-right corner, and select Profile from the menu.
Basic information about yourself is already set. Move your mouse over the default profile picture and click on the Change profile picture button:
Select on your hard drive the semadmin.jpg picture which is available under the data-consolidation/pictures folder of the tutorial resources you downloaded previously. Click UPLOAD.
Optionally, customize your user preferences like date format, language, etc (Settings tab) and your login or role (Advanced tab) and save.
Import industries
In this section, you will import Industry reference data values from an excel spreadsheet:
In the B2B Tutorial application, click on Industries in the Navigation Drawer, and then select Import from the actions menu.
Select the data-consolidation\datasets\industries-data.xlsx file from the tutorials resources that you downloaded.
Review the import contents and then click CONTINUE:
On the Define mappings screen, keep the default mappings and click CONTINUE:
Review the Import Summary and then click FINISH to import the data:
Review the imported data and click FINISH. This submits the data to the certification process so that all users will have access to it:
Wait until the toaster at the bottom-left corner of the screen indicates "Changes successfully applied", and then click on CLICK TO REFRESH to see the imported data:
Import companies
Repeat similar steps to import companies:
Click on Companies (Master) in the Navigation Drawer and select Import Master Records from the actions menu:
Select the data-consolidation\datasets\companies-data.xlsx file from the tutorials resources you downloaded.
Review the import contents and then click CONTINUE:
Click CONTINUE on the next screen to confirm the mapping.
Click FINISH on the Summary page:
Review the imported data and then click FINISH.
When the toaster indicates "Changes successfully applied", click on CLICK TO REFRESH to see the imported Master data. Note that 95 master records have been imported:
Click on Companies (Golden) in the Navigation Drawer to see the golden data created from the master records. Observe that 74 Golden records were created (compared to the 95 Master records), which means that some records matched and merged. In the next tutorial unit, you will refine the match rules.
Use Global Search
Global search allows you to search for records across multiple entities. In this tutorial, we configured the Global Search to search in both Industries and Companies.
Click on Global Search in the Navigation Drawer to access the feature.
Type health in the Search text field and click the Search button:
The application finds companies and industries which use the word "health" in their names:
Search for chevron. Currently, xDM finds many Chevron records because the source records fail to match with your PerfectMatch rule. You will improve this soon!
Congratulations!
You successfully used the application, loaded, and searched data.
You successfully built your first data consolidation application.
What we've covered
You created a model with entities, attributes, relationships, and match rules.
You configured default actions and added new actions to build your user experience.
You deployed your model and application to a data location.
You used your application to import, search, and browse data.
What's next?
In the next unit of the Data consolidation tutorial track, Standardize, Match, and Merge you will learn all the mechanics to write your own match rules for your own data requirements.