This tutorial teaches you how to build your first data consolidation application.

You will load data from multiple operational systems into the data hub where it will be matched and merged. In this first unit, you will create a very simple model with a simple match rule. Later, you will build on this foundation.

What you'll learn

• Creating the logical model
• Creating the user experience
• Deploying the application
• Using the application

Before you start

This tutorial is the first unit of the Data Consolidation track.

In this tutorial, we assume that you already set up a working Semarchy xDM environment and that you have design-time privileges to connect to the Semarchy Application Builder.

If you have not yet set up Semarchy xDM, follow first the Quick Install tutorials from the main tutorials menu.

GO TO TUTORIALS

Additional setup information can also be found in the installation guide.

Click on the button DOWNLOAD TUTORIAL RESOURCES to get the required resources, including the images and datasets used within this tutorial.

DOWNLOAD TUTORIAL RESOURCES

The estimated duration of this unit is about 1 hour and 40 minutes.

Semarchy xDM uses an entity-relationship diagram to help you create the logical model of your application.

What you'll learn

• Creating the model
• Creating the entities and add their attributes
• Creating lists of values (LOVs)

Create a model

To get started, you need to create a new logical model. The logical model will hold all of the metadata for your entities, attributes, rules, forms, collections, and applications.

1. On the Application Builder main page, click on "New Model...":

2. Enter B2BTutorial in the Name field and click Finish.

3. Right-click on Diagrams and select Add Diagram:

4. Enter MainDiagram in the Name field and click Finish.

Add the Company entity

You are now ready to create the Company entity. At the end of the creation process, it will be configured as follows:

To add a new entity to your model, the fastest way is to add it to your diagram.

1. From the Diagram editor, click Add Entity, and then click on the diagram itself:

2. Enter the following values in the first page of the wizard, and then click Next:

• Name: Company
• Entity type: Fuzzy Matched (which is best suited for data consolidation use cases)
• Select Historize Golden Records and Historize Master Records

3. On the next page, enter the following values to configure the primary key of your entity, and then click Finish:

• Name: ID
• Golden ID Generation: Sequence
• Starts With: 1000001 (Golden IDs will be auto-incremented starting from this value)
• ID generation: Manual (incoming companies will use their own ID generated by the source systems)

Add a simple attribute

1. On the Diagram editor, right-click on the Company entity, and then select Add Simple Attribute:

2. Enter the following values, and then click Finish:

• Name: CompanyName
• Type: String (Built-in Type)
• Length: 128
• Select Mandatory
• Mandatory Validation Scope: Pre-consolidation only

3. Save all your work: click the Save all current contents button on the top of the Model Design View.

Add Lists of Values

You are now going to add 2 new Lists of Values (LOV) that you will use for the Region and Country attributes.

Create the Region LOV

1. Right-click on List of Values in the Model Design View and select Add List of Values.

2. Enter RegionLOV as Name, then click the Add Value button to add the first value of your LOV:

3. Enter APAC for the Code and keep the auto-generated Label unmodified:

4. Do the same for the other values:

5. Click Save.

Add the Region attribute to the Company entity

Let's now add to the Company entity the attribute that will use the Region LOV.

1. Go back to the Main Diagram: select the Main Diagram editor tab or double-click on Diagrams>Main Diagram in the Model Design View:

2. Right-click on the Company entity and select Add Simple Attribute.

3. Enter the following values and click Finish:

• Name: Region
• Type: RegionLOV [List of values]
• Keep Mandatory unselected.

Create the Country LOV

We will now create the CountryLOV list of values using an Excel file.

1. Proceed as before to create this second list of values.

2. Back to the List of Values editor, click the Import values button:

3. Click the Browse button and select (or drag and drop) the data-consolidation\datasets\countries-lov.xlsx file from the Tutorial resources you downloaded earlier.

4. Click Finish.
5. Review the imported values:

6. Save your work.

Add the Country attribute to the Company entity

Add the Country simple attribute to your entity the same way you did for Region, using the following values:

• Name: Country
• Type: CountryLOV [List of values]
• Unselect Mandatory

Add the other attributes

It is now time to add the other attributes to your entity.

1. Following the same instructions you used for CompanyName, add the following attributes to the Company entity:

2. Return to the MainDiagram and click Show data types in the button bar to display the data types next to attribute names. Review your entity and check that all the required attributes have been added:

3. Save your work.

Assign an icon to the entity

It is a best practice to assign icons to your entities. These icons will visually guide users in several places in your application.

1. From the MainDiagram, click on the Company entity.
2. In the Properties view below the editor, identify the Icon property within the Name and Description tab, and then click Select Image:

3. Type "domain" in the search bar and press Enter to filter the images.
4. Select the mdi/domain.svg image:

5. Save your work.

Congratulations

You have successfully completed the first step of this tutorial by creating the main entity.

What we've covered

• Creating a model and a diagram
• Creating an entity and adding attributes
• Setting an icon on your entity

In the next step, you will add a second entity and link it with the Company entity.

You are now ready to create the Industry entity as described below:

Industries in this model are reference data used to classify Companies. Here you will use the top level of the NAICS industry classification standard.

In the next steps, you will:

• Create a Basic entity and add its attributes.
• Add a reference to describe the one-to-many relationship between a Company and an Industry.

Create the Industry Entity

To create the Industry entity, follow the same process as for the Company entity:

1. Add the entity to the diagram with the following values and click Next:

• Name: Industry
• Entity Type: Basic

2. Configure the entity's primary key attribute on the next page:

• Name: ID
• ID Generation: Manual
• Type: String [Built-in Type]
• Length: 20

3. Add the Title and ShortDescription attributes as you did previously for Company.

Assign an icon

Proceed the same way you did for Company to assign an icon for the Industry entity:

1. From the MainDiagram, click on the Industry entity.
2. In the Properties view, identify the Icon property within the Name and Description tab, and click Select Image:

3. Type "format-list-bulleted-type" in the search bar to find and select mdi/format-list-bulleted-type.svg:

Create the Company-Industry reference

From the Main Diagram, create the reference relationship named CompaniesIndustry, from Company to Industry:

1. Click Add Reference in the Palette.
2. Click on the Company entity and drag the line to the Industry entity:

4. Leave all values unmodified on the Create New Reference Relationship dialog, and then click Finish.

Congratulations

You have successfully completed the second step of this tutorial by adding another entity to your model and creating a reference relationship.

What we've covered

• Creating a basic entity and adding attributes
• Setting an icon on the entity
• Creating relationships between entities

In the next step, you will configure record matching and survivorship.

In this step, you will configure the match and survivorship rules that will be used by your application.

What you'll learn

• The basics of matching
• Creating a matcher which defines the merge thresholds
• Creating your first match rule
• Defining the survivorship rules

Matching data in Semarchy xDM

Matching in Semarchy xDM is performed on master records coming from source systems. It is a fundamental step of the Certification Process that aims at consolidating these master records into unique golden Records.

In the case of Fuzzy-Matched entities (such as the Company entity you created in the first step), Master and Golden records use different identifiers:

• Master records are referred to by their IDs in source systems
• Golden records use IDs generated as defined in your entity.

Match Rules define how incoming source records match together.

You control the rule details and the thresholds defining when matches merge and when merged records should be confirmed.

Once master records have been matched, Survivorship Rules define how their attributes will consolidate into the resulting golden record.

Create a match rule

It is now time to create your first match rule.

1. In the Application Builder Model Design View, go to Entities>Company>Matcher.
2. Right-click on Matcher and select Define SemQL Matcher:

3. The description is optional. Click Finish.

4. On the SemQL Matcher editor, click Add Match Rule:

5. Set the Name of your first match rule to PerfectMatch.

6. Scroll down the editor and identify the Match Condition field. Click Edit Expression:

7. Copy and paste the following SemQL Condition in the SemQL editor, and then click OK.

Record1.CompanyName = Record2.CompanyName 
AND Record1.Address1 = Record2.Address1 
AND Record1.PostalCode = Record2.PostalCode 
AND Record1.Country = Record2.Country

8. Click on SemQLMatcher - Company in the breadcrumb above the MatchRule editor to return to the SemQL Matcher editor:

9. Scroll down to the Merge Policy section of the editor. Keep the default value for all thresholds. With this configuration, all matches will merge.

10. Save your work.

Later you will add some nuance to handle fuzzy matches and suggestions.

Add publishers

Publishers are applications that provide source data to the data hub. They are commonly used in survivorship rules, but can also be used in match rules, enrichers, and in other locations in your model.

In this section, you will define 4 publishers, one for each source system that will submit data to xDM.

1. Right-click on Publishers in the Model Design View and select Add Publisher.

2. Enter CRM for the Name. Keep the Publisher Code and Label fields unmodified, and then click Finish.

3. Repeat the previous steps to create 3 other publishers:

• ERP
• FINANCE
• WEB

4. Save your work.

Configure a survivorship rule

It is time to set up your first survivorship rule. A default rule was created automatically with the entity. You will edit this rule to define the level of trust you place in each of your publishers.

1. To configure the default survivorship rule, go to Entities>Company>Survivorship Rules in the Model Design View and double-click DefaultRule.

2. Scroll down to the Consolidation Rule section.
3. Set Consolidation Strategy to Preferred Publisher to select and order your publishers manually.

4. Click the Select Publishers button next to the Prefered Publishers property:

• Add publishers in the following order: FINANCE, ERP, WEB, CRM.
  You can use the Move Up/Move Down buttons to reorder the publishers.

• Click Finish.

5. Click the Edit Expression button next to the Ranking Expression property:

6. Enter the following SemQL expression, to consolidate the value from the most recent record in case of a tie, and then click OK.

UpdateDate DESC

7. Finally, set the Override Strategy property to Override - until next user change. With this configuration, changes by the data stewards are allowed and take precedence over the source systems:

8. Save your work.

Later you will extend these to handle more complex requirements.

Congratulations

You have successfully completed the third step of this tutorial by configuring matching and survivorship rules.

What we've covered

• Creating simple matching rules
• Declaring publishers
• Configuring simple survivorship rules

In the next step, you will start to define the user experience for your application. You will learn how to add forms, collections, and more.

Semarchy xDM uses Material Design as the visual language for defining a business user interface. The user interface uses several application components that you will define in the next sections. These include display cards, forms, collections, steppers, search forms, action sets, business views, application actions, and more.

What you'll learn

• The different application components you can define in xDM
• Generating default application components
• Editing what was generated

Application components

Data Applications in Semarchy xDM rely on several components that you can directly generate and configure in the Application Builder. The following examples are taken from the Product Retail demo application.

• Display cards provide a compact and visual representation of a record.

• Collections configure how multiple records are displayed in a table or list or grid.

• Forms define how one record is displayed in a form to browse or author the data.

• Steppers determine the guided data authoring experience.
• Action Sets are data actions available for one or multiple records.

• Business Views tie all of the above together to form a consumable view.

• The Navigation Drawer allows navigation to business views.

Focus on Display Cards

Display Cards define a visual representation of a record in a nicely formatted way. They can be used in collections or in fields to offer a compact view of this entity.

When generating application components, you will be prompted to define how you want your records to be displayed in Display Cards. You will define SemQL expressions that will translate into formatted text for your primary text and secondary text. Here is how Display Cards will be configured for the Company and Industry entities:

• Company Display Card:

• Industry Display Card:

Generate components

You will now use the Generate Application Components wizard to automatically generate most of the components you will need to enable user interaction with your data.

1. In the Model Design View of the Application Builder, right-click on the Company entity under Entities and select Create Application Components.

2. Keep all default settings and press Next.

3. Observe on the next page that a default Primary Text Expression was set: SEM_TO_CHAR2(ID). This expression returns the ID converted to a string.

4. Set the Primary Text Expression:

• Click the Edit Expression button at the upper-right corner of the text area

• Remove the default expression.
• Double-click the CompanyName attribute in the list of attributes on the left to add it to the expression.
• Press OK.

5. Set the Secondary Text Expression: follow the same procedure as for the Primary Text Expression and select the FDN_Industry attribute, or paste FDN_Industry into the text area.
6. Click Finish.

7. Repeat the instructions from step 1 for the Industry entity, using the following expressions for the Display Card:

• Primary Text Expression: Title
• Secondary Text Expression: ID

8. Save your work.

Explore what was generated

The wizard has generated all the required objects for the Company entity:

• Display Card
• Collection
• Form
• Stepper
• Action Set
• Business View
• Navigation Drawer

Expand the Company entity to see all generated components:

Scroll down in the Model Design View and go to Applications>DefaultApplication.

Double-click on Folders and Actions and observe that a default list of menu items was initialized, including a BrowseCompany action:

Improve your application

In this section, you will improve your application before deploying it. You will:

• Enable the import of master records instead of golden records
• Enable easy browsing of master records as well as golden records
• Configure the Global Search feature
• Set icons and labels for your application and its entities.

Enable master data import

You are now going to configure the Import action for the Company entity so that it supports importing master records instead of golden records, which is the default setting.

1. Navigate to Entity>Company>Actions Sets and double-click on CompanyActionSet to open the editor for the generated Action Set.

2. Select the ImportAuthorCompanies action in the Actions list. This opens the Properties view Inspector below the editor:

3. In the Name and Definition tab of the Properties view, set the Label property for this action to Import Master Records

4. Select the Action Configuration tab, and then set Imported Record Type to Master Records

5. Save the Action Set.

Customize golden data browsing

By default, an action was generated to browse golden records of the Company entity. You will customize the label and color of this action, and add a similar action for the master records.

1. Navigate to Applications>DefaultApplication in the Model Design View and double-click on Folders and Actions:

2. Select the BrowseCompany action.
3. In the Name and Definition tab of the Properties view, set the Name of the action to BrowseCompanyGolden.

4. Select the Display Properties tab and then set the following properties:

• Label: Companies (Golden)
• Icon Color: md:amber

Enable master data browsing

To allow users to browse master records, you need to create a new action:

1. In the Folders and Actions editor, click Add Action:

2. Enter BrowseCompanyMaster for the Name of your new action, and then click Next.

3. Configure the Business View to browse as follows, and then click Next:

• Business View: select Company [Business View]
• View Type: select Master Data

4. Configure Display Properties as follows, and then click Finish:

• Select Use Custom Label
• Label: Companies (Master)

5. Click on the new BrowseCompanyMaster action and set the Icon Color property to md:blue in the Display Properties tab of the Properties view:

6. Customize the order of the actions in your menu: Select the BrowseCompanyMaster action, and then click Move Up to move it above BrowseIndustry:

Configure Global Search

The Global Search Configuration defines which business views are searched from the Global Search page, and how they are searched.

You will now configure the Global Search search for both Companies and Industries.

1. Click on Global Search Configuration.
2. In the Global Search Configuration editor, click Add Business View to Search above the Searched Business Views list:

3. Enter the following values, and then click OK:

• Name: Company
• Business View: Company [Business View]

4. Repeat the two previous steps to add Industry to the Global Search, using the following values:

• Name: Industry
• Business View: Industry [Business View]

5. Save your work.

Brand your application

Now you will set some application properties to make things clear for your data stewards:

1. Double-click Applications>DefaultApplication in the Model Design View.

2. Configure the B2B Tutorial Application by setting the following properties:

• Name: B2BTutorial
• Label: B2B Tutorial
• Title: B2B Tutorial
• Avatar: images://mdi/domain.svg (copy and paste the value, or click on the Select Image button and search for "domain" to find that image)
• Sort Method: Position. This option allows manual ordering of the navigation items.

3. Save your model.
4. Right-click on the root node in the Model Design View - element B2BTutorial [0.0] - and select Validate. No error or warning should be reported.

Congratulations

You have successfully completed the fourth step of this tutorial by configuring your first application!

What we've covered

• Generating application components
• Configuring entities' display cards
• Branding your application
• Configuring Global Search
• Validating your model

In the next step, you will learn how to deploy this application and access it for the first time.

Semarchy xDM allows you to deploy your model to a Data Location. It will create all the physical tables and columns in a relational database (PostgreSQL, Oracle, or SQL Server), and generate the user interface of your application.

What you'll learn

• Create a Datasource
• Create a Data Location
• Deploy the model to this Data Location
• Connect to the application

Create a Datasource

Before creating a Data Location and deploying your model, you must create a Platform Datasource to connect to the SEMARCHY_EMP_TUTORIAL database schema that was created in the Quick Install tutorials.

1. In the Application Builder toolbar, expand the user menu, and then select Welcome Page.

2. Select Configuration.

3. Scroll down to the Connections section and click Datasources.

4. Click the Add datasource floating action button.

5. Enter the following values and then click CREATE:

• Name: EMPLOYEE_TUTORIAL
• Database Type: select the value that corresponds to your database.

6. Enter the following connection parameters in the editor:

• The URL is populated with a default value. Check that the default host, port and database name in the connection URL are correct.
• Driver: keep the default value.
• Username: semarchy_b2b_tutorial.

7. Enter the password of the database schema:

• Click the Edit Secret button next to the Password field.

• Keep the current value of the Secure with field (or select another encryption method if you have configured any), enter semarchy_b2b_tutorial in the Value of "Password" field, and then click OK.

8. Click Test this Datasource in the button bar above the editor:

9. The Datasource test successful message should be displayed. Click OK.

If another message is displayed, identify the error and fix the connection parameters.

10. Click the Save and apply datasource modification button on the datasource editor.

11. Open the user menu on the top-right corner and then select All applications to go back to the Welcome Page.

Create the Data Location and deploy the application

In this section, you will create the B2BTutorial Data Location and deploy your model to this data location:

1. Click the Management button in the Application Builder bar to access the Management perspective.

2. Right-click on Data Locations and select New Data Location.

3. Enter the following values, and then click Finish:

• Name: B2BTutorial
• Data Source Name: select java:comp/env/jdbc/B2B_TUTORIAL
• Deployed Model Edition: select B2BTutorial [0.0] [Model Edition]

4. Wait until deployment completes.

Connect to the application

In this section, you will connect to the application that you just deployed:

1. In the Application Builder toolbar, expand the user menu, and then select Welcome Page.

2. From the Welcome Page, click the B2B Tutorial application icon.

3. If you have not yet completed the profile for the semadmin user in a previous tutorial, the following popup appears. Click CONTINUE.

4. Enter the following information for semadmin and click CONTINUE:

• First name: Semarchy
• Last name: Admin
• Email: semadmin@semarchy.com

5. On the next screen, press GET STARTED.
6. Discover the user interface which was built for your application:

• The Navigation Drawer contains the actions that you configured in the Folders and Actions.
• The Global Search form is displayed by default.

7. Click Companies (Golden) in the Navigation Drawer: Your application contains no customers yet. You will fix that soon.

Congratulations

You have successfully deployed and tested your first application.

What we've covered

• Deploy a model into a Data Location.
• Connect to the generated application.

In the next part, you will learn how to use this application to start loading data and browsing the results.

Now that you have successfully deployed your model and application, it is time to load some data.

In a data consolidation hub, data would be loaded via data integration technologies like REST APIs or SQL. In this tutorial, we load data via spreadsheets. But you will have the chance to do more realistic data integration in later tutorials!

Semarchy xDM generated all the code for you to import data from spreadsheets, match and merge records, and perform value survivorship.

For more information about using applications, you can refer to the Customer B2C Demo that showcases a fully-functional customer data application.

User profile

As a first time user, you should update your profile information and preferences.

Please follow these steps:

1. In the B2B Tutorial application, open the user menu on the upper-right corner, and select Profile from the menu.

2. Basic information about yourself is already set. Move your mouse over the default profile picture and click on the Change profile picture button:

3. Select on your hard drive the semadmin.jpg picture which is available under the data-consolidation/pictures folder of the tutorial resources you downloaded previously. Click UPLOAD.

4. Optionally, customize your user preferences like date format, language, etc (Settings tab) and your login or role (Advanced tab) and save.

Import industries

In this section, you will import Industry reference data values from an excel spreadsheet:

1. In the B2B Tutorial application, click on Industries in the Navigation Drawer, and then select Import from the actions menu.

2. Select the data-consolidation\datasets\industries-data.xlsx file from the tutorials resources that you downloaded.
3. Review the import contents and then click CONTINUE:

4. On the Define mappings screen, keep the default mappings and click CONTINUE:

5. Review the Import Summary and then click FINISH to import the data:

6. Review the imported data and click FINISH. This submits the data to the certification process so that all users will have access to it:

7. Wait until the toaster at the bottom-left corner of the screen indicates "Changes successfully applied", and then click on CLICK TO REFRESH to see the imported data:

Import companies

Repeat similar steps to import companies:

1. Click on Companies (Master) in the Navigation Drawer and select Import Master Records from the actions menu:

2. Select the data-consolidation\datasets\companies-data.xlsx file from the tutorials resources you downloaded.
3. Review the import contents and then click CONTINUE:

4. Click CONTINUE on the next screen to confirm the mapping.
5. Click FINISH on the Summary page:

6. Review the imported data and then click FINISH.

7. When the toaster indicates "Changes successfully applied", click on CLICK TO REFRESH to see the imported Master data. Note that 95 master records have been imported:

8. Click on Companies (Golden) in the Navigation Drawer to see the golden data created from the master records. Observe that 74 Golden records were created (compared to the 95 Master records), which means that some records matched and merged. In the next tutorial unit, you will refine the match rules.

Use Global Search

Global search allows you to search for records across multiple entities. In this tutorial, we configured the Global Search to search in both Industries and Companies.

1. Click on Global Search in the Navigation Drawer to access the feature.
2. Type health in the Search text field and click the Search button:

3. The application finds companies and industries which use the word "health" in their names:

4. Search for chevron. Currently, xDM finds many Chevron records because the source records fail to match with your PerfectMatch rule. You will improve this soon!

Congratulations!

You successfully used the application, loaded, and searched data.

You successfully built your first data consolidation application.

What we've covered

• You created a model with entities, attributes, relationships, and match rules.
• You configured default actions and added new actions to build your user experience.
• You deployed your model and application to a data location.
• You used your application to import, search, and browse data.

What's next?

In the next unit of the Data consolidation tutorial track, Standardize, Match, and Merge you will learn all the mechanics to write your own match rules for your own data requirements.