In this tutorial, you will see how you can implement data quality rules in Semarchy xDM to standardize your data and detect duplicates.

What you'll learn

Before you start

This tutorial is the second unit within the Data Authoring track:

Before following this tutorial, you must complete the first unit, Build your First Data Consolidation Application.

The estimated duration of this unit is about 1 hour.

Semarchy xDM allows you to create Enrichers to normalize, standardize, and enrich data loaded or authored in your application.

Several enrichers can be defined for an entity. They are executed in sequence.

You will learn how to define two types of enrichers:

In the next sections, you will be guided to:

  1. Create a SemQL enricher to standardize first and last names.
  2. Add a default value to the hire date if it is empty.
  3. Create a complex type to store additional phone attributes.
  4. Use a Plug-in enricher to standardize phone numbers and populate these attributes.

What you'll learn

Add the StandardizeNames enricher

Let's use SemQL enrichers to perform basic data standardization on the employees' FirstName and LastName. The goal is to make sure that these attributes are always spelled with the first letter in upper case and the remaining letters in lower case.

  1. Open your HRTutorial model in the Application Builder and navigate to Entities > Employee > Enrichers.
  2. Right-click the Enrichers node and then select Add SemQL Enricher.

  1. Set the Name to StandardizeNames and then click Next.

  1. Select the FirstName and LastName attributes in the Available Attributes list, click Add to add them to the Used Attributes list, and then click Finish.

  1. Enter the SemQL expression for each attribute in the Enricher Expressions section:

  1. Save your work.

Add the DefaultHireDate enricher

You will now create another SemQL enricher to assign a default date to the HireDate attribute if no date was provided.

  1. Navigate again to Entities > Employee > Enrichers.
  2. Right-click the Enrichers node and select Add SemQL Enricher.

  1. Set the Name to DefaultHireDate and click Next.
  2. Add HireDate to the Used Attributes list and click Next.

  1. To ensure that the enricher is triggered only if HireDate is not set, set the Filter to the following SemQL expression, and then click Finish: HireDate is null

  1. Set the Expression of the HireDate attribute with the following expression in the Enricher Expressions section: CURRENT_DATE()

  1. Save your work.

Add the StandardizePhone enricher

In this section, you will use the Phone Enricher Plug-in to standardize and enrich phone numbers. This plug-in exposes several outputs given a phone number and a country of origin.

Add the PhoneType Complex Type

You first need to create a Complex Type that will be populated by the phone enricher.

  1. Right-click the Complex Types node in the Model Design View, and then select Add SemQL Enricher.

  1. Set the Name to PhoneType and click Finish.

  1. Select the Definition Attributes finger tab and then click the Add Definition Attribute button.

  1. Enter the following values and then click Finish:

  1. Repeat the same steps to add all the following attributes:

Attribute

Datatype

Notes

Country

String(2)

ISO-3166-1 country code for this phone

EnrichedPhone

String(128)

Standardized phone number

LineType

String(128)

Type of line guessed by the plug-in

Carrier

String(128)

Name of the carrier when available

Location

String(128)

Name of the guessed location of the phone

Timezones

String(128)

Time zones for this phone

IsPossible

String(128)

Whether this phone is possible

IsValid

String(128)

Whether this phone is valid

  1. Check the result and save your work.

  1. Go back to the Details finger tab, click the Define... button, and then click the Define Display Name button.
  2. Click Next in the first step of the wizard.
  3. Add the EnrichedPhone attribute to the Selected Attributes list, and then click Finish.

  1. Save your work.
  2. Expand the Employee entity under Entities > Employee in the Model Design View, and then right-click Attributes and select Add Complex Attribute.

  1. Enter the following values, and then click Finish.

  1. Save your work.

Add the phone enricher

The Phone Enricher Plug-in accepts several inputs and produces several outputs such as the standardized phone or the geolocation of the phone line:

  1. Navigate again to Entities > Employee in the Model Design View.
  2. Expand the Employee entity, right-click Enrichers, and select Add API Enricher.

  1. Enter the following values and then click Finish:

  1. Scroll down to the Inputs section of the editor, and then click the Define Inputs button.

  1. Select the Region Code, Input Phone Number and Enriched Phone Format inputs, click the Add button to add them to the Used Inputs list, and then click Finish.

  1. Set the following expressions in the Expression column for each input listed in the Inputs section:

  1. Scroll down to the Outputs section and click the Define Outputs button:

  1. Add the following attributes to the Attributes Used list, and then click Finish:

  1. Select the following values in the Output Name column for each Attribute Name from the Outputs list:

Attribute Name

Output Name

EnrichedPhone.Carrier

Carrier Name

EnrichedPhone.EnrichedPhone

Enriched Phone Number

EnrichedPhone.IsPossible

Possible Phone Number

EnrichedPhone.IsValid

Valid Phone Number

EnrichedPhone.LineType

Phone Line Type

EnrichedPhone.Location

Geocoding Data

EnrichedPhone.Timezones

Time Zones

  1. Check the enricher configuration in the Inputs and Outputs sections, and then save your work.

Congratulations

You have completed the first part of this tutorial by adding data standardization rules to your model.

What we've covered

The next step will focus on adding validation and match rules!

Semarchy xDM provides various types of data validation rules when authoring data. These rules act as a firewall to prevent bad data from being entered or imported in the data hub.

Some of the rules are implicit, such as mandatory attributes, lists of values, or referential integrity. Others can be set explicitly, such as SemQL validations, Plug-in validations, and match rules.

In this step, you will learn how to use SemQL to validate your data and ensure data consistency.

You will be guided to:

What you'll learn

Validate end dates

An employee's end date can either be null if the employee is still in the company, or it must be greater than the hire date.

Let's add a validation that applies this rule.

  1. Go to Entities > Employee > Validations in the Model Design View of the Application Builder.
  2. Right-click Validations and select Add SemQL Validation.

  1. Fill in the wizard with the following values and then click Finish:

  1. Save your work.

Validate emails

You will now add a rule that makes the email mandatory only if the employee is a contractor.

  1. Navigate again to Entities > Employee > Validations in the Model Design View.
  2. Right-click Validations, and then select Add SemQL Validation.

  1. Fill in the wizard with the following values and click Finish:

  1. Save your work.

Prevent duplicates

It is a best practice to prevent users from creating duplicate records. Here, you will create a match rule that will be used to prevent the creation of an employee record if another employee with the exact same first and last name already exists within the same subsidiary.

  1. Go to Entities > Employee > Matcher in the Model Design View.
  2. Right-click Matcher and select Define SemQL Matcher.

  1. Set the Description to Prevent duplicate employees and click Finish.
  2. Click the Add Match Rule button in the Match Rules section of the SemQL Matcher Editor.

  1. Set the Name of your match rule to MatchOnName.

  1. Scroll down to the Matching section, and then click the Edit Expression button.

  1. Add the following SemQL Condition and click OK:
Upper(Record1.FirstName) = Upper(Record2.FirstName)
and Upper(Record1.LastName) = Upper(Record2.LastName)
and Record1.Subsidiary = Record2.Subsidiary

  1. Save your work.

Congratulations

You have successfully completed the second part of this tutorial by adding validation and match rules to your model.

What we've covered

The next part will focus on wiring enrichers and validation rules in the user interface and deploying a new version of your application.

Semarchy xDM gives you the flexibility to control exactly which rules should apply when users are authoring data using a Stepper.

In the next sections, you will learn how to make the enrichers and validation rules effective in the user-facing application. You will be guided to:

  1. Modify the EmployeeForm to add the missing phone enricher attributes.
  2. Modify the AuthorEmployee stepper to activate the rules.
  3. Deploy your changes.

Modify the EmployeeForm

You will now modify the EmployeeForm to add the enriched phone attributes and reorganize the form fields.

  1. In the Application Builder, navigate to Entity > Employee > Form, and then double-click EmployeeForm.

  1. Select the Phone attribute in the form's tree view, and then use the Move Up button to move this attribute between LastName and Salutation.

  1. Expand the EnrichedPhone attribute in the Attributes list, and then select all attributes it is composed of (hold the shift key + select the first and last attribute):

  1. Drag and drop the selected attributes to the Form's tree view between Phone and Salutation:

  1. Enter the following values in the Label column these new attributes:

  1. Select the FDN_Department attribute and use the Move Up button to move it between Title and HireDate (alternatively, drag and drop the attribute at the expected position).

  1. Make the EnrichedPhone_EnrichedPhone attribute read-only:

  1. Repeat the same procedure to make the following attributes read-only:

  1. Save your work.

Modify the AuthorEmployee stepper

Before you deploy your changes, you need to modify the AuthorEmployee stepper to enable the validations and enrichers that you defined in the previous steps.

  1. Go to Entities > Employee > Steppers in the Model Design View, and then double-click AuthorEmployees.

  1. Configure the validations to perform on stepper finish:

  1. Scroll up to the Steps section and select the second step: Employee.

  1. In the Properties view, open the Step Transition Validations finger tab, select the first row of the table (DETECT_DUPS), and then select Warn in the On Step Exit column.

  1. Make all other validations blocking for steps transitions:

  1. Configure all validations on data change:

  1. Configure validations on form open:

  1. Set up the execution of enrichers:

  1. Save your work.

Deploy the model changes

You are now ready to deploy your changes, have a first glance at the employee creation form, and see the impact of your changes.

  1. In the Application Builder, right-click on the root node in the Model Design View, corresponding to the HRTutorial [0.0] model, and then select Validate. The validation report should raise no error.

  1. Go to the Management perspective of the Application Builder.

  1. Right-click on your data location EmployeeTutorial and select Deploy Model Edition.

  1. Click Finish in the wizard to deploy your model changes.

Congratulations

You have successfully completed the third part of this tutorial by plugging in the validations and enrichers and deploying the changes to your application.

What we've covered

The next part will showcase all your rules in action in your application.

In this step of the tutorial, you will try several combinations for creating employees to see your enrichers and validations in action in the user-facing interface.

Enrichers in action

In the first step of this tutorial, you created 3 enrichers to standardize names, enrich phone information, and set the default hire date. Let's see them in action:

  1. Open your application from the Welcome Page.

  1. Select Employees in the Navigation Drawer, open the actions menu, and then select Create.

  1. Set the First Name to jack and the Last Name to BOLT-HarriSson. Notice how the StandardizeNames enricher re-formats values automatically.

  1. Set the Phone to 0478963556 and the Country to FR. Notice the outputs of the phone enricher:

  1. Scroll down to see that the Hire Date field was automatically set to the current date.
  2. Discard your changes: click again on Employees in the Navigation Drawer, and then click DISCARD ALL on the popup that appears.

Validations in action

In the first part of this tutorial, you added 2 validations on top of the built-in validations and enabled them in the AuthorEmployee stepper. Let's see them in action.

  1. Select Employees in the Navigation Drawer of your application, and then select Create from the actions menu.
  2. Set the First Name to Jack and the Last Name to Bolt, and then click FINISH.

  1. Two validation issues are raised. Click CANCEL.

  1. Fill in the following fields:
  1. Set End Date to a date before the hire date. Notice the error message.

  1. Select Is Contractor. Notice the error message due to a missing email address.

  1. Fix the data in error:
  1. Click FINISH to submit the creation of your record.
  2. Wait until the toaster at the bottom-left corner of the screen indicates "Changes successfully applied", and then select CLICK TO REFRESH:

  1. Jack Bolt is now available in the list of employees.

Matching in action

In the first step of this tutorial, you created a match rule to prevent from creating duplicate employees based on their name and subsidiary.

  1. Select Employees in the Navigation Drawer, and then select Create from the actions menu.
  2. Fill in the following fields and then click FINISH.
  1. Notice the error message stating that a duplicate has been found, and then select the record to see the details.

  1. Click RESOLVE DUPLICATE ISSUE.

  1. Select the second record (existing employee with the same name and subsidiary) by clicking on its avatar, and then click REPLACE.

  1. The editing form for the existing record is displayed in replacement of the creation form.

  1. Complete this record with the Phone and Phone Country you entered on the other record, and then click FINISH.
  1. The existing record is now updated with phone information and the creation of a duplicate was avoided.

Congratulations

You have successfully completed the fourth part of this tutorial by using the employee creation form to see the rules in actions.

What we've covered

You have completed the second unit of the Data Authoring track by implementing your first rules for data quality and deduplication.

What we've covered

What's next?

In the next part, you will learn how to customize your user interface by designing display cards, forms, and collections.