In this tutorial, you will see how you can implement data quality rules in Semarchy xDM to standardize your data and detect duplicates.

What you'll learn

• Designing enrichers to standardize and augment your data
• Implementing validations and duplicates detection rules to check your data
• Plugging the rules in the user interface and deploy your changes
• Testing your rules in the user application

Before you start

This tutorial is the second unit within the Data Authoring track:

Before following this tutorial, you must complete the first unit, Build your First Data Consolidation Application.

The estimated duration of this unit is about 1 hour.

Semarchy xDM allows you to create Enrichers to normalize, standardize, and enrich data loaded or authored in your application.

Several enrichers can be defined for an entity. They are executed in sequence.

You will learn how to define two types of enrichers:

• SemQL Enrichers that define the rules using the SemQL language.
• API enrichers that use Java to perform more complex data transformations or external API calls (for example, to look up information from a remote micro-service such as Google Maps).

In the next sections, you will be guided to:

1. Create a SemQL enricher to standardize first and last names.
2. Add a default value to the hire date if it is empty.
3. Create a complex type to store additional phone attributes.
4. Use a Plug-in enricher to standardize phone numbers and populate these attributes.

What you'll learn

• Creating SemQL enrichers
• Adding a Complex Type
• Creating API enrichers

Add the StandardizeNames enricher

Let's use SemQL enrichers to perform basic data standardization on the employees' FirstName and LastName. The goal is to make sure that these attributes are always spelled with the first letter in upper case and the remaining letters in lower case.

1. Open your HRTutorial model in the Application Builder and navigate to Entities > Employee > Enrichers.
2. Right-click the Enrichers node and then select Add SemQL Enricher.

3. Set the Name to StandardizeNames and then click Next.

4. Select the FirstName and LastName attributes in the Available Attributes list, click Add to add them to the Used Attributes list, and then click Finish.

5. Enter the SemQL expression for each attribute in the Enricher Expressions section:

• For the FirstName attribute: INITCAP(FirstName)
• For the LastName attribute: INITCAP(LastName)

6. Save your work.

Add the DefaultHireDate enricher

You will now create another SemQL enricher to assign a default date to the HireDate attribute if no date was provided.

1. Navigate again to Entities > Employee > Enrichers.
2. Right-click the Enrichers node and select Add SemQL Enricher.

3. Set the Name to DefaultHireDate and click Next.
4. Add HireDate to the Used Attributes list and click Next.

5. To ensure that the enricher is triggered only if HireDate is not set, set the Filter to the following SemQL expression, and then click Finish: HireDate is null

6. Set the Expression of the HireDate attribute with the following expression in the Enricher Expressions section: CURRENT_DATE()

7. Save your work.

Add the StandardizePhone enricher

In this section, you will use the Phone Enricher Plug-in to standardize and enrich phone numbers. This plug-in exposes several outputs given a phone number and a country of origin.

Add the PhoneType Complex Type

You first need to create a Complex Type that will be populated by the phone enricher.

1. Right-click the Complex Types node in the Model Design View, and then select Add Complex Type.

2. Set the Name to PhoneType and click Finish.

3. Select the Definition Attributes finger tab and then click the Add Definition Attribute button.

4. Enter the following values and then click Finish:

• Name: Country
• Length: 2

5. Repeat the same steps to add all the following attributes:

6. Check the result and save your work.

7. Go back to the Details finger tab, click the Define... button, and then click the Define Display Name button.
8. Click Next in the first step of the wizard.
9. Add the EnrichedPhone attribute to the Selected Attributes list, and then click Finish.

11. Save your work.
12. Expand the Employee entity under Entities > Employee in the Model Design View, and then right-click Attributes and select Add Complex Attribute.

13. Enter the following values, and then click Finish.

• Name: EnrichedPhone
• Type: PhoneType [Complex Type]

14. Save your work.

Add the phone enricher

The Phone Enricher Plug-in accepts several inputs and produces several outputs such as the standardized phone or the geolocation of the phone line:

1. Navigate again to Entities > Employee in the Model Design View.
2. Expand the Employee entity, right-click Enrichers, and select Add API Enricher.

3. Enter the following values and then click Finish:

• Select Java Plug-in.
• Select the Semarchy Phone Enricher in the drop-down list.
• Set the Name to StandardizePhone.

4. Scroll down to the Inputs section of the editor, and then click the Define Inputs button.

5. Select the Region Code, Input Phone Number and Enriched Phone Format inputs, click the Add button to add them to the Used Inputs list, and then click Finish.

6. Set the following expressions in the Expression column for each input listed in the Inputs section:

• Region Code: COALESCE(EnrichedPhone.Country, 'US')
• Input Phone Number: Phone
• Enriched Phone Format: 'INTERNATIONAL'

7. Scroll down to the Outputs section and click the Define Outputs button:

8. Add the following attributes to the Attributes Used list, and then click Finish:

• EnrichedPhone.Carrier
• EnrichedPhone.EnrichedPhone
• EnrichedPhone.IsPossible
• EnrichedPhone.IsValid
• EnrichedPhone.LineType
• EnrichedPhone.Location
• EnrichedPhone.Timezones

9. Select the following values in the Output Name column for each Attribute Name from the Outputs list:

10. Check the enricher configuration in the Inputs and Outputs sections, and then save your work.

Congratulations

You have completed the first part of this tutorial by adding data standardization rules to your model.

What we've covered

• Creating a SemQL enricher to standardize first and last names
• Setting a default value to the hire date using another SemQL enricher
• Creating a complex type to hold additional phone attributes
• Using a Plug-in enricher to standardize phone numbers and populate those attributes.

The next step will focus on adding validation and match rules!

Semarchy xDM provides various types of data validation rules when authoring data. These rules act as a firewall to prevent bad data from being entered or imported in the data hub.

Some of the rules are implicit, such as mandatory attributes, lists of values, or referential integrity. Others can be set explicitly, such as SemQL validations, Plug-in validations, and match rules.

In this step, you will learn how to use SemQL to validate your data and ensure data consistency.

You will be guided to:

• Create a SemQL validation rule to check that the hire date of an employee is before his contract end date.
• Create a SemQL validation rule to make the email mandatory when the employee is a contractor.
• Create a SemQL match rule to prevent the creation of duplicate employees.

What you'll learn

• Creating SemQL validations
• Creating match rules

Validate end dates

An employee's end date can either be null if the employee is still in the company, or it must be greater than the hire date.

Let's add a validation that applies this rule.

1. Go to Entities > Employee > Validations in the Model Design View of the Application Builder.
2. Right-click Validations and select Add SemQL Validation.

3. Fill in the wizard with the following values and then click Finish:

• Name: CheckEndDate
• Description: End date must be greater than the hire date
• Condition: (EndDate is null) OR (EndDate > HireDate)

4. Save your work.

Validate emails

You will now add a rule that makes the email mandatory only if the employee is a contractor.

1. Navigate again to Entities > Employee > Validations in the Model Design View.
2. Right-click Validations, and then select Add SemQL Validation.

3. Fill in the wizard with the following values and click Finish:

• Name: CheckContractorEmail
• Description: Contractors must have an email address
• Condition: (IsContractor = '1' and Email is not null) OR IsContractor = '0'

4. Save your work.

Prevent duplicates

It is a best practice to prevent users from creating duplicate records. Here, you will create a match rule that will be used to prevent the creation of an employee record if another employee with the exact same first and last name already exists within the same subsidiary.

1. Go to Entities > Employee > Matcher in the Model Design View.
2. Right-click Matcher and select Define SemQL Matcher.

3. Set the Description to Prevent duplicate employees and click Finish.
4. Click the Add Match Rule button in the Match Rules section of the SemQL Matcher Editor.

5. Set the Name of your match rule to MatchOnName.

6. Scroll down to the Matching section, and then click the Edit Expression button.

7. Add the following SemQL Condition and click OK:

Upper(Record1.FirstName) = Upper(Record2.FirstName)
and Upper(Record1.LastName) = Upper(Record2.LastName)
and Record1.Subsidiary = Record2.Subsidiary

8. Save your work.

Congratulations

You have successfully completed the second part of this tutorial by adding validation and match rules to your model.

What we've covered

• You created a SemQL validation to check that the hire date is before the end date.
• You created another SemQL validation to set email as mandatory when the employee is a contractor.
• You created a SemQL match rule to prevent the creation of duplicate employees.

The next part will focus on wiring enrichers and validation rules in the user interface and deploying a new version of your application.

Semarchy xDM gives you the flexibility to control exactly which rules should apply when users are authoring data using a Stepper.

• Enrichers can be triggered on data changes or when entering or exiting a step.
• Validations can be triggered while users are entering data in forms, or when data is submitted to your application.

In the next sections, you will learn how to make the enrichers and validation rules effective in the user-facing application. You will be guided to:

1. Modify the EmployeeForm to add the missing phone enricher attributes.
2. Modify the AuthorEmployee stepper to activate the rules.
3. Deploy your changes.

Modify the EmployeeForm

You will now modify the EmployeeForm to add the enriched phone attributes and reorganize the form fields.

1. In the Application Builder, navigate to Entity > Employee > Form, and then double-click EmployeeForm.

2. Select the Phone attribute in the form's tree view, and then use the Move Up button to move this attribute between LastName and Salutation.

3. Expand the EnrichedPhone attribute in the Attributes list, and then select all attributes it is composed of (hold the shift key + select the first and last attribute):

4. Drag and drop the selected attributes to the Form's tree view between Phone and Salutation:

5. Enter the following values in the Label column these new attributes:

• EnrichedPhone_Country: Phone Country
• EnrichedPhone_EnrichedPhone: Enriched Phone
• EnrichedPhone_LineType: Line Type
• EnrichedPhone_Carrier: Carrier
• EnrichedPhone_Location: Location
• EnrichedPhone_Timezones: Time Zones
• EnrichedPhone_IsPossible: Possible?
• EnrichedPhone_IsValid: Valid?

6. Select the FDN_Department attribute and use the Move Up button to move it between Title and HireDate (alternatively, drag and drop the attribute at the expected position).

7. Make the EnrichedPhone_EnrichedPhone attribute read-only:

• Select the EnrichedPhone_EnrichedPhone attribute.
• Select the General finger tab in the Properties view.
• Select Read-Only.

8. Repeat the same procedure to make the following attributes read-only:

• EnrichedPhone_EnrichedPhone
• EnrichedPhone_LineType
• EnrichedPhone_Carrier
• EnrichedPhone_Location
• EnrichedPhone_Timezones
• EnrichedPhone_IsPossible
• EnrichedPhone_IsValid

9. Save your work.

Modify the AuthorEmployee stepper

Before you deploy your changes, you need to modify the AuthorEmployee stepper to enable the validations and enrichers that you defined in the previous steps.

1. Go to Entities > Employee > Steppers in the Model Design View, and then double-click AuthorEmployees.

2. Configure the validations to perform on stepper finish:

• Scroll down to the Validations section of the editor.
• Select the first row of the table (DETECT_DUPS) and then select Warn in the On Stepper Finish column.

• Select all the other rows: hold the shift key, select the CheckContractorEmail row, and then select the last row of the table. Right-click the selection and then select Block. This makes all the selected validations blocking the completion of the employee stepper.

3. Scroll up to the Steps section and select the second step: Employee.

4. In the Properties view, open the Step Transition Validations finger tab, select the first row of the table (DETECT_DUPS), and then select Warn in the On Step Exit column.

5. Make all other validations blocking for steps transitions:

• Select all rows starting from the second one (CheckContractorEmail).
• Right-click this selection and then select Block.

6. Configure all validations on data change:

• Select the Form Validations finger tab.
• Select all rows in the table.
• Right-click the selection and select On Data Change.

7. Configure validations on form open:

• Select the following rows (hold the Ctrl key + select): CheckContractorEmail, CheckEndDate, Department|MANDATORY, FirstName, HireDate, LastName and Subsidiary|MANDATORY.
• Right-click this selection.
• Select On Form Open.

8. Set up the execution of enrichers:

• Select the Form Enrichers finger tab.
• Select On Data Change for StandardizeNames and StandardizePhone.
• Select On Form Open for DefaultHireDate.

9. Save your work.

Deploy the model changes

You are now ready to deploy your changes, have a first glance at the employee creation form, and see the impact of your changes.

1. In the Application Builder, right-click on the root node in the Model Design View, corresponding to the HRTutorial [0.0] model, and then select Validate. The validation report should raise no error.

2. Go to the Management perspective of the Application Builder.

3. Right-click on your data location EmployeeTutorial and select Deploy Model Edition.

4. Click Finish in the wizard to deploy your model changes.

Congratulations

You have successfully completed the third part of this tutorial by plugging in the validations and enrichers and deploying the changes to your application.

What we've covered

• You modified the Employee form to add missing attributes.
• You modified the AuthorEmployee stepper to activate rules.
• You deployed your changes.

The next part will showcase all your rules in action in your application.

In this step of the tutorial, you will try several combinations for creating employees to see your enrichers and validations in action in the user-facing interface.

Enrichers in action

In the first step of this tutorial, you created 3 enrichers to standardize names, enrich phone information, and set the default hire date. Let's see them in action:

1. Open your application from the Welcome Page.

2. Select Employees in the Navigation Drawer, open the actions menu, and then select Create.

3. Set the First Name to jack and the Last Name to BOLT-HarriSson. Notice how the StandardizeNames enricher re-formats values automatically.

4. Set the Phone to 0478963556 and the Country to FR. Notice the outputs of the phone enricher:

5. Scroll down to see that the Hire Date field was automatically set to the current date.
6. Discard your changes: click again on Employees in the Navigation Drawer, and then click DISCARD ALL on the popup that appears.

Validations in action

In the first part of this tutorial, you added 2 validations on top of the built-in validations and enabled them in the AuthorEmployee stepper. Let's see them in action.

1. Select Employees in the Navigation Drawer of your application, and then select Create from the actions menu.
2. Set the First Name to Jack and the Last Name to Bolt, and then click FINISH.

3. Two validation issues are raised. Click CANCEL.

4. Fill in the following fields:

• Subsidiary: ACME EU
• Title: Senior Consultant
• Department: Consulting

5. Set End Date to a date before the hire date. Notice the error message.

6. Select Is Contractor. Notice the error message due to a missing email address.

7. Fix the data in error:

• Set End Date to one month after the Hire Date.
• Set Email to jack.bolt@acme-eu.com.

8. Click FINISH to submit the creation of your record.
9. Wait until the toaster at the bottom-left corner of the screen indicates "Changes successfully applied", and then select CLICK TO REFRESH:

10. Jack Bolt is now available in the list of employees.

Matching in action

In the first step of this tutorial, you created a match rule to prevent from creating duplicate employees based on their name and subsidiary.

1. Select Employees in the Navigation Drawer, and then select Create from the actions menu.
2. Fill in the following fields and then click FINISH.

• First Name: JACK
• Last Name: BOLT
• Phone: 0478897889
• Phone Country: FR
• Subsidiary: ACME EU
• Department: Consulting

3. Notice the error message stating that a duplicate has been found, and then select the record to see the details.

4. Click RESOLVE DUPLICATE ISSUE.

5. Select the second record (existing employee with the same name and subsidiary) by clicking on its avatar, and then click REPLACE.

6. The editing form for the existing record is displayed in replacement of the creation form.

7. Complete this record with the Phone and Phone Country you entered on the other record, and then click FINISH.

• Phone: 0478897889
• Phone Country: FR

8. The existing record is now updated with phone information and the creation of a duplicate was avoided.

Congratulations

You have successfully completed the fourth part of this tutorial by using the employee creation form to see the rules in actions.

What we've covered

• How enrichers standardize and enrich data while creating an employee
• How validations prevent users from creating bad or incomplete data
• How match rules avoid duplicate employee creation

You have completed the second unit of the Data Authoring track by implementing your first rules for data quality and deduplication.

What we've covered

• Enrichers: You created enrichers to standardize your data.
• Validations: You added data validation rules to prevent bad data.
• Deployment: You enabled your rules and deployed your changes.
• Use: You used your application to see the rules in action.

What's next?

In the next part, you will learn how to customize your user interface by designing display cards, forms, and collections.