This tutorial teaches you how to load data in Semarchy xDM using the REST API. You will learn to update existing golden records and create a new record simulating a source system sending in a new record. This tutorial is based on the Customer B2C demo application, so you will get experience dealing with emails, phone numbers, and address data when learning how the REST API works.

What you'll learn

Before you start

This tutorial is the last within the Integration track, which is composed of SQL-based and REST-based tutorials.

Before following this tutorial, you must have set up Semarchy xDM and completed the Customer B2C Demo tutorial. Additionally, you must complete the 5 other tutorials from the Integration track so that you have an integration job and Postman already set up, and are familiar with core integration concepts.

GO TO TUTORIALS

The estimated duration of this unit is about 30 minutes.

Enjoy this tutorial!

In this step you will check:

Check the integration job

In this tutorial, you will use the Load and Submit Data function in the REST API. To use this function, you need to have an integration job set up in your model.

To check that you have an integration job correctly set up:

  1. In the Application Builder, go to the CustomerB2CDemo model.
  2. Under Model Design, go to the Jobs node.
  3. Confirm that you have the integration job INTEGRATE_ALL that contains all of the entities in the CustomerB2CDemo model.

If you do not have the INTEGRATE_ALL job correctly set up, please review the Load data via SQL tutorial in the Integration track. You can access the tutorial by clicking the link below and look for the "Create an integration job" step.

GO TO LOAD DATA VIA SQL TUTORIAL

Check Postman

You will use the Postman application to load and update data in xDM via the REST API. If you don't have Postman set up yet, refer to the Set up a REST client for Semarchy xDM tutorial in the Integration track.

SET UP A REST CLIENT FOR SEMARCHY XDM

To test that Postman is correctly set up:

  1. Open up the Postman application.
  2. Check in the Authorization tab that your authentication parameters are correct:


  1. Substitute your public IP address or public DNS address and port in this request URL, and insert it in the URL field:

http://[public_dns_or_ip_address]:[port]/semarchy/api/rest/query/CustomerB2CDemo/Person/GD

  1. Check the GET method is selected, click Send and check that results are returned in the bottom Body section:

If your Postman application is working correctly, then you're ready to proceed to the next section. If you are not able to connect or if your request fails, refer to the tutorial Set up a REST client for Semarchy xDM for step-by-step guidance for setting up Postman.

Congratulations

Now that you have confirmed that an Integration Job and Postman are correctly set up, in the next section you will use the REST API documentation to begin constructing your payload to update data in xDM.

Before we begin constructing REST calls to load data, let's look at the REST API documentation that Semarchy xDM generates for you. This is a fantastic resource that will help you construct your REST URLs to reduce the risks of typos and other human errors.

The Retrieve Data via the REST API tutorial provides additional guidance on the REST API documentation. Review the tutorial if you want more information:

GO TO RETRIEVE DATA VIA THE REST API TUTORIAL

Get the sample payload

In this tutorial, you will use the CREATE_LOAD_AND_SUBMIT action to load data in one step. This action is equivalent to the Continuous Loads feature when loading data via SQL where xDM handles getting a load ID and submitting the load for you. You can learn how to use Continuous Loads in the Semarchy documentation and by following the Load Data via SQL tutorial.

GO TO LOAD DATA VIA SQL TUTORIAL

You will now dive into the REST API documentation to grab the sample CREATE_LOAD_AND_SUBMIT request so you can load data.

To begin:

  1. Go to the REST API documentation URL. Remember to substitute your public ID address or public DNS address instead of using our server's information: http://[public_dns_or_ip_address]:[port]/semarchy/api/rest/api-ui/index.html?domain=data&data-location=CustomerB2CDemo
  2. Click on PUBLISH DATA in the left menu.
  3. Click on the Create a load option in the left menu:


  1. In the action drop-down menu, choose CREATE_LOAD_AND_SUBMIT.

    You will see a lot of helpful resources to guide you on how to build the request payload. For example, if you drill into persistOptions all the way down to validations to see ValidationType, you can see all the possible values for ValidationType.

  1. Scroll down further to find the Responses heading. These are sample responses to help you understand the possible responses you may receive when you load data. If your load fails, look up the response for troubleshooting clues.
  2. Look for the Request Samples.

  1. The sample payload has placeholder values to give you examples of values you need to substitute from your environment.

This payload sample is the recommended starting point for loading data later in this tutorial. You will return to this sample payload when we build the request in Postman.

Congratulations

You learned how the REST API documentation works. Let's summarize:

In the next step, you will customize the payload and make a POST request to load data via the REST API in Postman.

The ability to load data into xDM in near real-time is critical to delivering value to the business. For example, if your organization has a call center application that needs to update customer information in Semarchy xDM in near real-time, the REST API is the best way to integrate these two applications.

In the rest of this tutorial, we will focus on loading customer information via the REST API as well as triggering data enrichments and validations to ensure the new information meets your organization's data quality standards.

What you'll learn

Background

Imagine the scenario where a customer calls into the customer service center to update her information. This section will guide you through how to check the customer's information for data quality issues before persisting it in xDM.

Let's begin by pulling up the golden customer record of Nola Girvan. To find her information:

  1. Open the Customer B2C application.
  2. Go to the Customers business view.
  3. Use the search filter to search on "nola".


  1. Open the record for Nola Girvan. Go to the CONTACT INFO tab. Notice that the Valid Email Domain field for this customer shows NO.

When Nola Girvan's record was first loaded, xDM enriched her email address using the Semarchy Email Enricher. The enricher flagged Nola Girvan's email as invalid.

Nola received a postcard in the mail asking her to call the customer service center to update her information to ensure she receives the latest promotions by email.

Nola calls the customer service center to update her email address from nola@girvan.com to nola.girvan@gmail.com. When the customer service representative attempts to update her record in the system, the information can first flow through to Semarchy xDM to check the data quality before saving this information permanently. This maintains high data quality standards, which can be a challenge in typical CRM, marketing, and call center systems.

In the Customer B2C Demo model, there is a validation rule, InvalidEmail, that checks whether an email address is valid. This validation rule uses the Semarchy Email Validator, which accepts an email address as an input and returns a boolean value to indicate whether the email is valid. The validator performs syntax checks. It is also configured to perform MX records lookup to check the domain.

Set up Postman

Open your Postman application. As you build your request in Postman, you will learn how to use the POST method for loading and where to edit the payload.

To check whether Nola's new email is valid using the validation rule, InvalidEmail:

  1. Copy and paste the request URL from the REST API documentation into Postman. Remember to substitute your public IP address or public DNS address: http://[host]:[port]/semarchy/api/rest/loads/CustomerB2CDemo
  2. Change the method from GET to POST. This change should cause the Body tab to become enabled.
  3. Click the Body tab.

  1. Now let's modify the payload to call the REST API with an invalid email address.


For your convenience, we modified the sample payload for you. Copy the payload below and paste it into the Postman text editor - alternatively, you can modify the existing payload you copied from the REST API documentation.

{
    "action":"CREATE_LOAD_AND_SUBMIT",
    "programName":"UPDATE_DATA_REST_API",
    "loadDescription":"Update customer information reported through call center",
    "jobName":"INTEGRATE_ALL",
    "persistOptions":{
        "defaultPublisherId":"MKT",
        "optionsPerEntity":{
            "Person":{
                "enrichers":[],
                "validations":[
                                {
                                        "validationName": "InvalidEmail",
                                        "validationType": "PLUGIN"
                                }
                                        ],
                "queryPotentialMatches":false
            }
        },
        "missingIdBehavior":"GENERATE",
        "persistMode":"NEVER"
    },
    "persistRecords":{
        "Person":[
            {
                "SourceID":"1368178",
                "SourceEmail":"nola.girvan@gmail.co"
            }
        ]
    }
}

Note the following points of interest:

  1. Click Send in Postman:

The response you receive from Semarchy xDM should look like the following:

{
    "status": "PERSIST_CANCELLED",
    "load": {
        "loadId": 99,
        "loadStatus": "CANCELED",
        "loadCreator": "semadmin",
        "loadCreationDate": "2018-11-17T01:40:55.317Z",
        "programName": "UPDATE_DATA_REST_API",
        "loadDescription": "Update customer information reported through call center",
        "numberOfJobExecutions": 0,
        "submitInterval": -1,
        "submittable": true,
        "loadType": "EXTERNAL_LOAD"
    },
    "records": {
        "Person": [
            {
                "entityName": "Person",
                "recordValues": {
                    "CleansedEmail": "nola.girvan@gmail.com",
                    "Address.Country": "US",
                    "ValidEmailDomain": "1",
                    "NormalizedFirstName": "NOLA",
                    "PersonType": "CUSTOMER",
                    "NormalizedLastName": "GIRVAN",
                    "SourceEmail": "nola.girvan@gmail.co",
                    "Address.Street": "1620 Paonia Ave",
                    "DeleteOperation": null,
                    "StandardizedPhone": "(719) 574-6245",
                    "Address.State": "CO",
                    "Address.PostalCode": "80915",
                    "ValueStatus": "NORMAL",
                    "Nickname": "nola",
                    "DateOfBirth": "1980-04-10",
                    "DeleteAuthor": null,
                    "FirstName": "Nola",
                    "Address.City": "Colorado Springs",
                    "SourceID": "1368178",
                    "DeletePath": null,
                    "DeleteDate": null,
                    "PhoneticFirstName": "NL",
                    "PhoneticLastName": "JRFN",
                    "SourcePhone": "719-574-6245",
                    "PublisherID": "MKT",
                    "PhoneGeocodingData": "Colorado Springs, CO",
                    "LastName": "Girvan"
                },
                "failedValidations": [
                    {
                        "validationType": "PLUGIN",
                        "validationName": "InvalidEmail"
                    }
                ],
                "potentialMatches": []
            }
        ]
    }
}

Look for the failedValidations object. This response from xDM shows that the email address failed the InvalidEmail enricher. This is expected. If there was no error, meaning the email address is valid, then the failedValidations object would be empty.

You now know how to identify if the data you are loading has violated a validation rule via the REST API. In the next section, you will load Nola's valid email address (meaning the typo is fixed). Then you will be able to compare how the response differs when the email is valid.

Congratulations

You successfully check where an email address is valid via the Semarchy Email Validator using a REST call. Let's summarize what you achieved:

In the previous section, you learned how to check whether an email address is valid. Now that the customer service representative discovered the email address typo and has corrected it to be nola.girvan@gmail.com, you are ready to load this new email address into xDM.

What you'll learn

Load data into xDM

Imagine that the customer service representative wants to update Nola's email address to nola.girvan@gmail.com. To try the request again to see how the results differ:

  1. Modify the payload. First, fix the email by adding the trailing m to SourceEmail so that it reads: "SourceEmail":"nola.girvan@gmail.com"
  2. Next, fix the persistMode from NEVER to IF_NO_ERROR_OR_MATCH so that it reads: "persistMode":"IF_NO_ERROR_OR_MATCH". You should have a payload that looks like the following:
{
    "action":"CREATE_LOAD_AND_SUBMIT",
    "programName":"UPDATE_DATA_REST_API",
    "loadDescription":"Update customer information reported through call center",
    "jobName":"INTEGRATE_ALL",
    "persistOptions":{
        "defaultPublisherId":"MKT",
        "optionsPerEntity":{
            "Person":{
                "enrichers":[],
                "validations":[
                                {
                                        "validationName": "InvalidEmail",
                                        "validationType": "PLUGIN"
                                }
                                        ],
                "queryPotentialMatches":false
            }
        },
        "missingIdBehavior":"GENERATE",
        "persistMode":"IF_NO_ERROR_OR_MATCH"
    },
    "persistRecords":{
        "Person":[
            {
                "SourceID":"1368178",
                "SourceEmail":"nola.girvan@gmail.com"
            }
        ]
    }
}
  1. Click Send. The response you receive from Semarchy xDM should look like the following:
{
    "status": "PERSISTED",
    "load": {
        "loadId": 100,
        "loadStatus": "PENDING",
        "loadCreator": "semadmin",
        "loadCreationDate": "2018-11-17T01:42:05.504Z",
        "programName": "UPDATE_DATA_REST_API",
        "loadDescription": "Update customer information reported through call center",
        "loadSubmitDate": "2018-11-17T01:42:05.561Z",
        "batchSubmitter": "semadmin",
        "batchId": 66,
        "integrationJobName": "INTEGRATE_ALL",
        "integrationJobQueueName": "Default",
        "numberOfJobExecutions": 0,
        "submitInterval": -1,
        "submittable": true,
        "loadType": "EXTERNAL_LOAD"
    },
    "records": {
        "Person": [
            {
                "entityName": "Person",
                "recordValues": {
                    "CleansedEmail": "nola.girvan@gmail.com",
                    "Address.Country": "US",
                    "ValidEmailDomain": "1",
                    "NormalizedFirstName": "NOLA",
                    "PersonType": "CUSTOMER",
                    "NormalizedLastName": "GIRVAN",
                    "SourceEmail": "nola.girvan@gmail.com",
                    "Address.Street": "1620 Paonia Ave",
                    "DeleteOperation": null,
                    "StandardizedPhone": "(719) 574-6245",
                    "Address.State": "CO",
                    "Address.PostalCode": "80915",
                    "ValueStatus": "NORMAL",
                    "Nickname": "nola",
                    "DateOfBirth": "1980-04-10",
                    "DeleteAuthor": null,
                    "FirstName": "Nola",
                    "Address.City": "Colorado Springs",
                    "SourceID": "1368178",
                    "DeletePath": null,
                    "DeleteDate": null,
                    "PhoneticFirstName": "NL",
                    "PhoneticLastName": "JRFN",
                    "SourcePhone": "719-574-6245",
                    "PublisherID": "MKT",
                    "PhoneGeocodingData": "Colorado Springs, CO",
                    "LastName": "Girvan"
                },
                "failedValidations": [],
                "potentialMatches": []
            }
        ]
    }
}

You will know that the job succeeded if you see the status indicates the record was persisted: "status": "PERSISTED"

  1. Retrieve the values of the loadId and the batchId in the REST response (in our example, respectively 42 and 31):

  1. Check the status of the Integration Batch:

  1. Let's check the Customer B2C application to confirm that Nola's email successfully updated.

Great job! You successfully updated Nola's email and fixed it from an invalid to a valid email address.

Congratulations

You successfully loaded data by updating an email address via using a REST call. Let's summarize what you achieved:

Now that you know how to load data via the REST API, in the next section, you will try another data quality check. Instead of performing a dry run of a validation rule, you will perform a dry run of an enricher and then eventually persist the data.

In the previous two steps, you learned how to validate incoming data to boost data quality. You did it by first checking data against a validation rule and then persisting the data when the updated information had no errors.

Another way to improve data quality is by enriching data. In this section, you will focus on enriching customer information via the REST API to ensure the new information meets your organization's data quality standards. Then you will load this record via the REST API.

What you'll learn

Background

Imagine in this second scenario, a customer calls into the customer service center to update her phone number. This section will guide you through how to check the customer's information for data quality issues before persisting it in xDM.

Let's begin by pulling up the golden customer record of Gail Taulman. To find her information:

  1. Open the Customer B2C application.
  2. Go to the Customers business view.
  3. Use the search filter to search on "gail".


  1. Open the record for Gail Taulman. Go to the CONTACT INFO tab. Notice that the Source Phone field for this customer shows the phone number but is missing the area code. As a result, the Phone Geocoding Data field is empty.

When Gail Taulman's record was first loaded, xDM tried to enrich her phone number using the Semarchy Phone Enricher. The enricher was not able to geocode the phone number because Gail's phone number was missing the area code. This is not acceptable for most national or even regional organizations because a missing area code means the organization won't be able to reach the customer.

As a result, Gail received a postcard in the mail asking her to call the customer service center to update her information to ensure that she receives calls if there are ever any issues with her account.

Gail calls the customer service center to update her phone number's area code to 770. When the customer service representative attempts to update her record in the system, the information can first flow through to Semarchy xDM to check the data quality before saving this information permanently. This maintains high data quality standards, which can be a challenge in typical CRM, marketing, and call center systems.

In the Customer B2C Demo model, an enricher named StandardizePhone is set up on the Person entity,.

This enricher rule uses the Semarchy Phone Enricher, which accepts a phone number as an input and returns many different outputs, including a standardized version (supporting different international formatting standards) of the phone number, the region code, the status, the line type (mobile, landline, etc.), and many other possible outputs.

Use the phone enricher

The Customer B2C demo model already contains the StandardizePhone enricher. This enricher standardizes the phone number to the NATIONAL format, e.g. (123) 456-7890 and provides geocoding data.

To check Gail's updated phone number, let's run the enricher via the REST API:

  1. Paste the request URL that you copied from the REST API documentation into Postman. Remember to substitute your public IP address or public DNS address:

http://[host]:[port]/semarchy/api/rest/loads/CustomerB2CDemo

  1. Double check the method is set to POST.

  1. In the Body tab, select the "raw" radio button. This should give you a text editor panel. Paste the following payload:
{
    "action":"CREATE_LOAD_AND_SUBMIT",
    "programName":"UPDATE_DATA_REST_API",
    "loadDescription":"Update customer information reported through call center",
    "jobName":"INTEGRATE_ALL",
    "persistOptions":{
        "defaultPublisherId":"CRM",
        "optionsPerEntity":{
            "Person":{
                "enrichers":[
                    "StandardizePhone"
                ],
                "validations":[],
                "queryPotentialMatches":false
            }
        },
        "missingIdBehavior":"GENERATE",
        "persistMode":"IF_NO_ERROR_OR_MATCH"
    },
    "persistRecords":{
        "Person":[
            {
                "SourceID":"1418027",
                "SourcePhone":"7709694377"
            }
        ]
    }
}

Note the following points of interest:

"optionsPerEntity":{
            "Person":{
                "enrichers":[
                    "StandardizePhone"
                ]

  1. Click Send. The response you receive from Semarchy xDM should look like the following:
{
    "status": "PERSISTED",
    "load": {
        "loadId": 102,
        "loadStatus": "PENDING",
        "loadCreator": "semadmin",
        "loadCreationDate": "2018-11-19T20:37:33.981Z",
        "programName": "UPDATE_DATA_REST_API",
        "loadDescription": "Update customer information reported through call center",
        "loadSubmitDate": "2018-11-19T20:37:34.064Z",
        "batchSubmitter": "semadmin",
        "batchId": 67,
        "integrationJobName": "INTEGRATE_ALL",
        "integrationJobQueueName": "Default",
        "numberOfJobExecutions": 0,
        "submitInterval": -1,
        "submittable": true,
        "loadType": "EXTERNAL_LOAD"
    },
    "records": {
        "Person": [
            {
                "entityName": "Person",
                "recordValues": {
                    "CleansedEmail": "gail@taulman.com",
                    "Address.Country": "US",
                    "ValidEmailDomain": "1",
                    "NormalizedFirstName": "GAIL",
                    "PersonType": "PROSPECT",
                    "NormalizedLastName": "TAULMAN",
                    "SourceEmail": "gail@taulman.com",
                    "Address.Street": "5094 Westbrook Rd",
                    "DeleteOperation": null,
                    "StandardizedPhone": "(770) 969-4377",
                    "Address.State": "GA",
                    "Address.PostalCode": "30291",
                    "ValueStatus": null,
                    "Nickname": "gail",
                    "DateOfBirth": "1971-12-10",
                    "DeleteAuthor": null,
                    "FirstName": "Gail",
                    "Address.City": "Union City",
                    "SourceID": "1418027",
                    "DeletePath": null,
                    "DeleteDate": null,
                    "PhoneticFirstName": "KL",
                    "PhoneticLastName": "TLMN",
                    "SourcePhone": "7709694377",
                    "PublisherID": "CRM",
                    "PhoneGeocodingData": "Georgia",
                    "LastName": "Taulman"
                },
                "failedValidations": [],
                "potentialMatches": []
            }
        ]
    }
}
  1. Let's analyze the response. The "SourcePhone": "7709694377" shows you the input you sent to xDM via the REST API payload. The enricher updated the two fields "StandardizedPhone": "(770) 969-4377" and "PhoneGeocodingData": "Georgia". Not only has the enricher formatted the phone number it identified the phone as being from Georgia, which is consistent with Gail's address of Union City, GA.

  1. Access again the Contact info tab for the Customer Gail Taulman and observe that Source Phone, Standardized Phone and Geocoding Data have been updated:

You now know how to enrich data via the REST API. Furthermore, you know the difference between a validation rule and an enricher.

Congratulations

You learned how to load data in xDM using the REST API while enriching a record. Let's summarize:

Great job going through this tutorial!

You will find more information about Integration jobs, such as how to query and load data using SQL as well as how to retrieve via the REST API in this Integration track.

Well done! You have run your first REST POST requests to load data into Semarchy xDM via the REST API.

In this tutorial, you learned:

You will find more Integration information, including how to load data using SQL and how to use the REST API in the Integration tutorial series.

There are also tutorials about how to design applications in the Data Authoring and Data Consolidation tutorial tracks.

Go back to the main menu to see more tutorials.

GO TO TUTORIALS