Publish Data Using the REST API
The REST API supports publishing data changes and data deletion to a data hub.
Overview
The REST API provides the capabilities to manage loads, which includes querying, creating, submitting and canceling loads. It also supports persisting records in existing loads. Finally, it provides a shortcut to Load and Submit data in a single request.
Query Existing Loads
Method |
GET |
|---|---|
Base URL |
|
URL |
|
Supported Parameters |
|
Response Format |
The response contains a count of loads, or the list of loads meeting the criteria. The information returned for each load depends on the status of the load. Example 1. Query Existing Loads: sample response.
|
Load Type
The loadType indicates the nature of the load:
-
An external load (
EXTERNAL_LOAD) which can be submitted (submittable=true). -
A continuous load (
CONTINUOUS_LOAD) which cannot be manually submitted but is automatically submitted everysubmitIntervalseconds. -
A load attached to application activities, which cannot be submitted:
-
WORKFLOW_SUBMITcorresponds to a submit performed by a workflow. -
LEGACY_WORKFLOWcorresponds to a submit performed by a legacy workflow. It replaces theWORKFLOWload type, which is deprecated -
DIRECT_AUTHORINGcorresponds to a submit performed by a direct authoring action. -
DIRECT_DELETEcorresponds to a submit performed by a direct delete action. -
DIRECT_DUPS_CONFIRMcorresponds to a submit performed by a direct duplicates confirmation action. -
DIRECT_DUPS_MANAGEMENTcorresponds to a submit performed by a direct duplicates manager action.
-
Load Status
The possible values for the Load Status are listed in the table below.
| Load Status | Description |
|---|---|
RUNNING |
The load is currently running. |
CANCELED |
The load was canceled (CancelLoad) |
PENDING |
The load was submitted. A batch was created and is waiting for the batch poller to pick it. |
SCHEDULED |
The batch was taken into account by the batch poller. The job is queued by the engine. |
PROCESSING |
The batch’s job is currently being processed by the engine. |
SUSPENDED |
The job is suspended, either by an administrator or due to an error. It awaits for administrator intervention. |
WARNING |
The job completed successfully, but some records have caused validation errors. |
DONE |
The job completed successfully with no validation errors. |
ERROR |
The job did not complete successfully, it was canceled by an administrator. |
Integration Job
If a job is attached to the load (the load was submitted), then the integrationJob object provides details about this job, including its start date, current task, duration and any error that may occur in this job.
It also includes the notificationStatus, which indicates whether the notifications were successfully sent.
The currentTask corresponds to the running task for RUNNING jobs and the last executed task for KILLED or SUSPENDED jobs
|
Use the API to monitor jobs which are SUSPENDED or in ERROR to report possible issues with the integration. Combine this endpoint with the capability to manage loads to automate job restart.
|
Query a Load
Method |
GET |
|---|---|
Base URL |
|
URL |
|
Response Format |
The response contains the load identified by Example 2. Query one Load: sample response.
|
Initialize a Load
Method |
POST |
|---|---|
Base URL |
|
URL |
|
Request Payload |
The request contains the Example 3. Load creation: sample request.
|
Response Format |
The response contains the load information, including the load ID, load type, and an indication of the status. Example 4. Load creation: sample response.
|
Load Data
To load data in a given load, the URL must contain the Load ID that was returned at load creation time, or the Load ID/Name of a continuous load.
| Data is loaded in GD tables for basic entities and in MD tables for ID and Fuzzy matched entities. |
Method |
POST |
||||
|---|---|---|---|---|---|
Base URL |
|
||||
URL |
|
||||
Request Payload |
The request contains the
Example 5. Load data: sample request.
|
||||
Response Format |
The response contains, in the Example 6. Load submission: sample response.
|
Configure Data Loads
Persist Options
When loading one or more records you can configure the following elements in the persistOptions element:
-
For each entity:
-
enrichers: Defines the enrichers that should be executed before persisting the records. By default, no enricher is executed. Possible values are:-
JOB_PRE_CONSO: Run all the enrichers configured with a Pre-Consolidation Only or Pre and Post Consolidation scope. -
ALL: Run all enrichers defined for the entity. -
A list of enricher names in the following format:
[ "<enricher_name>", ... ]
-
-
validations: Defines the validations that should be executed after the enrichers. By default, no validation is executed. Possible values are:-
JOB_PRE_CONSO: Run all validations configured with a Pre-Consolidation Only or Pre and Post Consolidation scope. -
ALL: Run all the validations defined for the entity. -
A list of validations with their name and type in the following format:
[ { "validationType": "<validation_type>", "validationName": "<Validation_name>" }, ... ]Possible
validationTypevalues areCHECK,PLUGIN,MANDATORY,LOV,FOREIGNorUNIQUE.
-
-
queryPotentialMatchesRules: Defines the match rules to use to detect potential matches. By default, no match detection is performed. Possible values are:-
ALL: Run all the match rules defined for the entity. -
A list of match rule names, in the following format:
[ "<match_rule_name>", ... ]
-
-
queryPotentialMatchesHighestScoreOnly: When set to true, only the match found with the highest match score is returned in the response. Otherwise, all matches found are returned.The
queryPotentialMatchesparameter is deprecated and is replaced withqueryPotentialMatchesRulesandqueryPotentialMatchesHighestScoreOnly. -
queryPotentialMatchesBaseExpressions: Defines the set of base attributes to include in the response for the potential matches found. Possible values are:-
NONE: No attributes. -
USER_ATTRS(default): All entity attributes, except the built-in attributes and references. -
VIEW_ATTRS: All entity attributes, except references. Includes the built-in attributes. -
ID: Only the identifier attributes. Depending on the entity type and the matched record’s location,PublisherID,SourceID, the golden ID, and/or the primary key will be returned.
-
-
queryPotentialMatchesExpressions: Expressions to include to include in the response for the potential matches found, in addition to the base attributes (queryPotentialMatchesBaseExpressions). These expressions are in the following format:[alias]:[semql_expression]. -
responsePayloadRecordsBaseExpressions: Defines the set of base attributes to include in the response for the persisted records. Possible values are:-
NONE: No attributes. -
USER_ATTRS(default): All entity attributes, except the built-in attributes and references. -
VIEW_ATTRS: All entity attributes, except references, including the built-in attributes. -
ID: Only the identifier attributes. Depending on the entity type,PublisherID,SourceID, and/or the primary key are returned.
-
-
responsePayloadRecordsExpressions: Expressions to include in the response for the persisted records, in addition to the base attributes (responsePayloadRecordsBaseExpressions). These expressions are in the following format:[alias]:[semql_expression].
-
-
For the entire load:
-
missingIdBehavior: Option to define whether to generate IDs when they are not provided in the payload. Possible values areGENERATEto generate the ID orFAILto fail loading if the ID is missing. -
persistMode: Flag to define whether the records should be persisted or not. Possible values are:-
IF_NO_ERROR_OR_MATCH (Default value): Persist a record if no validation error was raised and no potential match was found.
-
ALWAYS: Always persist a record.
-
NEVER: Never persist a record.
-
-
responsePayload: This flag specifies the content of the response payload. Possible values are:-
RECORDS (Default value): Details of the persisted records.
-
SUMMARY: Count of persisted records.
-
SUMMARY_AND_RECORDS: Count and details of the persisted records.
-
-
Update Existing Records
The REST API allows to update Golden records for basic entities and Master records for ID and Fuzzy matched entities.
Single Update
You update an existing record by providing its ID when loading the data:
-
For basic entities, the ID attribute of the record to update.
-
For ID and Fuzzy-matched entities, the Source ID and Publisher ID of the master record to update.
If you persist a record in a load with an existing record ID, a copy of the record is checked out and changes are applied only to the fields provided in the request body. Other attributes still keep their value.
Mass Update
With the MASS_UPDATE_DATA action, you can use the endpoint to update multiple records within the same payload.
Method |
POST |
||
|---|---|---|---|
Base URL |
|
||
URL |
|
||
Request Payload |
The request contains the
For each entity you can set the following options:
Example 7. Mass Update Data: sample request to mass update records.
|
||
Response Format |
The response contains the status of the request, the load information as well as a the summary or the list of all the records updated as part of this request. Example 8. Mass update data: sample response.
|
Use the restApiMassUpdateFetchBatchSize system property to change the fetch batch size when mass updating records. The default value is 1000.
|
Set the Auditing Fields
By default, the auditing fields, Creator, Updator, CreateDate, UpdateDate are automatically set to the current user name and the current date when the user publishes the data. These values cannot be set by the user.
However, a user may be configured to push values in these auditing fields to publish data, for example, in the past, or on behalf of other users.
A user with a role having the Allow Publishing as user in API option selected in a model privilege grant is able to set the audit fields (Creator, Updator, CreateDate, UpdateDate) while publishing data via the REST API.
Enrich, Validate and Detect Matches
When loading or updating data, you can choose to execute enrichers, validations, and matchers for each entity.
Enrich and Validate
You may define, for each entity you want to configure, under the optionsPerEntity element, one element named after the entity.
For each entity, you can:
-
give a list of
enrichersto run, with their enricher names, -
give a list of
validationsto execute, with their validationType and validationName.
Detect Matches
When loading data, indicate with the queryPotentialMatchesRules whether the platform should check for duplicates according to the matching rules defined for the entity.
When checking for duplicates, master records potentially matching an incoming record are returned in the response, to help identify the reason for the matches.
You can define the attributes returned to only include those required for a specific use case, with two additional properties:
-
queryPotentialMatchesBaseExpressionsdefines the set of base attributes to include in the master records detected as potential matches:-
NONE: no attributes. -
USER_ATTRS: all attributes, except built-in attributes and references. -
VIEW_ATTRS: all attributes, except references. This includes built-in attributes. -
ID: only the identifier attributes. Depending on the entity type and the matched record’s location, PublisherID, SourceID, the golden ID, and/or the primary key will be returned.
-
-
queryPotentialMatchesExpressionsdefines a list of expressions to return in addition to the set of base attributes. These expressions are in the following format:<alias>:<semql_expression>.
For example, the following request looks for potential matches.
{
"action":"PERSIST_DATA",
"persistOptions": {
"defaultPublisherId": "CRM",
"optionsPerEntity": {
"Person": {
"queryPotentialMatchesRules": ["SameExactEmailMatchName"], (1)
"enrichers":["CleanseEmail"],
"validations":[],
"queryPotentialMatchesBaseExpressions": "NONE", (2)
"queryPotentialMatchesExpressions": { (2)
"Name": "Concat(FirstName, ' ', LastName)",
"Email": "CleansedEmail",
"Golden ID": "Gold_ID",
"Master ID": "ID"
}
}
},
"missingIdBehavior": "FAIL",
"persistMode": "NEVER" (3)
},
"persistRecords": {
"Person": [
{
"SourceID": "99998",
"FirstName": "John",
"LastName": "Doe",
"DateOfBirth": "1974-01-25",
"SourceEmail": "jass@ellerbusch.com"
}
]
}
}| 1 | Trigger potential matches detection for the records using one match rule. Note that since the matcher uses enriched values, the CleanseEmail enricher is also triggered. |
| 2 | Select the information returned for the potential matches. Since queryPotentialMatchesBaseExpressions is set to NONE, only the expressions defined in the queryPotentialMatchesExpressions are returned. |
| 3 | This value for PersistMode never persists the records. This call only finds potential matches. |
The response to this request is as follows:
{
"status": "PERSIST_CANCELLED",
"load": {
...
},
"records": {
"Person": [
{
"entityName": "Person",
"recordValues": {
"CleansedEmail": "jass@ellerbusch.com",
"SourceEmail": "jass@ellerbusch.com",
"DateOfBirth": "1974-01-25",
"FirstName": "John",
"SourceID": "99998",
"PublisherID": "CRM",
"LastName": "Doe",
...
},
"failedValidations": [],
"potentialMatches": [
{
"matchRuleName": "ExactEmailMatch",
"matchScore": 74,
"matchedRecordLocation": "MD",
"matchedRecordId": {
"SourceID": "1320830",
"PublisherID": "CRM"
},
"matchedRecordData": {
"Name": "Jass Ellerbusch",
"Email": "jass@ellerbusch.com",
"Golden ID": 10002,
"Master ID": "CRM.1320830"
}
}
]
}
]
}
}Data Loading Behavior
When invoked with a payload, the REST operation will run the enrichers, validations and matchers for each record, depending on the entity configuration.
It will then return:
-
The enriched data
-
a list of validation errors if any
-
a list of potential matches detected by the matching rules.
The record may be persisted or not at that stage, depending on the persistMode option:
-
If set to
ALWAYS, then the records are persisted even if they have errors and potential matches. -
If set to
NEVER, then the records are not persisted. Use this option to perform a dry-run to test your records. -
If set to
IF_NO_ERROR_OR_MATCH(default value), then the records are persisted if no validation error was raised and no potential match was found.
Publish Deletions
You can use the endpoint to load data, with the DELETE_DATA action, in order to publish record deletions.
Method |
POST |
||||||
|---|---|---|---|---|---|---|---|
Base URL |
|
||||||
URL |
|
||||||
Request Payload |
The request contains the
Submitting other information than these IDs in this property is considered an error. Example 11. Delete data: sample request to delete a golden record.
|
||||||
Response Format |
The response contains the status of the request, the load information as well as a list of all the records deleted as part of this request (including child records deleted by a cascade). Example 12. Delete data: sample response.
The record deletion
|
Submit a Load
To submit a load, the URL must contain the Load ID that was returned at load creation time.
Method |
POST |
|---|---|
Base URL |
|
URL |
|
Request Payload |
The request contains the Example 13. Load submission: sample request.
|
Response Format |
The response contains the load information, including the load ID, batch ID, and an indication of the status. Example 14. Load submission: sample response.
|
Cancel a Load
To cancel a load, the URL must contain the Load ID that was returned at load creation time.
Method |
POST |
|---|---|
Base URL |
|
URL |
|
Request Payload |
The request contains only the Example 15. Load cancellation: sample request.
|
Response Format |
The response contains load ID as well as an indication of the status. Example 16. Load cancellation: sample response.
|
Load and Submit Data
Using the REST API, it is possible to create a load, load data (or request deletions) and submit the load in a single request.
Method |
POST |
||||||
|---|---|---|---|---|---|---|---|
Base URL |
|
||||||
URL |
|
||||||
Request Payload |
The request contains the Example 17. Load and Submit: sample request.
|
||||||
Response Format |
The response contains load ID as well as an indication of the status. Example 18. Load and Submit: sample response.
|
Manage a Load
When a load was submitted, it is still possible to manage it using the REST API.
Method |
POST |
|---|---|
Base URL |
|
URL |
|
Request Payload |
The request contains the Example 19. Load submission: sample request.
|
Response Format |
The response contains the load information and its new state. If the requested operation is not possible, an error is returned. |
| Combine this endpoint with the capability to query loads for automating production monitoring, for example to automatically resume jobs that are suspended. |