Manage REST clients

REST clients extend the capabilities of Semarchy xDM to process data by calling external services via their REST API.

You can define a REST client for the entire Semarchy xDM platform in Semarchy Configuration, and then use this client in all the models in the repository.

REST clients overview

When Semarchy calls a REST client to enrich a record, the client sends a request sent to the external service.
This request is defined by:

  • An HTTP URL, which is accessed using an HTTP method (GET, POST, etc.).

  • (Optional) HTTP headers, (e.g., to pass authentication information).

  • (Optional) A body, typically in JSON format.

The elements of the request can be parameterized by Request Inputs, passed to the REST client. These inputs are typically data from the record being enriched.

The external service returns a response that contains:

  • An HTTP status code (e.g., 200: success, 404: not found)

  • A response body, typically in JSON format from which response outputs will be extracted. These outputs are mapped to attributes in the records to enrich.

Declare a REST client

To create a REST client:

  1. In Semarchy Configuration, select REST Clients in the navigation drawer.

  2. In the REST Clients editor toolbar, select Add New new > New REST Client RestClient. The Create New REST Client dialog opens.

  3. Enter the following information:

    • Name: Name of the REST client. When a model uses a REST client, it refers to this client by this name.

    • Label: User-friendly label for the REST client. This label appears in the Workbench.

    • Description: Provide a meaningful description for the model designers who will use this client.

  4. Click Next.

  5. Select the HTTP Method for the REST endpoint.

  6. Enter the URL of the REST endpoint.
    You can make parts of this URL variable by replacing constant values with request inputs, using the ${request_input_name} syntax. For example, if the service URL contains a variable part to pass an email address, you can use the syntax below to define this address as an input of the REST client:

    https://information.acme.com/information?email_adr=${email}
  7. Enter the HTTP Headers possibly required by the endpoint. Similarly to the URL, you can use variable inputs in the headers.

  8. Select Body Format for the request: use NONE if the endpoint does not require a body, JSON for a JSON payload and BINARY for a binary payload.

  9. In the Body field, compose the JSON or binary payload required by the endpoint. Similarly to the URL, you can use variable inputs in the body.

  10. Click Finish. The REST client editor opens.

  11. Review and update the Name, Label and Description of the client, as well as the HTTP Request parameters.

  12. If you have used inputs using the ${request_input_name} in the wizard, they are automatically added to the Request Inputs list.
    You can also manage these inputs manually:

    • To detect new inputs added to the URL, headers or body, click Detect Inputs RequestInput Detect in the Request Inputs table toolbar.

    • To create a new input, click Add Input RequestInput in the Request Inputs table toolbar

    • To delete unused request inputs, select them and then click Delete delete in the Request Inputs table toolbar

  13. In the Request Inputs table, provide for each input:

    • A user-friendly Label and a meaningful Description for the model designer who will use the REST client.

    • A Data Type for the input. Note that this is the expected type of the input value. For more information, see Request input datatype and formatting below.

  14. In the Response Output table, click Add Output ResponseOutput.
    The new output appears in the list.

    • Define a Name, Label and Description for this output. The label and description appear in the Workbench for model designers. The Name is used by an enricher to refer to the output.

    • Set the Value Selector. This selector is a JSONPath expression, starting from the root of the response payload and pointing to a single value in the payload.

    • Select the Data Type of the output.

    • The Behavior on Absent Value defines whether the enricher should fail or return the default value if the JSONPath expression returns no value for the payload.

      The behavior defined for absent values does not apply to null values. Null values received in the response payload are returned by the REST client as they are, without any processing or alteration.

    • The Behavior on Data Type Mismatch defines whether the enricher should fail or return the default value if the JSONPath expression returns a value that is not compatible with the output data type.

    • Set a Default Value or leave it empty to have a null default value.

  1. Repeat the previous step for each output returned by the REST client.
    For more information on outputs, see Response processing.

  2. In the Advanced Configuration section of the editor, set the following parameters:

    • Timeout for the HTTP request. After this timeout, the call is considered failed.

    • Max Retries: number of times the request should be attempted. The retries occur only if the service reports a code corresponding to a temporary failure condition. These include HTTP 404, 410, 429 and 503.

    • Success Status Codes: a comma-separated list of HTTP response status (e.g., 200, 201, 202) and response status patterns (e.g., 2xx) considered successful. The default is 2xx.

    • Error Selector: JSONPath expression returning a possible error to search in the response payload. Note that this path is evaluated even for a successful response. If it returns a value in that case, the error is raised.

  3. Press Control+S (or Command+S on macOS) to save the REST client.

Before proceeding, make sure to test the REST client.

Request input datatype and formatting

When using a request input in the REST client definition, the datatype defines how values for this input are processed and inserted in the request:

Automatic formatting:

  • Binary inputs are automatically encoded in BASE64 when used in the URL, the headers or in a JSON body. In a binary body, a binary input is not formatted.

  • Date and timestamp inputs are formatted as strings in the ISO_8601 format (YYYY-MM-DD for dates, YYYY-MM-DDTHH:mm:ss.SSSZ for timestamps).

  • Number inputs are formatted as strings with a dot as the decimal separator and no thousand separators.

  • Boolean inputs are not formatted and will appear as true or false.

The inputs are then substituted by their formatted values in the request.

During that replacement:

  • Encoding and escaping are automatically handled in the URLs and JSON.

  • Quotes are added in the JSON Body around strings, binary, dates and timestamps.

  • Input with null values will be replaced by null in the JSON payload.

Formatting in a JSON Body.

In the example below:

  • fullName and comment are string. They are automatically escaped and surrounded by quotes.

  • height in a numeric input. It is formatted with no quotes.

  • Picture is a binary. It is encoded in Base64 and surrounded by quotes.

{
    "Name"      : ${fullName},
    "Comment "  : ${comment},
    "Height"    : ${height},
    "Photo"     : ${picture}
}

A possible result will be:

{
    "Name"      : "John \"Joe\" Doe",
    "Comment "  : null,
    "Height"    : 180.2,
    "Photo"     : "data:image/png;base64,iVBOR...."
}
Example of URL Encoding

If you pass the value "Mountain View California" for the address input in the following URL:

https://information.acme.com/address?addr=${address}

The address input will be automatically encoded in the URL:

https://information.acme.com/address?addr=Mountain%20View%20California

Response processing

The REST client extracts from the response payload:

  • The output values, using Value Selectors.

  • A possible error message, using the Error Selector.

Both these selectors use JSONPath, a JSON query language.

The full JSONPath syntax supported by Semarchy xDM is detailed at https://github.com/json-path/JsonPath.
JSONPath supports retrieving arrays of data. For the value selectors, if an array is returned by the JSON Path, the first element of the array is extracted in the output.
Simple JSONPath examples

With the following payload

{
    "customers":  [
        {
        "first name": "Joe",
        "last name": "Celko",
        "age": 58,
        },
        {
        "first name": "Roger",
        "last name": "Moore",
        "age": 69,
        }
    ],
    "customers_count": 2
}

The following JSONPath expressions return the following values:

  • $.customers_count returns '2'

  • $.customers[0].age returns '58'

  • $['customers'][0]['age'] returns '58' (different syntax)

  • $['customers'][1]['last name'] returns 'Moore'

Test the REST client

To test a REST client:

  1. In the REST Clients editor toolbar, select Manage manage > Test.
    The REST client test dialog opens.

  2. Enter values for the request inputs, and then click Execute.

The REST client calls the endpoints, processes the outputs and shows the resulting values.

Import and export REST clients

REST Clients are stored in the repository. When moving a model to a different repository (for example, from a development to a test environment), you must also move the REST clients used by this model.

  • In the source environment, click Export RestClient Export in the REST Clients editor to export the client definition to a file.

  • In the target environment, in the REST Clients editor toolbar, select Add New new > Import RestClient Import to import the exported client definition.

You can also use the platform REST API to automate the export/import of the REST clients across environments.