Configure data notifications

This page describes how to configure data notifications with Semarchy xDM.

Prepare the named query

Data notification messages are structured after named queries. Data propagated by a data notification is filtered using the related named query’s filter and named query parameters.

  • To filter records created or updated within a range of batch IDs, the named query must include the following elements:

    • A filter having two parameters—​minimum batch ID and maximum batch ID—​to define the range of batch IDs. A typical named query filter for data notifications looks as follows:

      BatchID > :QUERY_PARAM_PREVIOUS_BATCH
and BatchID <= :QUERY_PARAM_CURRENT_BATCH

    • Two named query parameters for minimum batch ID and maximum batch ID used in the filter:

      • CURRENT_BATCH (number) using the binding name QUERY_PARAM_CURRENT_BATCH.

      • PREVIOUS_BATCH (number), using the binding name QUERY_PARAM_PREVIOUS_BATCH.

  • To filter data depending on the date and time when integration batches ran, the following named query parameters can be added to the named query and used in its filter:

    • CURRENT_BATCH_TIMESTAMP (timestamp) using the binding name QUERY_PARAM_CURRENT_BATCH_TIMESTAMP.

    • PREVIOUS_BATCH_TIMESTAMP (timestamp), using the binding name QUERY_PARAM_PREVIOUS_BATCH_TIMESTAMP.

  • To filter data depending on the date and time when the data notification is triggered, the following named query parameter can be added to the named query and used in its filter:

    • NOTIFICATION_TIMESTAMP (timestamp) using the binding name QUERY_PARAM_NOTIFICATION_TIMESTAMP.

Create a data notification

Data notifications are configured within data locations. To create a data notification:

  1. In the Management view, expand the Data Locations node, and then expand the data location for which you want to configure a data notification.

  2. Right-click the Data Notifications node and click Create new data notification New Data Notification. The Create New Data Notification wizard opens.

  3. Enter the following information:

    • Name: internal name of the data notification.

    • Named Query: click Edit to open the Select Named Query picker. Select the named query that will define the notification dataset content. Click OK to close the picker.

      To make a new named query visible in the Select Named Query picker, refresh your application (on a development instance) or redeploy the model (on a production instance).

    • Target Type: select the type of channel meant to receive data notifications. There are three types of targets:

      • JMS (Java Message Service) queue or topic.

      • Kafka topic.

      • REST API endpoint.

  4. Click Finish. The Data Notification editor opens.

  5. (Optional) Enter a description.

  6. Configure the content of data notifications in the Dataset section of the editor:

    1. (Optional) Set the Run as User property with the user required to resolve model variables from the named query.

    2. Select the View Type to define the type of data sent in the notification. For more information, see View Types.

      Not all view types are available for data notifications. For example, views bound to loads and batches are not listed.

    3. Set the root records limit to restrain the number of records sent in a notification. If more records are retrieved from the named query, the notification will automatically fail.

  7. Configure the named query parameters. Parameters from the selected named query are listed in the Named Query Parameters table view.

    1. (Optional) Click Refresh to update the list of parameters if the named query was updated.

    2. (Optional) Click Delete to remove parameters that you do not want to map.
      For each parameter, set the following properties:

      • Parameter Type: type mapped to the named query parameter, which can be either:

        • Literal: the query parameter value is a literal text value.

        • Variable: the query parameter value is the evaluation of the provided variable. The following variables are available:

          Variable Name Definition

          HIGH_BATCH_ID

          ID of the current batch running the integration job.

          HIGH_BATCH_TIMESTAMP

          Timestamp when the batch with HIGH_BATCH_ID was submitted.

          LOW_BATCH_ID

          ID of the previous batch that ran the integration job.

          LOW_BATCH_TIMESTAMP

          Timestamp when the batch with LOW_BATCH_ID was submitted.

          NOTIFICATION_TRIGGER_TIMESTAMP

          Timestamp when the data notification is triggered.

      • Parameter Value:

        • If Parameter Type is set to Literal: populate with the literal value to use.

        • If Parameter Type is set to Variable: populate with the name of the variable.

To only return records created or updated within a range of batch IDs, a configuration needs to map the following named query parameters with built-in variables:

  • CURRENT_BATCH: set with the HIGH_BATCH_ID built-in variable that contains the highest batch ID in the data location.

  • PREVIOUS_BATCH: set with the LOW_BATCH_ID variable that contains the current value of the low batch ID.

The CURRENT_BATCH and PREVIOUS_BATCH named query parameters are named following the named query prerequisites.

To filter a named query depending on batch execution timestamps, the following built-in variables are available and can be mapped with named query parameters (see named query prerequisites):

  • PREVIOUS_BATCH_TIMESTAMP: contains the timestamp of the previous recent batch (identified by the low batch ID).

  • CURRENT_BATCH_TIMESTAMP: contains the timestamp of the most recent batch (identified by the high batch ID).

To filter a named query depending on the data notification firing timestamp, the built-in NOTIFICATION_TRIGGER_TIMESTAMP variable is available and can be mapped with a named query parameter. It provides the timestamp when the current data notification instance was triggered.

  1. Configure the Trigger. Data notifications can be triggered:

    • On schedule: a scheduled event defined with a cron expression will send the notification. To configure the on-schedule trigger:

      1. Set the Trigger on property to Schedule.

      2. Click Edit next to the Cron Expression property to open the Cron Schedule dialog.

      3. Select a predefined frequency or select Cron Expression to enter a custom expression, and then click OK.

    • On batch completion: the data notification is executed once an integration job completes. To configure the on-batch-completion trigger:

      1. Set the Trigger on property to Batch complete.

      2. (Optional) Click Edit to open the Watched Entities selector. The data notification will be triggered only if the batch processes the selected entities. Leave this property empty to watch all entities.

  1. Configure the notification messages in the Message section:

    • Records Per Message: number of records per message. If the total number of records returned by the named query is higher than this value, the query results will be split over multiple messages. An empty value means that all records will be sent in one message (default: empty).

    • Notify On Empty Dataset: defines whether a notification should be sent if the named query returns no result (default: unselected).

      Selecting this option is highly recommended when testing connectivity so that a notification is sent even for an empty dataset.

    • Max Retries: maximum number of retries in case of error when sending messages to the target. After all retries, the notification is considered failed and future notifications are automatically suspended (default: 1).

    • Retry Interval: Duration in seconds between two retries (default: 60).

  1. Set Target parameters. Depending on the selected target, the following properties are editable:

    • For JMS:

      • Connection Factory: name of the object used by the JMS client to create a connection with a JNDI provider. The default value is ConnectionFactory.

      • JMS Destination: name of the JMS queue or topic the notification messages must be sent to. Prefix the name with queue/ or topic/ depending on the destination type. If no prefix is provided, queue/ is assumed.

      • JMS Username: name of the user required to connect to the JMS destination. Leave empty if no authentication is required by the broker (default: empty).

      • JMS Password: password required to connect to the JMS destination. Leave empty if no authentication is required by the broker (default: empty).

    • For Kafka:

      • Kafka Topic: name of the Kafka topic notification messages must be sent to.

      • Kafka Message Key: key to use for Kafka messages (if empty, the name of the data notification will be used).

        The URL of the Kafka broker and other connection parameters will be set in the Target Properties section.

    • For REST:

      • REST Request URL: URL of the REST endpoint. The value must include HTTP/S.

      • REST Timeout: timeout for the HTTP request in seconds. After this timeout, the notification is considered failed and future notifications are automatically suspended.

  1. Configure target properties: to cover all technical requirements of the middleware or downstream applications to which the notification messages are sent, additional properties can be provided in the Target Properties section using key-value pairs.

    • For a REST target, these properties will be passed as HTTP headers of the request.

    • Kafka and JMS targets use these properties to configure the connection.

      To add a property:

      1. Click the Add target property Add Target Property button. A new line is added to the Target Properties table view.

      2. Enter the Property Name (name of the JMS/Kafka property or HTTP header) or click the Select property Select Property button to select the property within a list of predefined keys per target type.

      3. Enter the Property Value. If required (e.g., in the case of a password), this value can be transformed into a secret by clicking the Make secret Make Secret button.

    If the target of the notification is a Kafka topic, you must configure the broker’s URL by adding a property with the predefined key bootstrap.servers.

  1. Validate and save. The configuration of a data notification is validated every time it is saved or when the user clicks the data notif validate Validate button in the editor. A data notification cannot be saved until all errors are fixed.

  2. Test and prepare the data notification for execution. Once the data notification is saved, the following actions are available:

    • Send Manual Notification: sends a data notification on a defined range of batch IDs, for test or replay purposes.

    • Set Low Batch ID: allows to set the value of the LOW_BATCH_ID variable.

    • Activate: activates the data notification trigger.

Configure data notifications using the REST API

Endpoints are available on the Semarchy xDM REST API to consult and configure data notifications.

For more details, see the REST API documentation.