Configure data notifications

Prepare the named query

Data Notification messages are structured after a Named Query.

To be used by a Data Notification, a Named Query must filter records created or updated within a range of Batch IDs and therefore should include the following elements:

  • A filter using two parameters - Minimum Batch ID and Maximum Batch ID - to define the range of Batch IDs. A typical Named Query filter for Data Notifications looks as follows:

  • Two Named Query Parameters for the minimum Batch ID and the maximum Batch ID used in the filter:

    • CURRENT_BATCH (Number), using the Binding Name QUERY_PARAM_CURRENT_BATCH .

    • PREVIOUS_BATCH (Number), using the Binding Name QUERY_PARAM_PREVIOUS_BATCH.

    Additional Named Query parameters can be configured if needed.

Create a data notification

Data Notifications are configured within Data Locations. To create a Data Notification:

  1. In the Management view, expand the Data Locations node, and then expand the data location for which you want to configure a data notification. Right-click the Data Notifications node and select data notification New Data Notification. The Create New Data Notification wizard opens.

  2. Enter the following information:

    • Name: Internal name of the data notification.

    • Named Query: Click Edit to open the Select Named Query picker. Select the named query that will define the notification dataset content. Click OK to close the picker.

    To make a new Named Query visible in the Select Named Query picker, make sure you refresh your application (on a development instance) or redeploy the model (on a production instance).
    • Target Type: Select the type of channel that will receive data notifications. There are three types of targets:

      • JMS (Java Message Service) queue or topic.

      • Kafka topic.

      • REST API endpoint.

  3. Click Finish. The Data Notification editor opens.

  4. Enter an optional Description.

  5. Configure the content of data notifications in the Dataset section of the editor:

    • Optionally set the Run as User property with the user required to resolve Model Variables from the named query.

    • Select the View Type to define the type of data (see View Types) sent in the notification.

    Not all view types are available for data notifications. For example, views bound to loads and batches are not listed.

    • Set the Root Records Limit to restrain the number of records sent in a notification. If more records are retrieved from the named query, the notification will automatically fail.

  6. Configure the Named Query Parameters:
    Parameters from the selected named query are listed in the Named Query Parameters table view. You can click Refresh to update the list of parameters if the Named Query was updated, or click Delete to remove parameters that you don’t want to map.
    For each parameter you must set the following properties:

    • Parameter Type: Type mapped to the named query parameter, that can be either:

      • Literal: The query parameter value is a literal text value.

      • Variable: The Query parameter value is the evaluation of the provided variable.

    • Parameter Value:

      • If Parameter Type is set to Literal: populate with the literal value to use.

      • If Parameter Type is set to Variable: populate with the name of the variable.
        To return only records created or updated within a range of Batch IDs, a typical configuration will, at least, map the following Named Query Parameters with built-in variables:

        • CURRENT_BATCH: Set with the HIGH_BATCH_ID variable.

        • PREVIOUS_BATCH: Set with the LOW_BATCH_ID variable.

    The CURRENT_BATCH and PREVIOUS_BATCH Named Query Parameters are named following the Named Query prerequisites and are set using built-in variables:

    • LOW_BATCH_ID contains the current value of the low batch ID.

    • HIGH_BATCH_ID contains the highest batch ID in the Data Location.

  1. Configure the Trigger. Data Notifications can be triggered:

    • On schedule: A scheduled event defined with a CRON expression will send the notification. To configure the on schedule trigger:

      1. Set the Trigger on property to Schedule.

      2. Click Edit next to the Cron Expression property to open the Cron Schedule dialog.

      3. Select a predefined frequency or select Cron Expression to enter a custom expression, and then click OK.

    • On batch completion: The Data Notification is executed once an integration job completes. To configure the on batch completion trigger:

      1. Set the Trigger on property to Batch complete.

      2. Optionally, click Edit to open the Watched Entities selector. Data Notification will be triggered only if the batch processes the selected entities. Leave this property empty to watch all entities.

  1. Configure the notification messages in the Message section:

    • Records Per Message: Number the records per message. If the total number of records returned by the Named Query is higher than this value, the query results will be split over multiple messages. An empty value means that all records will be sent in one message (default: empty).

    • Notify On Empty Dataset: Defines whether a notification should be sent if the Named Query returns no result (default: unselected).

      Selecting this option is highly recommended when testing connectivity so that a notification is sent even for an empty dataset.
    • Max Retries: Maximum number of retries in case of error when sending messages to the target. After all retries, the notification is considered failed and future notifications are automatically suspended (default: 1).

    • Retry Interval: Duration in seconds between 2 retries (default: 60).

  1. Set Target parameters. Depending on the selected target, the following properties are editable:

    • For JMS:

      • JMS Destination: Name of the JMS queue or topic the notification messages must be sent to. Prefix the name with queue/ or topic/ depending on the destination type. If no prefix is provided, queue/ is assumed.

      • JMS Username: Name of the user required to connect to the JMS destination. Leave empty if no authentication is required by the broker (default: empty).

      • JMS Password: Password required to connect to the JMS destination. Leave empty if no authentication is required by the broker (default: empty).

    • For Kafka:

      • Kafka Topic: Name of the Kafka topic the notification messages must be sent to.

    The URL of the Kafka broker and other connection parameters will be set in the Target Properties section.
    • For REST:

      • REST Request URL: URL of the REST endpoint. The value must include HTTP/S.

      • REST Timeout: Timeout for the HTTP request in seconds. After this timeout, the notification is considered failed and future notifications are automatically suspended.

  2. Configure Target Properties: To cover all technical requirements of the middlewares or downstream applications the notification messages are sent to, additional properties can be provided in the Target Properties section using key-value pairs.

    • For a REST target, these properties will be passed as HTTP Headers of the request.

    • Kafka and JMS targets use these properties to configure the connection.

    To add a property:

    1. Click the data notif add property Add Target Property button. A new line is added to the Target Properties table view.

    2. Enter the Property Name (name of the JMS/Kafka property or HTTP header) or click the data notif select property Select Property button to select the property within a list of predefined keys per target type.

    3. Enter the Property Value. If required (for instance in the case of a password), this value can be transformed into a Secret by clicking the make secret Make Secret button.

    If the target of the notification is a Kafka topic, you must configure the broker’s URL by adding a property with the predefined key bootstrap.servers.
  1. Validate and save. The configuration of a Data Notification is validated every time it is saved or the user clicks the data notif validate Validate button in the editor. A Data Notification cannot be saved until all errors are fixed.

  2. Test and prepare the Data Notification for execution. Once the Data Notification has been saved, the following actions are available:

    • Send Manual Notification: Sends a Data Notification on a defined range of Batch IDs, for test or replay purposes.

    • Set Low Batch ID: Allows to set the value of the Low Batch ID variable.

    • Activate: Activates the Data Notification trigger.

Configure data notifications using the REST API

Endpoints are available on the Semarchy xDM REST API to consult and configure Data Notifications.

For more details, refer to the REST API documentation.