External Load

Publishing source data for certification into golden data is performed as a transaction. Such publisher transaction is called an External Load.

External Load Overview

An external load lifecycle is described below:

  1. Initialize the External Load
  2. Load Data
  3. Submit the External Load

The middleware can also Cancel the External Load to abort the external load instead of submitting it.

Important: A load is always performed on the most current data edition, which is the only open data edition of the data location. Previous data edition are closed, and cannot be modified. As a consequence, it is not possible to submit loads on these past editions.

Submitting a Load

Loads and Batches

Loading data in the hub is done within an External Load, which is a Convergence for MDM transaction identified by a sequential Load ID.

An External Load represents a source data load transaction.

When an External Load is submitted with an Integration Job Name, a Batch – identified by a Batch ID – is created, and the Job starts processing the data published in this load.

A Batch represents a transaction certifying loaded data and writing in the hub the resulting golden data.

Note: Both Loads and Batches can be reviewed from the Data Location perspective in the Semarchy workbench.

Process after a Submit

When an external load is submitted, the following operations take place:

  1. The platform creates a batch and returns to the submitter the Batch ID
  2. The integration batch poller picks up the batch on its schedule:
    1. It creates a Job instance using the Job Definition which name is provided in the submit action.
    2. It moves the job into the Queue specified in the job definition
  3. The Execution engine processes this job in the queue.

Even when multiple loads take place simultaneously, the sequence into which the external loads are submitted defines the order into which the data is processed by the integration jobs and golden data certified from this source data.