Managing Execution

The Execution Engine processes the jobs submitted by the integration batch poller. It orchestrates the certification process for golden data.

Understanding Jobs, Queues and Logs

Jobs are processing units that run in the execution engine. There are two main types of jobs running in the execution engine:

Jobs are processed in Queues. Queues work in First-In-First-Out (FIFO) mode. When a job runs in the queue, the next jobs are queued and wait for their turn to run. To run two jobs in parallel, it is necessary to distribute them into different queues.

Queues are grouped into Clusters. There is one cluster per Data Location, named after the data location.

System Queues and Clusters

Specific queues and cluster exist for administrative jobs:

Job Priority

As a general rule, integration jobs are processed in their queues, and have the same priority.
There are two exceptions to this rule:

Model Edition Deployment Job

When a new model edition is deployed and requires data structure changes, DDL commands are issued as part of a job called DB Install<model edition name>. This job is launched in the in the SEM_SYS_QUEUE queue of the data location cluster.

This job modifies the tables used by the DML statements from the integration jobs. As a consequence, it needs to run while no integration job runs. This job takes precedence over all other queued jobs in the cluster, which means that:

  1. Jobs currently running in the cluster are completed normally.
  2. All the queues in the cluster, except the SEM_SYS_QUEUE are moved to a BLOCKED status. Queued jobs remain in the queue and are no longer executed.
  3. The model edition deployment job is executed in the SEM_SYS_QUEUE.
  4. When this job is finished, the other queues return to the READY status and resume the processing of their queued jobs.

This execution model guarantees a minimum downtime of the integration activity while avoiding conflicts between integration jobs and model edition deployment.

Platform Maintenance Job

If a job is queued in the SEM_SYS_CLUSTER/SEM_SYS_QUEUE queue, it takes precedence over all other queued jobs in the execution engine.
This means that:

  1. Jobs currently running in all the clusters are completed.
  2. All the clusters and queues except the SEM_SYS_CLUSTER/SEM_SYS_QUEUE are moved to a BLOCKED status. Queued jobs are no longer executed in these queues/clusters.
  3. The job in the in the SEM_SYS_CLUSTER/SEM_SYS_QUEUE is executed.
  4. When this job is finished, the other queues are moved to the READY status and resume the processing of their queued jobs.

This execution model guarantees a minimal disruption of the platform activity while avoiding conflicts between the platform activity and maintenance operations.

Queue Behavior on Error

When a job running in a queue encounters a run-time error, it behaves differently depending on the queue configuration:

Warning: The integration jobs are processed in a FIFO mode, a job that is failed automatically or cancelled by user choice cannot be restarted. To resubmit the source data for certification, the external load needs to be resubmitted entirely as a new load.

Warning: The integration job performs a commit after each task. As a consequence, when a job fails or is suspended, already processed entities have their golden data certified and committed in the hub.

Tip: A user can explicitly choose to halt a running job by suspending it. When such a use operation is performed, the job is considered in error and can be restarted or canceled.

Suspending a job on error is the preferred configuration under the following assumptions:

  1. All the data in a batch needs to be integrated as one single atomic operation. For example, due to referential integrity, it is not possible to integrate contacts without customers and vice versa. Suspending the job guarantees that it can be continued – after fixing the cause of the error – with the data location preserved in the same state.
  2. Batches and integration jobs are submitted in a precise sequence that represents the changes in the source, and need to be processed in the order they were submitted. For example, missing a data value change in the suspended batch that may impact the consolidation of future batches. Suspending the job guarantees that the jobs are processed in their exact submission sequence, and no batch is skipped without an explicit user choice.

There may be some cases when this behavior can be changed:

Queue Status

A queue is in one the following statuses:

A cluster can be in one the following statuses:

Managing the Execution Engine and the Queues

Accessing the Execution Engine

To access the execution engine:

  1. In the Administration view, double-click the Execution Engine node.

The Execution Engine editor opens.

The Execution Engine Editor

This editor displays the list of queues, grouped by clusters. If a queue is currently pending on a suspended job, it appear in red.

From the Execution Engine editor, you can perform the following operations:

Changing the Queue Behavior on Error

Note: See the Troubleshooting Errors and the Queue Behavior on Error sections section for more information about queue behavior on error and error management.

To change the queue behavior on error:

  1. In the Administration view, double-click the Execution Engine node. The Execution Engine editor opens.
  2. Select or de-select the Suspend on Error option for a queue to set its behavior on error or on a cluster to set the behavior of all queues in this cluster.
  3. Press CTRL+S to save the configuration. This configuration is immediately active.

Opening an Execution Console for a Queue

The execution console provides the details of the activity of a given queue. This information is useful to monitor the activity of jobs running in the queue, and to troubleshoot errors.

Note: The content of the execution console is not persisted. Executions prior to opening the console are not displayed in this console. Besides, if the console is closed, its content is lost.

To open the execution console:

  1. In the Administration view, double-click the Execution Engine node. The execution engine editor appears.
  2. Select the queue, right-click and select Open Execution Console.

The Console view for this queue opens. Note that it is possible to open multiple execution consoles to monitor the activity of multiple queues.

In the Console view toolbar you have access to the following operations:

Suspending a Job Running in a Queue

To restart a suspended job in a queue:

  1. In the Administration view, double-click the Execution Engine node. The execution engine editor appears..
  2. Select the queue that contains one Running Job.
  3. Right-click and then select Suspend Job.

The job is suspending and the queue switches to the SUSPENDED status.

Warning: Suspending the job is an operation that should be performed with care, as respecting the sequence of the submitted job have strong impact on the consistency of the data in the hub.

Restarting a Suspended Job in a Queue

To restart a suspended job in a queue:

  1. In the Administration view, double-click the Execution Engine node. The execution engine editor appears. The suspended queue appears in red.
  2. Select the suspended queue.
  3. Right-click and then select Restart Job.

The job restarts from the failed step. If the execution console for this queue is open, the details of the execution are shown in the Console.

Canceling a Suspended Job in a Queue

To cancel a suspended job in a queue:

  1. In the Administration view, double-click the Execution Engine node. The execution engine editor appears. The suspended queue appears in red.
  2. Select the suspended queue.
  3. Right-click and then select Cancel Job.

The job is cancelled, the queue become READY and starts processing queued jobs.
In the job logs, this job appears in Error status.

Managing Jobs Logs

The job logs display the jobs being executed or executed in the past by the execution engine. Reviewing the job logs allows you to monitor the activity of these jobs and troubleshoot execution errors.

Accessing the Job Logs

To access the logs:

  1. Open the Administration Console perspective.
  2. In the Administration View, double click the Job Logs node.
  3. The Job Logs editor opens.

The Job Logs Editor

From this editor you can review the job execution logs and drill down into these logs.

The following actions are available from the Job Logs editor toolbar.

Drilling Down into the Logs

The Job Logs editor displays the log list. This view includes:

To drill down into the logs:

  1. Double-click on a log entry in the Job Logs editor.
  2. The Job Log editor open. It displays all he information available in the job logs list, plus:
  3. Double-Click one entity in the Tasks list to drill down into the Task Group Log corresponding to this entity. The Task Group Log for the entity shows the details of the entity task, and the list of task groups performed for the entity. These tasks groups represent the successive operations performed for the given entity. For example: Enrich and Standardize, Validate Source Data, etc.
  4. Double-click one of the task groups in the Task Log list to drill down into one of the tasks group. Each tasks may contain one of more tasks or child task groups. For example, the Enrich and Standardize task group contains the log of the enrichers executed for the given entity.
  5. Double click of the task group to drill down down into the Task Log.
  6. The task log shows the task statistics, and provides a link to the Task Definition.

By drilling down into the task groups down to the task, it is possible to monitor the activity of a job, and review in the definition the executed code or plug-in.

Filtering the Logs

To create a job log filter:

  1. In the Job Logs editor, click the Apply and Manage User Defined Filters button and then select Search. The Define Filter dialog opens.
  2. Provide the filtering criteria:
  3. Click the Save as Preferred Filter option and enter a filter name to save this filter.

Saved filters appear when you click the Apply and Manage User Defined Filters button.
You can enable of disable a filter by marking it as active or inactive from this menu. You can also use the Apply All and Apply None to enable/disable all saved filters.

Note: Filters are saved in the user preferences and can be shared using preferences import/export.

To manage job log filters:

  1. Click the Apply and Manage User Defined Filters button, then select Manage Filters. The Manage User Filters editor opens.
  2. From this editor, you can add, delete or edit a filter, and enable disable filters for the current view.
  3. Click Finish to apply your changes.

Purging the Logs

You can purge selected job logs or all job logs returned by a filter.

To purge selected job logs:

  1. In the Job Logs editor, select the job logs that you want to purge. Press the CTRL key to select multiple lines or the SHIFT key to select a range of lines.
  2. Click the Purge Selection button.
  3. Click OK in the confirmation window.

The selected job logs are deleted.

To purge filtered job logs:

  1. In the Job Logs editor, click the Purge using a Filter button.

The jobs logs are purged.

Note: It is possible to trigger job logs purges through web services. The Administration Service exposes such operations.

Troubleshooting Errors

When a job fails, depending on the configuration of the queue into which this job runs, it is either in a Suspended or Error status.

The status of the job defines the possible actions on this job.

You have several capabilities in Semarchy Convergence for MDM to help you troubleshooting issues. You can drill down in the erroneous task to identify the issue or restart the job with the Execution Console activated

To troubleshoot an error:

  1. Open the Job Logs.
  2. Double-click the log entry marked as Suspended or in Error.
  3. Drill down into the Task Log, as explained in the Drilling Down into the Logs section.
  4. In the Task Log, review the Message.
  5. Click the Task Definition link to open the task definition and review the SQL Statements involved, or the plug-in called in this task.

Scheduling Data Purges

Data Purge helps you maintain a reasonable storage volume for the MDM hub and the repository by pruning historical data and job logs.

Introduction to Data Purge

The MDM hub stores the lineage of the certified golden data, as well as the changes that led to this golden data.
Preserving the lineage and history is a master data governance requirement. It is key in a regulatory compliance focus. However, keeping this information may also create a large volume of data in the hub storage.

To make sure lineage and history are preserved according to the data governance and compliance requirements, model designers define Data Retention Policy in the model. To keep a reasonable volume of information, administrators have to schedule regular Purges for this data.

Purges are managed by the Purge Scheduler. This service manages purge schedules, and triggers the appropriate purge job on the execution engine to prune the lineage and history according to the Data Retention Policy.

The purges delete the following elements of the lineage and history:

Optionally, the job logs can also be deleted as part of the purge process.

Accessing the Purge Scheduler

To access the purge scheduler:

  1. Open the Administration Console perspective.
  2. In the Administration View, double click the Purge Scheduler node. The Purge Scheduler editor opens.

This editor displays the scheduled purges. From the Purge Scheduler editor, you can stop, start or restart the Purge Scheduler service.

Creating a Purge Schedule

To create a purge schedule:

  1. In the Purge Scheduler editor toolbar, click the New Purge Schedule button. The Data Branch Purge Scheduling wizard opens.
  2. Select the data branches that you want to purge and then click the Add >> Button.
  3. Click the Next button.
  4. Set the schedule for the purge with a purge frequency ( Monthly, Weekly, Daily) or as a Cron Expression.
  5. Select the Purge Repository Logs option to prune the logs related to the purged history and lineage.
  6. Click Finish to close the wizard.
  7. Press CTRL+S to save the editor.

Note: Regardless of the frequency of the purges scheduled by the administrator, the data history retained is as defined by the model designer in the data retention policies.

Note: It is possible to trigger data purges through web services. The Administration Service exposes such operations.