Semarchy xDM Application Management Guide

Perspectives

There are two perspectives in Semarchy Application Builder:

You switch from one perspective to another by clicking the Design or Management button in the Application Builder header.

Tree Views

When a perspective opens, a tree view showing the objects you can work with in this view appears on the left hand-side of the screen.
In this tree view you can:

Outline

Certain perspectives includes an Outline view that appears on the left hand-side of the screen.
This view shows the object currently opened in the editor (and all its child objects) in a tree view.

You can use the same expand, double-click, right-click actions in the outline as in the tree view.

Editors

An object currently being viewed or edited appears in an editor in the central part of the screen.
You can have multiple editors opened at the same time, each editor appearing with a different tab.

Working with Other Views

Other views (for example: Progress, Validation Report) appear in certain perspectives. These views are perspective-dependent.

You can open or re-open a view using the Show View…​ User Menu item.

User Preferences

User preferences are available to configure the Application Builder behavior.

Model Editions

Model Changes are developed, managed and deployed as Model Editions. This version control mechanism allows you to freeze versions (called Editions) of a model at design-time and deploy them for run-time in a Data Location.

A data location always runs a given model edition. This means that this data location contains data organized according to the model structure in the given edition, and that golden data in this data location is processed and certified according to the rules of the given edition.

Model editions are identified by a version number. This version number format is <branch>.<edition>. The branch and model numbers start at zero and are automatically incremented as you create new branches or editions.

Open and Closed Model Editions

A Model Edition is at a given point of time either Open or Closed for editing.
Branching allows you to maintain two or more parallel Branches (lines) of model editions.

Actions on Model Editions

Model Editions support the following actions:

These tasks are explained and details in the next chapters of this guide.

Typical Model Lifecycle

A typical model lifecycle is described below.

Month    : January       March      April   May        June     August
Branch 0 : [0.0] -***-> [0.1] -*-> [0.2] ----------*-> [0.3] ----------->
Branch 1 :                    +-branching-> [1.0] -**-**-****-> [1.1] --->

Considerations for Models Editions Management

The following points should be taken into account when managing the model editions lifecycle:

Initial Setup and Deployment in Development

The following steps are required when creating a new model in a development environment.

The model is deployed and ready for testing. Integration specialists can load data into the data location and use the generated data management applications to view and manage the data.

Making Changes in Development

After the first development round, designers will repeatedly make changes to the model edition and test them in the development environment.

To test these changes in the development data location:

The updated model is immediately ready for testing. After the update:

Releasing the Model

When the model is complete and tested, it is ready for release.

To release a model:

Developing Iteratively after a Release

When you close a model edition, a new model edition (for example, with version number [0.1]) is automatically created.

You can proceed with your next project iteration, starting with this new model edition:

If fixes are required on a previously released model edition, model managers can branch this old model edition, modify, then release it.

Creating a New Model

Creating a new model creates the model with the first model branch and model edition.

To create a new model:

Closing and Creating a New Model Edition

This operation closes the latest open model edition in a branch and opens a new one. The closed model edition is frozen and can no longer be edited, but can be deployed to production environments.

To close and create a new edition:

Branching a Model Edition

Branching a model edition enables restarting and modifying a closed edition of a model. Branching creates a new branch based on a given edition, and opens a first edition of this branch.

To create a new branch:

Translation Bundles

Strings translation is performed using Translation Bundles, attached to model editions. A translation bundle is a properties file that contains a list of key/value pairs corresponding to the strings localized in a given locale. The translation bundle file is named translations_<locale>.properties, where <locale> is the locale of the translation.

The following example is a sample of a translation bundle file for the English language (translations_en.properties). In this file, the label for the Employee entity is the string Staff Member, and its description is A person who works for our company.

...
Entity.Employee.Label=Staff Member
Entity.Employee.Description=A person who works for our company.
Attribute.Employee.FirstName.Label=First Name
Attribute.Employee.FirstName.Description=First name of the employee
Attribute.Employee.Picture.Label=<New Key TODO>
...

Translating a Model

To translate a model:

To export translation bundles:

To import translation bundles:

The translations for the model edition in the selected languages are updated with those contained in the translation bundles. If the Cleanup Removed Keys During Import was selected, translations in these languages no longer used in the model are removed.

Translation and Model Edition Lifecycles

The lifecycle of the translations is entirely decoupled from the model edition and deployment lifecycle:

Data Location Types

There are two types of data locations. The type is selected when the data location is created and cannot be changed afterwards:

The data location types are:

Data Location Contents

A Data Location contains the hub data, stored in the schema accessed using the data location’s datasource. This schema contains database tables and other objects generated from the model edition.

The data location also refers three type of jobs (stored in the repository):

Moving Models at Design-Time

At design-time, it is possible to move models from one repository to another design repository using Export/Import:

Deploying to a Remote Location

Frequently, the deployment environment is separated from the development environment. For example, the development/QA and production sites are located on different networks or locations. In such cases, it is necessary to use export and import to transfer the model edition before performing the deployment in production.

A remote deployment consists in moving a closed model edition:

Data Location Status

A data location status is:

When moving a data location to a Maintenance status, the currently processed batches continue until completion. Loads submitted after the data location is moved to Maintenance will fail. They can be kept open and submitted later, when the data location is restored to a ready status.

System Queues and Clusters

Specific queues and cluster exist for administrative jobs:

Job Priority

As a general rule, integration jobs are processed in their queues, and have the same priority.
There are two exceptions to this rule:

Queue Behavior on Error

When a job running in a queue encounters a run-time error, it behaves differently depending on the queue configuration:

Suspending a job on error is the preferred configuration under the following assumptions:

There may be some cases when this behavior can be changed:

Queue Status

A queue is in one the following statuses:

A cluster can be in one the following statuses:

Accessing the Execution Engine

To access the execution engine:

The Execution Engine Editor

This editor displays the list of queues grouped by clusters. Queue currently pending on suspended jobs appear in red.

The list of queues and clusters displays the following information:

From the Execution Engine editor, you can perform the following operations:

Stopping and Starting the Execution Engine

To stop and start the execution engine:

Changing the Queue Behavior on Error

To change the queue behavior on error:

Opening an Execution Console for a Queue

The execution console provides the details of the activity of a given queue. This information is useful to monitor the activity of jobs running in the queue, and to troubleshoot errors.

To open the execution console:

In the Console view toolbar you have access to the following operations:

Suspending a Job Running in a Queue

To restart a suspended job in a queue:

Restarting a Suspended Job in a Queue

To restart a suspended job in a queue:

Canceling a Suspended Job in a Queue

To cancel a suspended job in a queue:

Stopping and Starting the Integration Batch Poller

To stop and start the integration batch poller:

Configuring the Integration Batch Poller

The integration batch poller configuration determines the frequency at which submitted batches are picked up for processing.

To configure the integration batch poller:

In the Advanced section, set optionally the following logging parameters:

Accessing the Job Logs

To access the logs:

The Job Logs Editor

From this editor you can review the job execution logs and drill down into these logs.

The following actions are available from the Job Logs editor toolbar.

Drilling Down into the Logs

The Job Logs editor displays the log list. This view includes:

To drill down into the logs:

By drilling down into the task groups down to the task, it is possible to monitor the activity of a job, and review in the definition the executed code or plug-in.

Filtering the Logs

To create a job log filter:

Saved filters appear when you click the Apply and Manage User-Defined Filters button.
You can enable of disable a filter by marking it as active or inactive from this menu. You can also use the Apply All and Apply None to enable/disable all saved filters.

To manage job log filters:

Purging the Logs

You can purge selected job logs or all job logs returned by a filter.

To purge selected job logs:

To purge filtered job logs:

The jobs logs are purged.

Notifications Servers Types

Notifications recipients may be users or systems. The type of notification sent as well as the recipient depends on the type of notification server configured.

Each notification server uses a Notification Plug-in that:

Semarchy xDM is provided with several built-in notification plug-ins:

A single notification server having either the JavaMail or SMTP type can be used to send Workflow Notifications. This server is flagged as the Workflow Notification Server

Any servers can be used to send Job Notifications. Each Job Notification Policy specifies the notification server it uses.

Configuring Notification Servers

Notification servers are configured at platform-level. Refer to the Configuring Notification Servers section in the Semarchy xDM Administration Guide for more details about notification servers configuration.

Configuring a Job Notification Policy

With a notification server configured, it is possible to create notification policies using this server.

To create a notification policy:

Using Groovy for Notifications

The Groovy scripting language is used to customize the notification. See http://groovy-lang.org/documentation.html for more information about this language.

ErrorCount > 0

BatchStatus == 'DONE' && WorkflowName.find("Product") != null

EntityNames.contains("Customers")  || EntityNames.contains("Contacts")

File Name: NotificationFile_${BatchId}$.txt

Job ($JobName) is finished as: $BatchStatus.

Job ($JobName) is complete.
<% if (BatchStatus != 'DONE')  { %> Please reviews the completed batch : $BatchStatus. <% } %>

<p>Job ($JobName) is complete.</p>
<p>Entities:</p>
<ul>
<% EntityNames.each() { entityName-> %>
    <li>$entityName</li>
<% } %>
</ul>

Job Notification Properties

The following table lists the properties available for job notifications.

Introduction to Data Purge

The data location stores the lineage and history of the certified golden data, that is the data that led to the current state of the golden data.

Preserving the lineage and history is a master data governance requirement. It is key in a regulatory compliance focus. However, keeping this information may also create a large volume of data in the hub storage.

To make sure lineage and history are preserved according to the data governance and compliance requirements, model designers will want to define Data Retention Policy for the model.

When a model is deployed to a data location, a Purge Job is automatically created to handle data pruning according to the retention policy. The purge job prunes the lineage and history data according to the retention policy. Optionally, it prunes the job logs, batches, loads, direct authoring, duplicate manager and workflow instances when all their data is purged.

To keep a reasonable volume of information, data location managers have to schedule regular executions of this job.

Configuring a Purge Schedule

To create a purge schedule:

Cron fields

The fields are as follows:

Special Characters

The Special Characters available in cron expressions are listed below:

Examples