Table of Contents

Welcome to Semarchy xDM.
This guide contains information about administering and monitoring Semarchy xDM.

Preface

Overview

Using this guide, you will:

  • Understand the Semarchy xDM architecture and components.
  • Learn how to manage the various component of the architecture.
  • Learn how to manage run-time and troubleshoot errors.
  • Learn how to enable and manage a secure environment for Semarchy xDM.

Audience

This document is intended for administrators managing and configuring Semarchy xDM in a data management initiative.

To discover Semarchy xDM, you can watch our tutorials.
The Semarchy xDM Documentation Library, including the development, administration and installation guides is available online.

Document Conventions

This document uses the following formatting conventions:

ConventionMeaning

boldface

Boldface type indicates graphical user interface elements associated with an action, or a product specific term or concept.

italic

Italic type indicates special emphasis or placeholder variable that you need to provide.

monospace

Monospace type indicates code example, text or commands that you enter.

Other Semarchy Resources

In addition to the product manuals, Semarchy provides other resources available on its web site: http://www.semarchy.com.

Obtaining Help

There are many ways to access the Semarchy Technical Support. You can call or email our global Technical Support Center (support@semarchy.com). For more information, see http://www.semarchy.com.

Feedback

We welcome your comments and suggestions on the quality and usefulness of this documentation.
If you find any error or have any suggestion for improvement, please mail support@semarchy.com and indicate the title of the documentation along with the chapter, section, and page number, if available. Please let us know if you want a reply.

Introduction to Semarchy xDM

Semarchy xDM is the Intelligent Data Hub platform for Master Data Management (MDM), Reference Data Management (RDM), Application Data Management (ADM), Data Quality, and Data Governance.
It provides all the features for data quality, data validation, data matching, de-duplication, data authoring, workflows, and more.

Semarchy xDM brings extreme agility for defining and implementing data management applications and releasing them to production. The platform can be used as the target deployment point for all the data in the enterprise or in conjunction with existing data hubs to contribute to data transparency and quality.
Its powerful and intuitive environment covers all use cases for setting up a successful data governance strategy.

Architecture Overview

The Semarchy xDM architecture includes:

  • The Semarchy xDM Server, a Java EE application deployed and running in an application server. This application serves:
    • The Application Builder, Dashboard Builder, Setup and Configuration user interfaces: These web application are used by designers and administrators to create, manage and administer the models and applications designed in Semarchy xDM.
    • Data Management Applications and Dashboard Applications: These web applications are used by business users to browse and manage data and visualize metrics dashboards.
    • A Rest API to perform programmatically data integration, management and administrative operations.
  • The Repository that stores all the metadata used by Semarchy xDM. This includes the data models as well as the definition of the data management and dashboard applications. The repository is hosted in a database schema. A given Semarchy xDM server is always attached to a single repository.
  • The Data Locations that contain data for the data models. This data include the golden, master and source data, with all the lineage and history. A data location is hosted in a database schema. Multiple data locations can be attached to a single repository.

The following sections detail the component of this architecture.

Semarchy xDM Server

The Semarchy xDM server is a Java EE application deployed and running in a supported application server. It stores all its information in the repository. One application is always attached to a single repository, and connects this repository using a JDBC datasource named SEMARCHY_REPOSITORY. This datasource is configured in the application server.

The Semarchy xDM application is used at design-time to design:

  • Data models and applications, using the Application Builder,
  • Dashboard applications, using the Dashboard Builder.

At run-time, the application:

  • serves the Data Management Application,
  • serves the Dashboard Applications,
  • serves the REST API to manage data and platform operations,
  • runs the jobs that certify golden data.

Certification Process

Semarchy xDM creates or modifies golden data using data feeds from:

  • Data feeds from source applications (the Publishers). Such data is published programmatically using the SQL or REST APIs.
  • Data creation or changes performed by users in the Data Management applications.

The certification process is triggered as explained below:

  1. The Publishers or the Data Management application submit to the data hub a batch of source data. Such data is submitted as a Load that contains multiple datasets to process together.
  2. The Integration Batch Poller polls integration batches at regular intervals. When a new batch is detected, the batch poller requests the Execution Engine to start the Integration Job associated with this batch.
  3. The Integration Job is created from a Job Definition and executed by the Execution Engine. Data is passed through the different steps of the certification process, and golden data is certified from the source data.
The various steps of the certification process are detailed in the Certification Process Design chapter of the Semarchy xDM Developer’s Guide.

Repository

The repository contains the design-time and run-time information for a given Semarchy xDM Server instance.

Repository Contents

The repository stores the following information:

  • For Data Management Models
    • The entities, attributes, etc.
    • The model versions: branches, editions, etc.
    • The platform configuration & security information: roles, privileges, notification servers, preferences, etc.
    • The data locations information: deployed model, job definitions, notifications, etc.
    • Run-time information: Engine queues, logs, etc.
  • For Dashboard Applications
    • The dashboard application metadata.

A repository is stored in a database schema accessed from the application using a JDBC datasource named SEMARCHY_REPOSITORY.

The repository should never be accessed directly via SQL queries. Access to the Semarchy xDM information must be performed through the Semarchy user interfaces.

Repository Types

There are two types of repositories:

  • Design: All design-time and run-time operations are possible in this type of repository.
  • Deployment: With this repository type, you can only import closed model editions and cannot edit them.

The deployment repositories are suitable for production sites. Model transfer from design to deployment repositories is handled via incremental export/import of closed model editions. Refer to the Planning the Installation chapter in the Semarchy xDM Installation Guide for examples of repository deployment patterns.

The repository type is selected at creation time and cannot be modified afterwards.
Both type of repositories can be used indifferently for Semarchy xDM Dashboards applications.

Data Locations

When a data management hub must be available for run-time (for testing or production purposes), it is generated from a data model defined in the repository, and deployed in a data location. Data Locations contain the deployed data hubs. Each hub contains the golden data, the source data pushed by publishers in the hub, and the various stages of this data in the certification process.

The data location is hosted in a database schema and accessed via a JDBC datasource defined in the application server. A data location refers to the datasource via its JNDI URL.

Data locations are only used for Semarchy models designed in the Application Builder. Dashboard Applications created in Semarchy xDM Dashboards do not require a data location.

Data Locations, Repositories and Models

A data location is attached to a repository: You can declare as many data locations as you want in a repository, but a data location is always attached to a single repository. It is not possible to have a data location attached to two repositories at the same time.

You may deploy several model editions successively in a data location, but only one model edition is deployed and is active in the data location at a certain point in time.

Data Location Contents

A Data Location contains the hub data, stored in the schema accessed using the data location’s datasource. This schema contains database tables and other objects generated from the model edition.

The data location also refers three type of jobs (stored in the repository):

  • Installation Jobs: The jobs for creating or modifying in a non-destructive way the data structures in the schema.
  • Integration Jobs: The jobs for certifying data in these data structures, according to the model job definitions.
  • Purge Jobs: The jobs for purging the logs, data lineage and history according to the retention policies.

Data Location Types

There are two types of data locations:

  • Development Data Locations: A data location of this type supports deploying open or closed model editions. This type of data location is suitable for testing models in development and quality assurance environments.
  • Production Data Location: A data location of this type supports deploying only closed model editions. This type of data location is suitable for deploying MDM hubs in production environments.
The type is selected when the data location is created and cannot be changed afterwards.

Refer to the Planning the Installation chapter in the Semarchy xDM Installation Guide for examples of data location deployment patterns.

Data Structures and Certification Process

A deployed model edition is made of a Data Structure and a set of Integration Jobs.

  • The Data Structure of the MDM Hub is a set of tables stored in the database schema of the data location. This structure contains the landing tables for the loads pushed in the hub by the publishers, the golden records tables and the tables handled by the certification process to create golden records from the source records.
  • The Integration Jobs are sequences of tasks stored in the repository, which perform the certification for data entering the hub. They are specific the deployed model edition. When a model edition is deployed in a data location, the integration job definition for this model edition replace previous job definitions.
Data Structure and Integration Jobs work together and are deployed simultaneously by the model edition deployment process.
Data Structure Details

The data structure is implemented to support all the successive model editions deployed in the data location. The data structure is created when the first model edition is deployed, and is altered for new model editions. Changes to the structure are only incremental and non-destructive.

For example, the GD_CUSTOMER table holds the data for all successive design states of the Customer entity. If a new FaxNumber attribute is added to the entity and deployed with model edition 1.1, the FAX_NUMBER column is created to contain the fax number data. If this attribute is removed in subsequent model editions (1.2 and so forth), the column for this attribute remains in the data structures, when these model editions are deployed.

Platform Components

The Semarchy xDM platform contains several components described in the sections below.

Integration Batch Poller

The Integration Batch Poller polls the integration batches submitted to the platform on a defined schedule, and starts the Integration Jobs on the Execution Engine.

Execution Engine

The Execution Engine processes the Integration Jobs submitted by the Integration Batch Poller. It orchestrates the certification process to generate the golden data. This engine sequences the jobs in Clusters and Queues.

The execution engine logs the activity of the platform and manages notification policies.

Security

The application uses role-based security for accessing Semarchy xDM features. The users and roles used to connect to the application must be defined in the security realm as part of the application server configuration and then declared in Semarchy xDM.

Role base security is used in Semarchy xDM to define the access privileges to the features of the platform (Platform-Level Security), as well as the privileges to access and modify data in the data locations (Model-Level Security).

Introduction to Semarchy Configuration

Platform-level Administrative tasks are performed in a dedicated user interface called Semarchy Configuration. This section explains how to work with it.

Connecting

To access Semarchy Configuration, you need an URL, a user name and password that have been provided by your Semarchy xDM administrator.

To log in to Semarchy Configuration:

  1. Open your web browser and connect to the URL provided to you by your administrator. For example http://<host>:<port>/semarchy/ where <host> and <port> represent the name and port of host running the Semarchy xDM application. The Login Form is displayed.
  2. Enter your user name and password.
  3. Click Log In. The Semarchy xDM Welcome page opens.
  4. Click the Configuration button. Semarchy xDM Configuration opens.

To log out of Semarchy Configuration:

  1. In the Semarchy Configuration User Menu, select Log Out.

Semarchy Configuration

Semarchy Configuration allows to view and administer the following components:

  • Applications Configuration: Global parameters for all applications.
  • Image Libraries: Manage image libraries and images in these libraries.
  • Logging Configuration: Configure the platform logging (trace) for debugging purposes.
  • Notification Servers: Add, remove and configure servers used to send job notifications and application emails.
  • Plug-ins: View, add or update user-created plug-ins.
  • Roles: Declare in Semarchy xDM the application server roles, and grant them with platform-level privileges.
  • Variable Value Providers: Configure the system queried by Semarchy xDM to retrieve values for model variables.

Managing the Repository

The repository contains the design-time and run-time information for the Semarchy xDM server.

Understanding Repositories

The repository is created when Semarchy xDM is installed. The type of the repository (Design Repository or Deployment Repository) is set also at creation time, and the application always connects to a single repository.

The repository creation process is detailed in the Semarchy xDM Installation Guide.

The type of a repository defines the capabilities of this repository:

  • A Design Repository allows you to perform all design-time and run-time operations.
  • A Deployment Repository only allows run-time operations. You can import closed model editions in such repository but cannot edit them.
Typical Patterns for repository deployment are detailed in the Planning the Installation chapter of the Semarchy xDM Installation Guide. Simple and advanced deployment tasks are explained in the Deployment chapter of the Semarchy xDM Developer’s Guide.

Repository Administration Tasks

Viewing the Repository and System Information

Semarchy xDM exposes the repository and system details in the About dialog.

To view the repository and system details:

  1. In the Semarchy Configuration User menu, select About.
  2. In the About dialog:
    • The License Information link displays the current license information and allow for Updating the License Key.
    • The Repository Information link displays the repository details (including name and version).
    • The System Information link displays the platform system details and may be used for support purposes.

Updating the License Key

Semarchy xDM stores in the repository the license information and the license key provided to you for evaluation or when you purchased the product. You can update an expired license key with a new one using the following procedure.

You must be logged with a user having the semarchyAdmin role or Platform Administration to perform license key update tasks. Without this privilege, you are only able to view the license key.

To update the license key:

  1. In the Semarchy Configuration User menu, select About.
  2. In the About dialog select the License Information link.
  3. In the License Key Information dialog, click the Upload License Key File… button.
  4. Use the Browse button to select the license key file.
  5. If the selected license key recognized as a valid one, you can click the OK button to register the license key in the repository.
A temporary license key must be updated when it expires. When such license key expires, the repository content is preserved as is, but the application is no longer accessible and a popup window will prompt you for a new license key when you log in.

Configuring the Platform

The platform contains of several components that can be managed from Semarchy Configuration.

Configuring Notifications Servers

Notifications tell users or applications when a job completes or when operations are performed into workflows, for example, when task is assigned to a role.

There are two types of notifications:

  • Job Notifications issued under certain conditions when an integration job completes. These notifications are used for administration, monitoring, or integration automation. These notifications are configured with Notification Policies in the data locations.
    For more information about notification policies, see the Semarchy xDM Application Management Guide.
  • Workflow Notifications are emails sent to users when operations are performed in a workflow. They are configured in workflow transitions and tasks.
    For more information about workflow notifications, see the Semarchy xDM Developer’s Guide.

Both families of Notifications are issued via Notification Servers.

Notifications Servers Types

Notifications recipients may be users or systems. The type of notification sent as well as the recipient depends on the type of notification server configured.

Each notification server uses a Notification Plug-in that:

  • defines the configuration parameters for the notification server,
  • defines the configuration and form of the notification,
  • sends the notifications via the notification servers.

Semarchy xDM is provided with several built-in notification plug-ins:

  • JavaMail: The notification is sent in the form of an email via a Mail Session server configured in the application server, and referenced in the notification server. For more information about configuring Mail Session, see the Semarchy xDM Installation Guide.
  • SMTP: The notification is sent in the form of an email via a SMTP server entirely configured in the notification server.
  • File: The notification is issued as text in a file stored in a local directory or in a FTP/SFTP file server.
  • HTTP: The notification is issued as a GET or POST request sent to a remote HTTP server. Use this server type to call a web service with the notification information.
  • JMS: The notification is issued as a JMS message in a message queue.
It is possible to develop additional plug-ins to issue other type of notifications. See the Semarchy xDM Plug-in Development Guide for more information about plug-in development.

A single notification server having either the JavaMail or SMTP type can be used to send Workflow Notifications. This server is flagged as the Workflow Notification Server

Any servers can be used to send Job Notifications. Each Job Notification Policy specifies the notification server it uses.

Creating a Notification Server

To create a notification server:

  1. In Semarchy Configuration, select the Overview editor, and then click the Notification Servers link. The Notification Servers editor opens.
  2. Select the Notification Servers list, right click and select image New Notification Server. The Create New Notification Server wizard opens.
  3. Enter the following workflow parameters:
    • Name: Internal name of the notification server.
    • Label: User-friendly label for the server.
    • Plug-in ID: Select one of the available notification server plug-in.
    • Workflow Notification Server: Select this option to use this notification server by default in the workflows. This options can be selected only if the Plug-in ID is JavaMail or SMTP.
  4. Click Next.
  5. In the second wizard page, enter the configuration information for your type of server:
    • JavaMail:
      • JNDI URL: JNDI URL of the Java Mail Session service available in the application server. This URL is typically java:comp/env/mail/Session if the Mail Session service is declared as mail/Session in the application server.
      • From User: Email address of the sender of the notifications from this server. This address is also used in the reply-to address for notification emails.
      • Password If this server requires specific authentication, enter a password for this server.
    • SMTP:
      • SMTP Host Name and SMTP Port: Name or address, and port of the SMTP host.
      • From User: Email address of the sender of the notifications from this server. This address is also used in the reply-to address for notification emails.
      • Authentication Required: If this server requires specific authentication, select the Authentication Required option and enter a User Name and Password for this server, and indicate whether it uses TLS or SSL.
      • Additional SMTP Properties Enter additional properties as property=value pairs.
    • File:
      • File System: Select the file system of the file server. FILE for a local server, FTP or SFTP for a remote server.
      • Host, Port, Login, Password are required to connect an FTP or SFTP server.
      • Root Path: Provide the root path for storing the notification file. For example c:\work\notifications for Windows or /work/notifications for UNIX/Linux
    • HTTP:
      • Scheme: Specify whether the HTTP request should be done using HTTP or HTTPS
      • Host, Port and optionally Login, Password are used to connect the HTTP server.
      • Base Path: Root path appended added after the host and port in the URL. For example: / or /rest/api/.
      • Use System Properties: Check this option to use the system-defined properties to configure the HTTP connection. This option allows using a proxy configuration defined in the Java parameters for the application server.
      • Proxy Host, Proxy Port, Proxy Login and Proxy Password are used to configure the connection through an HTTP proxy.
      • Headers: Enter additional HTTP headers as property=value pairs.
    • JMS:
      • Connection Factory URL: JNDI URL of the factory used to create a connection to the JMS destination. The URL is typically java:comp/env/jms/ConnectionFactory if the connection factory is declared as jms/ConnectionFactory in the application server.
      • Login and Password used when initiating the JMS connection.
  6. Press CTRL-S to save the configuration.

Testing a Notification Server

After configuring the notification server, it is recommended to run a test email on this server.

To test a notification server:

  1. In the Notification Servers editor, select the notification server you want to test, right-click and select Test Configuration.
  2. The next steps depend on the type of notification servers:
    • File, HTTP, JMS: No further operation is needed. A connection attempt is made on the notification server.
    • JavaMail and SMTP: Provide a comma-separated list of email addresses and then click OK. An email is sent via the notification server to these recipients.

Configuring Variable Value Providers

Semarchy xDM uses variables defined in models to enforce certain data governance policies for a user’s session.
For more information about model variables, see the Semarchy xDM Developer’s Guide.

A Variable Value Provider is a system that can be queried by Semarchy xDM to retrieve the values for these variables. Typically, this system is a server containing information about the user connected to Semarchy xDM.

Two type of variable value providers are supported out-of-the-box:

  • Datasource Variable Provider: This variable value provider is a relational database that is accessed through a JDBC datasource. Semarchy xDM can issue SQL statements against this database to retrieve variable values. For example, an employee database that can be queried to retrieve the country of the connected user.
  • LDAP Variable Provider: This variable value provider is a directory server that is accessed using the LDAP protocol. Semarchy xDM can issue queries against this directory server to retrieve variable values. For example, an LDAP directory that can be used to retrieve the organizational unit of the connected user.

Variable value providers are configured in the repository, and can be used by any model in this repository.

When working with a deployment repository, make sure to configure the variable value providers used in the models before importing or deploying them in this repository.

Creating a Variable Value Provider

To create a variable value provider:

  1. In Semarchy Configuration, select the Overview editor, and then click the Variable Value Providers link. The Variable Value Providers editor opens.
  2. Select the Variable Value Providers list, right-click and then select image New Variable Value Provider. The Install Variable Value Provider wizard opens.
  3. Enter the following information:
    • Name: Internal name of the variable value provider.
    • Label: User-friendly label for the variable value provider. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.
  4. Select the Plug-in ID corresponding to the variable value provider type: LDAP Variable Provider or Datasource Variable Provider.
  5. Click Next.
  6. Click the image Edit Expression button.
  7. In the the Variable Value Configuration dialog, enter the configuration information.
    This information differs depending on the selected Plug-In.
    • For a Datasource Variable Value Provider, enter the following information:
      • JNDI Data Source Name: Select a JDBC datasource in the list.
        This datasource must be defined in the application server. It is used to connect the database acting as the variable value provider.
    • For an LDAP Variable Value Provider, enter the following information:
      • LDAP Host: Name or IP address of the LDAP server.
      • LDAP Port: Listening port of the LDAP server. The port is typically 389 for non-SSL connections, and 636 for SSL connections.
      • Use SSL: Check this options to use SSL to connect to the LDAP server.
      • User: Name of the user used to retrieve data from the LDAP Server. Note that this user should have read privileges to the LDAP structure.
      • Password: This user’s password.
  8. Click OK to close the Variable Value Configuration dialog.
  9. Click Finish.

The variable value provider is added to the list.

Testing the Variable Value Provider Configuration

After configuring a new variable value provider, it is recommended to test its configuration.

to test a variable value provider configuration:

  1. In the Variable Value Providers editor, select of the variable value provider in the list.
  2. Right-click and select Test Configuration.

A message indicates whether the connection test was successful or not.

The configuration test only tests the connection information, but does not check the privileges granted to the user to retrieve the values from the provider.

Managing Image Libraries

Image libraries store images and icons used by applications. Libraries are stored in the repository, are global to the platform and can be used by all applications. Designers use, in applications, images and icons from the libraries by reference, using image library URLs.

Default Image Libraries

When the repository is created, built-in image libraries are seeded with default Semarchy xDM images and icons:

  • mdi is the library for all the Material Design icons and assets. This library cannot be modified.
  • default contains Semarchy branding images as well as the default images used by applications, such as the default images for the actions. Icons provided in addition to the mdi library are in this library. This library cannot be modified.
  • demo<application_name>_ contains the images/icons used in the <application_name> demonstration application. Such a library is seeded when a demo application is installed.
The default image libraries content changes with new versions of Semarchy xDM. When an image is removed or changed in the default libraries, it is indicated as deprecated in the validation report for models using it. Model designers should update their models to use newer versions of such images.

Browsing Image Libraries

The Image Libraries editor, available in Semarchy Configuration, allows administrators to view and manage the image libraries.

To browse the image libraries:

  • Open Semarchy Configuration.
  • In the Overview, click the Image Libraries link.
    The Image Libraries editor opens.

Use the filter box in the image libraries editor to reduce the number of images displayed. You can use the '%' sign to replace any number of character.

Managing Image Libraries

Image libraries are created and modified using file import. You can import two types of files:

  • Image files, for which you must select a target library.
  • Zip files, into which images are organized into folders. When a Zip file is imported, one library is created for each folder at the root of the Zip file, and images in this folder are imported into the library. Exporting images creates a file that follows the same format.

To import images:

  • Open the image libraries editor.
  • Click the Import Image Library button. The import dialog opens.
  • Drop either a Zip file or multiple image files into the dialog, or click Browse to select a Zip file or multiple image files.
  • If you import image files, select the Target Library for these images. Select Create a New Library to create a new library for these images.
  • Click OK to import the images.

To delete images:

  • Open the image libraries editor.
  • Click an image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.
  • Click the Delete Selected Images button, and then confirm the deletion.
If you delete all the images from a library, the library disappears.

To export images:

  • Open the image libraries editor.
  • Click the image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.
  • Click the Export Selected Images button.

A zip file is downloaded, containing all the selected images stored into folders named after their parent library.

Using this export mechanism, you can move an image library to a remote repository, or reorganize the library before reimporting it.
If an image is removed from a library while being used in a model, it will raise a warning in the model validation report. Designers should fix such stale image references and update their models to point to valid images.

Managing Plug-ins

Semarchy xDM allows extending its capabilities using Java code and external APIs. Using the Open Plug-in Architecture, existing services or information systems can contribute to the master data processing and enrichment. You can extend the Enrichment and Validation capabilities in Semarchy xDM through user-defined plug-ins.

For detailed information about plug-in development and packaging, see the Semarchy xDM Plug-in Development Guide.

A Plug-in is delivered as a jar file bundle that must be deployed in each Semarchy xDM server instance running integration jobs that use the plug-in. You do not need to restart the server to take new or updated bundles into account.

These bundles are tagged with a version number. As a consequence, updating an existing plug-in with a newer version of this plug-in will automatically make the platform work with the newer plug-in version. The deployment process installs a new plug-in or replaces an existing plug-in version with a new one.

To deploy a plug-in:

  1. In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.
  2. Click the image Install or Update Plug-in button in the upper right corner of the Plug-ins editor. The Install/Update Plug-ins dialog opens.
  3. Click the Browse button and select the plug-in binary file. For example: com.acme.phoneStandardizer_1.0.0.jar.
  4. Click OK. A Status window shows the number of plug-ins installed or updated.
  5. Your session is closed to take this new plug-in into account. Restart the session.
  6. In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.

The plug-in now appears in the list, and can be used in the models and the integration jobs.

Make sure to install the plug-ins required by the jobs of a model before deploying the model. If a job starts before its required plug-ins are installed, then it will fail. The plug-in can be installed and the job resumed after the installation.

To uninstall a plug-in:

  1. In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.
  2. Select the plug-in in the list.
  3. Click the image Uninstall Selected Plug-ins button in the editor’s toolbar.

Managing Applications Configuration

The global behavior of the data management applications is configured from Semarchy Configuration.

Changes performed in this configuration apply to all the applications started in the instance of Semarchy xDM.
The semarchyAdmin role is required to configure the global application parameters.

To set the Applications Configuration:

  1. In Semarchy Configuration, select the Overview editor, and then click the Applications Configuration link. the Applications Configuration editor opens.

This editor displays the global application parameters:

  • Configure the Server Base URL of your Semarchy xDM installation. This URL is seeded when creating the repository and used in workflow notifications to provide links to the server.
  • Set the CSV Export Limit and Excel Export Limit to define the maximum export size allowed in each format. Note that generating export files is resource consuming on the server. It is recommended to test the scalability of the new export limits.
  • Set the Autocomplete Max Results value to define the maximum number of results that should be fetched by the autocomplete components in the applications. These dynamic drop drown lists fetch records as you type a search strings.
  • Change the Header Logo that appears for all applications and in the welcome page by uploading a picture. Note that this picture is scaled to a height of 96 pixels.
    You can also change the Favicon used for this page.

Managing Custom Translations

Semarchy xDM allows translating the strings you define in applications (for example, the labels of the business views or fields) as part of the model and application localization effort.

In addition, strings built in Semarchy, visible in all data management and dashboard applications, such as 'Inbox', 'Filter' or 'Global Search', can also be modified in the platform configuration with Custom Translations.

With custom translations, you can:

  • Override existing built-in strings. For example, replace 'Inbox' by 'My Tasks' in the Inbox editor title.
  • Create translations for new languages for the built-in strings.
Custom translations are configured at platform-level and apply to all the applications served by the platform.
For more information about model translation, see the Model Localization section in the Semarchy xDM Application Management Guide.

The translation process consists of:

  1. Exporting existing languages' localization file, or blank localization files for new languages. These files are properties files, which can be modified using a text editor.
  2. Editing these files using a text editor.
  3. Import the modified files.

Exporting Custom Translations

To export custom translations:

  1. In Semarchy Configuration, select the Overview editor and then click on the image Custom Translations link.
    The Custom Translations editor opens and lists the custom translations that you have configured.
  2. If you want to modify existing custom translations, select these translations in the list.
  3. Click on the image Export Translations…​ button in the editor toolbar.
  4. In the Export Translation Bundles dialog, select the translation bundles that you want to export. The Hide Languages with no Custom Translations option masks the languages for which you have no custom translation.
  5. Click OK
    The selected translation bundles are exported in a zip file.

Custom Translations Export File Format

The translations are exported as a compressed file that contains properties files (containing key=value pairs).

These files are organized into folders:

  • The /commons folder contains property files for all the languages with strings common to the dashboard and data management applications.
  • The /dashboards folder contains property files for all the languages with strings specific to the dashboards component.
  • The /applications folder contains the property files for all the languages with strings specific to the data management applications.
After editing the custom translation properties files, make sure to compress these files in a zip file that follows the exact same structure, as it is expected by the import process.
Languages and Variants

The property files have a suffix corresponding to the language code and an optional variant code.

Example 1. Example of language and variant

en is the suffix for the English language, and en_US is the suffix for United States variant of this language. When a key needs to be translated:

  • Users having their browser configured in the English (United States) variant will use the en_US translation if it is available, or else use the English en translation.
  • Users having their browser in English will directly use the English en translation.

In addition, specific properties files with the en_default and en_default suffixes represent the default translations built-in Semarchy xDM for English and French. These are here for informational purposes and are ignored during the import process.

Language translation fallback

You do not need to define all translations in the custom translations since the string localization process has a fallback process.

In this process, translations are searched in the following order:

  1. In the language variant custom translation (for example fr_FR for the French (France) language variant)
  2. In the language custom translation (for example fr for the French language)
  3. In the language built-in translation (for example fr_default), if any exist for the user language.
  4. In the custom English translation (en).
  5. In the built-in English translation (en_default).

Semarchy xDM includes built-in translations for English and French.
Using the fallback mechanism, you can create custom translations for these two languages to override the built-in translations.

The translations that you provide for these base languages automatically override, in the fallback mechanism, the built-in ones.

To override a single string in English, you need to provide a custom translation for the English language that contains only this string.

Importing Custom Translations

To import custom translations:

  1. In Semarchy Configuration, select the Overview editor and then click on the image Custom Translations link.
    The Custom Translations editor opens.
  2. Click on the image Import Translations…​ button in the editor toolbar.
  3. Click the Add button and select a zip file containing custom translations.
    The editor shows in the Languages to import the custom translations available in the file.
  4. Select the property files that you want to import, and then click Finish
    The selected translation bundles are imported.

When importing a custom translation for a language that is not translated yet, a new translation is created. If a custom translation already exists for a given language, it is removed and replaced entirely by the new one.

Configuring the Logging

Logging is used to troubleshoot the Semarchy xDM platform issues and monitor the platform activity.

The default logging configuration set up in Semarchy works in most cases. Most of the times, you do not need to change this configuration.
You can reset the logging configuration to the default values by clicking the Restore Logging Configuration to Default button in the Logging Configuration editor toolbar.

By default, the important log events appear in the Semarchy Configuration Error view. To access this error view, in the Semarchy Configuration user menu, choose the Window > Show View…​ > Others and then select General > Error Log.

Semarchy xDM uses Apache Log4J to log its activity. This section provide key concepts and configuration samples to work with Log4J. Always refer to the Log4J Manual for reference information.

Log4J works with three key concepts: Appenders, Loggers and Log Levels.

  • The Appenders deliver log events to a destination. For example they write events to a file on the application server file system, to a message queue or to the Semarchy Configuration Errors view.
  • The Loggers capture log events emitted by software certain components in Semarchy xDM.
  • The Log Levels define at the same time the granularity and importance of log events.

Each log events is emitted with a certain log level, is captured by a logger that sends it to an appender for delivery.

Setting the Logging Configuration

Appenders, loggers and log levels are configured through the Logging Configuration editor.

To set the Logging Configuration:

  1. In Semarchy Configuration, select the Overview editor, and then click the Logging Configuration link. The Logging Configuration editor opens.
  2. Change the configuration in the editor.
  3. Press CTRL-S to save this editor.
    The new logging configuration is immediately taken into account.

Understanding Log Levels

The Log Level define the granularity and importance of a log event. The log level is one of the following: TRACE, DEBUG, INFO, WARN, ERROR, FATAL.

These values are ordered from highest to lowest granularity, but also in order of importance: The DEBUG level is very granular and most messages are not relevant in the normal course of operations, whereas the ERROR and FATAL messages will appear less frequently but are more important.

When you capture events, via a logger, at a certain level, all events of higher importance are also captured. For example, when setting a logger to INFO, then the WARN, ERROR, and FATAL events are also captured.

When parameterizing a logger, it is possible to set the level to ALL to capture all log events, and to OFF to disable log capture.
INFO is typically a good level to start with, as it captures information messages as well as possible warnings and errors.

Defining Appenders

Appenders deliver log events to a destination. Appenders are defined with a name and an appender class (that is, a Java class the writes log events to a file, to a JMS message queue, to an email, etc).
Appender classes may have one or more properties to configure their behavior. They also support Layout properties to define the format of the logged event.

In the logging configuration, the appender is defined with the following syntax:

log4J.appender.<appender_name> = <appender_class>

Each property for this appender is defined with the following syntax:

log4J.appender.<appender_name>.<property_name> = <property_value>
Sample Appenders Configuration

Sample appender configurations are provided in this section.

Configuring an appender called FILE to write to a daily rolling log file.
# Appender Class
log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender 	(1)

# Appender Properties
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd'.log'		(2)
log4j.appender.FILE.file=${user.home}/.semarchy/logfile.log 	(2)

log4j.appender.FILE.append=true 				(3)
log4j.appender.FILE.encoding=UTF-8 				(3)

# Appender Layout Configuration
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout 	(4)
log4j.appender.FILE.layout.ConversionPattern=%d{yyyy-MM-dd HH\:mm s S}-%t-%x-%-5p-%-10c\:%m%n (4)
1The DailyRollingFileAppender appender class creates a daily logging file. You can also use the FileAppender to write to a simple file.
2The daily log file is located in the .semarchy/ sub-directory of the user home directory. Its name is logfile.log to which the date, formatted with the pattern, is concatenated.
3These properties configure the appender behavior: Add log events at the end of the daily file, and use UTF-8 encoding.
4Logged event are formatted using the PatternLayout formatting class and a conversion pattern.
This pattern renders log events in the following format:
2010-03-23-main—​DEBUG-log4jExample:Here is a debug message
Configuring an appender called PDE to log to the Semarchy Configuration Error view
# Appender writing to the {mdm-short-product-name} Configuration Error view
log4j.appender.PDE=com.semarchy.commons.log4j.appender.pde.PDEAppender 	(1)
log4j.appender.PDE.Threshold=INFO 					(2)
1This appender class logs events into the Semarchy Configuration Error view.
2Only log events more important that this threshold will be considered by the appender. Make sure to align this value with the loggers that send events to this appender.
Configuring an appender called SMTP to send log events by email
# Appender sending emails via an SMTP host (usually for important error messages)
log4j.appender.SMTP = org.apache.log4j.net.SMTPAppender 		(1)
log4j.appender.SMTP.SMTPHost = <smtp_host_name> 			(2)
log4j.appender.SMTP.SMTPPort = <smtp_host_port> 			(2)
log4j.appender.SMTP.From = email@my-domain.com  			(2)
log4j.appender.SMTP.To = admin1@my-domain.com,admin2@my-domain.com 	(3)
log4j.appender.SMTP.Subject = Log Event 				(3)
log4j.appender.SMTP.layout =  org.apache.log4j.TTCCLayout 		(4)
1This appender class sends log events by email.
2Mail Server configuration. Note that additional properties may be set for the SMTP user, password, protocol, etc.
3Email recipients and subject.
4Log events are formatted using the TTCC Layout, a detailed format containing the time, thread, category and nested diagnostic context.

Defining Loggers

Loggers are named and organized as a hierarchy of classes emitting log events.
For example, com.semarchy is the parent of com.semarchy.mdm. The highest logger in the hierarchy is the rootLogger.

You configure loggers with a log level, and attach them to one or more appenders. Any event emitted by the class that is more important than (or as important as) the given log level is captured by the logger and sent to the appender.

Use the following syntax to attach a logger to appenders with a given log level:

log4j.logger.<loggerName> = <level>, <appender_name> [,<appender_name>, …]

For example, to send DEBUG, INFO, WARN, ERROR and FATAL events emitted by the com.semarchy.platform.setup.IPlatformManager class to the appender named PDE:

log4j.logger.com.semarchy.platform.setup.IPlatformManager=DEBUG, PDE

Logger inherit their appenders additively from their parents in the hierarchy. They can override the level provided by their parent.

In the following configuration, the com.semarchy.platform.setup.IPlatformManager logger inherits from the PDE appender of the com.semarchy logger, and sends ERROR events to the PDE appender in addition to the SMTP appender.

log4j.logger.com.semarchy=INFO, PDE
log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP

To prevent such inheritance, use the following syntax:

log4j.additivity.<logger_name> = false

For example, in the the following configuration, com.semarchy.platform.setup.IPlatformManager will no longer log into the PDE appender in addition to SMTP.

log4j.logger.com.semarchy=INFO, PDE
log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP
log4j.additivity.com.semarchy.platform.setup.IPlatformManager=false

The following configuration is the default one for Semarchy xDM.

Sample logger configuration for Semarchy xDM.
log4j.logger.com.semarchy=INFO
log4j.logger.com.semarchy.commons.sql=INFO
log4j.logger.com.semarchy.mdm.datahub.services.query.datamgr.IDataManager=INFO
log4j.logger.com.semarchy.platform.engine=INFO
log4j.logger.com.semarchy.platform.engine.PluginExecution=INFO
log4j.logger.com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor=INFO
log4j.logger.com.semarchy.platform.product.notification.JobNotificationHandler=ERROR
log4j.logger.com.semarchy.platform.setup.IPlatformManager=INFO
log4j.logger.org=INFO
log4j.logger.org.apache.aries.blueprint=WARN
log4j.logger.org.apache.commons.beanutils.converters=WARN
log4j.logger.org.apache.cxf=WARN
log4j.logger.org.apache.cxf.services = WARN
log4j.logger.org.apache.directory.shared.asn1.ber.Asn1Decoder=ERROR
log4j.logger.org.ops4j.pax.logging=WARN
log4j.logger.org.quartz=WARN
log4j.logger.org.springframework.jdbc.core.JdbcTemplate=INFO
log4j.rootLogger=INFO, PDE

The following table lists the various loggers available in Semarchy xDM:

Logger NameDescription

com.semarchy

Root logger for the Semarchy platform.

com.semarchy.commons.sql

TRACE logs SQL queries made by the Semarchy Application Builder and Configuration user interfaces.

com.semarchy.mdm.datahub.services.query.datamgr.IDataManager

Logs database operations performed by the platform for the MDM applications while browsing data. DEBUG logs all queries and execution times.

com.semarchy.platform.engine.core.impl.DefaultStandaloneEngineImpl

Logs execution engine events. DEBUG logs all submitted jobs, and ERROR can be used to troubleshoot engine issues.

com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor

Logs execution engine job processing. Use DEBUG to see execution engine details.

com.semarchy.platform.engine.PluginExecution

Logs enricher and validation plugins execution. DEBUG logs plugins feedback, TRACE logs the processing of every row.

com.semarchy.platform.integration.polling.IntegrationLoadDequeuer

Logs the Batch Poller activity. DEBUG logs every polled interval.

com.semarchy.platform.product.notification.JobNotificationHandler

Logs job notifications. ERROR traces notification failures.

com.semarchy.platform.setup.IPlatformManager

Logs platform status changes. INFO traces normal status changes. ERROR traces abnormal statuses.

org.apache.cxf.services

INFO traces REST requests, responses, and faults between the MDM applications and the server.

 org.springframework.jdbc.core.JdbcTemplate

DEBUG traces the SQL queries made by the MDM applications and REST API calls as well as Dashboards-related SQL queries.

Managing the Security

The application uses role-based security and privilege grants for accessing the Semarchy xDM features as well as the data contained in the data location.

Understanding the Security Model

Platform-Level and Model-Level Security

There are two levels of security in Semarchy xDM:

  • Platform-Level Security defines access to the features of the platform. For example, access to the administrative features, or access to the design-time capabilities. Platform-level security sets platform users’ privileges (who can design models, monitor executions, manage security, etc.).
  • Model-Level Security defines security privileges to access and modify data in the data locations. Defining these privileges is a data governance decision and should be defined as part of the data governance initiative. Defining Model Security is covered in the Securing Data chapter of the Semarchy xDM Developer’s Guide.

Role-Based Security

Both levels of security are role-based:

  • The Privileges (platform level/model level) are granted to Roles in Semarchy xDM.
  • These Roles are declared in Semarchy xDM. The roles declared in Semarchy xDM must map roles that pre-exist in the application server. These application server roles are created and granted to application server users as part of the application server configuration
  • Users logging in to Semarchy xDM use their application server credentials.

Depending on the application server hosting the Semarchy xDM server, the roles/user association may be made through a concept of group: A user belongs to a group and the role is granted to the group.
Depending on its configuration, the application server may delegate user authentication and management in general to a security provider (SSO, LDAP, etc…).

Note that roles are not only used for security purposes. They are also used as email aliases for email notifications.

Security Context

When you log in to Semarchy:

  1. You enter the user and password in the Semarchy xDM login window.
  2. This information is passed to the application server.
  3. The application server itself or its security provider (SSO, LDAP, etc.) authenticates the user, gets the list of roles associated to this user (possibly via groups) and returns this list of roles in the session’s Security Context.
  4. Semarchy xDM starts a session with this security context, allowing:
    • Certain platform features depending on the Platform-Level Privileges granted to the roles.
    • Certain data access/modification capabilities depending on the Model-Level Privileges granted to the roles.
Role Names are Case-Sensitive
The role names defined in Semarchy must exactly match the role names returned by the security provider.
For example, for a Tomcat installation, you define in the tomcat-user.xml file a role named FINANCEADMIN, and in Semarchy, you declare a FinanceAdmin role. These two roles will not match. Tomcat users with the FINANCEADMIN role will not be granted the privileges defined in Semarchy for the FinanceAdmin role.
Semarchy xDM enforces security at several layers in the application. Insufficient privileges for a user will reflect in the user interface as missing elements or disabled menus. In the REST API, insufficient privileges to perform an operation will cause an error with the operation.
Semarchy xDM does not store the users, password and and user/roles associations. All this critical information remains in the application sever or in the enterprise security provider.

Privilege Precedence

Privileges apply in order of precedence: Read/Write then Read then None. As a consequence, a user always has the best privileges associated to all his roles.

For example: The user John has two user-defined roles granted to him:

  • ModelDesigner role has Read privileges for Application Management and Read/Write for Application Design.
  • ProductionManager has Read/Write privileges for Application Management and None on for Application Design

The resulting privileges for John are Read/Write for both Application Management and Application Design.

Privileges Description

The following table describes the platform privileges you can grant to a role:

Platform PrivilegeDescription

Platform Administration

Grants access to the Semarchy Configuration interface to view or configure the Repository, Notification Servers, Variable Value Providers, Image Libraries, Plug-ins, Logging Configuration and Applications Configuratiom.

This privilege does not give access to the Roles configuration. Only a user with the SemarchyAdmin role can create and modify roles.

Application Design

Grants access to all the components of the Model Design perspective in the Application Builder to view or design models.
Grants also access to the model creation/export, as well as model editions management (close a model editions, create a branch, manage the translations), and the Image Library.

Application Management

Grants access to all components for model and application management (the Management view in the Application Builder), including deploying model editions, creating and configuring the data locations (notification policies, continuous loads, etc), the batch poller, the execution engine, the job logs and the purge schedules. This privilege also allows upgrading the data locations, and configuring Variable Value Providers.

Dashboard Management

Grants access to the Dashboard Applications Builder to create Dashboard Applications.

Built-in Roles

The following roles are built in the platform:

  • semarchyConnect: This role must be granted for a user to log in. It should be granted by default to all users connecting to Semarchy xDM.
  • semarchyAdmin : This role has full access to all the features of the platform. semarchyAdmin is the only role that gives you access to the Roles in the Semarchy Configuration user interface.
When a creating a new model, a model-level privilege grant is automatically created for the semarchyAdmin role, giving this role full access to the data. By modifying this privilege grant, the model designer can reduce the privileges of the semarchyAdmin role on the data.
Be cautious when granting the semarchyAdmin role. This role defines a super user who can create roles, grant privileges and update the license information.

Managing Roles and Privileges

Creating the Roles and Users in the Application Server Security Realm

Before declaring a new role in Semarchy xDM, make sure that this role is defined in the application server and that a user is granted with this role and the semarchyConnect role to log in to Semarchy xDM.

The role/user creation procedure depends on the application server hosting Semarchy xDM. Please refer to your application server documentation for more information.

An example is given below for creating a role and a user for Apache Tomcat.

To configure a new role and user for Semarchy xDM:

  1. Stop the Apache Tomcat Server using the stop the Apache Tomcat server using <tomcat>/bin/shutdown.bat (Windows) or <tomcat>/bin/shutdown.sh (UNIX/Linux), where <tomcat> is the Apache Tomcat installation folder.
  2. Edit the <tomcat>/conf/tomcat-users.xml file.
  3. In the <tomcat-users> section, add the following lines (<password> is the password for this user):
     <role rolename="MDMDev">
     <user username="john" password="<password>" roles="semarchyConnect,MDMDev"/>
  4. Save the file.
  5. Restart the Apache Tomcat server using <tomcat>/bin/startup.bat (Windows) or <tomcat>/bin/startup.sh (UNIX/Linux).

A new role MDMDev is created. The user john is also created with the semarchyConnect built-in role and the MDMDev role.

Declaring the Roles in Semarchy xDM

To create new role:

  1. In Semarchy Configuration, select the Overview editor, and then click the Roles link.
  2. In the Roles editor, right-click Roles table and select image New Role. The Install Role wizard opens.
  3. Enter the following information:
    • Name: Role name. This role name must exactly match the role name in the application server security configuration. For example: MDMDev.
    • Label: User-friendly label for the role. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.
    • Email(s): Enter a comma-separated list of email addresses of recipients for notifications sent to this role.
  4. Click Next.
  5. Select the privileges to grant to this role. For example: Model Design: Read/Write, Job and Job Log Administration: Read.
  6. Click Finish.
    The role is created. You can connect a user with this role to test the set of privileges.
Make sure to use a role name that matches exactly (with the same case) a role name defined in the application server configuration.

Sample Roles

You can use the following role examples in a typical Semarchy xDM configuration:

Platform PrivilegeDeveloperProduction UserAdministrator

Platform administration

Read

Read

Read/Write

Application Design

Read/Write

Read

Read

Application Management

Read

Read/Write

Read/Write

Dashboard Management

Read/Write

None

Read

These roles are given as examples and should be adapted to your environment’s requirements.

Scripting Administration

All platform administration tasks are exposed by the REST API to enable automated administration and environment setup.

Model and application management operations such as creating data locations or managing model edition are also available in the REST API. See the Scripting Deployment and Management chapter in the Semarchy xDM Application Management Guide for more information.

Scripted Administration Overview

The REST API exposes endpoints for the following administrative operations:

  • Repository: Create and upgrade repositories, get the platform status and repository information.
  • License: Read current license information and upload a new license file.
  • Roles: Export and import (replace all) roles in the platform.
  • Image Libraries: List available libraries, export and import image libraries content in a zip format.
  • Plugins Bundles: List, delete, import and export bundles containing enricher and validation plugins.
  • Notification Servers and Variable Value Providers: List, get, set or delete the configuration of all or some of these servers.
  • Users: List, create, update or delete all or specific user profiles with their preferences.
  • Configuration: configure the platform and the logging configuration.
For details about using the endpoints, see the built-in REST API documentation.

REST API Documentation

The REST API exposes its built-in documentation as a link in the Welcome page. In addition, this documentation is available for tools as an OpenAPI specification.

To Access the REST API Documentation:

  1. Log in to Semarchy xDM as a user with one of the Application Design, Application Management and Platform Administration platform privileges.
  2. Click the REST API link on the Welcome page.
  3. In the upper-right menu, select Platform Administration.

The documentation exposes the endpoint description and provides request and response samples.