Welcome to Semarchy xDM.
This guide contains information about administering and monitoring Semarchy xDM.
Preface
Overview
Using this guide, you will:
- Understand the Semarchy xDM architecture and components.
- Learn how to manage the various component of the architecture.
- Learn how to manage run-time and troubleshoot errors.
- Learn how to enable and manage a secure environment for Semarchy xDM.
Audience
Document Conventions
This document uses the following formatting conventions:
Convention | Meaning |
---|---|
boldface | Boldface type indicates graphical user interface elements associated with an action, or a product specific term or concept. |
italic | Italic type indicates special emphasis or placeholder variable that you need to provide. |
| Monospace type indicates code example, text or commands that you enter. |
Other Semarchy Resources
In addition to the product manuals, Semarchy provides other resources available on its web site: http://www.semarchy.com.
Obtaining Help
There are many ways to access the Semarchy Technical Support. You can call or email our global Technical Support Center (support@semarchy.com). For more information, see http://www.semarchy.com.
Feedback
We welcome your comments and suggestions on the quality and usefulness
of this documentation.
If you find any error or have any suggestion for improvement, please
mail support@semarchy.com and indicate the title of the documentation
along with the chapter, section, and page number, if available. Please
let us know if you want a reply.
Introduction to Semarchy xDM
Semarchy xDM is the Intelligent Data Hub platform for Master Data
Management (MDM), Reference Data Management (RDM), Application Data Management
(ADM), Data Quality, and Data Governance.
It provides all the features for data
quality, data validation, data matching, de-duplication, data authoring,
workflows, and more.
Semarchy xDM brings extreme agility for defining and implementing data management applications and releasing them to production. The platform can be used as the target deployment point for all the data in the enterprise or in conjunction with existing data hubs to contribute to data transparency and quality.
Its powerful and intuitive environment covers all use cases for setting up a successful data governance strategy.
Architecture Overview
The Semarchy xDM architecture includes:
- The Semarchy xDM Server, a Java EE application deployed and running in an application server. This application serves:
- The Application Builder, Dashboard Builder, Setup and Configuration user interfaces: These web application are used by designers and administrators to create, manage and administer the models and applications designed in Semarchy xDM.
- Data Management Applications and Dashboard Applications: These web applications are used by business users to browse and manage data and visualize metrics dashboards.
- A Rest API to perform programmatically data integration, management and administrative operations.
- The Repository that stores all the metadata used by Semarchy xDM. This includes the data models as well as the definition of the data management and dashboard applications. The repository is hosted in a database schema. A given Semarchy xDM server is always attached to a single repository.
- The Data Locations that contain data for the data models. This data include the golden, master and source data, with all the lineage and history. A data location is hosted in a database schema. Multiple data locations can be attached to a single repository.
The following sections detail the component of this architecture.
Semarchy xDM Server
The Semarchy xDM server is a Java EE application deployed and running in a supported application server. It stores all its information in the
repository. One application is always attached to a single repository,
and connects this repository using a JDBC datasource named
SEMARCHY_REPOSITORY
. This datasource is configured in the
application server.
The Semarchy xDM application is used at design-time to design:
- Data models and applications, using the Application Builder,
- Dashboard applications, using the Dashboard Builder.
At run-time, the application:
- serves the Data Management Application,
- serves the Dashboard Applications,
- serves the REST API to manage data and platform operations,
- runs the jobs that certify golden data.
Certification Process
Semarchy xDM creates or modifies golden data using data feeds from:
- Data feeds from source applications (the Publishers). Such data is published programmatically using the SQL or REST APIs.
- Data creation or changes performed by users in the Data Management applications.
The certification process is triggered as explained below:
- The Publishers or the Data Management application submit to the data hub a batch of source data. Such data is submitted as a Load that contains multiple datasets to process together.
- The Integration Batch Poller polls integration batches at regular intervals. When a new batch is detected, the batch poller requests the Execution Engine to start the Integration Job associated with this batch.
- The Integration Job is created from a Job Definition and executed by the Execution Engine. Data is passed through the different steps of the certification process, and golden data is certified from the source data.
Repository
The repository contains the design-time and run-time information for a given Semarchy xDM Server instance.
Repository Contents
The repository stores the following information:
- For Data Management Models
- The entities, attributes, etc.
- The model versions: branches, editions, etc.
- The platform configuration & security information: roles, privileges, notification servers, preferences, etc.
- The data locations information: deployed model, job definitions, notifications, etc.
- Run-time information: Engine queues, logs, etc.
- For Dashboard Applications
- The dashboard application metadata.
A repository is stored in a database schema accessed from the application using a JDBC datasource named SEMARCHY_REPOSITORY
.
Repository Types
There are two types of repositories:
- Design: All design-time and run-time operations are possible in this type of repository.
- Deployment: With this repository type, you can only import closed model editions and cannot edit them.
The deployment repositories are suitable for production sites. Model transfer from design to deployment repositories is handled via incremental export/import of closed model editions. Refer to the Planning the Installation chapter in the Semarchy xDM Installation Guide for examples of repository deployment patterns.
Data Locations
When a data management hub must be available for run-time (for testing or production purposes), it is generated from a data model defined in the repository, and deployed in a data location. Data Locations contain the deployed data hubs. Each hub contains the golden data, the source data pushed by publishers in the hub, and the various stages of this data in the certification process.
The data location is hosted in a database schema and accessed via a JDBC datasource defined in the application server. A data location refers to the datasource via its JNDI URL.
Data Locations, Repositories and Models
A data location is attached to a repository: You can declare as many data locations as you want in a repository, but a data location is always attached to a single repository. It is not possible to have a data location attached to two repositories at the same time.
You may deploy several model editions successively in a data location, but only one model edition is deployed and is active in the data location at a certain point in time.
Data Location Contents
A Data Location contains the hub data, stored in the schema accessed using the data location’s datasource. This schema contains database tables and other objects generated from the model edition.
The data location also refers three type of jobs (stored in the repository):
- Installation Jobs: The jobs for creating or modifying in a non-destructive way the data structures in the schema.
- Integration Jobs: The jobs for certifying data in these data structures, according to the model job definitions.
- Purge Jobs: The jobs for purging the logs, data lineage and history according to the retention policies.
Data Location Types
There are two types of data locations:
- Development Data Locations: A data location of this type supports deploying open or closed model editions. This type of data location is suitable for testing models in development and quality assurance environments.
- Production Data Location: A data location of this type supports deploying only closed model editions. This type of data location is suitable for deploying MDM hubs in production environments.
Refer to the Planning the Installation chapter in the Semarchy xDM Installation Guide for examples of data location deployment patterns.
Data Structures and Certification Process
A deployed model edition is made of a Data Structure and a set of Integration Jobs.
- The Data Structure of the MDM Hub is a set of tables stored in the database schema of the data location. This structure contains the landing tables for the loads pushed in the hub by the publishers, the golden records tables and the tables handled by the certification process to create golden records from the source records.
- The Integration Jobs are sequences of tasks stored in the repository, which perform the certification for data entering the hub. They are specific the deployed model edition. When a model edition is deployed in a data location, the integration job definition for this model edition replace previous job definitions.
Data Structure Details
The data structure is implemented to support all the successive model editions deployed in the data location. The data structure is created when the first model edition is deployed, and is altered for new model editions. Changes to the structure are only incremental and non-destructive.
For example, the GD_CUSTOMER
table holds the data for all successive design states of the Customer entity.
If a new FaxNumber attribute is added to the entity and deployed with model edition 1.1, the FAX_NUMBER
column is created to contain the fax number data. If this attribute is removed in subsequent model editions (1.2 and so forth), the column for this attribute remains in the data structures, when these model editions are deployed.
Platform Components
The Semarchy xDM platform contains several components described in the sections below.
Integration Batch Poller
The Integration Batch Poller polls the integration batches submitted to the platform on a defined schedule, and starts the Integration Jobs on the Execution Engine.
Execution Engine
The Execution Engine processes the Integration Jobs submitted by the Integration Batch Poller. It orchestrates the certification process to generate the golden data. This engine sequences the jobs in Clusters and Queues.
The execution engine logs the activity of the platform and manages notification policies.
Security
The application uses role-based security for accessing Semarchy xDM features. The users and roles used to connect to the application must be defined in the security realm as part of the application server configuration and then declared in Semarchy xDM.
Role base security is used in Semarchy xDM to define the access privileges to the features of the platform (Platform-Level Security), as well as the privileges to access and modify data in the data locations (Model-Level Security).
Introduction to Semarchy Configuration
Platform-level Administrative tasks are performed in a dedicated user interface called Semarchy Configuration. This section explains how to work with it.
Connecting
To access Semarchy Configuration, you need an URL, a user name and password that have been provided by your Semarchy xDM administrator.
To log in to Semarchy Configuration:
- Open your web browser and connect to the URL provided to you by your
administrator. For example
http://<host>:<port>/semarchy/
where<host>
and<port>
represent the name and port of host running the Semarchy xDM application. The Login Form is displayed. - Enter your user name and password.
- Click Log In. The Semarchy xDM Welcome page opens.
- Click the Configuration button. Semarchy xDM Configuration opens.
To log out of Semarchy Configuration:
- In the Semarchy Configuration User Menu, select Log Out.
Semarchy Configuration
Semarchy Configuration allows to view and administer the following components:
- Applications Configuration: Global parameters for all applications.
- Image Libraries: Manage image libraries and images in these libraries.
- Logging Configuration: Configure the platform logging (trace) for debugging purposes.
- Notification Servers: Add, remove and configure servers used to send job notifications and application emails.
- Plug-ins: View, add or update user-created plug-ins.
- Roles: Declare in Semarchy xDM the application server roles, and grant them with platform-level privileges.
- Variable Value Providers: Configure the system queried by Semarchy xDM to retrieve values for model variables.
Managing the Repository
The repository contains the design-time and run-time information for the Semarchy xDM server.
Understanding Repositories
The repository is created when Semarchy xDM is installed. The type of the repository (Design Repository or Deployment Repository) is set also at creation time, and the application always connects to a single repository.
The type of a repository defines the capabilities of this repository:
- A Design Repository allows you to perform all design-time and run-time operations.
- A Deployment Repository only allows run-time operations. You can import closed model editions in such repository but cannot edit them.
Repository Administration Tasks
Viewing the Repository and System Information
Semarchy xDM exposes the repository and system details in the About dialog.
To view the repository and system details:
- In the Semarchy Configuration User menu, select About.
- In the About dialog:
- The License Information link displays the current license information and allow for Updating the License Key.
- The Repository Information link displays the repository details (including name and version).
- The System Information link displays the platform system details and may be used for support purposes.
Updating the License Key
Semarchy xDM stores in the repository the license information and the license key provided to you for evaluation or when you purchased the product. You can update an expired license key with a new one using the following procedure.
To update the license key:
- In the Semarchy Configuration User menu, select About.
- In the About dialog select the License Information link.
- In the License Key Information dialog, click the Upload License Key File… button.
- Use the Browse button to select the license key file.
- If the selected license key recognized as a valid one, you can click the OK button to register the license key in the repository.
Configuring the Platform
The platform contains of several components that can be managed from Semarchy Configuration.
Configuring Notifications Servers
Notifications tell users or applications when a job completes or when operations are performed into workflows, for example, when task is assigned to a role.
There are two types of notifications:
- Job Notifications issued under certain conditions when an integration job completes. These notifications are used for administration, monitoring, or integration automation. These notifications are configured with Notification Policies in the data locations.
For more information about notification policies, see the Semarchy xDM Application Management Guide. - Workflow Notifications are emails sent to users when operations are performed in a workflow. They are configured in workflow transitions and tasks.
For more information about workflow notifications, see the Semarchy xDM Developer’s Guide.
Both families of Notifications are issued via Notification Servers.
Notifications Servers Types
Notifications recipients may be users or systems. The type of notification sent as well as the recipient depends on the type of notification server configured.
Each notification server uses a Notification Plug-in that:
- defines the configuration parameters for the notification server,
- defines the configuration and form of the notification,
- sends the notifications via the notification servers.
Semarchy xDM is provided with several built-in notification plug-ins:
- JavaMail: The notification is sent in the form of an email via a Mail Session server configured in the application server, and referenced in the notification server. For more information about configuring Mail Session, see the Semarchy xDM Installation Guide.
- SMTP: The notification is sent in the form of an email via a SMTP server entirely configured in the notification server.
- File: The notification is issued as text in a file stored in a local directory or in a FTP/SFTP file server.
- HTTP: The notification is issued as a GET or POST request sent to a remote HTTP server. Use this server type to call a web service with the notification information.
- JMS: The notification is issued as a JMS message in a message queue.
A single notification server having either the JavaMail or SMTP type can be used to send Workflow Notifications. This server is flagged as the Workflow Notification Server
Any servers can be used to send Job Notifications. Each Job Notification Policy specifies the notification server it uses.
Creating a Notification Server
To create a notification server:
- In Semarchy Configuration, select the Overview editor, and then click the Notification Servers link. The Notification Servers editor opens.
- Select the Notification Servers list, right click and select New Notification Server. The Create New Notification Server wizard opens.
- Enter the following workflow parameters:
- Name: Internal name of the notification server.
- Label: User-friendly label for the server.
- Plug-in ID: Select one of the available notification server plug-in.
- Workflow Notification Server: Select this option to use this notification server by default in the workflows. This options can be selected only if the Plug-in ID is JavaMail or SMTP.
- Click Next.
- In the second wizard page, enter the configuration information for your type of server:
- JavaMail:
- JNDI URL: JNDI URL of the Java Mail Session service available in the application server. This URL is typically
java:comp/env/mail/Session
if the Mail Session service is declared asmail/Session
in the application server. - From User: Email address of the sender of the notifications from this server. This address is also used in the reply-to address for notification emails.
- Password If this server requires specific authentication, enter a password for this server.
- JNDI URL: JNDI URL of the Java Mail Session service available in the application server. This URL is typically
- SMTP:
- SMTP Host Name and SMTP Port: Name or address, and port of the SMTP host.
- From User: Email address of the sender of the notifications from this server. This address is also used in the reply-to address for notification emails.
- Authentication Required: If this server requires specific authentication, select the Authentication Required option and enter a User Name and Password for this server, and indicate whether it uses TLS or SSL.
- Additional SMTP Properties Enter additional properties as
property=value
pairs.
- File:
- File System: Select the file system of the file server.
FILE
for a local server,FTP
orSFTP
for a remote server. - Host, Port, Login, Password are required to connect an
FTP
orSFTP
server. - Root Path: Provide the root path for storing the notification file. For example
c:\work\notifications
for Windows or/work/notifications
for UNIX/Linux
- File System: Select the file system of the file server.
- HTTP:
- Scheme: Specify whether the HTTP request should be done using
HTTP
orHTTPS
- Host, Port and optionally Login, Password are used to connect the HTTP server.
- Base Path: Root path appended added after the host and port in the URL. For example:
/
or/rest/api/
. - Use System Properties: Check this option to use the system-defined properties to configure the HTTP connection. This option allows using a proxy configuration defined in the Java parameters for the application server.
- Proxy Host, Proxy Port, Proxy Login and Proxy Password are used to configure the connection through an HTTP proxy.
- Headers: Enter additional HTTP headers as
property=value
pairs.
- Scheme: Specify whether the HTTP request should be done using
- JMS:
- Connection Factory URL: JNDI URL of the factory used to create a connection to the JMS destination. The URL is typically
java:comp/env/jms/ConnectionFactory
if the connection factory is declared asjms/ConnectionFactory
in the application server. - Login and Password used when initiating the JMS connection.
- Connection Factory URL: JNDI URL of the factory used to create a connection to the JMS destination. The URL is typically
- JavaMail:
- Press CTRL-S to save the configuration.
Testing a Notification Server
After configuring the notification server, it is recommended to run a test email on this server.
To test a notification server:
- In the Notification Servers editor, select the notification server you want to test, right-click and select Test Configuration.
- The next steps depend on the type of notification servers:
- File, HTTP, JMS: No further operation is needed. A connection attempt is made on the notification server.
- JavaMail and SMTP: Provide a comma-separated list of email addresses and then click OK. An email is sent via the notification server to these recipients.
Configuring Variable Value Providers
Semarchy xDM uses variables defined in models to enforce
certain data governance policies for a user’s session.
For more information about model variables, see the Semarchy xDM Developer’s Guide.
A Variable Value Provider is a system that can be queried by Semarchy xDM to retrieve the values for these variables. Typically, this system is a server containing information about the user connected to Semarchy xDM.
Two type of variable value providers are supported out-of-the-box:
- Datasource Variable Provider: This variable value provider is a relational database that is accessed through a JDBC datasource. Semarchy xDM can issue SQL statements against this database to retrieve variable values. For example, an employee database that can be queried to retrieve the country of the connected user.
- LDAP Variable Provider: This variable value provider is a directory server that is accessed using the LDAP protocol. Semarchy xDM can issue queries against this directory server to retrieve variable values. For example, an LDAP directory that can be used to retrieve the organizational unit of the connected user.
Variable value providers are configured in the repository, and can be used by any model in this repository.
Creating a Variable Value Provider
To create a variable value provider:
- In Semarchy Configuration, select the Overview editor, and then click the Variable Value Providers link. The Variable Value Providers editor opens.
- Select the Variable Value Providers list, right-click and then select New Variable Value Provider. The Install Variable Value Provider wizard opens.
- Enter the following information:
- Name: Internal name of the variable value provider.
- Label: User-friendly label for the variable value provider. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.
- Select the Plug-in ID corresponding to the variable value provider type: LDAP Variable Provider or Datasource Variable Provider.
- Click Next.
- Click the Edit Expression button.
- In the the Variable Value Configuration dialog, enter the configuration information.
This information differs depending on the selected Plug-In.- For a Datasource Variable Value Provider, enter the following information:
- JNDI Data Source Name: Select a JDBC datasource in the list.
This datasource must be defined in the application server. It is used to connect the database acting as the variable value provider.
- JNDI Data Source Name: Select a JDBC datasource in the list.
- For an LDAP Variable Value Provider, enter the following information:
- LDAP Host: Name or IP address of the LDAP server.
- LDAP Port: Listening port of the LDAP server. The port is typically 389 for non-SSL connections, and 636 for SSL connections.
- Use SSL: Check this options to use SSL to connect to the LDAP server.
- User: Name of the user used to retrieve data from the LDAP Server. Note that this user should have read privileges to the LDAP structure.
- Password: This user’s password.
- For a Datasource Variable Value Provider, enter the following information:
- Click OK to close the Variable Value Configuration dialog.
- Click Finish.
The variable value provider is added to the list.
Testing the Variable Value Provider Configuration
After configuring a new variable value provider, it is recommended to test its configuration.
to test a variable value provider configuration:
- In the Variable Value Providers editor, select of the variable value provider in the list.
- Right-click and select Test Configuration.
A message indicates whether the connection test was successful or not.
Managing Image Libraries
Image libraries store images and icons used by applications. Libraries are stored in the repository, are global to the platform and can be used by all applications. Designers use, in applications, images and icons from the libraries by reference, using image library URLs.
Default Image Libraries
When the repository is created, built-in image libraries are seeded with default Semarchy xDM images and icons:
- mdi is the library for all the Material Design icons and assets. This library cannot be modified.
- default contains Semarchy branding images as well as the default images used by applications, such as the default images for the actions. Icons provided in addition to the mdi library are in this library. This library cannot be modified.
- demo<application_name>_ contains the images/icons used in the <application_name> demonstration application. Such a library is seeded when a demo application is installed.
Browsing Image Libraries
The Image Libraries editor, available in Semarchy Configuration, allows administrators to view and manage the image libraries.
To browse the image libraries:
- Open Semarchy Configuration.
- In the Overview, click the Image Libraries link.
The Image Libraries editor opens.
Use the filter box in the image libraries editor to reduce the number of images displayed. You can use the '%' sign to replace any number of character.
Managing Image Libraries
Image libraries are created and modified using file import. You can import two types of files:
- Image files, for which you must select a target library.
- Zip files, into which images are organized into folders. When a Zip file is imported, one library is created for each folder at the root of the Zip file, and images in this folder are imported into the library. Exporting images creates a file that follows the same format.
To import images:
- Open the image libraries editor.
- Click the Import Image Library button. The import dialog opens.
- Drop either a Zip file or multiple image files into the dialog, or click Browse to select a Zip file or multiple image files.
- If you import image files, select the Target Library for these images. Select Create a New Library to create a new library for these images.
- Click OK to import the images.
To delete images:
- Open the image libraries editor.
- Click an image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.
- Click the Delete Selected Images button, and then confirm the deletion.
To export images:
- Open the image libraries editor.
- Click the image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.
- Click the Export Selected Images button.
A zip file is downloaded, containing all the selected images stored into folders named after their parent library.
Managing Plug-ins
Semarchy xDM allows extending its capabilities using Java code and external APIs. Using the Open Plug-in Architecture, existing services or information systems can contribute to the master data processing and enrichment. You can extend the Enrichment and Validation capabilities in Semarchy xDM through user-defined plug-ins.
For detailed information about plug-in development and packaging, see the Semarchy xDM Plug-in Development Guide.
A Plug-in is delivered as a jar file bundle that must be deployed in each Semarchy xDM server instance running integration jobs that use the plug-in. You do not need to restart the server to take new or updated bundles into account.
These bundles are tagged with a version number. As a consequence, updating an existing plug-in with a newer version of this plug-in will automatically make the platform work with the newer plug-in version. The deployment process installs a new plug-in or replaces an existing plug-in version with a new one.
To deploy a plug-in:
- In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.
- Click the Install or Update Plug-in button in the upper right corner of the Plug-ins editor. The Install/Update Plug-ins dialog opens.
- Click the Browse button and select the plug-in binary file. For
example:
com.acme.phoneStandardizer_1.0.0.jar
. - Click OK. A Status window shows the number of plug-ins installed or updated.
- Your session is closed to take this new plug-in into account. Restart the session.
- In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.
The plug-in now appears in the list, and can be used in the models and the integration jobs.
To uninstall a plug-in:
- In Semarchy Configuration, select the Overview editor, and then click the Plug-ins link.
- Select the plug-in in the list.
- Click the Uninstall Selected Plug-ins button in the editor’s toolbar.
Managing Applications Configuration
The global behavior of the data management applications is configured from Semarchy Configuration.
To set the Applications Configuration:
- In Semarchy Configuration, select the Overview editor, and then click the Applications Configuration link. the Applications Configuration editor opens.
This editor displays the global application parameters:
- Configure the Server Base URL of your Semarchy xDM installation. This URL is seeded when creating the repository and used in workflow notifications to provide links to the server.
- Set the CSV Export Limit and Excel Export Limit to define the maximum export size allowed in each format. Note that generating export files is resource consuming on the server. It is recommended to test the scalability of the new export limits.
- Set the Autocomplete Max Results value to define the maximum number of results that should be fetched by the autocomplete components in the applications. These dynamic drop drown lists fetch records as you type a search strings.
- Change the Header Logo that appears for all applications and in the welcome page by uploading a picture. Note that this picture is scaled to a height of 96 pixels.
You can also change the Favicon used for this page.
Managing Custom Translations
Semarchy xDM allows translating the strings you define in applications (for example, the labels of the business views or fields) as part of the model and application localization effort.
In addition, strings built in Semarchy, visible in all data management and dashboard applications, such as 'Inbox', 'Filter' or 'Global Search', can also be modified in the platform configuration with Custom Translations.
With custom translations, you can:
- Override existing built-in strings. For example, replace 'Inbox' by 'My Tasks' in the Inbox editor title.
- Create translations for new languages for the built-in strings.
The translation process consists of:
- Exporting existing languages' localization file, or blank localization files for new languages. These files are properties files, which can be modified using a text editor.
- Editing these files using a text editor.
- Import the modified files.
Exporting Custom Translations
To export custom translations:
- In Semarchy Configuration, select the Overview editor and then click on the Custom Translations link.
The Custom Translations editor opens and lists the custom translations that you have configured. - If you want to modify existing custom translations, select these translations in the list.
- Click on the Export Translations… button in the editor toolbar.
- In the Export Translation Bundles dialog, select the translation bundles that you want to export. The Hide Languages with no Custom Translations option masks the languages for which you have no custom translation.
- Click OK
The selected translation bundles are exported in a zip file.
Custom Translations Export File Format
The translations are exported as a compressed file that contains properties files (containing key=value
pairs).
These files are organized into folders:
- The
/commons
folder contains property files for all the languages with strings common to the dashboard and data management applications. - The
/dashboards
folder contains property files for all the languages with strings specific to the dashboards component. - The
/applications
folder contains the property files for all the languages with strings specific to the data management applications.
Languages and Variants
The property files have a suffix corresponding to the language code and an optional variant code.
en
is the suffix for the English language, and en_US
is the suffix for United States variant of this language.
When a key needs to be translated:
- Users having their browser configured in the English (United States) variant will use the
en_US
translation if it is available, or else use the Englishen
translation. - Users having their browser in English will directly use the English
en
translation.
In addition, specific properties files with the en_default
and en_default
suffixes represent the default translations built-in Semarchy xDM for English and French. These are here for informational purposes and are ignored during the import process.
Language translation fallback
You do not need to define all translations in the custom translations since the string localization process has a fallback process.
In this process, translations are searched in the following order:
- In the language variant custom translation (for example
fr_FR
for the French (France) language variant) - In the language custom translation (for example
fr
for the French language) - In the language built-in translation (for example
fr_default
), if any exist for the user language. - In the custom English translation (
en
). - In the built-in English translation (
en_default
).
Semarchy xDM includes built-in translations for English and French.
Using the fallback mechanism, you can create custom translations for these two languages to override the built-in translations.
The translations that you provide for these base languages automatically override, in the fallback mechanism, the built-in ones.
Importing Custom Translations
To import custom translations:
- In Semarchy Configuration, select the Overview editor and then click on the Custom Translations link.
The Custom Translations editor opens. - Click on the Import Translations… button in the editor toolbar.
- Click the Add button and select a zip file containing custom translations.
The editor shows in the Languages to import the custom translations available in the file. - Select the property files that you want to import, and then click Finish
The selected translation bundles are imported.
When importing a custom translation for a language that is not translated yet, a new translation is created. If a custom translation already exists for a given language, it is removed and replaced entirely by the new one.
Configuring the Logging
Logging is used to troubleshoot the Semarchy xDM platform issues and monitor the platform activity.
By default, the important log events appear in the Semarchy Configuration Error view. To access this error view, in the Semarchy Configuration user menu, choose the Window > Show View… > Others and then select General > Error Log.
Semarchy xDM uses Apache Log4J to log its activity. This section provide key concepts and configuration samples to work with Log4J. Always refer to the Log4J Manual for reference information.
Log4J works with three key concepts: Appenders, Loggers and Log Levels.
- The Appenders deliver log events to a destination. For example they write events to a file on the application server file system, to a message queue or to the Semarchy Configuration Errors view.
- The Loggers capture log events emitted by software certain components in Semarchy xDM.
- The Log Levels define at the same time the granularity and importance of log events.
Each log events is emitted with a certain log level, is captured by a logger that sends it to an appender for delivery.
Setting the Logging Configuration
Appenders, loggers and log levels are configured through the Logging Configuration editor.
To set the Logging Configuration:
- In Semarchy Configuration, select the Overview editor, and then click the Logging Configuration link. The Logging Configuration editor opens.
- Change the configuration in the editor.
- Press CTRL-S to save this editor.
The new logging configuration is immediately taken into account.
Understanding Log Levels
The Log Level define the granularity and importance of a log event. The log level is one of the following: TRACE
, DEBUG
, INFO
, WARN
, ERROR
, FATAL
.
These values are ordered from highest to lowest granularity, but also in order of importance: The DEBUG
level is very granular and most messages are not relevant in the normal course of operations, whereas the ERROR
and FATAL
messages will appear less frequently but are more important.
When you capture events, via a logger, at a certain level, all events of higher importance are also captured. For example, when setting a logger to INFO
, then the WARN
, ERROR
, and FATAL
events are also captured.
ALL
to capture all log events, and to OFF
to disable log capture.INFO
is typically a good level to start with, as it captures information messages as well as possible warnings and errors.Defining Appenders
Appenders deliver log events to a destination. Appenders are defined with a name and an appender class (that is, a Java class the writes log events to a file, to a JMS message queue, to an email, etc).
Appender classes may have one or more properties to configure their behavior. They also support Layout properties to define the format of the logged event.
In the logging configuration, the appender is defined with the following syntax:
log4J.appender.<appender_name> = <appender_class>
Each property for this appender is defined with the following syntax:
log4J.appender.<appender_name>.<property_name> = <property_value>
Sample Appenders Configuration
Sample appender configurations are provided in this section.
# Appender Class log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender (1) # Appender Properties log4j.appender.FILE.DatePattern='.'yyyy-MM-dd'.log' (2) log4j.appender.FILE.file=${user.home}/.semarchy/logfile.log (2) log4j.appender.FILE.append=true (3) log4j.appender.FILE.encoding=UTF-8 (3) # Appender Layout Configuration log4j.appender.FILE.layout=org.apache.log4j.PatternLayout (4) log4j.appender.FILE.layout.ConversionPattern=%d{yyyy-MM-dd HH\:mm s S}-%t-%x-%-5p-%-10c\:%m%n (4)
1 | The DailyRollingFileAppender appender class creates a daily logging file. You can also use the FileAppender to write to a simple file. |
2 | The daily log file is located in the .semarchy/ sub-directory of the user home directory. Its name is logfile.log to which the date, formatted with the pattern, is concatenated. |
3 | These properties configure the appender behavior: Add log events at the end of the daily file, and use UTF-8 encoding. |
4 | Logged event are formatted using the PatternLayout formatting class and a conversion pattern.This pattern renders log events in the following format: 2010-03-23-main—DEBUG-log4jExample:Here is a debug message |
# Appender writing to the {mdm-short-product-name} Configuration Error view log4j.appender.PDE=com.semarchy.commons.log4j.appender.pde.PDEAppender (1) log4j.appender.PDE.Threshold=INFO (2)
1 | This appender class logs events into the Semarchy Configuration Error view. |
2 | Only log events more important that this threshold will be considered by the appender. Make sure to align this value with the loggers that send events to this appender. |
# Appender sending emails via an SMTP host (usually for important error messages) log4j.appender.SMTP = org.apache.log4j.net.SMTPAppender (1) log4j.appender.SMTP.SMTPHost = <smtp_host_name> (2) log4j.appender.SMTP.SMTPPort = <smtp_host_port> (2) log4j.appender.SMTP.From = email@my-domain.com (2) log4j.appender.SMTP.To = admin1@my-domain.com,admin2@my-domain.com (3) log4j.appender.SMTP.Subject = Log Event (3) log4j.appender.SMTP.layout = org.apache.log4j.TTCCLayout (4)
1 | This appender class sends log events by email. |
2 | Mail Server configuration. Note that additional properties may be set for the SMTP user, password, protocol, etc. |
3 | Email recipients and subject. |
4 | Log events are formatted using the TTCC Layout, a detailed format containing the time, thread, category and nested diagnostic context. |
Defining Loggers
Loggers are named and organized as a hierarchy of classes emitting log events.
For example, com.semarchy
is the parent of com.semarchy.mdm
. The highest logger in the hierarchy is the rootLogger
.
You configure loggers with a log level, and attach them to one or more appenders. Any event emitted by the class that is more important than (or as important as) the given log level is captured by the logger and sent to the appender.
Use the following syntax to attach a logger to appenders with a given log level:
log4j.logger.<loggerName> = <level>, <appender_name> [,<appender_name>, …]
For example, to send DEBUG
, INFO
, WARN
, ERROR
and FATAL
events emitted by the com.semarchy.platform.setup.IPlatformManager
class to the appender named PDE
:
log4j.logger.com.semarchy.platform.setup.IPlatformManager=DEBUG, PDE
Logger inherit their appenders additively from their parents in the hierarchy. They can override the level provided by their parent.
In the following configuration, the com.semarchy.platform.setup.IPlatformManager
logger inherits from the PDE appender of the com.semarchy
logger, and sends ERROR
events to the PDE
appender in addition to the SMTP
appender.
log4j.logger.com.semarchy=INFO, PDE log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP
To prevent such inheritance, use the following syntax:
log4j.additivity.<logger_name> = false
For example, in the the following configuration, com.semarchy.platform.setup.IPlatformManager
will no longer log into the PDE
appender in addition to SMTP
.
log4j.logger.com.semarchy=INFO, PDE log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP log4j.additivity.com.semarchy.platform.setup.IPlatformManager=false
The following configuration is the default one for Semarchy xDM.
log4j.logger.com.semarchy=INFO log4j.logger.com.semarchy.commons.sql=INFO log4j.logger.com.semarchy.mdm.datahub.services.query.datamgr.IDataManager=INFO log4j.logger.com.semarchy.platform.engine=INFO log4j.logger.com.semarchy.platform.engine.PluginExecution=INFO log4j.logger.com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor=INFO log4j.logger.com.semarchy.platform.product.notification.JobNotificationHandler=ERROR log4j.logger.com.semarchy.platform.setup.IPlatformManager=INFO log4j.logger.org=INFO log4j.logger.org.apache.aries.blueprint=WARN log4j.logger.org.apache.commons.beanutils.converters=WARN log4j.logger.org.apache.cxf=WARN log4j.logger.org.apache.cxf.services = WARN log4j.logger.org.apache.directory.shared.asn1.ber.Asn1Decoder=ERROR log4j.logger.org.ops4j.pax.logging=WARN log4j.logger.org.quartz=WARN log4j.logger.org.springframework.jdbc.core.JdbcTemplate=INFO log4j.rootLogger=INFO, PDE
The following table lists the various loggers available in Semarchy xDM:
Logger Name | Description |
---|---|
| Root logger for the Semarchy platform. |
| TRACE logs SQL queries made by the Semarchy Application Builder and Configuration user interfaces. |
| Logs database operations performed by the platform for the MDM applications while browsing data. DEBUG logs all queries and execution times. |
| Logs execution engine events. DEBUG logs all submitted jobs, and ERROR can be used to troubleshoot engine issues. |
| Logs execution engine job processing. Use DEBUG to see execution engine details. |
| Logs enricher and validation plugins execution. DEBUG logs plugins feedback, TRACE logs the processing of every row. |
| Logs the Batch Poller activity. DEBUG logs every polled interval. |
| Logs job notifications. ERROR traces notification failures. |
| Logs platform status changes. INFO traces normal status changes. ERROR traces abnormal statuses. |
| INFO traces REST requests, responses, and faults between the MDM applications and the server. |
| DEBUG traces the SQL queries made by the MDM applications and REST API calls as well as Dashboards-related SQL queries. |
Managing the Security
The application uses role-based security and privilege grants for accessing the Semarchy xDM features as well as the data contained in the data location.
Understanding the Security Model
Platform-Level and Model-Level Security
There are two levels of security in Semarchy xDM:
- Platform-Level Security defines access to the features of the platform. For example, access to the administrative features, or access to the design-time capabilities. Platform-level security sets platform users’ privileges (who can design models, monitor executions, manage security, etc.).
- Model-Level Security defines security privileges to access and modify data in the data locations. Defining these privileges is a data governance decision and should be defined as part of the data governance initiative. Defining Model Security is covered in the Securing Data chapter of the Semarchy xDM Developer’s Guide.
Role-Based Security
Both levels of security are role-based:
- The Privileges (platform level/model level) are granted to Roles in Semarchy xDM.
- These Roles are declared in Semarchy xDM. The roles declared in Semarchy xDM must map roles that pre-exist in the application server. These application server roles are created and granted to application server users as part of the application server configuration
- Users logging in to Semarchy xDM use their application server credentials.
Depending on the application server hosting the Semarchy xDM server, the roles/user association may be made through a
concept of group: A user belongs to a group and the role is granted to
the group.
Depending on its configuration, the application server may delegate user
authentication and management in general to a security provider (SSO,
LDAP, etc…).
Security Context
When you log in to Semarchy:
- You enter the user and password in the Semarchy xDM login window.
- This information is passed to the application server.
- The application server itself or its security provider (SSO, LDAP, etc.) authenticates the user, gets the list of roles associated to this user (possibly via groups) and returns this list of roles in the session’s Security Context.
- Semarchy xDM starts a session with this security
context, allowing:
- Certain platform features depending on the Platform-Level
Privileges granted to the roles.
- Certain data access/modification capabilities depending on the Model-Level Privileges granted to the roles.
- Certain platform features depending on the Platform-Level
Privileges granted to the roles.
The role names defined in Semarchy must exactly match the role names returned by the security provider.
For example, for a Tomcat installation, you define in the
tomcat-user.xml
file a role named FINANCEADMIN, and in Semarchy, you declare a FinanceAdmin role. These two roles will not match. Tomcat users with the FINANCEADMIN role will not be granted the privileges defined in Semarchy for the FinanceAdmin role.Privilege Precedence
Privileges apply in order of precedence: Read/Write then Read then None. As a consequence, a user always has the best privileges associated to all his roles.
For example: The user John has two user-defined roles granted to him:
- ModelDesigner role has Read privileges for Application Management and Read/Write for Application Design.
- ProductionManager has Read/Write privileges for Application Management and None on for Application Design
The resulting privileges for John are Read/Write for both Application Management and Application Design.
Privileges Description
The following table describes the platform privileges you can grant to a role:
Platform Privilege | Description |
---|---|
Platform Administration | Grants access to the Semarchy Configuration interface to view or configure the Repository, Notification Servers, Variable Value Providers, Image Libraries, Plug-ins, Logging Configuration and Applications Configuratiom. This privilege does not give access to the Roles configuration. Only a user with the SemarchyAdmin role can create and modify roles. |
Application Design | Grants access to all the components of the Model
Design perspective in the Application Builder to view or design models. |
Application Management | Grants access to all components for model and application management (the Management view in the Application Builder), including deploying model editions, creating and configuring the data locations (notification policies, continuous loads, etc), the batch poller, the execution engine, the job logs and the purge schedules. This privilege also allows upgrading the data locations, and configuring Variable Value Providers. |
Dashboard Management | Grants access to the Dashboard Applications Builder to create Dashboard Applications. |
Built-in Roles
The following roles are built in the platform:
- semarchyConnect: This role must be granted for a user to log in. It should be granted by default to all users connecting to Semarchy xDM.
- semarchyAdmin : This role has full access to all the features of the
platform. semarchyAdmin is the only role that gives you access to the Roles in the
Semarchy Configuration user interface.
Managing Roles and Privileges
Creating the Roles and Users in the Application Server Security Realm
Before declaring a new role in Semarchy xDM, make sure that this role is defined in the application server and that a user is granted with this role and the semarchyConnect role to log in to Semarchy xDM.
An example is given below for creating a role and a user for Apache Tomcat.
To configure a new role and user for Semarchy xDM:
- Stop the Apache Tomcat Server using the stop the Apache Tomcat
server using
<tomcat>/bin/shutdown.bat
(Windows) or<tomcat>/bin/shutdown.sh
(UNIX/Linux), where<tomcat>
is the Apache Tomcat installation folder. - Edit the
<tomcat>/conf/tomcat-users.xml
file. - In the
<tomcat-users>
section, add the following lines (<password>
is the password for this user):<role rolename="MDMDev"> <user username="john" password="<password>" roles="semarchyConnect,MDMDev"/>
- Save the file.
- Restart the Apache Tomcat server using
<tomcat>/bin/startup.bat
(Windows) or<tomcat>/bin/startup.sh
(UNIX/Linux).
A new role MDMDev
is created. The user john
is also created
with the semarchyConnect
built-in role and the MDMDev
role.
Declaring the Roles in Semarchy xDM
To create new role:
- In Semarchy Configuration, select the Overview editor, and then click the Roles link.
- In the Roles editor, right-click Roles table and select New Role. The Install Role wizard opens.
- Enter the following information:
- Name: Role name. This role name must exactly match the role name
in the application server security configuration. For example:
MDMDev
. - Label: User-friendly label for the role. Note that as the Auto
Fill box is checked, the Label is automatically filled in. Modifying
this label is optional.
- Email(s): Enter a comma-separated list of email addresses of recipients for notifications sent to this role.
- Name: Role name. This role name must exactly match the role name
in the application server security configuration. For example:
- Click Next.
- Select the privileges to grant to this role. For example: Model Design: Read/Write, Job and Job Log Administration: Read.
- Click Finish.
The role is created. You can connect a user with this role to test the set of privileges.
Sample Roles
You can use the following role examples in a typical Semarchy xDM configuration:
Platform Privilege | Developer | Production User | Administrator |
---|---|---|---|
Platform administration | Read | Read | Read/Write |
Application Design | Read/Write | Read | Read |
Application Management | Read | Read/Write | Read/Write |
Dashboard Management | Read/Write | None | Read |
Scripting Administration
All platform administration tasks are exposed by the REST API to enable automated administration and environment setup.
Scripted Administration Overview
The REST API exposes endpoints for the following administrative operations:
- Repository: Create and upgrade repositories, get the platform status and repository information.
- License: Read current license information and upload a new license file.
- Roles: Export and import (replace all) roles in the platform.
- Image Libraries: List available libraries, export and import image libraries content in a zip format.
- Plugins Bundles: List, delete, import and export bundles containing enricher and validation plugins.
- Notification Servers and Variable Value Providers: List, get, set or delete the configuration of all or some of these servers.
- Users: List, create, update or delete all or specific user profiles with their preferences.
- Configuration: configure the platform and the logging configuration.
REST API Documentation
The REST API exposes its built-in documentation as a link in the Welcome page. In addition, this documentation is available for tools as an OpenAPI specification.
To Access the REST API Documentation:
- Log in to Semarchy xDM as a user with one of the Application Design, Application Management and Platform Administration platform privileges.
- Click the REST API link on the Welcome page.
- In the upper-right menu, select Platform Administration.
The documentation exposes the endpoint description and provides request and response samples.