Semarchy xDM Installation Guide

Hardware Requirements

The Semarchy xDM server runs as an application in a supported application server. The hardware requirements are those of the application Server.
Refer to your application server documentation for more information about the required hardware requirements.

Software Requirements

This section contains a list of software requirements for this release of Semarchy xDM.

Semarchy xDM Server

The Semarchy xDM Server is a Java EE application deployed and running in a supported application server.

This server provides several access methods:

The Semarchy xDM server stores its information in a repository. One application is always attached to a single repository, and connects this repository using a JDBC datasource named SEMARCHY_REPOSITORY configured in the application server.

The Semarchy xDM application is used at design-time to design models and applications, and deploy them. At run-time, it also manages the processes involved to schedule and execute the certification process in the hub.

The application uses role-based security for accessing Semarchy xDM features. The users and roles used to connect to the application must be defined in the security realm of the application server. Configuring the roles and users is part of the application configuration.

Repository

The repository stores the following information:

A repository is stored in an database/schema accessed from the application using a JDBC datasource named SEMARCHY_REPOSITORY.

Data Locations

Data managed by Semarchy is stored in a data location. The data location contains data structured after the model entities and attributes, and runs certification jobs generated after the rules (enricher, validation, matching, etc) defined in the model. A given model edition is deployed into a data location to host data matching the model structure and rules. At run-time, the data location contains the the golden, master and source data, with all the lineage and history.

The data location is hosted in an database/schema and accessed via a JDBC datasource defined in the application server. A data location refers to the datasource via its JNDI URL.

Pattern #1: Development, QA/UAT, and Production in Different Sites

This pattern assumes that the development, QA/UAT (Quality Assurance, User Acceptance Tests), and production sites are located on different networks or sites.

For this pattern, three repositories are created instead:

With this configuration:

Pattern #2: Development/QA/UAT and Production in Different Sites

This pattern is similar to Pattern #1 but assumes that development and QA/UAT are co-located in one site, and production is located in a remote location.

For this pattern, two repositories are created:

With this configuration:

Pattern #3: Single Repository and Project

This pattern assumes that a single project is designed through a development/QA/Production lifecycle.

For this pattern:

In this pattern, a single repository contains the development, QA and production editions of the models. Model versioning allows freezing and delivering to the next stage (and next deployment location) a model as it moves along its lifecycle.

Pattern #4: Single Repository, Multiple Projects.

This pattern is similar to the previous one but assumes that several projects/models are managed in the same repository.

For this pattern:

The organization is the same as in pattern #3, but a set of data locations exists for each project managed in the single repository.

Reference Architecture for High-Availability

In a clustered deployment, only one instance of the Semarchy xDM application manages and runs the certification processes. This instance is the Active Instance. A set of clustered Semarchy xDM applications serves users accessing the user interfaces (Application Builder, Dashboard Builder, xDM Discovery, Configuration or the data management applications) as well as applications accessing data locations via integration points. These are Passive Instances. The back-end databases hosting the repository and data locations are deployed in a database cluster, and an HTTP load balancer is used as a front-end for users and applications.

The reference architecture for such a configuration is described in the following figure:

This architecture is composed of the following components:

In this architecture:

Configuring Semarchy xDM for High-Availability

General Considerations

Configure the database with a charset that supports all languages, such as AL32UTF8 for Oracle and UTF8 for PostgreSQL. Semarchy xDM uses specific characters for storing internal information, and applications in Semarchy xDM natively support multi-lingual data without preventing users from entering accented characters (or Cyrillic or Arabic or Chinese). A database configured for a specific language or with a limited charset may not function optimally with Semarchy xDM.

Oracle

Repositories and data locations should be located in separate schemas. However, they do not necessary need to be located in the same database.

Semarchy xDM ships with an Oracle JDBC driver (ojdbc8.jar) for Oracle Database version 12c Release 2 (12.2.x). This driver is strongly recommended for all recent database versions. If you are using an older Oracle version (11g), it is recommended to review the compatibility of this driver with your Oracle database version and possibly install to an older driver version instead (ojdbc6 or ojdbc7).

com.semarchy.mdm.runtime.data.InvalidDataAccessResourceUsageException: java.lang.RuntimeException: Unexpected DB value .... (Class oracle.sql.TIMESTAMP for logicalType TIMESTAMP)

<Resource
	name="...
	...
	connectionProperties="oracle.jdbc.J2EE13Compliant=true"
	...
/>

PostgreSQL

Repositories and data locations should be located in separate schemas.

SQL Server

The configuration presented above uses logins defined in the database. Change this script to use Windows or AD logins as needed.

Semarchy xDM does not support schemas for SQL Servers. One database is required for each repository and data location, and each user used to connect these databases should have the db_owner (dbo) role.

Be cautious of the collation when configuring the database instance:

SQL Server repository and data locations databases should also be configured as follows for Semarchy xDM:

Repository

The following considerations should be taken into account when sizing the repository database/schema:

Data Location

The following considerations should be taken into account when sizing the data location databases/schema:

A recommended original sizing is to add the source data volume pushed for each entity by all publishers plus one data authoring (the overall input) and multiply it by a factor of 10. It is recommended after the original sizing to monitor the size of the data location in the normal course of operations and adjust the sizing appropriately.

Data Retention Policies can be created to define the volume of data to retain in the data locations, and Data Purges can be scheduled to trigger the pruning of unnecessary data. Defining Retention Policies is covered in the Securing Data chapter of the Semarchy xDM Developer’s Guide. Data Purges are described in the Managing Execution chapter of the Semarchy xDM Administration Guide.

Datasource Tuning Considerations

When configuring datasources for Semarchy xDM, more specifically in production environments, you must take into account several considerations listed below.

Installing the Database JDBC Drivers

Install the JDBC drivers to connect the repository and data location databases, as well as the additional drivers required for the databases accessed by the xDM Dashboards charts and dashboards, or profiled by xDM Discovery.

To install the JDBC drivers:

Installing the Mail Session Libraries

This configuration is required for mail notifications using JEE Mail Session.

To install the Java Mail Libraries:

Installing additional libraries not included in JDK 11

If running Tomcat with a JDK 11, you must copy to the <tomcat>/lib/ folder libraries required for Semarchy that are not provided in this version of the JDK. This step is not required for a JDK 8.

You will find these libraries, listed below, in the `temp/mdm-server/additional-libraries/`folder.

com.sun.activation.jakarta.activation_*.jar
com.sun.istack.commons-runtime_*.jar
com.sun.xml.fastinfoset.FastInfoset_*.jar
jakarta.jws-api_*.jar
jakarta.xml.bind-api_*.jar
jakarta.xml.soap-api_*.jar
org.apache.servicemix.specs.jaxws-api-*.jar
org.glassfish.jaxb.runtime_*.jar
org.glassfish.jaxb.txw2_*.jar
org.jvnet.staxex.stax-ex_*.jar

LDAP

Semarchy xDM supports authenticating as well as roles retrieval from an external directory, such as LDAP or Active Directory. The information entered in the login form is passed to the external directory, which returns, if the user is valid one, this user’s roles.

To delegate the authentication to an LDAP directory, add to the semarchy.xml configuration file a Tomcat JNDI Realm definition as shown in the example below. This configuration must be customized to match your LDAP directory configuration.

<Realm className="org.apache.catalina.realm.JNDIRealm"
	connectionURL="ldap://ldaphost.mydomain.com:389"		(1)
	userPattern="uid={0},ou=users,ou=people,dc=myCompany,dc=com"	(2)
	roleBase="ou=groups,ou=people,dc=myCompany,dc=com"		(3)
	roleName="cn"							(3)
	roleSearch="(member={0})" />					(3)

The parameters of the realm must be customized to your configuration:

OpenID and OpenID Connect

OpenID is a mechanism used to delegate authentication to another provider. With OpenID configured, you can use Google, Yahoo, Facebook or Twitter accounts to authenticate to Semarchy xDM, to replace or in addition to the login form.

There are two main "implementations" of OpenID supported by the authentication providers: OpenID 2.0 and OpenID Connect.

The next sections explain how to configure OpenID 2.0 or OpenID Connect for Tomcat, and how to mix these authentication schemes with a Login Form.

Configuring OpenID 2.0

OpenID is used as an authentication service, which does not includes the roles. A realm (LDAP, File, or other) is usually defined to manage and serve the roles of the users identified with OpenID.

Configuring OpenID 2.0 with Tomcat requires the Open ID Authenticator for Tomcat, which extends Tomcat authentication to support OpenID 2.0.

To configure authentication with OpenID:

<!-- OpenID Valve configuration for Authentication -->
<Valve className="org.bsworks.catalina.authenticator.openid.OpenIDAuthenticator"
	singleProviderURI="https://me.yahoo.com"
	loginNameAttributeType="http://axschema.org/contact/email"
	allowedClaimedIDPattern="https://me.yahoo.com/a/.+"
	hostBaseURI="http://mdm_host:port" /> 	(1)

<!--Realms configuration -->
<Realm className="org.bsworks.catalina.authenticator.openid.OpenIDRealm">
	<!--
	This realm provides authorizations for the OpenID-authenticated users.
	It uses LDAP with each username equal to the OpenID username.
	Any Tomcat realm can be used for this purpose.
	-->
	<Realm  className="org.apache.catalina.realm.JNDIRealm"
		connectionURL="ldap://ldaphost.mydomain.com:389"			(2)
		userPattern="uid={0},ou=users,ou=people,dc=myCompany,dc=com"	(3)
		userPassword="uid"						(3)
		roleBase="ou=groups,ou=people,dc=myCompany,dc=com"
		roleName="cn"
		roleSearch="(member={0})" />
</Realm>

<!-- OpenID Valve configuration for Authentication -->
<Valve className="org.bsworks.catalina.authenticator.openid.OpenIDAuthenticator"
	loginNameAttributeType="http://axschema.org/contact/email"
	singleProviderURI="https://me.yahoo.com"
	allowedClaimedIDPattern="https://me.yahoo.com/a/.+"
	hostBaseURI="http://mdm_host:myport" />

<!--
	Login form configuration, allowing to display a OpenID authentication button
	pointing to OpenID Provider (Yahoo) given by parameter value
-->
<Parameter name="SingleSignOn"
           value="openid_identifier=https://me.yahoo.com"
           override="true" />

<!--Realms configuration -->
<Realm className="org.bsworks.catalina.authenticator.openid.OpenIDRealm">
	<!--
	The first realm provides authorizations for the OpenID-authenticated users.
	This realm uses LDAP with each username equal to the OpenID username.
	Any Tomcat realm can be used for this purpose.
	-->
	<Realm  className="org.apache.catalina.realm.JNDIRealm"
		connectionURL="ldap://ldaphost.mydomain.com:389"
		userPattern="uid={0},ou=users,ou=people,dc=myCompany,dc=com"
		userPassword="uid"
		roleBase="ou=groups,ou=people,dc=myCompany,dc=com"
		roleName="cn"
		roleSearch="(member={0})" />

	<!--
	The second realm (and subsequent ones) are used as fallbacks for
	authentication and authorization.
	Local realms can be used as local authentication providers,
	or fallback providers if the OpenID authentication fails.
	-->
	<Realm 	className="org.apache.catalina.realm.JNDIRealm"
		connectionURL="ldap://ldaphost.mydomain.com:389"
		userPattern="uid={0},ou=users,ou=people,dc=myCompany,dc=com"
		roleBase="ou=groups,ou=people,dc=myCompany,dc=com"
		roleName="cn"
		roleSearch="(member={0})" />

</Realm>

Configuring OpenID Connect

OpenID Connect is a newer implementation, and authentication providers move over time from OpenID 2.0 to OpenID Connect. It is supported by example by Google, Amazon Cognito, Microsoft Azure AD, ADFS, and OKTA.

Configuring OpenID Connect with Tomcat requires the OpenID Connect Authenticator for Tomcat, which extends Tomcat authentication to support OpenID Connect.

To configure authentication with OpenID:

<!-- Valve configuration for OpenID Connect authentication -->

<Valve className="org.bsworks.catalina.authenticator.oidc.tomcat85.OpenIDConnectAuthenticator"
	providers="[{				(1)
		name: Google,
		issuer: https://accounts.google.com,
		clientId:     xxxxx,	(2)
		clientSecret: xxxxxx	(2)
	}]"
	additionalScopes="email"	(3)
	usernameClaim="email"		(4)
	noForm="true" 			(5)
	hostBaseURI="http://mdm_host:port" (6)
	<!-- logoutUrl="http://myhost.myport/semarchy/logout.do" --> (7)
	landingPage="/"/>

<!--
The following LDAP realm provides the authorizations of the OpenID Connect-authenticated users.
Any Tomcat realm can be used for this purpose.
-->
<Realm className="org.apache.catalina.realm.JNDIRealm"	(1)
	connectionURL="ldap://ldaphost.mydomain.com:389"
	userPattern="uid={0},ou=users,ou=people,dc=myCompany,dc=com"
	roleBase="ou=groups,ou=people,dc=myCompany,dc=com"
	roleName="cn"
	roleSearch="(member={0})"

    <CredentialHandler className="com.semarchy.tool.jee.tomcat.CaseInsensitiveCredentialHandler"/> (2)
	/>

<Valve className="com.semarchy.tool.jee.tomcat.OpenIdConnectAuthenticator"
	providers="[{		(1)
		name: 'Microsoft Azure AD',
		issuer:       xxxx,	(2)
		clientId:     xxxx,	(2)
		clientSecret: xxxxx (2)
	}]"
	usernameClaim="email"	(3)
	additionalScopes="email groups" (4)
	hostBaseURI="http://myhost:myport"
	noForm="true"
	groupClaim="groups" (5)
	groupSeparator=","

	<!-- Role mapping is supported if required -->
	roleMappingEnabled = "true" (6)
	keepMappedRoles="false"
	keepUnmappedRoles="false"
	regexEnabled="true"

	landingPage="/"

	<!-- logoutUrl="http://myhost.myport/semarchy/logout.do" --> (7)
	/>

<!-- Realm using the roles extracted from the ID Token -->
<Realm className="com.semarchy.tool.jee.tomcat.OpenIdConnectRealm" />

<!-- Valve configuration for mixed OpenID Connect authentication -->

<Valve className="org.bsworks.catalina.authenticator.oidc.tomcat85.OpenIDConnectAuthenticator"
	providers="[{				(1)
		name: Google,
		issuer: https://accounts.google.com,
		clientId:     xxxxx,
		clientSecret: xxxxxx
	}]"
	usernameClaim="email"
	additionalScopes="email"
	noForm="false" 			(2)
	hostBaseURI="http://mdm_host:port"
	landingPage="/"/>

<!--
	This realm provides authentication for form-authenticated users as well as authorization
	for OpenID Connect and form-authenticated users.
	These users are stored in an "OpenIDDatabase" file resource, declared in server.xml.
	This file contains username and roles for all users. Only users authenticated by form
	will use the passwords stored in this files.
-->
	<Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="OpenIDDatabase"/>

<!--
The resource named "OpenIDDatabase" used in the realm configuration
must be declared as the /conf/openid-users.xml file stored in the
application server's file system.
-->
<Resource name="OpenIDDatabase" auth="Container"
          type="org.apache.catalina.UserDatabase"
          description="User database"
          factory="org.apache.catalina.users.MemoryUserDatabaseFactory"
          pathname="conf/openid-users.xml" />

<user username="john.doe@mydomain.com" password="xxxx" roles="semarchyConnect,businessUser"/>
<user username="local_admin" password="xxxx" roles="semarchyConnect,semarchyAdmin"/>

Windows Authentication Using Waffle

Windows Authentication (SSO) is supported in Tomcat using the Waffle (Windows Authentication Framework) component.
Using this mechanism, the user connected to the windows machine is used to authenticate to Semarchy xDM.

To configure Windows Authentication:

To enable Windows authentication, use the following configuration.

<Valve className="com.semarchy.tool.jee.tomcat.RoleMappingNegotiateAuthenticator"
	principalFormat="fqn"
	roleFormat="fqn"
	protocols="NTLM"
	/>

<Realm className="waffle.apache.WindowsRealm" />

To mix Login Form and Windows authentication use the following configuration:

<parameter name="SingleSignOn"
           value="action=j_negotiate_check"
           override="true" />

<Valve className="com.semarchy.tool.jee.tomcat.RoleMappingMixedAuthenticator"
	principalFormat="fqn"
	roleFormat="fqn"
	protocols="NTLM"
	/>

<Realm className="waffle.apache.WindowsRealm" />

Using the Tomcat Role Mapper

When the service providing the authorizations returns groups of users (if it does not support the concept of Roles) or role names that cannot exactly match the roles declared in Semarchy xDM (for example, if they include spaces or special characters), you have to configure a mapping between these role names and the Semarchy xDM role names.

Semarchy xDM provides a specific component for this purpose, called the Role Mapper.

To configure the Role Mapper:

<Realm className="com.semarchy.tool.jee.tomcat.RoleMappingRealm">
	<!-- This is the Realm to which role mapping is applied. -->
	<Realm ClassName="org.apache.catalina.realm.JNDIRealm"
	...
	/>
</Realm>

<directory_group>=<semarchy_role_1>,<semarchy_role_2>,...

AdministratorsGroup=semarchyConnect,semarchyAdmin
UsersGroup=semarchyConnect
Data Stewards=demoDataStewards

MDM\u0020Users=semarchyConnect
Global\u005CStewards=semarchySteward

SUPP_.* = Supplier 	(1)
SUPP_(.*) = $1		(2)
(.*) = $1		(3)

<Realm className="com.semarchy.tool.jee.tomcat.RoleMappingRealm"
       keepMappedRoles="false"
       keepUnmappedRoles="false"
       regexEnabled="true"
       rolesMappingPathName="/home/user/map.properties">

Using the Tomcat Wrapper Realm

To simplify the configuration of user authentication and authorizations in Apache Tomcat, Semarchy xDM comes with a Wrapper realm that supports two nested realms:

<Realm className="com.semarchy.tool.jee.tomcat.AuthAndRolesRealm">

    <!-- First realm is for authentication -->
    <Realm className="org.apache.catalina.realm.JNDIRealm"
           connectionURL="ldap://ldaphost.mydomain.com:389"
           userPattern="uid={0},ou=users,ou=people,dc=myCompany,dc=com"
           />

    <!-- Second realm is for authorizations -->
    <Realm className="org.apache.catalina.realm.JDBCRealm"
           driverName="oracle.jdbc.driver.OracleDriver"
           connectionURL="jdbc:oracle:thin:@dbserver:1521:ora11"
           userTable="users"
           userNameCol="user_name"
           userCredCol="user_name"
           userRoleTable="user_roles"
           roleNameCol="role_name"

		<!-- Credential handler for case-sensitivity differences -->
		<CredentialHandler className="com.semarchy.tool.jee.tomcat.CaseInsensitiveCredentialHandler"/>

           />
</Realm>

Configuring JMS Using the JNDI Lookup Factory

To configure the JNDI Lookup Factory:

<!-- The Connection Factory encapsulates a set of connection configuration parameters.
     It is used to create a connection with the JMS provider. -->

<Resource name="jms/<connection_factory_name>"			(1)
	auth="Container"
	type="javax.jms.ConnectionFactory"
	factory="com.semarchy.tool.jee.tomcat.jndi.JndiLookupFactory"
	jndiKey="<connection_factory_jndi_location_in_provider>"(2)
	initialCtxFactory="<initial_context_factory>"		(3)
	providerUrl="<provider_url>"				(4)
	username = "<jms_server_login>"				(5)
	password = "<jms_server_password>"			(5)
/>
<!-- Depending on the JMS provider, you must provide additional parameters.
     For example:
     java.naming.security.protocol = "ssl"
-->

The Connection Factory resource definition uses the following parameters:

<!-- A destination is a JMS Queue or Topic to send notifications to. -->

<Resource name="jms/<destination_name>" 			(1)
	auth="Container"
	type="javax.jms.Queue"					(2)
	factory="com.semarchy.tool.jee.tomcat.jndi.JndiLookupFactory"
	jndiKey="<destination_jndi_location_in_provider>" 	(3)
	initialCtxFactory="<initial_context_factory>" 		(4)
	providerUrl="<provider_url>" 				(5)
	username = "<jms_server_login>"				(6)
	password = "<jms_server_password>"			(6)
/>
<!-- Depending on the JMS provider, you must provide additional parameters.
     For example:
     java.naming.security.protocol = "ssl"
-->

The JMS destination resource definition uses the following parameters:

Using JMS Destinations from the JNDI Lookup Factory

When configuring the Notification Server and Job Notification Policy, or order to use the JMS destination defined:

Both those values must be prefixed with java:comp/env/ when used.

Installing the Database JDBC Driver

Install the JDBC drivers to connect the repository and data location databases, as well as the additional drivers required for the the datasources accessed by xDM Dashboards or xDM Discovery.

To install the JDBC drivers:

Installing the Database JDBC Driver

Install the JDBC drivers to connect the repository and data location databases, as well as the additional drivers required for the databases accessed by the xDM Dashboards charts and dashboards, or profiled by xDM Discovery.

To install the JDBC drivers:

Configuring the Security Realm and Semarchy xDM Administrator

To configure the security realm:

To configure the semadmin user:

Configuring Groups/Roles Mappings

The configured realm uses the default Java Authorization Contract for Containers (JACC) provider included in Glassfish. This JACC provider does not support dynamic roles, and mandates that the mappings between Groups and Roles are defined in the deployed application descriptor file.

To define the groups/roles mappings:

<security-role-mapping>
 <role-name>semarchyConnect</role-name>
 <group-name>semarchyConnectGroup</group-name>
</security-role-mapping>
<security-role-mapping>
 <role-name>semarchyAdmin</role-name>
 <group-name>semarchyAdminGroup</group-name>
</security-role-mapping>

Installing the Database JDBC Driver

Install the JDBC drivers to connect the repository and data location databases, as well as the additional drivers required for the databases accessed by the xDM Dashboards charts and dashboards, or profiled by xDM Discovery.

To install the JDBC drivers:

Installing the Pooling Libraries

This configuration uses connection pooling available in DBCP.

To install the Connection Pooling Libraries:

Installing the Database JDBC Driver

Oracle WebLogic comes with a JDBC Driver for the Oracle database. For PostgreSQL, SQL Server and the databases accessed from xDM Dashboards or xDM Discovery, you must add the driver file to the classpath variable of the WebLogic Server.