Table of Contents

Welcome to Semarchy xDM.
This guide contains information about upgrading Semarchy xDM from previous releases.

Preface

Overview

Using this guide, you will learn how to plan and perform the upgrade of Semarchy xDM for development and production environments.

Audience

This document is intended for administrators interested in upgrading the Semarchy xDM platform from previous versions.

To discover Semarchy xDM, you can watch our tutorials.
The Semarchy xDM Documentation Library, including the development, administration and installation guides is available online.

Document Conventions

This document uses the following formatting conventions:

ConventionMeaning

boldface

Boldface type indicates graphical user interface elements associated with an action, or a product specific term or concept.

italic

Italic type indicates special emphasis or placeholder variable that you need to provide.

monospace

Monospace type indicates code example, text or commands that you enter.

Other Semarchy Resources

In addition to the product manuals, Semarchy provides other resources available on its web site: http://www.semarchy.com.

Obtaining Help

There are many ways to access the Semarchy Technical Support. You can call or email our global Technical Support Center (support@semarchy.com). For more information, see http://www.semarchy.com.

Feedback

We welcome your comments and suggestions on the quality and usefulness of this documentation.
If you find any error or have any suggestion for improvement, please mail support@semarchy.com and indicate the title of the documentation along with the chapter, section, and page number, if available. Please let us know if you want a reply.

Planning the Upgrade

Before the Upgrade

Documentation Review

Before starting the upgrade, you should review the following documents:

  • The Semarchy xDM Release Notes provides the latest information about the Semarchy xDM Release, including new features and bug fixes.
  • The Semarchy xDM Installation Guide provides the procedures for installing and configuring Semarchy xDM. In this guide, you should review the system requirements for this new release.
  • Review the major changes that may impact your upgrade, as well as the required actions listed in Pre-Upgrade Actions. Make sure to review these before starting the upgrade process.
  • Depending on your current version and the upgrade version, some actions may be required after the upgrade process. Review these Post-Upgrade Actions before starting the upgrade process.

Understanding Major, Minor and Patch Versions

Version numbers in Semarchy xDM are expressed in the following format: <major_version>.<minor_version>.<patch_version>. For example, version 1.3.2.

The upgrade may be a major upgrade, a minor upgrade or a patch. The differences between the old version number and the new version number define the type of upgrade.

  • A Major Upgrade takes place as soon as the major version differs. Major versions include major feature changes, and typically require repository and data locations upgrade. For example, upgrading from 1.3.2 to 2.0.1 is a major version upgrade.
  • A Minor Upgrade takes place when first digit remains the same but the second differs. Minor versions include minor feature changes, and typically require repository and data locations upgrade. For example, upgrading from 2.0.3 to 2.1.0 is a minor version upgrade.
  • A Patch is applied when the patch version only differs. Patches do not require repository and data location upgrade. For example, upgrading from 2.0.0 to 2.0.3 is patching.

Although the process is the same for all types of upgrade, be aware that the repository and data location upgrade steps may be skipped for patches.

Unless specified otherwise, the upgrade path between two version of Semarchy xDM is direct. You do not need to install intermediate versions.

Select the Upgrade Path

The upgrade supports two paths:

  • In-Place Upgrade consists in installing the new version of Semarchy xDM in place of the previous version and upgrading the existing repositories. With this method, you can revert the environment to its original state by restoring the backup of the server folders and database schemas.
  • Out-of-Place Upgrade consists in replicating the existing Semarchy xDM environment and upgrading the copy. The original environment remains unchanged.

You may choose one of the other path for your infrastructure. This choice depends on your IT infrastructure policies for upgrading.

Pre-Upgrade Actions

The following changes may impact your configuration, please review them carefully.

Upgrading from versions prior to 5.1.0

New License Management

Semarchy xDM introduces in this release a new system to manage licenses. This new license system simplifies the management of licenses and xDM instances at an enterprise level. With the license system, new or upgraded xDM instances run during a grace period, during which they must be declared and activated.

The new license management applies to newly created xDM 5.1 instances, as well as instances upgraded to version 5.1.

Recommended Action
Before starting the upgrade, administrators should plan the upgrade of their various environments with the following facts in mind:

  • The repository upgrade process erases the existing license key in xDM instances when upgrading from a version prior to version 5.1.0.
  • Such instances have a 7 days grace period during which they work normally. After that period, they will stop working.
  • Before the grace period is over, administrators should activate the instance, using the instruction described in the Managing the License section in the Semarchy xDM Administration Guide.

Our global Technical Support Center (support@semarchy.com) team is available for assistance during the upgrade process and to help planning the license change.

Browser Support

Semarchy xDM discontinues the support for old web browsers. The updated list of supported web browser is available in the Client Requirements section of Semarchy xDM Installation Guide.

Recommended Action
Administrators should inform all Semarchy xDM users of the new client requirements.

Oracle JDBC Datasource Configuration

Starting with this version, the JDBC connections to an Oracle Database hosting a repository or data location must be made with the oracle.jdbc.J2EE13Compliant property set to true.

When this option is not set, errors such as the following one will be raised in the application log.

com.semarchy.mdm.runtime.data.InvalidDataAccessResourceUsageException: java.lang.RuntimeException: Unexpected DB value .... (Class oracle.sql.TIMESTAMP for logicalType TIMESTAMP)

Required Actions

Administrators should review their datasources configuration and make sure to have them configured with this property.
For example, for a Tomcat datasource, the resource must contain the following:

<Resource
  name="...
  ...
  connectionProperties="oracle.jdbc.J2EE13Compliant=true"
  ...
/>

New Indexes on MD Tables

The data location upgrade in version 5.1.0 creates new performance indexes for all the MD_xxx tables, using the B_SUGGMERGEID column.

For performance optimization, database administrators may have already created manually such indexes, which may enter in conflict with the new ones, created during the data location upgrade.

Required Actions

Administrators should check all MD_xxx tables in data locations for customized indexes manually created on the B_SUGGMERGEID column. If such indexes exist, they should be dropped before the data location upgrade.

Upgrading from versions prior to 5.0.0

Infrastructure & System Requirements

The following changes in the system requirements may impact your setup:

  • Apache Tomcat 7.x-8.0.x and Oracle WebLogic 12c Release 1 (12.1.x) application servers are no longer supported
  • Oracle Database version 10g is not longer supported for the repository and data locations.
  • Oracle JDBC driver version is updated to version 12.2.x (ojdbc8.jar).
  • PostgreSQL JDBC driver version is updated to version 42.2.5.

Other changes to the Semarchy xDM web application:

  • The web application uses JEE 7
  • Support for Java 11.

Recommended Action

  • Review the system requirements in the Semarchy xDM Installation Guide.
  • If your infrastructure uses a java machine, database or application server version that is no longer supported, make sure to plan for an upgrade of your infrastructure prior to upgrading to this release.
  • Make sure to upgrade older driver versions to the newer ones.
  • Review the installation guide to review possible changes required to switch to Java 11.

Upgrading from versions prior to 4.0.0

This section applies to all installations upgrading from a version prior to version 4.0.0

Infrastructure & System Requirements

The following changes in the system requirements may impact your setup:

  • Semarchy xDM now requires Java 8. A Java Runtime Environment (JRE) or Development Kit (JDK) version 1.8.0 or above is now required.
  • The following application servers are no longer supported as they do not support Java 8.
    • Glassfish 3.x (4.x is the minimum required version)
    • JBoss 7.x (Wildfly 8 is the minimum required version)

Required Action
If your current infrastructure does not meet the new requirements, upgrade your infrastructure prior to upgrading Semarchy xDM.

Browser Support

Semarchy xDM discontinues the support for old web browsers. The updated list of supported web browser is available in the Client Requirements section of Semarchy xDM Installation Guide.

Recommended Action
Administrators should inform all Semarchy xDM users of the new client requirements.

SOAP Web Services Decommission

SOAP web services available in previous versions of Semarchy xDM are decommissioned in this release. A new REST API is made available for new users, providing similar integration capabilities for querying data in the hub and publishing data into the MDM hub. Note that certain features available in the SOAP Web Services are not available in the REST API, such as workflow management and administrative services.

Recommended Action
Existing users relying on the SOAP Web Services, and willing to continue using them temporarily before moving to the new REST API should contact our global Technical Support Center (support@semarchy.com).

Pulse Dashboard Not Unavailable

This version does not include the Pulse dashboards in the generated application. Pulse is replaced by Semarchy xDM Discovery for profiling and Semarchy xDM Dashboards for metrics.

Recommended Action
Existing users currently using Pulse should contact our global Technical Support Center (support@semarchy.com) for more information about these new components.

Data Edition Decommission

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provide access to the current state of the data in the hub, based on the deployed model edition.

The upgrade process prunes the data in closed data editions from the data locations. Only data from the latest data edition is preserved in each data location.

Recommended Action
Existing users willing to archive the contents from previous data editions should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.

Error Storage

Error storage has been changed in this release. The golden errors and source error tables no longer store the entire history of errors, but only the latest error instances.

The data location upgrade process prunes the history of errors on golden and source records, preserving only the latest error instances.

Recommended Action
Existing users willing to preserve the history of errors should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.

Upgrading

This chapter describes the various steps of the upgrade process.

Stop All Server Instances and Connections

Before upgrading, you must stop all applications and user connections to Semarchy xDM. During the upgrade:

  • No application should access the Semarchy xDM services and APIs.
  • No user should access the Semarchy user interfaces for design-time or data management operations.
  • No user or application should access the repository or data locations’ database schemas.

You should contact your database administrator and middleware administrator to make sure that the unavailability of Semarchy xDM during the upgrade process is correctly managed.

Stop all Semarchy xDM Servers. This is done either by:

  • Stopping Semarchy xDM from your application server administration console.
  • Stopping the application server if it is dedicated to Semarchy xDM only.

Backup Your Installation

Before upgrading, you should make sure that:

  • The schema(s) containing the Semarchy xDM repositories are backed up; Use the database utilities for performing such backup operations.
  • The schema(s) containing the Data Locations are backed up; Use the database utilities for performing such backup operations.
  • The Semarchy xDM installation folders in the application servers are backed up.
  • Any other relevant content (plug-in Jars, etc.) is also backed up.
Before going further, make sure that all the required backup are done, and that the entire Semarchy xDM environment is stopped and not accessed by any user or application.

Duplicate Database Schemas (Out of Place Upgrade)

If you have decided to perform an out-of-place upgrade, you must replicate (Copy) all the schemas (the repositories, data locations, etc.) using the database utilities.
The new product version, considered as a separate installation, will point to this copy of the schemas.

Use a clear naming convention for these new schemas to be able to identify them easily. For example, if the original schemas are called SEM_REPOSITORY, SEM_DLOC1, etc., the new one may be called SEM_2_2_REPOSITORY, SEM_2_2_DLOC1, etc.
With PostgreSQL, when replicating the databases, make sure that the search_path parameter is correctly set to $user, public, extensions in the database copies.

Install the New Server Version

Perform the following operation for every application server into which Semarchy xDM was deployed.

When installing a new version for in-place or out-of-place upgrade, pay attention to the application server and additional libraries versions installed in the current installation. This includes libraries related to security and authentication (role mapper, Waffle, etc.) These libraries are frequently compatible with a certain version of the application server and work with your current configuration. Do not systematically upgrade the application server and those libraries when upgrading Semarchy. If you decide to upgrade these libraries and the application server, make sure to review and test the compatibility and configuration.

Installing for In-Place Upgrade

To install the new version for in-place upgrade:

  1. Deploy the new Semarchy xDM Web Archive file on top of the previous version.
Refer to your application server documentation for more information about re-deploying applications on top of previous installations. The simple instructions for Apache Tomcat are provided below.

To perform an in-place upgrade for Apache Tomcat.

  1. Upload the new semarchy.war file to a temporary directory in the Tomcat server, for example /tmp (Unix/Linux) or c:\temp (windows).
  2. Make a copy of the existing configuration file (<tomcat>/conf/Catalina/localhost/semarchy.xml) to the same directory on the server.
  3. Connect to the Apache Tomcat Manager (http://<tomcat_host>:<tomcat_port>/manager/).
  4. In the applications list, note the Context Path of the existing Semarchy xDM application, and then click the Undeploy button to un-deploy it.
  5. In the Deploy directory of WAR file located on the server section, type in the context path and enter the file paths to the semarchy.xml and semarchy.war files.
    For example:
    • Context Path (required): /semarchy.
    • XML Configuration file URL: /tmp/semarchy.xml (Unix/Linux) or c:\temp\ semarchy.xml (windows)
    • WAR or Directory URL: /tmp/semarchy.war (Unix/Linux) or c:\temp\ semarchy.war (windows)
  6. Click the Deploy button.

Installing for Out-of-Place Upgrade

To install the new version for out-of-place upgrade:

  1. Deploy the new version Semarchy xDM Web Archive file as a new deployment, separated from the previous version.
When performing an out-of-place update, make sure to deploy the new version as a new deployment. If this new deployment takes place in the application server that already contains the previous version, make sure to change the deployment name during the process.
Refer to your application server documentation for more information about deployment.

Configure the New Server

Perform the following operation for each deployment of the new Semarchy xDM version.

Configuration for In-Place Upgrade

For in-place upgrade, the updated deployment reuses the configuration already in place. Check this configuration in the application server console or configuration files. The following points should be checked:

  • The datasources for the database schemas are set up and working.
  • The JavaMail Session servers are configured and working.
  • Security is configured.

Configuration for Out-of-Place Upgrade

To configure the new version in an out-of-place upgrade:

  1. Configure new JDBC datasources for the new deployment only. The datasources include those used to connect the Repository, Data Locations, etc. They should:
    • Point to the copies of the original schemas (e.g.: SEM_2_2_REPOSITORY, SEM_2_2_DLOC1).
    • Be local to the new deployment. Do not use global datasources
    • Use the same names as the datasources used by the old installation.
  2. Make sure that the existing JavaMail Session resource is configured or available for the new deployment.
  3. Make sure that the same security environment is configured for the new deployment.
Datasources should be configured locally for the deployment. For example, the repository datasource must be called SEMARCHY_REPOSITORY; Declaring this datasource at deployment-level avoids having multiple Semarchy xDM (possibly in different versions) connecting the same repository schema at the same time, which would lead to an erroneous behavior.
By using datasources local to the new deployment, you make sure that the new deployment connects exclusively to the copy of the schemas. Similarly, the old version should use local datasources to connect only the original schemas.

Start One New Server Instance

Perform this operation for a single new deployed version instance. If you have multiple repositories, you should repeat this operation for each repository.
Start your application server or connect to your application server and start the deployed version.

Upgrade the Repository

Repository upgrade is required only for major and minor versions. For patches, repository upgrade is not needed and the Repository Upgrade dialog does not appear when you connect.
The repository upgrade is an operation that cannot be undone. Make sure you have made a backup of the repository schema before proceeding.

Before Upgrading the Repository

The Oracle database has introduced in version 11g and 12c a new feature called Automatic Column Group Detection that automatically creates extended statistics for column groups. These extended statistics prevent the upgrade process to rename certain columns in the repository, causing an ORA-54032: column to be renamed is used in a virtual column expression error to appear during the repository upgrade process.

Before upgrading a repository hosted in an Oracle 11g or 12c instance, it is required to drop these statistics. They will be automatically re-created by the database as needed.

To drop unwanted extended statistics for repositories hosted in Oracle 11g and 12c:

  1. Connect to the database hosting the repository schema with a query tool (SQL Developer, SQL*Plus), using the repository schema user.
  2. Run the script provided below:
begin
  for xrec in (
    select
      TABLE_NAME, EXTENSION
    from
      USER_STAT_EXTENSIONS
    where
      TABLE_NAME like 'MTA%'
      and (
        EXTENSION like '%FROMEDITION%'
        or EXTENSION like '%TOEDITION%'
      )
  )
  loop

    DBMS_STATS.DROP_EXTENDED_STATS (
      ownname=>null,
      tabname=>xrec.table_name,
      extension=>TO_CHAR(xrec.extension)
    );

  end loop;
end;

Once the unwanted extended statistics are dropped, proceed with the repository upgrade.

Upgrading the Repository

To upgrade the repository:

  1. Connect to the new instance you have started, using a user account with the semarchyAdmin role. At the first connection, the Repository Upgrade dialog appears.
  2. Review the dialog.
  3. Click the checkbox and then click the Upgrade button.

The repository upgrade starts.

The repository upgrade process can only be triggered by a user with the semarchyAdmin role.

Upgrade the Data Locations

Data locations upgrade is required only for major and minor versions. For patches, data location upgrade is not needed.
The data location upgrade is an operation that cannot be undone. Make sure you have made a backup of the data location schemas before proceeding.
Users will be unable to access data locations that need an upgrade, or data locations being upgraded. Applications using these data locations will appear disabled on the Semarchy xDM welcome page. To reduce application downtime, it is recommended to plan data location upgrade after the repository upgrade.
It is not possible to deploy a model edition in a data location that requires an upgrade. If a data location requires an upgrade, Semarchy xDM will prevent all deployments in this data location.
Development data locations upgrade is possible only if the current model edition deployed is valid. Before upgrade a development data location make sure that the deployed model does not raise any design-time validation errors.
Development data locations upgrade is possible is the current deployed model corresponds to the latest model state. For such data locations, make sure that the latest version of the model is deployed before performing a data location upgrade.

Before Upgrading the Data Locations

Before upgrading a data location hosted in an Oracle 11g or 12c instance, it is required to drop the extended statistics that may prevent column renaming. For more information, refer to Before Upgrading the Repository.

To drop unwanted extended statistics:

  1. For each data location:
    1. Connect to the database hosting the data location schema with a query tool (SQL Developer, SQL*Plus), using the data location schema user.
    2. Run the script provided below:
begin
  for xrec in (
    select
      TABLE_NAME, EXTENSION
    from
      USER_STAT_EXTENSIONS
    where (
        TABLE_NAME like 'GI_%'
        or TABLE_NAME like 'GD_%'
        or TABLE_NAME like 'SD_%'
	or TABLE_NAME like 'MI_%'
      )
      and (
        EXTENSION like '%B_IS_VALIDATED%'
        or EXTENSION like '%B_STATUS%'
	or EXTENSION like '%B_HASSUGGMERGE%'
      )
  )
  loop

    DBMS_STATS.DROP_EXTENDED_STATS (
      ownname=>null,
      tabname=>xrec.table_name,
      extension=>TO_CHAR(xrec.extension)
    );

  end loop;
end;

Once the unwanted extended statistics are dropped, proceed with the upgrade.

Upgrading the Data Locations

To upgrade the data locations:

  1. Connect to the new instance you have started, using a user account with the semarchyAdmin role or having a role with the Application Management privilege.
  2. In the Welcome page, click Application Builder.
  3. In the Management perspective, a warning in the Management view indicates the list of data locations attached to this repository that require an update.
  4. Click the link under this list to perform the upgrade. The Data Location Upgrade dialog appears.
  5. Review the dialog.
  6. Click the checkbox and then click the Upgrade button.

The upgrade starts. When it completes, the window is refreshed and and warning no longer appears.

The data location upgrade process can only be triggered by a user with the semarchyAdmin role or with the Application Management privilege.

Restart All Server Instances

The upgrade is now finished.
You can restart all the Semarchy xDM instances in the new version, and open the access through the user interface, services, APIs and database tables.

Post-Upgrade Actions

This chapter describes the actions required after the upgrade process was complete. These actions depend on the current version of Semarchy xDM and the version you upgrade to.

If no action is indicated in this chapter for your versions, then no action is required after restarting the Semarchy xDM instances.
Certain actions performed on the model (for example, changes in validations, enrichers or any aspect of the certification process), which in essence are changes applied to your model, require that you re-deploy the model.

Upgrading from versions prior to 5.1.0

This section applies to all installations upgrading from a version prior to version 5.1.0.

New License Management

Semarchy xDM introduces in this release a new system to manage licenses.

The repository upgrade process erases the existing license key in xDM instances when upgrading from a version prior to version 5.1.0. Such instances have a 7 days grace period during which they work normally. After that period, they will stop working.

Required Action

Before the grace period is over, administrators should activate the instance, using the instruction described in the Managing the License section in the Semarchy xDM Administration Guide.

Our global Technical Support Center (support@semarchy.com) team is available for assistance during the upgrade process and to help planning the license change.

New Component: Semarchy xDM Discovery

Semarchy xDM Discovery enables data architects and business users to gather metrics and profile any source data to prepare a data management initiative.

When connecting the welcome page, users with the appropriate privileges will see a new xDM Discovery link to access the user interface for Semarchy xDM Discovery.

Suggested Actions
Teams willing to use Semarchy xDM Discovery in their projects should review the additional steps required to configure datasources for this component in the Semarchy xDM Installation Guide, and should grant the appropriate privilege to the platform roles, as explained in the Semarchy xDM Administration Guide, to access this feature.

Using Semarchy xDM Discovery is explained in the Semarchy xDM Discovery User’s Guide.

Select All in Actions

In previous releases, only the Delete, Confirm Duplicates, Review and Confirm Duplicates and Review Duplicates Suggestions actions had Enable All properties to execute the action on all records when no selection was made.

Starting with this release, this implicit select all is replaced by an explicit Select All capability that applies to all actions. With this feature, users can explicitly select all the filtered records in a collection, possibly unselect some of them and then start an action with the selection.

The number of records that can be processed at the same time may be limited at design-time for each action using a new optional Items Limit property, available in the Action Configuration properties. A warning is raised to the user when he triggers the action with a selection larger than this limit.

When running an action on a set of records selected by "Select All", the Condition defined on the action as well as the Model Privileges Grants still apply to automatically filter the records when the action starts. The user is warned if the action cannot process all the selected records due to the condition or a missing privilege.

Actions are automatically upgraded to use this new feature:

  • The Items Limit is set to null (no limit) for Export actions.
  • The Items Limit set to 1000 for Copy and Edit actions
  • The Items Limit is set to null (no limit) for existing Delete and duplicate management actions having Enable Delete All Records, Enable Confirm All or Enable Review All selected.
  • The Items Limit is set to 1000 for existing Delete and duplicate management actions having Enable Delete All Records, Enable Confirm All or Enable Review All de-selected.
  • The Enable Delete All Records, Enable Confirm All and Enable Review All properties are deprecated and no longer appears in the Application Builder. Designers willing to force actions to run only on single record selections only should use the Support Multiple Selection property.

Recommended Actions
Application designers should review the configuration of the actions after the upgrade and inform the application users of this functional change.

Split Duplicate Propagation

A new Split Duplicate Propagation is added to the references configuration to define how splitting records in the referenced entity affects the referencing records. This property applies when both the referenced and referencing entities are fuzzy matched.

Setting this property to Reset Matching forces rematching child records when their parent is split in a duplicate manager.

The child records are re-matched, ignoring their previous match groups and user decisions. The resulting golden records are re-distributed under their respective parent records. When re-matched, the referencing records are all re-merged into new golden records with new Golden IDs, and lose all their value overrides.

Suggested Actions
Model designers willing to use this feature should enable this option and re-deploy their model.

SQL Server Certified for Production

This release introduces full support for SQL Server, including production certification. The configuration of the SQL Server databases used for the repository and data locations need to be reviewed to make sure they comply with the requirements described in the Database-Specific Considerations section in the Semarchy xDM Installation Guide.

The QUOTED_IDENTIFIER and READ_COMMITTED_SNAPSHOT should be set to ON for these databases.

Required Actions
Database administrators should check the configuration of these parameters for the databases using the following command:

SELECT NAME, IS_QUOTED_IDENTIFIER_ON, IS_READ_COMMITTED_SNAPSHOT_ON FROM SYS.DATABASES

They should possibly modify the databases configuration using the following commands:

ALTER DATABASE <database_name> SET QUOTED_IDENTIFIER ON;
ALTER DATABASE <database_name> READ_COMMITTED_SNAPSHOT ON;

Upgrading from versions prior to 5.0.1

This section applies to all installations upgrading from a version prior to version 5.0.0.

Source Reference Validations

In previous releases, a child record referencing a parent record detected as erroneous during the pre-consolidation validation passed the certification process and was published with a null reference.

Starting with this release, when a child record references a parent record detected as erroneous during the pre-consolidation validation, then this child record is also rejected in the pre-consolidation validation by the reference validation. If a previous valid version of the parent record exists in the hub, then the child record is attached to that version instead. During post-consolidation validation, a reference is always considered valid if the referenced record exists (flagged as erroneous or not).

Required Actions
Model designers should review their model validations and references for possible impact on error management. They must re-deploy their models in order to update the jobs and apply this change in behavior. Those willing to preserve the old behavior must define the -Dmdm.sourceFkValidation.allowRefToNewRecordsWithErrors=true system property.

Upgrading from versions prior to 5.0.0

This section applies to all installations upgrading from a version prior to version 5.0.0.

New Component: Semarchy xDM Dashboards

Semarchy xDM Dashboards is the component of Semarchy xDM for designing visualizations and dashboards that create insight by combining your business data and the data stored and managed in the Semarchy xDM intelligent data hubs.

When connecting the welcome page, users with the appropriate privileges will see a new Dashboard Builder link to access the design-time user interface for dashboards.

Suggested Actions
Teams willing to use Semarchy xDM Dashboards in their projects should review the additional steps required to configure datasources for this component in the Semarchy xDM Installation Guide, and should grant the appropriate privilege to the platform roles, as explained in the Semarchy xDM Administration Guide, to access this feature.

Using Semarchy xDM Dashboards is explained in the xDM Dashboards Designer’s Guide.

User Interface Reorganization

The user interfaces available in Semarchy xDM have been reorganized:

The Workbench is now split into two user interfaces:

  • Configuration for all platform-level administrative tasks.
  • Application Builder, with two perspectives:
    • Design for model design.
    • Management for model editions and deployment/data locations management tasks.

A new Dashboard Builder user interface is available for designing dashboard applications.

Suggested Actions
Team with documented processes that use the previous platform user interface organization should review their documentation accordingly.

Platform Privilege Changes

The platform-level privileges have been reorganized to match the user interfaces reorganization in the Semarchy xDM platform as well as the security best practices.

The roles defined in previous releases are automatically upgraded to these new privileges, using the following rules:

  • Roles with the old Data Location privilege are granted the new Application Management privilege.
  • Roles with the old Model Design privilege are granted the new Application Design privilege.
  • Roles with one of the old Plug-ins Administration or Logging Configuration privileges are granted the new Platform Admin privilege.
  • Roles with one of the old Job and Job Log Administration or Execution Engine privileges are granted the new Application Management privilege.

Recommended Actions
Administrators should review the new privilege grants and make sure they meet their security requirements and best practices.

Model Privilege Changes

In previous releases of Semarchy xDM, roles with a model privilege grant with the Grant full access to the model selected had also access to the REST API, as if the Grant access to integration web services was selected too.

Starting with this release, Grant full access to the model does not automatically grants access to the REST API, and the Grant access to integration web services must be explicitly selected in the model privilege grant.
The upgrade process automatically takes care of checking this option for all model privilege grants with Grant full access to the model.

Recommended Actions
Model designers should review the model privilege grants and update as needed the access to the integration API.

Lookup enricher multiple outputs support

The Semarchy Lookup Enricher supports multiple outputs, which changes its configuration. The configuration of plug-in enrichers using this component is automatically upgraded.

For more information, review the Semarchy Lookup Enricher in the Semarchy xDM Plug-in Reference Guide.

Required Actions
Designers using this enricher in their models must re-deploy their model to update their job for the upgraded configuration.

REST API record update change

In previous releases, publishing with the REST API a record update for a fuzzy and id matched entity incorrectly set to null the attributes not present in the payload.

This behavior has been aligned (in bug fix MDM-7819) on the basic entity behavior. Only the attributes present in the payload are taken into account to update the existing record. Attributes absent from the payload keep the value of the original record.

Required Actions
Integration designers relying on this behavior in their integration flows should review and update these flows to take into account the behavior change.

Streaming Data Import

To enable large files imports, a streaming data import feature has been implemented for CSV and XLSX file formats. The XLS format for import is deprecated since it does not support streaming. XLS files are processed as CSV files.

Suggested Actions
Designers should inform the end-users of these changes.

REST API Fix & Stateless Mode

This release introduces a memory leak fix (MDM-8086) to the REST API and makes calls stateless to reduce resource consumption and increase security.

This fix requires on Tomcat instances to remove references to semarchy.commons.web.configuration_*.jar in the tomcat.util.scan.StandardJarScanFilter.jarsToSkip property found in the Catalina/catalina.properties file.

Required Actions
If upgrading a Tomcat instance on premises on in the cloud, review the catalina.properties file and remove, if needed, semarchy.commons.web.configuration_*.jar from the classes listed in the tomcat.util.scan.StandardJarScanFilter.jarsToSkip property.

Upgrading from versions prior to 4.4.0

This section applies to all installations upgrading from a version prior to version 4.4.0.

Search on Business View Child Collections

In this version, business views support searching and filtering every collection in the business view. In previous releases, filtering was only supported on the root collection of the business view. This new feature implies the following changes:

  • Search configuration no longer appears in the business view editor directly.
  • Each business entity of the business view (available through the Transitions tree-table) has a field named Search Configurations in its Display Properties to define the search methods available for the business entity.
  • The business view search configurations configured in previous releases are automatically moved by the upgrade process to the root business entity of each business view.

Suggested Actions
Designers willing to enable search in child collections should modify their business views accordingly.

New Date Datatype

In this version, a new Date datatype replaces Datetime to store dates with no time or timestamp. The Timestamp datatype remains to store date and time with timestamp information. The upgrade process automatically converts Datetime attributes to the new Date datatype in the model and data locations. As a consequence, data columns using the old Datetime type are automatically trimmed to store a Date.

Recommended Actions
Designers using attributes with a Datetime datatype to store date AND time information should modify their models prior to data location upgrade to change the datatype of these attributes from Date to Timestamp.

Integration specialists should review their data flows reading from or writing to the MDM hub and make sure they correctly take into account this datatype change.

The DATETIME type is deprecated for plug-in enrichers and validators, and the new DATE type is added..

Recommended Actions
Plug-in developers should review their plug-in code and replace DATETIME parameters, input and outputs by DATE or TIMESTAMP.

Upgrading from versions prior to 4.3.0

This section applies to all installations upgrading from a version prior to version 4.3.0.

Data Location Purge Schedules

In this version, the data purge is modified with the following changes:

  • The purge now takes into account all the data, including source data, source authoring data, deleted data traces, as well as data related to canceled external loads, direct authoring, duplicate manages or workflows.
    For all these, the data purge is performed according to the retention policy defined for the entities in the model.
  • A purge job is always generated when deploying a model, to handle at least the data purge for canceled external loads, direct authoring, duplicate manages or workflows.
  • Finally, the configuration of the purge schedule is moved to each data location.

In addition:

  • The upgrade process automatically creates under each data location a purge schedule equivalent to the old scheduled purges and then removes the old purge schedules.
  • For data locations for which no purge was previously configured a purge is scheduled every Sunday at midnight to handle the purge of the canceled loads.

Recommended Actions
Administrators should review the purge schedules and update them according to their production systems' requirements and organization.

Master Value Picking

A new option is available for matching entities, on duplicates managers and form steps. Enable Master Value Picking enables users overriding the value of consolidated golden records to select values from the master records in addition to being able to enter their own values.

When upgrading to this version, this option is automatically activated for duplicate managers, but not for form steps.

Recommended Actions
Designers should review that master value picking is relevant for their users, and modify the settings of the duplicate managers and steppers' form steps accordingly.

Hierarchy Tree Views

This version introduces a new visualization for business views. Hierarchies views provide a user-friendly way for business users to navigates hierarchies (of products, organization, cost centers, employees) with an intuitive user interface.

Hierarchies are not configured by default during the upgrade.

Recommended Actions
Designers willing to configure hierarchy tree views should review and update their business view configurations accordingly. See the Configuring the Hierarchy in a Business View section in the Semarchy xDM Developer’s Guide for more information about this feature.

Entity Historization

In this version, it is possible to enable automated historization to keep track of all changes on master and golden data for each entity. Applications can be configured to allow browsing the records history.

The upgrade does not automatically enable historization for the entities.

Suggested Actions
Model designers willing to enable this feature should alter the entities to support historization, and re-deploy their model. See the Historizing Entity Data section in the Semarchy xDM Developer’s Guide for more information about this feature.

ID and Fuzzy Matched Entity Delete

This version introduces the capability to perform soft and hard deletes for records in all entity types.

This option is not automatically enabled by the upgrade.

Suggested Actions
Model designers willing to enable delete for entities should modify their entities accordingly. See the Entity Records Hard and Soft Delete section in the Semarchy xDM Developer’s Guide for more information about this feature.

Authoring Master Records and Errors

Starting with version 4.3.0, ID and fuzzy matched entities support authoring (creating, editing, importing) source or erroneous records on behalf of publishers. This option is enabled when configuring authoring actions for ID and fuzzy matched entities.

Existing action sets are not modified by the upgrade process to support this feature.

Suggested Actions
Model designers willing to use this features should modify their models accordingly. See the Creating Action Sets and the Authoring Data in Applications sections for more information about this feature.

Integration

In this release, the following changes take place in the integration interfaces and should be reviewed by the integration specialists:

  • SDE and SDL views available via the REST API are respectively renamed SD4L and SD4LK.
  • the DeleteType built-in attribute, corresponding to the B_DELETETYPE column now tracks golden deleted when they lose all their master records with the LEGLESS_DELETE value.
  • New GH and MH tables and views store the golden and master history. New GH4B and MH4B views allow browsing golden and master data at the time a given batch was completed.
  • The type of the repository and data location columns storing boolean values is changed from VARCHAR(1) to CHAR(1) in Oracle.

Forms and Collections with UUID in Display Cards

After upgrading from a version prior to 4.3.0, any form or collection for an entity with a Primary Key attribute of type UUID, and using this UUID in the display card, fails to load with the following exception:

java.sql.SQLSyntaxErrorException: ORA-00932: inconsistent datatypes: expected CHAR got BINARY

Display cards created after versions 4.3.0 do not have this issue since the UUID is automatically wrapped in a SEM_UUID_TO_CHAR function call.

Required Actions
Teams facing this issue should modify their display cards to wrap the UUID in a SEM_UUID_TO_CHAR function call, and re-deploy the applications

Upgrading from versions prior to 4.2.0

This section applies to all installations upgrading from a version prior to version 4.2.0.

Data Entry Workflows

After a model upgrade, the data entry workflows that existed in version prior to version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The data authoring experience in version 4 supports two methods:

  • Direct Authoring, into which users edit and save records, without having to create a workflow. This method uses a guided authoring UI based on a design-time object called a Stepper.
  • Workflow Authoring, into which users start a Workflow with tasks and transitions. Each task uses a stepper for modifying the data.

Workflows and steppers are triggered using actions defined in action sets.

Refer to the Working with Applications chapter in the Semarchy xDM Developer’s Guide for more information about steppers, workflows and action sets.

Required Action
A wizard helps model designer upgrade their original workflows into steppers and workflows. Model designers must use this wizard to recover and convert their workflows.

Upgrading Data Entry Workflows

To upgrade data entry workflows:

  1. In the Semarchy Application Builder, open the model edition containing the workflows to upgrade.
  2. Right-click the model node in the Model Design view and then select Upgrade v3.x Data Entry Workflows…​.
  3. Review the notice in the About Workflow Upgrade step and then click Next.
  4. The Workflow Upgrade step shows the existing data entry workflows, with the list of business views that they use for data entry.
    • Select Upgrade for each workflow that you want to upgrade.
    • Select the Upgrade Target for each of these workflows. This target defines how the workflow will be upgraded:
      • Stepper: This option is available only for single-task workflows using a single business view and having no validation or enrichment configured on the cancel transition. The wizard converts the business view into a stepper, reproducing its hierarchical structure and data entry forms. It also adds to this stepper the enrichers and validations defined on the submit transition.
      • Workflow: This option is available for all workflows. The wizard converts each business view used into a task to a stepper, as in the previous option. In addition, it converts the workflow to the new format, using for each task the newly created steppers. Enrichment and validations are added to the steppers and/or workflow transitions to reproduce the original behavior.
    • Select Actions to automatically create, in the entity’s default actions set, the authoring actions (create, edit, etc.) corresponding to the options enabled in the original workflow. These actions use either the stepper or workflow created by the wizard.
    • Select Delete to delete the original workflow. Note that this action cannot be undone.
  5. Click Finish. The upgrade wizard converts the original workflows according to your choices.
You can only upgrade workflows in open model editions.
You can run again the workflow upgrade process on any workflow that has not been deleted by the wizard.
The Delete option deletes the selected data authoring workflows. This operation cannot be undone.
When all v3.x data entry workflows are converted and deleted, the File > Upgrade v3.x Data Entry Workflows menu option automatically disappears.

Recommended Actions
Review and test the new workflows and steppers after the upgrade to make sure the new behavior corresponds to the original functional requirements.

Duplicate Management Workflows

After a model upgrade, the duplicate management workflows that existed in version prior to version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The duplicate management experience in version 4 now uses a dedicated design-time artifact, the Duplicate Manager, duplicate management actions defined in action sets to review, confirm, merge or split duplicates and suggestions.

Refer to the Working with Applications chapter in the Semarchy xDM Developer’s Guide for more information about duplicate managers and action sets.

Required Action
A wizard helps you upgrade original workflows into duplicate managers after completing the model upgrade. Model designers must use this wizard to recover and convert their workflows.

Upgrading Duplicate Management Workflows

To upgrade duplicate management workflows:

  1. In the Semarchy Application Builder, open the model edition containing the workflows to upgrade.
  2. Right-click the model node in the Model Design view and then select Upgrade v3.x Duplicate Management Workflows…​.
  3. Review the notice in the About Workflow Upgrade step and then click Next.
  4. The Workflow Upgrade step shows the existing duplicate management workflows.
    • Select Upgrade for each workflow that you want to upgrade. Each workflow upgrades to a duplicate manager, using the default display card, form, and collection from the entity.
    • Select Delete to delete the original workflow. Note that this action cannot be undone.
    • Select Actions to automatically create, in the entity’s default actions set, the duplicate management actions using the duplicate manager created by the wizard.
  5. Click Finish. The upgrade wizard converts the original workflows according to your choices.
You can only upgrade workflows in open model editions.
You can run again the workflow upgrade process on any workflow that has not been deleted by the wizard.
The Delete option deletes the selected duplicate management workflows. This operation cannot be undone.
When all v3.x duplicate management workflows are converted and deleted, the File > Upgrade v3.x Duplicate Management Workflows menu option automatically disappears.

Recommended Actions
Review and test the new duplicate management actions after the upgrade to make sure the new behavior corresponds to the original functional requirements.

Survivorship and Override for Matching Entities

ID and fuzzy matched entities (also known as UDPK and SDPK entities) now support overriding data values on golden records. When data authoring operations are performed in these entities, the data authored (Authoring Data) by the users is clearly separated from the data loaded from external publishers (Source Data).

The consolidators have evolved into Survivorship rules, and support two phases:

  • Consolidation (using a Consolidation Rule), merge the data loaded from the publishers in the SD_ tables.
  • Override (using an Override Rule) apply possible data overrides authored by users on top of the consolidated data.

The survivorship rules allow implementing simply the pattern into which users fix consolidated values and/or create directly golden records in matching entities, without having to rely on matching and consolidation rules. There as now used only for the data loaded from external publishers (Source Data)

For more information about survivorship and override, refer to the Survivorship and the Certification Process sections in the Semarchy xDM Developer’s Guide.
Consolidators Upgrade

The upgrade process automatically performs the following changes in your models:

  • Consolidators upgrade to Survivorship Rules:
    • For Record-Level Consolidators, a single default survivorship rule is created with a consolidation rule corresponding to the original consolidator.
    • For Field-Level Consolidators, one survivorship rule is created per field, with a consolidation corresponding to the field’s original consolidator.
    • For all the upgraded survivorship rule, the override strategy is set by default to No Override.
  • Consolidation strategies upgrade:
    • The Any Value consolidation strategy is converted to a Custom Ranking strategy with no Raking Expression.
    • Other strategies remain unchanged.
  • Additional Changes
    • An additional Master ID Survivorship Rule is automatically created to consolidate the master record IDs into the golden record, with the Custom Raking consolidation strategy and no raking expression. This corresponds to Any Value.

Recommended Actions
Model designers should review the new survivorship rules and modify them according to their functional requirements, possibly adding override rules.

They should also consider:

  • Aggregating multiple single-attribute rules into multi-attribute rules, for attributes sharing the same consolidation strategy and sharing the same override strategy.
    Keep in mind that attributes within the same survivorship rule are always overridden simultaneously. As a consequence, an override on one of their attributes causes an override on all of them.
  • Creating a default survivorship per entity that applies to all attributes not taken into account by other rules.
Changes for the Certification Process

Publishers dedicated to data entry are no longer supported. The authoring data is already separated from the publisher data. These publishers should no longer be used in any of the rules involved in the certification process.

Required Action
Model designers should review and re-design matchers and consolidators handling a combination of consolidation and authoring. They should split the logic into the two elements of the survivorship rules:

  • Consolidation Rules must only handle data flows consolidated from the publishers
  • Override Rules must only handle user changes performed after the consolidation phase.

Data authored by users are not enriched or validated for ID and fuzzy matching entities as part of the certification process. These records should be enriched and validated as part of the steppers into which users author the data.

Required Action
Model designers should review the enrichers and validations applied to data from data entry publishers, and modify the steppers used for authoring this data in order to enforce enrichment and validation at this level. For example, when the stepper completes.

Changes for Data Integration
  • SD_ tables now only contain source data from application publishers and should only be used in that sense. New SA_ tables contain records created for data authoring and overrides performed on golden data.

Required Action
Integration specialists should review data integration flows intended to publish data on behalf of users (using a data entry publisher), and consider publishing data in the SA_ tables instead of the SD_ tables.

Documentation Property

In this version, a Documentation property is added to entity attributes, complex types' definition attributes, references, matching and survivorship rules. The documentation property supports the Markdown syntax for rich text. It provides detailed information to end-users using the generated application.

The Description property is not only used for describing/documenting model artifacts for model designers. Descriptions are no longer visible to end-users.

The documentation of an attribute/reference is used by default in all forms fields based on this attribute/reference. It is possible to override this inherited documentation in the form field.

The upgrade process automatically copies the content of the Description properties (which was visible to end-users before) to the Documentation property.

Recommended Actions
Model designers should review the documentation property on entity attributes, complex types' definition attributes, references, matching and survivorship rules, possibly enhance it with markdown content. They should also review field documentation and possibly override the value inherited from the entity attributes.

Other Changes

Changes for Data Integration

Designers and integration specialists should be aware of the following table structure breaking changes:

  • Basic entities' SD_ and SE_ tables are renamed SA_ and AE_. Their structures remain unchanged.
  • GI_ tables no longer have a B_ERROR_STATUS column.

Required Action
Model designers and integration specialists must review all components, scripts or data integration flows using these tables and columns and update them accordingly

Upgrading from versions prior to 4.0.0

This section applies to all installations upgrading from a version prior to version 4.0.0

Data Editions Decommission

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provide access to the current state of the data in the hub, based on the deployed model edition.

Changes for Deployment
  • The Data Editions view is renamed Data Locations and reorganized for a single deployed model and single data edition.
  • The deployment process for models and the management operations for data locations is simplified:
    • All operations related to data branch and data editions disappear.
    • Only one model edition is deployed at a time in a data location, and only data edition relies on this model.
    • All model edition operations on data location, that is installing a new model edition, updating an open model edition, or switching to an old model edition are replaced by a single Deploy Model Edition operation.
    • The Set Status to Maintenance and Set Status to Ready on the data location replace the Set Data Edition Status to Maintenance and Set Data Edition Status Back to Open.
  • Successive model edition deployments now appear in the Data Location view in a Deployment History node, showing the integration, installation and purge jobs deployed with the model edition.

Recommended Actions
Production teams should review and update their deployment process instructions to match the simplified process and administration console changes.

Changes for Data Locations
Existing data in closed data editions is removed from the data locations. Only data from the latest data edition is preserved in each data location.

Required Action
Model designers and integration specialists should review any component that accesses or uses data using data editions, taking into account the removal of all previous data editions.

Changes for the Certification Process
  • SemQL still supports the FromEdition, ToEdition and BranchID built-in attributes in existing clauses, but they are made deprecated, and will always return BranchID=0, FromEdition=0, and ToEdition=null.
  • The :V_DATABRANCH and :V_DATAEDITION built-in platform variables remain but are deprecated and both return 0.
  • The DataEdition Job Notification property is deprecated and returns the default value 0.

Recommended Actions
Designers should review their SemQL clauses and remove deprecated built-in attributes.

Changes for Data Integration

The B_BRANCHID, B_FROMEDITION, B_TOEDITION columns are dropped from all tables.

Required Action
Model designers and integration specialists should review any component that accesses or uses the data location tables in SQL, and remove any use of these columns: Integration jobs, SQL statements, PL/SQL functions called from SemQL, etc.

The PL/SQL INTEGRATION_LOAD integration package remains backward compatible. New versions of the functions not using data branch and edition are added to and replace the old ones.

Recommended Action
Integration specialists should review any component that uses these functions, and remove deprecated parameters.

For more information about table structures and the integration package, refer to the Semarchy xDM Integration Guide.

Post-Consolidation Validations

Post-consolidation validations, running on consolidated data, are no longer removing golden records. They only flag these records as erroneous, yet keeping them as golden records.

Changes for Applications

Golden data now includes records failing post-consolidation validations. A new ErrorStatus built-in attribute indicates whether the golden record has passed successfully or not validations. Possible values include:

  • VALID if the record has no error.
  • ERROR if the record has errors
  • a <NULL> value also indicates a record with no error.  

Required Action
Model designers should review their business views or queries to filter (using the ErrorStatus = 'VALID' OR ErrorStatus IS NULL SemQL clause) golden data and exclude golden records failing post-consolidation validation if such validations are used.

Changes for Data Integration

Designers and integration specialists should be aware of the following table structure breaking changes:

  • B_STATUS column is removed from the GI_ and MI_ tables.
  • A new B_ERROR_STATUS column is added to the GD_ table to flag the golden records failing a post-consolidation validation. Possible values include:
    • VALID if the record has no error.
    • ERROR if the record has errors
    • a <NULL> value also indicates a record with no error.  

Required Action
Integration specialists should now assume that the GD_ tables must be filtered to only read the valid golden records (where B_ERROR_STATUS = 'VALID' or B_ERROR_STATUS IS NULL).

Error Storage

The storage of errors has changed in this release:

  • The error tables (SE_ and GE_) no longer host the entire erroneous record, but only the list of errors with a reference to the erroneous records (which data is stored in the SD_ and GD_ tables).
  • Both SD_ and GD_ tables now have a B_ERROR_STATUS flag indicating the source or golden record status. Possible values are:
    • VALID if the record has no errors,
    • ERROR if the record has errors,
    • RECYCLED for source records only, if the record was recycled and considered valid
    • OBSOLETE_ERROR for source records only, if this record had errors but a newer version of the record fixed them.
    • a <NULL> value also indicates a record with no error.  
  • The B_STATUS columns are renamed B_ERROR_STATUS.
  • The SE_ and GE_ tables no longer store the entire history of errors, but only the latest error instances.

Required Action
Integration specialists should review their error processing flows or scripts and respectively join data from the SE/SD tables (on B_PUBID, B_SOURCEID and B_LOADID) and GE/GD tables (on the Golden ID) in order to retrieve erroneous data along with the errors. They should also take into account the fact that the error tables only store the latest error instances.

Applications Upgrade

The structure and the content of the applications and application components have considerably changed between versions 3.x and 4.x. This section details how the repository upgrade converts original artifacts into new ones.

Display Cards

Display Cards replace the Display Name for entities.

The upgrade process automatically creates a default display card from the display name, reproducing in the primary text the concatenated string of the display name.

Refer to the Display Cards section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should enhance this display card, possibly configuring a secondary text, an avatar and/or an image.

Collections

Collections replace Table Views for entities.

The upgrade process automatically upgrades all table views to collection supporting only a table view. The tables columns are upgraded to the new version 4 components with a configuration reproducing the original appearance of the table.

Refer to the Collections section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should enhance the upgraded collections, possibly configuring a display card, and alternate list and grid views. They should also review the resulting tables and change the columns' configuration to improve the appearance of their collections.

Form Views

Forms replace Forms Views for entities. In addition, forms now support a new layouting system (using the flex model), multiple tables, embedded collection and richer components.

The upgrade process automatically upgrades all form views to forms with a single tab. The form fields are upgraded to the new version 4 components with a configuration reproducing the original appearance.

The upgrade process preserves the forms organization into sections, but does not preserve the layout of fields and sections in the form.
Refer to the Forms section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should review and enhance the upgraded forms, fixing the layout and splitting long forms into multiple tabs. They should also review the field components' configuration to improve the appearance of their forms.

Business Views

Business Objects and Business Object Views are now merged into a single Business View design-time object, moved under the entity. This new artifact supports the features of both business objects and business object views.

The upgrade process automatically converts business objects and business object views to business views, recreating the references to the upgraded forms and collections.

Refer to the Business Views section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should review and enhance the upgraded business views. They can, for example, configure transitions for the embedded collections created when improving the upgraded forms.

Applications

Applications are now organized with folders and actions (such as browse, inbox, etc.) and a navigation drawer containing shortcuts to certain actions.

The upgrade process automatically upgrades the original application organization and re-creates the folder and actions structure to provide similar features.

Refer to the Application Actions and Folders section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should review and enhance the experience in the upgraded applications by adding icons, images, and colors to the actions created, and by adding advanced actions such as error browsing or record creation.

Upgrading from versions prior to 3.3.0

This section applies to all installations upgrading from a version prior to version 3.3.0.

Incoming Records Not Merging Singleton Golden Records

Customers who have upgraded to version 3.2 prior to version 3.2.4 may experience an issue with singleton golden records (for fuzzy matching entities) already in the hub not being merged with incoming records that match them. This issue is caused by a defect in the upgrade mechanism.

Required Action
Customers upgrading from version 3.2.0 through 3.2.4 should refer to the Knowledge Base Article that explains how to fix an existing upgrade.

New Reserved Keywords in SemQL

New capabilities added to the SemQL language introduce the following reserved keywords:

  • HAS: This token is used as an equivalent to the HAVE keyword as in any <child_entity_role> has ( <condition_on_child_entity> ).
  • NULLS FIRST, LAST: These keywords allow for NULLS FIRST and NULLS LAST clauses in custom ranking consolidators.

Recommended Action
After the upgrade, model designers should review and validate their models and make sure that no physical object is named using the new reserved keywords.

Integration Job Execution Scheme

The Integration Job changes from a "by entity" to a "by Phase" execution scheme. This means that instead of processing entirely each entity, for example, Customers then Contact, it now processes entirely each phase (Enrichment for all entities, then validation for all entities, etc). This change does not impact the outcome of the integration job.

Required Action
No action is required. Possible database operations started outside of Semarchy and assuming specific step sequencing, should be reviewed and updated as needed.

Upgrading from versions prior to 3.2.0

This section applies to all installations upgrading from a version prior to version 3.3.0.

Matching Process

In versions prior to 3.2, a matcher includes a single matching condition.
Starting with version 3.2, a matcher is composed of several matching rules and supports scoring for automated merge and confirmation. To reflect this new matcher and provide performance enhancements, the data certification job was re-designed and the data location structures have been modified.

Matching Rule Upgrade

When a model is upgraded, existing matchers are converted to the new matchers in order to deliver the same matching behavior:

  • The existing matcher for each entity is converted to a matcher with a single matching rule called Default, with a matching score of 100. This rule contains the old matcher’s binning expressions and matching condition.
  • The Merge Policy is configured with all the thresholds set to 0, so that all duplicate groups detected automatically merge.
  • The Auto-Confirm Policy is configured as follows:
    • Auto-confirm golden records threshold is set to 100. Merged duplicate groups are never automatically confirmed.
    • Auto-confirm singletons is checked. Singletons are automatically confirmed.

Recommended Action
After the upgrade, model designers should review their matchers to benefit from the new capabilities and enable automated merge and confirmation for the business users and data stewards.

Data Location Upgrade

When a data location is upgraded, the data structures are modified to support new matching rules, scoring, suggested merges and auto-confirmation. See the Table Structures section in the Semarchy xDM Integration Guide for more information about these tables.

Recommended Action
Customers consuming golden and master records from GD and MD tables and using duplicate management-related columns should review the new structures and propagate changes in their data integration flows.

Data Administrators should gather statistics on the data location schemas to make sure that subsequent processing use statistics updated with the updated structure and values.

In addition, the existing records are upgraded with the following rules:

  • Golden records keep their existing IDs and confirmed/validated state.
  • Confirmed or unconfirmed golden records keep their status. This status is moved from the B_IS_VALIDATED column in GD tables to the new B_ISCONFIRMED column, and a new B_CONFIRMATIONSTATUS column provides a more detailed confirmation status in MD and GD tables.
  • Partially confirmed golden records (that is golden records with only part of their master records confirmed) are upgraded to Not Confirmed (B_ISCONFIRMED=0 and B_CONFIRMATIONSTATUS=NOT_CONFIRMED).
  • The existing duplicate management workflows remain, but the duplicates listed in these workflows disappear.
  • Golden records split and confirmed by the user are upgraded in an exclusion group.
  • A confidence score is automatically computed.

Recommended Action
Administrators and data stewards should review the upgrade rules listed above and should test the upgrade prior to applying it to their production environment. After the upgrade, data stewards should review the confirmation state of the duplicates and execute duplicate management workflows to confirm duplicates as needed.

Customized Search Forms

In versions prior to 3.2, all the default built-in search methods (full text, advanced, by form and SemQL) were available in the business object views. Starting with 3.2, it is possible to create customized forms and define which of the search methods or customized forms are available in applications.

Recommended Action
After the upgrade, model designers should review and customize the search methods available in their applications.

Database Functions

Starting with version 3.2, it is possible to declare user-defined PL/SQL functions. By declaring these functions, designers make them available in the SemQL editor and avoid SemQL warnings raised by these functions.

Recommended Action
After the upgrade, model designers should declare all PL/SQL functions used in their SemQL clauses. By doing so, model validation will no longer raise warning for these functions.

Reserved Physical Column Names

Prior to version 3.2, the attributes' physical column names could take any value. Starting with version 3.2.0, it is no longer possible to define physical column names starting with B_ as they may collide with reserved column names.

Models with physical column names violating this rule no longer pass model validation and each column will raise a validation error such as Invalid physical name: <column_name> matches the reserved pattern B_*.

Required Action
Prior to the upgrade, administrators should refer to the knowledge base article that explains how to detect, rename and upgrade the offending attributes.

Upgrading from versions prior to 3.1.0

This section applies to all installations upgrading from a version prior to version 3.3.0.

Model Privilege Grants for SemarchyAdmin

In versions prior to 3.1, the semarchyAdmin role had a hardcoded full access to the data on all models.
Starting with version 3.1 the privilege grant for this role is no longer hardcoded and can be modified.

When a creating a new model, or upgrading an existing one, a model-level privilege grant is automatically created for the semarchyAdmin role with Grant full access to the model selected to keep the same level of privileges.

Recommended Action
After the upgrade, model designers may want to modify this privilege grant to reduce the privileges of the semarchyAdmin role on the data and prevent platform administrators from accessing data in the hub.

Model Privileges Grants for the Integration Services

In versions prior to 3.1, accessing the integration services required a role having platform-level read/write privileges to the data location. Starting with version 3.1, a specific option called Grant access to integration services appears in the model privilege grants to define whether a role can access this model’s data via the services.

This privilege applies to the SOAP and REST Service APIs.

When upgrading, roles that could access the integration services thanks to their privileges on the data locations will automatically have this option checked for the models deployed in the data location.

Recommended Action
After the upgrade, model designers may want to review and modify this privilege grant to prevent access to the integration services to certain roles.

Upgrading from versions prior to 3.0.0

Upgrading from a Semarchy xDM version before version 3.0 is not supported.