Post-Upgrade Actions

In version 2023.1.0, the Read-Only property available on form fields is replaced with Authoring Mode that allows model designers to define if each field is displayed using its editable, read-only, or browsing variant (see Form Fields Properties).

When migrating to 2023.1.0 or above, the Authoring Mode property of existing form fields is automatically initiated according to the following rules:

Additionally, the Read-Only property available on stepper form steps is replaced in version 2023.1.0 with Form Fields Mode. This property allows model designers to override the Authoring Mode for all the fields of a given step (see Form Steps Flow).

When migrating to 2023.1.0 or above, the Form Fields Mode property of existing stepper form steps is initiated according to the following rules:

Starting from this version, the logging in Semarchy xDM relies on Log4J 2, which is configured using an XML configuration format, in replacement of the key-value property pairs previously used. It is therefore required to migrate the existing startup and platform logging configurations to that new Log4J 2 format.

As mentioned in the pre-upgrade actions, after the upgrade, Semarchy xDM platform is configured with the Internal Identity Provider only and the administrator user that was created during the upgrade process.

As mentioned in the pre-upgrade actions, the datasources used by the Data locations, xDM Dashboard, xDM Discovery, the Variable Value Providers and the enrichment/validation Plugins are no longer configured at the application server level and must be defined in the Semarchy xDM platform as platform datasources.

The upgrade process automatically creates empty datasources for the components that refer to application server datasources. However, it does not recover the datasource configurations.

The referenced JNDI datasources are upgraded into empty platform datasources with a normalized name. The normalized name is derived from the JNDI name by performing the following operations:

For example, in a data location, a reference to a JNDI datasource named java:comp/env/jdbc/DATA_LOCATION_1 is upgraded to a platform datasource named DATA_LOCATION_1. That datasource is empty and must be configured by the administrator.

The following changes have been made to the REST API Authentication:

The Enable Import, Enable Import-Update and Restrict Import to Update properties available on workflow transitions, steppers, and actions are replaced by two properties:

The Batch Update Size properties of enricher plugins now automatically takes the value 1000 when left empty. This value provides better performances in most situations.

The Master Records' OldMatchGroupID attribute (B_OLDMATCHGRP column in the MI table) is deprecated.

With the new Integrated Authentication feature, the user and role tables (MTA_USER, MTA_USER_ROLES) stored in the repository have been moved and their structure has been changed. The new tables storing users and roles are named IDM_USER and IDM_USER_ROLE.

The following changes have occurred in the built-in variable:

This release changes how case sensitivity is handled for database functions declared in the model.

Starting with this release, you can declare the function names and schemas in any case. However, a model validation prevents having two functions with the same qualified name and a different case.

PostgreSQL and Oracle are case-insensitive for all function calls. PostgreSQL assumes that all object names are in lowercase, and Oracle assumes that all object names are in uppercase. The product behavior does not change for models based on these two databases.

An SQL Server database may be configured to handle object names in a case-sensitive manner. Starting with this release, Semarchy xDM assumes, for SQL Server, case-sensitivity for all function calls, including the SemQL built-in functions. Function calls performed in SemQL with a function name that does not match exactly the case of the built-in function definition (as it appears in the SemQL editor) now raise warnings in the model validation, since they may fail at run-time, depending on the database server configuration.

A new Melissa Plug-in for Semarchy xDM provides enrichers to fix and complete contact data for US/Canada using the Personator service and to validate international addresses in 240 countries using the Global Address Verification service.

Existing enrichers using the legacy Personator Enricher are automatically upgraded to use the new enricher as part of the automated model upgrade process.

The upgrade automatically configures the enricher to use the new plug-in capabilities, as described below:

The Matchcode Global Plug-in for Semarchy xDM, using GBGRoup Matchcode Global to provide an enricher for international postal addresses is removed in this release.

In xDM Discovery, the Row Sampling table property is deprecated and replaced by the Maximum Distinct Values Stored and Maximum Patterns Stored properties with default values of 50. The value set in Row Sampling is automatically copied to these new properties by the upgrade process.

Semarchy xDM introduces in this release a new system to manage licenses.

The repository upgrade process erases the existing license key in xDM instances when upgrading from a version before version 5.1.0. Such instances have a 7 days grace period during which they work normally. After that period, they will stop working.

Semarchy xDM Discovery enables data architects and business users to gather metrics and profile any source data to prepare a data management initiative.

When connecting the welcome page, users with the appropriate privileges will see a new xDM Discovery link to access the user interface for Semarchy xDM Discovery.

In previous releases, only the Delete, Confirm Duplicates, Review and Confirm Duplicates and Review Duplicates Suggestions actions had Enable All properties to execute the action on all records when no selection was made.

Starting with this release, this implicit select all is replaced by an explicit Select All capability that applies to all actions. With this feature, users can explicitly select all the filtered records in a collection, possibly unselect some of them and then start an action with the selection.

The number of records that can be processed at the same time may be limited at design-time for each action using a new optional Items Limit property, available in the Action Configuration properties. A warning is raised to the user when he triggers the action with a selection larger than this limit.

Actions are automatically upgraded to use this new feature:

A new Split Duplicate Propagation is added to the references configuration to define how splitting records in the referenced entity affects the referencing records. This property applies when both the referenced and referencing entities are fuzzy matched.

Setting this property to Reset Matching forces rematching child records when their parent is split in a duplicate manager.

The child records are re-matched, ignoring their previous match groups and user decisions. The resulting golden records are re-distributed under their respective parent records. When re-matched, the referencing records are all re-merged into new golden records with new Golden IDs and lose all their value overrides.

This release introduces full support for SQL Server, including production certification. The configuration of the SQL Server databases used for the repository and data locations need to be reviewed to make sure they comply with the requirements described in database-specific considerations.

The QUOTED_IDENTIFIER and READ_COMMITTED_SNAPSHOT should be set to ON for these databases.

In previous releases, a child record referencing a parent record detected as erroneous during the pre-consolidation validation passed the certification process and was published with a null reference.

Starting with this release, when a child record references a parent record detected as erroneous during the pre-consolidation validation, then this child record is also rejected in the pre-consolidation validation by the reference validation. If a previous valid version of the parent record exists in the hub, then the child record is attached to that version instead. During post-consolidation validation, a reference is always considered valid if the referenced record exists (flagged as erroneous or not).

Semarchy xDM Dashboard is the component of Semarchy xDM for designing visualizations and dashboards that create insight by combining your business data and the data stored and managed in the Semarchy xDM intelligent data hubs.

When connecting the welcome page, users with the appropriate privileges will see a new Dashboard Builder link to access the design-time user interface for dashboards.

The user interfaces available in Semarchy xDM have been reorganized:

The Workbench is now split into two user interfaces:

A new Dashboard Builder user interface is available for designing dashboard applications.

In previous releases of Semarchy xDM, roles with a model privilege grant with the Grant full access to the model selected had also access to the REST API, as if the Grant access to integration web services was selected too.

Starting with this release, Grant full access to the model does not automatically grant access to the REST API, and the Grant access to integration web services must be explicitly selected in the model privilege grant.
The upgrade process automatically takes care of checking this option for all model privilege grants with Grant full access to the model.

The Semarchy Lookup Enricher supports multiple outputs, which changes its configuration. The configuration of plug-in enrichers using this component is automatically upgraded.

In previous releases, publishing with the REST API a record update for a fuzzy and id matched entity incorrectly set to null the attributes not present in the payload.

This behavior has been aligned (in bug fix MDM-7819) on the basic entity behavior. Only the attributes present in the payload are taken into account to update the existing record. Attributes absent from the payload keep the value of the original record.

To enable large file imports, a streaming data import feature has been implemented for CSV and XLSX file formats. The XLS format for import is deprecated since it does not support streaming. XLS files are processed as CSV files.

This release introduces a memory leak fix (MDM-8086) to the REST API and makes calls stateless to reduce resource consumption and increase security.

This fix requires on Tomcat instances to remove references to semarchy.commons.web.configuration_*.jar in the tomcat.util.scan.StandardJarScanFilter.jarsToSkip property found in the Catalina/catalina.properties file.

In this version, business views support searching and filtering every collection in the business view. In previous releases, filtering was only supported on the root collection of the business view. This new feature implies the following changes:

In this version, a new Date datatype replaces Datetime to store dates with no time or timestamp. The Timestamp datatype remains to store date and time with timestamp information. The upgrade process automatically converts Datetime attributes to the new Date data type in the model and data locations. As a consequence, data columns using the old Datetime type are automatically trimmed to store a Date.

The DATETIME type is deprecated for plug-in enrichers and validators, and the new DATE type is added.

In this version, the data purge is modified with the following changes:

Besides:

A new option is available for matching entities, on duplicates managers and form steps. Enable Master Value Picking enables users overriding the value of consolidated golden records to select values from the master records in addition to being able to enter their own values.

When upgrading to this version, this option is automatically activated for duplicate managers, but not for form steps.

This version introduces a new visualization for business views. Hierarchies views provide a user-friendly way for business users to navigates hierarchies (of products, organization, cost centers, employees) with an intuitive user interface.

Hierarchies are not configured by default during the upgrade.

In this version, it is possible to enable automated historization to keep track of all changes on master and golden data for each entity. Applications can be configured to allow browsing the history of the records.

The upgrade does not automatically enable historization for the entities.

This version introduces the capability to perform soft and hard deletes for records in all entity types.

This option is not automatically enabled by the upgrade.

Starting with version 4.3.0, ID and fuzzy matched entities support authoring (creating, editing, importing) source or erroneous records on behalf of publishers. This option is enabled when configuring authoring actions for ID and fuzzy matched entities.

Existing action sets are not modified by the upgrade process to support this feature.

In this release, the following changes take place in the integration interfaces and should be reviewed by the integration specialists:

After upgrading from a version before 4.3.0, any form or collection for an entity with a Primary Key attribute of type UUID, and using this UUID in the display card, fails to load with the following exception:

Display cards created after version 4.3.0 do not have this issue since the UUID is automatically wrapped in a SEM_UUID_TO_CHAR function call.

After a model upgrade, the data entry workflows that existed in a version before version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The data authoring experience in version 4 supports two methods:

Workflows and steppers are triggered using actions defined in action sets.

After a model upgrade, the duplicate management workflows that existed in a version before version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The duplicate management experience in version 4 now uses a dedicated design-time artifact, the Duplicate Manager, duplicate management actions defined in action sets to review, confirm, merge or split duplicates and suggestions.

ID and fuzzy matched entities (also known as UDPK and SDPK entities) now support overriding data values on golden records. When data authoring operations are performed in these entities, the data authored (Authoring Data) by the users is separated from the data loaded from external publishers (Source Data).

The consolidators have evolved into Survivorship rules, and support two phases:

The survivorship rules allow implementing simply the pattern into which users fix consolidated values and/or create directly golden records in matching entities, without having to rely on matching and consolidation rules. There as now used only for the data loaded from external publishers (Source Data)

In this version, a Documentation property is added to entity attributes, complex types' definition attributes, references, matching, and survivorship rules. The documentation property supports the Markdown syntax for rich text. It provides detailed information to end-users using the generated application.

The Description property is not only used for describing/documenting model artifacts for model designers. Descriptions are no longer visible to end-users.

The documentation of an attribute/reference is used by default in all forms fields based on this attribute/reference. It is possible to override this inherited documentation in the form field.

The upgrade process automatically copies the content of the Description properties (which was visible to end-users before) to the Documentation property.

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provides access to the current state of the data in the hub, based on the deployed model edition.

Post-consolidation validations, running on consolidated data, are no longer removing golden records. They only flag these records as erroneous, yet keeping them as golden records.

The storage of errors has changed in this release:

The structure and the content of the applications and application components have considerably changed between versions 3.x and 4.x. This section details how the repository upgrade converts original artifacts into new ones.

Customers who have upgraded to version 3.2 before version 3.2.4 may experience an issue with singleton golden records (for fuzzy matching entities) already in the hub not being merged with incoming records that match them. This issue is caused by a defect in the upgrade mechanism.

New capabilities added to the SemQL language introduce the following reserved keywords:

The Integration Job changes from a "by entity" to a "by Phase" execution scheme. This means that instead of processing entirely each entity, for example, Customers then Contact, it now processes entirely each phase (Enrichment for all entities, then validation for all entities, etc). This change does not impact the outcome of the integration job.

In versions before 3.2, a matcher includes a single matching condition.
Starting with version 3.2, a matcher is composed of several matching rules and supports scoring for automated merge and confirmation. To reflect this new matcher and provide performance enhancements, the data certification job was re-designed and the data location structures have been modified.

In versions before 3.2, all the default built-in search methods (full text, advanced, by form, and SemQL) were available in the business object views. Starting with 3.2, it is possible to create customized forms and define which of the search methods or customized forms are available in applications.

Starting with version 3.2, it is possible to declare user-defined PL/SQL functions. By declaring these functions, designers make them available in the SemQL editor and avoid SemQL warnings raised by these functions.

Before version 3.2, the attributes' physical column names could take any value. Starting with version 3.2.0, it is no longer possible to define physical column names starting with B_ as they may collide with reserved column names.

Models with physical column names violating this rule no longer pass model validation and each column will raise a validation error such as Invalid physical name: <column_name> matches the reserved pattern B_*.

In versions before 3.1, the semarchyAdmin role had a hardcoded full access to the data on all models.
Starting with version 3.1 the privilege grant for this role is no longer hardcoded and can be modified.

When creating a new model, or upgrading an existing one, a model-level privilege grant is automatically created for the semarchyAdmin role with Grant full access to the model selected to keep the same level of privileges.

In versions before 3.1, accessing the integration services required a role having platform-level read/write privileges to the data location. Starting with version 3.1, a specific option called Grant access to integration services appears in the model privilege grants to define whether a role can access this model’s data via the services.

This privilege applies to the SOAP and REST Service APIs.

When upgrading, roles that could access the integration services thanks to their privileges on the data locations will automatically have this option checked for the models deployed in the data location.