Welcome to Semarchy xDM.
This guide contains information about upgrading Semarchy xDM from previous releases.

Preface

Audience

This document is intended for administrators interested in upgrading the Semarchy xDM platform from previous versions.

If you want to learn about MDM or discover Semarchy xDM, you can watch our tutorials.
The Semarchy xDM Documentation Library, including the development, administration and installation guides is available online.

Document Conventions

This document uses the following formatting conventions:

Convention Meaning

boldface

Boldface type indicates graphical user interface elements associated with an action, or a product specific term or concept.

italic

Italic type indicates special emphasis or placeholder variable that you need to provide.

monospace

Monospace type indicates code example, text or commands that you enter.

Other Semarchy Resources

In addition to the product manuals, Semarchy provides other resources available on its web site: http://www.semarchy.com.

Obtaining Help

There are many ways to access the Semarchy Technical Support. You can call or email our global Technical Support Center (support@semarchy.com). For more information, see http://www.semarchy.com.

Feedback

We welcome your comments and suggestions on the quality and usefulness of this documentation.
If you find any error or have any suggestion for improvement, please mail support@semarchy.com and indicate the title of the documentation along with the chapter, section, and page number, if available. Please let us know if you want a reply.

Overview

Using this guide, you will learn how to plan and perform the upgrade of Semarchy xDM for development and production environments.

Planning the Upgrade

Before the Upgrade

Documentation Review

Before starting the upgrade, you should review the following documents:

  • The Semarchy xDM Release Notes provides the latest information about the Semarchy xDM Release, including new features and bug fixes.

  • The Semarchy xDM Installation Guide provides the procedures for installing and configuring Semarchy xDM. In this guide, you should review the system requirements for this new release.

  • Review the major changes that may impact your upgrade, as well as the required actions listed in Pre-Upgrade Actions. Make sure to review these before starting the upgrade process.

  • Depending on your current version and the upgrade version, some actions may be required after the upgrade process. Review these Post-Upgrade Actions before starting the upgrade process.

Understanding Major, Minor and Patch Versions

Version numbers in Semarchy xDM are expressed in the following format: <major_version>.<minor_version>.<patch_version>. For example, version 1.3.2.

The upgrade may be a major upgrade, a minor upgrade or a patch. The differences between the old version number and the new version number define the type of upgrade.

  • A Major Upgrade takes place as soon as the major version differs. Major versions include major feature changes, and typically require repository and data locations upgrade. For example, upgrading from 1.3.2 to 2.0.1 is a major version upgrade.

  • A Minor Upgrade takes place when first digit remains the same but the second differs. Minor versions include minor feature changes, and typically require repository and data locations upgrade. For example, upgrading from 2.0.3 to 2.1.0 is a minor version upgrade.

  • A Patch is applied when the patch version only differs. Patches do not require repository and data location upgrade. For example, upgrading from 2.0.0 to 2.0.3 is patching.

Although the process is the same for all types of upgrade, be aware that the repository and data location upgrade steps may be skipped for patches.

Unless specified otherwise, the upgrade path between two version of Semarchy xDM is direct. You do not need to install intermediate versions.

Select the Upgrade Path

The upgrade supports two paths:

  • In-Place Upgrade consists in installing the new version of Semarchy xDM in place of the previous version and upgrading the existing repositories. With this method, you can revert the environment to its original state by restoring the backup of the application folders and database schemas.

  • Out-of-Place Upgrade consists in replicating the existing Semarchy xDM environment and upgrading the copy. The original environment remains unchanged.

You may choose one of the other path for your infrastructure. This choice depends on your IT infrastructure policies for upgrading.

Pre-Upgrade Actions

The following changes may impact your configuration, please review them carefully.

Upgrading from versions prior to 4.0.0

This section applies to all installations upgrading from a version prior to version 4.0.0

Infrastructure & System Requirements

The following changes in the system requirements may impact your setup:

  • Semarchy xDM now requires Java 8. A Java Runtime Environment (JRE) or Development Kit (JDK) version 1.8.0 or above is now required.

  • The following application servers are no longer supported as they do not support Java 8.

    • Glassfish 3.x (4.x is the minimum required version)

    • JBoss 7.x (Wildfly 8 is the minimum required version)

Required Action
If your current infrastructure does not meet the new requirements, upgrade your infrastructure prior to upgrading Semarchy xDM.

Browser Support

Semarchy xDM discontinues the support for old web browsers. The updated list of supported web browser is available in the Client Requirements section of Semarchy xDM Installation Guide.

Recommended Action
Administrators should inform all Semarchy xDM users of the new client requirements.

SOAP Web Services Decommission

SOAP web services available in previous versions of Semarchy xDM are decommissioned in this release. A new REST API is made available for new users, providing similar integration capabilities for querying data in the hub and publishing data into the MDM hub. Note that certain features available in the SOAP Web Services are not available in the REST API, such as workflow management and administrative services.

Recommended Action
Existing users relying on the SOAP Web Services, and willing to continue using them temporarily before moving to the new REST API should contact our global Technical Support Center (support@semarchy.com).

Pulse Dashboard Not Unavailable

This version does not include the Pulse dashboards in the generated application.

Recommended Action
Existing users currently using Pulse Dashboards should contact our global Technical Support Center (support@semarchy.com) for more information about Pulse.

Data Edition Decommission

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provide access to the current state of the data in the hub, based on the deployed model edition.

The upgrade process prunes the data in closed data editions from the data locations. Only data from the latest data edition is preserved in each data location.

Recommended Action
Existing users willing to archive the contents from previous data editions should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.

Error Storage

Error storage has been changed in this release. The golden errors and source error tables no longer store the entire history of errors, but only the latest error instances.

The data location upgrade process prunes the history of errors on golden and source records, preserving only the latest error instances.

Recommended Action
Existing users willing to preserve the history of errors should export this information from the data location tables using scripting or data integration flows, or keep a backup of the data location contents performed before the upgrade.

Upgrading

This chapter describes the various steps of the upgrade process.

Stop All Application Instances and Connections

Before upgrading, you must stop all applications and user connections to Semarchy xDM. During the upgrade:

  • No application should access the Semarchy xDM services and APIs.

  • No user should access the Semarchy Workbench for design-time or data management operations.

  • No user or application should access the repository or data locations’ database schemas.

You should contact your database administrator and middleware administrator to make sure that the unavailability of Semarchy xDM during the upgrade process is correctly managed.

Stop all Semarchy xDM Applications. This is done either by:

  • Stopping the Semarchy xDM application from your application server administration console.

  • Stopping the application server if it is dedicated to Semarchy xDM only.

Backup Your Installation

Before upgrading, you should make sure that:

  • The schema(s) containing the Semarchy xDM repositories are backed up; Use the Oracle Database utilities for performing such backup operations.

  • The schema(s) containing the Data Locations are backed up; Use the Oracle Database utilities for performing such backup operations.

  • The Semarchy xDM installation folders in the application servers are backed up.

  • Any other relevant content (plug-in Jars, etc.) is also backed up.

Before going further, make sure that all the required backup are done, and that the entire Semarchy xDM environment is stopped and not accessed by any user or application.

Duplicate Database Schemas (Out of Place Upgrade)

If you have decided to perform an out-of-place upgrade, you must replicate (Copy) all the schemas (the repositories, data locations, etc.) using the Oracle Database utilities.
The new product version, considered as a separate installation, will point to this copy of the schemas.

Use a clear naming convention for these new schemas to be able to identify them easily. For example, if the original schemas are called SEM_REPOSITORY, SEM_DLOC1, etc., the new one may be called SEM_2_2_REPOSITORY, SEM_2_2_DLOC1, etc.

Install the New Application Version

Perform the following operation for every application server into which the Semarchy xDM application was deployed.

When installing a new application version for in-place or out-of-place upgrade, pay attention to the application server and additional libraries versions installed in the current installation. This includes libraries related to security and authentication (role mapper, Waffle, etc.) These libraries are frequently compatible with a certain version of the application server and work with your current configuration. Do not systematically upgrade the application server and those libraries when upgrading the Semarchy application. If you decide to upgrade these libraries and the application server, make sure to review and test the compatibility and configuration.

Installing for In-Place Upgrade

To install the new application version for in-place upgrade:

  1. Deploy the new version Semarchy xDM Web Application Archive file on top of the previous version.

Refer to your application server documentation for more information about re-deploying applications on top of previous installations. The simple instructions for Apache Tomcat are provided below.

To perform an in-place upgrade for Apache Tomcat.

  1. Backup the <tomcat>/conf/Catalina/localhost/semarchy.xml file.

  2. Connect to the Apache Tomcat Manager (http://<tomcat_host>:<tomcat_port>/manager/).

  3. In the Deploy War File section, click the Select File button and then select the new semarchy.war file.

  4. Click the Deploy button.

  5. On the Tomcat server machine, restore the backup of the semarchy.xml file in the <tomcat>/conf/Catalina/localhost/ folder.

Installing for Out-of-Place Upgrade

To install the new application version for out-of-place upgrade:

  1. Deploy the new version Semarchy xDM Web Application Archive file as a new application, separated from the previous version.

When performing an out-of-place update, make sure to deploy the new version as a new application. If this new deployment takes place in the application server that already contains the previous version, make sure to change the application name during the deployment.
Refer to your application server documentation for more information about deploying new applications.

Configure the New Application

Perform the following operation for each deployment of the new Semarchy xDM application version.

Configuration for In-Place Upgrade

For in-place upgrade, the updated deployment reuses the configuration already in place. Check this configuration in the application server console or configuration files. The following points should be checked:

  • The datasources for the database schemas are set up and working.

  • The JavaMail Session servers are configured and working.

  • Security is configured.

Configuration for Out-of-Place Upgrade

To configure the new application version in an out-of-place upgrade:

  1. Configure new JDBC datasources for the new application deployment only. The datasources include those used to connect the Repository, Data Locations, etc. They should:

    • Point to the copies of the original schemas (e.g.: SEM_2_2_REPOSITORY, SEM_2_2_DLOC1).

    • Be local to the new application. Do not use global datasources

    • Use the same names as the datasources used by the old installation.

  2. Make sure that the existing JavaMail Session resource is configured or available for the new application.

  3. Make sure that the same security environment is configured for the new application.

Datasources should be configured locally for the application. For example, the repository datasource must be called SEMARCHY_REPOSITORY; Declaring this datasource at application-level avoids having multiple Semarchy xDM applications (possibly in different versions) connecting the same repository schema at the same time, which would lead to an erroneous behavior.
By using datasources local to the new application, you make sure that the new application connects exclusively to the copy of the schemas. Similarly, the old version should use local datasources to connect only the original schemas.

Start One New Application Instance

Perform this operation for a single new application version instance. If you have multiple repositories, you should repeat this operation for each repository.
Start your application server or connect to your application server and start the deployed application.

Upgrade the Repository

Repository upgrade is required only for major and minor versions. For patches, repository upgrade is not needed and the Repository Upgrade dialog does not appear when you connect.
The repository upgrade is an operation that cannot be undone. Make sure you have made a backup of the repository schema before proceeding.

Before Upgrading the Repository

The Oracle database has introduced in version 11g and 12c a new feature called Automatic Column Group Detection that automatically creates extended statistics for column groups. These extended statistics prevent the upgrade process to rename certain columns in the repository, causing an ORA-54032: column to be renamed is used in a virtual column expression error to appear during the repository upgrade process.

Before upgrading a repository hosted in an Oracle 11g or 12c instance, it is required to drop these statistics. They will be automatically re-created by the database as needed.

To drop unwanted extended statistics for repositories hosted in Oracle 11g and 12c:

  1. Connect to the database hosting the repository schema with a query tool (SQL Developer, SQL*Plus), using the repository schema user.

  2. Run the script provided below:

begin
  for xrec in (
    select
      TABLE_NAME, EXTENSION
    from
      USER_STAT_EXTENSIONS
    where
      TABLE_NAME like 'MTA%'
      and (
        EXTENSION like '%FROMEDITION%'
        or EXTENSION like '%TOEDITION%'
      )
  )
  loop

    DBMS_STATS.DROP_EXTENDED_STATS (
      ownname=>null,
      tabname=>xrec.table_name,
      extension=>TO_CHAR(xrec.extension)
    );

  end loop;
end;

Once the unwanted extended statistics are dropped, proceed with the repository upgrade.

Upgrading the Repository

To upgrade the repository:

  1. Connect to the new application instance you have started, using a user account with the semarchyAdmin role. At the first connection, the Repository Upgrade dialog appears.

  2. Review the dialog.

  3. Click the checkbox and then click the Upgrade button.

The repository upgrade starts. When it completes, the Overview window appears.

The repository upgrade process can only be triggered by a user with the semarchyAdmin role.

Upgrade the Data Locations

Data locations upgrade is required only for major and minor versions. For patches, data location upgrade is not needed, and the Overview page will not show any data location requiring an update.
The data location upgrade is an operation that cannot be undone. Make sure you have made a backup of the data location schemas before proceeding.
Development data locations upgrade is possible only if the current model edition deployed is valid. Before upgrade a development data location make sure that the deployed model does not raise any design-time validation errors.
Development data locations upgrade is possible is the current deployed model corresponds to the latest model state. For such data locations, make sure that the latest version of the model is deployed before performing a data location upgrade.

Before Upgrading the Data Locations

Before upgrading a data location hosted in an Oracle 11g or 12c instance, it is required to drop the extended statistics that may prevent column renaming. For more information, refer to Before Upgrading the Repository.

Certain columns are renamed in the data location when upgrading from a version before 3.2.0 to a version after.

To drop unwanted extended statistics:

  1. For each data location:

    1. Connect to the database hosting the data location schema with a query tool (SQL Developer, SQL*Plus), using the data location schema user.

    2. Run the script provided below:

begin
  for xrec in (
    select
      TABLE_NAME, EXTENSION
    from
      USER_STAT_EXTENSIONS
    where (
        TABLE_NAME like 'GI_%'
        or TABLE_NAME like 'GD_%'
      )
      and (
        EXTENSION like '%B_IS_VALIDATED%'
      )
  )
  loop

    DBMS_STATS.DROP_EXTENDED_STATS (
      ownname=>null,
      tabname=>xrec.table_name,
      extension=>TO_CHAR(xrec.extension)
    );

  end loop;
end;

Once the unwanted extended statistics are dropped, proceed with the upgrade.

Upgrading the Data Locations

To upgrade the data locations:

  1. Connect to the new application instance you have started, using a user account with the semarchyAdmin role.

  2. In Overview page, a warning in the Administration section indicates the list of data locations attached to this repository that require an update.

  3. Click the link under this list to perform the upgrade. The Data Location Upgrade dialog appears.

  4. Review the dialog.

  5. Click the checkbox and then click the Upgrade button.

The upgrade starts. When it completes, the Overview window is refreshed and and warning no longer appears.

The data location upgrade process can only be triggered by a user with the semarchyAdmin role.

Restart All Application Instances

The upgrade is now finished.
You can restart all the Semarchy xDM instances in the new version, and open the access through the user interface, services, APIs and database tables.

Post-Upgrade Actions

This chapter describes the actions required after the upgrade process was complete. These actions depend on the current version of Semarchy xDM and the version you upgrade to.

If no action is indicated in this chapter for your versions, then no action is required after restarting the Semarchy xDM application instances.
Certain actions performed on the model (for example, changes in validations, enrichers or any aspect of the certification process), which in essence are changes applied to your model, require that you re-deploy the model.

Upgrading from versions prior to 4.2.0

This section applies to all installations upgrading from a version prior to version 4.2.0.

Data Entry Workflows

After a model upgrade, the data entry workflows that existed in version prior to version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The data authoring experience in version 4 supports two methods:

  • Direct Authoring, into which users edit and save records, without having to create a workflow. This method uses a guided authoring UI based on a design-time object called a Stepper.

  • Workflow Authoring, into which users start a Workflow with tasks and transitions. Each task uses a stepper for modifying the data.

Workflows and steppers are triggered using actions defined in action sets.

Refer to the Working with Applications chapter in the Semarchy xDM Developer’s Guide for more information about steppers, workflows and action sets.

Required Action
A wizard helps model designer upgrade their original workflows into steppers and workflows. Model designers must use this wizard to recover and convert their workflows.

Upgrading Data Entry Workflows

To upgrade data entry workflows:

  1. In the Semarchy Workbench menu, open the model edition containing the workflows to upgrade.

  2. Select File > Upgrade v3.x Data Entry Workflows…​.

  3. Review the notice in the About Workflow Upgrade step and then click Next.

  4. The Workflow Upgrade step shows the existing data entry workflows, with the list of business views that they use for data entry.

    • Select Upgrade for each workflow that you want to upgrade.

    • Select the Upgrade Target for each of these workflows. This target defines how the workflow will be upgraded:

      • Stepper: This option is available only for single-task workflows using a single business view and having no validation or enrichment configured on the cancel transition. The wizard converts the business view into a stepper, reproducing its hierarchical structure and data entry forms. It also adds to this stepper the enrichers and validations defined on the submit transition.

      • Workflow: This option is available for all workflows. The wizard converts each business view used into a task to a stepper, as in the previous option. In addition, it converts the workflow to the new format, using for each task the newly created steppers. Enrichment and validations are added to the steppers and/or workflow transitions to reproduce the original behavior.

    • Select Actions to automatically create, in the entity’s default actions set, the authoring actions (create, edit, etc.) corresponding to the options enabled in the original workflow. These actions use either the stepper or workflow created by the wizard.

    • Select Delete to delete the original workflow. Note that this action cannot be undone.

  5. Click Finish. The upgrade wizard converts the original workflows according to your choices.

You can only upgrade workflows in open model editions.
You can run again the workflow upgrade process on any workflow that has not been deleted by the wizard.
The Delete option deletes the selected data authoring workflows. This operation cannot be undone.
When all v3.x data entry workflows are converted and deleted, the File > Upgrade v3.x Data Entry Workflows menu option automatically disappears.

Recommended Actions
Review and test the new workflows and steppers after the upgrade to make sure the new behavior corresponds to the original functional requirements.

Duplicate Management Workflows

After a model upgrade, the duplicate management workflows that existed in version prior to version 4.0.0 are not automatically upgraded. They are preserved in a hidden form in the model.

The duplicate management experience in version 4 now uses a dedicated design-time artifact, the Duplicate Manager, duplicate management actions defined in action sets to review, confirm, merge or split duplicates and suggestions.

Refer to the Working with Applications chapter in the Semarchy xDM Developer’s Guide for more information about duplicate managers and action sets.

Required Actions
A wizard helps you upgrade original workflows into duplicate managers after completing the model upgrade. Model designers must use this wizard to recover and convert their workflows.

Upgrading Duplicate Management Workflows

To upgrade duplicate management workflows:

  1. In the Semarchy Workbench menu, open the model edition containing the workflows to upgrade.

  2. Select File > Upgrade v3.x Duplicate Management Workflows…​.

  3. Review the notice in the About Workflow Upgrade step and then click Next.

  4. The Workflow Upgrade step shows the existing duplicate management workflows.

    • Select Upgrade for each workflow that you want to upgrade. Each workflow upgrades to a duplicate manager, using the default display card, form, and collection from the entity.

    • Select Delete to delete the original workflow. Note that this action cannot be undone.

    • Select Actions to automatically create, in the entity’s default actions set, the duplicate management actions using the duplicate manager created by the wizard.

  5. Click Finish. The upgrade wizard converts the original workflows according to your choices.

You can only upgrade workflows in open model editions.
You can run again the workflow upgrade process on any workflow that has not been deleted by the wizard.
The Delete option deletes the selected duplicate management workflows. This operation cannot be undone.
When all v3.x duplicate management workflows are converted and deleted, the File > Upgrade v3.x Duplicate Management Workflows menu option automatically disappears.

Recommended Actions
Review and test the new duplicate management actions after the upgrade to make sure the new behavior corresponds to the original functional requirements.

Survivorship and Override for Matching Entities

ID and fuzzy matched entities (also known as UDPK and SDPK entities) now support overriding data values on golden records. When data authoring operations are performed in these entities, the data authored (Authoring Data) by the users is clearly separated from the data loaded from external publishers (Source Data).

The consolidators have evolved into Survivorship rules, and support two phases:

  • Consolidation (using a Consolidation Rule), merge the data loaded from the publishers in the SD_ tables.

  • Override (using an Override Rule) apply possible data overrides authored by users on top of the consolidated data.

The survivorship rules allow implementing simply the pattern into which users fix consolidated values and/or create directly golden records in matching entities, without having to rely on matching and consolidation rules. There as now used only for the data loaded from external publishers (Source Data)

For more information about survivorship and override, refer to the Survivorship and the Certification Process sections in the Semarchy xDM Developer’s Guide.
Consolidators Upgrade

The upgrade process automatically performs the following changes in your models:

  • Consolidators upgrade to Survivorship Rules:

    • For Record-Level Consolidators, a single default survivorship rule is created with a consolidation rule corresponding to the original consolidator.

    • For Field-Level Consolidators, one survivorship rule is created per field, with a consolidation corresponding to the field’s original consolidator.

    • For all the upgraded survivorship rule, the override strategy is set by default to No Override.

  • Consolidation strategies upgrade:

    • The Any Value consolidation strategy is converted to a Custom Ranking strategy with no Raking Expression.

    • Other strategies remain unchanged.

  • Additional Changes

    • An additional Master ID Survivorship Rule is automatically created to consolidate the master record IDs into the golden record, with the Custom Raking consolidation strategy and no raking expression. This corresponds to Any Value.

Recommended Actions
Model designers should review the new survivorship rules and modify them according to their functional requirements, possibly adding override rules.

They should also consider:

  • Aggregating multiple single-attribute rules into multi-attribute rules, for attributes sharing the same consolidation strategy and sharing the same override strategy.
    Keep in mind that attributes within the same survivorship rule are always overridden simultaneously. As a consequence, an override on one of their attributes causes an override on all of them.

  • Creating a default survivorship per entity that applies to all attributes not taken into account by other rules.

Changes for the Certification Process

Publishers dedicated to data entry are no longer supported. The authoring data is already separated from the publisher data. These publishers should no longer be used in any of the rules involved in the certification process.

Required Actions
Model designers should review and re-design matchers and consolidators handling a combination of consolidation and authoring. They should split the logic into the two elements of the survivorship rules:

  • Consolidation Rules must only handle data flows consolidated from the publishers

  • Override Rules must only handle user changes performed after the consolidation phase.

Data authored by users are not enriched or validated for ID and fuzzy matching entities as part of the certification process. These records should be enriched and validated as part of the steppers into which users author the data.

Required Actions
Model designers should review the enrichers and validations applied to data from data entry publishers, and modify the steppers used for authoring this data in order to enforce enrichment and validation at this level. For example, when the stepper completes.

Changes for Data Integration
  • SD_ tables now only contain source data from application publishers and should only be used in that sense. New SA_ tables contain records created for data authoring and overrides performed on golden data.

Required Actions
Integration specialists should review data integration flows intended to publish data on behalf of users (using a data entry publisher), and consider publishing data in the SA_ tables instead of the SD_ tables.

Documentation Property

In this version, a Documentation property is added to entity attributes, complex types' definition attributes, references, matching and survivorship rules. The documentation property supports the Markdown syntax for rich text. It provides detailed information to end-users using the generated application.

The Description property is not only used for describing/documenting model artifacts for model designers. Descriptions are no longer visible to end-users.

The documentation of an attribute/reference is used by default in all forms fields based on this attribute/reference. It is possible to override this inherited documentation in the form field.

The upgrade process automatically copies the content of the Description properties (which was visible to end-users before) to the Documentation property.

Recommended Actions
Model designers should review the documentation property on entity attributes, complex types' definition attributes, references, matching and survivorship rules, possibly enhance it with markdown content. They should also review field documentation and possibly override the value inherited from the entity attributes.

Other Changes

Changes for Data Integration

Designers and integration specialists should be aware of the following table structure breaking changes:

  • Basic entities' SD_ and SE_ tables are renamed SA_ and AE_. Their structures remain unchanged.

  • GI_ tables no longer have a B_ERROR_STATUS column.

Required Actions
Model designers and integration specialists must review all components, scripts or data integration flows using these tables and columns and update them accordingly

Upgrading from versions prior to 4.0.0

This section applies to all installations upgrading from a version prior to version 4.0.0

Data Editions Decommission

The data edition feature is removed in this release. A data location now hosts one deployed model edition at a time and provide access to the current state of the data in the hub, based on the deployed model edition.

Changes for Deployment
  • The Data Editions view is renamed Data Locations and reorganized for a single deployed model and single data edition.

  • The deployment process for models and the management operations for data locations is simplified:

    • All operations related to data branch and data editions disappear.

    • Only one model edition is deployed at a time in a data location, and only data edition relies on this model.

    • All model edition operations on data location, that is installing a new model edition, updating an open model edition, or switching to an old model edition are replaced by a single Deploy Model Edition operation.

    • The Set Status to Maintenance and Set Status to Ready on the data location replace the Set Data Edition Status to Maintenance and Set Data Edition Status Back to Open.

  • Successive model edition deployments now appear in the Data Location view in a Deployment History node, showing the integration, installation and purge jobs deployed with the model edition.

Recommended Actions
Production teams should review and update their deployment process instructions to match the simplified process and administration console changes.

Changes for Data Locations
Existing data in closed data editions is removed from the data locations. Only data from the latest data edition is preserved in each data location.

Required Action
Model designers and integration specialists should review any component that accesses or uses data using data editions, taking into account the removal of all previous data editions.

Changes for the Certification Process
  • SemQL still supports the FromEdition, ToEdition and BranchID built-in attributes in existing clauses, but they are made deprecated, and will always return BranchID=0, FromEdition=0, and ToEdition=null.

  • The :V_DATABRANCH and :V_DATAEDITION built-in platform variables remain but are deprecated and both return 0.

  • The DataEdition Job Notification property is deprecated and returns the default value 0.

Recommended Actions
Designers should review their SemQL clauses and remove deprecated built-in attributes.

Changes for Data Integration

The B_BRANCHID, B_FROMEDITION, B_TOEDITION columns are dropped from all tables.

Required Action
Model designers and integration specialists should review any component that accesses or uses the data location tables in SQL, and remove any use of these columns: Integration jobs, SQL statements, PL/SQL functions called from SemQL, etc.

The PL/SQL INTEGRATION_LOAD integration package remains backward compatible. New versions of the functions not using data branch and edition are added to and replace the old ones.

Recommended Action
Integration specialists should review any component that uses these functions, and remove deprecated parameters.

For more information about table structures and the integration package, refer to the Semarchy xDM Integration Guide.

Post-Consolidation Validations

Post-consolidation validations, running on consolidated data, are no longer removing golden records. They only flag these records as erroneous, yet keeping them as golden records.

Changes for Applications

Golden data now includes records failing post-consolidation validations. A new ErrorStatus built-in attribute indicates whether the golden record has passed successfully or not validations. Possible values are VALID (if the record has no errors) or ERROR (if the record has errors).

Required Action
Model designers should review their business views or queries to filter (using the ErrorStatus = 'VALID' SemQL clause) golden data and exclude golden records failing post-consolidation validation if such validations are used.

Changes for Data Integration

Designers and integration specialists should be aware of the following table structure breaking changes:

  • B_STATUS column is removed from the GI_ and MI_ tables.

  • A new B_ERROR_STATUS column is added to the GD_ table to flag the golden records failing a post-consolidation validation. Possible values are VALID or ERROR.

Required Action
Integration specialists should now assume that the GD_ tables must be filtered to only read the valid golden records (where B_ERROR_STATUS = 'VALID').

Error Storage

The storage of errors has changed in this release:

  • The error tables (SE_ and GE_) no longer host the entire erroneous record, but only the list of errors with a reference to the erroneous records (which data is stored in the SD_ and GD_ tables).

  • Both SD_ and GD_ tables now have a B_ERROR_STATUS flag indicating the source or golden record status. Possible values are:

    • VALID if the record has no errors,

    • ERROR if the record has errors,

    • RECYCLED for source records only, if the record was recycled and considered valid

    • OBSOLETE_ERROR for source records only, if this record had errors but a newer version of the record fixed them.

  • The B_STATUS columns are renamed B_ERROR_STATUS.

  • The SE_ and GE_ tables no longer store the entire history of errors, but only the latest error instances.

Required Action
Integration specialists should review their error processing flows or scripts and respectively join data from the SE/SD tables (on B_PUBID, B_SOURCEID and B_LOADID) and GE/GD tables (on the Golden ID) in order to retrieve erroneous data along with the errors. They should also take into account the fact that the error tables only store the latest error instances.

Applications Upgrade

The structure and the content of the applications and application components have considerably changed between versions 3.x and 4.x. This section details how the repository upgrade converts original artifacts into new ones.

Display Cards

Display Cards replace the Display Name for entities.

The upgrade process automatically creates a default display card from the display name, reproducing in the primary text the concatenated string of the display name.

Refer to the Display Cards section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should enhance this display card, possibly configuring a secondary text, an avatar and/or an image.

Collections

Collections replace Table Views for entities.

The upgrade process automatically upgrades all table views to collection supporting only a table view. The tables columns are upgraded to the new version 4 components with a configuration reproducing the original appearance of the table.

Refer to the Collections section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should enhance the upgraded collections, possibly configuring a display card, and alternate list and grid views. They should also review the resulting tables and change the columns' configuration to improve the appearance of their collections.

Form Views

Forms replace Forms Views for entities. In addition, forms now support a new layouting system (using the flex model), multiple tables, embedded collection and richer components.

The upgrade process automatically upgrades all form views to forms with a single tab. The form fields are upgraded to the new version 4 components with a configuration reproducing the original appearance.

The upgrade process preserves the forms organization into sections, but does not preserve the layout of fields and sections in the form.
Refer to the Forms section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should review and enhance the upgraded forms, fixing the layout and splitting long forms into multiple tabs. They should also review the field components' configuration to improve the appearance of their forms.

Business Views

Business Objects and Business Object Views are now merged into a single Business View design-time object, moved under the entity. This new artifact supports the features of both business objects and business object views.

The upgrade process automatically converts business objects and business object views to business views, recreating the references to the upgraded forms and collections.

Refer to the Business Views section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should review and enhance the upgraded business views. They can, for example, configure transitions for the embedded collections created when improving the upgraded forms.

Applications

Applications are now organized with folders and actions (such as browse, inbox, etc.) and a navigation drawer containing shortcuts to certain actions.

The upgrade process automatically upgrades the original application organization and re-creates the folder and actions structure to provide similar features.

Refer to the Application Actions and Folders section in the Semarchy xDM Developer’s Guide for more information.

Recommended Actions
Model designers should review and enhance the experience in the upgraded applications by adding icons, images, and colors to the actions created, and by adding advanced actions such as error browsing or record creation.

Upgrading from versions prior to 3.3.0

This section applies to all installations upgrading from a version prior to version 3.3.0.

Incoming Records Not Merging Singleton Golden Records

Customers who have upgraded to version 3.2 prior to version 3.2.4 may experience an issue with singleton golden records (for fuzzy matching entities) already in the hub not being merged with incoming records that match them. This issue is caused by a defect in the upgrade mechanism.

Required Action
Customers upgrading from version 3.2.0 through 3.2.4 should refer to the Knowledge Base Article that explains how to fix an existing upgrade.

New Reserved Keywords in SemQL

New capabilities added to the SemQL language introduce the following reserved keywords:

  • HAS: This token is used as an equivalent to the HAVE keyword as in any <child_entity_role> has ( <condition_on_child_entity> ).

  • NULLS FIRST, LAST: These keywords allow for NULLS FIRST and NULLS LAST clauses in custom ranking consolidators.

Recommended Action
After the upgrade, model designers should review and validate their models and make sure that no physical object is named using the new reserved keywords.

Integration Job Execution Scheme

The Integration Job changes from a "by entity" to a "by Phase" execution scheme. This means that instead of processing entirely each entity, for example, Customers then Contact, it now processes entirely each phase (Enrichment for all entities, then validation for all entities, etc). This change does not impact the outcome of the integration job.

Recommended Action
No action is required. Possible database operations started outside of Semarchy and assuming specific step sequencing, should be reviewed and updated as needed.

Upgrading from versions prior to 3.2.0

This section applies to all installations upgrading from a version prior to version 3.3.0.

Matching Process

In versions prior to 3.2, a matcher includes a single matching condition.
Starting with version 3.2, a matcher is composed of several matching rules and supports scoring for automated merge and confirmation. To reflect this new matcher and provide performance enhancements, the data certification job was re-designed and the data location structures have been modified.

Matching Rule Upgrade

When a model is upgraded, existing matchers are converted to the new matchers in order to deliver the same matching behavior:

  • The existing matcher for each entity is converted to a matcher with a single matching rule called Default, with a matching score of 100. This rule contains the old matcher’s binning expressions and matching condition.

  • The Merge Policy is configured with all the thresholds set to 0, so that all duplicate groups detected automatically merge.

  • The Auto-Confirm Policy is configured as follows:

    • Auto-confirm golden records threshold is set to 100. Merged duplicate groups are never automatically confirmed.

    • Auto-confirm singletons is checked. Singletons are automatically confirmed.

Recommended Action
After the upgrade, model designers should review their matchers to benefit from the new capabilities and enable automated merge and confirmation for the business users and data stewards.

Data Location Upgrade

When a data location is upgraded, the data structures are modified to support new matching rules, scoring, suggested merges and auto-confirmation. See the Table Structures section in the Semarchy xDM Integration Guide for more information about these tables.

Recommended Action
Customers consuming golden and master records from GD and MD tables and using duplicate management-related columns should review the new structures and propagate changes in their data integration flows.

Data Administrators should gather statistics on the data location schemas to make sure that subsequent processing use statistics updated with the updated structure and values.

In addition, the existing records are upgraded with the following rules:

  • Golden records keep their existing IDs and confirmed/validated state.

  • Confirmed or unconfirmed golden records keep their status. This status is moved from the B_IS_VALIDATED column in GD tables to the new B_ISCONFIRMED column, and a new B_CONFIRMATIONSTATUS column provides a more detailed confirmation status in MD and GD tables.

  • Partially confirmed golden records (that is golden records with only part of their master records confirmed) are upgraded to Not Confirmed (B_ISCONFIRMED=0 and B_CONFIRMATIONSTATUS=NOT_CONFIRMED).

  • The existing duplicate management workflows remain, but the duplicates listed in these workflows disappear.

  • Golden records split and confirmed by the user are upgraded in an exclusion group.

  • A confidence score is automatically computed.

Recommended Action
Administrators and data stewards should review the upgrade rules listed above and should test the upgrade prior to applying it to their production environment. After the upgrade, data stewards should review the confirmation state of the duplicates and execute duplicate management workflows to confirm duplicates as needed.

Customized Search Forms

In versions prior to 3.2, all the default built-in search methods (full text, advanced, by form and SemQL) were available in the business object views. Starting with 3.2, it is possible to create customized forms and define which of the search methods or customized forms are available in applications.

Recommended Action
After the upgrade, model designers should review and customize the search methods available in their applications.

PL/SQL Functions

Starting with version 3.2, it is possible to declare user-defined PL/SQL functions. By declaring these functions, designers make them available in the SemQL editor and avoid SemQL warnings raised by these functions.

Recommended Action
After the upgrade, model designers should declare all PL/SQL functions used in their SemQL clauses. By doing so, model validation will no longer raise warning for these functions.

Reserved Physical Column Names

Prior to version 3.2, the attributes' physical column names could take any value. Starting with version 3.2.0, it is no longer possible to define physical column names starting with B_ as they may collide with reserved column names.

Models with physical column names violating this rule no longer pass model validation and each column will raise a validation error such as Invalid physical name: <column_name> matches the reserved pattern B_*.

Required Action
Prior to the upgrade, administrators should refer to the knowledge base article that explains how to detect, rename and upgrade the offending attributes.

Upgrading from versions prior to 3.1.0

This section applies to all installations upgrading from a version prior to version 3.3.0.

Model Privilege Grants for SemarchyAdmin

In versions prior to 3.1, the semarchyAdmin role had a hardcoded full access to the data on all models.
Starting with version 3.1 the privilege grant for this role is no longer hardcoded and can be modified.

When a creating a new model, or upgrading an existing one, a model-level privilege grant is automatically created for the semarchyAdmin role with Grant full access to the model selected to keep the same level of privileges.

Recommended Action
After the upgrade, model designers may want to modify this privilege grant to reduce the privileges of the semarchyAdmin role on the data and prevent platform administrators from accessing data in the hub.

Model Privileges Grants for the Integration Services

In versions prior to 3.1, accessing the integration services required a role having platform-level read/write privileges to the data location. Starting with version 3.1, a specific option called Grant access to integration services appears in the model privilege grants to define whether a role can access this model’s data via the services.

This privilege applies to the SOAP and REST Service APIs.

When upgrading, roles that could access the integration services thanks to their privileges on the data locations will automatically have this option checked for the models deployed in the data location.

Recommended Action
After the upgrade, model designers may want to review and modify this privilege grant to prevent access to the integration services to certain roles.

Upgrading from versions prior to 3.0.0

Upgrading from a Semarchy xDM version before version 3.0 is not supported.