Semarchy xDM 2023.1 Release Notes

Feature Highlights

Version Naming

Semarchy xDM’s versions will now use the <year>.<quarter>.<patch> pattern. The current version will be the first to use this pattern and will be called 2023.1.0.

Data-Driven Workflows

Semarchy xDM version 2023.1 comes with a major enhancement to augment user collaboration for effective data management through Data-Driven Workflows.

Workflow Builder

In the new graphical Workflow Builder module, designers can create rich data-driven workflows that allow divergent paths depending on record data, user data, or workflow metadata.

The building blocks for Workflows are steps, linked by transitions to move the workflow from one step to the next. Workflows are initiated through a Start Event, which defines the overall parameters and metadata of the workflow. Workflows also support automations, such as Submit to publish record data into the hub.

User tasks allow users to view or modify data records through steppers. Designers can now configure the stepper’s action bar with more options and controls for both moving through the stepper as well as moving through the workflow itself.

Workflows are triggered using actions defined in data applications. These actions can be added to the navigation drawer, to folders, or to the actions menu of an entity. The workflow startup experience, including available or mandatory fields, can be configured at design time.

Simplifying and streamlining the building process enables true collaboration between architects, designers, and even business users in the design of workflows. The no-code approach to designing and building elaborate workflows leads to faster turnaround and implementation, with a more transparent understanding of the workflows across the organization.

Data-Driven with SemQL

There are a number of functionalities in workflows that can be carried out dynamically. Certain steps now support data-driven properties with a built-in SemQL editor, through which designers access workflow data and metadata, functions, and variables.

For example, Routers allow designers to define data-driven rules using SemQL to control workflow behavior based on data.

SemQL may also be used to automatically assign tasks to a user or team.

Finally, SemQL is used to define the user associated with submitting records to the hub: the workflow initiator, the last person to have approved the record, or a data steward, for example.

Data-driven workflows are a powerful tool for designers to adapt processes to the intricacies of the business-directed solution they are providing and the data necessary for that solution. Further efficiency is also achieved by ensuring that the right information is available to the right people to accomplish a given task at the right time.

Parallel Blocks

Parallel blocks run simultaneous branches within the workflow, which may contain multiple steps, including user tasks and routers. With this capability, multiple teams and users can collaborate and work simultaneously on different parts of the records being modified in a workflow.

Parallel blocks allow workflows to meet any process optimization needs, and to respond dynamically to different or evolving business requirements across the organization, particularly when diverse business units are responsible for distinct aspects of data management.

My Tasks

In the navigation drawer, the Inbox has been replaced with My Tasks, providing an enriched view of all the user’s tasks and workflows in a single place. Users are able to quickly access and understand all their tasks and workflows with a single view that includes the data and metadata (such as status, current step, and assignees) attached to these tasks. Finally, users can also review past workflows that they initiated.

The workflow side panel provides users with a detailed view of a given workflow, whether in progress or completed. The history tab shows the user details of the selected workflow, allowing them to drill down into parallel blocks, see who has carried out previous steps, if they have made any comments, or provided any attachments at each step.

My Tasks provides data stewards and business users with a bird’s-eye view of the workflows they are involved in, contributing to their day-to-day efficiency.

Notifications

When a task is to be carried out in a workflow, email notifications can be configured and sent automatically to the user or team to which the task is assigned. These notifications are dynamic in that they may contain record data and workflow details.

Automated notifications alert users when action needs to be taken, while their rich content provides details regarding the task. This combination of automation and informativity augments productivity.

Authoring and User Experience

Read-Only and Browsing Mode for Form Fields

A new Authoring Mode property on form fields define whether fields should display using their editable, read-only, or browsing variant. This property replaces the Read-Only property. In addition, a Form Fields Mode property on form steps within steppers enables forcing all fields in a step to the read-only or browsing variant.

Using these properties, designers have better control over the appearance of fields, and can quickly design display-friendly review steps.

Keyboard Navigation

Keyboard shortcuts in Semarchy xDM applications increases accessibility and efficiency in performing actions and navigating in user views.

Data Notifications

Data Notifications benefit from several enhancements:

  • Configuration now supports filtering the propagated changes using timestamps in addition to batch IDs.

  • Configuration capabilities are enriched with new parameters to define a custom Connection Factory for JMS targets or Kafka key for Kafka targets.

Setup and Administration

Record Type-Specific Retention Policy

Record type-specific retention allows designers to control how long the different types of lineage and historical data (source data, source errors, history, or deletions, for example) are retained in the hub, globally or per entity.

Monitoring Execution Engine

New REST API endpoints (/appbuilder/execution-engine) are available to monitor, start and stop the execution engine and batch poller.

Other Noticeable Changes

Other noticeable changes include:

  • Configure the list of identity providers available per node in a Semarchy cluster using statup parameters.

  • A download link is automatically created in browsing forms for binary attributes' content that cannot be displayed by the browser.

  • Support for Snowflake as datasources for Dashboards

Change Log

Version 2023.1.2

Known Issues

  • MDM-7254: With PostgreSQL, the application server hosting the Semarchy xDM instance and the database server must be in the same timezone to avoid inconsistent dates in the integration loads.

  • MDM-7917: A boolean field default value is set to null when creating a record and to false when editing it.

  • MDM-7949: A model with a physical column name using a reserved name raises a validation error but can be deployed.

  • MDM-8034: When stopping Semarchy xDM using the Tomcat Manager, a SEVERE message is logged with org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks…​. This message is a known Tomcat Issue that can be ignored.

  • MDM-8703: Data import from Excel does not properly handle dates before Feb 28, 1900.

  • MDM-8977: Changing a reference to a fuzzy entity while authoring a master record is not taken into account.

  • MDM-9184: Existing FS_ and FP_ reference column values in a basic entity are overwritten by an enricher when this record is modified by the API or by authoring.

  • MDM-10193: When a record in an entity is updated, records with the same ID in inheriting entities are also updated.

  • MDM-10269: Using the SEM_NUMBER_TO_CHAR function with a string argument does not raise a model validation error but fails with an "invalid number" error in the log.

  • MDM-10319: Authoring the same record in two different browser windows makes the second authoring fail with a java.lang.IllegalStateException: Current Step (products) should be a form step error.

  • MDM-12415: Long Integer values higher than 9223372036854700 raise errors when authoring (Invalid ID for data type (longInteger)) or browsing (<entity name> not found) records.

  • MDM-12996: Login and SSO buttons only work when double-clicked on iOS phones and tablets.

  • MDM-13934: The Component Type drop down list in the form editor of the Application Builder contains an unknown semDisplayCard_Label value that raises an Unhandled event loop exception when selected.

  • MDM-13937: Selecting one workflow definition using the table view in Workflow Builder selects all instances.

Bug Fixes

  • MDM-13946: Duplicate managers performance improvement.

  • MDM-14151: Clicking a row in My Tasks, My Workflows, or the Administration UI unexpectedly refreshes the table and scrolls it to the top of the page.

  • MDM-14205: Column Values histograms in xDM Discovery show duplicate values counts.

  • MDM-14206: Authoring fields do not reflect model privileges grants in the context of workflow user tasks.

  • MDM-14212: Starting a workflow instance from a record having a UUID primary key fails with an unexpected server error.

  • MDM-14221: Duplicate managers performance optimization.

  • MDM-14240: Model documentation export fails when the Semarchy instance runs with JDK 17.

  • MDM-14245: Third party module upgrade: Apache Tomcat (xDM Preconfigured).

Version 2023.1.1

Known Issues

  • MDM-7254: With PostgreSQL, the application server hosting the Semarchy xDM instance and the database server must be in the same timezone to avoid inconsistent dates in the integration loads.

  • MDM-7917: A boolean field default value is set to null when creating a record and to false when editing it.

  • MDM-7949: A model with a physical column name using a reserved name raises a validation error but can be deployed.

  • MDM-8034: When stopping Semarchy xDM using the Tomcat Manager, a SEVERE message is logged with org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks…​. This message is a known Tomcat Issue that can be ignored.

  • MDM-8703: Data import from Excel does not properly handle dates before Feb 28, 1900.

  • MDM-8977: Changing a reference to a fuzzy entity while authoring a master record is not taken into account.

  • MDM-9184: Existing FS_ and FP_ reference column values in a basic entity are overwritten by an enricher when this record is modified by the API or by authoring.

  • MDM-10193: When a record in an entity is updated, records with the same ID in inheriting entities are also updated.

  • MDM-10269: Using the SEM_NUMBER_TO_CHAR function with a string argument does not raise a model validation error but fails with an "invalid number" error in the log.

  • MDM-10319: Authoring the same record in two different browser windows makes the second authoring fail with a java.lang.IllegalStateException: Current Step (products) should be a form step error.

  • MDM-12415: Long Integer values higher than 9223372036854700 raise errors when authoring (Invalid ID for data type (longInteger)) or browsing (<entity name> not found) records.

  • MDM-12996: Login and SSO buttons only work when double-clicked on iOS phones and tablets.

  • MDM-13934: The Component Type drop down list in the form editor of the Application Builder contains an unknown semDisplayCard_Label value that raises an Unhandled event loop exception when selected.

  • MDM-13937: Selecting one workflow definition using the table view in Workflow Builder selects all instances.

  • MDM-14240: Model documentation export fails when the Semarchy instance runs with JDK 17.

New Features

  • MDM-13972: Semarchy xDM is now certified with Java 17.

  • MDM-14020: Semarchy xDM is now certified with Eclipse Temurin Open JDK.

Bug Fixes

  • MDM-13355: A dashboard query using the PATINDEX function fails on SQL Server with an Unable to retrieve data from the server error.

  • MDM-13892: Some cells are not correctly aligned in data tables.

  • MDM-13901: In Dashboard Builder, a Missing mandatory value in databaseType error is issued when validating a newly created datasource.

  • MDM-13926: With Safari, the Primary Text of entity Display Cards is not visible.

  • MDM-13950: ID form fields set with the auto or force-readonly authoring mode display an empty value on authoring forms when the related entity attribute is a UUID.

  • MDM-13972: Semarchy xDM version 2023.1.0 fails to start with Java 17 JDK.

  • MDM-14022: Duplicates Managers performance enhancement.

  • MDM-14038: After an upgrade to 2023.1.0, the SEMARCHY_REPOSITORY_READONLY datasource is not available as a Dashboard datasource.

  • MDM-14044: After an upgrade to 2023.1.0, the upgraded SemQL in the Authoring Mode form field property is incorrect and raises an ORA-00932 error in Oracle.

  • MDM-14054: Object form fields fail to display content from an URL when browsing.

  • MDM-14070: A form section configured with a horizontal layout incorrectly uses a vertical layout.

  • MDM-14071: The Object form field displays the raw content when configured with a SemQL visibility condition.

  • MDM-14076: Reference fields defined with a non-writable expression are not displayed in authoring forms and prevent other fields in the same container from being displayed.

  • MDM-14093: Object form fields fail to display content from an URL when authoring.

  • MDM-14096: In Dashboard applications, the Cancel button of the Export Records dialog is not localized.

  • MDM-14108: The priority set while performing a transition in Data-Driven Workflows is not saved on the workflow instance.

  • MDM-14116: A newly created Dashboard Query raises an unexpected error when validated under certain conditions.

  • MDM-14117: Unexpected margins cause the content of collection columns to be truncated.

  • MDM-14124: The Workflow Builder link incorrectly appears on the Welcome Page for users with no sufficient privileges.

  • MDM-14125: Third-party library upgrade: Kafka Client.

  • MDM-14127: Third-party library upgrade: Commons File Upload.

  • MDM-14138: When a form is opened in a stepper, a nested exception is com.microsoft.sqlserver.jdbc.SQLServerException: Incorrect syntax near the keyword 'order' is raised with SQL Server under certain conditions.

  • MDM-14146: Enabling the Tomcat security manager causes server startup and integration jobs to raise a java.security.AccessControlException: access denied error.

  • MDM-14163: Master records cannot be moved to an existing golden record in Merge or Split duplicate managers using the graph view.

  • MDM-14171: Email notifications are not sent by Data-Driven workflows if the entity’s display card has no secondary text.

  • MDM-14204: Email notifications are not sent by Data-Driven Workflows if the primary or secondary text of the entity’s display card contains user-dependent variables.

  • MDM-14210: Third-party library upgrade: net.minidev and nimbus-jose-jwt.

Version 2023.1.0

Breaking Changes

  • MDM-6475: Following the replacement of the Read-Only form field property with Authoring Mode, fields containing computed expressions (SemQL) will display as Read-Only authoring fields when Authoring Mode is set to auto. Such fields were previously displayed as browsing fields. To display computed fields as browsing fields on authoring forms, designers should set their Authoring Mode to force-browsing in the Application Builder after migrating this version.

  • MDM-13565: Support for Java 8 has been discontinued. Only Java 11 is supported.

Known Issues

  • MDM-7254: With PostgreSQL, the application server hosting the Semarchy xDM instance and the database server must be in the same timezone to avoid inconsistent dates in the integration loads.

  • MDM-7917: A boolean field default value is set to null when creating a record and to false when editing it.

  • MDM-7949: A model with a physical column name using a reserved name raises a validation error but can be deployed.

  • MDM-8034: When stopping Semarchy xDM using the Tomcat Manager, a SEVERE message is logged with org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks…​. This message is a known Tomcat Issue that can be ignored.

  • MDM-8703: Data import from Excel does not properly handle dates before Feb 28, 1900.

  • MDM-8977: Changing a reference to a fuzzy entity while authoring a master record is not taken into account.

  • MDM-9184: Existing FS_ and FP_ reference column values in a basic entity are overwritten by an enricher when this record is modified by the API or by authoring.

  • MDM-10193: When a record in an entity is updated, records with the same ID in inheriting entities are also updated.

  • MDM-10269: Using the SEM_NUMBER_TO_CHAR function with a string argument does not raise a model validation error but fails with an "invalid number" error in the log.

  • MDM-10319: Authoring the same record in two different browser windows makes the second authoring fail with a java.lang.IllegalStateException: Current Step (products) should be a form step error.

  • MDM-12415: Long Integer values higher than 9223372036854700 raise errors when authoring (Invalid ID for data type (longInteger)) or browsing (<entity name> not found) records.

  • MDM-12996: Login and SSO buttons only work when double-clicked on iOS phones and tablets.

  • MDM-13901: Datasource validation in Dashboard Builder returns an error for wrongly missing Database Type property after creation.

  • MDM-13934: The Component Type drop down list in the form editor of the Application Builder contains an unknown semDisplayCard_Label value that raises an Unhandled event loop exception when selected.

  • MDM-13937: Selecting one workflow definition using the table view in Workflow Builder selects all instances.

New Features

  • MDM-6475: A new Authoring Mode property on form fields allows designers to define whether fields should use their browsing or read-only variant, or if their mode should be determined automatically depending on the attribute’s type and content. Similarly, a new Form Fields Mode property on stepper steps allows designers to force all fields for a given step to use their browsing or read-only variant.

  • MDM-8148: Data retention policies can be configured at a finer level for source data, source errors, history, and deletions. This is possible for the default retention policy as well as for individual policies for entities.

  • MDM-10690: A new Address.City attribute is now available in the OpenStreetMap enricher’s outputs.

  • MDM-11013: Object fields on browsing forms now show a download link when their content cannot be previewed instead of triggering an automated download.

  • MDM-11111: Object and Hyperlink fields on browsing forms now support Long Text attributes and data URLs.

  • MDM-11122: A new Quotation Character parameter in the import wizard options allows end-users to specify the quotation character used in the imported file.

  • MDM-11458: Reference fields now show More values available - type to filter at the bottom of the drop-down list when more than 50 records are available.

  • MDM-11469: Invalid datasources are now visible and available for selection in Discovery. An error message is displayed when an invalid datasource is selected.

  • MDM-11570: The app-builder/data-locations/<data-location-name>/ REST API endpoint now restarts the installation job in the case where the data location is in INSTALL_JOB_FAILED status.

  • MDM-11716: A new Time (HH:mm:ss) column in the job logs export shows task durations in hours/minutes/seconds.

  • MDM-12140: New NOTIFICATION_TRIGGER_TIMESTAMP, HIGH_BATCH_TIMESTAMP, and LOW_BATCH_TIMESTAMP built-in variables are available for data notifications to filter data depending on the notification firing timestamp and/or batch executions timestamps.

  • MDM-12170: A new Connection Factory property allows configuring a custom connection factory for JMS data notification targets.

  • MDM-12256: The app-builder/data-locations/<data-location-name>/set-status REST API endpoint now returns a HTTP 409 response with a specific message when the data location is in a status that is not compliant with the requested status change.

  • MDM-12310: Oracle 21c is now supported for Semarchy xDM repository and data locations.

  • MDM-12352: The page informing users about HTTP 500 errors now shows a log reference that can be retrieved in logs.

  • MDM-12411: The REST Request URL property for REST data notifications can now be more than 128 characters long.

  • MDM-12436: Snowflake is now supported for creating datasources for dashboards.

  • MDM-12546: New REST API endpoints allow monitoring, stopping and starting the batch execution engine.

  • MDM-12271: Different groups of identity providers (authentication methods) may be enabled per Semarchy xDM node. The xdm.idp.availableidps property in the startup configuration can be used to configure the list of identity providers for a given node.

  • MDM-13373: Third party library upgrade: kafka-clients

  • MDM-13455: The official xDM Docker image now runs Tomcat with a semarchy_user non-root user.

  • MDM-13532: AWS Aurora PostgreSQL versions 14.x are now supported for Semarchy xDM repository and data locations.

  • MDM-13857: A new Kafka Message Key property allows defining a custom value for the Kafka key used in data notifications sent to a Kafka target.

Bug Fixes

  • MDM-11129: Duplicate Management entity actions have no default label.

  • MDM-11255: Opening the filter panel of a child business entity and navigating back to the parent business entity using the breadcrumb prevents opening any record in the parent business view until the page is refreshed.

  • MDM-11664: Submitting a record that uses a LoV with no values causes the integration batch to fail.

  • MDM-11935: The Add Datasource dialog does not contain a Close button if no Discovery-compliant datasource is found.

  • MDM-11941: Sessions do not timeout when badges are enabled on application actions.

  • MDM-11974: Pressing any key opens the drop-down list for Reference, Chips, and Menu form fields, which prevents using the shift and tab keys for backward navigation.

  • MDM-12120: The $EntityNames job notification property is empty for deletions.

  • MDM-12148: The web.xml file provided with the xDM Preconfigured version prevents using à, é, or ç characters in passwords.

  • MDM-12163: SemQL validation on ID field keeps rejecting after modification the erroneous value that was initially provided.

  • MDM-12198: Search field drop-down list unexpectedly shows on the results page when the search form is submitted using the keyboard.

  • MDM-12245: Merge or Split action on SQLServer fails with a SQLException if a Golden Record has over 500 master records.

  • MDM-12299: Keyboard navigation improvements

  • MDM-12309: Form field documentation is not shown in the side panel for form fields with SemQL computed values.

  • MDM-12393: Zip file upload security enhancement.

  • MDM-12408: The Override and Reset Override buttons are still available for form fields set to read-only.

  • MDM-12432: Object form fields set to read-only still allow uploading a file on authoring forms.

  • MDM-12539: Field label color set to theme:accent is not applied.

  • MDM-12543: Validations for boolean fields in steppers are not triggered.

  • MDM-12549: Special characters / \ ; % in username cause user creation with the REST API to fail with a javax.ws.rs.ClientErrorException: HTTP 404 Not Found error. These characters are now authorized but /`and `\ require additional setup:

    • To allow the slash / character to be used in usernames, the encodedSolidusHandling property must be set to decode for every Connector in the server.xml file of the xDM instance.

    • To allow the backslash \ character, a new org.apache.catalina.connector.CoyoteAdapter.ALLOW_BACKSLASH must be set to true when starting xDM.

  • MDM-12564: UUID field values are shown as NaN in advanced filter drop-down lists.

  • MDM-12677: The Min. Date value set on a Date Picker field is rejected with a validation error when entered by a user on an authoring form.

  • MDM-12720: Values of long text attributes are not retrieved when entering an advanced filter criterion.

  • MDM-12821: Optimization of CPU and memory usage for model deployment on data locations containing large user defined functions.

  • MDM-12900: The field timeline on browsing forms shows values from failed integration batches.

  • MDM-13045: Third party library upgrade: AngularJS

  • MDM-13075: Using environment variables in the named query attributes of a data notification makes instances of this notification fail with a com.semarchy.mdm.datahub.datanotif.DataNotificationException$QueryRecordsException error.

  • MDM-13112: In Configuration, values in the Latest Authentication Date column are sorted alphabetically instead of being sorted in chronological order.

  • MDM-13171: Relative width between containers is ignored on authoring and browsing forms when the user reduces the width of browser window.

  • MDM-13179: Application theming color defined with notation different from hexadecimal RGB (#rrggbb) prevents fields from being displayed on authoring forms.

  • MDM-13224: Match rules using the SEM_EDIT_DISTANCE_SIMILARITY function with PostgreSQL raise a org.postgresql.util.PSQLException: ERROR: division by zero error.

  • MDM-13263: Background color on Reference collection columns is displayed as gradient instead of solid color.

  • MDM-13287: Checkbox authoring fields with long helper text are not displayed and cannot be selected.

  • MDM-13343: Checkbox authoring fields do not show their full helper text in a tooltip.

  • MDM-13357: In Dashboard Builder, renaming a dashboard without actually changing its name removes it from the list of dashboards.

  • MDM-13417: Clicking the value of a Hyperlink column in a collection that contains several columns with this type always opens the content of the first Hyperlink column.

  • MDM-13459: Drop-down lists on form filters do not open after saving a filter.

  • MDM-13475: Zipped SVG images are not imported in image libraries.

  • MDM-13481: The horizontal scrollbar is not available on collections accessed with Firefox.

  • MDM-13495: On the Users collection in Configuration, the Select All functionality ignores the filter previously set by the user.

  • MDM-13505: Third party module upgrade for xDM Quick Start virtual machines on AWS, Azure, and GCP and Azure solution template: Apache Tomcat

  • MDM-13518: Third party library upgrade: d3-color

  • MDM-13640: Starting xDM on a Mac M1 CPU using an arm64 JDK fails with a org.osgi.framework.BundleException: Could not resolve module: com.semarchy.engine.plugins.convergence.email error.

  • MDM-13643: Saving the target properties of a Kafka data notification fails with a java.lang.NoClassDefFoundError: Could not initialize class com.semarchy.mdm.datahub.datanotif.producers.KafkaMessageProducer error.

  • MDM-13669: In the datasource editor, clicking the header of the Driver Properties section after entering a property value as a secret raises an Error: Minified React error #31 error in the user interface.

  • MDM-13771: The French label of the Kafka Topic data notification property is incorrect.

  • MDM-13864: Third party library upgrade: SnakeYAML