Semarchy xDI 5.3 Release Notes

This document provides the latest information about Semarchy xDM Data Integration version 5.3, including new features and Bug Fixes.

Feature Highlights

Semarchy xDI version 5.3 is a major release with multiple improvements. This new version adds a new mechanism to simplify the installation of Components, an improved user experience, security features such as the external value resolvers, and of course integration features such as target and qualify filters, and new components.

New Licensing

Semarchy xDI comes with a new licensing mechanism. A fresh Designer setup now has a built-in 7-days trial period after which it must be registered.

Registration can take place in online, offline mode, or performed against a license server installed within your network. For scripted deployments, administrators can pre-set the license configuration for their users via system properties.

The new licensing mechanism simplifies Designer registration while supporting all specific infrastructure or security requirements.

Simplified Component Installation

Semarchy xDI Designer now includes a built-in mechanism to easily download the components and connectors to integrate and work with your systems.

Semarchy provides a central update site, pre-configured in Designer, from which users can download and install with a few clicks components certified by Semarchy. This process automatically installs all the required dependencies. Users can see and install all the components available, but can only use those allowed by their license.

Semarchy Branding and Product Experience

All components (Analytics, Designer, Runtime, and components) are rebranded to Semarchy, including all the graphical user interfaces.

Configuration parameters, including environment variables, options, driver URLs, class names, HTTP endpoints are also rebranded. Previous names are still supported for backward compatibility.

The entire product and components documentation is now consolidated in a single location, available online from the Semarchy website. It is released separately from product components for more frequent updates and improvements.

Improved User Experience

Semarchy xDI Designer is now certified with the latest versions of Windows, Linux, and macOS.

It includes multiple UX enhancements to accelerate your integration flow design experience, including reorganized toolbars, contextual menus and editors, clearer labels for properties and actions, etc.

Multiple Metadata Wizards have been enhanced, not only in terms of user experience but also by adding features to ease connecting and reverse-engineering. For example:

  • The Database reverse-engineering wizard indicates the possible causes when it finds no table. The reverse-engineering process also takes into account foreign keys referencing tables in separate schemas.

  • The JSON wizard now supports reverse-engineering from a JSON Schema.

  • The XML wizard can now reverse-engineer XML data stored in the clipboard, and infer datatypes when sourcing from XML data.

  • The Excel wizard is enhanced to help configure the JDBC driver properties. Excel datastores now support Row Number and Row Count pseudo-columns, added as Property Field nodes in an Excel datastore.

  • The HTTP REST metadata wizard now reverse-engineers cookies headers from the response. It also supports XML reverse-engineering from XML schemas in addition to JSON Objects and JSON Arrays.

  • Google BigQuery metadata now supports partitions reverse-engineering.

The Query Editor is greatly enhanced with new features: The Ctrl+Enter shortcut to run a query, sorting, column ordering, copying the contents of a column/cell/row to the clipboard as well as exporting to CSV.

In the Mappings editor, when mapping the first field from a source datastore the user is proposed to map only this field or automatically all source fields matching by name.

In the Processes editor, a new "Arrange all (with Collapse)" action collapses all steps while arranging them in the diagram.

The Expression Editor has been enhanced with improved syntax highlighting and enhanced detection and navigation for metadata, session, and environment variables, as well as user-defined functions.

Target and Qualify Filters

Target Filters are new filters that can be added to target datastores in mappings, allowing to filter the data targeted by updates. This feature is available and improves the performances of integration templates for technologies supporting this capability (Teradata, HSQL).

A new Qualify property on filters allows configuring the use of the QUALIFY clause for filter conditions. This type of clause filters the result of the ordered analytical/window functions on the data. This feature is available and improves the filtering performances for technologies supporting this capability (Teradata, Snowflake).

Extended Actions

A new Execute SFTP Command action allows running commands on an SFTP server.

Get Files with SCP and Send Files with SCP have a new "SCP Preserve File Date" property to preserve file timestamps after the transfer. These actions also support processed files publishing in a table, and Get Files with SCP supports binding downloaded files to actions.

The SQL File Export now supports exporting named objects instead of SQL queries resultsets, and a new "SQL Exp SQL Query File Encoding" property to define the encoding of the query file.

The Execute Delivery action now supports limiting the number of parallel running sessions.

Modules Improvements

With the Dynamic Module Refresh, Modules added or modified in a runtime no longer require a runtime restart to be taken into account. This behavior is configured using the moduleRefreshMode configuration parameter. Administrators can also force an agent to reload its modules using the reload-modules runtime command.

In addition, Designer automatically creates at startup all the modules considered complete in the local runtime. This option may be configured in the Designer using the xdi.designer.module.automaticCreation boolean.

Secrets Management

The management and securitization of sensitive data (secrets) in Semarchy xDI are enhanced with External Value Resolvers and Ciphering Keys.

External Value Resolvers is the new feature that allows designers to retrieve sensitive values stored in external secret stores such as Hashicorp Vault, Google Cloud Secret Manager, Amazon Secret Manager, Microsoft Azure Key Vault, etc. External Value Resolvers are configured in the runtime configuration file.
Designers can refer to external values in the metadata definitions, and xDI retrieves these values from the external stores and uses them when needed.

Semarchy xDI Production Analytics now supports the capability to define Ciphering Keys per environment and runtimes. These keys are used to generate deliveries with the encryption configured for the runtimes in these environments. In addition, you can select the key used to encrypt sensitive information stored in the Analytics database.
The Runtime also supports configuring a ciphering key to encrypt or decrypt passwords stored in the deliveries.

Log Level and Inheritance

Delivery execution and scheduling from Analytics, from the runtime command or from an action, now supports explicit configuration of the Session Log Level.

A new runtime configuration property enables Log Level Inheritance for child sessions to use the log level of their parent session.

New Semarchy xDM Component

The new Semarchy xDM Component supports bi-directional data integration for a Semarchy xDM data hub. Using this component optimized by the Semarchy teams, you can publish source data into Semarchy data locations, and consume data data to distribute it to other applications. This component includes Semarchy xDM dedicated tools to manage loads and sessions and take into account the Semarchy data certification process in your integration flows.

Components Highlights

A new Kafka Structured technology streamlines reading and writing structured messages to the Kafka event streaming platform. Using this technology, you configure both the connectivity to Kafka as well as message structure (Avro, JSON, etc). The structure is used to reverse-engineer, validate, integrate the messages. Reject topics are supported for messages that do not comply with the structure.

New SAP SQL Anywhere component to integrate data from and to the RDBMS formerly known as Sybase SQ.

The JMS and AMQP technologies now appear as dedicated components that can be installed from the component update site.

Excel now supports reverse-engineer and read/write to Excel Tables. The Excel driver also supports explicitly defining via the pathType=folder|file property whether it accesses a single Excel file or a folder containing multiple Excel files, each file accessed named after a Catalog.

Salesforce component is enhanced with a new template to integrate data using the Bulk API v2 from Salesforce.

The JSON integration template allows configuring how empty arrays or objects should be written in the output node.

The integration templates for REST and SOAP APIs now support configuring retries, functional response codes, and error messages capture to handle errors occurring on API calls.

Components installed in Designer now automatically contribute to the list of statistics, automatically adding those relevant for their technologies. These statistics can be enabled/disabled from the Designer’s preferences.

Noticable Changes

  • In the Designer Runtime view, the default protocol used to connect runtimes is HTTP and no longer RMI.

  • Analytics is no longer certified for Tomcat version before 8.5.x. Review the system requirements for the list of certified application servers.

  • Sybase ASE is no longer certified for Analytics. Review the system requirements for the list of certified databases.

  • Semarchy xDI Designer requires a JDK 11.0.2. Earlier version of Java 11 fail to connect to the license server. Review the system requirements for more details.

Change Log

For previous releases' release notes and changelogs, refer to the stambia.org website.

5.3.0

Semarchy xDI Designer

New Features
  • DI-128 : JSON Metadata: Reverse-engineering JSON schemas

  • DI-207 : Mappings: Improved "Arrange All (Horizontal Layout)" to better handle layout when using "Show source set" or "Show dataset links".

  • DI-399 : Processes: Clarify visual indicator on action parameters automatically set by a Metadata Link

  • DI-1627 : Web Service Publishing dedicated component.

  • DI-1628 : Scheduler dedicated component.

  • DI-2190 : Database reverse-engineering wizard provides indicates possible causes when no table is returned.

  • DI-2377 : Designer: Target platform upgrade

  • DI-2477 : New metadata wizards automatically select all the listed elements for selective reverse-engineering.

  • DI-2480 : HTTP REST: Reverse-engineering XML structures from XML schemas

  • DI-2488 : Runtime View: Use HTTP instead of RMI to connect runtimes

  • DI-2635 : Processes: The Action properties view header now displays the action type in addition to the step name

  • DI-2641 : Reverse-engineer foreign keys referencing tables in another schema when the other table and schema are available in the metadata

  • DI-2873 : Mappings: When mapping the first field from a source datastore the user is proposed to map only this field or automatically all source fields matching by name.

  • DI-2874 : Mappings: Target Filters can be added to target datastores

  • DI-2973 : HTTP REST metadata wizard supports reverse-engineering the response Set-Cookies headers.

  • DI-3051 : XML Metadata Wizard: Automatically infer during reverse-engineering datatypes from XML Data

  • DI-3383 : New mds:escape('TextToEncode','TargetFormat') XPath function to encode strings.

  • DI-3391 : XML Metadata Wizard: Reverse-engineering the structure from XML data stored in the clipboard.

  • DI-3395 : Modules Management Dialog: Improved scrollbar for a long list of modules

  • DI-3415 : An XPath expression evaluation now shows in the console the executed expression before the result

  • DI-3420 : XPath Functions: support parameter descriptions

  • DI-3422 : Improve how the nodes that do not exist anymore in Metadata are displayed in Mappings that were using them, to regroup them and make them more easily identifiable

  • DI-3430 : An XPath expression evaluation now shows in the console lists of items with line breaks

  • DI-3466 : Processes: New "Arrange all (with Collapse)" action in the process editor to collapse all expanded steps when arranging.

  • DI-3477 : Package Configuration editor: Select only processes or mappings when adding new items

  • DI-3478 : Package Configuration editor: Models selection dialog support filtering by pattern

  • DI-3484 : Internal Resources View: Use the tree layout by default

  • DI-3560 : Expressions editor enhanced with improved syntax highlighting and enhanced metadata detection and navigation.

  • DI-3631 : Metadata, Mapping and Process: Improve title and description of the creation wizards

  • DI-3701 : Allow Components to contribute to Designer monitored statistics

  • DI-3706 : Reload new or modified modules dynamically without restarting the runtime

  • DI-3707 : Improve UX for the Export Models and dependencies dialog

  • DI-3711 : Designer automatically creates for the local runtime all the modules considered complete.

  • DI-3714 : Mappings: Templates are now labeled as "Template" and no longer "Action Process" for clarity.

  • DI-3802 : Improve the title of the Reference Content Selection wizard for clarity

  • DI-3998 : Query editor usability enhancements

  • DI-4053 : Query Editor menu renamed to "Launch Query Editor"

  • DI-4141 : Legacy conf.ecg is no longer created in new workspaces, and replaced by Runtime Metadata.

  • DI-4244 : Metadata Editor: renamed the advanced tabs for clarity

  • DI-4250 : Rebranding: UX Elements

  • DI-4259 : Metadata Editor: Only show the tabs in advanced developer mode for clarity

  • DI-4290 : New Qualify property on filters allows configuring the use of the QUALIFY clause for filter conditions

  • DI-4325 : UX: Remove unnecessary views

  • DI-4326 : UX: Remove unnecessary menu items

  • DI-4327 : UX: "Execute" menu renamed to "Run"

  • DI-4329 : UX: Move the Navigator View to the proper view category

  • DI-4339 : UX: Remove unused perspectives

  • DI-4340 : UX: Add the name of the perspective

  • DI-4341 : UX: Remove unnecessary toolbar items

  • DI-4342 : UX: Datastore context menu reviewed in the mapping editor

  • DI-4346 : UX: Field context menu reviewed in the mapping editor

  • DI-4351 : UX: Reorder buttons in the toolbars

Bug Fixes
  • DI-216 : Unable to enter the "~" character in Designer with some operating systems

  • DI-318 : Designer UI stops refreshing in certain situations, when adding or removing values in Metadata unbounded attributes, forcing to close and reopen the Metadata to see the changes

  • DI-3002 : Metadata Editor: The "Open Metadata Reference" button does not work for attributes with multiple values.

  • DI-3088 : Module editor incorrectly allows carriage returns in the name field.

  • DI-3135 : Mappings: No warning when changing a field on an unnamed metadata property.

  • DI-3295 : Designer hangs at startup due to initialization issues in the cache

  • DI-3384 : Support special characters in workspace object names

  • DI-3400 : Runtimes are not displayed in the Navigator View when one of them has an invalid port defined

  • DI-3447 : SQL To Parameters Action: The sample script is not added when the action is created from a table drag and drop.

  • DI-3476 : Package Configuration editor: Filtering performance issues when the workspace contains a large number of mappings and processes

  • DI-3483 : Internal Resources View: Resources are unexpectedly displayed without a path when switching to the list layout

  • DI-3486 : Internal Resources: In the "Import Resource Into…​" action, the "OK" button should be disabled when no target folder is selected

  • DI-3523 : Mappings: The "Run" button is disabled in some situations, even after fixing the mapping errors

  • DI-3945 : EMF Compare: In the Git Staging view, the comparison editor on a newly added file opens with errors

  • DI-4023 : When renaming a mapping, the underlying generated process is not renamed accordingly

  • DI-4074 : Metadata Editor unexpectedly becomes dirty when clicking on attributes even when no modification is performed

  • DI-4306 : Runtime View: The list of runtimes is unexpectedly cleared when rebuilding the cache

Semarchy xDI Runtime

New Features
  • DI-496 : SCP Get/Send File Actions: Store the list of processed files in a database.

  • DI-546 : Execute Delivery Action: New parameter to configure the session log level

  • DI-547 : Execute delivery command: New parameter to configure the session log level

  • DI-548 : Child sessions can now inherit from their parent’s log level

  • DI-549 : Schedule delivery command: New parameter to configure the session log level

  • DI-618 : Session Name, Delivery name, and Process ID are logged when starting a session

  • DI-845 : Stored Procedures: Support numeric and timestamp datatypes

  • DI-1975 : External Value Resolver support in runtime.

  • DI-1979 : Support user-defined return codes in the execute delivery command

  • DI-2058 : Execute Delivery Action: New parameter to limit the number of parallel sessions

  • DI-2467 : New "Execute SFTP Command" action

  • DI-3296 : SOAP (WSDL): Configure retries, functional response codes and error messages capture to handle errors occurring on API calls.

  • DI-3358 : Get/Send Files with SCP Actions: New "SCP Preserve File Date" property to preserve file timestamps after the transfer.

  • DI-3361 : SQL File Export Action: New "SQL Exp SQL Query File Encoding" property to define the encoding of the query file.

  • DI-3379 : SQL File Export Action: support exporting named objects instead of SQL queries resultsets

  • DI-3439 : SCP Get File Action: Added support to bind to actions the downloaded files.

  • DI-3559 : Automatically create the target folder when publishing a delivery to a file delivery repository.

  • DI-3706 : Reload new or modified modules dynamically without restarting the runtime

  • DI-3708 : New "reload-modules" runtime command to reload modules.

  • DI-3760 : SSH, SCP and SFTP actions supporting libraries update.

  • DI-4151 : Startdelivery script started with no argument or with the -h or -help option displays the command help.

  • DI-4153 : JSON writing - ability to choose how to write empty objects and arrays

  • DI-4250 : Rebranding: UX Elements

  • DI-4254 : Rebranding: Runtime environment variables and options

Bug Fixes
  • DI-791 : Database pool information is incorrectly logged in the "rdbmsLog.log" file instead of the "jdbc.log" logger.

  • DI-2897 : Status attribute does not filter correctly user log events

  • DI-3085 : Execute Delivery Action should use a single request to initialize and run the session.

  • DI-3384 : Support special characters in workspace object names

  • DI-3781 : The runtime logs java.lang.NullPointerException errors if receiving connection requests while not fully started.

  • DI-4032 : A database connection test appears successful even when the database is not available or the connection information incorrect

  • DI-4139 : The automatic commit at the end of the session, if the session is successful, is not performed when the process contains actions with no transaction name

Semarchy xDI Production Analytics

Bug Fixes
  • DI-3400 : Runtimes are not displayed in the Navigator View when one of them has an invalid port defined

  • DI-4032 : A database connection test appears successful even when the database is not available or the connection information incorrect

New Features
  • DI-3437 : UX Improvement in the Authorization dialog

  • DI-4117 : Improved error message when building a delivery from an invalid package

  • DI-4255 : Rebranding: Analytics environment variables and options

  • DI-4329 : UX: Move the Navigator View to the proper view category

5.3.1

Semarchy xDI Designer

Bug Fixes
  • DI-5247 : The Designer fails to start with a License has been tempered error when using a license with an expiration date in online mode.

  • DI-5038 : Copy/pasting a Mapping that uses User Defined Functions creates a broken mapping that cannot be opened.

  • DI-4733 : REST and HTTP REST metadata reverse-engineering adds new nodes instead of updating existing ones when performed multiple times.

Semarchy xDI Runtime

New Features
  • DI-3450 : Updated the purgeLog.sql sample to use NOT EXISTS clauses instead of NOT IN for better performances.

Bug Fixes
  • DI-5285 : The Mail User parameter of the Send Email Action is incorrectly made mandatory.

  • DI-5230 : The Execute Delivery action does not start the sub-delivery when Deliv Memory Mode is set to false.

  • DI-5019 : An empty delivery file in the runtime delivery repository prevents the deployment any delivery, with an unavailable runtime error.

  • DI-4970 : The Kafka and HTTP REST components return generic errors in the Session Logs and detailed errors in the console.

  • DI-4879 : An error in a bind action used in a loop causes the the next loop iterations to throw a NullPointerException.

Semarchy xDI Production Analytics

New Features
  • DI-2254 : Analytics' Session Detail Editor now shows the child sessions started by Execute Delivery actions.

Bug Fixes
  • DI-4859 : Analytics login fails with an HTTP-500 error after upgrading from a version before 3.8 if the database contains missing session report parent links.

License Service

Bug Fixes
  • LS-347 : The License Server start and stop shell scripts contain Windows carriage returns, which causes issues on Linux/Unix.

  • LS-346 : When started as a Windows service with Java 11, the License Server stops after a few seconds with no error message.

  • LS-340 : The License Service fails to start as a Windows Service due to inconsistent relative working directory path resolution.

5.3.2

Semarchy xDI Designer

New Features
  • DI-5352 : Designer 7-days trial license: update the default permissions.

  • DI-5242 : EMF Compare: new shell command to run a logical merge.

  • DI-5213 : Designer Welcome Page: Update the links of the overview page.

  • DI-5205 : User-defined functions: Ability to escape the comma characters embedded inside the value of a function parameter.

Bug Fixes
  • DI-5444 : EMF Compare: When using the compare editor on a mapping containing stages, "java.lang.UnsupportedOperationException" error is displayed.

  • DI-5442 : EMF Compare: An erroneous false-positive conflict may be raised in some situations when cherry-pick/merging/pulling, although there are no real conflicts.

  • DI-5397 : EMF Compare: when the same attribute is added with different values from two distinct branches, the logical merge is unexpectedly not detecting it as a conflict.

  • DI-5330 : HTTP REST wizard: Authorization requests are unexpectedly not sent through the defined proxy.

  • DI-5311 : The designer may deploy the libraries of a previous component version in the modules instead of the upgraded component libraries.

  • DI-5018 : Switching the configuration in the designer does not correctly switch the runtime connected in runtime view.

Semarchy xDI Runtime

New Features
  • DI-2616 : A child delivery started with memory mode is now started within the same java virtual machine as its parent delivery, when the parent is run from startDelivery.bat(.sh) scripts.

Bug Fixes
  • DI-5051 : Execute delivery process action: "An illegal reflective access operation has occurred" error is thrown when running an autonomous delivery that uses WSDL mappings.

  • DI-4913 : Web Service Publication: Cannot publish a delivery as webservice when there is another delivery running on the runtime.

Semarchy xDI Production Analytics

Bug Fixes
  • DI-5346 : When trying to deploy a delivery that uses a certified action, "java.lang.ClassNotFoundException: com.indy.engine.Global cannot be found" error is displayed.

  • DI-5322 : Migration wizard: when migrating an Analytics V2 repository, metadata configurations might be migrated without node names in some situations.

  • DI-5318 : Session Reports: Filters have no effect when added while a session node is expanded.

  • DI-5265 : Session Reports: Refresh button has no effect when a session node is expanded.

License Service

Bug Fixes
  • LS-342 : Update Log4j third-party library (CVE-2021-44832).

5.3.3

Semarchy xDI Designer

No changes in this version.

Semarchy xDI Runtime

New Features
  • DI-5611 : Handle the java option "--add-opens" when defined by the user.

Bug Fixes
  • DI-5612 : When using Java 11, the Runtime doesn’t start with the error "ERROR com.indy.engine.main - When trying to get version" or starts with an empty console and empty logs.

  • DI-5610 : When using Java 8, the Runtime doesn’t start and throws the error "Unrecognized option: --add-opens=java.base/java.net=ALL-UNNAMED".

Semarchy xDI Production Analytics

No changes in this version.

License Service

No changes in this version.

5.3.4

Semarchy xDI Designer

Bug Fixes
  • DI-5751 : EMF Compare: in some scenarios the merge operation fails, unexpected conflicts are detected but not visible due to a NullPointerException.

  • DI-5636 : GitLogicalMerge and GitLogicalPull shell commands were unavailable.

  • DI-5465 : When saving a process imported from the internal resources which contains both "Meta-INF" and other nodes such as "xslVariableName", the latter are unexpectedly removed, causing errors such as "Variable has not been declared".

  • DI-5419 : Double-quotes the value of the "Name" property of a field node in a File metadata causes "Element type 'field' must be followed by either attribute specifications" error.

  • DI-5417 : Unable to empty a metadata password field.

Semarchy xDI Runtime

Bug Fixes
  • DI-5633 : The error "Unable to locate wrapper.jar" is displayed when trying to install the runtime as a windows service.

  • DI-5620 : At startup, the Runtime throws a "NullPointerException" when a Vault External Value Resolver is configured.

  • DI-5502 : The "Runtime Release Version" number that is displayed by the "versions" runtime shell command is the technical version number instead of the public version number.

  • DI-5403 : Sql File Export: Generating multiple positional files concurrently randomly produces wrong, corrupted files.

Semarchy xDI Production Analytics

New Features
  • DI-5270 : Sort package managers alphabetically in the administration editor.

Bug Fixes
  • DI-5321 : Errors such as "null value in column 'attr_db_id'" are thrown when migrating a repository from Analytics 2.x.x that contains attributes that use a nodeId that no longer exists.

License Service

Bug Fixes
  • DI-5639 : Upgrade the spring third-party library version to 5.3.18 (CVE-2022-22965).

  • DI-5625 : The license server didn’t start when using a custom certificate associated with several subdomain names or wildcarded subdomain name.

5.3.5

Semarchy xDI Designer

Bug Fixes
  • DI-5681 : EMF Compare: "UnsupportedOperation" error is displayed when comparing a process with local history.

  • DI-5829 : EMF Compare: "java.lang.ClassCastException" exception raised when comparing a process/mapping with double-click.

  • DI-5793 : EMF Compare: in some scenarios the merge operation fails, unexpected conflicts are detected but not visible due to a NullPointerException.

  • DI-5778 : From the designer, some sessions cannot be accessed when trying to view their details, with the error "Unable to find altIds in the runtime for session".

  • DI-5742 : Performance issue when rendering the selective reverse-engineering page in the various wizards.

Semarchy xDI Runtime

Bug Fixes
  • DI-5778 : From the designer, some sessions cannot be accessed when trying to view their details, with the error "Unable to find altIds in the runtime for session".

Semarchy xDI Production Analytics

No changes in this version.

License Service

Bug Fixes
  • DI-5769 : When a custom certificate is defined, the shutdown script of the license server fails with "unable to find valid certification path to requested target" error.