Semarchy xDI 5.3 Release Notes

This document provides the latest information about Semarchy xDM Data Integration version 5.3, including new features and Bug Fixes.

Feature Highlights

Semarchy xDI version 5.3 is a major release with multiple improvements. This new version adds a new mechanism to simplify the installation of Components, an improved user experience, security features such as the external value resolvers, and of course integration features such as target and qualify filters, and new components.

New Licensing

Semarchy xDI comes with a new licensing mechanism. A fresh Designer setup now has a built-in 7-days trial period after which it must be registered.

Registration can take place in online, offline mode, or performed against a license server installed within your network. For scripted deployments, administrators can pre-set the license configuration for their users via system properties.

The new licensing mechanism simplifies Designer registration while supporting all specific infrastructure or security requirements.

Simplified Component Installation

Semarchy xDI Designer now includes a built-in mechanism to easily download the components and connectors to integrate and work with your systems.

Semarchy provides a central update site, pre-configured in Designer, from which users can download and install with a few clicks components certified by Semarchy. This process automatically installs all the required dependencies. Users can see and install all the components available, but can only use those allowed by their license.

Semarchy Branding and Product Experience

All components (Analytics, Designer, Runtime, and components) are rebranded to Semarchy, including all the graphical user interfaces.

Configuration parameters, including environment variables, options, driver URLs, class names, HTTP endpoints are also rebranded. Previous names are still supported for backward compatibility.

The entire product and components documentation is now consolidated in a single location, available online from the Semarchy website. It is released separately from product components for more frequent updates and improvements.

Improved User Experience

Semarchy xDI Designer is now certified with the latest versions of Windows, Linux, and macOS.

It includes multiple UX enhancements to accelerate your integration flow design experience, including reorganized toolbars, contextual menus and editors, clearer labels for properties and actions, etc.

Multiple Metadata Wizards have been enhanced, not only in terms of user experience but also by adding features to ease connecting and reverse-engineering. For example:

  • The Database reverse-engineering wizard indicates the possible causes when it finds no table. The reverse-engineering process also takes into account foreign keys referencing tables in separate schemas.

  • The JSON wizard now supports reverse-engineering from a JSON Schema.

  • The XML wizard can now reverse-engineer XML data stored in the clipboard, and infer datatypes when sourcing from XML data.

  • The Excel wizard is enhanced to help configure the JDBC driver properties. Excel datastores now support Row Number and Row Count pseudo-columns, added as Property Field nodes in an Excel datastore.

  • The HTTP REST metadata wizard now reverse-engineers cookies headers from the response. It also supports XML reverse-engineering from XML schemas in addition to JSON Objects and JSON Arrays.

  • Google BigQuery metadata now supports partitions reverse-engineering.

The Query Editor is greatly enhanced with new features: The Ctrl+Enter shortcut to run a query, sorting, column ordering, copying the contents of a column/cell/row to the clipboard as well as exporting to CSV.

In the Mappings editor, when mapping the first field from a source datastore the user is proposed to map only this field or automatically all source fields matching by name.

In the Processes editor, a new "Arrange all (with Collapse)" action collapses all steps while arranging them in the diagram.

The Expression Editor has been enhanced with improved syntax highlighting and enhanced detection and navigation for metadata, session, and environment variables, as well as user-defined functions.

Target and Qualify Filters

Target Filters are new filters that can be added to target datastores in mappings, allowing to filter the data targeted by updates. This feature is available and improves the performances of integration templates for technologies supporting this capability (Teradata, HSQL).

A new Qualify property on filters allows configuring the use of the QUALIFY clause for filter conditions. This type of clause filters the result of the ordered analytical/window functions on the data. This feature is available and improves the filtering performances for technologies supporting this capability (Teradata, Snowflake).

Extended Actions

A new Execute SFTP Command action allows running commands on an SFTP server.

Get Files with SCP and Send Files with SCP have a new "SCP Preserve File Date" property to preserve file timestamps after the transfer. These actions also support processed files publishing in a table, and Get Files with SCP supports binding downloaded files to actions.

The SQL File Export now supports exporting named objects instead of SQL queries resultsets, and a new "SQL Exp SQL Query File Encoding" property to define the encoding of the query file.

The Execute Delivery action now supports limiting the number of parallel running sessions.

Modules Improvements

With the Dynamic Module Refresh, Modules added or modified in a runtime no longer require a runtime restart to be taken into account. This behavior is configured using the moduleRefreshMode configuration parameter. Administrators can also force an agent to reload its modules using the reload-modules runtime command.

In addition, Designer automatically creates at startup all the modules considered complete in the local runtime. This option may be configured in the Designer using the xdi.designer.module.automaticCreation boolean.

Secrets Management

The management and securitization of sensitive data (secrets) in Semarchy xDI are enhanced with External Value Resolvers and Ciphering Keys.

External Value Resolvers is the new feature that allows designers to retrieve sensitive values stored in external secret stores such as Hashicorp Vault, Google Cloud Secret Manager, Amazon Secret Manager, Microsoft Azure Key Vault, etc. External Value Resolvers are configured in the runtime configuration file.
Designers can refer to external values in the metadata definitions, and xDI retrieves these values from the external stores and uses them when needed.

Semarchy xDI Production Analytics now supports the capability to define Ciphering Keys per environment and runtimes. These keys are used to generate deliveries with the encryption configured for the runtimes in these environments. In addition, you can select the key used to encrypt sensitive information stored in the Analytics database.
The Runtime also supports configuring a ciphering key to encrypt or decrypt passwords stored in the deliveries.

Log Level and Inheritance

Delivery execution and scheduling from Analytics, from the runtime command or from an action, now supports explicit configuration of the Session Log Level.

A new runtime configuration property enables Log Level Inheritance for child sessions to use the log level of their parent session.

New Semarchy xDM Component

The new Semarchy xDM Component supports bi-directional data integration for a Semarchy xDM data hub. Using this component optimized by the Semarchy teams, you can publish source data into Semarchy data locations, and consume data data to distribute it to other applications. This component includes Semarchy xDM dedicated tools to manage loads and sessions and take into account the Semarchy data certification process in your integration flows.

Components Highlights

A new Kafka Structured technology streamlines reading and writing structured messages to the Kafka event streaming platform. Using this technology, you configure both the connectivity to Kafka as well as message structure (Avro, JSON, etc). The structure is used to reverse-engineer, validate, integrate the messages. Reject topics are supported for messages that do not comply with the structure.

New SAP SQL Anywhere component to integrate data from and to the RDBMS formerly known as Sybase SQ.

The JMS and AMQP technologies now appear as dedicated components that can be installed from the component update site.

Excel now supports reverse-engineer and read/write to Excel Tables. The Excel driver also supports explicitly defining via the pathType=folder|file property whether it accesses a single Excel file or a folder containing multiple Excel files, each file accessed named after a Catalog.

Salesforce component is enhanced with a new template to integrate data using the Bulk API v2 from Salesforce.

The JSON integration template allows configuring how empty arrays or objects should be written in the output node.

The integration templates for REST and SOAP APIs now support configuring retries, functional response codes, and error messages capture to handle errors occurring on API calls.

Components installed in Designer now automatically contribute to the list of statistics, automatically adding those relevant for their technologies. These statistics can be enabled/disabled from the Designer’s preferences.

Noticable Changes

  • In the Designer Runtime view, the default protocol used to connect runtimes is HTTP and no longer RMI.

  • Analytics is no longer certified for Tomcat version before 8.5.x. Review the system requirements for the list of certified application servers.

  • Sybase ASE is no longer certified for Analytics. Review the system requirements for the list of certified databases.

  • Semarchy xDI Designer requires a JDK 11.0.2. Earlier version of Java 11 fail to connect to the license server. Review the system requirements for more details.

Change Log

For previous releases' release notes and changelogs, refer to the stambia.org website.

5.3.0

Semarchy xDI Designer

New Features
  • DI-128 : JSON Metadata: Reverse-engineering JSON schemas

  • DI-207 : Mappings: Improved "Arrange All (Horizontal Layout)" to better handle layout when using "Show source set" or "Show dataset links".

  • DI-399 : Processes: Clarify visual indicator on action parameters automatically set by a Metadata Link

  • DI-1627 : Web Service Publishing dedicated component.

  • DI-1628 : Scheduler dedicated component.

  • DI-2190 : Database reverse-engineering wizard provides indicates possible causes when no table is returned.

  • DI-2377 : Designer: Target platform upgrade

  • DI-2477 : New metadata wizards automatically select all the listed elements for selective reverse-engineering.

  • DI-2480 : HTTP REST: Reverse-engineering XML structures from XML schemas

  • DI-2488 : Runtime View: Use HTTP instead of RMI to connect runtimes

  • DI-2635 : Processes: The Action properties view header now displays the action type in addition to the step name

  • DI-2641 : Reverse-engineer foreign keys referencing tables in another schema when the other table and schema are available in the metadata

  • DI-2873 : Mappings: When mapping the first field from a source datastore the user is proposed to map only this field or automatically all source fields matching by name.

  • DI-2874 : Mappings: Target Filters can be added to target datastores

  • DI-2973 : HTTP REST metadata wizard supports reverse-engineering the response Set-Cookies headers.

  • DI-3051 : XML Metadata Wizard: Automatically infer during reverse-engineering datatypes from XML Data

  • DI-3383 : New mds:escape('TextToEncode','TargetFormat') XPath function to encode strings.

  • DI-3391 : XML Metadata Wizard: Reverse-engineering the structure from XML data stored in the clipboard.

  • DI-3395 : Modules Management Dialog: Improved scrollbar for a long list of modules

  • DI-3415 : An XPath expression evaluation now shows in the console the executed expression before the result

  • DI-3420 : XPath Functions: support parameter descriptions

  • DI-3422 : Improve how the nodes that do not exist anymore in Metadata are displayed in Mappings that were using them, to regroup them and make them more easily identifiable

  • DI-3430 : An XPath expression evaluation now shows in the console lists of items with line breaks

  • DI-3466 : Processes: New "Arrange all (with Collapse)" action in the process editor to collapse all expanded steps when arranging.

  • DI-3477 : Package Configuration editor: Select only processes or mappings when adding new items

  • DI-3478 : Package Configuration editor: Models selection dialog support filtering by pattern

  • DI-3484 : Internal Resources View: Use the tree layout by default

  • DI-3560 : Expressions editor enhanced with improved syntax highlighting and enhanced metadata detection and navigation.

  • DI-3631 : Metadata, Mapping and Process: Improve title and description of the creation wizards

  • DI-3701 : Allow Components to contribute to Designer monitored statistics

  • DI-3706 : Reload new or modified modules dynamically without restarting the runtime

  • DI-3707 : Improve UX for the Export Models and dependencies dialog

  • DI-3711 : Designer automatically creates for the local runtime all the modules considered complete.

  • DI-3714 : Mappings: Templates are now labeled as "Template" and no longer "Action Process" for clarity.

  • DI-3802 : Improve the title of the Reference Content Selection wizard for clarity

  • DI-3998 : Query editor usability enhancements

  • DI-4053 : Query Editor menu renamed to "Launch Query Editor"

  • DI-4141 : Legacy conf.ecg is no longer created in new workspaces, and replaced by Runtime Metadata.

  • DI-4244 : Metadata Editor: renamed the advanced tabs for clarity

  • DI-4250 : Rebranding: UX Elements

  • DI-4259 : Metadata Editor: Only show the tabs in advanced developer mode for clarity

  • DI-4290 : New Qualify property on filters allows configuring the use of the QUALIFY clause for filter conditions

  • DI-4325 : UX: Remove unnecessary views

  • DI-4326 : UX: Remove unnecessary menu items

  • DI-4327 : UX: "Execute" menu renamed to "Run"

  • DI-4329 : UX: Move the Navigator View to the proper view category

  • DI-4339 : UX: Remove unused perspectives

  • DI-4340 : UX: Add the name of the perspective

  • DI-4341 : UX: Remove unnecessary toolbar items

  • DI-4342 : UX: Datastore context menu reviewed in the mapping editor

  • DI-4346 : UX: Field context menu reviewed in the mapping editor

  • DI-4351 : UX: Reorder buttons in the toolbars

Bug Fixes
  • DI-216 : Unable to enter the "~" character in Designer with some operating systems

  • DI-318 : Designer UI stops refreshing in certain situations, when adding or removing values in Metadata unbounded attributes, forcing to close and reopen the Metadata to see the changes

  • DI-3002 : Metadata Editor: The "Open Metadata Reference" button does not work for attributes with multiple values.

  • DI-3088 : Module editor incorrectly allows carriage returns in the name field.

  • DI-3135 : Mappings: No warning when changing a field on an unnamed metadata property.

  • DI-3295 : Designer hangs at startup due to initialization issues in the cache

  • DI-3384 : Support special characters in workspace object names

  • DI-3400 : Runtimes are not displayed in the Navigator View when one of them has an invalid port defined

  • DI-3447 : SQL To Parameters Action: The sample script is not added when the action is created from a table drag and drop.

  • DI-3476 : Package Configuration editor: Filtering performance issues when the workspace contains a large number of mappings and processes

  • DI-3483 : Internal Resources View: Resources are unexpectedly displayed without a path when switching to the list layout

  • DI-3486 : Internal Resources: In the "Import Resource Into…​" action, the "OK" button should be disabled when no target folder is selected

  • DI-3523 : Mappings: The "Run" button is disabled in some situations, even after fixing the mapping errors

  • DI-3945 : EMF Compare: In the Git Staging view, the comparison editor on a newly added file opens with errors

  • DI-4023 : When renaming a mapping, the underlying generated process is not renamed accordingly

  • DI-4074 : Metadata Editor unexpectedly becomes dirty when clicking on attributes even when no modification is performed

  • DI-4306 : Runtime View: The list of runtimes is unexpectedly cleared when rebuilding the cache

Semarchy xDI Runtime

New Features
  • DI-496 : SCP Get/Send File Actions: Store the list of processed files in a database.

  • DI-546 : Execute Delivery Action: New parameter to configure the session log level

  • DI-547 : Execute delivery command: New parameter to configure the session log level

  • DI-548 : Child sessions can now inherit from their parent’s log level

  • DI-549 : Schedule delivery command: New parameter to configure the session log level

  • DI-618 : Session Name, Delivery name, and Process ID are logged when starting a session

  • DI-845 : Stored Procedures: Support numeric and timestamp datatypes

  • DI-1975 : External Value Resolver support in runtime.

  • DI-1979 : Support user-defined return codes in the execute delivery command

  • DI-2058 : Execute Delivery Action: New parameter to limit the number of parallel sessions

  • DI-2467 : New "Execute SFTP Command" action

  • DI-3296 : SOAP (WSDL): Configure retries, functional response codes and error messages capture to handle errors occurring on API calls.

  • DI-3358 : Get/Send Files with SCP Actions: New "SCP Preserve File Date" property to preserve file timestamps after the transfer.

  • DI-3361 : SQL File Export Action: New "SQL Exp SQL Query File Encoding" property to define the encoding of the query file.

  • DI-3379 : SQL File Export Action: support exporting named objects instead of SQL queries resultsets

  • DI-3439 : SCP Get File Action: Added support to bind to actions the downloaded files.

  • DI-3559 : Automatically create the target folder when publishing a delivery to a file delivery repository.

  • DI-3706 : Reload new or modified modules dynamically without restarting the runtime

  • DI-3708 : New "reload-modules" runtime command to reload modules.

  • DI-3760 : SSH, SCP and SFTP actions supporting libraries update.

  • DI-4151 : Startdelivery script started with no argument or with the -h or -help option displays the command help.

  • DI-4153 : JSON writing - ability to choose how to write empty objects and arrays

  • DI-4250 : Rebranding: UX Elements

  • DI-4254 : Rebranding: Runtime environment variables and options

Bug Fixes
  • DI-791 : Database pool information is incorrectly logged in the "rdbmsLog.log" file instead of the "jdbc.log" logger.

  • DI-2897 : Status attribute does not filter correctly user log events

  • DI-3085 : Execute Delivery Action should use a single request to initialize and run the session.

  • DI-3384 : Support special characters in workspace object names

  • DI-3781 : The runtime logs java.lang.NullPointerException errors if receiving connection requests while not fully started.

  • DI-4032 : A database connection test appears successful even when the database is not available or the connection information incorrect

  • DI-4139 : The automatic commit at the end of the session, if the session is successful, is not performed when the process contains actions with no transaction name

Semarchy xDI Production Analytics

Bug Fixes
  • DI-3400 : Runtimes are not displayed in the Navigator View when one of them has an invalid port defined

  • DI-4032 : A database connection test appears successful even when the database is not available or the connection information incorrect

New Features
  • DI-3437 : UX Improvement in the Authorization dialog

  • DI-4117 : Improved error message when building a delivery from an invalid package

  • DI-4255 : Rebranding: Analytics environment variables and options

  • DI-4329 : UX: Move the Navigator View to the proper view category

Base Components - Version 3.0.0 (Component Pack)

New Features
  • DI-128: JSON Metadata: Reverse-engineering JSON schemas

  • DI-496: SCP Get/Send File Actions: Store the list of processed files in a database.

  • DI-546: Execute Delivery Action: New parameter to configure the session log level

  • DI-2058: Execute Delivery Action: New parameter to limit the number of parallel sessions

  • DI-2467: New "Execute SFTP Command" action

  • DI-2641: Reverse-engineer foreign keys referencing tables in another schema when the other table and schema are available in the metadata

  • DI-3296: SOAP (WSDL): Configure retries, functional response codes and error messages capture to handle errors occurring on API calls.

  • DI-3358: Get/Send Files with SCP Actions: New "SCP Preserve File Date" property to preserve file timestamps after the transfer.

  • DI-3361: SQL File Export Action: New "SQL Exp SQL Query File Encoding" property to define the encoding of the query file.

  • DI-3379: SQL File Export Action: support exporting named objects instead of SQL queries resultsets

  • DI-3439: SCP Get File Action: Added support to bind to actions the downloaded files.

  • DI-3456: File Metadata: Only allow valid values for the "expression" property of computed fields

  • DI-3558: Actions supporting encoding properties now list sample encoding values.

  • DI-3701: Allow Components to contribute to Designer monitored statistics

  • DI-4053: Query Editor menu renamed to "Launch Query Editor"

  • DI-4153: JSON writing - ability to choose how to write empty objects and arrays

  • DI-4220: XML / JSON Metadata - sort alphabetically the property fields for clarity

  • DI-4727: Rebranding: Templates and sample projects

  • DI-4731: Rebranding: Template messages

  • DI-4800: HTTP Security and HTTP Proxy are now included as base components

  • DI-4813: Rebranding: Drivers classes and URLs

  • DI-4850: Target Filter - Implement target filters in HSQL Templates

  • DI-4962: Improved component dependencies and requirements management

Bug Fixes
  • DI-4023: When renaming a mapping, the underlying generated process is not renamed accordingly