Base Component Release Notes

This page lists the main features added to the Base component.

Feature Highlights

Version 2.2.5

Template REJECT HSQL

A new parameter named "Create index on verified table" has been added to create indexes on verified tables improves performance.

Version 2.2.4

Display order of the various items in Database

When expanding the Hierarchy of a Schema from Project Explorer, columns and datastores are mixed together due to the same Display Order specified on both nodes. Furthermore, under a schema, the datastores and query folders may be mixed together

The display order of the various items in database has been reviewed, such as datastores, columns, query folder, and more to make sure that they are displayed in a comprehensive order in Designer.

Version 2.2.3

This version contains some improvements which can be found on the full changelog.

Version 2.2.2

Temporary exchange files

Some components use temporary exchange files to handle the data received and sent and most of the stored files use XML format.

Sometimes, special characters raise issues when being stored inside this exchange file such as "\", "\f","\","\b"…​

The used exchange file format has been upgraded from XML 1.0 format to XML 1.1 format to avoid this issue.

Version 2.2.1

This version contains some improvements which can be found on the full changelog.

Version 2.2.0

JMS Process Action

The display name of the JMS Send Message Action has been changed to be consistent between the one displayed in the palette and the one defined when the action was created.

Concat Files Process Action

Append Line Separator

Addition of a new parameter "Append Line Separator" to specify if a separator should be added before writing the new data when using append mode.

The line separator character used is the separator defined in already existing "File Separator" parameter.

With these settings, a line separator is added in existing file before appending new data.

Sort Order

Addition of a new parameter "Order Sort" to choose in which order the files should be processed.

Three values are possible:

  • none - default value - same behavior as before, which depends on the order in which the file system returns the files

  • alphabetical ascending - order alphabetically with ascent order

  • alphabetical descending - order alphabetically with descendent order

Use Proxy Security Metadata in WSDL Metadata

Proxy information in WSDL Metadata used to be defined manually in different attributes directly in the WSDL Metadata.

To homogenize with the Proxy Security Metadata which exists from a few versions now, the WSDL Metadata now allows to select the Proxy Metadata node to be used.

The legacy proxy attributes have been moved to deprecated tab.

You can now select a Proxy Metadata node in WSDL Metadata.

The Proxy Metadata must have been created and defined before, to see it in the list.

You can also directly drag and drop the Proxy Metadata node inside the attribute to select it.

The Reverse REST wizard which can be used in WSDL Metadata has been improved to support selecting the Proxy Node also:

Version 2.1.1

JSON and XML Metadata - New Property fields

New Property fields available on XML and JSON Metadata, named "nodeAbsolutePosition" and "nodeUUID".

Version 2.1.0

File Metadata - Property Fields

List of available values on property fields

In File Metadata, when creating property fields, a new attribute named "Property" has been added on Property Field node to list the available values of "property fields".

When the "Property" attribute is specified, then the selected value is used as property field

When the attribute is not specified, then the name of the property field is used as property field.

Property fields as source or as target

Some property fields can be used as source and some as target in Mapping.

We have defined this in the metadata so that in a mapping it is not possible to use as a source a property field intended to be used as a target, for example.

Sort flat files

Sorting data on a flat file has been added in this version, like what is currently possible for JSON and XML.

  1. On the metadata, create a property field of type sortkey

  2. On the mapping, simply map this field to sort your data with a tag : SORT_ASC or SORT_DESC.

It works for flat files and hierarchical / multi-record files.

Default sort order is ascending.
If the parameter "Order by Expression" is set on the Template, the property field will take precedence.

Process Actions

SCP Actions

The SCP Actions require sometime additional properties to connect. On the SSH Metadata, a new parameter named "SSH Additional Connection Properties" has been added.

You can define in this parameter additional connection properties in the form of Java Properties. (key=value lines separated with line returns)

On the SCP Actions, this parameter is filled in automatically from the Metadata parameter when there is a metadata link on the action.

SQL File Export Process Action: Generate header

A new value named "always" is available for "generate header" parameter. This allows to indicate that the header should always be added, even in append mode.

The "generate header" boolean parameter has been transformed to a list of values with "true","false" and "always"

The change has been also performed on File export template.

SQL File Export Process Action: String delimiter

A new parameter named "Sql Exp String Delimiter Replacement" has been added to replace the string delimiter that is found inside the data when exporting data to file.

Version 2.0.7

Minor improvements and fixed issues

This version contains some minor improvements and fixed issues, which can be found in the complete changelog.

Version 2.0.6

Send Mail Action

file content encoding

With the Send Mail action, a file can be parameter as mail content. But the Send Email action cannot automatically apply the file encoding.

A new "Mail content file encoding" parameter has been added, to define the encoding to use to read the content of the file specified by the "Mail content file" parameter.

extra connection properties

A new key/value attribute is added on Outgoing Server to define extra connection properties.

A new parameter which allows to pass extra connection properties with the syntax of java like properties has been added on Send Email Process Action, this parameter is automatically set by the Metadata attribute when a metadata link is dropped on the action.

For instance to specify the host name of an HTTP web proxy server that will be used for connections to the mail server:

mail.smtp.proxy.host=localhost

Other improvements and fixed issues

This version also contains some other improvements and fixed issues, which can be found in the complete changelog.

Version 2.0.5

INTEGRATION FILE (File export) Template

Default numeric format

The file export action supports the definition of a numeric format. But this parameter was not managed.

A new parameter has been added on the Template, that allows to define a value for the action parameter and use the Metadata value when it is not defined.

With this format, it will give numeric values such as 1234.56

If no value is defined (default), the "Default numeric format" value defined in Metadata will be used.
String delimiter and string delimiter hexadecimal

A new parameter to alternatively define the string delimiter as character instead of hexadecimal representation has been added.

The Template works correctly with previous version but the parameter requires the hexadecimal value for the string delimiter.

For clarity, the following changes have been performed

  • The legacy "String Delimiter" has been renamed to by "String Delimiter (hexa)" parameter

  • A new parameter "String Delimiter" has been added, which allows to define the characters directly

If both parameters are used at the same time then the value of the string delimiter parameter is used first.

Integration Rdbms to Rdbms (no load)

New parameter - Fetch Size

Addition of a new parameter "Fetch size" to configure the number of lines used when querying the source database, to optimize memory management.

New parameter - Split Parallel Degree

Addition of a new parameter "Split Parallel Degree" to configure to enable parallel load of the source.

This parameter requires that a numeric column on the source has a tag 'SPLIT_BY' defined in its Metadata.

WDSL Improvements

Connection and Read Timeouts

This new version allows to define Connection and Read Timeouts at different levels.

You can define them globally on the WSDL Metadata, or for each operation, and override them also in Mapping through new parameters on Templates.

This offers agility to define timeouts accordingly to your requirements.

Definition in Metadata

In Metadata, you will find those new attributes under the "Advanced" tab.

The timeouts can be specified globally on the root node, applying to all the operations defined in the Metadata.

They can also be specified directly on an operation node, if you want to define different timeouts on the various operations.

Note that it overrides the default timeouts defined on the root node, if any.

Definition in Template

You can also define the timeouts directly in Mapping, through the corresponding two Template parameters.

Timeouts defined in Templates take precedence over the timeouts defined in Metadata.

If no timeout is defined on the Template, it will use the value from Metadata.

OAuth2 improvements

New parameter to specify how credentiels are sent

Some OAuth2 authentication servers require that the credentials are sent alongside the token generation.

When this is the case, those credentials were legacily sent as parameters by Stambia

However, some servers require those credentials to be sent differently.

A new attribute named "Send Client Credentials" has therefore been added in Metadata to define how client id and client secret should be sent, with the support of a new mode to send them in Basic Auth header.

This attribute is available for the following "Flow Type", which may require it depending on the OAuth2 server implementation:

  • Resources Owner Password Credentials Grant

  • Client Credentials Grant

This new attribute proposes the list of the following values:

  • Send Client Id as parameter

  • Send Client Id and Client Secret as parameters

  • Send Client Id and Client Secret as Basic Auth header Backward compatibility

Legacy "Send Client Id" and "Send Client Secret" have been moved to a deprecated tab:

When working with a Metadata using the legacy attributes, the default value for "Send Client Credentials" attribute is automatically adapted with the according value.

Change Data Capture (CDC)

Multiple improvements have been performed to homogenize the usage of Change Data Capture (CDC) in the various Components.

Parameters have been homogenized, so that all Templates should now have the same CDC Parameters, with the same support of features.

Multiple fixes have also been performed to correct CDC issues. Refer to the changelog for the exact list of changes.

Version 2.0.2

INTEGRATION File

Following improvements have been performed on this Template:

  • Parameter "Order ByExpression" now supports XPath expressions

  • Adding a new statistic: STAT_INSERT

Version 2.0.1

LDAP TOOLS

Improvements
  • On the LDAP TOOL INTEGRATOR, adding a new property "Match Ldap Attributes with", like on the LDAP EXTRACTOR (search). This parameter defines how the template matches the ldap attributes with the file’s fields :

  • On the LDAP TOOL EXTRACTOR (search), adding a new parameter to work with large volumes of data : "Page Size".

  • On the LDAP TOOL EXTRACTOR (search), to lighten exports, useless attributes are ignored.

Fixed issues
  • On the LDAP TOOL EXTRACTOR (search), when the tool was used several times, an error occurred.

  • On the LDAP TOOL EXTRACTOR (search), the parameter countLimit was not taken into account.

Load XML to Rdbms

A new parameter named 'Create Load Indexes" has been added to create load indexes.

When 'Create Load Indexes' is set to true, this parameter controls the syntax used to create the indexes.

It can be used to adjust the syntax according to the underlying technology.

It is a string that can contain the following special keywords.

Name Description

[indexName]

It’s the name of the Index. It must be dynamic.

[schemaName]

It’s the Schema Name of the Work Schema

[catalogName]

It’s the Catalog Name of the Work Schema

[tableName]

It’s the name of the Table without any prefix

[tablePath]

It’s the full name of the Table

[Columns]

It’s the list of the Columns to be indexes

Change Log

Version 5.3.11 (Component Pack)

Bug Fix

  • DI-5998: The tech:mappingName() XPath expression returns an empty value.

Version 5.3.9 (Component Pack)

New Features

  • DI-6382: The Outgoing Server (SMPTP) and Incoming Account nodes in the email metadata now support Microsoft Outlook’s modern authentication system based on a token mechanism. See Getting Started with Email Server

Version 5.3.8 (Component Pack)

New Features

  • DI-6249: The core RDBMS product has been improved. The information about the databases' support for indexing is now included within the technology definition.

  • DI-5896: A new template has been added to write JSON files using the JSON Lines Format: INTEGRATION Rdbms to JSON Lines.

Bug Fixes

  • DI-6218: The Replicator Rdbms tool fails to correctly generate the names of the tables that use delimiters in their names.

  • DI-6181: The Replicator Rdbms tries to create indexes even when target databases do not support indexing.

  • DI-6158: The Template INTEGRATION File (FileExport) template does not respect the case of the source column names when generating the file header.

Version 5.3.7 (Component Pack)

Bug Fixes

  • DI-5850: The warning "WARNING_FIELDS_NOT_FOUND" is unexpectedly displayed when mapping from a source file a field with a dash in its name.

Version 5.3.6 (Component Pack)

New Features

  • DI-6047: Provide a dedicated metadata that contains the built-in runtime entry within the internal resources, which replaces the hardcoded entry previously in the preferences and related view.

Version 5.3.5 (Component Pack)

New Features

  • DI-5836: Add a new generic XPath function for databases that allows generating a NULL constraint DDL.

  • DI-5741: Improve the core RDBMS product to allow defining in technology definitions if a datatype supports the NULL or NOT NULL constraints at creation.

Version 5.3.4 (Component Pack)

Bug Fixes

  • DI-5573: Template - REJECT Hsql - Missing SELECT keyword in the query checking alternate keys.

  • DI-5416: Template - Unload Rdbms to File - the clean step does not clean the temporary file when the split parameter is not specified.

Version 3.0.1 (Component Pack)

Bug Fixes

  • DI-5276: Template - Replicator Rdbms - the SQL query generated from the action "createIndexOnWorkTable" was incorrect, the schema name was unexpectedly doubled

Version 3.0.0 (Component Pack)

New Features

  • DI-128: JSON Metadata: Reverse-engineering JSON schemas

  • DI-496: SCP Get/Send File Actions: Store the list of processed files in a database.

  • DI-546: Execute Delivery Action: New parameter to configure the session log level

  • DI-2058: Execute Delivery Action: New parameter to limit the number of parallel sessions

  • DI-2467: New "Execute SFTP Command" action

  • DI-2641: Reverse-engineer foreign keys referencing tables in another schema when the other table and schema are available in the metadata

  • DI-3296: SOAP (WSDL): Configure retries, functional response codes and error messages capture to handle errors occurring on API calls.

  • DI-3358: Get/Send Files with SCP Actions: New "SCP Preserve File Date" property to preserve file timestamps after the transfer.

  • DI-3361: SQL File Export Action: New "SQL Exp SQL Query File Encoding" property to define the encoding of the query file.

  • DI-3379: SQL File Export Action: support exporting named objects instead of SQL queries resultsets

  • DI-3439: SCP Get File Action: Added support to bind to actions the downloaded files.

  • DI-3456: File Metadata: Only allow valid values for the "expression" property of computed fields

  • DI-3558: Actions supporting encoding properties now list sample encoding values.

  • DI-3701: Allow Components to contribute to Designer monitored statistics

  • DI-4053: Query Editor menu renamed to "Launch Query Editor"

  • DI-4153: JSON writing - ability to choose how to write empty objects and arrays

  • DI-4220: XML / JSON Metadata - sort alphabetically the property fields for clarity

  • DI-4727: Rebranding: Templates and sample projects

  • DI-4731: Rebranding: Template messages

  • DI-4800: HTTP Security and HTTP Proxy are now included as base components

  • DI-4813: Rebranding: Drivers classes and URLs

  • DI-4850: Target Filter - Implement target filters in HSQL Templates

  • DI-4962: Improved component dependencies and requirements management

Bug Fixes

  • DI-4023: When renaming a mapping, the underlying generated process is not renamed accordingly

Version 2.2.5 (Base Component)

Bug Fixes

  • DI-5034: Databases - A Foreign Key could not be created manually in a sub-Metadata File

Version 2.2.4 (Base Component)

New Features

  • DI-4942: Databases - review to display order of the various items such as datastores, columns, query folder, and more to make sure they are displayed in a comprehensive order in Designer

Version 2.2.3 (Base Component)

New Features

  • DI-4852: RDBMS databases technologies - new attribute "Technical Columns Prefix" available, which will be supported by Templates gradually

Version 2.2.2 (Base Component)

Bug Fixes

  • DI-3971: Metadata Products - "Explicit Join in Bracket" attribute which is defined in products should be displayed with "true" as default value in the interface when no value is set, as this is the value used behind the scenes by Designer when no value is defined

  • DI-4348: Temporary exchange files using XML format will now use XML 1.1 format to avoid issues with some special characters

Version 2.2.1 (Base Component)

Bug Fixes

  • DI-4415: Reject message for Not Null checks unexpectedly changed for database rejects, which could cause some misunderstanding

  • DI-4419: Template - Replicator Rdbms - Columns of work table in synchronization mode were not generated in the correct order in statements

  • DI-4421: Template - Replicator Rdbms - Incorrect delete statement generated when deleting from MySQL database

Version 2.2.0 (Base Component)

New Features

  • DI-3978: WSDL Metadata - support to alternatively define the Proxy settings through a proxy node from Proxy Metadata, and making it supported in Reverse REST Wizard

  • DI-1164: Concat Files Process Action - when using append mode, a new additional parameter has been added to define if a separator should be placed or not between the concatenated files

  • DI-2965: Concat Files Process Action - new parameter to choose the order in which the files are processed

  • DI-3757: Foreign Key - ability to select in the "referenced column" attribute, a column from a table which is on another schema

Bug Fixes

  • DI-2604: JMS Send Message From File Process Action - the display name has been modified to be coherent between the one displayed in the palette and the one defined when creating the action

  • DI-3262: Reject constraints - "User Message", "Reject Code" and "Security Level" attributes have been moved back to standard tab for clarity and to make them also appear again in Mapping when consulting constraint properties

  • DI-2642: Metadata - "CreationDdl" XPath expression does not produce the correct schema in foreign key reference clause

Version 2.1.1 (Base Component)

New Features

  • DI-3471: XML and JSON Metadatas - The new Property Field named "nodeAbsolutePosition" has been added

  • DI-3472: XML and JSON Metadatas - The new Property Field named "nodeUUID" has been added

Version 2.1.0 (Base Component)

New Features

  • DI-1970: File Metadata - ability to sort data when writing files through a new Property Field named "sortKey", which supports SORT_DESC and SORT_ASC tags to define the sort order

  • DI-2031: File Metadata - A list of possible values for “Property field” has been added in new attribute named Property to allow user to select value he want to use

  • DI-2033: File Metadata - definition of which property fields can be used as source or target

  • DI-2420: SQL File Export Process Action - A new parameter named "SQL_EXP_STRING_DELIMITER_REPLACEMENT" has been added to allow to replace the string delimiter that is found inside the data when exporting data to file

  • DI-2657: Template - Integration Rdbms to XML - The WRITE_KEY_VALUES parameter uses a reference: this is changed to an XPath retrieving the parameter value, to allow overriding this value from other Templates which are reusing it

  • DI-811: SCP Actions and SSH Metadata - Added a new "SSH Additional Connection Properties" parameter to define specific connection properties

  • DI-1426: SQL File Export Process Action - new value available for "generate header" parameter to allow indicating that the header should always be added (even in append mode)

  • DI-1716: NTLM Authentication - adding a dedicated wizard on NTML security node, and support NTLM security node in HTTP Security Metadata

  • DI-2479: XML Metadata - generation of XSD file is now in a dedicated wizard named "XSD Generation Tool"

Bug Fixes

  • DI-2618: LOAD Xml to Rdbms Template: correction of the error message "Unable to load class: com.stambia.avro.reader.visitor.AvroDatatTypeConverter" with Avro

  • DI-2697: JSON Property fields - the "nodeName" property field was not appearing in some other Metadata which are reusing JSON technology, such as HTTP REST Metadata

Version 2.0.7 (Base Component)

New Features

  • DI-2538: Mapping technology (map.tech) - support cdcMaxTimestamp option

  • DI-2731: Mapping technology (map.tech) - New parameter on XPath function tech:workName() which is available on steps to be able to enabled / disable object delimiters

Bug Fixes

  • DI-2709: Mapping technology (map.tech) - Staging Mask was not working when using [sessionTimestampName] on a database which has object name truncation size

  • DI-2714: Template - Unload Rdbms To File - fix split column prefix which is incorrect when mapping is multi-target

  • DI-2732: Template - UNLOAD Rdbms to File - generated temporary file names may unexpectedly contain object delimiters

  • DI-2777: Template - Load Rdbms to Rdbms - Index names should not contain a schema on some target databases (such as MySQL)

Version 2.0.6 (Base Component)

New Features

  • DI-3396: Databases - A new attribute named "Report Schema" have been added

  • DI-3507: Template - Integration Rdbms to Xml - Improved WRITE_KEY_VALUE parameter to be reusable when this Template is used in another Template

  • DI-3710: Template - Load Xml To Rdbms - support custom loading mechanism with streamHook

  • DI-3138: Email Metadata and Send Email Process Action - A new key/ value attribute is added on Outgoing Server to define extra connection properties

Bug Fixes

  • DI-3053: Read Email Process Action - MAIL_EXTRA_PROPERTIES was ignored during read email action.

  • DI-3403: INTEGRATION Rdbms to Wsdl - The expression to get the Metadata attribute "Send Client Credential" has been corrected to be passed properly to the Template

  • DI-778: Send Mail Action - Added a new "Mail content file encoding" parameter to define the encoding to use to read the content of the file specified by the "Mail content file" parameter

Version 2.0.5 (Base Component)

New Features

  • DI-133: WSDL Metadata - Remove the name attribute on Fault node which is not required anymore, to have a consistent behaviour with Input and Output nodes which does not have name attribute either

  • DI-1357: INTEGRATION Rdbms to Rdbms (no load) - Addition of a new parameter "Fetch size" to configure the number of lines of each fetch of data when querying the source database

  • DI-1550: INTEGRATION File (FileExport) Template - new parameter to alternatively define the string delimiter as character instead of hexadecimal representation

  • DI-1767: TOOL Ldap Extractor (search) - new parameter 'attributesFormattedAsSidString'

  • DI-1769: UNLOAD Rdbms to file supports defining if boolean values should be exported as numeric values or boolean values

  • DI-1771: INTEGRATION Rdbms to Rdbms (no load) - Addition of a new parameter "Split Parallel Degree" to enable parallel loading of source data

  • DI-1893: UNLOAD Rdbms to File: add support for split

  • DI-1894: Add a tooltip on "Enable CDC" attribute of an Rdbms column

  • DI-1903: Mapping generation (map.tech) - CDC improvement - tech:hasCdc() XPath function now supports defining a custom "sourceSetName", which will, most notably, allow the support of CDC on Mapping stages

  • DI-1904: Mapping generation (map.tech) - CDC improvement - new XPath function has been added on a step to improve the retrieval of CDC filter part (tech:cdcFilterPart2())

  • DI-1905: Mapping generation (map.tech) - CDC improvement - The test to generate the CDC Filter part on a source now also checks the fact that a CDC source exists

  • DI-2194: INTEGRATION File (FileExport) Template - the default numeric format specified on Metadata is now used automatically to define the numeric format used by the Template, and a new parameter on the Template allows to override it

  • DI-2849: Support in product definition to define a new group by syntax using aliases

  • DI-2862: RDBMS databases Metadata - new attributes "Default Table Storage Clause" and "Default Index Storage Clause" on temporary objects tab

  • DI-2864: Generic Templates - support the "Default Table Storage Clause" and "Default Index Storage Clause" attributes which are defined in Metadata

  • DI-1788: Templates updated - support the "Default Table Storage Clause" and "Default Index Storage Clause" attributes which are defined in Metadata

  • DI-1912: Templates updated - support having CDC sources on Templates which were not supporting it (such as staging templates)

  • DI-1910: Templates updated - New parameter 'Cdc Subscriber' on Templates on which it was not handled yet

  • DI-1909: Templates updated - New Parameters 'Unlock Cdc Table' and 'Lock Cdc Table' to configure the behaviour of CDC tables locking

  • DI-1139: Web Services invocation - addition of new attributes in Metadata and Templates to declare a Connection Timeout and a Read Timeout

  • DI-1828: Web Service authentication with OAuth2 - a new parameter named "Send Client Credentials" has been added to define how client id and client secret should be sent, with the support of a new mode to send them in Basic Auth header

Bug Fixes

  • DI-820: INTEGRATION File Template - when using "generate header" parameter alongside with "truncate=true" and "line separator=after", the generate header line was unexpectedly missing a line separator after it

  • DI-1605: INTEGRATION HSQL - lock and Unlock CDC steps should be executed only on sources which are not loaded through a load step

  • DI-1666: SSH Metadata - [techCode] and [productCode] XPath functions were unexpectedly returning "FTP" instead of "SSH"

  • DI-1764: INTEGRATION Hsql - 'Append Mode' parameter is ignored in some situations

  • DI-1948: Mapping generation (map.tech) - CDC improvement - Mapping stages were not taken into account by tech:hasCdcSource(), leading to issues when working with CDC

  • DI-1949: Mapping generation (map.tech) - CDC improvement - unable to distinguish directly journalized sources and sources involving journalization, which can lead to CDC actions being executed in some situations while they shouldn’t

  • DI-1908: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates on Lock / Unlock CDC steps

  • DI-1907: Templates updated - The 'Cdc Subscriber' parameter was ignored in some Templates when querying the source data

Version 2.0.4 (Base Component)

New Features

  • DI-1761: Mapping Transparent Stage - support using transparent stage feature with "Integration Rdbms to Wsdl" Template

  • DI-200: SQL To Parameters Process Action: a sample script is now provided in the body when creating the action

  • DI-523: New Process Action "Raise Error" available to raise an error with a defined message

  • DI-1967: Email Metadata and Read Email Process Action: on Incoming account, new attribute to provide additional key / value connection properties