External value resolvers

Overview

In data integration processes, it is a good practice to keep sensitive information outside of your metadata. You can keep that information out of your metadata by using configuration parameters from external stores, which are then retrieved during execution. For example, you could store URLs, passwords, tokens, access keys, and more.

External value stores are configured as an External Value Resolver in Semarchy xDI Runtime.

Semarchy xDI supports multiple stores through External Value Resolver components, including HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager. It also offers a Simple External Value Resolver that stores values in a text file.

When designing your processes, use specific syntax to retrieve parameters directly from external stores.

Configure external value resolvers

Configure external value resolvers in the Runtime configuration file, engineParameters.xml, using the following structure per resolver:

Example 1. Configuration template
<externalValueResolver>
  <...>
  <resolver
    prefix="resolver-prefix"
    class="resolver-plugin-class"
    module="resolver-module"
  >
    <parameter name="parameter-name" value="parameter-value"/>
    <parameter name="parameter-name" value="parameter-value"/>
    <parameter name="parameter-name" value="parameter-value"/>
    <...>
  </resolver>
  <...>
</externalValueResolver>

Each resolver you add has:

  1. A resolver prefix

  2. A Java class

  3. A module name

  4. Parameters

You can externalize the resolver configuration to a separate file.

Prefix

The prefix is a unique name you give to the value store, which identifies it later.

The prefix is appended to the front of parameters retrieved from the store. It is good practice to add a trailing slash character / to the prefix for readability.

You can also have a single resolver without a prefix. If it does not have a prefix, the resolver is used by default when retrieving external values.

Class

The class attribute must contain the Java class of the resolver plug-in. Each resolver plug-in connects to a specific kind of store.

The list of supported resolvers is further in this article, along with their parameters.

Module

The module attribute must contain the name of an xDI module, so that the runtime has the files needed to communicate with the external value store.

Parameters

Each external value resolver has its own options. Define all needed options in a parameter tag, using name and value attributes.

External configuration

You can also keep your external value resolvers in a separate configuration file, referenced by a tag in the main engineParameters.xml configuration file. The path to this separate file is relative to the location of engineParameters.xml.

Example 2. Main configuration file
<!-- External Value Resolver Configuration	-->
<externalValueResolver file="./external-value-resolvers.xml" />
Example 3. External configuration file external-value-resolvers.xml
<externalValueResolver>
  <...>
  <resolver
    prefix="resolver-prefix"
    class="resolver-plugin-class"
    module="resolver-module">
    <parameter name="parameter-name" value="parameter-value"/>
    <parameter name="parameter-name" value="parameter-value"/>
    <parameter name="parameter-name" value="parameter-value"/>
    <...>
  </resolver>
  <...>
</externalValueResolver>

Use external value resolvers

To retrieve a value from an external store value, and use it in a metadata object in Semarchy xDI Designer, enter special reference syntax in specify in Designer fields:

%ext{<resolver-prefix><key-path>}ext%

The syntax has three parts:

  • Opening and closing tags: %ext{ and }ext%.

  • <resolver-prefix> is the resolver prefix you configured. If you leave it blank, the default resolver is used.

  • <key-path> is the path to the value in the store. Certain resolvers define another startup path automatically added before this one.

For example, suppose you want to use Google Cloud Secret Manager, configured with a gcp/ prefix, to retrieve the credentials/products/db001_password value in a password field. In this scenario, enter the following text in the field:

%ext{gcp/credentials/products/db001_password}ext%

Supported resolvers

File resolver

The file external value resolver is in Semarchy xDI by default. It retrieves values stored in a text file as key=value pairs, as in the following example:

db001_username=john
db002_username=bob

Supported parameters

Parameter Required Description

filePath

Yes

Full or relative path to a text file containing values. This parameter is mandatory.

refreshMode

No

Defines when the Runtime should reload the file. Possible values are:

  • dynamic (default): the file is automatically reloaded when it changes.

  • onStartup: xDI Runtime loads the file when it starts. A restart is required to register this change.

  • `dynamic

Sample configuration

<externalValueResolver>
  <resolver
    prefix="simple/"
    class="com.semarchy.xdi.externalvalueresolver.impl.SimpleExternalValueResolverPlugin">
    <parameter name="filePath" value="text-file-path"/>
    <parameter name="refreshMode" value="dynamic|onStartup"/>
  </resolver>
</externalValueResolver>

HashiCorp Vault resolver

This resolver retrieves values stored in a HashiCorp Vault server.

Install

To use this resolver, first install the HashiCorp Vault component in Designer. Then create a module for this technology.

When configuring the resolver, reference the module name in the module="" attribute.

Supported parameters

Table 1. General parameters
Parameter Required Description

host

Yes

HashiCorp Vault server hostname or IP address.

backendVersion

Yes

Vault backend API version. Possible values are KV_1 or KV_2.

port

No

HashiCorp Vault server port. Defaults to 80 for HTTP, and 443 for HTTPS.

scheme

No

HTTP scheme used to connect the server. Possible values are http or https. Defaults to https.

startPath

No

Path in which to look for values on the Vault server. This start path is a prefix for all retrieved keys.

For example, if you define the start path as secret/corp/, the production/secret_key_a value is retrieved from secret/corp/production/secret_key_a.

authMethod

Yes

HashiCorp Vault authentication method. Possible values are appRole or token.

If you set the authMethod parameter to appRole, also configure the following parameters:

Table 2. appRole parameters
Parameter Required Description

roleId

Yes

HashiCorp Vault Role Id.

secretId

Yes*

HashiCorp Vault Secret Id. You must encrypt the secret ID using the Runtime Encrypt command beforehand.

Set either this parameter, or the uncryptedSecretId parameter, but not both. Using this parameter is preferable to enhance the security of your integration flows.

uncryptedSecretId

Yes*

HashiCorp Vault Secret Id, as plain text. This parameter is a plain text alternative to the secretId parameter. To keep your integration flows more secure, avoid using this parameter where possible.

Set either this parameter, or the secretId parameter, but not both.

namespace

No

HashiCorp Vault namespace to use, if any.

If you set the authMethod parameter to token, also configure the following parameters:

Table 3. token parameters
Parameter Required Description

token

Yes*

Vault authentication token. You must encrypt the token using the Runtime Encrypt command beforehand.

Set either this parameter, or the uncryptedToken parameter, but not both. Using this parameter is preferable to enhance the security of your integration flows.

uncryptedToken

Yes*

Vault authentication token, as plain text. This parameter is a plain text alternative to the token parameter. To keep your integration flows more secure, avoid using this parameter where possible.

Set either this parameter, or the uncryptedToken parameter, but not both.

Sample configuration

<resolver
  prefix="vault/"
  class="com.semarchy.xdi.externalvalueresolver.vault.VaultExternalValueResolverPlugin"
  module="ExternalValueResolver HashiCorp">
  <parameter name="host" value="vault-host"/>
  <parameter name="authMethod" value="token"/>
  <parameter name="backendVersion" value="KV_2"/>
  <parameter name="port" value="8080"/>
  <parameter name="scheme" value="http"/>
  <parameter name="token" value="encrypted-token"/>
  <!-- parameter name="uncryptedToken" value="unencrypted-token"/ -->
  <!-- parameter name="namespace" value="vault_entreprise_namespace" -->
  <parameter name="startPath" value="secret/corp/"/>

</resolver>

Google Cloud Secret Manager resolver

This resolver retrieves values stored in Google Cloud Secret Manager.

Install

To use this resolver, first install the Google Secret Manager component in Designer. Then create a module for this technology.

When configuring the resolver, reference the module name in the module="" attribute.

Usage notes

The resolver only returns the most recent value for a given secret.

Semarchy xDI uses the Google Auth library to send Google Credentials. You must set the GOOGLE_APPLICATION_CREDENTIALS environment variable for it to work.

Supported parameters

Parameter Required Description

projectId

Yes

Unique ID of the project in the Google Cloud Platform.

Sample configuration

<resolver
  prefix="gcp/"
  class="com.semarchy.xdi.externalvalueresolver.google.GoogleCloudSecretManagerExternalValueResolverPlugin"
  module="ExternalValueResolver Google">
<parameter name="projectId" value="gcp-project-id"/>
</resolver>

AWS Secrets Manager resolver

This resolver retrieves values stored in AWS Secrets Manager.

Install

To use this resolver, first install the Amazon Secret Manager component in Designer. Then create a module for this technology.

When configuring the resolver, reference the module name in the module="" attribute.

Usage notes

Semarchy xDI uses the AWS default credentials provider chain. It relies on the environment to provide the AWS credentials.

By default, AWS credential files are in C:\Users\<UserName>\.aws\credentials under Windows, or ~/.aws/credentials on macOS or Linux. The file contains credentials as shown in the following example:

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

Supported parameters

Parameter Required Description

awsRegion

Yes

Which AWS region to use. This property must be a value from the AWS SDK region list.

Sample configuration

Example 4. AWS Secrets Manager External Value Resolver Configuration
<resolver
  prefix="aws/"
  class="com.semarchy.xdi.externalvalueresolver.amazon.AmazonSecretsManagerExternalValueResolverPlugin"
  module="ExternalValueResolver Amazon"
>
<parameter name="awsRegion" value="eu-west-1"/>
</resolver>

Microsoft Azure Key Vault resolver

This resolver retrieves values stored in Azure Key Vault.

Install

To use this resolver, first install the Microsoft Azure Key Vault component in Designer. Then create a module for this technology.

When configuring the resolver, reference the module name in the module="" attribute.

Usage notes

To access Azure Blob Storage, Semarchy xDI uses the Azure Default Credential provider. It relies on the environment to provide the Azure credentials.

For example, credentials can be retrieved automatically from environment variables.

Supported parameters

Parameter Required Description

keyVaultName

Yes

ID of the Azure project containing the secrets. This parameter is mandatory.

Sample configuration

<resolver
  prefix="azure"
  class="com.semarchy.xdi.externalvalueresolver.azure.AzureSecretManagerExternalValueResolverPlugin"
  module="ExternalValueResolver Azure">
<parameter name="keyVaultName" value="keyvault01"/>
</resolver>