External Value Resolvers

Overview

You can use in data integration processes configuration parameters stored into external stores. These values are retrieved at execution time. External value stores are configured as External Value Resolver in the Semarchy xDI Runtime.

At design time, you use a specific syntax for the values stored in external stores. When needed, Semarchy xDI retrieves these values from the external stores and uses them.

External Value Resolvers are recommended to store sensitive information. For example, URLs, passwords, tokens, or keys to access databases or systems. This avoids keeping that sensitive information in the Metadata.

Semarchy xDI supports multiple stores via External Value Resolver components, including HashiCorp Vault, AWS Secrets Manager Azure Key Vault, Google Cloud Secret Manager. It also supports a Simple External Value Resolver that stores values in a text file.

Configure External Value Resolvers

External Value Resolvers are configured in the Runtime Configuration File (engineParameters.xml) using the following syntax:

Example 1. External Value Resolver Configuration Template
<externalValueResolver>
  <...>
  <resolver
    prefix="resolver-prefix"
    class="resolver-plugin-class"
  >
    <parameter name="parameter-name" value="parameter-value"/>
    <parameter name="parameter-name" value="parameter-value"/>
    <parameter name="parameter-name" value="parameter-value"/>
    <...>
  </resolver>
  <...>
</externalValueResolver>

You can configure multiple resolvers using this syntax. Each resolver is configured with:

  • A prefix that uniquely identifies the store. This prefix is used to specify a value retrieved from the store. If you do not specify a prefix, the resolver will be used by default to retrieve external values. Only one resolver can be defined without a prefix.

    The prefix is used as is before the referenced value. It is a good practise to use a trailing slash / on the prefix for readability.

  • The class is the Java class of the resolver plugin. Each resolver plugin addresses a specific store.

  • Multiple parameters, provided with a name and a value These parameters configure the resolver.

The Supported Resolvers with their parameters are listed below.

Use External Value Resolvers

To use in a metadata a value stored in an external store and retrieve it via an External Value Resolver, specify in the Semarchy xDI Designer field a reference to that value with the following syntax:

%ext{[<resolver-prefix>]<key-path>}ext%

In this syntax:

  • <resolver-prefix> is the prefix configured for the resolver. This prefix is optional. When unspecified, the default resolver (with no prefix) is used.

  • <key-path> is the path to the value in the store. Note that certain resolvers define a startup path the is automatically added before that key path.

Use a value stored in Google Cloud Secret Manager

For example, to use the credentials/products/db001_password value stored in a Google Cloud Secret Manager configured as the gcp/ external value provider, in the database metadata password field, set the following:

%ext{gcp/credentials/products/db001_password}ext%

Supported Resolvers

File

The simple file external value resolver is included by default. It retrieves values stored in a text file. This file is a key/value properties file, as in the example below:

db001_username=john
db002_username=bob
Example 2. Simple External Value Resolver Configuration
<externalValueResolver>
  <resolver
    prefix="simple/"
    class="com.semarchy.xdi.externalvalueresolver.impl.SimpleExternalValueResolverPlugin"
  >
    <parameter name="filePath" value="text-file-path"/>
    <parameter name="refreshMode" value="dynamic|onStartup"/>
  </resolver>
</externalValueResolver>

This resolver supports the following parameters:

Parameter Description

filePath

Path to a text file containing the values. This parameter is mandatory.

refreshMode

This optional parameter defines when the Runtime should reload the file. Possible values are:

  • dynamic (default): The file is automatically reloaded when changes are detected.

  • onStartup: The file is loaded when the Runtime starts. A restart is required to take changes into account.

HashiCorp Vault

This resolver retrieves values stored in a HashiCorp Vault server.

This resolver is not included by default. You must install the HashiCorp Vault component prior to using this resolver.
Example 3. HashiCorp Vault External Value Resolver Configuration
<resolver
  prefix="vault/"
  class="com.semarchy.xdi.externalvalueresolver.vault.VaultExternalValueResolverPlugin"
>
  <parameter name="host" value="vault-host"/>
  <parameter name="authMethod" value="token"/>
  <parameter name="backendVersion" value="KV_2"/>
  <parameter name="port" value="80"/>
  <parameter name="scheme" value="http"/>
  <parameter name="token" value="encrypted-token"/>
  <!-- parameter name="uncryptedToken" value="unencrypted-token"/ -->
  <!-- parameter name="namespace" value="vault_entreprise_namespace" -->
  <parameter name="startPath" value="secret/corp/"/>

</resolver>

This resolver supports the following parameters:

Parameter Description Related Auth Method

host

Hashicorp Vault server hostname or IP address. This parameter is mandatory.

backendVersion

Vault backend API version. Possible values are KV_1 or KV_2. This parameter is mandatory.

port

Hashicorp Vault server port. This parameter is optional, and port 80 is used by default for HTTP, 443 for HTTPS.

scheme

HTTP scheme used to connect the server (http or https). This parameter is optional and defaults to https.

authMethod

Hashicorp Vault authentication method. This parameter is mandatory. Possible values are appRole or token.

roleId

Hashicorp Vault Role Id.

appRole

secretId

Hashicorp Vault Secret Id. The secret id must be encrypted using the Runtime Encrypt command. You can also specify an unencrypted secret id, which is not recommended for security reasons.

appRole

uncryptedSecretId

Hashicorp Vault Secret Id. This secret id must be defined as plain text. It is a plain text alternative to the secretId parameter. It is not recommended for security reasons.

appRole

namespace

Hashicorp Vault Namespace. This parameter defines the namespace to use if any and is optional.

appRole

token

Vault authentication token. This token must be encrypted using the Runtime Encrypt command. You can also specify an unencrypted token, which is not recommended for security reasons.

token

uncryptedToken

Vault authentication token. This token must be defined as plain text. It is a plain text alternative to the token parameter. It is not recommended for security reasons.

token

startPath

Path into which the values are looked for in the Vault server. This path prefixes all the retrieved keys. For instance, if you define the secret/corp/ start path, production/secret_key_a is retrieved from secret/corp/production/secret_key_a.

Google Cloud Secret Manager

This resolver retrieves secret values stored in Google Cloud Secret Manager.

This resolver is not included by default. You must install the Google Secret Manager component before using this resolver.
For a given secret, only the latest value is returned.
Example 4. Google Cloud Secret Manager External Value Resolver Configuration
<resolver
  prefix="gcp/"
  class="com.semarchy.xdi.externalvalueresolver.google.GoogleCloudSecretManagerExternalValueResolverPlugin" module="External Value Resolver Google"
>
<parameter name="projectId" value="gcp-project-id"/>
</resolver>

This resolver supports the following parameters:

Parameter Description

projectId

The ID of the project in the Google Cloud Platform. This parameter is mandatory.
Credentials Configuration

Semarchy xDI uses the GoogleCredentials to build the Google Credentials and so relies on the GOOGLE_APPLICATION_CREDENTIALS environment variable, which must be set.

AWS Secrets Manager

This resolver retrieves secret values stored in AWS Secrets Manager.

This resolver is not included by default. You must install the Amazon Secret Manager component before using this resolver.
Example 5. AWS Secrets Manager External Value Resolver Configuration
<resolver
  prefix="aws/"
  class="com.semarchy.xdi.externalvalueresolver.amazon.AmazonSecretsManagerExternalValueResolverPlugin"
  module="External Value Resolver Amazon"
>
<parameter name="awsRegion" value="eu-west-1"/>
</resolver>

This resolver supports the following parameters:

Parameter Description

awsRegion

AWS region to use. This property must be a valid input for AWS Region. This parameter is mandatory.
Credentials Configuration

Semarchy xDI uses the AWS Default Credential Provider Chain and therefore relies on the system/environment for the AWS credentials.

For example, credentials are retrieved automatically from the C:\Users\<UserName>\.aws\credentials` (windows) or ~/.aws/credentials (macOS/Linux) file. which contains your credentials as shown in the below example:

[default]
aws_access_key_id = YOUR_AWS_ACCESS_KEY_ID
aws_secret_access_key = YOUR_AWS_SECRET_ACCESS_KEY

Azure Key Vault

This resolver retrieves secret values stored in Azure Key Vault.

This resolver is not included by default. You must install the Azure Key Vault component before using this resolver.
Example 6. Azure Key Vault External Value Resolver Configuration
<resolver
  prefix="azure"
  class="com.semarchy.xdi.externalvalueresolver.azure.AzureSecretManagerExternalValueResolverPlugin"
  module="External Value Resolver Azure">
<parameter name="keyVaultName" value="keyvault01"/>
</resolver>

This resolver supports the following parameters:

Parameter Description

keyVaultName

ID of the Azure project containing the secrets. This parameter is mandatory.
Credentials Configuration

Semarchy xDI relies on the system/environment for the Azure Credentials, including:

  • AZURE_CLIENT_ID : Azure Client ID.

  • AZURE CLIENT_SECRET : Azure Client Secret.

  • AZURE_TENANT_ID : Azure Tenant ID.