Semarchy Data Management
This source extracts metadata from Semarchy xDM Data Management
Overview
This source connects to the Semarchy Data Management server to retrieve the data location assets (entities, attributes, certification jobs, etc.), then to the underlying database hosting this data location to retrieve the underlying physical assets (tables, columns).
The underlying data location database is configured as an inner_source.
This source supports:
-
Metrics retrieval for data location assets. For example, the number of golden records for entities.
-
Stateful Ingestion for both data location and underlying physical assets.
-
Data Profiling to collect table, row, and column statistics for the underlying physical assets.
-
Set the Domain for the underlying physical assets.
-
Filter Assets for the underlying physical assets.
Sample Recipe
source:
type: semarchy-xdm
config:
xdm_base_url: 'http://localhost:8080'
xdm_dataloc: CustomerB2CDemo
xdm_api_key: <api-key>
# xdm_api_username: <user>
# xdm_api_password: <password>
# disable_ssl_verification: true
inner_source:
# Configure the inner source depending on the underlying
# data location database.
type: postgres
config:
host_port: localhost:5432
database: semarchyDemoDatabase
username: username
password: password
include_tables: true
include_views: true
profiling:
enabled: true
profile_table_level_only: false
schema_pattern:
allow:
- semarchy_customer_b2c_mdm
sink:
# sink configParameters
The following table lists the source parameters.
Parameter |
Description |
||
|
Base URL of the Semarchy Data Management application server. |
||
|
Name of the data location to harvest. |
||
|
Semarchy Data Management API Key parameter to connect to Data Management server. Use this API Key instead of the
|
||
|
Semarchy Data Management user. You can also use the |
||
|
This user’s password. |
||
|
Option to disable SSL verification. |
||
|
Database source configuration corresponding to the underlying database of the data location. This configuration is a regular PostgreSQL, Oracle or Microsoft SQL Server source configuration.
|