Welcome to Semarchy xDM.
This guide contains information about using the product to design and develop an MDM project.

Preface

Audience

This document is intended for users interested in using Semarchy xDM for their Enterprise Master Data Management Initiatives.

If you want to learn about MDM or discover Semarchy xDM, you can watch our tutorials.
The Semarchy xDM Documentation Library, including the development, administration and installation guides is available on-line.

Document Conventions

This document uses the following formatting conventions:

Convention Meaning

boldface

Boldface type indicates graphical user interface elements associated with an action, or a product specific term or concept.

italic

Italic type indicates special emphasis or placeholder variable that you need to provide.

monospace

Monospace type indicates code example, text or commands that you enter.

Other Semarchy Resources

In addition to the product manuals, Semarchy provides other resources available on its web site: http://www.semarchy.com.

Obtaining Help

There are many ways to access the Semarchy Technical Support. You can call or email our global Technical Support Center (support@semarchy.com). For more information, see http://www.semarchy.com.

Feedback

We welcome your comments and suggestions on the quality and usefulness of this documentation.
If you find any error or have any suggestion for improvement, please mail support@semarchy.com and indicate the title of the documentation along with the chapter, section, and page number, if available. Please let us know if you want a reply.

Overview

This guide contains information about using the product to design and develop an MDM project.

Using this guide, you will learn how to:

  • Use the Semarchy Workbench to design and develop an MDM project.

  • Design the logical model representing your business entities.

  • Design the certification process for certifying golden data from data published from source systems.

  • Deploy the logical models and run the certification processes developed in Semarchy Workbench.

Introduction to Semarchy xDM

Semarchy xDM is designed to support any kind of Enterprise Master Data Management initiative. It brings an extreme flexibility for defining and implementing master data models and releasing them to production. The platform can be used as the target deployment point for all master data of your enterprise or in conjunction with existing data hubs to contribute to data transparency and quality with federated governance processes. Its powerful and intuitive environment covers all use cases for setting up a successful master data governance strategy.

Semarchy xDM is based on a coherent set of features for all Master Data Management projects.

Unified User Interface

Data architects, business analysts, data stewards and business users all share the same point of entry in Semarchy xDM through the single Semarchy Workbench user interface accessible through any type of browser. This interface uses rich perspectives to suit every user role and allows team collaboration. The advanced meta-data management capabilities enhance the usability of this intuitive interface.

A Unique Modeling Framework

Semarchy xDM includes a unique modeling framework that combines both ER and OO modeling. Data architects and business analysts use this environment to define semantically complete models that will serve as references for the enterprise. These models include the description of the business entities as well as the rules associated with them.

The Modeling Framework supports:

  • Logical Data Modeling: The expression of the logical data model semantics and rules by business analysts. This includes:

    • The target data model (Entities / Attributes / Relations / List of Values, etc.).

    • The rules for data quality (validations, referential integrity validation, list of values​​, etc.).

    • The data access privileges associated with user roles

  • Certification Process Logical Design. This includes:

    • The definition of applications that publish data to the Hub (Publisher)

    • The rules to enrich and standardize data

    • The rules to match and to identify groups of similar records

    • The survivorship rules to produce the reference data (Golden Data)

  • Applications. This includes:

    • Creating, organizing and branding user-facing applications

    • Designing display cards, forms and collections to display the data from the entities

    • Designing search forms to search this data

    • Assembling the entities into compound Business Views

    • Designing steppers for authoring data.

    • Designing duplicate managers to manually merge and split groups of similar records.

    • Creating workflows.

Version Management

The innovative Semarchy xDM technology supports an infinite number of metadata versions. The collaborative process between the different governance teams set the rules to close version of models to keep full traceability or to run multiple projects in parallel.
During the modeling phase, the data architect and business analysts create their metadata definition until the first semantically data model is finished. This model is then frozen in an edition and eventually delivered to production. Subsequent iterations of the data model are automatically opened. This allows Semarchy xDM users to replay the entire cycle of metadata definition. At any time, project managers may choose to branch the developments in order to develop two versions of the model in parallel.

Golden Data Certification

Semarchy xDM supports the integration of data from any source in batch mode, asynchronous or synchronous mode. The platform manages the lifecycle of data published in the hub. This data is pushed by publishers through existing middleware (ETL, SOA, EAI, Data Integration, etc.). The platform provides standard APIs such as SQL for ETLs, REST API, etc for real-time publishing. The certification process follows a sequence of steps to produce and make available certified reference data (Golden Records). These steps apply all the logical rules defined in the modeling phase to certify at all times the integrity, quality and uniqueness of the reference data.

With its Open Plug-in Architecture Semarchy xDM can delegate stages of the certification process to any component present in the information system infrastructure.

Generated Applications

Semarchy xDM supports Application generation. Applications provide secured and filtered access to the data stored in the hub using business-friendly Web interfaces. Applications also support data management operations and user collaboration.

Golden Data Consumption

The certified data is stored in a relational database, which allows Semarchy xDM to benefit from higher performance and scalability. The data is made available to consumers across multiple channels to allow a non-intrusive integration with the information system:

  • SQL access to reference data using JDBC or ODBC.

  • REST access to reference data.

Introduction to the Semarchy Workbench

Logging In to the Semarchy Workbench

To access Semarchy Workbench, you need an URL, a user name and password that have been provided by your Semarchy xDM administrator.

To log in to the Semarchy Workbench:

  1. Open your web browser and connect to the URL provided to you by your administrator. For example http://<host>:<port>/semarchy/ where <host> and <port> represent the name and port of host running the Semarchy xDM application. The Login Form is displayed.

  2. Enter your user name and password.

  3. Click Log In. The Semarchy xDM Welcome page opens.

  4. Click the Workbench button. The Semarchy xDM workbench opens on the Overview perspective.

Logging Out of the Semarchy Workbench

To log out of the Semarchy Workbench:

  1. In the Semarchy Workbench menu, select File > Log Out.

Opening a Model Edition

A model edition is a version of a model. This edition can be in a closed or open status. Only open editions can be edited.
Opening a model edition connects the workbench to this edition.

Before opening a model edition, you have to create the model and its first edition. To create a new model and manage model editions, refer to the Models Management chapter.

To open a model edition:

  1. In the Semarchy Workbench menu, select File > Open Model Edition

  2. In the Open a Model Edition dialog, expand the node for your model and then select a branch in the branch tree. The list of Model Editions for this branch is refreshed with the editions in this branch.

  3. Select an edition in this list and then click Finish.

  4. The Semarchy Workbench changes to the Model Edition perspective and displays the selected model edition.

Working with the Semarchy Workbench

The Semarchy Workbench is the graphical interface used by all Semarchy xDM users. This user interface exposes information in panels called Views and Editors .
A given layout of views and editors is called a Perspective .

The Semarchy Workbench

Working with Perspectives

There are several perspectives in Semarchy Workbench:

  • Overview: this perspective is the landing page in the Workbench. It allows you to access all the other perspectives and the applications.

  • Model Design: this perspective is used to edit or view a model.

  • Data Locations: this perspective is used to create data locations and deploy model editions in these locations.

  • Model Administration: this perspective is used to manage model editions and branches.

  • Administration Console: this perspective is used to administer Semarchy xDM and monitor run-time activity.

Working with Tree Views

When a perspective opens, a tree view showing the objects you can work with in this view appears.
This view appears on the left hand-side of the screen.
In this tree view you can:

  • Expand and collapse nodes to access child objects.

  • Double-click a node to open the object’s editor.

  • Right-click a node to access all possible operations with this object.

Working with the Outline

Certain perspectives includes an Outline view. This view shows the object open in the editor (and all its child objects) in a tree view.
This view appears on the left hand-side of the screen.
You can use the same expand, double-click, right-click actions in the outline as in the tree view.

Working with Editors

An object currently being viewed or edited appears in an editor in the central part of the screen.
You can have multiple editors opened at the same time, each editor appearing with a different tab.

Editor Organization

Editors are organized as follows:

  • The editor has a local toolbar which is used for editor specific operations. For example, refreshing or saving the content of an editor is performed from the toolbar.

  • The editor has a breadcrumb that allows navigating up in the hierarchy of objects.

  • The editor has a sidebar which shows the various sections and child objects attached to an editor. You can click in this sidebar to jump to one of these sections.

  • Properties in the editors are organized into groups, which can be expanded or collapsed.

Saving an Editor

When the object in an editor is modified, the editor tab is displayed with a star in the tab name. For example, Contact* indicates that the content of the Contact editor has been modified and needs to be saved.
To save an editor, either:

  • Click the Save button in the Workbench toolbar.

  • Use the CTRL+S key combination.

  • Use the File > Save option in the Workbench menu.

You can also use the File > Save All menu option or Save All toolbar button to save all modified editors.

Closing an Editor

To close an editor, either:

  • Click the Close (Cross) icon on the editor’s tab.

  • Use the File > Close option in the Workbench menu.

  • Use the Close option in the editor’s context menu (right-click on the editor’s tab).

You can also use the File > Close All menu option or the Close All option in the editor’s context menu (right-click on the editor’s tab) to close all the editors .

Accelerating Editing with CamelCase

In the editors and dialogs in Semarchy Workbench, the Auto Fill checkbox accelerates object creation and editing.
When this option is checked and the object name is entered using CamelCase, the object Label as well as the Physical Name is automatically generated.
For example, when creating an entity, if you type ProductPart in the name, the label is automatically filled in with Product Part and the physical name is set to PRODUCT_PART.

Duplicating Objects

The workbench supports duplicating model objects such as forms, collection, search forms, business views, etc.

To duplicate an object:

  1. Select the object to duplicate in the Model Edition tree view.

  2. Right-click and then select Duplicate.

A copy of the object is created.

It is also possible to duplicate a group of objects. When duplicating a group of objects that reference one another, the references in the copied objects are moved to the copies of the original objects.

For example:

  • When copying a single business view BV1 that references a collection C1, the copy of the business View (BV2) still references the collection C1.

  • When copying a business view BV1 with a collection C1 that it references (the group includes BV1 and C1), the copy of the business view BV2 references the copy of the collection C2.

To duplicate a group of objects:

  1. Select the multiple objects to duplicate in in the Model Edition tree view. Press the CTRL key to enable multiple selection.

  2. Right-click and then select Duplicate.

A copy of the group of objects is created. The links are made, if possible, to the copied objects.

Deleting Objects

The Semarchy xDM model is designed to maintain its consistency.

When deleting an object (for example using the Delete context menu action) that is referenced by other objects, an alert Unable to Delete Object appears.

This dialog indicates which references prevent the object deletion. Remove these references before deleting this object.

The Model Diagram

The Model Diagram Editor shows a graphical representation of a portion of the model or the entire model.
Using this diagram, you can create entities and references in a graphical manner, and organize them as graphical shapes.

The diagram is organized in the following way:

  • The Diagram shows shapes representing entities and references.

  • The Toolbar allows you:

    • to zoom in and out in the diagram.

    • to select an automatic layout for the diagram and apply this layout by clicking the Auto Layout button.

    • to select the elements to show in the diagram (attributes, entities, labels or names, foreign attributes, etc)

  • The Palette provides a set of tools:

    • Select allows you to select, move and organize shapes in the diagram. This selection tool allows multiple selection (hold the Shift or CTRL keys).

    • Add Reference and Add Entity tools allow you to create objects.

    • Add Existing Entities allows you to create shapes for existing entities.

After choosing a tool in the palette, the cursor changes. Click the Diagram to use the tool. Note that after using an Add… tool, the tool selection reverts to Select.

For more information about the model diagrams, see the Diagrams section in the Logical Modeling chapter.

The Workflow Diagram

The workflow diagram shows a graphical representation of a workflows.
Using this diagram, you can design tasks and transitions for a workflow.

The diagram is organized in the following way:

  • The Diagram shows shapes representing tasks (rectangles) and transitions (links), as well as the start and end events (round shapes).

  • The Toolbar allows you:

    • to zoom in and out in the diagram.

    • to select an automatic layout for the diagram and apply this layout by clicking the Auto Layout button.

    • to select the elements to show in the diagram (name/labels on the tasks and transitions).

    • to validate the workflow. The validation report shows the errors of the workflow.

  • The Palette provides a set of tools:

    • Select allows you to select, move and organize shapes in the diagram. This selection tool allows multiple selection (hold the Shift or CTRL keys).

    • Add Task and Add Transition tools allow you to modify the workflow.

  • The Properties view allows you to edit the properties of the task or transition selected in the diagram.

In this diagram, you can drag the transition arrows to change the source and target tasks of a transition.

For more information about the workflow diagrams, see the Workflows section in the Working with Applications chapter.

Working with Other Views

Other views (for example: Progress, Validation Report) appear in certain perspectives. These views are perspective-dependent.

Workbench Preferences

User preferences are available to configure the workbench behavior. These preferences are stored in the repository and are specific to each user connecting to the workbench. The preferences are applied regardless of the client computer or browser used by the user to access the workbench.

Setting Preferences

Use the Window > Preferences dialog pages to set how you want the workbench to operate.

You can browse the Preferences dialog pages by looking through all the titles in the left pane or search a smaller set of titles by using the filter field at the top of the left pane. The results returned by the filter will match both Preference page titles and keywords such as general and stewardship.

The arrow controls in the upper-right of the right pane enable you to navigate through previously viewed pages. To return to a page after viewing several pages, click the drop-down arrow to display a list of your recently viewed preference pages.

The following preferences are available in the Preferences dialog:

  • General Preferences:

    • Date Format: Format used to display the date values in the workbench. This format uses Java’s Simple Date Format patterns.

    • DateTime Format: Format used to display the date and time values in the workbench. This format uses Java’s Simple Date Format patterns.

    • Link Perspective to Active Editor: Select this option to automatically switch to the perspective related to an editor when selecting this editor.

Exporting and Importing User Preferences

Sharing preferences between users is performed using preferences import/export.

To export user preferences:

  1. Select File > Export. The Export wizard opens.

  2. Select Export Preferences in the Export Destination and then click Next.

  3. Click the Download Preferences File link to download the preferences to your file system.

  4. Click Finish to close the wizard.

To import user preferences:

  1. Select File > Import. The Import wizard opens.

  2. Select Import Preferences in the Import Source and then click Next.

  3. Click the Open button to select an export file.

  4. Click OK in the Import Preferences Dialog.

  5. Click Finish to close the wizard.

Importing preferences replaces all the current user’s preferences by those stored in the preferences file.

Working with SemQL

This section provides a quick introduction to the SemQL language. For a detailed description of this language with examples, refer to the Semarchy xDM SemQL Reference Guide.

SemQL is a language to express declarative rules in Semarchy Workbench.

SemQL Language Characteristics

The SemQL Language has the following characteristics:

  • The syntax is similar to the SQL language and most SemQL functions map directly to database functions.

  • SemQL is converted on the fly and executed by the hub database.

  • SemQL uses Qualified Attribute Names instead of columns names. The code remains implementation-independent.

Qualified Attribute Names

A Qualified Attribute Name is the path to an attribute from the current entity being processed.
This path not only allows accessing the attributes of the entity, but also allows access to the attributes of the entities related to the current entity.

Examples:

  • FirstName: Simple attribute of the current Employee entity.

  • InputAddress.PostalCode: Definition Attribute (PostalCode ) of the InputAddress Complex Attribute used in the current Customer entity.

  • CostCenter.CostCenterName: Current Employee entity references the CostCenter entity and this expression returns an employee’s cost center name

  • CostCenter.ParentCostCenter.CostCenter: Same as above, but the name is the name of the cost center above in the hierarchy. Note that ParentCostCenter is the referenced role name in the reference definition.

  • Record1.CustomerName: CustomerName of the first record being matched in a matcher process. Record1 and Record2 are predefined qualifiers in the case of a matcher to represent the two records being matched.

SemQL Syntax

The SemQL Syntax is similar to the SQL language for creating expressions , conditions and order by clauses.

  • Expressions contain functions and operators, and return a value of a given type.

  • Conditions are expressions returning true or false. They support AND, OR, NOT, IN, IS NULL, LIKE, REGEXP_LIKE, Comparison operators (=, !=, >, >=, <, <=), etc.

  • Order By Clauses are used to sort records in a set of results using an arbitrary combination of Expressions sorted ascending or descending

In expressions, conditions and order by clauses, it is possible to use the SemQL functions. The list of SemQL functions is provided in the SemQL Editor

SemQL is used to create expressions, conditions and order by clauses. SELECT, UPDATE or INSERT queries are not supported, as well as joins, sub-queries, aggregates, in-line views and set operators.

SemQL Examples

Example 1. Enricher Expressions
  • FirstName = InitCap(FirstName)

  • Name = InitCap(FirstName) || Upper(FirstName)

  • City = Replace(Upper(InputAddress.City),'CEDEX','')

In these examples, InitCap, Upper and Replace are SemQL functions. The concatenate operator || is also a SemQL operator.

Example 2. Validation Conditions

The following condition checks the quality of the InputAddress complex attribute of the customer entity:

InputAddress.Address is not null and ( InputAddress.PostalCode is not null or InputAddress.City is not null)

In this example, the IS NOT NULL, AND and OR SemQL operators are used to build the condition.

Example 3. Matcher

The following binning expression groups customers by their Country/PostalCode:

InputAddress.Country || InputAddress.PostalCode

The following matching condition matches two customer records by name, address and city name similarity:

SEM_EDIT_DISTANCE_SIMILARITY( Record1.CustomerName, Record2.CustomerName ) > 65
and SEM_EDIT_DISTANCE_SIMILARITY( Record1.InputAddress.Address, Record2.InputAddress.Address ) > 65
and SEM_EDIT_DISTANCE_SIMILARITY( Record1.InputAddress.City, Record2.InputAddress.City ) > 65

In this last example, SEM_EDIT_DISTANCE_SIMILARITY is a SemQL function. Record1 and Record2 are predefined names for qualifying the two records to match.

The SemQL Editor

The SemQL editor can be called from the Workbench when a SemQL expression, condition or clause needs to be built.

The SemQL Editor

This editor is organized as follows:

  • Attributes available for the expression appear in the left panel. Double-click an attribute to add it to the expression.

  • Functions declared in SemQL appear in the left bottom panel, grouped in function groups. Double-click a function to add it to the expression. You can declare your own database functions in the model so that they appear in the list of functions.

  • Messages appear in the right bottom panel, showing parsing errors and warnings.

  • Description for the selected function or attribute appear at the bottom of the editor.

  • The Toolbar enables you to to indent the code or hide/display the various panels of the editor and to undo/redo code edits.

Declaring Database Functions and Procedures

You can use customized functions in SemQL implemented in the database hosting the data location. You declare these functions in the model to have them appear in the list of functions. You can also use database procedures declared in the model, as Triggers in Applications.

Both the functions and procedures are declared using the same mechanism.

Note that:

  • Functions that are not declared can still be used in SemQL, but will not be recognized by the SemQL parser and will cause validation warnings.

  • Only procedures can be used as triggers, as triggers do not expect a returned value.

To declare a database function or procedure:

  1. Right-click the Database Functions node in the Model Edition view and select Add Database Function. The Create New Database Function wizard opens.

  2. In the Create New Database Function wizard, enter the following values:

    • Name: Name of the function or procedure. This name must exactly match the name of the function in the database. Note that if this function is part of a package, the function name must be prefixed by the package name.

    • Schema: The schema containing this function/package.

    • Categories: Enter or select the function categories of the SemQL Editor into which this function should appear.

    • Check the Procedure option if you want to declare a procedure.

  3. Click Finish to close the wizard. The Database Function editor opens.

  4. In the Description field, enter a detailed description for the function.

  5. Click the Add Argument button in the Function Arguments section to declare an argument for the function.
    The new argument is added to the Function Arguments table.

    1. Edit the Name of this argument in the Function Arguments table.

    2. Select whether this argument is Mandatory and whether it is an Array of values.

  6. Repeat the previous step to declare all the arguments.

  7. Use the Move Up and Move Down buttons to order the argument according to your function implementation.

  8. Press CTRL+S to save the editor.

  9. Close the editor.

Only declare the Schema if the function is available in this given schema in all the environments (development, test, productions) into which the model will be deployed. If it is not the case, it is recommended not to use a schema name and to create synonyms to make your function available in all environments from the data location schema.
A Mandatory argument cannot follow a non-mandatory one. An Array argument must always be the last argument in the list.

Working with Properties

Literal vs. SemQL Properties

When designing a model or application, most properties support literal values. For example, the Label of a field is a literal value. Certain properties in application components also support SemQL values, which are values based on the record data.

When a field supports literal and SemQL, a value type drop down appears aside the field to select the type of value. (Literal or SemQL).

  • If you select Literal, the property editing component adapts to the property type (boolean, string, etc).

  • If you select SemQL, the property editing component adapts gives you access to a SemQL editor. Note that if the property is a boolean, then the SemQL editor requires a condition, and not an expression.

Example 4. Literal vs. SemQL Properties

In the example below, the Visible property is configured as a literal. As this property is a boolean, it appears as a checkbox. In that case, Visible is always true, and this field is always visible.

Literal Property Example

If the same property is now configured as SemQL, you can provide a SemQL expression to define its value. In this second case, Visible is true if the value of the Label attribute is not null. The field will be visible or invisible depending on the data.

SemQL Property Example

In both these example, look at the Relative Width or Component Type properties. They only support literal values and do not have the value type selector.

Colors

Certain properties store color values. For example, the Label Color for a form field, or the Icon Color for an application action. Colors in Semarchy xDM applications are defined using three possible formats:

  • CSS Color: css: <css-color>, where <css-color> is a CSS color value.
    Examples:

    • css: rgb(255, 255, 155)

    • css: rgba(255, 255, 155, 0.5)

    • css: #FFFFFF

    • css: grey

  • Theme Color: theme: <intention>[ <hue>], where <intention> is one of primary, accent, warn and <hue> is one of default, hue-1, hue-2, hue-3. See Material Design Color for more information.
    Examples

    • theme: primary

    • theme: primary hue-1

  • Material Design Palette Color: md:<color>[ <shade>], where <color> is one of named Material Design color and <shade> is one of the hues defined for this color (500, A500…). See Material Design Color for more information.
    Examples:

    • md: deep-purple

    • md: indigo A500

Images

Certain properties and values store images or references to images or icons. For example, the Primary Image or Avatar for a display card, or the Icon Color for an application action. An image or icon in a Semarchy xDM application is displayed from one of the following source types:

  • Binary Data, stored in the hub. This is the case for form or collection attributes images that display attributes with a binary data type. The form or collection component is configured with the Image Source property set to Content.

  • External URL, referring to images stored on another server. This is the case for form or collection images that display attributes containing an URL string. The form or collection component is configured with the Image Source property set to URL. Such an URL starts with http:// or https://

  • Image Library URL. This is the case for icons selected in the application, or form or collection image that display a literal or computed value containing an URL starting with images://. An Image Library URL is structured as follows: images://<library-name>/<file-name>. See Managing Image Libraries for more information about images and image libraries.

Composing Image URLs

It is possible to compose external or image library URLs using SemQL.

For example, the myApp image library contains icons named priority_1.svg, priority_2.svg, etc.
The following SemQL expression, used in the customer display card avatar, will make the avatar appear with an icon from the image library that depends on the value of the cust_priority attribute.

'images://myApp/priority_' || cust_priority || '.svg'

Markdown

Markdown is a markup language with a simple text formatting syntax. Semarchy xDM is able to render Markdown text as HTML in the applications:

  • Certain properties, such as Documentation, support this language.

  • The Markdown component can be used in your applications to render and author Markdown text stored in entities' attributes.

Semarchy xDM support the following Markdown flavors:

Refer to this example to get started with the markdown syntax, and to the CommonMark and GFM specifications for a detailed reference of the markdown language.

Working with Plug-ins

Plug-ins allow extending the capabilities of the Semarchy xDM using Java code and external APIs.
Plug-ins are used to implement enrichers or validations not feasible in SemQL.

The plug-ins have the following characteristics:

  • All plug-ins take Inputs (mapped on attributes) and Parameters, which may be mandatory (or not).

  • Enricher Plug-ins return Outputs (mapped on attributes to enrich). A subset of these outputs may be used for enriching attributes.

  • Validation Plug-ins return a Boolean value indicating whether or not the values of the input are valid.

Examples of plug-ins:

  • Enricher: Geocoding using an external API (Google, Yahoo, etc.)

  • Validation: Pattern Matching, Validating against an external API, etc.

More information:

Logical Modeling

Logical modeling allows defining the business entities that make up the model.

Introduction to Logical Modeling

In Semarchy xDM, modeling is performed at the logical level. You do not design physical objects such as tables in a model, but instead design logical business entities independently from their implementation.

Logical Model Objects

Logical modeling includes creating the following objects:

  • Customized types such as User-Defined Types, Complex Types and List of Values reused across the model.

  • Entities representing the data records with their Attributes.

  • Reference Relationships linking the entities of the model, for example to build hierarchies.

  • Constraints such as Validations, Unique Keys and Mandatory attributes.

  • Display Cards, Forms, Search Forms and Collections defining how the entities are used in an Application. These objects are explained in the Working with Applications chapter.

  • Named Queries defining structured integration endpoints, accessed to consume data using a REST API.

In the modeling phase, certification process objects such as Enrichers or Validations are created. Creating these objects is described in the Certification Process Design chapter. After completing the modeling phase, you can create Applications to access the data stored in your model.

Objects Naming Convention

When designing a logical model, it is necessary to enforce a naming convention to guarantee good readability of the model and a clean implementation.
There are several names and labels used in the objects in Semarchy xDM:

  • Internal Names (Also called Names) are used mainly by the Model designers and are unique in the model. They can only contain alphanumeric characters, underscores and must start with a letter.

  • Physical Names and Prefixes are used to create the objects in the database corresponding to the logical object. These can only contain uppercase characters and underscores.

  • Labels and Descriptions are visible to the users (end-users and data stewards) consuming data through the UI. These user-friendly labels and descriptions can be fixed at later stages in the design. They are externalized and can be localized (translated) in the platform.

The following tips should be used for naming objects:

  • Use meaningful Internal Names. For example, reference relationships should all be named after the pattern <entity name><relation verb><entity name> , like CustomerHasAccountManager .

  • Do not try to shorten internal names excessively. They may become meaningless. For example, using CustAccMgr instead of CustomerHasAccountManager is not advised .

  • Use the CamelCase for internal names as it enables the use of the Auto fill feature. For example, ContactBelongsToCustomer, GeocodedAddressType.

  • Define team naming conventions that accelerate object type identification. For example, types and list of values can be post-fixed with their type such as GeocodedAddressType, GenderLOV.

  • Define user-friendly Labels and Descriptions. Internal Names are for the model designers, but labels and descriptions are for end users.

The following table lists a typical naming convention you can adapt to your project.

Object Name Pattern Examples

Model

[Model Name]

StoresAndSuppliersMDM, ItemCatalog

Entity

[Entity Name]

Product, Item, Customer

Attribute

[Attribute Name]

IsProductAvailable

Reference

[From Entity]Has[To Entity][Role]

EmployeeHasEmployeeManager

Reference Role To

[Role Name In Singular]

Manager

Reference Role From

[Role Name In Plural]

ManagedEmployees

Complex Type

[ComplexType Name]Type

AddressType

User-Defined Type

[Type Name]Type

SimpleStringType

Publisher (Code)

[CODE]

MDM

Publisher (Name)

[Publisher Name]

MDM Application

Job

[JOB_NAME][JOB_TYPE]

INITIAL_LOAD_FULL, CERTIFY_PRODUCT_SIMPLE

Form

[Form Name or Kind]Form

DefaultForm, ProductAuthoringForm, ProductReviewForm, MediaSimpleForm, ItemsMainAttributesForm

Collection

[Collection Name or Kind]Collection

DefaultCollection, AllAttributesCollection

List Of Values

[LOV Name]LOV

CurrencyLOV, ProductTypesLOV

Business View

[Business View Name]([optional Classification])

Products, ProductsByCategory, EmployeeHierarchy

Transition Name

[Reference Role From]

ManagedEmployees

Stepper

Author[Entity Name]s

AuthorProducts, AuthorCustomers

Stepper Collection Step

[Entity Name]s or [Referencing Role Name]

Products, Customers, RelatedItems

Stepper Form Step

[Entity Name]

Product, Customer, Item

Action Set

[Action Set Name Or Purpose]ActionSet

CustomerActionSet, ProductImportExportActionSet, ProductCreateEditActionSet

Application

[Application Name]

ProductSearch, ProductDesign, ReferenceDataManagement

Folder

[Folder Name]

ProductsManagement

Workflow

[Workflow Name]

CreateOrModifyCustomers, InitiateFullProductCreation, RequestProductLabelChange

Workflow Task

[Task Purpose]

CreateProductBasicInformation, AddTechnicalDetailsForProduct, VerifyProductCreationGuidelines

Workflow Transition

[Transition Verb/Action]

SendDataForApproval, RejectRequestAndAskForMoreInformation, AcceptAndSendToMarketing

Model Validation

A model may be valid or invalid. An invalid model is a model that contains a number of design errors or warnings.

When a model contains errors:

  • You cannot deploy this model

  • You cannot close this model edition.

  • Applications from this model can no longer be accessed. This is frequent at design-time, as you deploy the model first, perform multiple changes while refreshing the application.

If you face issues with a model or an application, run a model validation to make sure that the model does not have new issues.

To validate the model:

  1. In the Model Edition view of the Model Design Perspective, select the model node at the root of the tree. You can alternately select on entity to validate only this entity.

  2. Right-click and select Validate.

  3. If you have unsaved editors, select those to save when prompted.

  4. The validation process starts. At the end of the process, the list of issues (informations, errors and warnings) is displayed in the Validation Report view.

In the validation report:

  • Double-click an issue to open the object causing this issue.

  • Right-click and select Validation Details to view the details of the selected issue.

  • Right-click and select Export Validation Report to export all the validations in a CSV file.

It is recommended to regularly run the validation on the model or on specific entities. Validation may guide you in the process of designing a model. You can perform regular validations to assess how complete the model really is, and you need to pass model validation before deploying or closing a model edition.

Generating the Model Documentation

When a model is complete or under development, it is possible to generate a documentation set for this model.

This documentation set may contain the following documents:

  • The Logical Model Documentation, which includes a description of the logical model and the rules involved in the certification process.

  • The Applications Documentation, which includes a description of the applications of the model and their related components (views, business view, etc).

  • The Physical Model Documentation, which includes a description of the physical tables generated in the data location when the model is deployed. This document is a physical model reference document for the integration developers.

The documentation is generated in HTML format and supports hyperlink navigation within and between documents.

To generate the model documentation:

  1. In the Model Edition view of the Model Design Perspective, select the model node at the root of the tree.

  2. Right-click and select Export Model Documentation.

  3. In the Model Documentation Export dialog, select the documents to generate.

  4. Select the appropriate Encoding and Locale for the exported documentation. The local defines the language of the generated documentation.

  5. Click OK to download the documentation.

The documentation is exported in a zip file containing the selected documents.

It is possible to export the model documentation only for a valid model.

Types

Several types can be used in the Semarchy xDM models:

  • Built-in Types are part of the platform. For example: string, integer, etc.

  • List of Values (LOVs) are a user-defined list of string codes and labels. For example: Gender (M:Male, F:Female), VendorStatus (OK:Active, KO:Inactive, HO:Hold).

  • User-Defined Types are a user restriction on a built-in type. For example the GenericNameType type can be defined as a String(80) and the ZipCodeType can be used as an alias for Decimal (5,0).

  • Complex Types are a customized composite type made of several Definition Attributes using Built-in Type, User-Defined Type or a List of Values. For example, an Address complex type has the following definition attributes: Street Number, Street Name, Zip Code, City Name and Country.

All these types, as well as the user-defined types, are reused across the entire model.

Lists of values, user-defined types or complex types are used by reference in the rest of the model. Changes performed to such a type directly impact the entities and attributes using this type. To list the attributes using a type and analyze the impact of changing a type, open the editor for this type and then select the Used in item in the left sidebar.

Built-in Types

Built-in types are provided out of the box in the platform.

Built-in types include:

  • Numeric Types:

    • ByteInteger: 8 bytes signed. Range [–128 to 127]

    • ShortInteger: 16 bytes signed. Range [–32,768 to –32,767]

    • Integer: 32 bytes signed. Range [–2^32 to (2^32-1)]

    • LongInteger: 64 bytes signed. Range [–2^64 to (2^64-1)]

    • Decimal: Number(Precision,Scale), where Precision in [1-38] and Scale in [–84-127]

  • Text Types:

    • String: Length smaller than 4000 characters.

    • LongText: No size limit – Translated to a CLOB in the database

  • Date Types:

    • Date: Date with no time or timezone.

    • Timestamp: Date and Time (to fractional digits of a Second) with timezone. See the note below for more information about timestamps and time zones.

  • Binary Types:

    • Binary: Store any type of binary content (image, document, movie, etc.) with no size limit – Translated to a BLOB in the database

  • Misc. Types:

    • UUID: 16 bytes Global Unique ID

    • Boolean: 1 character containing either `1' (true) or `0' (false).

Timestamps Values and Time Zones Conversion

Timestamp values contain the date and time, and are therefore subject to time zones conversions:

  • Timestamps are stored in the repository and data locations in the time zone of the application server.

  • Timestamps used in applications:

    • Users view or author timestamps in the timezone defined in their user profile. The values are automatically converted from/to the application server timezone when read/written to the database.

    • Import or export is an exception. Users import or export timestamps in the application server timezone, independently from their profile timezone. This allows exporting timestamps in one timezone and reimporting them in a different timezone without altering the values.

  • Timestamps in integration:

    • The REST API uses the ECMAScript standard format (YYYY-MM-DDTHH:mm:ss.sssZ) for timestamps. This format includes an explicit timezone part and conversion is performed automatically.

    • Integration specialists using SQL to consume data from or publish data to the MDM hub should be aware that conversion may happen depending on their driver configuration. It is recommended to have the integration component located in the same timezone as the application server to avoid such conversion.

List of Values

List of Values (LOVs) are a user-defined list of code and label pairs.
They are limited to 1,000 entries and can be imported from a Microsoft Excel Spreadsheet.

Each value in the list has a Code, which is stored in the hub. A value is usually displayed to the users using its Label, supported by its Description and Image URL.

Examples:

  • Gender (M:Male, F:Female)

  • VendorStatus (OK:Active, KO:Inactive, HO:Hold)

Lists of values are limited to 1,000 entries. If a list of value needs to contain more than 1,000 entries, you should consider implementing in the form of an entity instead.

To create a list of values:

  1. Right-click the List of Values node and select Add List of Values…. The Create New List of Values wizard opens.

  2. In the Create New List of Values wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the list of values.

    • Label: User-friendly label in this field. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Length: Length of the code for the LOV.

  3. Click Finish to close the wizard. The List of Values editor opens.

  4. In the Description field, optionally enter a description for the user-defined type.

  5. Add values to the list using the following process:

    1. In the Values section, click the Add Value button. The Create New LOV Value dialog appears.

    2. In this dialog, enter the following values:

      • Code: Code of the LOV value. This code is the value stored in an entity attribute.

      • Label: User-friendly label displayed for a field having this value.

      • Description: Long description of this value.

    3. Click Finish to close the dialog.

  6. Repeat the previous operations to add more values. You can select a line in the list of value and click the Delete button to delete this line. You can also double-click a value to edit its properties, including the description and image URL.

  7. Press CTRL+S to save the editor.

  8. Close the editor.

List of values can be entered manually as described above and can be translated.

In addition, you can also import values or translated values from a Microsoft Excel Spreadsheet.
This spreadsheet must contain only one sheet with four columns containing the Code, Label, Image URL and Description values. Note that the first line of the spreadsheet will be ignored in the import process.

To import a list of values from an excel spreadsheet:

  1. Open the editor for the list of value.

  2. Expand the Values section.

  3. In the Values section, click the Import Values button. The Import LOV Values wizard appears.

  4. Use the Open button to select a Microsoft Excel spreadsheet.

  5. Select Merge to merge the imported list with the existing one. If you un-select this option, the imported list replaces the existing one, possibly removing values that do not exist in the import file.

  6. Click Next . The changes to perform are computed and a report of object changes is displayed.

  7. Click Finish to perform the import. The Import LOV wizard closes.

  8. Press CTRL+S to save the editor.

  9. Close the editor.

User-Defined Types

User-Defined Types (UDTs) are a user restriction on a built-in type. They can be used as an alias to a built-in type, restricted to a given length/precision.

Examples:

  • GenericNameType type can be defined as a String(80)

  • ZipCodeType can be used as an alias for Decimal(5,0).

To create a user-defined type:

  1. Right-click the User-defined Types node and select Add User-defined Type…. The Create New User-defined Type wizard opens.

  2. In the Create New User-defined Type wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the user-defined type

    • Label: User-friendly label in this field. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Built-in Type: Select a type from the list.

    • Length, Precision, Scale. Size for this user-defined type. The fields available depend on the built-in type selected. For example a String built-in type will only allow entering a Length.

  3. Click Finish to close the wizard. The User-defined Type editor opens.

  4. In the Description field, optionally enter a description for the user define type.

  5. Press CTRL+S to save the editor.

  6. Close the editor.

Complex Types

Complex Types are a customized composite type made up of several Definition Attributes using Built-in Type, User-Defined Type or a List of Values.

For example, an Address complex type has the following definition attributes: Street Number, Street Name, Zip Code, City Name and Country.

To create a complex type:

  1. Right-click the Complex Types node and select Add Complex Type…. The Create New Complex Type wizard opens.

  2. In the Create New Complex Type wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

  3. Click Finish to close the wizard. The Complex Type editor opens.

  4. In the Description field, optionally enter a description for the complex type.

  5. Select the Definition Attributes item in the editor sidebar.

  6. Repeat the following steps to add definition attributes to this complex type:

    1. Select the Add Definition Attribute… button. The Create New Definition Attribute wizard opens.

    2. In the Create New Definition Attribute wizard, check the Auto Fill option and then enter the following values:

      • Name: Internal name of the object.

      • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

      • Physical Column Name: Name of the physical column containing the values for this attribute. This column name is prefixed with the value of the Physical Prefix specified on the entity complex attribute of this complex type.

      • Type: List of values, built-in or user-defined type of this complex attribute.

      • Length, Precision, Scale. Size for this definition attribute. The fields available depend on the built-in type selected. For example a String built-in type will only allow entering a Length. If a list of values or a user-defined type was selected, these values cannot be changed.

      • Mandatory: Check this box to make this definition attribute mandatory when the complex type is checked for mandatory values.

    3. Click Finish to close the wizard. The new definition attribute appears in the list. You can double-click the attribute in the list to edit it further and edit its advanced properties. It is recommended to define the user facing Documentation for the attribute. Use plain text or the Markdown syntax for rich text. This text provides detailed documentation for the attribute and appears in the documentation sidenav.

  7. Press CTRL+S to save the Complex Type editor.

  8. Close the editor.

A complex type has the following advanced properties that impact its behavior:

  • Mandatory: When an entity attribute is checked for mandatory values, and this attribute uses a complex type, each of the definition attributes of this complex type with the mandatory option selected are checked.

  • Searchable: This option defines whether this attribute is used for searching.

  • Translated: Options reserved for a future use.

  • Multi-Valued: This option applies to definition attributes having the type list of values. Checking this box allows the definition attribute to receive several codes in the list of values, separated by the Value Separator provided. For example, a multi-valued Diplomas field can receive the DM, DP, DPM codes meaning that that contact is Doctor of Medicine, Pharmacy and Preventive Medicine.

Complex Type Display Name

When a complex attribute value needs to be displayed in a compact form (for example, in a table, or in a single field), the Display Name is used.

The display name defines how a complex attribute is displayed in compact form. It is a concatenation of several definition attributes, separated by a Separator.
For example, the GeocodedAddress complex type contains a large number of attributes, from the simple StreetNumber down to the longitude and latitude. A display name would be for example: StreetNumber StreetName City Country .

To create or modify a complex attribute display name:

  1. Right-click the complex attribute node and select Define Display Name…. The Modify Display Name wizard opens.

  2. In the Modify Display Name wizard, enter the following values:

    • Separator: String that will separate the selected definition attributes in the display name.

  3. Click Next.

  4. In the Display Name Attributes page, select the Available Attributes you want to add and click the Add >> button to add them to the Selected Attributes.

  5. Use the Move Up and Move Down buttons to order the selected attributes.

  6. Click Finish to close the wizard.

  7. Press CTRL+S to save the Display Name editor.

  8. Close the editor.

Only one display name can be created for a given complex type.

Entities

Entities are the key components of the logical modeling. They are not database tables, but they represent Business Entities of the domain being implemented in the MDM Project. Example of entities: Customers, Contacts, Parties, etc.

Entity Characteristics

Entities have several key characteristics. They are made of Attributes and References.

Attributes

Entities have a set of properties, called Attributes. These attributes can be either:

  • Simple Attributes using built-in types, user-defined types or lists of values created in the model.

  • Complex Attributes using complex types created in the model.

For example, the Contact entity may have the following attributes:

  • FirstName and LastName: Simple attributes using the user-defined type called GenericNameType

  • Comments: Simple attribute using the built-in type LongText.

  • Gender: Simple attributes based on the GenderLov list of values.

  • Address: Complex Attribute using the GeocodedAddress complex type.

Entity Type

Each entity has a given entity type. This type expresses the entity capabilities for match/merge and authoring. Entity types are:

  • Basic: Basic entities do not support match and merge of records. They assume that data comes from a single data source. This entity type is suitable for entities for which data is authored (or imported) exclusively in the hub, or for simple reference data entities.

  • ID Matched (formerly known as UDPK): ID matched entities support match and merge of records. They assume that data comes from many data sources, and similar records share an identifier common to all sources. Records in entities using ID Matching are matched if they have the same ID and then merged into golden records. This entity type is well suited when there is a true unique identifier for all the applications communicating with the MDM hub.

  • Fuzzy Matched (formerly known as SDPK): Fuzzy matched entities support match and merge of records. They assume that records comes from many data sources that do not share a common identifier. Records need to be matched using their data content. Records in entities using Fuzzy Matching are matched using a set of match rules defined in a Matcher.

The choice of an Entity Type is important. Please take into account the following differentiators when creating an entity.
Basic Entity
  • Basic means that this entity does not support match and merge.

  • With this type of entity, you must assume that all data comes from a single (de-duplicated) source or is authored exclusively in the MDM hub.

  • When authoring or loading data in a basic entity, you simply overwrite the existing golden record with the new data, possibly keeping track of the changes. There is no notion of multiple master records merging into a golden record.

  • This type of entity is particularly suitable for reference data or classification data, which are typically managed only in the hub.

  • Due to the simple nature of these entities, the certification process is simpler and faster for basic entity records than for ID or fuzzy matched entity records.

  • A Matcher can be defined for the entity, for detecting potential duplicates when manually creating records in the hub.

Use Basic Entities for data that only exist in the hub, or for data coming from a single data source into which there are no duplicates.

ID Matched Entity
  • ID Matching assumes that the multiple applications providing data to this entity share a common ID. This ID can be used as the unique identifier, even for the golden records.

  • This ID is stored into a single attribute which will be the golden data Primary Key. If the ID in the information system is composed of several columns, you must concatenate these values into the PK column.

  • As this ID is common to all systems, similar records are always matched using the ID. A Survivorship Rule defines how they are consolidated into a single golden records, and how users can override the consolidated values.

  • Although ID matching is faster than fuzzy matching, it still requires consolidating multiple records into a single golden record. The certification process for ID matched records is slower than for basic entity records.

  • A Matcher can be defined for the entity, for detecting potential duplicates when manually creating records in the hub.

  • When authoring an ID matched entity record in a stepper, you may create a new record that only exists in the hub, or override the consolidated values. The Survivorship Rule defines how data creation or override takes place for attributes.

Use ID Matching when you need to match and merge records from various sources and have a truly unique and shared identifier for all these sources.

Fuzzy Matched Entity
  • Fuzzy Matching means that applications in the enterprise have different IDs, and Semarchy xDM needs to generate a unique identifier (Primary Key - PK) for the golden records. This PK can be either a sequence or a Unique ID (UUID).

  • Similar records may exist various systems, representing the same master data, with different IDs. These similar records must be matched using fuzzy matching methods that compare their content.

  • A Matcher defines how similar master records are matched. A Survivorship Rule defines how they are consolidated into a single golden records, and how users can override the consolidated values.

  • Duplicate Managers define the user interfaces into which users review, merge or split groups of matching records.

  • Due to the complex processing involved to fuzzy-match and then merge records, the certification process for fuzzy-matched records is slower than for ID matched or basic entity records.

  • When authoring a fuzzy matched entity record in a stepper, you may create a new record that only exists in the hub, or override the consolidated values. The Survivorship Rule defines how data creation or override takes place for attributes.

Use Fuzzy Matching when you need to match and merge records from various sources and do not have a shared identifier for all these sources.

ID Generation

The entity type impacts the method used for generating the record IDs:

  • Basic Entities: The Golden Record Primary Key is the ID sent or authored when creating a record. When creating records, this ID may be:

    • Manually provided by users

    • Automatically generated using a Sequence, Universally Unique Identifier generator (UUID) or a SemQL expression.

  • ID Matched Entities: The Golden Record Primary Key is also the ID that exists in the source systems. When creating source/master records, this ID may be:

    • Manually provided by users.

    • Automatically generated using a Sequence, Universally Unique Identifier generator (UUID) or a SemQL expression.

  • Fuzzy Matched Entities: The Golden Record Primary Key is managed and always generated by the system, using a Sequence or UUID.
    When creating source/master records, this source/master ID may be:

    • Manually entered by users

    • Automatically generated using a Sequence, Universally Unique Identifier generator (UUID) or a SemQL expression.

When generating IDs automatically using a Sequence in data entry forms for an ID Matched entity, you must take into account records pushed by other publishers (using for example a data integration tool).
These publishers may use the same IDs for the same entity, and in this case the records will match by ID. If you want to separate records entered manually from other publishers' records and avoid unexpected matching, configure your sequence using the Start With option to start beyond the range of IDs used by the other publishers.
An ID generated with a SemQL expression is immutable. This ID will not change after the initial record creation even if the value of the attributes used in the expression change. Such an ID is created when a record form is saved for the first time, for example when a record is imported from a file containing no IDs.
References

Entities are related using Reference Relationships. A reference relationship defines a relation between two entities. For example, an Employee is related to a CostCenter by the EmployeeHasCostCenter relation.

Constraints

Data quality rules are created in the design of an entity. These constraints include:

  • Mandatory columns

  • List of Values range check

  • Unique Key

  • Record level Validations.

  • Reference Relationships

These constraints are checked on the source records and the consolidated records as part of the certification process. They can also be checked to enforce data quality in data authoring.

Inheritance

Entities can extend other entities (Inheritance). An entity (child) can be based on another entity (parent).
For example, the PersonParty and OrganizationParty entities inherit from the Party entity.
They share all the attributes of their parent but have specificities.

When inheritance is used:

  • The child entity inherits the following elements: Attributes, Unique Keys, Validations, Enrichers and References.

  • Matchers and Survivorship Rules are not inherited

  • It is not possible to modify the entity type. The child inherits from the parent entity type.

  • It is possible to add elements to the child entity: Attributes, Unique Keys, Validations, Enrichers and References.

  • The underlying physical tables generated for the child entities and parent entity are the same. They contain a superset of all the attributes in the cluster of entities.

  • Deleting an inherited attribute on a child entity removes it from the parent entity, and by extension from all the child entities inheriting this attribute from the parent.

  • You cannot modify the entity inheritance from the Entity editor and must use the Alter Entity option. See Altering an Entity for more information.

  • With entities inheriting attributes from a parent and having additional attributes, you can order and delete (non-inherited) attributes from the entity’s Inherited Attributes (All) section. It is not possible to order additional attributes before the inherited attributes. It is not possible to delete inherited attributes.

Inheritance Limitations & Recommendations

Although inheritance looks like a good way to avoid repeating design artifacts on entities that look the same, the interest of inheritance is frequently counterbalanced by certain limitations:

  1. Navigating through references defined on a parent (abstract) entity cannot differentiate the type of the inheriting entities. As reference navigation frequently mandates differentiating these entities, designers tend to create references directly on the inheriting entities, eliminating the benefits of inheritance.

  2. Records from two entities inheriting from the same abstract parent entity cannot be matched. For example, if creating a AbstractOrganization abstract entity, then two inheriting PrivateOrganization and PublicOrganization entities, you cannot create a matcher that looks for duplicates between PrivateOrganization and PublicOrganization.

  3. SemQL does not give you access to the type of the instance you are currently manipulating, and does not provide, from the abstract entity, access to the inheriting entities' attributes. As a consequence, you cannot aggregate rules at the abstract level.

  4. Multiple inheritance, that is an entity inheriting from multiple abstract parents, is not supported.

Due to these limitations, using inheritance reduces the capability of a model to evolve on the long run. It is not recommended to use inheritance unless you have a full understanding of these limitations.

Experience shows that a simpler approach using a single entity that holds all attributes, along with conditions to run rules or customize UI artifacts, brings the similar benefits and greater flexibility to the model. This approach is recommended in most cases instead of inheritance.

Certification Rules

In addition to the display characteristics, an entity is designed with certification rules describing how master data is created and certified from the source data published by the source applications.
These characteristics are detailed in the Certification Process Design chapter.

Entity Records Hard and Soft Delete

It is possible to configure an entity to support record deletion by selecting the Delete Enabled option for this entity.

This allows two types of deletes on the entity record, configured on each delete operation:

  • Hard Delete Physically deletes records from the hub database. Data cannot be recovered.

  • Soft Delete Logically deletes records by moving them to a deleted records storage.

When an entity support deletion, the propagation of the deletes must be taken into account when configuring the reference relationships to this entity.

In an application using entities with delete enabled, you may configure Delete actions in the entities' action sets.

Delete is handled by the jobs generated when deploying the model edition. A change in the delete or delete propagation configuration requires that you re-deploy the model edition.
Historizing Entity Data

You can historize an entity to keep track of data changes and enable browsing this entity data at any point in time.

You can historize Golden Data for all entities, and historize Master Data for ID and Fuzzy Matched entities. Historization is configured when creating the entity, or later on by altering an existing entity.

When an application uses entities with historization enabled, it is possible to enable the history browsing feature in the application configuration and optionally define a role required to use that feature.

History uses specific table structures and is handled by the jobs generated when deploying the model edition. A change in the history configuration requires that you re-deploy the model edition.
History tracing starts only after the model is re-deployed. If you activate the historization for an entity already containing some data, its history will remain empty after the re-deployment, the current state of the golden data is not copied to the history. Only subsequent changes will be traced in the history.
You can run again data loads in order to force re-creating the history, or copy the current state of the golden data in the history to seed the history with the current data state.

Creating an Entity

Creating a New Entity

To create an entity:

  1. Right-click the Entities node and select Add Entity…. The Create New Entity wizard opens.

  2. In the Create New Entity wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Plural Label: User-friendly label for this entity when referring to several instances. The value for the plural label is automatically generated from the Label value and can be optionally modified.

    • Extends Entity: Select an entity you want to extend in the context of Inheritance. Leave this field to an empty value if this entity does not extend an existing one.

      • If the entity extends an existing one, the remaining options cannot be changed as they are inherited. Click Finish to close the wizard and then press CTRL+S to save the editor.

      • If you do not use inheritance, proceed to the next step.

    • Physical Table Name: This name is used to name the physical table that will be created to store information about this entity. For example, if the physical table is CUSTOMER, then the golden data is stored in a GD_CUSTOMER table.

    • Entity Type: Select the type for this entity. Basic, ID Matched or Fuzzy Matched. See Entity Type for more information.

    • Historize Golden Records: Select this option to keep track of the history of the golden data for this entity. See Historizing Entity Data for more information.

    • Historize Master Records: Select this option to keep track of the history of the master data for this entity. This option is available for ID Matched and Fuzzy Matched entities, if the Historize Golden Records is selected. See Historizing Entity Data for more information.

  3. Click the Next button.

  4. In the Primary Key Attribute screen, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the primary key attribute.

    • Label: User-friendly label for this primary key attribute.

    • Physical Column Name: This name is used to name the physical column that will be created to store values for this attribute.

    • For fuzzy matched entities only, select the Golden ID Generation method:

      • Sequence: Use this option to automatically generate the Golden ID as a sequential number. You can specify a startup value for the sequence in the Starts With field.

      • UUID: Use this option to automatically generate the Golden ID as a Universally Unique Identifier.

    • Select the ID Generation method (for all entities):

      • Sequence: Use this option to automatically generate the ID as a sequential number. You can specify a startup value for the sequence in the Starts With field.

      • UUID: Use this option to automatically generate the ID as a Universally Unique Identifier.

      • SemQL: Use this option to automatically generate the ID with a SemQL Expression, using the entities' attributes.

      • Manual: Use this option to manually provide the ID.

    • If the ID Generation method is set to SemQL or Manual, you can then choose the Type and Length for the ID attribute. For fuzzy matched entities the type is automatically set to string.

  5. Click Finish to close the wizard. The Entity editor opens.

  6. In the Description field, optionally enter a description for this entity.

  7. Optionally select an Icon for the entity. This icon is used in the applications to represent records from this entity when no other icon is specified.

  8. Select or unselect the Delete Enabled option to enable or disable record deletion for this entity.

  9. Press CTRL+S to save the editor.

When an entity is created, it contains no attributes. Simple Attributes and Complex Attributes can be added to the entity now.

You cannot modify the entity type or primary key directly from the entity editor. To change such key properties of the entity after creating it, you must use the Alter Entity option. See Altering an Entity for more information.
Adding a Simple Attribute

To add a simple attribute:

  1. Expand the entity node, right-click the Attributes node and select Add Simple Attribute…. The Create New Simple Attribute wizard opens.

  2. In the Create New Simple Attribute wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Physical Column Name: This name is used to name the physical column that will be created to store values for this attribute.

    • Type: Select the type of the attribute. This type can be a built-in type, a user-defined type or a list of values.

    • Length, Precision, Scale. Size for this definition attribute. The fields available depend on the built-in type selected. For example a String built-in type will only allow entering a Length. If a list of values or a user-defined type was selected, these values cannot be changed.

    • Mandatory: Check this box to make this attribute mandatory.

    • Mandatory Validation Scope: This option is available if the Mandatory option was selected. Select whether this attribute should be checked for null values pre and/or post consolidation. For more information, refer to the Certification Process Design chapter.

    • LOV Validation Scope: This option is available if the selected type is a list of values. It defines whether the attribute’s value should be checked against the codes listed in the LOV. For more information, refer to the Certification Process Design chapter.

  3. Click Finish to close the wizard. The Simple Attribute editor opens.

  4. In the Description field, optionally enter a description for the simple attribute.

  5. In the Documentation field, optionally enter the user-facing documentation for this attribute. Use plain text or the Markdown syntax for rich text. This text provides detailed documentation for the attribute and appears in the documentation sidenav.

  6. Select the Searchable option to use this attributes when performing a full-text search on this entity.

  7. Press CTRL+S to save the editor.

Adding a Complex Attribute

To add a complex attribute:

  1. Expand the entity node, right-click the Attributes node and select Add Complex Attribute…. The Create New Complex Attribute wizard opens.

  2. In the Create New Complex Attribute wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Physical Prefix: This name is used to prefix the physical column that will be created to store values for this complex attribute. The column name is <Physical Prefix>_<Definition Attribute Physical Column Name>

    • Complex Type: Select the complex type of the attribute.

    • Mandatory Validation Scope: This option is available if the Mandatory option was selected for at least one of the definition attributes of the selected complex type. Select whether the mandatory definition attributes of the complex type should be checked for null values pre and/or post consolidation. For more information, refer to the Certification Process Design chapter.

    • LOV Validation Scope: This option is available if at least one of the definition attributes of the complex type is a list of values. It defines whether the definition attribute’s value should be checked against the codes listed in the LOV. For more information, refer to the Certification Process Design chapter.

  3. Click Finish to close the wizard. The Complex Attribute editor opens.

  4. In the Description field, optionally enter a description for the complex attribute.

  5. In the Documentation field, optionally enter the user-facing documentation for this attribute. Use plain text or the Markdown syntax for rich text. This text provides detailed documentation for the attribute and appears in the documentation sidenav.

  6. Press CTRL+S to save the editor.

Working with Attributes

It is possible to edit, order or delete attributes in an entity from the Attributes list in the entity editor.

To order the attributes in an entity:

  1. Open the editor for the entity.

  2. Select the Attributes item in the sidebar.

  3. Select an attribute in the Attributes list and use the Move Up and Move Down buttons to order this attribute in the list.

To delete attributes from an entity:

  1. Open the editor for the entity.

  2. Select the Attributes item in the sidebar.

  3. Use the Delete buttons to remove the attribute from the list.

  4. Click OK in the confirmation dialog.

Altering an Entity

Altering an entity allows modifying the key properties of an entity, including its type, inheritance and primary key attribute.

To alter an entity:

  1. Select the entity node, right-click and select Alter Entity. The Modify Entity wizard opens.

  2. In the wizard, modify the properties of the entity using the same process used for Creating a New Entity.

When altering an entity, it is strongly recommended to validate the model to make sure that the other elements of the model do not conflict with the modified entity definition. You must also re-deploy the model edition in order to make the changes active in the deployment data location.

Reference Relationships

Reference Relationships functionally relate two existing entities. One of them is the referenced, and one is referencing.
For example:

  • In an EmployeeHasCostCenter reference, the referenced entity is CostCenter, the referencing entity is Employee.

  • In a CostCenterHasParentCostCenter reference, the referenced entity is CostCenter, the referencing entity is also CostCenter. This is a self-reference.

They are used for:

  • Organizing records into hierarchies: For example, a hierarchy of cost centers is built using a self-reference called CostCenterHasParentCostCenter linking the CostCenter entity to itself.

  • Displaying lists of child records referencing a given record. For example, a CostCenter can appear with a list of Employee belonging to this cost center according to the EmployeeHasCostCenter relation.

  • Navigating to records referenced through a relation. For example, navigate from an Employee record to the CostCenter he belongs to.

  • Referential Integrity. Referential integrity is enforced as part of the golden data certification process. References can be used as constraints on records. For example, to check that an Employee is attached to an existing CostCenter, or a Contact is attached to an existing Customer.

  • Propagating or preventing deletion. For example, to define that a Customer with existing Contracts cannot be deleted, or that when a Product is delete, all its Parts should also be deleted.

A reference is expressed in the model in the form of a foreign attribute that is added to the referencing entity.

To create a reference relationship:

  1. Right-click the Reference Relationships node in the model and select Add Reference…. The Create New Reference Relationship wizard opens.

  2. In the Create New Reference Relationship wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Physical Name: Name of the database indexes created for optimizing access via this reference.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Validation Scope: Select whether the referential integrity should be checked pre and/or post consolidation. For more information, refer to the Certification Process Design chapter.

  3. In the Referencing [0..*] group, enter the following values:

    • Referencing Entity: Select the entity which references the referenced (parent) entity. For example, in an EmployeeHasCostCenter relation, it is the Employee entity.

    • Referencing Role Name: Name used to refer to this entity from the referenced entity. For example, in an EmployeeHasCostCenter relation, it is the Employees.

    • Referencing Role Label: User-friendly label to refer to one record of this entity from the referenced entity. For example, in an EmployeeHasCostCenter relation, it is the Reporting Employee .

    • Referencing Role Plural Label: User-friendly label to refer to a list of records of this entity from the referenced entity. For example, in an EmployeeHasCostCenter relation, it is the Reporting Employees (plural) . In the Referenced [0..1] group, enter the following values:

    • Referenced Entity: Select the entity which is referenced. For example, in an EmployeeHasCostCenter relation, it is the CostCenter entity.

    • Referenced Role Name: Name used to refer to this entity from the referencing entity. For example, in an EmployeeHasCostCenter relation, it is the CostCenter . This name is also the name given to the foreign attribute added to the referencing entity to express this relation.

    • Referenced Role Label: User-friendly label to refer to this entity from the referencing entity. For example, in an EmployeeHasCostCenter relation, it is the Cost Center.

    • Mandatory: Define whether the reference to this entity is mandatory for the child. For example, in an EmployeeHasCostCenter relation, this option must be checked as an employee always belong to a cost center. For a CostCenterHasParentCostCenter reference, the option should not be checked as some cost centers may be at the root of my organization chart.

    • Physical Name: Name of the physical column created for the foreign attribute in the tables representing the referencing entity.

  4. Click Finish to close the wizard. The Reference Relationship editor opens.

  5. In the Description field, optionally enter a description for the Reference Relationship.

  6. Define the user facing Documentation for the reference. Use plain text or the Markdown syntax for rich text. This text provides detailed documentation for the reference and appears in the documentation sidenav.

  7. In the Delete Propagation field, define how this reference affects deletion. The possible values are:

    • Restrict: Prevent referenced record deletion if a referencing record exists. If a child is found through this relationship while attempting to delete the parent (or an ancestor), the whole delete operation will be aborted or not allowed.

    • Nullify: Set the reference to null on the referencing record when referenced record is deleted. If a child is found through this relationship while attempting to delete the parent, its relationship to the parent is set to null. This value cannot be selected for mandatory relationships

    • Cascade: Delete referencing records when referenced record is deleted. If a child is found through this relationship while attempting to delete the parent, an attempt to delete this child and all its children recursively will be made, until all children are finally deleted. If any of the child relationships - at any level are preventing the delete, the record and all its children are not deleted.

  8. Press CTRL+S to save the editor.

  9. Close the editor.

Data Quality Constraints

Data Quality Constraints include all the rules in the model that enforce a certain level of quality on the entities. These rules include:

  • Mandatory Attributes: An attribute must not have a null value. For example, the Phone attribute in the Customer entity must not be null.

  • References (Mandatory References): An entity with a non-mandatory reference must have a valid referenced entity or a null reference (no referenced entity). For mandatory references, the entity must have a valid reference and does not allow null references.

  • LOV Validation: An attribute with an LOV type must have all its values defined in the LOV. For example, the Gender attribute of the Customer entity is a LOV of type GenderLOV. It must have its values in the following range: [M:Male, F:Female].

  • Unique Keys: A group of column that has a unique value. For example, for a Product entity, the pair ProductFamilyName, ProductName must be unique.

  • Validations: A formula that must be valid on a given record. For example, a Customer entity must have either a valid Email or a valid Address.

The Mandatory Attributes and LOV Validations are designed when creating the Entities. The references are defined when creating Reference Relationships.
In this section, the Unique Keys and Validations are described. Refer to the previous sections of the chapter for the other constraints.

More information:

Unique Keys

A Unique Key defines a group of attributes that should be unique for an entity.

  1. Expand the entity node, right-click the Unique Keys node and select Add Unique Key…. The Create New Unique Key wizard opens.

  2. In the Create New Unique Key wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Validation Scope: Select whether the unique key should be checked pre and/or post consolidation. For more information, refer to the Certification Process Design chapter.

  3. Click Next.

  4. In the Key Attributes page, select the Available Attributes you want to add and click the Add >> button to add them to the Key Attributes.

  5. Use the Move Up and Move Down buttons to order the selected attributes.

  6. Click Finish to close the wizard. The Unique Key editor opens.

  7. In the Description field, optionally enter a description for the Unique Key.

  8. Press CTRL+S to save the editor.

  9. Close the editor.

In the certification process unique keys are checked after the match and consolidation process, on the consolidated (merged) records. Possible unique key violations are not checked on the incoming (source records).
Unique keys can be checked in data authoring to verify whether records with the same key already exist.

Validations

A record-level validation validates the values of a given entity record against a rule. Several validations may exist on a single entity.

There are two types of validation:

  • SemQL Validations express the validation rule in the SemQL language. These validations are executed in the hub’s database.

  • Plug-in Validations use a Plug-in developed in Java. These validations are executed by Semarchy xDM. Controls that cannot be achieved within the database (for example that involve calling an external API) can be created using plug-in validations.

SemQL Validation

To create a SemQL validation:

  1. Expand the entity node, right-click the Validations node and select Add SemQL Validation…. The Create New SemQL Validation wizard opens.

  2. In the Create New SemQL Validation wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Description: optionally enter a description for the SemQL Validation.

    • Condition: Enter the SemQL condition that must be true for a valid record. You can use the image Edit Expression button to open the SemQL Editor .

    • Validation Scope: Select whether the SemQL Validation should be checked pre and/or post consolidation. For more information, refer to the Certification Process Design chapter.

  3. Click Finish to close the wizard. The SemQL Validation editor opens.

  4. Press CTRL+S to save the editor.

  5. Close the editor.

Plug-in Validation
Before using a plug-in validation, make sure the plug-in was added to the platform by the administrator. For more information, refer to the Semarchy xDM Administration Guide.

To create a plug-in validation:

  1. Expand the entity node, right-click the Validations node and select Add Plug-in Validation…. The Create New Plug-in Validation wizard opens.

  2. In the Create New Plug-in Validation wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Plug-in ID: Select the plug-in from the list of plug-ins installed in the platform.

    • Validation Scope: Select whether the Validation should be checked pre and/or post consolidation. For more information, refer to the Certification Process Design chapter.

  3. Click Finish to close the wizard. The Plug-in Validation editor opens. The Plug-in Params and Plug-in Inputs tables show the parameters and inputs for this plug-in.

  4. Only the parameters that are mandatory for the plug-in to work are listed in the Plug-in Params. You can add the parameters you need to set to the list:

    1. In the Plug-in Params table, click the Define Plug-in Parameters button.

    2. In the Parameters dialog, select the Available Parameters you want to add and click the Add >> button to add them to the Used Parameters.

    3. Click Finish to close the dialog.

  5. Set the values for the parameters:

    1. Click the Value column in the Plug-in Params table in front of a parameter. The cell becomes editable.

    2. Enter the value of the parameter in the cell, and then press Enter.

    3. Repeat the previous steps to set the value for the parameters.

  6. Only the inputs that are mandatory for the plug-in to work are listed in the Plug-in Inputs. You can add the inputs you need to set to the list:

    1. In the Plug-in Inputs table, click the Define Plug-in Inputs button.

    2. In the Input Bindings dialog, select the Available Inputs you want to add and click the Add >> button to add them to the Used Inputs.

    3. Click Finish to close the dialog.

  7. Set the values for the inputs:

    1. Double-Click the Expression column in the Plug-in Inputs table in front an of input. The SemQL editor opens.

    2. Edit the SemQL expression using the attributes to feed the plug-in input and then click OK to close the SemQL Editor.

    3. Repeat the previous steps to set an expression for the inputs.

  8. Optionally, you can use Advanced Plug-in Configuration properties to optimize and configure the plug-in execution.

  9. Press CTRL+S to save the editor.

  10. Close the editor.

Diagrams

A Diagram is a graphical representation of a portion of the model or the entire model.
Using the Diagram, not only can you make a model more readable, but you can also create entities and references in a graphical manner, and organize them as graphical shapes.

It is important to understand that a diagram only displays shapes which are graphical representations of the entities and references. These shapes are not the real entities and reference, but graphical artifacts in the diagram:

  • When you double click on a shape from the diagram, you access the actual entity or reference via the shape representing it.

  • It is possible to remove a shape from the diagram without deleting the entity or reference.

  • You can have multiple shapes representing the same entity in a diagram. This is typically used for readability reasons. All these shapes point to the same entity.

  • If you delete an entity or reference, the shapes representing it automatically disappear from the diagrams.

Creating Diagrams

To create a diagram:

  1. Right-click the Diagrams node and select Add Diagram…. The Create New Diagram wizard opens.

  2. In the Create New Diagram wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • In the Description field, optionally enter a description for the Diagram.

  3. Click Finish to close the wizard. The Diagram editor opens.

Working with Entities and References

In this section, the creation/deletion of entities and references from the diagram is explained.

To create an entity using the diagram:

  1. In the Palette, select Add Entity.

  2. Click the diagram. The Create New Entity wizard opens.

Follow the entity creation process described in the Creating an Entity section.

The entity is created and a shape corresponding to this entity is added to the diagram.
Note that you can also create, edit and delete attributes from the diagram. Select an attribute or entity and use the context menu options.

To create a reference using the diagram:

  1. In the Palette, select Add Reference.

  2. Select the referencing entity in the diagram. Keep the mouse button pressed, and move the cursor to the referenced entity.

  3. Release the mouse button. The Create New Reference wizard opens. It is pre-filled based on the two entities.

Follow the reference relationship process described in the Reference Relationships section.

The reference is created and a shape corresponding to this reference is added to the diagram.

To delete a reference or an entity from the diagram:

  1. In the diagram, select the entity or reference you want to delete.

  2. Right-click and select Delete.

  3. Click OK in the Confirm Delete dialog.

The reference or entity, as well as the shape in the diagram disappear.

Deleting an entity or reference cannot be undone.

Working with Shapes

In this section, the creation/deletion of shapes in the diagram without changing the real entity or reference is explained.

To add existing entities to the diagram:

  1. In the Palette, select Add Existing Entity.

  2. Click the diagram. The Selection Needed dialog opens showing the list of entities in the diagram.

  3. Select the entities to add to the diagram.

  4. Click OK. The shapes for the selected entities are added to the diagram.

You can repeat this operation if you want to add multiple shapes for an entity in the diagram.

To add existing references to the diagram:
It is not possible to manually create a shape for reference in a diagram.
When an entity is added to the diagram, shapes for the references relating this entity to entities already in the diagram are automatically added.

To remove a shape from the diagram:

  1. In the diagram, select the shape representing the entity of reference you want to delete.

  2. Right-click and select Remove Shape.

The shape disappears from the diagram. The entity or reference is not deleted.

Database Reverse-Engineering

Reverse-engineering can be used to quickly populate a model from table structures stored in a schema.
During this process, entities are created with attributes named after the column names.

This process connects to a database schema using a JDBC datasource that must be previously configured on the application server for the Semarchy xDM application. Contact the application server administrator to declare this datasource.
Reverse-engineering is provided as a one-shot method to quickly populate the initial model from a database schema. The entities created using a reverse-engineering are by default fuzzy matched entities. Foreign keys existing between the tables in the schema are not reverse-engineered as references in the model. Besides, the reverse-engineering process is not incremental.

To perform a database reverse-engineering:

  1. Select the Entities node in the model.

  2. Right-click and select Database Reverse-Engineering. The Reverse-Engineer Tables to Entities appears.

  3. Select the Datasource to Reverse-Engineer from the list.

  4. Click Next. The list of tables and columns is reverse-engineered. An Entity Preview page shows the list of tables and columns and the suggested logical names for these.

    • The Logical Name column corresponds to the internal names of the entities and attributes generated by the reverse-engineering process.

    • The Physical Name column is filled in with the physical name of the table and column in the database

  5. Select only the entities and attributes you want to reverse-engineer.

  6. Modify the logical and physical names in this page.

  7. Press Finish . The entities and attributes are created according to your definition.

The entities created via a reverse-engineering process are provided as is. It is recommended to review their definition, particularly the Entity Type .

Model Variables

Model Variables contain values that are used to customize the user experience or parameterize an integration job. Variable values are local to the user session or executed job. They are set either via a job parameter (for jobs), or retrieved from remote servers (declared as Variable Value Providers) when the user opens his session. Variables can be used in SemQL filters and expressions created at design and run-time.

For more information about Variable Value Providers, see the Configuring Variable Value Providers section in the Semarchy xDM Administration Guide.

SemQL provides built-in variables that contain information about the load or batch being processed or about the user connected or performing the operations. For example, the user name, its roles, the email or phone number he has entered in his profile.
Refer to the Built-in Platform Variables section in the Semarchy xDM SemQL Reference Guide for more information.

Creating Model Variables

Before creating model variables make sure that the variable value providers, from which you want to retrieve information, are declared for your Semarchy xDM instance.

To create a model variable:

  1. Right-click the Model Variables node and select Add Model Variable…. The Create New Model Variable wizard opens.

  2. In the Create New Model Variable wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Variable Type: select the data type of the variable: boolean, number or string.

    • Variable Value Provider: select the variable value provider that will be queried to retrieve the values for this variable.

  3. Click Next.

  4. Select the image Edit Expression button.

  5. In the Variable Lookup Query dialog, enter the query to retrieve the variable value:

    • For a Datasource Variable Value Provider, enter a SQL Query. You can use :V_USERNAME (connected user name) or another built-in variable as a bind variable in this query.

    • For an LDAP Variable Value Provider, enter the parameters of the LDAP search:

      • Base DN: The name of the base object entry (or possibly the root) relative to which the search is to be performed. For example: dc=myCompany.

      • Filter: Criteria to use for selecting elements within the scope. For more information about LDAP filters, see LDAP Search Filter Syntax.

      • Attribute: The attribute from the returned result to set in the variable value.

      • Search Scope: The depth of the search. Possible search scope values are Base Object (search just the object specified in the Base DN), Single Level (search entries immediately below the base DN), or Whole Subtree (the entire sub-tree starting at the base DN).

  6. Click Finish to close the wizard. The Model Variable editor opens.

  7. Press CTRL+S to save the editor.

  8. Close the editor.

Variable Lookup Queries

The variable lookup query defined in a model variable retrieves information from a variable value provider. This information can be made specific to the connected user using for example the V_USERNAME or V_USER_ROLES built-in variables .

For example, the V_USERNAME built-in variable stores the connected user name, and can be referred to in the variable lookup queries using the following syntax:

  • :V_USERNAME in SQL queries. For example, SELECT COUNTRY FROM USERINFO WHERE USER=:V_USERNAME retrieves the COUNTRY for the connected user name (variable binding using the :V_USERNAME syntax) from a USERINFO table.

  • {V_USERNAME} in LDAP Filters. For example, the filter (&(objectClass=person)(cn={V_USERNAME})) will select persons (elements of the objectClass person) for which the common name ( cn) contains the currently connected user name (variable binding using the {V_USERNAME} syntax).

Similarly, the V_USER_ROLES variable stores a Comma-separated list of roles of the connected user.

Refer to the Built-in Platform Variables section in the Semarchy xDM SemQL Reference Guide for a list of all built-in variables.
Lookup queries should return a single value (column) and a single result (record). If the query returns multiple results or multiple values, only the first value of the first result is taken into account and set in the variable value.
It is not possible to use a variable in the lookup query of another variable.

Testing Model Variables

After creating a new model variable, it is recommended to test it.

To test a model variable:

  1. In the Model Edition view, double-click the model variable. The Model Variable editor for this variable opens.

  2. In the editor toolbar, click the image Retrieve Current Value button.

  3. The variable value refreshed for the current session appears in the Current Value field of the editor. If the variable cannot be refreshed, an error is issued.

Using Model Variables

Model Variables can be used:

  • In user sessions: They are set when the user accesses an application, using a variable value provider. In this context, variables are used to parameterize the user experience (for example, in filters restricting the user privileges).

  • In integration jobs: In an integration job, a variable value is usually set using a job parameter. If no job parameter is set, the value is set using the variable value provider. In this context, variables are used to parameterize the job execution (for example, in an enricher’s filter expression to prevent the enricher from processing any record depending on the value).

Model Variable can be used, for example, in the following SemQL expressions:

In these SemQL expressions, you can bind the model variable using the :<variable name> syntax. You can also use in these expressions the built-in :V_USERNAME bind variable.

Example 5. Using a variable to customize the user experience.

To create a privilege grant allowing the connected user to see only his own record in the Employee master data:

  • In the model, create a variable called CurrentUserEmail, refreshed from the LDAP directory attribute email filtered with (&(objectClass=person)(cn={V_USERNAME})).

  • Create the privilege grant on the Employee entity, filtered with the following SemQL Expression: EmailAddress=:CurrentUserEmail

The connected user will be granted these privileges only for the master data record matching this expression.

Example 6. Using a variable to parameterize an enricher’s execution.

To parameterize an enricher’s execution depending on a job parameter value:

  • In the model, create a variable called RUN_ENRICHER, refreshed from datasource value provider, using the SELECT '0' FROM DUAL database query.

  • In the enricher’s filter, enter :RUN_ENRICHER = 1.

In you set a job parameter named RUN_ENRICHER to 1 for the job running this enricher, then this enricher will run. Otherwise, it will process no record.

Variable values are not persisted. They are retrieved when the user connects, and disposed at the end of the user session. If the content of the Variable Value Provider (Remote LDAP Directory or Database) is modified, the changes are taken into account only when the user re-connects to Semarchy xDM.
When a model variable is used in a job, but no corresponding job parameter is set for the job, then the variable takes its value from the variable value provider, using as the connected user the one that has started the job.

Named Queries

Named queries define integration endpoints to consume structured information from the MDM hub using a REST API. Each named query is exposed as a REST endpoint, supporting complex structures as query parameters.

A named query is structured as an Object (linked to an entity) containing Properties.

Property have a given type: * Value: A simple value, computed from SemQL expressions using the entity’s attributes. * Multi Records: A filtered and sorted list of records related to the entity. For example, the list of Contacts attached to the Customer. * Single Record: A single records the entity relates to. For example, the AccountManager to which the Customer is attached.

Multi Records and Single Record are also Objects, and have therefore their own properties.

Typically: * Values represent attributes of the entity * Multiple Records represent referencing (child) records with their properties. * Single-Records represent referenced (parent) records with their properties.

A Multiple-Records or Single-Record property can refer to an object that already exist in the named query. Use this capability to create recursive structures.

Creating a Named Query

To create a named query:

  1. Right-click the Named Queries node and select Add Named Query…. The Create New Named Query wizard opens.

  2. In the Create New Named Query wizard, enter the following values:

    • Name: Internal name of the object.

    • Description: Description of the named query.

    • Entity: select the entity at the root of the named query.

  3. Click Next. In the second wizard step, provide the Name for the root object of the query.

  4. Click Next. In the third wizard step, select:

    • Simple Attributes, to create properties of type Value

    • Foreign Attributes, to create properties of type Single-Record

    • Reference Relationship, to create properties of type Multiple-Records

  5. Click Finish to close the wizard. The Named Query editor opens.

  6. Configure an optional Filter condition applied to the data retrieved by the query.

  7. Configure an optional Sort expression applied to the data returned by the query.

  8. Press CTRL+S to save the editor.

Adding a Property

To add a property to an object:

  1. In the Object Properties tree table, select the object you want to add a property to.

  2. Click the Add Property button in the tree table toolbar. The Create New Object Property wizard opens.

  3. In the Create New Object Property wizard, enter the following values:

    • Property Type: Select the type of the property.

    • SemQL Expression: Enter or build the SemQL expression corresponding to a single record, multi-record or value expression, according to the property type.

    • Name: Provide a name for the property.

  4. If you create a Value property, click Finish to close the wizard. The property is added to the object.

  5. If you create a Multi Record or Single Record property:

    1. Click Next.

    2. In the second wizard step, select the object the Multi Record or Single Record property should point to:

      • Create New Object to create and name a new object.

      • Use Existing Object to point to an object that already exists in the named query.

    3. Click Finish to close the wizard. The property is added to the object.

    4. Select the Multi Record or Single Record property that you just created.

    5. In the Properties view, you can configure:

      • A Filter condition applied to the data returned in a Single or Multi Record property.

      • A Sort expression applied to the records returned in a Multi Record property.

You can also use the Add Properties action to add multiple properties by selecting simple attributes, foreign attributes or references relationships.

Creating Query Parameters

A query parameter is used to parameterize the query. It is defined for the query and can be used in the filter conditions, sort expression and the properties' SemQL expressions.

To add a query parameter:

  1. In the Named Query editor, scroll down to the Query Parameters table, and then click the Add Query Parameter button. The Create New Query Parameter wizard opens.

  2. In the Create New Query Parameter wizard, enter the following values:

    • Name: Internal name of the object.

    • Binding Name: Name used to refer to this variable in the filter conditions, sort and SemQL expressions.

    • Parameter Data Type: Type of the parameter.

    • Default Value: Configure a default value for the parameter.

    • Mandatory: If the parameter has no default value, indicate whether it is mandatory for the query.

    • Click Finish the parameter is added to the list.

To use a query parameter, simple refer to it by using its Binding Name prefixed by a colon (:).

For example, to filter products using the ID passed in a parameter which binding name is QUERY_PARAM_SEARCHED_ID:

ProductID = :QUERY_PARAM_SEARCHED_ID

Using Named Queries

Named queries are used as REST API endpoints to consume data. The Semarchy xDM Integration Guide explains how to use named queries.

Certification Process Design

Introduction to the Certification Process

The Certification Process creates consolidated and certified Golden Records from various sources:

  • Source Records, pushed into the hub by middleware systems on behalf of upstream applications (known as the Publishers).
    Depending on the type of the entity, these records are either converted to golden records directly (basic entities) or matched and consolidated into golden records (ID and fuzzy matched entities). When matched and consolidated, these records are referred to as Master Records the golden record they have contributed to create are referred to as Master Based golden records.

  • Source Authoring Record, authored by users in the MDM applications.
    When a user authors data in an MDM application, depending on the entity type and the application design, he performs one of the following operations:

    • He creates new golden records or updates existing golden records that exist only for the hub, and do not exist in any of the publishers. These records are referred to as Data Entry Based golden records This pattern is allowed for all entities, but basic entities support only this pattern.

    • He creates or updates master records on behalf of publishers, submitting these records to matching and consolidation. This pattern is allowed only for ID and fuzzy matched entities.

    • He overrides golden values resulting from the consolidation of records pushed by publishers. This pattern is allowed only for ID and fuzzy matched entities.

  • Delete Operations, made by users on golden records from entities with delete enabled.

  • Matching Decisions, taken by data stewards for fuzzy matched entities, using duplicates managers. Such decisions include confirming, merging or splitting groups of matching records as well as accepting/rejecting suggestions.

The certification process takes these various sources, applies the rules and constraints defined in the model in order to create, update or delete the golden data that business users browse using the MDM applications and that downstream application consume from the hub.

This process is automated and involves several phases, automatically generated from the rules and constraints, which are defined in the model based on the functional knowledge of the entities and the publishers involved.

The following sections describe the details of the certification process for ID, fuzzy matched and basic entities, and the delete process for all entities.

Certification Process for ID and Fuzzy Matched Entities

The certification process involves the following steps:

  1. Enrich and Standardize Source Data: Source Authoring Records created or updated on behalf of publishers and Source Records are enriched and standardized using the SemQL and Plug-in Enrichers executed Pre-Consolidation.

  2. Validate Source Data: The enriched and standardized records are checked against the various Constraints executed Pre-Consolidation. Erroneous records are ignored for the rest of the processing and the errors are logged

    Note that source authoring records are enriched and validated only for basic entities. For ID and fuzzy matched, source authoring records are not enriched and validated.

  3. Match and Find Duplicates: For fuzzy matched entities, this step matches pairs of records using a Matcher and creates groups of matching records (match groups). For ID matched entities, matching is simply made on the ID value.
    The matcher works as follows:

    • It runs a set of Match Rules. Each rule has two phases: first, a binning phase creates small bins of records. Then a matching phase compares each pair of records within these small bins to detects duplicates.

    • Each match rule has a Match Score that expresses how strongly the pair of records matches. A pair of records that match according to one or more rules is given the highest Match Score of all these rules.

    • When a match group is created, an overall Confidence Score is computed for that group. According to this score, the group is marked as a suggestion or immediately merged, and possibly confirmed. These automated actions are configured in the Merge Policy and Auto-Confirm Policy of the matcher.

    • Matching Decisions taken by users on match groups are applied at that point, superseding the matcher’s choices.

  4. Consolidate Data: This step consolidates match group duplicates into single consolidated records. The Consolidation Rules created in the Survivorship Rules defines how the attributes consolidate.

  5. Enrich Consolidated Data: The SemQL and Plug-in Enrichers executed Post-Consolidation run to standardize or add data to the consolidated records.

  6. Publish Certified Golden Data: This step finally publishes the Golden Records for consumption.

    • This step applies possible overrides from Source Authoring Record , according to the Override Rules defined in the Survivorship Rules.

    • This step also creates or updates Data Entry Based golden records (that exist only in the MDM), from Source Authoring Records.

  7. Validate Golden Data: The quality of the golden records is checked against the various Constraints executed on golden records (Post-Consolidation). Note that unlike the pre-consolidation validation, it does not remove erroneous golden records from the flow but flags them as erroneous. The errors are also logged .

  8. Historize Data: Golden and master data changes are added to their history if historization is enabled.

Source Authoring Records are not enriched or validated for ID and fuzzy matched entities as part of the certification process. These records should be enriched and validated as part of the steppers into which users author the data.

Certification Process for Basic Entities

The certification process involves the following steps:

  1. Enrich and Standardize Source Data: During this step, the Source Records and Source Authoring Records are enriched and standardized using SemQL and Plug-in Enrichers executed Pre-Consolidation.

  2. Validate Source Data: The quality of the enriched source data is checked against the various Constraints executed Pre-Consolidation. Erroneous records are ignored for the rest of the processing and the errors are logged .

  3. Publish Certified Golden Data: This step finally publishes the Golden Records for consumption .

  4. Historize Data: Golden data changes are added to their history if historization is enabled.

Note that:

  • Basic entities do not separate Source Records from Source Authoring Records. Both follow the same process.

  • Source data for basic entities does not pass through enrichers or validations executed post-consolidation.

Deletion Process

A Delete Operation (for basic, ID or fuzzy matched entities) involves the following steps:

  1. Propagate through Cascade: Extends the deletion to the child records directly or indirectly related to the deleted ones with a Cascade configuration for Delete Propagation.

  2. Propagate through Nullify: Nullifies child records related to the deleted ones with a Nullify configuration for the Delete Propagation.

  3. Compute Restrictions: Removes from deletion the records having related child records and a Restrict configuration for the Delete Propagation. If restrictions are found, the entire delete is canceled as a whole.

  4. Propagate Delete to owned Master Records to propagate deletion to the master records attached to deleted golden records. This step only applies to ID and fuzzy matched entities.

  5. Publish Deletion: Tracks record deletion, with the record values for soft deletes only, and then removes the records from the golden and master data. When doing a hard delete, this step deletes any trace of the records in every table. The only trace of a hard delete is the ID (without data) of the deleted master and golden records. Deletes are tracked in the history for golden and master records, if historization is configured.

Is is not necessary to configure in the job all the entities deletion should cascade to. The job generation automatically detects the entities that must be included for deletion based on the entities managed by the job.

Rules Involved in the Process

The rules involved in the process include:

  • Enrichers: Sequence of transformations performed on the source and/or consolidated data to make it complete and standardized.

  • Data Quality Constraints: Checks carried out on the source and/or consolidated data to isolate or flag erroneous rows. These include Referential Integrity, Unique Keys, Mandatory attributes, List of Values, SemQL/Plug-in Validations

  • Matcher: This rule applies to Fuzzy Matched entities only. It is a set of match rules that bin (group) then match similar records to detect them as duplicates. The resulting duplicate clusters are merged (consolidated) and confirmed depending on their confidence score.

  • Survivorship Rule: This rule defines, for Fuzzy Matched and ID Matched Entities, how the golden record values are computed. It is composed of:

    • A Consolidation Rule, defining how to consolidate values from duplicate records (detected by the matcher) into a single (golden) record.

    • An Override Rule, defining how values authored by users override the consolidated value in the golden record.

Integration Jobs

When all the rules are defined, or more Integration Jobs can be defined for the model.
The integration job will run to perform the certification process, using the hub’s database engine for most of the processing (including SemQL processing) and Semarchy xDM for running the plug-ins code.

Integration jobs are triggered to integrate data published in batch by data integration/ETL tools, or to process data managed by users in applications.

When pushing data in the hub, a data integration or ETL product performs the following:

  1. It requests a Load ID to identify the data load and initiate a transaction with Semarchy xDM.

  2. It loads data in the landing tables of Semarchy xDM, possibly from several sources identified as Publishers.

  3. It submits the load identified by the Load ID, and when submitting the load, it provides the name of the Integration Job that must be executed to process the incoming data.

Similarly, when a user starts a stepper:

  1. A transaction is created and attached to the stepper instance and is identified by a Load ID.

  2. The user performs the data authoring operations in the graphical user interface. All the data manipulations are performed within the transaction. The user can save data in this transaction and resume these data changes later.

  3. When the stepper is finished, the transaction is submitted. This triggers the Integration Job specified in the stepper definition.

Workflows

Workflows allow users to collaborate for data authoring. Each task of a workflow is processed by a user who modifies and reviews data using a stepper.

When a workflow completes, an integration job is triggered to process the changes and choices made by the user.

Workflow Lifecycle

A workflow is a set of tasks, linked by transitions.
A workflow instance (also named activity) is initiated from an Application. When it runs, it executes a single task at a time. A task is assigned to a role declared in Semarchy xDM. Such a task is assigned to a user having this role. This user then can operations on the data in the workflow. When the task is finished, the user completes the task and moves it through a transition to another task.

The task may finish the workflow either via a submit or a cancel operation. The cancel operation cancels all data changes, and the submit operation submits them to the hub. After a submit operation, an integration job is started to certify the changes performed in the workflow.

Transaction

A workflow carries along a Load Transaction (equivalent to an external load) which contains the data changes performed by the user. This transaction attached to the workflow. Changes performed in this transaction exist only for the transaction and the workflow until the workflow is submitted or canceled

Publishers

For ID Matched and Fuzzy Matched entities, Publishers are applications that provide source data to the MDM Hub. They identify themselves using a code when publishing data into the hub.
The publisher does not represent the technical provider of the data (the ETL or Data Integration product), but the source of the data (The CRM Application, the Sales Management system, etc.). Examples of publishers: CRM, Sales, Marketing, Finance, etc.

The consolidation rules perform certain choices depending on the publishers, and the publishers are tracked to identify the origin of the data certified by Semarchy xDM.

Identifying clearly and declaring the publishers is important in the design of the certification process. Make sure to identify the publishers when starting an MDM project.
You do not need to define a dedicated publisher for the data authored directly in the generated MDM applications. This data is of the automatically identified by the system as authored by users.

To create a publisher:

  1. Right-click the Publishers node and select Add Publisher…. The Create New Publisher wizard opens.

  2. In the Create New Publisher wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Code: Code of the publisher. This code is used by the certification process pushing data to the hub, to identify records originating from this publisher.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

  3. Active: Check this box to make this publisher active. An inactive publisher is simply declared but not used in the consolidation rules.

  4. Click Finish to close the wizard. The Publisher editor opens.

  5. In the Description field, optionally enter a description for the Publisher.

  6. In the Color property, optionally set a color for the publisher. This color is used for this publisher in applications when needed, instead of a generated color.

  7. Press CTRL+S to save the editor.

  8. Close the editor.

Enrichers

Enrichers normalize, standardize and enrich data loaded or authored in the hub. Additional Post-Consolidation enrichers can apply to consolidated data resulting from the match and merge process.

Enrichers have the following characteristics:

  • Several enrichers can be defined for an entity, and are executed in a sequence. The order in which they are defined in the model will be the order in which they will be executed

  • Enrichers can be enabled or disabled for integration jobs. Enrichers disabled for the jobs can be used for data authoring.

  • Enrichers can be configured to run on the source data (pre-consolidation) and/or the consolidated (post-consolidation) data.

  • Enrichers running pre-consolidation apply to all incoming data. It is possible to define a filter on each enricher. Only the filtered records are modified by the enricher.

  • Only pre-consolidation enrichers are allowed for basic entities.

There are two types of enrichers:

  • SemQL Enrichers express the enrichment rule in the SemQL language. These enrichers are executed in the hub’s database.

  • Plug-in Enrichers use a Plug-in developed in Java. These enrichers are executed by Semarchy xDM. Transformation that cannot be carried out in within the database (for example that involve calling an external API) can be created using plug-in enrichers.

Creating SemQL Enrichers

A SemQL Enricher enriches several attributes of an entity using attributes from this entity, transformed using SemQL expressions and functions.
You will find SemQL examples for enrichers in the Introduction to the Semarchy Workbench chapter.

To create a SemQL enricher:

  1. Expand the entity node, right-click the Enrichers node and select Add SemQL Enricher…. The Create New SemQL Enricher wizard opens.

  2. In the Create New SemQL Enricher wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

  3. Click Next.

  4. In the Enricher Expressions page, select the Available Attributes you want to enrich and click the Add >> button to add them to the Used Attributes.

  5. Click Next.

  6. Optionally click the image Edit Expression button to open the expression editor to define a filter. The enricher will only enrich those of the records respecting this filter. Skip this task if you want to enrich all the records.

  7. Click Finish to close the wizard. The SemQL Enricher editor opens.

  8. In the Description field, optionally enter a description for the SemQL Enricher.

  9. Select the Enrichment Scope for this enricher. The scope may be Pre-Consolidation Only, Post-Consolidation Only, Pre and Post Consolidation or None (not executed in the jobs).

  10. Set the enricher expressions:

    1. In the Enricher Expressions table, select the Expression column for the attribute you want to enrich and then click the image Edit Expression button. The SemQL editor opens.

    2. Create a SemQL expression to load the attribute to enrich, and then click OK to close the SemQL Editor.

    3. Repeat the previous steps to set an expression for each attribute to enrich.

  11. Press CTRL+S to save the editor.

  12. Close the editor.

Creating Plug-in Enrichers

A Plug-in Enricher enriches several attributes of an entity using attributes from this entity, transformed using a plug-in developed in Java.
A plug-in enricher takes:

  • a list of Plug-in Inputs: These are attributes possibly transformed using SemQL.

  • a list of Plug-in Parameters values.

It returns a list of Plug-in Outputs which must be mapped on the entity attributes.

Attributes are mapped on the input to feed the plug-in and on the output to enrich the entity with the resulting data transformed by the plug-in.

Before using a plug-in enricher, make sure the plug-in was added to the platform by the administrator. For more information, refer to the Semarchy xDM Administration Guide.

To create a plug-in enricher:

  1. Expand the entity node, right-click the Enrichers node and select Add Plug-in Enricher…. The Create New Plug-in Enricher wizard opens.

  2. In the Create New Plug-in Enricher wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Plug-in ID: Select the plug-in from the list of plug-ins installed in the platform.

  3. Click Next.

  4. Optionally click the image Edit Expression button to open the expression editor to define a filter. The enricher will only enrich those of the records respecting this filter. Skip this task if you want to enrich all the records.

  5. Click Finish to close the wizard. The Plug-in Enricher editor opens. The Plug-in Params, Plug-in Inputs and Plug-in Outputs tables show the parameters and inputs/outputs for this plug-in.

  6. Select the Enrichment Scope for this enricher. The scope may be Pre-Consolidation Only, Post-Consolidation Only, Pre and Post Consolidation or None (not executed in the jobs).

  7. Only the parameters that are mandatory for the plug-in to work are listed in the Plug-in Params. You can add the parameters you need to set to the list:

    1. In the Plug-in Params table, click the Define Plug-in Parameters button.

    2. In the Parameters dialog, select the Available Parameters you want to add and click the Add >> button to add them to the Used Parameters.

    3. Click Finish to close the dialog.

  8. Set the values for the parameters:

    1. Click the Value column in the Plug-in Params table in front of a parameter. The cell becomes editable.

    2. Enter the value of the parameter in the cell, and then press Enter.

    3. Repeat the previous steps to set the value for other parameters.

  9. Only the inputs that are mandatory for the plug-in to work are listed in the Plug-in Inputs. You can add the inputs you need to set to the list:

    1. In the Plug-in Inputs table, click the Define Plug-in Inputs button.

    2. In the Input Bindings dialog, select the Available Inputs you want to add and click the Add >> button to add them to the Used Inputs.

    3. Click Finish to close the dialog.

  10. Set the values for the inputs:

    1. Click the Expression column in the Plug-in Inputs table in front an input and then click the image Edit Expression button. The SemQL editor opens.

    2. Edit the SemQL expression using the attributes to feed the plug-in input and then click OK to close the SemQL Editor.

    3. Repeat the previous steps to set an expression for other inputs.

  11. Select the attributes to bind no the Plug-in Outputs:

    1. In the Plug-in Outputs table, click the Define Plug-in Outputs button.

    2. In the Output Bindings dialog, select the Available Attributes you want to enrich and click the Add >> button to add them to the Attributes Used.

    3. Click Finish to close the dialog.

  12. For each attribute in the Plug-in Outputs table, select in the Output Name column the plug-in output you want to use to enrich the attribute shown in the Attribute Name column.

  13. Optionally, you can use Advanced Plug-in Configuration properties to optimize and configure the plug-in execution.

  14. Press CTRL+S to save the editor.

  15. Close the editor.

Advanced Plug-in Configuration

The enricher and validation plug-ins provide options for optimizing and configuring their execution.
The following properties appear in the Advanced Configuration section of the editor:

  • Max Retries: If the execution of the plug-in fails, it is repeated for this number of times.

  • Behavior on Error: If the execution still fails after the Max Retries have been attempted, the plug-in either skips the current record, skips the entire enrichment task, or stops the whole job, depending on this property.

  • Thread Pool Size: This property defines the number of parallel threads used by this plug-in. This option is taken into account if the plug-in used is thread safe and declared as such.

Pre and Post-Consolidation Validation

Several validations can be defined per Entity. Validations check attribute values and reject invalid records. All validations are executed on each record.

Validations can take place pre-consolidation and/or post-consolidation:

  • Pre-Consolidation: Applies to source data, after the enrichment phase and before the matching phase. All the source records pass through all the pre-consolidation checks and records failing one check are isolated from the certification flow. All the errors for each record are raised.

  • Post-Consolidation: Applies to the golden data. All the consolidated records pass through all the post-consolidation checks and records failing one check are flagged as erroneous. They are not removed from the flow. All the errors for each record are raised. Note that post-consolidation validations are not supported for Basic Entities.

Pre-Consolidation validations remove erroneous source records from the certification process, and their data does not contribute to the golden records. Post-Consolidation validations only flag erroneous golden records. They still appear as Golden Records with Errors.
Unique Keys are only checked post-consolidation, as they only make sense on consolidated records.

Pre vs. Post Validation

Choosing the scope of a validation has impact on the certification process.
The following examples will illustrate the impact of the choice of the pre or post consolidation validation.

Example 7. Post-Consolidation Validation

A CheckNullRevenue validation checks that Revenue is not null for a Customer entity.
With this entity:

  • Customer data is published from the CRM and Sales applications.

  • Only the Sales publisher loads revenue data. CRM leaves this attribute null.

  • The consolidation needs critical information from the CRM application (email, name, address, etc…)

Validation scope impact on data certification:

  • If CheckNullRevenue is executed pre-consolidation, all data from the CRM is rejected, as revenue is null. No data from the CRM is matched or merged and the golden records are incomplete.

  • If CheckNullRevenue is executed post-Consolidation, records from both CRM and Sales are matched and merged, and the resulting records are flagged as erroneous if they still have a null revenue.

Example 8. Pre-Consolidation Validation

The matching process for the Customer entity uses the GeocodedAddress to match customers from all the sources. This value should be very accurate.

An IsValidGeocodedAddress validation checks that GeocodedAddress is not empty and GeocodedAddress.Quality is high enough after the enrichment phase. If the GeocodedAddress is empty or not good enough, then these customers should not be processed further as they are not fit for the matching phase.

In this example, IsValidGeocodedAddress should be executed Pre-consolidation to reject the records with addresses not good enough for accurate matching, before the matching phase.

Matching

The matching phase detects the duplicates in order to consolidate them into a single golden record.

Matching works differently for Fuzzy Matched and ID Matched entities.

  • Fuzzy Matched Entities use a Matcher to automatically detect duplicates using fuzzy matching algorithms.

  • ID Matched Entities perform an exact match on the user-provided ID value, as this ID is a primary key that is unique across all systems.

Basic entities do not support matching and consolidation. ID Matched entities do not support matching (it is made using the ID), but support record consolidation.

You can define matchers on Basic and ID Matched Entities. They are used for the sole purpose of detecting duplicates when creating new records. Such matchers interactively warns the user when a new entry matches existing records.

Understanding Match and Merge

When using ID Matching, the outcome of the matching process is extremely predictable. Indeed, two records with the same ID will match and will be merged according to the Consolidation Rule.

When using Fuzzy Matching, the outcome of the matching process is less predictable, as it is based on fuzzy match rules and algorithms that depend on the data. This section explains how match and merge work in this context.

Match Rules and Scoring

First, the matching must detect the duplicates, that is pairs of records that match because they are somehow similar.

Multiple Match Rules in a matcher allow you to define several conditions for considering two records a match. Each condition has a different Matching Score. This score represents the percentage of confidence you put in a match that occurs thanks to a rule.

Example 9. Match Rules and Scores

For example, the following rule (MATCH_RULE_1) defines two businesses that are exactly the same, and may have a score of 100:

Record1.CustomerName = Record2.CustomerName and
Record1.InputAddress.Address = Record2.InputAddress.Address and
...
Record1.InputAddress.City = Record2.InputAddress.City

The following rule (MATCH_RULE_2) would have for example a score of 85, as it detects businesses that have large number of similarities:

SEM_EDIT_DISTANCE_SIMILARITY( Record1.CustomerName, Record2.CustomerName ) > 85 and
SEM_EDIT_DISTANCE_SIMILARITY( Record1.InputAddress.Address, Record2.InputAddress.Address ) > 65 and
SEM_EDIT_DISTANCE_SIMILARITY( Record1.InputAddress.City, Record2.InputAddress.City ) > 65

The following rule (MATCH_RULE_3) would have a score of 20, as it detects two businesses with vaguely similar names in the same city:

SEM_EDIT_DISTANCE_SIMILARITY( Record1.CustomerName, Record2.CustomerName ) > 50
Record1.InputAddress.City = Record2.InputAddress.City

When two records match, they receive a match score equal to the highest score of all the rules that matched them (highest confidence).

For example, two records matching by the MATCH_RULE_2 (score 85) and MATCH_RULE_3 (score 20) would have a match score of 85 (the highest of 85 and 20)

Matching Groups & Group Confidence Score

Matching Groups re-group duplicate records that have matched. These groups are created using matching transitivity.

The matching mechanism is transitive, which means that:
If A matches B and B matches C, then A, B and C are in the same matching group.

Each matching group has a Confidence Score that expresses how confident you can be with that group of matching records.

This score is the average of the match scores in the group. Pairs in the group that have not matched by any rule are considered as having a score of zero.

Example 10. Computing the Group Confidence Score

For example:

  • A matches B according to MATCH_RULE_1 with a match score of 100

  • B matches C according to MATCH_RULE_3 with a match score of 20

  • A did not match C so their match score is 0.

A, B and C are in the same matching group according to the matching transitivity.

The matching group confidence score is 40: (100 + 20 + 0) / 3 = 40

Merging Groups into Golden Records

Depending on the confidence score, you may let the matcher automatically merge the match group. This operation creates a Golden Record from the group, then applies the Consolidation Rule to the group of records to define the values that are consolidated in the golden record.

If a match group is not merged automatically, because its confidence score is not high enough, it is flagged as a Merge Suggestion. In that case:

  • Incoming records are kept as singleton golden records.

  • Existing groups of records and golden records remain untouched.

Merge Suggestions are reviewed by data stewards with duplicate management actions, to decide whether or not to merge groups and create golden records.

The Merge Policy set when Creating a Matcher defines the confidence scores required to automatically merge groups in a variety of situations. See Automating Merge and Confirmation for more information about the merge policy.

Confirming Golden Record

As the values change in the source records, the match groups and golden records may change.

Example 11. Value changes impact on unconfirmed records

Renaming a business from "Micro Soft Incorporated" to "Micro Soft" is likely to have it match and merge with an existing "Microsoft" record, if we fuzzy-match by business name. The original "Micro Soft Incorporated" golden record would then cease to exist, as it would be merged within the "Microsoft" record.

Confirming a Golden Record consists in "freezing" the match group to avoid having it reconsidered every time data changes.

This is typically done by a data steward with a duplicate management action. The steward manually confirms the correct match groups and fixes the incorrect match groups.

Depending on the confidence score computed for a match group, you may also want to automatically confirm the golden record to avoid having the steward reviewing all the data.

The Auto-Confirm Policy set when Creating a Matcher defines the confidence score required to automatically confirm golden records.

It also allows you to set whether singletons (golden records composed of a single master record) should be automatically confirmed.

Over time, records may have the following Confirmation Status:

  • Not Confirmed: The golden record was never confirmed by the matcher or a user.

  • Confirmed: The golden record was entirely confirmed by the matcher or a user.

  • Partially Confirmed: Part of the match group that composes the golden record was confirmed by a user, but some masters in this match group are still not marked as confirmed.

  • Previously Confirmed: A record that was confirmed but which group has been modified by a user.

Regardless of the Confirmation status of a record, a data stewards is always able, with a duplicate management action, to manually split and merge duplicates.

Understanding Match Rules

Each Match Rule runs as a two phase process:

  • Binning: This phase uses a set of expressions to group the records into bins. Binning is a Divide and Conquer approach to avoid an excessive number of comparisons.

  • Matching: This phase uses a SemQL condition that compares all the pairs of records (Record1 is compared with Record2) within a bin. When this condition returns true, the pair of records are considered as duplicates.

The Binning Phase

The binning phase divides the source records into bins to allow the matching phase to run only within a given bin. As the matching phase can be a resource consuming one, reducing the number of comparisons is important for high performance matching.

Examples:

  • Instead of trying to match all customers, we will try to match customers only when they are located in the same country. Binning will take the Country attribute as the binning expression.

  • If we want to make the matching phase even more efficient, we could match customers with the same Country and SalesRegion.

Binning is completed using several expressions defined in the matcher. The records for which all binning expressions give the same results belong to the same bin.

Example 12. Binning by Country and Region

To perform binning for Customers with the same Country and Region’s first letter in the GeocodedAddress complex field, we would use:

  • Binning Expression #1: GeocodedAddress.Country

  • Binning Expression #2: SUBSTR(GeocodedAddress.Region,1,1)

Smaller Bins will mean faster processing, but, you must make sure that binning does not exclude possible matches.
Example 13. Incorrect Binning

Binning persons by the first four letters of their last name is not a good choice: Jones-Smith, Bill and Jonnes-Smith, Bill would end up into different bins, and would never be matched, although they are obviously similar.

For this specific case, you may consider a different attribute, or use SOUNDEX transformation on the name for binning.

The Matching Phase

The matching phase uses a condition that compares two records.
This condition uses two pseudo records named Record1 and Record2 corresponding to the two records being matched. If this condition is true, then the two records are considered as matched.

Example 14. Sample Matching Condition

The following matching condition matches customers having Customer names meeting the two following requirements

  • Sounding the same in English (SOUNDEX) OR with a similar name (using edit distance) by more than 80%.

  • Similar City name and address (using edit distance) by more 65%

( SOUNDEX (Record1.CustomerName) = SOUNDEX (Record2.CustomerName)
OR
SEM_EDIT_DISTANCE_SIMILARITY(Record1.CustomerName,Record2.CustomerName)>80 )
and SEM_EDIT_DISTANCE_SIMILARITY(Record1.InputAddress.Address, Record2.InputAddress.Address) > 65
and SEM_EDIT_DISTANCE_SIMILARITY( Record1.InputAddress.City, Record2.InputAddress.City ) > 65
Attributes Available for Match Rules

The match rules have access to a certain number of attributes:

  • Attributes from the entity being matched.

  • Attributes from related parent entities of the entity being matched.

  • Attributes from related child entities of the entity being matched.

Match rules used to match master records in the certification process are also used to detect existing matches when a user creates a new golden record from an application. If a match rule uses an attribute that is only available for master records and not for golden records (for example, PublisherID or SourceID), then this match rule is entirely ignored when detecting duplicates during golden record creation.
Matching on Parent Records

This capability is available for all entities. You can access attributes from the parent entities using regular SemQL expressions.
For example, if you have a Contact entity with a parent Customer entity, and want to match contacts when they work for customers with the same name, you would use the following syntax:

Record1.Customer.Name = Record2.Customer.Name

Note that in this expression, Customer is the role name of the customer entity in the relation.

Matching on Child Records

This capability is available only for fuzzy-matching entities. You must declare in the rule the child entity used for matching.
For example, if you have a Contact entity with a child EmailAddresses entity and you want to match contacts using their email addresses, then you must configure a match rule in the Contact matcher with:

  • Match on Child Records checked.

  • Child Records set to EmailAddresses

  • Binning Expressions and a Matching Condition using attributes of the EmailAddresses entity.

Note also that if any pair of child records matches according to the rule, then the two parent records are matched.
For example, the following expression used in the match rule will match contacts when they have at least one matching email address.

Record1.Address = Record2.Address

Automating Merge and Confirmation

The Merge Policy and Auto-Confirm Policy available when Creating a Matcher allow you to automatically merge and confirm detected match groups in certain situations.

Merge Policy Situations

In the merge policy, you define a confidence score threshold above which the match group is automatically merged into a golden record.

For example, if you set the value for Creating a golden record from new master records to 80, and a match group with a confidence score higher than 80 appears only from new master records, it is merged into a new golden record.
If a match group of new master records with a confidence score of 80 or lower appears, then is not merged automatically. It is proposed for merging to the data steward.

The various situations under which an automated merge may take place are listed and explained below:

  • Create a golden record from new master records: This is a frequent situation when new master records are loaded into the hub and matched/merged. The initial data loads enter in this situation.

  • Merge unconfirmed golden records: This situation occurs when existing master records attached to golden records that have not been confirmed are modified, causing the existing golden records to possibly merge all together. In this situation, if the two unconfirmed golden records merge, one ceases to exist, and the survivor may see its values modified.

  • Merge confirmed golden records: This situation occurs when existing master records attached to golden records that have been confirmed are modified, causing the existing golden records to possibly merge all together. In this situation, if the two golden records merge, a confirmed golden record may cease to exist, and the other one may see its values modified.

  • Merge unconfirmed with confirmed golden records: This situation occurs when existing unconfirmed golden records are about to merge with a golden record that has been confirmed. In this situation, unconfirmed golden records may cease to exist, and the confirmed golden record may see its values modified.

  • Add new master record to an unconfirmed golden records: This situation occurs when a new master record is about to be merged with a golden record that has not been confirmed. In this situation, the golden record values may change. This is typically the case for loads following the initial load.

  • Add new master record to a confirmed golden records: This situation occurs when a new master record is about to be merged with a golden record that has been confirmed. In this situation, the golden record values may change. This is typically the case for loads following the initial load.

  • Merge golden records previously split by the user: This situation occurs when two groups manually split by a data steward re-matches due a new record matching both groups. In this situation, existing golden records reviewed by the data steward may cease to exist.

A group may fall into several situations. In that case, its confidence score must exceed all the thresholds for the automated merge to happen.
Auto-Confirm Situations

There are two situations for automatically confirming merged golden records:

  • When the match group’s confidence score is above a certain threshold, the resulting golden record can be automatically marked as confirmed.

  • Singletons, that is golden records composed of a single master record, can be automatically confirmed. Note that singletons that have match suggestions (records matched but with a score not high enough to automatically merge) are not automatically confirmed.

For example, if a match group with a confidence score of 80 was automatically merged, and the Auto-Confirm Golden Record threshold was set to 79, then this group is also marked as confirmed. If the threshold is set to 85, then this group is merged but not marked as confirmed.

Pattern for Automating Merge & Confirmation
Pattern #1: No Unmonitored Change

In this pattern, the data steward should review all records. All of them are treated with the same importance. No change is made and no record is created without having a user confirming it.

Solution:

  • Set all the values in the Merge Policy and Auto-Confirm Policy to 100

  • Un-select Auto-Confirm Singletons.

Pattern #2: No Stewardship

In this pattern, the hub merges and confirms all content. Fixes will take place on demand on confirmed golden records.

Solution:

  • Set all the values in the Merge Policy and Auto-Confirm Policy to 0

  • Select Auto-Confirm Singletons.

Pattern #3: Delayed Stewardship

In this pattern, the hub merges all content and confirms no record. The steward monitors and confirms all records after they are merged.

Solution:

  • Set all the values in the Merge Policy to 0

  • Set Auto-Confirm Golden Records to 100

  • Un-select Auto-Confirm Singletons.

Alternately, to reduce stewardship overheard, you may select Auto-Confirm Singletons to avoid reviewing the singletons.

Pattern #4: Merge All then Review Suspicious Matches

In this pattern, the hub merges all content but the steward must review suspicious matches.

Solution:

  • Set all the values in the Merge Policy to 0

  • Set Auto-Confirm Golden Records to 80 (adjust this value to your match rules scores)

  • Select Auto-Confirm Singletons.

The steward will be able to review suspicious unconfirmed golden records (those under the 80% confidence score).

Pattern #5: Manually Merge Suspicious Matches

In this pattern, the hub merges and confirms confident matches but the steward must manually merge others.

Solution:

  • Set all the values in the Merge Policy to 80 (adjust this value to your match rules scores)

  • Set Auto-Confirm Golden Records to 80 (adjust this value to your match rules scores)

  • Select Auto-Confirm Singletons.

Pattern #6: Prevent Golden Deletion

In this pattern, the hub must prevent golden records from ceasing to exist without the steward approval, but allows other confident merge operations to take place automatically.

Solution:

  • Set the Merge confirmed golden records, Merge unconfirmed golden records, Merge unconfirmed with confirmed golden records, Merge golden records previously split by the user to 100.

  • Set other values in the Merge Policy to 80 (adjust this value to your match rules scores)

  • Set Auto-Confirm Golden Records to 80 (adjust this value to your match rules scores)

  • Select Auto-Confirm Singletons.

Creating a Matcher

Only one matcher can be created for each entity.

To create a matcher:

  1. Expand the entity node, right-click the Matcher node and select Define SemQL Matcher…. The Create New SemQL Matcher wizard opens.

  2. In the Description field, optionally enter a description for the Matcher.

  3. Click Finish to close the wizard. The SemQL Matcher editor opens.

  4. Define the Match Rules:

    1. Click the Add Match Rule button in the Match Rules tables. The Match Rule: NewRule editor opens.

    2. Give a Name and internal Description to the match rule.

    3. Give a user-friendly Label and Documentation. In the documentation, use plain text or the Markdown syntax for rich text. This text provides detailed documentation for the rule. It appears in the documentation sidenav.

    4. If you want to do matching on child records, check the Match on Child Records option, and select the Child Records you want to use for matching.

    5. Define the Binning Expressions:

      1. In the Binning Expressions table, click the Add Binning Expression button. The SemQL editor opens.

      2. Create a SemQL expression used to bin records for this entity, and then click OK to close the SemQL Editor.

      3. Repeat the previous steps to create all your binning expressions.

    6. Define the Match Condition and Match Score:

      1. In the Matching section, click the image Edit Expression button. The SemQL editor opens.

      2. Create a SemQL condition used to match records for this entity, and then click OK to close the SemQL Editor.

      3. Enter a value for the Match Score. This value should be between 1 and 100.

    7. Press CTRL+S to save the editor.

    8. Use the breadcrumb on top of the editor to return to the matcher. The new match rule appears in the list.

    9. Repeat the previous steps to create all the match rules.

  5. Define the Merge Policy: Set the minimum confidence score required for a match group to be merged in the various Merge Policy Situations.

  6. Define the Auto-Confirm Policy:

    1. Auto-confirm golden records: Minimum confidence score required for a match group to be automatically confirmed.

    2. Auto-confirm singletons: Select this option to have singletons (un-matched records) automatically confirmed.

  7. Press CTRL+S to save the editor.

  8. Close the editor.

Survivorship

Survivorship indicates which data survives in the golden records.

A Survivorship Rule defines, for attributes in Fuzzy Matched and ID Matched Entities, how golden record values are computed. It is composed of:

  • A Consolidation Rule, defining how to consolidate values from duplicate records (detected by the matcher) into a single (golden) record.

  • An Override Rule, defining how values authored by users override the consolidated value in the golden record.

A survivorship rule applies to a single attribute or to a set of attributes of a given entity.

Grouping Attributes in Survivorship Rules

When a rule has multiple attributes, all these attributes work together for the consolidation and override process:

  • Attributes in the group consolide together with the same Consolidation Rule.

  • When a user overrides one of these attributes, all of them are considered overriden. In the user interface, when the override buttons is clicked for one attribute, override is also toggled for all the attributes in the group.

  • When the override is removed (according to the Override Rule) from one of these attributes, all attributes in the group loose their override simultaneously.

The Default flag is defined for one and only one survivorship rule for an entity. The rule with the Default flag applies to all attributes not handled by any other rule of this entity.

Master ID Survivorship Rule

The Master ID Survivorship Rule is a specific rule identified by the following icon image.
This rule does not have an override strategy. It only defines how the ID of one of the master records attached to a golden record consolidates into this golden.

This master record is preferred when references to the golden record need to be re-attached to a master record. For example, if the golden record splits, references to the golden will preferably follow this master record.

Consolidation Rule

The Consolidation Rule defines, using a Consolidation Strategy, how to choose the best values in the consolidation process.

The consolidation strategies are listed in the table below. It also shows the equivalent SemQL expression for the strategy.

Strategy Description Expression

Custom Ranking

The SemQL Ranking Expression is used to rank duplicates, and the first value by rank is used.
The expression is an order by clause and can contain the specification of the ascending (ASC) or descending (DESC) order as well as the position of the NULL values (NULLS FIRST or NULLS LAST).

[semQL expression] , PubID ASC, SourceID ASC

Largest/Smallest Value

Values are sorted using their type-specific sort method (alphabetical for strings, for example).
For example: Mozart is larger than Beethoven (M is after B in the alphabet). Note that binary attributes do not support this consolidation strategy.

[value] ASC or [value] DESC

Longest/Shortest Value

The lengths of the values are ordered.
For example: Mozart is shorter than Beethoven (String size).

LENGTH([value]) ASC or LENGTH([value])DESC

Most Frequent Value

The first most frequent non null value.

Specific

Preferred Publisher

Publishers are manually ordered. The first one returning a value for the field is used.

Specific

For all strategies except Most Frequent Value, the Ranking Expression stores a SemQL Expression used to sort records in the event of an ambiguity for the strategy.
For example, when two fields having different values are duplicates from the same publisher and a Preferred Publisher strategy is used. The expression is an order by clause and can contain the specification of the ascending (ASC) or descending (DESC) order as well as the position of the NULL values (NULLS FIRST or NULLS LAST).
Only the Custom Ranking and Preferred Publisher strategies work for consolidation rules involving multiple attributes. If you want to use, for example, the Largest Value strategy for two attributes, then you must define two rules, one for each attribute.
A Skip Null option is available for the Custom Ranking and Preferred Publisher strategies to avoid null values in the consolidation process. For single-attribute rules, the first non-null value after ranking is consolidated. For multi-attributes rules, the values are picked from the first record (after ranking) having at least one of the attribute is not unll.
If you do not want to configure the consolidation, use the Custom Ranking strategy with no Ranking Expression set.

Override Rule

The Override Rule defines, using a Override Strategy, how to overwrite consolidated values with values authored by the users.

The override strategies are listed in the table below.

Strategy Description

Always Authored in the MDM

The value is always authored in the MDM application. Values coming from other sources (if any) are always ignored. This attribute remains null until a user explicitly changes the value. The attribute is therefore ignored for consolidation.

No Override

The value is always consolidated according to the consolidation rules. The application does not allow overriding data for this attribute.

Override - until consolidated value changes

Value override is allowed. Values coming from the publishers consolidate according to the consolidation rules. When an override is made, the value is maintained until the value consolidated from the publishers changes. When this happens, the new consolidated value wins against the authored value. The system reverts to the defined consolidation rules to arbitrate survivorship for the next value changes from the publishers.

Override - until next user changes

Value override is allowed. Values coming from the publishers consolidate according to the consolidation rules. When an override is made, the value is maintained until a user enters a new value or removes the override.

Defining a Survivorship Rule

Every entity is created with a Master ID Survivorship Rule and a Default Survivorship Rule using the Custom Ranking consolidation strategy with no ranking expression.

These default rules should be modified to reflect the entity requirements. You can also create new survivorship rules to handle specific attributes.

To create a survivorship rule:

  1. Right-click the Survivorship Rules under the entity and then select Add Survivorship Rule. The Create New Survivorship Rule wizard opens.

  2. In the Create New Survivorship Rule wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for the object.

    • Default Rule: Define this new rule as the default rule for the entity.

  3. Click Finish to close the wizard. The Survivorship Rule editor opens.

  4. In the Survivorship Rule editor, provide a Description for this rule and a Documentation. In the documentation, use plain text or the Markdown syntax for rich text. This text provides detailed documentation for the rule. It appears in the documentation sidenav.

  5. Select the attributes managed with this rule:

    1. In the Attributes table, select the image Define Survivorship Rule Attributes button.

    2. In the Create New Survivorship Rule Attribute dialog, double-click the attributes that you want to add.

    3. Click Finish

  6. Define the consolidation rule:

    1. Select a Consolidation Strategy.

    2. Set the parameters depending on the selected strategy:

      • Any Value, Largest Value , Longest Value, Most Frequent Value, Shortest Value and Smallest Value: No parameter is required.

      • Custom Ranking:

        1. On the Ranking Expression field, click the image Edit Expression button. The SemQL editor opens.

        2. Create a SemQL expression used to rank the records, and then click OK to close the SemQL Editor.

        3. Select the Skip Nulls option if you want to skip null values and pick the highest ranking not null value.

      • Preferred Publisher:

        1. On the Preferred Publisher field, click the image Select Publishers button. The Manage Survivorship Rule dialog opens.

        2. Double-click the first publisher in the Available Publishers list to add to the Publishers list.

        3. Repeat this operation to add the other publishers in the preference order.

        4. Use the Move Up and Move Down buttons to order the publishers.

        5. Click Finish to close the dialog.

        6. Select the Skip Nulls option if you want to skip null values returned by publishers.

  7. Set the Ranking Expression to sort records in the event of an ambiguity for the strategy.

    1. On the Ranking Expression field, click the image Edit Expression button. The SemQL editor opens.

    2. Create a SemQL expression used to handle consolidation disambiguation, and then click OK to close the SemQL Editor.

  8. Define the override rule:

    1. Select a Override Strategy.

  9. Press CTRL+S to save the editor.

  10. Close the editor.

Creating Integration Jobs

An Integration Job is a job executed by Semarchy xDM to integrate and certify source data into golden records. This job uses the rules defined as part of the certification process, and contains a sequence of Tasks running these rules. Each task addresses one entity, and performs several processes (Enrichment, Validation, etc.) for this entity.

Creating Jobs

To create a job:

  1. Right-click the Jobs node and select Add Job…. The Create New Job wizard opens.

  2. In the Create New Job wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Description: Optionally enter a description for the Job.

    • Queue Name: Name of the queue that will contain this job.

  3. Click Next.

  4. In the Tasks page, select the Available Entities you want to process in this job and click the Add >> button to add them to the Selected Entities.

  5. Click Finish to close the wizard. The Job editor opens.

  6. Select Tasks in the editor sidebar. In the list of Tasks, the entities involved in each task are listed, as well as the processes (Enrichers, Matchers, etc.) that will run for these entities.

  7. Use the Move Up and Move Down buttons to order the tasks.

  8. To edit the processes involved in one task:

    1. Double-click the entity Name in the Tasks table. The editor switches to the Task editor.

    2. Select the process you want to enable for this task.

    3. Use the editor breadcrumb to go back to the Job editor.

  9. Press CTRL+S to save the editor.

  10. Close the editor.

Jobs Parameters

Jobs can be parameterized for optimizing their execution.

To change a job parameter:

  1. In the job editor, select Job Parameters in the editor sidebar.

  2. In the Job Parameters table, click the Add Parameter button. The Create New Job Parameter wizard opens.

  3. In the Name field, enter the name of the parameter.

  4. In the Value field, enter the value for this parameter.

  5. Click Finish to close the wizard.

  6. Press CTRL+S to save the editor.

  7. Close the editor.

The following table lists the parameters available to customize the jobs.

Parameter Name Values Description

PARAM_RECYCLE_ERRORS

0 or 1

If this parameter is set to 1, error recycling is triggered and rejects from previous job executions are recycled in this job.

PARAM_ANALYZE_STATS

0, 1 or JSON object

If this parameter is set to 1, statistics collection is triggered in the MDM hub tables to optimize processing. This option is useful to accelerate the processing of large datasets. The default value for this parameter is 1.

For PostgreSQL, you can also provide a JSON object to configure how statistics are collected. A sample JSON object is provided below.

Sample configuration
{
"useVacuum": true,
"vacuumConfig": {
  "analyze": true,
  "full": true,
  "freeze": true,
  }
}

In this object, set useVacuum to true to use PostgreSQL’s VACUUM command instead of ANALYZE for statistics collection. The default value is false.

If VACUUM is used, configure it with the vacuumConfig object:

  • analyze: Set to true to use the ANALYZE option. The default value is true.

  • full: Set to true to use the FULL option. The default value is false.

  • freeze: Set to true to use the FREEZE option. The default value is false.

PARAM_AGGREGATE_JOB_ENRICHERS

0 or 1

If this parameter is set to 1, consecutive SemQL enrichers are merged into a single SQL statement when executed. This applies to all entities.

PARAM_AGGREGATE_ENTITY_ENRICHERS_<entity>

0 or 1

If this parameter is set to 1, consecutive SemQL enrichers are merged into a single SQL statement when executed. This applies only to the entity which name is provided as <entity>.

PARAM_AGGREGATE_JOB_PLUGIN_ENRICHERS

0 or 1

If this parameter is set to 1, consecutive plug-in enrichers are piped, so that the output of an enricher is directly passed in memory to the next one. This applies to all entities.

PARAM_AGGREGATE_ENTITY_PLUGIN_ENRICHERS_<entity>

0 or 1

If this parameter is set to 1, consecutive plug-in enrichers are piped, so that the output of an enricher is directly passed in memory to the next one. This applies to all entities. This applies only to the entity which name is provided as <entity>.

PARAM_PLUGIN_ENRICHERS_UPDATE_BATCH_SIZE

Number

When the plugin execution and database are fast, the network can be the bottleneck. A batch update is a group of updates sent together to the database in one batch, rather than sending the updates one by one.This JDBC batch update size is used by plug-in enrichers when writing records to the database. A high value for this parameter reduces the number of network access to write the data but increases the memory load on the server. This parameter applies to plug-ins running individually. For chained plug-ins, it applies to the last plug-in in the chain, which writes enriched data to the database.

Enricher Aggregation

Enrichers support aggregation for faster processing. When not aggregated, enrichers run one after the other, reading, modifying and then updating the data into the database. With aggregation, you avoid excessive database read/write operations by forcing several enrichers to run in a single operation.

  • Multiple consecutive SemQL Enrichers are aggregated using the PARAM_AGGREGATE_JOB_ENRICHERS and PARAM_AGGREGATE_ENTITY_ENRICHERS_<entity_name> parameters. This aggregation converts the multiple SemQL enrichers into a single SQL statement processed by the database, preventing successive database read/writes.

  • Multiple consecutive Plug-in Enrichers are aggregated using the PARAM_AGGREGATE_JOB_PLUGIN_ENRICHERS and PARAM_AGGREGATE_ENTITY_PLUGIN_ENRICHERS_<entity_name> parameters. This aggregation creates a processing chain that processes data in one pass in memory, avoiding successive database read/writes.

Enricher aggregation follows the rules listed below:

  • Only successive enrichers of the same type (SemQL or Plug-in) can be aggregated. For example, if you have the following sequence of enrichers SEMQL_1, SEMQL2, PLUGIN_1, PLUGIN_2, SEMQL3, you can aggregate SEMQL_1 with SEMQL2 and PLUGIN_1 with PLUGIN_2.

  • Plug-in enricher aggregation will stop when:

    • A plug-in enricher has a filter that uses an attribute updated by a previous enricher in the chain.

    • A plug-in enricher has an input that contains a complex SemQL expression with attributes updated by a previous enricher in the chain.

    • A plug-in enricher has one of the Thread Pool Size, Max Retry or Behavior on Error options set to a different value than a previous enricher in the chain.

Jobs Sequencing and Parallelism

A Job is a sequence of tasks. These tasks must be ordered to handle referential integrity. For example, if you process the Contact entity and then the Customer entity, you may start processing new contacts attached to customers that do not exist yet. You should preferably process the customers then the contacts.

Jobs are themselves executed sequentially in defined Queues in a FIFO (First-In First-Out) mode.
If two jobs can run simultaneously, they should be in different queues. For example, if two jobs address two different areas in the same model, then these jobs can run simultaneously in different queues.

It is not recommended to configure jobs processing the same entity to run in different queues. Instances of these jobs running simultaneously in two different queues may write conflicting changes to this entity, causing major data inconsistencies.

Designing Integration Jobs

It is possible to create jobs specific to the data loads performed by the data integration batches, and dedicated jobs for data authoring and duplicate management operations.

Integration Jobs for Data Integration

Data published in batch may target several entities.

It is recommended to define jobs specific to the data loads targeting the hub:

  • Such jobs should include all the entities loaded by the data load process.

  • In addition, it is recommended to include the entities referencing the entities loaded by the data load process.

Integration Jobs for Authoring

A stepper (used directly or in a workflow) manipulates data in several entities.

If a stepper (or the workflow using it) is created without a reference to a job, a job is automatically generated and attached to that stepper when the model is deployed. This generated job processes all the entities involved in that stepper.

It is also possible to define a specific job for the stepper:

  • Such job should process all the entities involved in the stepper.

  • In addition, if some of these are fuzzy matched entities, the job should process all the entities referencing these entities.

Integration Jobs for Duplicate Management

A duplicate management action handles duplicates detected on a specific entity.

It is possible to define a specific jobs for each duplicate management action:

  • Such job should process the entity involved in the duplicate management workflow.

  • In addition, if the entity involved in the duplicate management workflow is a fuzzy matched entity, the job should process all the entities referencing this entity.

Working with Applications

Applications provide business users and data stewards with a customized experience to access and manage their data.

Introduction to Applications

Applications Features

The model contains entities describing the master data of the enterprise. Business users and data stewards require dedicated Applications to access the master data in the hub, according to their role, privileges, and requirements.

These applications may:

  • Display relevant information organized in a business-friendly way.

  • Support data management operations, such as data authoring.

  • Support user collaboration for data management operations.

For example, if the model contains customer, contact, employee and cost center entities:

  • An HR user would need an application showing the cost center and employee data organized with hierarchies of cost centers, hierarchies of managers/employees. This application would also allow authoring of employee information.

  • A Finance user would need an application to see the cost center data in a hierarchy of cost centers and perform changes on these cost centers.

  • A Data Steward would need access to the data and errors to fix them.

From a single holistic model that includes all the master data domains (customers, suppliers, products, HR, etc.), you can define several applications (Customer Data Hub, Product Information Management, HR Hub etc.) for business roles and responsibilities in the enterprise. Each application exposes to users only the elements relevant to their roles.

Browsing and Searching Data in Applications

The primary use for an application is browsing golden data. Browsing is done using Business Views, that organize the data from multiple entities. Business views use Display Cards, Collections and Forms to display this data.

To search for information in the application, users use either filtering within a business view or the global search capability. Searching involved built-in search methods or customized Search Forms.

When browsing a business view, users can triggers actions from Action Sets to dive into the data or trigger data management actions (authoring, duplicates management).

Authoring Data in Applications

The data visible in the applications reflect two streams of information :

  • External source systems that publish their data to the MDM hub.

  • Users that author (create, edit, import, etc) data in the MDM applications.

Publishing Data to the Hub

Data curated in external source systems (known as the Publishers) is published to the MDM using data integration. This data goes through the certification process to create golden data for the three types of entity (basic, ID and fuzzy matched):

  • For ID and fuzzy matched entities, the source data from the publishers becomes master data, which is matched and consolidated into golden data.

  • For basic entities, there is no notion of publisher and source data is directly converted to golden data with no matching and consolidation.

Authoring Data

Users can also author - that is create, import or edit - data for any entity in the applications. The ability to create, edit or import records is defined by the Action Set attached to the business view being browsed.

There are two ways of authoring data in an application:

  • Direct authoring using Steppers. In a stepper, a single user creates or modifies a batch of data records, and then submits this data to the certification process.
    This method provides a simple "edit and save" approach to authoring.

  • Collaborative authoring using Workflows. When a workflow task is assigned to a user, he authors and/or reviews the workflow data using the stepper attached to the task. When done, the user transitions the workflow to another user or submits the data.
    This method provides a more collaborative way of authoring, where multiple users with different roles can edit subsets of the information and perform review tasks.

The authoring data lifecycle differs depending on the entity type. The next sections explain the various patterns.

Authoring Basic Entities

With basic entities:

  • Creating a new record creates a golden record via the certification process.

  • Updating a golden record modifies it and submits the changes to the certification process.

  • Importing either creates a new record if the imported record ID is not found in the hub, or updates the existing record that is found.

  • An update performed on an erroneous record takes this record, modifies it and resubmits it to the hub.

Basic entity records authoring uses direct authoring or workflows, with all the capabilities of steppers. You can manage complex clusters of entity records (customers and contacts) within steppers.
Authoring ID or Fuzzy Matched Entities

With matched entities, there are two types of golden records:

  • Golden records consolidated from publishers' master records. Such records are referred to as Master-Based.

  • Golden records created directly in the applications. These records do not exist for any publisher. Such records are referred to as Data Entry-Based, as they do not have associated master records.

Matched entities authoring works on golden records or master records.

When authoring golden records:

  • Creating a new golden record creates a Data Entry-Based record, that only exist in the hub.

  • Updating a golden record or fixing a golden error depends on the record type:

    • If the record is Data Entry-Based, then the update is applied to the golden record via the certification process. Such a record can be entirely modified.

    • If the record is Master-Based, then an Override is applied to the consolidated golden record data. This override overwrites certain attributes according to the Survivorship Rules defined for the entity.

  • The import and mass update actions on golden records automatically take care of the record type, using the golden record update or override patterns as needed.

Golden records authoring uses direct authoring or workflows, with all the capabilities of steppers. You can manage complex clusters of entity records (customers and contacts) within steppers.

When authoring master records:

  • Creating, updating, mass-updating or importing master records or source errors creates or updates the publisher records on behalf of publishers, submitting them to matching and consolidation as if the new records or changes were submitted by the publisher. Such master records may be later on overwritten by records with the same ID sent by the same publisher.

Master records authoring supports only direct authoring, using steppers limited to their root entity only. This means that master records authoring is performed on an entity basis, and not on a cluster of records. Users can only manage records from one entity at a time (customers or contacts).
Master records authoring is more suitable for data stewards to create or fix records before the publishers in an interim period, that is before the publisher actually pushes those records. It is recommended to use golden record authoring for business users.
Managing Duplicates in Applications

Master records in fuzzy matched entities are automatically matched using rules defined in the entity matcher.

When these records are incorrectly matched, data stewards can supersede the matcher and manually match and merge records, additionally overriding values in the consolidated golden data. Similarly, when the matcher has suggested merges without actually applying them, stewards can review, approve or reject these suggestions.

The Duplicates Management operations use Duplicate Managers, triggered from action sets used when browsing fuzzy matched records.

Application Components

An application is made up of the following components:

  • A set of structured Business Views to browse data.

  • Forms and Collections used in business views to one or multiple records.

  • Search Forms filter or search the data in the business views or in the global search.

  • Action Sets defining the data management operations (creating new records, editing a record) available on business views.

  • Steppers executed to create or modify new records.

  • Duplicate Managers to review, merge or split groups of matching records and edit merged golden records.

  • Workflows composed of tasks for users to collaborate on data management.

  • Application Actions organized into a hierarchy of Folders. Examples of actions include:

    • Browsing or searching a business view

    • Searching the entire application using Global Search.

    • Opening the user’s' Inbox or Profile, or an URL.

  • The application Navigation Drawer contains shortcuts to frequently used actions or folders.

Certain application pages, such as the Profile or Inbox, are built-in and cannot be modified. All other elements are designed as part of the application design.
When a model is created or upgraded from an older Semarchy xDM version, a simple application called Default Application is automatically created.

Creating an Application

To create an application:

  1. Right-click the Applications node and select Add Application…. The Create New Application wizard opens.

  2. In the Create New Application wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label for is automatically filled in. Modifying this label is optional.

    • Required Role: The application is by default available for all users. To restrict access to this application to a specific role, select it from the drop-down list.

    • Default Action: Select the action that will be the Home Page of the application. As the application has no other component yet, you have the choice between the Global Search, the Inbox and the Root folder.

  3. Click Finish to close the wizard. The Application editor opens.

  4. In the Description field, optionally enter a description for the Application.

  5. Configure the application Branding. For this purpose, you can use images from the image libraries.

    • Title: The short name of the application that appears in the navigation drawer’s title.

    • Title Color: Color used for the application title text. Use this color to ensure that the text of the title displays correctly on top of the Cover image.

    • Avatar: An icon representing the application. It appear in the navigation drawer’s title, and is also used to represent this application on the Welcome page.

    • Logo: Large logo representing the application, for example on the Global Search page.

    • Cover: Background image of the navigation drawer’s title.

    • Favicon: Icon displayed in the browser tab.

  6. Configure the Display Properties for the application:

    • Sort Method: Defines the sort method for application folders and actions. They can be automatically sorted alphabetically according to their Label or organized in a fixed Position set at design-time.

  7. In the Features section:

    • Select Enable History Browsing to allow users to browse data at a specific point in time. This feature works with entities for which historization is activated.

    • You can limit history browsing access to users with the role selected in the History Browsing Required Role drop-down.

It is possible to create multiple applications for a single model and to support different views to this model.
For example, a simple application for users browsing the hub through the Business Views and another application supporting advanced data authoring and management operations.

Creating Application Components

Once the new application is created, designers can start creating the various components of the application:

  • Reusable components attached to the entities:

    1. Display Cards to define how entities data appear in compact form.

    2. Forms to define how a single entity record appears.

    3. Collections to define how multiple entity records appear.

    4. Search Forms to define how to search and filter the entities records.

    5. Steppers to define how records in entities are authored.

    6. Duplicate Managers to define how duplicate records are manually managed.

    7. Action Sets to define groups of data management actions available for entities.

    8. Business Views composed of related entities, using the forms, collection, display cards and search forms to display and search the data.

  • Reusable components attached to the model:

    1. Workflows for users to collaborate on data management operations.

  • Application components:

    1. Application Actions and Folders to organize the possible operations in the applications.

    2. Navigation Drawer containing shortcuts to applications and folders.

    3. Global Search Configuration for the global search.

The following sections explain the creation process for the various components listed above. You can also use the Create Application Components Wizard to accelerate their creation.

Create Application Components Wizard

The Create Application Components wizard accelerates the creation of the various application components required to browse and manage a given entity.

To automatically create application components for an entity:

  1. Right-click the entity node and select Creation Application Components.

  2. In the Creation Application Components wizard’s first page:

    • Select an existing Display Card or select <Create New> to create a new one using the values provided in the second step of the wizard.

    • Select an existing Form, a Collection, and an Action Set or select <Create New> to optionally create new instances of these items for the entity.

    • Select Enable Authoring to create a stepper and the various authoring actions for your entity.

    • Select Enable Duplicate Management (for a fuzzy matched entity) to create a duplicate manager and the various duplicate management actions for your entity.

    • Select Create Business View (for a fuzzy matched entity) to automatically create a business view.

    • Select an application in the Show In field into which the business view will be used.

  3. Click Next. In the second wizard page. In this page, you can edit the display card selected in the first page or define the properties for a new display card:

    • Use the SemQL editor to define the Primary Text Expression as well as the Secondary Text Expression

    • Select a Primary Image and an Avatar from the image libraries. You can also define these using SemQL expressions.

  4. Click Finish to close the wizard.

The wizard automatically performs the following operations:

  • It creates a new Display Card or updates the selected one with the values entered in the second step of the wizard.

  • It optionally creates a Collection using that display card. The collection supports automatically the list, table and grid views depending on the display card content:

    • The list uses two lines if the display card has a secondary text, and shows an avatar if the display card has a value for the avatar.

    • The table uses two lines if the display card has a secondary text, and contains all the attributes defined for the entity.

    • The grid view is configured if a primary image is configured in the display card. It uses two lines if the display card has a secondary text.

  • It optionally creates a Form with a single tab, which contains one field per attribute defined for the entity. Included attributes are:

    • The ID attribute expression

    • The following user-defined attributes in their order in the entity:

    • Atomic attributes

    • Complex atomic attributes: The complex type is therefore decomposed.

    • FDN attributes (and not FIDs)

  • If the Enable Authoring option was selected, a Stepper is created with a root collection step and a form step using the collection and form selected or created by the wizard.

  • If the Enable Duplicate Management option was selected, a Duplicate Manager is created using the collection and form selected or created by the wizard.

  • An Action Set with all common actions (export, etc), all possible authoring actions (edit, create, import, etc) if Enable Authoring was selected, and all possible duplicate management action if Enable Duplicate Management was selected. Authoring and duplicate management actions use the stepper and duplicate managers previously created.

  • A Business View with a single level using the Collection, Form, Display Card and Action Set previously created.

  • The business view is added to the selected application.

You can also accelerate the design of applications by duplicating objects such as the business views, form, collections and search forms.

Display Cards

Certain simple display properties are defined as part of the logical modeling effort: The Labels and Descriptions of the entities, attributes, types and references. These are used as default labels and descriptions when using entities, attributes or references in applications.

An entity contains several attributes (simple and complex). When an entity value needs to be displayed in a compact form (for example, in a collection, or in a single field), its Display Card is used.

The display card defines how an entity record appears in a list. It uses the entity’s attribute values to build the elements of the display card:

  • Primary Text: This text is displayed as the first line of the display card.

  • Secondary Text and Supporting Text: These texts appear as the second and third line when the component showing the display card requires them.

  • Primary Image: This picture is displayed for a grid list. If the primary image has a null value, it is replaced by the Fallback Image.

  • Avatar: This small picture is displayed in a round shape in a list.

Example 15. Display Card

The examples below show a display card used to show the same product record in a grid and a list.

Display card in a grid Display card in a list

In this example:

  • Primary Text is set to the Label attribute of the product.

  • Secondary Text concatenates the brand, family and sub-family of the product, using the following SemQL expression: FDN_Brand || ' (' || FDN_Family || ' - ' || FDN_SubFamily || ')'.

  • The Primary Image and the Avatar use the ImageURL attribute of the product.

To create an entity display card:

  1. Right-click the entity node and select Add Display Card.

  2. In the Create New Display Card wizard, enter the following values:

    • Name: Internal name of the object.

    • Default Display Card select this option to use this display card by default for the entity. Only one default display card can exist for an entity.

  3. Click Next.

  4. Use the SemQL editor to define the Primary Text Expression as well as the Secondary Text Expression

  5. Click Next.

  6. Use the SemQL editor to define the Primary Text Expression as well as the Secondary Text Expression

  7. Select a Primary Image and an Avatar from the image libraries. You can also define these using SemQL expressions.

  8. Click Finish to close the wizard.

  9. Press CTRL+S to save the Display Card editor.

  10. Close the editor.

Make sure you define at least one display card for the entities used in applications. A very basic display card uses the most representative attribute of the entity as the primary text.

Forms

Forms display the details for one record for a given entity.

A form is composed of:

  • Form Fields, displaying one SemQL expression built from the attributes of the record. For example, Customer Last Name or the Customer’s Account Manager Name are form fields.

  • Embedded Collections, displaying a list of records related to the current record. For example, the Email Addresses of the customer.

Fields and embedded collections are organized into Tabs and Sections. Non-visible Containers are also used to layout the fields and embedded collections in the form.

Creating a New Form

To create a form:

  1. Right-click the Forms node under an entity and select Add Form…. The Create New Form wizard opens.

  2. In the Create New Form wizard, enter the following values:

    • Name: Internal name of the object.

    • Default Form: Select this option if the form should be used as the default form for this entity.

  3. Click Finish to close the wizard. The Form editor opens.

When creating a form:

  • The Overview tab of the form editor contains the form properties

  • The Form tab of the form editor contains:

    • A tree table showing the containers, fields and collection composing the form. You can edit the name, label and value for each form component from the tree table.

    • A list of available Attributes. Drag these attributes into the form to create form fields.

    • The toolbar shows buttons to create, organize or delete tabs, sections, containers, form fields and embedded collections.

  • The Outline view shows a preview of the form layout. You can refresh this preview:

    • Manually, by clicking the Refresh Refresh button in the Outline view toolbar.

    • Automatically every time changes are made to the form, by toggling the Auto-synchronization Auto-Synchronization in the Outline view toolbar.

Designing the Form Layout

Form Layout Concepts

The form layout uses the Flex model, which accelerates the design and automatically adapts the form to the size of the screen at run-time.

In this model, elements arranged within a container depend on the Layout Direction of the container:

  • When the direction is Vertical, all elements are automatically arranged vertically and occupy all the horizontal space.

  • When the direction is Horizontal, all elements are arranged in a line, occupying equally the horizontal space.

    • If a Relative Width is defined for the elements in the form or a percentage or number that is a multiple of 5, or 33 or 66, then the elements occupy a horizontal space depending on this relative width. For example, setting the first element to 33 and the second to 66 creates a "1/3 + 2/3" layout.

    • At run-time, on smaller devices, the layout direction switches automatically to Vertical, and the elements stack up, avoiding horizontal scrolling.

Forms support three types of containers for layouts:

  • Tabs that only support the vertical direction.

  • Containers that support both vertical and horizontal directions, but have no label. These are non-visible containers, used for layout purposes.

  • Sections: They work like visible containers. They have a name and a visible label.

Example 16. Form layout

The mockup below illustrates the layout concepts:

Form Layout Example

In this example:

  • Tab_1 (vertical by default) contains Container_1 and Section_3. As Tab_1 is vertical, the contained elements are arranged vertically.

  • Section_3 (vertical) contains four form fields that stack vertically.

  • Container_1 (horizontal) contains Container_2 and Container_3, which appear side-by-side, each occupying half of the space.

  • Container_2 (vertical) contains two sections: Section_1 has a vertical direction and Section_2 is horizontal.

Creating Form Containers

To create a form tab:

  1. In the toolbar, select the Add Form Tab Add Form Tab button.

  2. Drag the button to the form tree table. A new tab appears.

  3. In the tree table, double-click the Name cell to edit the tab name.

  4. Repeat the operation for the Label.

  5. In the Properties view, optionally review and edit the tab properties:

    • Display Properties

      • Icon: Select an icon from the image library that can be used in addition or instead of the label.

      • Display Form Tab As: label, icon or icon and label

      • Visible: This boolean property indicates under condition the tab is visible.

      • Label Width: Width of the labels in this tab. This with can be expressed in pixels ('200px') but other css units are also supported. For example: '15%' or '100em'.

To create a form section:

  1. In the toolbar, select the Add Form Section Add Form Section button.

  2. Drag the button to the form tree table, in a tab or container. A new section appears.

  3. In the tree table, double-click the Name cell to edit the section name.

  4. Repeat the operation for the Label.

  5. In the Properties view, optionally review and edit the section properties:

    • Display Properties

      • Visible: This boolean property indicates under which condition the section is visible.

      • Layout Direction and Relative Width. See the form layout section for more information.

      • Label Width: Width of the labels in this section. This with can be expressed in pixels ('200px') but other css units are also supported. For example: '15%' or '100em'.

To create a form container:

  1. In the toolbar, select the Add Form Container Add Form Container button.

  2. Drag the button to the form tree table, in a tab or container. A new container appears.

  3. In the tree table, double-click the Name cell to edit the container name.

  4. In the Properties view, review and edit the container properties:

    • Display Properties

      • Layout Direction and Relative Width. See the form layout section for more information.

      • Label Width: Width of the labels in this container. This with can be expressed in pixels ('200px') but other css units are also supported. For example: '15%' or '100em'.

As you create containers to layout the form, use the Outline view so have a preview of the form layout.

Creating Form Fields

A form field, unlike a container, has a given Value, typically a SemQL expression.

To create a form field:

  1. In the Form tab of the Form editor, drag an attribute from the available Attributes list into a container in form tree table. The new form field is added into the container, with a Name and a Value equal to attribute name.
    You can also drag the Add Form Field Add Form Field button to the tree table to create a new form field with no value.

  2. In the tree table, double-click the Name cell to edit the field name.

  3. Repeat the operation for the Label.

  4. Click the Value cell, and then click the …​ (SemQL Editor) button to edit the field value.

You can select (by pressing the CTRL key) and then drag and drop multiple attributes from the attribute list.
Configuring Form Fields

A form field has a set of properties to configure its behavior and appearance.

The main property is the field’s Component Type. The component type is the UI component used to display and author this field in the form. Component types include Text, Image, Checkbox, Date Picker, etc.

You configure the field using properties common to all types (for example, the Label Color), and others are specific to the selected component type (for example, the Image Alignment).

  1. In the Form tab of the Form editor, select the form field to configure.

  2. In the Properties view, set the following properties:

    • Name and Definition

      • Name and Description : Define the name and technical description of this field.

      • Use Custom Label: Fields use by default the label and definition of their entity attribute, or a generated label for computed fields. Select this option to define a more specific label and documentation for the field.

      • Label: Define the label used on the form.

      • Documentation: Define the user-facing documentation for the field. Use plain text or the Markdown syntax for rich text. This text provides detailed documentation for the field. It appears in the documentation sidenav.

    • Display Properties

      • Component Type: Select a component type suitable for your field datatype.
        Component types are listed and described in Appendix A: Component Types and Properties

      • Visible: This boolean property indicates under which condition the field is visible.
        Note that this boolean value can be the result of a SemQL condition, which allows showing/hiding fields depending on record values. For example, You can hide specific fields (the Publisher for example) depending on the value of a built-in attribute called ViewType, which returns the code of the current view (GD for Golden, MD for Master, etc).

      • Relative Width: See the form layout section for more information.

      • Display Card: If the form field is a reference to a record, provide the display card used to display that reference.

      • Default Value Data Type: If your value is a SemQL expression, you can explicitly configure its datatype using this property. .In the Properties Tab, configure the other properties.
        All properties are listed in Appendix A: Component Types and Properties.

Common Form Fields Properties

The following properties are commonly used and when designing forms:

  • Label Position, Label Visible, Label Text Alignment, Label Color, Label Typography to position and style the label of a form field.

  • Icon, Icon Color to replace the label by an icons.

  • Text Typography, Text Alignment, Text Color, Background Color to style the field.

  • Display Format to provide specific formating for text values.

  • Height and Max Width to size components.

  • Display Type to chose the appearance of the component when it supports several.

  • Source Type to tell the component the nature of the value (for example, whether it is an image binary or a link to the image).

  • Helper Text, Editing Format to provide assistance on form fields when authoring.

All available properties are listed in Appendix A: Component Types and Properties.

Creating Embedded Collections

An embedded collection displays a list of records related to the record displayed in the form. For example, the Contacts related to the Company record being displayed.

To create an embedded collection:

  1. In the toolbar, select the Add Embedded Collection Add Embedded Collection button.

  2. Drag the button to the form tree table, in a tab or container. A new embedded collection appears.

  3. In the tree table, double-click the Name cell to edit the component name.

  4. Repeat the operation for the Label.

  5. Click the Value cell, and then click the …​ (SemQL Editor) button to edit the value.
    The Value for an embedded collection is the path to an entity related to the one displayed by the form. Select the path to this entity in the dialog and then click OK

  6. In the Properties view, optionally review and edit the embedded collection:

    • Name and Definition

      • Name, Label, Plural Label, Description and Markdown Documentation define by which name this collection appears and how it is described.

      • Action Set available on this collection. See Action Sets for more information about action sets.

    • Display Properties

      • Visible: This boolean property indicates under which condition the collection is visible.

      • Collection: Select the collection used to display the embedded collection.

      • Allow List, Table, Grid: Select which type of view is allowed for the embedded collection. Note that the selected type must be configured in the selected collection.

      • Relative Width: See the form layout section for more information.

      • Height: Height taken by the embedded collection, in pixels.

      • Filter: Filter applied to the data retrieved from the related entity.

      • Sort Expression: Sort expression applied to the data retrieved from the related entity.

      • User-Defined Sort: Check this box to allow users to customize the sort.

      • Limit: Maximum number or records retrieved (after filtering) from the related entity.

      • Display Record Count: Define whether the number of records should be displayed along the collection title.

The embedded collection configuration is finalized when you use the form in a business view. In the business view, you configure specific transitions for each embedded collection of the form. For more information, see Configuring Embedded Collection Transitions

Collections

Collections define how multiple records from the same entity appear.

A collection can appear in three different shapes, called View Types:

  • List uses the display card avatar, primary, secondary and supporting text values. It is suitable for simple lists and/or small devices

  • Grid uses the display card primary image, primary and secondary texts. It is suitable for records for which the image is the primary way of identifying a record.

  • Table corresponds to a regular table, into which the first column can also show the display card.

Example 17. Collection View Types

The following list view item uses the avatar image and two lines containing the primary and secondary text.

List view
View type: List

The following grid tile uses the primary image and two lines containing the primary and secondary text.

Grid view
View type: Grid

The following two-lines table row uses a set of columns including Product ID, Family, etc, plus a display card as the first column.

Table view
View type: Table with a display card column

A collection is typically composed of:

  • a Display Card used when displaying the collection as a list, a grid or for the first column of the table.

  • a list of Table Columns used when displaying the collection as a table.

Creating a New Collection

To create a collection:

  1. Right-click the Collection node under an entity and select Add Collection…. The Create New Collection wizard opens.

  2. In the Create New Collection wizard, enter the following values:

    • Name: Internal name of the object.

    • Default Collection: Select this option if the collection should be used as the default collection for this entity.

    • Display Card: Select one of the display cards of the entity. See Display Cards for more information.

  3. Click Finish to close the wizard. The Collection editor opens.

In the collection editor:

  • The Overview tab contains the collection properties

  • The Collection tab contains:

    • A table showing the table columns of the list. You can edit the name, label and value for each column from this table.

    • A list of Attributes available. Drag these attributes into the table to create columns.

    • The toolbar shows buttons to create, organize or delete table columns.

To configure the collection:

  1. In the Collection editor, select the Overview tab.

  2. Configure the properties for each view type:

    • Table

      • Allow Table: Select this option for this collection to support the table view type.

      • Line Nb.: Number of lines of each table row.

      • Display Card Column: Select this option to use the first table column to show the display card.
        Note that unlike the other column, this first column is locked at run-time and cannot be rearranged by users.

      • Display Card Column Label: Label used for the header of the display card column.

      • Display Card Column Default Width: Width of the display card column by default.

    • List

      • Allow List: Select this option for this collection to support the list view type.

      • Line Nb.: Number of lines of each list item.

      • Show Avatar: Select this option to display the avatar in the list.

      • Show Dividers: Select this option to display dividers between list items.

    • Grid

      • Allow Grid: Select this option for this collection to support the grid view type.

      • Line Nb.: Number of lines used for the text in the tiles.

      • Tile Height: Height of each tile in pixels, or the aspect ratio instead (e.g.: 16:9, 4:3, 1:1).

      • Number of Columns: Number of horizontal tiles in the grid.

      • Text Position: Position of the text in the tiles.

Creating Table Columns

If the Allow Table option is selected for the collection, then it can be displayed as a table, and you can add additional columns for this view type.

A table column has a given Value, typically a SemQL expression.

To create a table column:

  1. In the Collection tab of the Collection editor, drag an attribute from the available Attributes list into the table. The new column is added to the table, with a Name and a Value equal to attribute name.
    You can also drag the Add Table Column Add Table Column button to the tree table to create a new table column with no value.

  2. In the tree table, double-click the Name cell to edit the table column name.

  3. Repeat the operation for the Label.

  4. Click the Value cell, and then click the …​ (SemQL Editor) button to edit the table column value.

You can select (by pressing the CTRL key) and then drag and drop multiple attributes from the attribute list.
Configuring Table Columns

A table column has a set of properties to configure its behavior and appearance.

The main property is the column’s Component Type. The component type is the UI component used to display the value in the table. Component types include Text, Image, Checkbox, Date Picker, etc.

You configure the column using properties common to all types (for example, the Header Color), and others are specific to the selected component type (for example, the Image Alignment).

  1. In the Collection tab of the Collection editor, select the table column to configure.

  2. In the Properties view, set the following properties:

    • Name and Definition

      • Name, Label, Description: Define the name this field and the label used on the form.

    • Display Properties

      • Component Type: Select a component type suitable for your column datatype.
        Component types are listed and described in Appendix A: Component Types and Properties

      • Default Width: Default width of the column in pixels. Users can manually resize columns and reset them to this width.

      • Visible by Default: This boolean property indicates whether this column should be displayed initially in the table, or should only be in the list of available columns.

      • Display Card: If the table column is a reference to a record, provide here the display card used to display that reference.

      • Default Value Data Type: If your value is a SemQL expression, you can explicitly configure its datatype using this property. .In the Properties Tab, configure the other properties.
        All properties are listed in Appendix A: Component Types and Properties.

Common Table Columns Properties

The following properties are commonly used and when designing forms:

  • Header Color, Column Alignment to style the header and align the entire column.

  • Text Typography, Text Alignment, Text Color, Background Color to style the text is each cell.

  • Display Format to provide specific formatting for text values.

  • Max Width to size components.

  • Display Type to choose the appearance of the component when it supports several.

  • Source Type to tell the component the nature of the value (for example, whether it is an image binary or a link to the image).

All available properties are listed in Appendix A: Component Types and Properties.

Search Forms

A Search Form exposes several Search Parameters. When the user submits a search form, a search query is issued, using these parameters as bind values.

Search Forms are used to filter data in the business views or to search for records in the global search.

A search form exposes several Search Parameters to look for records in an entity. When the user submits the search form, a Search Query is issued, using these parameters as bind values.

To create a search form:

  1. Right-click the Search Forms node under an entity and select Add Search Form…. The Create New Search Form wizard opens.

  2. In the Create New Search Form wizard, enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label in this search form.

    • Description: Description of this search form.

  3. Click Finish to close the wizard. The Search Form editor opens.

  4. Define a Search Tip text that will be displayed in the search form.

  5. Add a search parameter:

    1. In the Search Parameters table, click the Add Search Parameter button. The Create New Search Parameter wizard opens.

    2. In the Create New Search Parameter wizard, enter the following values:

      • Name: Internal name of the object.

      • Binding: You use this binding string in the search form SemQL query condition to refer to this parameter. Note that as the Auto Fill box is checked, the Binding is automatically filled in. Modifying the binding is optional.

      • Label: User-friendly label for this search parameter. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

      • Use Type, Length, Scale and Precision to define logical data type of this parameter.

    3. Click Finish to close the wizard. The parameter appears in the Search Parameters table.

      • To edit a search parameter: Select the Properties view and then select the search parameter in the table. The Properties view shows all its properties, including the Name and Definition as well as the Search Parameters Display Properties.

      • To remove a search parameter, select the search parameter in the search parameters table, click the Delete button and then confirm the deletion.

    4. Use the Properties view to configure the search parameter.

  6. Repeat the previous step to create all your search parameters.

  7. Reorder search parameters in the table using the Move Up and Move Down buttons.

  8. Create a SemQL Condition for the search query by clicking the image Edit Expression button to open the SemQL Editor. Note that the SemQL editor displays the search form parameters bindings in the Variables section.

New search forms designed for an entity are not immediately available in the business views or applications using that entity. You must define a Search Configuration in the business views to enable or disable both the built-in search types and your own search forms.

Search Parameters Display Properties

The Display Type of a search parameter defines the component used to display this parameter in the search form. A Display Type is optionally configured through display properties.

The display type available for a search parameter depends on its logical data Type. For example, the Date Picker display type is available only for a DateTime search parameter.

The display types are listed below with their properties:

  • Text Field: Simple text field. This type is available for all logical data types.

  • Checkbox: Simple checkbox. This type is available for the Boolean logical data type.

  • Date Picker: Date (and time, for a timestamp datatype) selector. This display type is available for the Date and Timestamp logical types.

  • Drop Down List: Selectable list of elements. This display type is typically used to select a search parameter value from a list of values. It is configured using the following options:

    • Display Format: Defines whether the label and/or code of the list of values should be displayed.

    • Sort Order: Defines if the list of values should be sorted by the code or label.

  • Value Picker: Auto-complete component used to select filtered values from a given entity of the model. It is configured using the following options:

    • Lookup Entity: Entity from which the values are selected.

    • Filter Expression: SemQL expression used to filter the entity data. This expression can use other search parameters' values.

    • Bound Select Expression: SemQL expression defining the value returned by a value picker selection.

    • Primary Text Expression: SemQL expression defining the value displayed in the auto-complete items' first line.

    • Secondary Text Expression: SemQL expression defining the value displayed in the auto-complete items' second line.

    • Use Distinct: Check this option to only display distinct values in the auto-complete component.

    • Display Count: Check this option to display the count of records for each distinct value in the auto-complete. This option is available if Use Distinct is selected

    • Sort By Count: If Display Count is selected, this option allows sorting by ascending or descending record count.

    • Default Sort Expression: If Use Distinct is not selected, this SemQL expression defines the sort order of the records in the auto-complete.

The SemQL Condition defined for the search form can use attributes from the search entity as well as attributes from related (parent or child) entities. Refer to a search parameter with its Binding prefixed by a colon :.

For example, if a search form on the Customers entity has one parameter labeled Searched Name (binding: SEARCHED_NAME), the following SemQL condition filters customers by their name.

CustomerName like '%' || :SEARCHED_NAME ||  '%'

If we add to this form a second parameter labeled Has Influencer Contact (Binding: HAS_INFLUENCER_CONTACT), the following SemQL condition filters customers by their name and possibly, depending on the fact that they have one or more contacts with IsInfluencer = 1

CustomerName like '%' || :SEARCHED_NAME ||  '%'
and
( :HAS_INFLUENCER_CONTACT = 0
	or
  any Contacts have (IsInfluencer = 1)
)
For a detailed description of the SemQL language with examples, refer to the Semarchy xDM SemQL Reference Guide.

Steppers

When a user starts a authoring action (creating, editing, importing, etc.) on one or more records, he uses a Stepper.

Introduction to Steppers

A stepper defines the cluster of related records that is manipulated when authoring, as well as the wizard-like sequence of steps that drive the user through the authoring operation.

For example, a CreateContact stepper defines that:

  • Authoring Contacts consists in authoring Contacts and their attached Addresses. So the cluster of records contains several contacts and for each of them several addresses records.

  • Creating a contact goes through 3 steps: 1.General Info, 2.Addresses and 3.Comments. The second step (2.Addresses) is itself composed of two sub-steps: 1.Address Data and 2.Phone Numbers.

Collection and Form Steps

A stepper contains two types of steps:

  • Collection Steps show a set of records. The user may select one or more records and edit these records.

  • Form Steps show attributes of one record. A form step displays attributes using the layout and contents defined in a form section or tab. Multiple form steps are sequenced under a collection step, guiding users through the steps to create or modify a record.

When using a stepper to author master data on behalf of a publisher (See Authoring ID or Fuzzy Matched Entities for more information), only the first level of the stepper is used, and second level collections steps are ignored.
Triggers and Validations

As part of the data authoring flow, a steppers supports automated data transformations and data validations.

Automated transformations take place in the form of calls to Enrichers defined in the entities, or to Procedures declared in the model.

Data validations check that the data complies with data quality rules defined in the model, such as mandatory attributes, unique keys, SemQL validations, etc. When a data validation fails, it is raised to the user and may prevent that user from proceeding in the stepper.

Creating a New Stepper

To create a stepper:

  1. Right-click the Steppers node under an entity and select Add Stepper. The Create New Stepper wizard opens.

  2. In the Create New Stepper wizard, enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label in this stepper form.

    • On Finish Job: Normally this should be left empty. Semarchy xDM automatically generates a job for you. But in some advanced cases you may need the ability to specify a particular job to execute. In these cases you may select the job to execute when the user completes the stepper and submits the data.

    • Collection: Select the collection used to display the entity records at the root level of the stepper.

  3. Click Next.
    The Create Steps step of the wizard displays the forms defined for the entity, expanded to the section level.

  4. Select the tabs or sections from which you want to create steps:

    • Each selected form tab will appear as a form step showing all its child sections and form fields.

    • Each selected form section will appear as a form step showing all its form fields.

  5. Click Finish to close the wizard.

The Stepper editor opens. A root collection step is created in the Steps tree table, with child steps for the selected tabs and sections. Now you may configure the existing steps or add new steps.

Configuring Steps

Once the root level of the stepper has been defined, you can:

  • Add new collection steps to author child records.

  • Add new form step under existing form steps.

  • Configure the collection steps

  • Configure the stepper flow.

Adding a Collection Step

You can create a collection step to author records referencing the record currently handled by the stepper through a reference relationship.

For example, if a collection step manages Company records, you can create a child collection step to manage Contact records, as Contact references Company through a ContactBelongToCompany relationship.

To create a collection step:

  1. Identify the relationship you are interested in. The current collection step corresponds to the parent entity. Decide which child entity you want to add to the stepper and which reference joins the parent entity to the child entity.

  2. In the Stepper editor, scroll down to the Steps tree table.

  3. Select the collection step under which you want to create a collection step.

  4. In this table’s toolbar, click the Add Collection Step Add Collection Step button.
    The Create New Collection Step wizard opens.

  5. In the Create Collection Step wizard, enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label in this collection step.

    • Reference: Reference to follow to identify the child records to author.

    • Collection: Select the collection used to display the child records.

  6. Click Next.
    The Create Steps step of the wizard displays the forms defined for the child entity, expanded to the section level.

  7. Select the tabs or sections from which you want to create steps:

    • Each selected form tab will appear as a form step showing all its child section and form fields.

    • Each selected form section will appear as a form step showing all its form fields.

  8. Click Finish to close the wizard.
    A collection step is created, with child steps for the selected tabs and sections.

  9. Select this collection step and then use the Move Up and Move Down buttons in the toolbar to position the step.

Adding Form Steps

You can create additional form steps under collection steps.

To add form steps:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the collection step under which you want to create a form step.

  3. In this table’s toolbar, click the Add Form Step Add Form Step button.
    The Create New Form Step wizard opens.

  4. In the Create Form Step wizard, the tabs or sections from which you want to create steps:

  5. Click Finish to close the wizard.
    New child steps are created for the selected tabs and sections.

  6. Select these steps and then use the Move Up and Move Down buttons in the toolbar to position them.

Configuring Collection Steps

The collection steps show records being authored. Its aspect and content is configurable through certain properties.

To configure a collection step:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the collection you want to edit.

  3. In the Properties view, set the following properties:

    • Name and Definition

      • Customized Display Card: Select a display card used to represent the authored record in a child step of this collection step. Typically this is a Name and ID to easily identify the record.

    • Collection Data. Note that the root collection step is a specific collection step for which these properties cannot be configured.

      • Reference: Reference to follow to identify the child records to author.

      • Filter: This SemQL condition filters the records to checkout when performing an edit operation that automatically adds to the stepper data child records.
        For example, if the edit operation using this stepper should only add child records whose Status attribute is not set to Obsolete, use the following SemQL expression: Status <> 'Obsolete'

    • Display

      • Collection: Collection used to display the records in the collection step.

      • Default Display Type: Default view type used for this collection.

      • Customized Sort: Select this option to sort the records in the collection according to the Sort Expression.

      • Sort Expression: Sort expression applied to the data in the collection.

      • User-Defined Sort: Check this box to allow users to customize the sort.

      • Allow Table, Allow List and Allow Grid: Select those of the views available for the collection. These must be supported by the selected Collection.

      • Custom Collection Label: Label (plural) used as the title of the collection. It defaults to the Plural Role Label of the Referencing Entity for the selected reference.

Configuring the Stepper Flow

The stepper flow defines:

  • The visibility of steps depending on the authoring situation (whether you are creating or editing a record)

  • Whether a step is read-only or allows authoring.

  • The availability or options in each situation.

Collection Steps:

The following properties of the collection steps allow designers to configure the flow of the stepper.

  • In Name and Definition

    • Enable Finish: Adds a finish button on the step to finish the sequence of step it belongs to. This is useful when all subsequent steps are optional. See the Tips section on the next page for additional advice.
      On the root collection step, this allows the user to submit the data in the stepper.

    • Enable on Parent Create: Makes this step visible when the parent record is being created, or, for the root collection step, if the stepper was started to create a record.

    • Enable on Parent Edit: Makes this step visible when the parent record is being edited, or, for the root collection step, if the stepper was started to edit a record.

    • Read-Only: Makes this step read-only, preventing any data change operation.

    • Linear: Forces the user to go through the child steps in their order. If this option is unchecked, users can jump to any of the child steps.

    • Delete Type: Define the type of delete to apply when a user selects to delete a record edited in the stepper. See Entity Records Hard and Soft Delete for more information.

  • In On Parent Create

    • Skip to Child Creation on Parent Create: Skip the collection step and jump directly to the first visible child step when the parent record is being created, or, for the root collection step, if the stepper was started to create a record.

    • Enable Create, Enable Edit, etc: Activate the checked action in the stepper when the parent record is being created, or, for the root collection step, if the stepper was started to create a record. Note that Enable Remove only removes created records from this stepper, but does not delete golden data from the hub.

  • In On Parent Edit

    • Skip to Child Editing on Parent Create: Skip the collection step and jump directly to the first visible child step when the parent record is being edited, or, for the root collection step, if the stepper was started to edit a record.

    • Enable Create, Enable Edit, etc: Activate the checked action in the stepper when the parent record is being edited, or, for the root collection step, if the stepper was started to edit a record. Note that Enable Remove only removes records from this stepper, leaving them untouched in the hub. The Enable Delete option allows deleting golden data from the hub.

Form Steps

The following properties of the form steps allow designers to configure the flow of the stepper.

  • In Name and Definition

    • Enable Finish: Adds a finish button on the step to finish the sequence of step it belongs to.

    • Enable on Parent Create: Makes this step visible when it is used in a record creation operation.

    • Enable on Parent Edit: Makes this step visible when it is used in a record editing operation.

    • Read-Only: Makes this step read-only.

    • Enable Master Value Picking: This option applies to form steps for ID or fuzzy matched entities. It enables users overriding a consolidated value to select the override from the master records in addition to being able to enter their own values.

Tips for Configuring the Stepper Flow

The following guidelines help configuring the flow of a stepper:

  • Use Read-Only to create review steps. Users cannot do anything on these steps except move to the previous or next steps.

  • Use the Enable Finish on steps to make all subsequent steps in the sequence optional:

    • For example, if in a sequence of three form steps, the third is optional, make sure to Enable Finish on the second one.

  • Uncheck Linear on collection steps to define a free-form sequence of child steps. Non-linear steppers are usually preferred to modify records and linear steppers are preferred to create records.

  • Uncheck Enable Create, Enable Edit, etc, options to simplify the choices given to the users on steps, but remove them carefully:

    • For example, when you create a stepper specifically to edit existing records, disable On Parent Edit > Enable Create.

    • If the user needs to create child record in the context of editing (for example, to create a new Address when editing an existing Contact), make sure to Enable Create accordingly.

  • Use the Skip to …​ options to shortcut the creation or editing experience. When this option is selected, the user will not see the collection and will not have to click on the Edit or Create action in the collection.

    • To simplify new Contact creation and have the user arrive directly on the first form step, check the Skip to Child Creation on Parent Create option on the root collection step of the stepper.

Configuring Triggers and Validations

In a stepper, it is possible to automate data changes using triggers and enrichers. It is also possible to run data validations and raise issues to the user.

Data changes or validations run when certain events take place in the stepper. For example, when the user enters a stepper or a step, modifies a field value or leaves a step.

Configuring Triggers and Enrichers

Automated data transformations include:

  • Stepper Triggers that execute database procedure calls when the stepper starts or finishes.

  • Step Triggers that execute enrichers or database procedure calls when the user enters, leaves or performs certain operations in the step.

  • Form Step Enrichers that run enrichers when the user opens a form step or modifies data in this form step.

Triggers refer either to Enrichers or Database Procedures.

Stepper Triggers

A stepper trigger can execute a procedure declared in the model. Enrichers cannot be attached to the stepper as procedures can. This is because enrichers always affect an individual record and must therefore be attached to the relevant step. Procedures may be attached to steps or to the stepper itself.

To create a stepper trigger:

  1. In the Stepper editor, scroll down to the Triggers table.

  2. In the table toolbar, select the Add Stepper Procedure Trigger Add Stepper Procedure Trigger button. The Create New Stepper Trigger wizard opens.

  3. In the Stepper Trigger wizard step, enter the following values:

    • Name: Internal name of the object.

    • Procedure: Select a Database function defined as a procedure in the model.

    • Event: Select the event that should cause this procedure to run.

  4. Click Next.
    The Configure Procedure step of the wizard displays the arguments of the selected procedure.

  5. For each argument, click the image Edit Expression button to create a SemQL expression that will be bound to the argument.

  6. Click Finish to close the wizard. A stepper trigger is created

  7. Select this stepper trigger and then use the Move Up and Move Down buttons in the toolbar to order the list of triggers. Triggers, for a given event, are executed in the list order.

Step Triggers

A step trigger can execute either a procedure or an enricher on specific step events.

To create a step trigger:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the step you want to modify.

  3. In the Properties view, select the Step Triggers finger tab.
    This tab shows a table with the list of triggers for the step.

  4. In the table toolbar, select the Add Step Procedure Trigger Add Step Procedure Trigger button or the the Add Step Enricher Trigger Add Step Enricher Trigger button. The Create New Step Trigger wizard opens.

  5. In the Step Trigger wizard step, enter the following values:

    • Name: Internal name of the object.

    • Procedure/Enricher: Select a database function declared as a procedure or an enricher.

    • Event: Select the event that should cause this procedure or enricher to run.

  6. If you create a Step Procedure Trigger:

    1. Click Next. The Configure Procedure step of the wizard displays the arguments of the selected procedure.

    2. For each argument, click the image Edit Expression button to create a SemQL expression that will be bound to the argument.

  7. Click Finish to close the wizard. A step trigger is created

  8. Select this step trigger and then use the Move Up and Move Down buttons in the toolbar to order the list of triggers. Triggers, for a given event, are executed in the list order.

Form Step Enrichers

A form step enricher executes an enricher when the user enters a form or modifies the value of a form field.

To configure step enrichers:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the form step you want to modify.

  3. In the Properties view, select the Form Step Enricher finger tab.
    This tab shows a table with the list of enrichers defined for the fields that appear for this form step.

  4. Click the Synchronize Form Step Enrichers Synchronize Form Step Enrichers button in the table toolbar to refresh the list with new enrichers.

  5. Check the appropriate option to run enrichers:

    • On Form Open: When the user opens the form. Note that the enricher is triggered every time the user opens the form, including when resuming the stepper or when returning from another step.

    • On Data Change: When the user changes the value of an attribute that is an input for the enricher.

Note that data change is detected when the user leaves the field, when a selection is made in the field (for example, for menus) or after a certain time when the user has performed a change but remains in the field.
Stepper Events

This section lists the various events that may cause triggers to start for steps and steppers.

The detailed list of events and their corresponding user actions are listed in Appendix B: Stepper Events Reference

Events for Steppers

A user may exit a stepper using different actions:

  • Finish: The user completes the stepper and submits the records for certification.

  • Close > Discard: The user completes the stepper and discards all its content. The stepper is canceled and cannot be resumed.

  • Close > Saved: The user completes the stepper and saves its content for later. He can resume this stepper later.

The following events occur once when a user enters or exits a stepper.

  • Stepper - Start: Run when the stepper starts

  • Stepper - Finish (Pre Validation): Run once, when the user finishes the stepper, before the stepper validations

  • Stepper - Finish (Post Validation): Run once, when the user finishes the stepper, after the stepper validations

  • Stepper - Close (Discard): Run once, when the user closes the stepper and discards data changes.

  • Stepper - Close (Saved): Run once, when the user closes the stepper and saves draft data

Events for Steps

A user may exit a step using different actions:

  • The user clicks the Continue or Back buttons.

  • The user clicks (for a non-linear stepper) a step in the header to jump to that step.

  • The user clicks the Close button to go one level up in the stepper.

The events behavior differ for collection steps and form steps:

  • Event on Steps apply to both collection steps and form steps.

  • Events specifically indicated for Collection Step do not apply to form steps.

  • Events used in form steps run once for the record managed in the form step.

  • Events used in collection steps have two flavors, indicated with (Each) or (Once):

    • (Each): This type of event runs once for each record in the collection step. It gives you access to the attribute of these records.

    • (Once): This type of event runs once for all record in the collection step. It gives you access to the attributes of the parent record of these records. From this parent, you can use SemQL’s any|all syntax.

For a detail of the sequence of events triggered for each user action, see Appendix B: Stepper Events Reference.

The following events occur depending on user actions for form and collection steps:

  • Step - Enter (Once): Run once, when a step is entered

  • Step - Enter (Each): Run for each record, when a step is entered

  • Collection Step - Add Child: Run whenever a child record is added to this collection (directly or through the ADD ANOTHER button of a child step)

  • Collection Step - Copy Child (Once): Run once, when a child record is copied in this collection. Note that you can use in such trigger the CopiedFrom built-in attribute which contains the ID of the source record that was copied.

  • Collection Step - Copy Child (Each): Run for each copied record, when a copy child record action is performed in this collection. Note that you can use in such trigger the CopiedFrom built-in attribute which contains the ID of the source record that was copied.

  • Collection Step - Edit Child (Once): Run once, when a child record is edited on this collection (directly or through the EDIT NEXT button of a child step)

  • Collection Step - Edit Child (Each): Run for each record, when a child record is edited on this collection (directly or through the EDIT NEXT button of a child step)

  • Collection Step - Import (Once): Run once when a user completes an import

  • Collection Step - Import (Each): Run for each imported record, when a user completes an import

  • Collection Step - Remove Child (Once): Run once, when child records are removed from this collection

  • Collection Step - Remove Child (Each): Run for each record, when child records are removed from this collection

  • Step - Continue (Once): Run once, when a user clicks the CONTINUE button

  • Step - Continue (Each): Run for each record, when a user clicks the CONTINUE button

  • Step - Back (Once): Run once, when a user clicks the BACK button

  • Step - Back (Each): Run for each record, when a user clicks the BACK button

  • Step - Close (Once): Run once, when a user clicks the CLOSE button on a step (whether to get back to the parent collection or to close the stepper)

  • Step - Close (Each): Run for each record, when a user clicks the CLOSE button on a step (whether to get back to the parent collection or to close the stepper)

  • Step - Exit (Pre validation - Once): Run once, when a user exits a step, before data validation

  • Step - Exit (Pre validation - Each): Run for each record, when a user exits a step, before data validation

  • Step - Exit (Post validation - Once): Run once, when a user exits a step, after data validation

  • Step - Exit (Post validation - Each): Run for each record, when a user exits a step, after data validation

  • Collection Step - Close Child (Once): Run whenever a child step of this collection is closed

  • Collection Step - Close Child (Each): Run whenever a child step of this collection is closed

Configuring Validations

Validation types include:

  • Stepper Validations check data in any entity modified in the stepper in order to warn or block the user. They run when the stepper finishes.

  • Step Validations check data in the entity modified in the step in order to warn or block the user. They run when the step completes.

  • Form Validations check data in the entity modified in the step in order to warn or block the user. They run when the user opens or modifies data in the form step.

Validations refer to the data quality rules defined for the entities, including:

  • FOREIGN to check a that a referenced value exists.

  • MANDATORY to check that a mandatory attribute or reference is not left null.

  • CHECK to run a SemQL or Plug-in Validation.

A special validation type called DETECT_DUPS is available for Stepper and Step Validations. When the user is creating a new record, this validation type run the matching rule defined for the entity and shows possible duplicates, allowing the user to replace the new record in the stepper by an existing duplicate.
Stepper Validations

A stepper validation runs a validation when the stepper completes and the user tries to apply the data changes.
If a validation fails, it is raised to the user, and may possibly prevent him from applying the data changes.

To configure stepper validations:

  1. In the Stepper editor, scroll down to the Validations table. This table shows the list of validations defined for the entities managed in the stepper.

  2. Click the Synchronize Stepper Finish Validations Synchronize Stepper Finish Validations button in the table toolbar to refresh the list with new validations.

  3. Select the validation you want to run or select multiple validations while keeping the <CTRL> key pressed.

  4. Right-click and then select the behavior of this validation:

    • Skip: Do not run this validation

    • Warn: Run this validation and raise a failure as an issue to the user. The user may choose to ignore the warning.

    • Block: Run this validation and raise a failure as a blocking issue to the user. As long as blocking issues exist, the user cannot proceed.

Step Validations

A step validation runs a validation when the user leaves a step.
If a validation fails, it is raised to the user, and may possibly prevent him from leaving the current step.

To configure step validations:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the step you want to modify.

  3. In the Properties view, select the Validations finger tab.
    This tab shows a table with the list of validations available for the entity managed in the step.

  4. Click the Synchronize Step Validations Synchronize Step Validations button in the table toolbar to refresh the list with new validations.

  5. Select the validation you want to run or select multiple validations while keeping the <CTRL> key pressed.

  6. Right-click and then select the behavior of this validation:

    • Skip: Do not run this validation

    • Warn: Run this validation and raise a failure as an issue to the user.

    • Block: Run this validation and raise a failure as a blocking issue to the user. As long as blocking issues exist, the user cannot proceed.

Form Validations

A form validation runs a validation when the user enter or modifies a form. Such a validation does not block the user, but raises a message under the field(s) that causes the validation failure. Combine the use of a form validation (which shows useful information but is non-blocking) with a step validation (blocking) if the user should be forced to fix an issue before proceeding.

To configure form validations:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the form step you want to modify.

  3. In the Properties view, select the Form Validations finger tab.
    This tab shows a table with the list of validations available for the entity managed in the step.

  4. Click the Synchronize Form Step Validations Synchronize Form Step Validations button in the table toolbar to refresh the list with new validations.

  5. Check the appropriate option to run validations:

    • On Form Open: When the user opens the form.

    • On Data Change: When the user changes the value of an attribute that is an input for the validation.

Configuring References Management

Reference Selection

If a form step contains Reference fields, then it is possible to configure the selection of these references.

This selection is performed using one of the following components:

  • An auto-complete field that lists selectable records filtered by a search expression as you type in.

  • A reference selection dialog that shows a collection of selectable records. This second option is better if you need to have a detailed view of the records to select.

To configure reference selection:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the form step containing the references.

  3. In the Properties view, select the References Selection finger tab.
    This tab shows a table with the list of references managed in the step.

  4. Click the Refresh References Refresh References button in the table toolbar to refresh the list with new references.

  5. Configure the references:

    • Reference Filter Type: Select None if you do not want the reference to be selectable. Auto-Complete displays an auto-complete component to select the reference. Collection displays the list of selectable records in a reference selection dialog.

    • Picker Filter: This filter restricts the possible selection for both the auto-complete and reference selection dialog. Such a filter is used for example when selecting the Customer referenced by a Contact, to filter only those of the Customers located in the same country as the Contact.

    • Sort Expression: This sort expression orders the data in the selectable records.

  6. If you choose the Collection type, you must configure the following options:

    • Picker Collection View: Select the collection used to show the selectable records in the reference selection dialog.

    • Search Configurations: Click this cell to select and order search methods that appear in an optional search step preceding the collection of selectable records. If no search method is selected as Available, then this seach step does not appear before the collection.

    • User-Defined Sort: Check this box to allow users to customize the sort in the reference selection dialog.

Many to Many Creation

If a child collection step points to a child entity that represent a many to many relation, then it is possible to configure selecting the record at the other side of the many to many relation to create the child records in the many to many entity.

For example, a ProductStepper stepper has a child collection step to create ProductMarkets. ProductMarket is an entity representing the many to many relation between Products and Markets. When the user creates ProductMarkets, he wants in fact to select Markets, and have ProductMarkets created. This configuration aims at giving this capability.

To configure many to many creation:

  1. In the Stepper editor, scroll down to the Steps tree table.

  2. Select the collection step that manages an entity representing the many to many relation.

  3. In the Properties view, select the Many to Many Relation finger tab.

  4. Select the Many to Many Relation checkbox, and then configure the Many to Many Creation options:

    • Target Foreign Attribute: Select the foreign attribute that representing the other side of the many to many relation.

    • Picker Collection View Select the collection to use to display the selectable records at the other side of the many to many relation.

    • Picker Filter: SemQL condition pre-filtering the selectable records at the other side of the many to many relation.

    • Search Configurations: Select the type of search used to filter the selectable records at the other side of the many to many relation.

    • Customized Sort: Select this option to sort the selectable records according to the Sort Expression.

    • Sort Expression: Sort expression applied to the data in the selectable records.

    • User-Defined Sort: Check this box to allow users to customize the sort.

    • Default Display Type: Default view type used for the collection showing the selectable records.

    • Allow Table, Allow List, Allow Grid: Select the views available for the collection. These must be supported by the selected Picker Collection.

Duplicate Managers

A Duplicate Manager defines the user interface into which data stewards review, merge, split groups of matching records, and override the values of the golden consolidated records.

A duplicate manager is used when the user triggers a duplicate management action with a selection of records. Actions include:

  • Review and Confirm Duplicates to confirm groups of fuzzy matching records after reviewing these groups into details.

  • Merge or Split Duplicates to reorganize groups of fuzzy matching records.

  • Review Duplicates Suggestions to accept, reject or reorganized suggestions made on groups of fuzzy matching records.

In a duplicate manager:

  • Actions on matching groups and suggestions are performed using two alternate views:

    • A Graph view, into which master records, golden records, and suggestions appear as graph nodes. This view helps data stewards understanding groups/suggestions as well as the match rules that relate master records. The content of the nodes is defined by a Display Card.

    • A Table view, into which master records, golden records, and suggestions appear as rows in a tree table. This view helps data stewards understanding groups/suggestions, even for very large groups and suggestions, but relations between master records (match rules) are not exposed. The appearance of the table is defined by a Table View Collection.

  • The user may select and add records to the duplicate management operation. He selects these records from a Collection, optionally sorted and filtered according to search configurations.

  • The user can visualize the details of a record as defined in a selected Form Tab.

  • The same form tab is also used in the Explain Record view that shows the master records values that consolidate to compose a golden record.

  • The form tab is also used as an authoring form to override the values of the consolidated golden record.

A duplicate manager provides features that are available to the user depending on the configuration of the actions in the action set that use this duplicates manager.

To create a duplicate manager:

  1. Right-click the Duplicate Managers node under an entity and select Add Duplicate Manager. The Create New Duplicate Manager wizard opens.

  2. In the Create New Duplicate Manager wizard, enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object.

    • On Finish Job: Normally this should be left empty. Semarchy xDM automatically generates a job for you. In some advanced cases, you may need the ability to specify a particular job to execute. In these cases, you may select the job to execute when the user completes the duplicate manager and submits the changes.

    • Collection: Select the collection used to select records to add to the duplicate manager.

    • Display Card: Select the display card used to represent the records in the graph.

    • Form Tab: Select the form tab used to show or edit values of a record.

  3. Click Finish to close the wizard. The Duplicate Manager editor opens.

  4. Configure how value overrides take place for golden records:

    • Enable Master Value Picking: This option enables users overriding a consolidated value to select the override from the master records in addition to being able to enter their own values.

  5. By default, only the Graph view is enabled for managing duplicates. If you want to enable the Table view, select a Table View Collection in the Display Section. Note that the selected collection should have the Allow Table and the Display Card Column options selected.

  6. Configure in the Golden Record Selection section how records are selected and added to the duplicate manager:

    • Collection: Select the collection used to show the list of records.

    • Search Configuration: Click the Edit button to select and order search methods available to filter the list of records. The user will be able to filter the records using the selected search methods before seeing them in the collection.

    • Select Customize Sort and enter a SemQL Sort Expression to sort the records in the collection. You can also enable User-Defined Sort for this collection.

    • Select the available display type for the collection using the Allow Table, Allow List and Allow Grid properties, and select one of these as the Default Display Type.

Workflows

Workflows enable business users to manage the data stored in the MDM hub using an application.

You can accelerate the design of applications by duplicating existing workflows.

Creating a New Workflow

To create a workflow:

  1. Right-click the Workflows node and select Add Workflow…. The Create New Workflow wizard opens.

  2. In the Create New Workflow wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the workflow.

    • Label: User-friendly label for this workflow. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Administrator Role: Role required to administer this workflow. Users with this role can perform any operation on this workflow.

    • Job: Select the job to execute when the workflow completes with a Submit operation. Do not select any job to have a job automatically generated for this workflow based on the manipulated data.

  3. Click Next. In the next page, configure the first task of the workflow:

    • Name: Internal name of the task.

    • Label: User-friendly label for this task. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Assigned to Role: Select the role to which this task is automatically assigned.

    • Entity: Select the entity which data is manipulated in the first task.

    • Authoring Stepper: Select a stepper from that entity that is used for the first authoring task.

  4. Click Finish to close the wizard. The Workflow editor opens.

The Workflow Editor

The workflow editor displays the workflow as a diagram. This diagram is used to configure, add tasks and transitions to the workflow.

This workflow contains by default the following elements:

  • A Start Event (circular shape) that represents the startup point for this workflow. All the start tasks are linked from this event.

  • An End Event (circular shape) that represents the completion point for this workflow. This event is preceded by two built-in tasks called Submit Data and Cancel Data that represent the submit and cancel operations that finish a workflow.

  • The first Task that is linked from the Start Event and that links to the Submit Data and Cancel Data built-in tasks.

It is possible to link the start event to your tasks, but your tasks cannot link directly to the end event. They must transition to the built-in Submit Data and Cancel Data tasks that finish the workflow.

Configuring the Workflow

The workflow can be configured from the Properties view. Click the background of the workflow diagram to open the workflow properties.

All workflow have the following properties sections:

  • Name and Definition: The Name, Label, Description, Administrator Role and Job options in this section are configured when creating the workflow and can be changed here. The additional Editable Properties allows you to define whether workflow properties (the label, for example) can be modified after startup.

  • Tasks: The list of tasks of the workflow.

  • Transitions: The list of transitions of the workflow.

Adding a Task

To add a task from the diagram:

  1. In the Workflow diagram, select the Add Workflow Task tool in the Palette.

  2. Click the diagram. The Create New Task wizard appears.

  3. In the Create New Task wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the task.

    • Label: User-friendly label for this task. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • Assigned to Role: Select the role to which this task is automatically assigned. It is the role required for the performer of this task.

    • Select the authoring method for this workflow task, that is one Entity and one Authoring Stepper from that entity. This authoring stepper is used when the task is started.

  4. Click Finish to create the task.

The new task appears in the diagram, with a transition automatically created to the Cancel built-in task. If it is the first task added to this workflow, it is linked from the start event.

Configuring a Task

A task selected in the workflow is configured from the Properties view.

You can configure the following properties:

  • In the Name and Definition section:

    • The Name, Label, Description, Assigned to Role, Entity and Authoring Stepper properties, configured when created the task, can be modified.

    • Assigner Role defines the role required for a user to be able to assign to a user a task pending with the Assigned to Role. A user who has the workflow Administrator Role is also able to perform the same operation.

  • In the Task Management section:

    • Allow Unassign enables the task assignee to release the task to the task’s Assign To Role.

    • Allow Reassign enables the task assignee to assign the task to a different user who has the Assign To Role.

    • Max Pending Duration (min) define the time after which a notification is sent when the task is left assigned to the role and not to a user.

    • Max Running Duration (min) define the time after which a notification is sent when the task is left assigned to a user.

    • Max Total Duration (min) define the time after which a notification is sent when the workflow remains on the task.

  • In the Email Notifications section, define the email notifications sent on task triggers.

Configuring Email Notifications in Tasks

When an event occurs in a task, it can trigger an email notification sent to a list of recipients.

To configure an email notification in a task:

  • Select the Email Notifications section of the workflow task properties. This table section shows the list of notifications for this task.

  • Click the Add Notification button Add Notification button in the properties panel toolbar. A new notification is added to the list.

  • Click the Name cell for this notification and edit this notification’s name.

  • Click the Subject cell and then the image Edit Expression button to edit the email subject. The subject supports variables that return contextual information about the workflow.

  • Click the Message cell and then the image Edit Expression button to edit the email body. the body supports variables as well as plain text or the Markdown syntax for rich text.

  • Click the Triggers cell to configure the event that should trigger this notification.
    A notification may be triggered when the task starts, finishes, is assigned/unassigned/reassigned, or when the max durations defined when configuring the task are reached.

  • Click the Recipient cell to configure the recipients for this notification.
    Recipients are contextual to the current task/workflow, or to a Select Task in the workflow. You can also define a list of emails in the Other Recipients.

Configuring Triggers, Enrichers and Validations in Tasks

A task of a workflow provides interactive feedback to the user according to the stepper selected for the task.

The triggers, enrichers and validations defined in the steps are executed, but the stepper-level Triggers and Validations, running at the end of the stepper, are not executed. They are replaced in workflows by the Triggers and Validations defined for the transitions.

Configuring the Submit and Cancel Tasks

The Submit and Cancel built-in tasks are specific corresponding to the workflow’s possible ends.

  • Cancel does not require configuration other than a transition pointing to it.

  • Submit supports only configuring email notifications with one single type of trigger called Job Completes. This trigger runs the notification when the submit job of the workflow completes.

Adding a Transition

A transition links to tasks in the diagram.

To add a transition from the diagram:

  1. In the Workflow diagram, select the Add Transition tool in the Palette.

  2. Select a task the diagram or the Start event. Keep the mouse button pressed, and move the cursor to the next task in the workflow, or the built-in Submit or Cancel tasks.

  3. Release the mouse button.

The transition is created and a link appears between the two elements in the diagram.

Transitions have a direction. If a transition goes from Task_A to Task_B, it only means that you can move in the workflow from Task_A to Task_B. If you want to move from Task_B to Task_A, then you must create another transition in the other direction.
It is possible to create multiple transitions between two tasks, even in the same direction. For example, you can have two transitions from Task_A to Task_B, each transition having a different configuration.
Configuring a Transition

A transition selected in the workflow is configured from the Properties view.

You can configure the following properties:

  • In the Name and Definition section:

    • The Name, Label and Description for the transition.

    • The Icon displayed along with the Label when selecting this transition.

    • The Required Role defines the role required for a user to view this transition.

    • Ask for Comments prompts for comments when the task completes.

  • In the Stepper Configuration section:

    • Stepper Mode defines how the stepper of the next task starts:

      • Create Single Record opens the stepper in a mode that allows the user to create only one record and then finish.

      • Edit Single Record opens the stepper in a mode that allows the user to review/author only one record (the first in the list) and then finish.

      • Multi-Records opens the stepper in a mode that allows the user to manipulate multiple records, according to the other properties, including the Multi-Record Start Action.

    • Multi-Record Start Action defines how a stepper started in a Multi-Records stepper mode behaves

      • Edit First Record opens the stepper on the first record, allowing to move to the next records for review/authoring.

      • Create New Record opens the stepper for the creation of a new record, allowing afterward to review/author the other records.

      • Import opens the stepper on the new records import wizard.

      • Import Update opens the stepper on the new records import wizard. These records are imported to update existing records.

      • Display Root Collection opens the stepper on list of records manipulated in the workflow.

    • Enable Create, Enable Edit, etc: Select the actions to allow in the stepper.

  • In the Assignees section:

    • Candidate Assignees defines the list of candidate assignees (users) for the next task. This list appears when a user completes the previous task via this transition.

      • No Candidate: No candidate users. The Assigned To role defined for the next task must be included to the list.

      • Members of Next Task’s Role: All the members of the Assigned To role defined for the next task.

      • Workflow Performers: All the users who have collaborated to this workflow.

      • Previous Task Performer: The user who has performed the previous task in the workflow.

      • All Performers of Selected Task: All the users who have collaborated to the Selected Task.

      • Last Performer of Selected Task: The user who has last worked on the Selected Task.

      • User Name from a Variable: User whose name is returned by the Assignee Name Variable.

    • Selected Task and Assignee Name Variable must be set depending on the Candidate Assignees configuration.

    • Include Assigned to Role adds to the list of users the Assign To Role defined for the next task. This option must be selected is Candidate Assignees is set to No Candidate.

The transition from the Start Event (circular shape) to the startup task of the workflow cannot be configured. The first task’s stepper automatically starts with a mode that depends on the action from the action set that was triggered and the record selection that was made. Transitions to the End Event (circular shape) cannot be edited.

Configuring Email Notifications in Transitions

When a transition is executed, it can trigger an email notification sent to a list of recipients.

To configure an email notification for a transition:

  • Select the Email Notifications section of the workflow transition properties. This table section shows the list of notifications for this transition.

  • Click the Add Notification button Add Notification button in the properties panel toolbar. A new notification is added to the list.

  • Click the Name cell for this notification and edit this notification’s name.

  • Click the Subject cell and then the image Edit Expression button to edit the email subject. The subject supports variables that return contextual information about the workflow.

  • Click the Message cell and then the image Edit Expression button to edit the email body. the body supports variables as well as plain text or the Markdown syntax for rich text.

  • Click the Recipient cell to configure the recipients for this notification.
    Recipients are contextual to the transition’s previous and next tasks, to the workflow, or to a Select Task in the workflow. You can also define a list of emails in the Other Recipients.

Configuring Triggers and Validations in Transitions

A transition of can perform data transformations and enforce data quality checks. These checks warn the user of possible data issues and can optionally block the transition.

The following sections are available in the Properties view for configuring data transformations and data quality checks on transitions:

  • The Triggers section contains a list of procedures executed before or after the validations when the transition executes.

  • The Validations section contains the list of validations applicable to the entities managed in the task. Each validation is configured to Warn the user, Block the transition, or you can simply Skip it.

Configuring Triggers in Transitions

A transition trigger can execute a procedure declared in the model.

To create a transition trigger:

  1. In the Transition properties, select the Triggers section.

  2. In the table toolbar, select the Add Trigger Add Trigger button. The Create New Workflow Transition Procedure wizard opens.

  3. In the Workflow Transition Procedure wizard step, enter the following values:

    • Name: Internal name of the object.

    • Procedure: Select a database function defined as a procedure in the model.

    • Trigger Type: Select whether the trigger should run before or after the validations defined for this transition.

  4. Click Next.
    The Configure Procedure step of the wizard displays the arguments of the selected procedure.

  5. For each argument, click the image Edit Expression button to create a SemQL expression that will be bound to the argument.

  6. Click Finish to close the wizard. A trigger is created

  7. Select this trigger and then use the Move Up and Move Down buttons in the toolbar to order the list of triggers. Triggers, for a given Trigger Type, are executed in the list order.

Configuring Validations in Transitions

A validation runs a data quality check when the transition executes.
If a validation fails, it is raised to the user, and may possibly prevent him from proceeding.

To configure transition validations:

  1. In the Transition properties, select the Validations section. This table shows the list of validations defined for the entities managed in the workflow.

  2. Click the Synchronize Stepper Finish Validations Synchronize Transition Validations button in the table toolbar to refresh the list with new validations.

  3. Select the validation you want to run or select multiple validations while keeping the <CTRL> key pressed.

  4. Right-click and then select the behavior of this validation:

    • Skip: Do not run this validation

    • Warn: Run this validation and raise a failure as an issue to the user. The user may choose to ignore the warning and proceed.

    • Block: Run this validation and raise a failure as a blocking issue to the user. As long as blocking issues exist, the user cannot proceed.

Action Sets

Action sets are groups of actions that appear together in a business view menu to modify one or more records.

Possible actions, depending on the entity type, include:

  • Create new records.

  • Edit selected records.

  • Import records to create new records or update existing records.

  • Copy selected records.

  • Mass-Update selected records or all records.

  • Confirm Duplicates to directly confirm groups of fuzzy matching records.

  • Review and Confirm Duplicates to confirm groups of fuzzy matching records after reviewing these groups into details.

  • Merge or Split Duplicates to reorganize groups of fuzzy matching records.

  • Review Duplicates Suggestions to accept, reject or reorganized suggestions made on groups of fuzzy matching records.

  • Delete selected records or all records.

  • Export selected records or all records to Excel or CSV format.

  • Browse Graph for a given record to graphically navigate the relations from and to this record.

  • Explain Record for a fuzzy matched record to graphically view the matches and consolidated values.

Actions that create or modify records (Create, Edit, Import, Copy and Mass-Update) require a Stepper or a Workflow to perform the operation. Similarly, actions that manage duplicates (Merge or Split Duplicates, Review Duplicates Suggestions) require a Duplicate Manager and are available only for Fuzzy Matched Entities.

Creating Action Sets

To create an action set:

  1. Right-click the Action Sets node under an entity and select Add Action set. The Create Action Set wizard opens.

  2. In the Create New Action Set wizard, enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this action that appears in the action menu.

  3. Select the Create Default Action to have the wizard seed default actions in the action set.

  4. Click Finish to close the wizard.

The Action Set editor opens, optionally with default actions created.

To add new actions:

  1. In the Action Set editor, scroll down to the Actions table.

  2. Click the Add Action Add Action button in the table toolbar.
    The Create New Action wizard opens.

  3. In the wizard:

    • Select a Stepper or a Workflow if you want to add data authoring actions. Note that stepper and workflow selections are mutually exclusive.

    • Select a Duplicate Manager if you want to add duplicate management actions.

    • Select the actions (Create, Edit, Import, etc). you want to add. Note that authoring actions are locked until you select a stepper that is valid for these actions, and certain duplicate management actions are unavailable unless you select a Duplicate Manager.

  4. Click Finish to close the wizard.
    The selected actions are added to the action set.

  5. Use the Move Up and Move Down buttons to reorder the actions in the action set. The items in the action menu will appear that order.

Valid Steppers for Actions

Entities and steppers support actions under certain conditions:

  • Create: Stepper’s Root collection step must have one child step enabled on parent create.

  • Edit, Mass-Update: Stepper’s root collection step must have one child step enabled on parent edit.

  • Import: Stepper’s root collection step must enable child import on parent create.

  • Copy: The entity ID generation must be UUID or Sequence and the stepper’s root collection step must enable child copy on parent edit.

  • Delete: The entity must have the Delete Enabled option selected.

Configuring Actions

To configure actions:

  1. In the Action Set editor, scroll down to the Actions table.

  2. Select an action in the table.

  3. In the Properties view, configure the action properties:

    • Name and Definition

      • Name, Label, and Description.

      • Required Role: Optionally select a role required to perform this action.

      • Icon: Icon representing the action in the menu.

    • In the Action Configuration tab the Condition (available for Delete, Mass-Update Edit, Copy and duplicate management actions): the SemQL condition that must be met by all select records for this action to be available.

    • If configuring a data authoring action, in the Action Configuration tab, set the following properties:

      • Stepper/Workflow (only for authoring actions): Stepper or workflow used to perform the action.

      • Support Multiple Creation (available for Create): Select this option to support cyclic record creation.

      • Created Record Type (available for Create): Select the type of records that should be created. See Authoring Data in Applications for more information about master and golden data creation.

        • Golden Records creates new golden records that only exist in the MDM and are unrelated to source publishers.

        • Master Records (only for ID and fuzzy matched entities) creates master records on behalf of publishers. These records require that you provide the publisher, and they are matched and merged. They may be updated later on by records pushed by publishers with the same source ID.

      • Imported Record Type (available for Import): Select the type of records that should be handled by the import. See Authoring Data in Applications for more information about golden and master data import.

        • Golden Records imports new golden records or changes to existing golden records that only exist in the MDM. If importing golden data on a record consolidated from publishers, such an import is considered an overrides.

        • Master Records (only for ID and fuzzy matched entities) imports new or change existing master records on behalf of publishers. These records require that you provide the publisher, and they are matched and merged. They may be updated later on by records pushed by publishers with the same source ID.

      • Default Publisher (only for Import and Create): This option is available for ID of fuzzy matched entities, when importing or creating master records. This is the publisher on behalf of which the master records are imported or created by default.

      • Allow Other Publishers: (only for Import and Create): This option is available for ID of fuzzy matched entities, when importing or creating master records. It allows the user to select the publisher on behalf of which the records are created, or load the publisher from a column when importing.

      • Support Multiple Selection (available for Edit, Copy or Delete): Select this option to support this action when multiple records are selected.

      • Available For Golden Data (available for Mass-Update and Edit): Select this option to support this action when browsing golden data. See Authoring Data in Applications for more information about golden data authoring.

      • Available for Master Data (available for Mass-Update and Edit): Select this option to support this action when browsing master data. See Authoring Data in Applications for more information about master data authoring.

      • Available for Erroneous Data (available for Mass-Update and Edit): Select this option to support this action when browsing errors. See Authoring Data in Applications for more information about erroneous data authoring.

      • Restrict Import To Update (available for Import): Select this option to restrict import to updating existing records. No new records will be created.

    • If configuring a duplicate management action, in the Action Configuration tab, set the following properties:

      • Duplicate Manager (available for duplicate management actions): Duplicate Manager used to perform this action.

      • Enable Edit (available for Review and Confirm Duplicates, Merge or Split Duplicates and Review Duplicates Suggestions): Select this option to allow editing the values merged into the golden records while managing the duplicates.

      • Enable Confirm All (available for Confirm Duplicates): Select this option to allow the action with no records selected, which corresponds to confirming all the records.

      • Enable Review All (available for Review and Confirm Duplicates and Review Duplicates Suggestions): Select this option to allow the action with no records selected, which corresponds to reviewing and confirming all the records.

      • Mandate Confirm All (available for Review and Confirm Duplicates): Select this option to force the user to confirm all the duplicates he has to review.

      • Mandate Resolve All Suggestion (available for Review Duplicates Suggestions): Select this option to force the user to confirm all the suggestions he has to review.

      • Prompt Golden ID (available for Review Duplicates Suggestions): Select this option to prompt the user for the surviving Golden ID when he merges two golden records while processing duplicates suggestion.

      • On Finish Job (available for Confirm Duplicates): Select the job to run when the duplicates confirmation is submitted. Note that if you leave this property empty, Semarchy xDM automatically generates a job for you.

    • If configuring a delete action, in the Action Configuration tab, set the following properties:

      • Enable Delete All Records (available for Delete): Select this option to allow the action with no records selected, which corresponds to deleting all the records.

      • Delete Type (available for Delete): Define the type of delete applied with this action. See Entity Records Hard and Soft Delete for more information.

      • On Finish Job (available for Delete): Select the job to run when record deletion is submitted. Note that if you leave this property empty, Semarchy xDM automatically generates a job for you.

    • If configuring an explain record action, in the Action Configuration tab, set the following properties:

      • Display Card: Select the display card used to represent the records in the graph.

      • Form Tab: Select the form tab used to show the values of a record.

Once the action sets are created, you can use them in business views to start data management operations, possibly using steppers or workflows.

Business Views

In an application, business users can browse the MDM Hub content using user-friendly views called Business Views.

A business view is a set of related business entities. It starts from a Root Business Entity which, through Transitions, supports navigating to other Business Entities. These business entities can also through transitions lead to other business entities, etc.

For example, the Company Hierarchy business view uses the Cost Center and Employee entities, and uses the relations that link employee to managers, cost centers to parent cost centers, and employees to cost centers for the transitions.

Business views rely on the model structure, as Business Entities are based on entities and Transitions are based on reference relationships. Business views are functional views on subsets of the overall model.

Creating Business Views

To create a business view:

  1. In the model, expand the node of entity that should be at the root of your business view.

  2. Right-click the Business Views node under this entity and select Add Business View…. The Create New Business View wizard opens.

  3. In the Create New Business View wizard, enter the following values:

    • Name: Internal name of the business view.

    • Label: User-friendly label for this business view. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

  4. Click Next. The second step of the wizard configures the business entity at the root of the business view.

    • Browsing Form: Select the form that is used for showing single records for this entity.

    • Root Filter: This SemQL condition filters the records of the root entity that should appear when opening the business view. For example, in a business view representing the hierarchy of Cost Centers, we should filter only the cost centers with no parent cost center.

    • Root Label: Label used for the root business entity. It defaults to the label of the root entity.

    • Root Plural Label: Plural Label used for the root business entity. It defaults to the plural label of the root business entity.

  5. Click Next. The third step of the wizard allows you to select reference relationships to the root business entity for which you want to create a transition.

    • Select all the references for which you want to create a transition.

  6. Click Finish.
    The business view editor opens on the new business view, with the transitions and child business entities created for the selected references.

  7. Optionally configure the business view by setting the following properties:

    • Name and Definition

      • Description: optionally enter a description for the business view.

      • Required Role: Optionally select a role required to open this business view. This role is required in addition to the privilege grants on the entities composing the business view.

    • Icon and Label

      • Icon: Select an icon from the image library to represent the root business entity.

Creating Transitions

To include more entities in a business view (for example, child Cost Centers or Employees reporting to the cost centers), it is necessary to create more transitions to other business entities.

To create transitions:

  1. In the Business View editor, scroll down to the Transition table.
    This table shows the business entities and the transitions that relate them.

  2. Select the parent business entity for the new transition.

  3. Click the Add Transition Add Transition button in the toolbar of the transitions table. The Create New Transition wizard opens.

  4. In the Create New Transition wizard, enter the following values:

    • Transition Path: Click the image Edit Expression to select a transition path. This path is a SemQL path to child records related to the parent entity.

    • Name: Enter the name for the transition.

  5. Click Next. The second step of the wizard configures the child business entity of the transition:

    • Filter: The business entity shows records related by the transition path to the parent business entity, filtered in addition using this condition.
      For example, a business view has a Company root business entity and a Contact child business entity. When showing a given customer, the business view will show only the contacts related the given customer. The Filter condition applies in addition to that, for example to show only the contacts with a phone number.

    • Select whether you want to create a new business entity or use an existing one.

    • If you choose to Create a New Business Entity, select an Entity and give a Name for the new business entity.

    • If you choose to Use an Existing Business Entity, select an existing Target Business Entity. Note that this option is only possible if this business view already contains a business entity pointing to the same entity as the transition path.

  6. Click Finish

Using Existing Business Entity for Infinite Hierarchies
It is possible to create hierarchies with an infinite depth by creating a transition from a business entity to itself. For example, to create a hierarchy of cost centers, create a business view with a Cost center as the root business entity (call it CostCenterBusinessEntity) and a root filter selecting only the cost centers with no parents (e.g.: ParentCostCenter is null). Then add a transition using the self-relation linking parent and child cost centers, and select the existing CostCenterBusinessEntity.
Configuring Embedded Collection Transitions

If the form used for a business entities contains embedded collections, then you can configure an embedded collection transition to configure which records are displayed in this embedded collection and how browsing takes place when clicking these records in the embedded collection.

To configure an embedded collection transition:

  1. In the Business View editor, scroll down to the Transition table.

  2. Select the parent business entity for the new the transition.

  3. Click the Configure Embedded Collection Transition Configure Embedded Collection Transition button in the toolbar of the transitions table. The Create New Transition wizard opens.

  4. In the Create New Transition wizard, set the following values:

    • Embedded Collection: Select one of the embedded collections of the form of your business entity.

    • Name: Enter the name for the transition.

  5. Click Next. The second step of the wizard configures the child business entity of the transition:

    • Select whether you want to create a new business entity or use an existing one.

    • If you choose to Create a New Business Entity, select an Entity and give a Name for the new business entity.

    • If you choose to Use an Existing Business Entity, select an existing Target Business Entity.

  6. Click Finish

Configuring Business Views

In addition to the structure based on business entities and transitions, the business view also includes options to configure how it appears in the application.

You typically configure the following:

  • In the Transition, you configure how the transition appears.

  • In the Business Entity, you configure how child records appear when displayed in a collection or individually.

  • In a business entity, you configure the action available in the menu by selecting an Action Set.

  • If a business entity uses a form with references to other entities, the Reference Browsing options define how to navigate these references.

  • If a business entity uses a form with embedded collections, the Embedded Collection options define how to navigate when clicking an item in these collections.

  • In the Business View Transition and Business Entity, you also configure the tree view showing the business view hierarchy.

Configuring Transitions

To configure a transition:

  1. In the Business View editor, scroll down to the Transition table.

  2. Select the transition in the table.

  3. In the Properties view, configure the following properties:

    • In Name and Definition, modify the Name, Transition Path, Filter and Target Business Entity of the transition.

    • Icons and Labels

      • Icon: Icon used to represent the child records of the transition.

      • Use Custom Label: Select this option to use a Label and a Plural Label different from those of the child entity.

    • Display Properties

      • Form Tab Condition: SemQL condition that must be met for the transition tab to appear in the business view.
        For example, you may only want to add the Contacts tab if there is a contact, which can be expressed with Any Contacts have ( 1=1 )

      • Display Form Tab: Select how the tab should appear. With the label, icon, or both.

Configuring Business Entities

To configure a business entity:

  1. In the Business View editor, scroll down to the Transition table.

  2. Select the business entity in the table.

  3. In the Properties view, configure the following properties:

    • Name and Definition

      • Modify the Name and modify the selected Entity.

    • Display Properties

      • Customized Display Card: Display card used to represent one record in the editor header for this business entity.

      • Browsing Collection: Collection used to display a list of records.

      • Browsing Form: Form used to display one record.

      • Outline Visible: Defines whether the outline should be displayed for this business entity.

      • Display Record Count: Defines when the record count should appear, after the customized display card in the editor header, for this business entity.

    • Collection Configuration

      • Customized Sort: Select this option to sort the records in the browsing collection according to the Sort Expression.

      • Sort Expression: Sort expression applied to the data in the browsing collection.

      • User-Defined Sort: Check this box to allow users to customize the sort.

      • Allow Table, Allow List, Allow Grid: Select the views available for the collection. These must be supported by the selected Browsing Collection.

Configuring an Action Set

To configure the actions available for a business entity:

  1. In the Business View editor, scroll down to the Transition table.

  2. Select the business entity in the table.

  3. In the Properties view, select the Name and Definition finger tab.

  4. Select an Action Set that will define which actions are available to manage the data on that business entity.

Configuring References Browsing

Reference Browsing options define how to navigate references appearing in the form defined for a business entity.

  1. In the Business View editor, scroll down to the Transition table.

  2. Select the business entity in the table.

  3. In the Properties view, select the References Browsing finger tab.
    This table lists the references of the browsing form selected for the business entity.

  4. Click the Refresh References Refresh References button in the Properties view toolbar to refresh the list of browsable references.

  5. For each reference listed, select a Browsing Target:

    • Not Browsable makes the reference not navigable.

    • Form opens a popup with the selected Form when the reference is clicked.

    • Business View opens the selected Target Business View in the same editor.

Configuring Embedded Collections Browsing

Embedded Collection Browsing options define how to navigate when a user clicks an item from an embedded collection placed in the form defined for a business entity.

  1. In the Business View editor, scroll down to the Transition table.

  2. Select the business entity in the table.

  3. In the Properties view, select the Embedded Collections finger tab.
    This table lists the embedded collections of the browsing form selected for the business entity.

  4. Click the Refresh Configuration Refresh Configuration button in the Properties view toolbar to refresh the list of embedded collections.

  5. For each collection listed, select a Browsing Target:

    • Not Browsable makes embedded collection items not navigable.

    • Form opens a popup with the selected Form when the collection item is clicked.

    • Business View opens the selected Target Business View in the same editor.

A business view can be searched at every level. You configure the search capabilities for each business entity of your business view.

Semarchy xDM provides the following built-in search types to look for data in business views:

  • Text: The text search looks up all attributes of the business entity marked as Searchable in the model that match your search pattern. Users can use the % wildcard to match a number of characters or _ to match a single character.

  • By Form: This method shows a default form with all the business entity attributes available, and pickers to select values to filter these attributes.

  • Advanced Search: Advanced search allows users to specify which attributes to search on and the operators to use for comparison.

  • SemQL: SemQL search allows users to use SemQL queries to search. With SemQL, users can use attributes in parent entities or child entities, as well as the SemQL library of functions.

In addition to these search types, you can design your own Search Forms.

The business entity search configuration defines the search methods (built-in search methods and search forms) available to filter the records of a business entity in a business view.

To define the search configuration:

  1. In the Business View editor, scroll down to the Transitions table.

  2. In this table, select the business entity you want to enable search for.

  3. In the Properties view, select the Display Properties finger tab.
    The Search Configuration property lists the search methods enabled for this business entity.

  4. Click the Edit button to open the Available Search Methods selection dialog.
    This table lists the available search methods (built-in and search forms) available for the root business entity.

  5. Select the Available checkbox for each search method you want to make available.

  6. Order the search methods using the Move Up and Move Down buttons.

Configuring the Hierarchy in a Business View

A business view can be browsed using a hierarchical tree view into which users can expand and select transitions and child records as tree view nodes.

To configure the hierarchical tree view:

  1. In the Business View editor, scroll down to the Hierarchy Configuration section.

    • Select Enable Hierarchy View to enable the hierarchy. This adds a "Toggle Hierarchy" button to the business view toolbar. Users use this button to show/hide the hierarchy.

    • Select Open Hierarchy by Default to show the hierarchy to the user the first time he browses the business view.

    • Select in Display Root Node As how the root node (representing the collection of root business entity records) should appear, that is with an icon and/or a label. The icon and label used for the root node are the Icon and Root Plural Label defined in the Icon and Label section of the business view editor.

  2. Select each transition in the Transitions table of the Business View editor, and then in the Properties view, select the Hierarchy Configuration tab.

    • Select Enable in Hierarchy to enable this transition in the hierarchy. If the transition is disabled, then it does not appear in the hierarchy. A record for which no child transition is enabled cannot be expanded in the hierarchy. This property supports SemQL, which means that you can configure the transition’s presence in the hierarchy based on the data values. For example, use any contacts have (1=1) to disable the transition to the contacts when there are no child contacts for a customer.

    • Select Transition Node Visible to show the transition node between the parent record and the child records. If this property is false, then the child records appear directly under the parent record with no intermediate node representing the transition itself. This property supports SemQL, which means that you can configure the transition visibility based on the data values.

    • Select Expand by Default to have this transition expand automatically when its parent is expanded.

    • Select in Display Transition Node As how the transition node (representing the collection of business entity records) should appear, that is with an icon and/or a label. The icon and label used for the transition node are the Icon and Root Plural Label defined in the Icon and Label tab of the transition properties.

  3. Select each business entity in the Transitions table of the Business View editor, and then in the Properties view, select the Display Properties tab.

    • Select a Record Node Display Card for the business view, that is the display card used to display in the hierarchy each record of this business entity. Note that only the primary text and the avatar (if it is defined) are used in the display card.

Tips for Creating Views and Objects

You can accelerate the design of applications by Duplicating Objects such as the business views, forms and collections.
Creating Views for Specific Purposes

It is possible to create different business views/collections and forms for specific purposes, or use advanced capabilities (particularly the Visible flags) to hide/show parts of these graphical components contextually.

It is not necessary to have forms dedicated to data authoring, as the form components support both browsing and authoring patterns.

Selecting Attributes for Forms and Collections

The form/collections used in business views and steppers must be carefully designed keeping in mind the user experience, and should include relevant form fields/columns.

Key Attributes

The key for a record depends on the context (master record, golden record, source record, etc.) as well as the type of entity. To make form design simpler and consistent, you can simply include the primary key attribute for the entity in the form or collection. Depending on the context and entity type, the correct attribute will be displayed. For example, for a source record of a fuzzy matched entity, the SourceID will be displayed.

Unless you want to specifically display one of the key attributes (Golden ID, SourceID, etc.), you should not specifically add one of these in the form or collection.

Steppers Forms

For data authoring purposes, certain attributes are required in the form used in the steppers:

  • The entity primary key attribute is usually recommended. Depending on the entity type and context, the appropriate attribute is automatically displayed. See the note about ID Generation and Steppers below for more information.

  • It is also recommended to include the mandatory fields in the data authoring forms unless null values are automatically handled by enrichers and triggers. Otherwise, mandatory value validation may reject data.

  • If the entity references another entity, it is recommended to include the reference (the foreign display name attribute) to enable attaching a record to its parent. For example an employee is attached to a cost center and a manager. The corresponding FDN_CostCenter and FDN_Manager attributes should be added to the form used to author the employee, for example to assign the employee to a specific cost center or manager.

ID Generation and Steppers

The ID Generation used for the entity managed at a level in a stepper forces certain rules in the step and form design.

  • If the ID requires a user input (Manual or SemQL):

    • The first form step under the collection step must include the ID attribute or the attributes required to generate the ID using the SemQL expression. This forces the user to enter the ID information.

    • At run-time, a placeholder (temporary) ID is generated to allow enrichment and validation in the first form step before the full user input. It is recommended not to use or store the ID in the first step as you might be using this placeholder ID. The ID should be considered final only after the step into which the user has entered the ID information.

  • If the ID does not require any user input (Sequence or UUID):

    • The ID is generated automatically when the form opens. The ID is not needed in the form and steps, other than for information.

When a form is used for the purpose of data authoring, read-only fields appear as not editable. In the same context, certain attributes are automatically hidden:

  • Attributes from related entities. For example, a form attribute defined with AccountManager.FirstName for a Customer entity will not appear in data authoring on the customer form, as it refers to a related entity. When browsing, it will show a given customer’s account manager name.

  • Attributes that use a computed SemQL expression. For example, a form attribute defined with CustomerName will allow for inputting this attribute’s value. If you define the form attribute with UPPER(CustomerName), it will appear when browsing and display the customer name in uppercase, but will disappear when the form is used for data authoring.

Application Actions and Folders

An application’s Actions, organized in a hierarchy of Folders. Action types include:

  • Browse a Business View to see its data or errors, possibly opening it with a search form.

  • Global Search, to search the entire application.

  • Inbox, to open the user’s inbox and see his work to do or done.

  • Open URL, to open any URL

  • Profile, to open the user’s profile editor.

  • Import, to start a data import operation in a stepper or workflow.

  • Create, to start creating new records using a stepper or workflow.

  • Browse Entities, to automatically generate a folder to browse entities' data using various pre-built views (golden, master, errors, etc).

Creating Folders

To create a folder:

  1. Double-click the Folders and Actions node under the application. The Application opens on the Folders and Actions tree table.
    This tree table shows the hierarchy of folders and actions. All folders and actions are under a Root folder which cannot be deleted.

  2. Select a folder in the hierarchy and then click the Add Folder Add Folder button in the toolbar. The Create New Application Folder wizard opens.

  3. In the Create New Application Folder wizard, enter the following values:

    • Name: Internal name of the folder.

    • Label: User-friendly label for this folder.

    • Icon: Icon used to display this folder in the navigation drawer

    • Image: Image used to display this folder as a tile in its parent folder.

  4. Click Finish to close the wizard. The folder is added to the folders hierarchy.

Creating Actions

To create an action:

  1. Double-click the Folders and Actions node under the application. The Application opens on the Folders and Actions tree table.

  2. Select a folder in the hierarchy and then click the Add Action Add Action button in the toolbar. The Create New Application Action wizard opens.

  3. In the Create New Application Action wizard, enter the following values:

    • Name: Internal name of the action.

    • Action Type: Select the type of the action.

  4. Click Next

  5. If the action type requires it, enter its configuration parameters and the click Next

    • Browse Business View:

      • Business View: Select the business view to open:

        • Golden Data is the data certified data in the hub.

        • Golden Data With Errors is golden data flagged as erroneous.

        • Master Data is the latest records from source systems (for ID and Fuzzy Matched entities).

        • Source Data is data loaded from source system or authored by users.

        • Source Data With Errors is data loaded from source system or authored by users, and rejected by the validations.

      • Use Search on Open: Select this option to open the business view with its configured search methods for the user to filter the data before seeing it.

      • View Type: Select the type of view to browse.

    • Open URL:

      • URL: URL to open.

      • Target: Target of the URL. You can open the URL in a new tab, the same tab or in an editor.

    • Create or Import:

      • Create Action or Import Action: Select the corresponding action in an action set to trigger.

    • Browse Entities Folder:

      • Lineage Enabled: Selection this option to enable lineage navigation (golden to master records, for example) in the generated views.

  6. In the Display Properties step, configure how the action appears:

    • Label: User-friendly label for this action.

    • Description: Description for this action. This description appears as a secondary text for this action.

    • Icon: Icon used to display this action in the navigation drawer

    • Image: Image used to display this action as a tile in its parent folder.

  7. Click Finish to close the wizard. The action is added to the folders hierarchy.

Organizing Actions and Folders

Use drag and drop, or the Move Up and Move Down buttons to organize folders and actions in the Folders and Actions tree table. Note that the order set in this tree table is applied in the application at run-time if you have selected Position for the application’s Sort Method. If you have selected the Label sort method, then items in folders are automatically sorted alphabetically using their label.

In the Properties view, you can configure the display properties of the action in the application. For example, the color of the icon and image alignment.

The Navigation Drawer, displayed on the left of the application, enables the definition of shortcuts to application actions, organized into groups. In the navigation drawer:

  • Navigation Groups contain Navigation Items pointing to application actions.

  • Folder Groups are pointers to application folders. Such a group directly lists all the actions stored in the referenced folder.

To create a navigation group:

  1. Double-click the Navigation Drawer node under the application. The Application opens on the Groups tree table.
    This table shows the folder and navigation groups, and the items under the navigation groups.

  2. Click the Add Group Add Group button in the toolbar. The Add a Group wizard opens.

  3. In the Add A Group wizard, enter the following values:

    • Name: Internal name of the group.

    • Label: User-friendly label for this group, as it will appear in the navigation drawer.

    • Show Label: Select this option to show the label for the group.

    • Show Divider: Select this option to show a divider before the group.

  4. Click Finish to close the wizard. The group is added to the navigation drawer.

  5. Use the Move Up and Move Down buttons to order the elements in the navigation drawer.

To create a navigation item:

  1. Double-click the Navigation Drawer node under the application. The Application opens on the Groups tree table.

  2. Select a Navigation Group in the list.

  3. Click the Add Item Add Item button in the toolbar. The Add an Item wizard opens.

  4. Select an Action from the application actions, and then click Finish.
    The item is added to the navigation drawer.

  5. Select the item and use the Move Up and Move Down buttons to order this item within its group.

A navigation item uses the label and icon defined in the application action.

To create a folder group:

  1. Double-click the Navigation Drawer node under the application. The Application opens on the Groups tree table.

  2. Click the Add Folder Group Add Folder Group button in the toolbar. The Add a Folder Group wizard opens.

  3. In the Add a Folder Group wizard, enter the following values:

    • Linked Folder: Select a folder in the application this folder group will point to.

    • Name: Internal name of the group.

    • Show Label: Select this option to show the label of the folder in the navigation drawer.

    • Show Divider: Select this option to show a divider before the group.

  4. Click Finish to close the wizard. The group is added to the navigation drawer.

  5. Use the Move Up and Move Down buttons to order the elements in the navigation drawer.

A folder group uses the label and icon defined in the application folder.

Global Search Configuration

The Global Search Configuration defines which business views are searched when a search string is entered in the global search page, and how they are searched.

To configure a global search configuration

  1. Double-click the Global Search Configuration node under the application. The Application opens on its Global Search Configuration section.

  2. Configure the global search page:

    • Select Display "Search Form" Dropdown to allow the user to select which of the business views he wants to search. If you do not display this dropdown, all business views will be searched.

    • Select Sort Business Views Alphabetically to have the business views sorted in the drop down and in the search results alphabetically. Otherwise, their position in the Searched Business View applies.

  3. Click the Add Business View to Search Add Business View to Search button in the toolbar. The Create New Global Search Business View wizard opens.

  4. In this wizard, enter the following values:

    • Business View: Business view from the application to search for.

    • Search Type: Type of search used to search the business view. You can use the Full-Text search type or select a search form defined for the root business entity of the business view.

    • Search Parameter: If a search form is selected in the Search Type the search text is bound to this search parameter used in the search form query.

    • Max Results: Maximum number of results displayed in the search results for this business view.

  5. Use the Move Up and Move Down buttons to order the searched business views in the table.

  6. Repeat the three previous steps to create additional search configurations for other business views.

Validating an Application

An application or a component of the application (business views, forms) must be validated to ensure its correct behavior after deployment and raise possible issues.

To validate the application or one component from the diagram:

  1. In the Model Design view, select the node corresponding to your application right-click and then select Validate.

  2. If you have unsaved editors, select those to save when prompted.

  3. The validation process starts. At the end of the process, the list of issues (errors and warnings) is displayed in the Validation Log view. You can click an error or waning to open the object causing this issue.

Opening an Application

You can connect to the deployed applications from the Welcome page.

To open an application:

  1. Open a new tab in your web browser and connect to the URL that was provided by your administrator. For example http://<host>:<port>/semarchy/ where <host> and <port> represent the name and port of the host running the Semarchy xDM application. If you are not logged in yet, the Login Form is displayed.

  2. Enter your user name and password.

  3. Click Log In. The Semarchy xDM Welcome page opens.

  4. In the welcome page, click one of these applications to open it.

If an application cannot be connected, for example if its data location is in maintenance or requires an upgrade, it appears with a disabled avatar.
You can bookmark the URL when connected to a given application to return directly to this application.
When you connect to an application for the first time, its structure is generated from the model and kept in a cache. To refresh this cache with changes performed in the model or in the application, click the Refresh option in the profile menu, in the upper right corner of the application.

Applications Common Configuration

All applications running in a Semarchy xDM instance share configuration properties such as the header logo or the export limits. These properties are configured from the Administration Console. See the Managing the Platform chapter in the Semarchy xDM Administration Guide for more information.

Managing Image Libraries

Image libraries store images and icons used by applications. Libraries are stored in the repository, are global to the platform and can be used by all applications. Designers use, in applications, images and icons from the libraries by reference, using Image Library URLs.

Default Image Libraries

When the repository is created, built-in image libraries are seeded with default Semarchy xDM images and icons:

  • mdi is the library for all the Material Design icons and assets. This library cannot be modified.

  • default contains Semarchy branding images as well as the default images used by applications, such as the default images for the actions. Icons provided in addition to the mdi library are in this library. This library cannot be modified.

  • demo<application_name>_ contains the images/icons used in the <application_name> demonstration application. Such a library is seeded when a demo application is installed.

The default image libraries content changes with new versions of Semarchy xDM. When an image is removed or changed in the default libraries, it is indicated as deprecated in the validation report for models using it. Model designers should update their models to use newer versions of such images.

Browsing Image Libraries

The Image Libraries editor, available in the Administration Console view, allows administrators and model designers to view and manage the image libraries.

The Model Design platform privilege is required to manage image libraries.

To browse the image libraries:

  • Open the Administration Console perspective.

  • In the Administration view, double-click the Image Libraries node.
    The Image Libraries editor opens.

Use the filter box in the image libraries editor to reduce the number of images displayed. You can use the '%' sign to replace any number of character.

Managing Image Libraries

Image libraries are created and modified using file import. You can import two types of files:

  • Image files, for which you must select a target library.

  • Zip files, into which images are organized into folders. When a Zip file is imported, one library is created for each folder at the root of the Zip file, and images in this folder are imported into the library. Exporting images creates a file that follows the same format.

To import images:

  • Open the image libraries editor.

  • Click the Import Image Library button. The import dialog opens.

  • Drop either a Zip file or multiple image files into the dialog, or click Browse to select a Zip file or multiple image files.

  • If you import image files, select the Target Library for these images. Select Create a New Library to create a new library for these images.

  • Click OK to import the images.

To delete images:

  • Open the image libraries editor.

  • Click an image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.

  • Click the Delete Selected Images button, and then confirm the deletion.

If you delete all the images from a library, the library disappears.

To export images:

  • Open the image libraries editor.

  • Click the image to select it. Repeat this operation for all images you want to delete. You can also click the Select All option in the toolbar to select all the filtered images.

  • Click the Export Selected Images button.

A zip file is downloaded, containing all the selected images stored into folders named after their parent library.

Using this export mechanism, you can move an image library to a remote repository, or reorganize the library before reimporting it.
If an image is removed from a library while being used in a model, it will raise a warning in the model validation report. Designers should fix such stale image references and update their models to point to valid images.

Models Management

Semarchy xDM supports out of the box metadata versioning.
When working with a model in the Semarchy Workbench, the developer works on a Model Edition (version of the model).

Model management includes all the operations required to manage the versions of the models.

Introduction to Metadata Versioning

A Model Edition is at a given point of time either Open or Closed for editing.
Branching allows you to maintain two or more parallel Branches (lines) of model editions.

Model Editions and Branches

  • An Open Model Edition can be modified, and is considered as being designed. When a milestone is reached and the design is considered complete, the model can be closed. When a model edition is closed, a new model edition is created and opened. This new model edition contains the same content as the closed edition.

  • A Closed Model Edition can no longer be modified and cannot be reopened on the same branch. You can only edit this closed edition by reopening it on a different branch.

  • When a model from a previously closed edition needs to be reopened for editing (for example for maintenance purposes), a new branch based on this old edition can be created and a first edition on this branch is opened. This edition contains the same content as the closed edition it originates from.

Version Numbers

A model has a version number that is automatically generated with the Branch ID number and Model Edition number. The branch and model numbers start at zero and are automatically incremented as you create new branches or model editions.
A model edition appears in the following format: <Model Name> [<branch>.<edition>]. For example, the first model edition in the first branch has the version [0.0]. The fourth edition of the CustomerAndFinancialMDM model in the second branch is named CustomerAndFinancialMDM [1.3].

A Chronological Example

The following example shows the chronological evolution of a model through editions and branching:

  • January: a new CustomerHub model is created with the first branch and the first edition. This is CustomerHub [0.0].

  • March: The design of this model is complete. The CustomerHub [0.0] edition is closed and deployed. The new model edition automatically opened is CustomerHub [0.1].

  • April: Minor fixes are completed on CustomerHub [0.1]. To deploy these to production, the model edition CustomerHub [0.1] is closed, deployed to production and a new edition CustomerHub [0.2] is created and is left open untouched for maintenance.

  • May: A new project to extend the original hub starts. In order to develop the new project and maintain the hub deployed in production, a new branch with a first edition in this branch is created, based on CustomerHub [0.1] (closed). Two models are now opened: CustomerHub [0.2] which will be used for maintenance, and CustomerHub [1.0] into which new developments are added.

  • June: Maintenance fixes need to take place on the hub deployed in production.CustomerHub [0.2] is modified, closed and sent to production. A new edition is created: Customer [0.3].

  • August: Then new project completes. CustomerHub [1.0] is now ready for release and is closed before shipping. A new edition CustomerHub [1.1] is created and opened.

The following schema illustrates the timeline for the edition and branches. Note that the changes in the two branches are totally decoupled. Stars indicate changes made in the model editions:

Month    : January       March      April   May        June     August
Branch 0 : [0.0] -***-> [0.1] -*-> [0.2] ----------*-> [0.3] ----------->
Branch 1 :                    +-branching-> [1.0] -**-**-****-> [1.1] --->

At that stage, two editions on two different branches remain and are open: CustomerHub [1.1] and CustomerHub [0.3].

Model Development Lifecycle

This section describes the overall model development and deployment lifecycle.

Initial Setup and Deployment in Development

The following steps are required when creating a new model in a development environment.

  1. Create a New Model. This operation automatically creates the first edition of the model (Typically, with edition number [0.0]).

  2. Design the first iteration of your model by creating the logical model, certification process rules and the MDM applications.

  3. Run a Model Validation when your model is stable and ready for the first tests.

  4. Create a development data location, using your model edition, to deploy and test your model.

The model is deployed and ready for testing. You can load data into the data location and use the applications to view and manage the data.

Making Changes in Development

After the first deployment and tests, you will repeatedly make changes to the model edition (logical model, certification process rules or MDM applications) and test them in the development environment.

To test these changes in the development data location:

  1. Run a Model Validation to make sure that the model is valid.

  2. Deploy again the model again. This replaces the existing model by the updated one.

The updated model is immediately ready for testing. After the update, make sure to:

  • Run the data loads for the updated jobs to reprocess the incoming data as needed.

  • Click the Refresh option in the profile menu (in the upper right corner of the application) to force a regeneration of the MDM application.

Releasing from Development

When the model is complete and tested, it is ready for release.

To release a model:

  1. Close the model edition. This operation automatically freezes the current model edition and opens a new one.

  2. Deploy the closed model edition using one of the following methods:

Developing Iteratively after a Release

When you close a model edition, a new model edition (for example, with version number [0.1]) is automatically created.

You can proceed with your next project iteration, starting with this new model edition:

  1. Make changes to this model edition in development until the next project iteration is ready for release.

  2. When ready, release this model edition from development.

If you need to make fixes to a previously released model edition, you can branch this old model edition, modify, then release it.

Working with Model Editions and Branches

Model edition and branching can be managed from the Model Administration perspective.

To access the data administration perspective:

  1. Select the Overview perspective in the toolbar.

  2. In the Overview, click the Manage Model Editions… link in the Administration section. The Model Administration perspective opens.

Creating a New Model

Creating a new model creates the model with the first model branch and model edition.

To create a new model:

  1. In the menu, select File > New > New Model…. The Create New Model wizard opens.

  2. In the Create New Model wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

  3. In the Description field, optionally enter a description for the Model.

  4. Click Finish to close the wizard. The new model is created, with a root branch and a first edition.

  5. Model Edition perspective opens on the new model.

A new model is automatically configured to run on the same target database technology as the repository. You can select a different Target Technology in the Model editor.

Closing and Creating a New Model Edition

This operation closes the latest open model edition in a branch and opens a new one. The closed model edition is frozen and can no longer be edited, but can be deployed to production environments.

It is only possible to close an open edition. This edition is the latest one from a branch.
It is only possible to close an edition that is valid.

To close and create a new edition:

  1. In the Model Editions view of the Data Administration Perspective, expand the model and the model branch containing the edition you want to close.

  2. Right-click the latest model edition of the branch (indicated as opened) and select Close and Create New Edition.

  3. Click OK to confirm closing the model edition.

  4. The Enter a comment for this new model edition dialog, enter a comment for the new model edition. This comment should explain why this new edition was created.

  5. Click OK. The model is validated, then a new model edition is created and appears in the Model Editions view.

Be cautious when closing a model edition. Closing an edition cannot be undone, and a closed edition cannot be reopened.

Branching a Model Edition

Branching a model edition enables restarting and modifying a closed edition of a model. Branching creates a new branch based on a given edition, and opens a first edition of this branch.

It is only possible to branch from closed model editions.
When creating a model, a first branch named <model_name>_root is created with the first model edition.

To create a new branch:

  1. In the Model Editions view of the Data Administration Perspective, expand the model and the model branch containing the edition from which you want to branch.

  2. Right-click the closed edition from which you want to branch and select Create Model Branch From this Edition. The Create New Model Branch wizard opens.

  3. In the Create New Model Branch wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

  4. In the Description field, optionally enter a description for the Model Branch.

  5. Click Finish to close the wizard.

  6. In the Model Branch Created dialog, click Yes to open the first edition of this new branch.

  7. The newly created edition opens.

Target Technology

A model is designed for a given database Target Technology (Oracle, PostgreSQL). Although most of the model content is technology-agnostic, the artifacts generated in the data location schema, as well as the certification process, will use capabilities specific to that technology.

When you create a model, it is automatically configured for the technology of the repository into which it is created. When you design the model, some of the capabilities, for example, the database functions available in SemQL, will depend on that technology.

You can configure the Target Technology value in the Model editor.

Make sure to check the model’s target technology when you start working with a model, or when you import a model from another repository. If you change this technology later, make sure to validate the model in order to have a list of possible issues due to the technology change.

Model Localization

When designing a model, labels, descriptions and other user-facing text strings, are entered to provide a user-friendly experience. These strings are natively externalized in Semarchy xDM, and can be translated (localized) in any language.

A user connecting an application created with Semarchy xDM will see these strings (label of the entities, attributes, list of values, etc.) translated in the locale of his web browser if such translation is available. If no translation is available in his locale for a given text, the default string (for example, the label or description specified in the model) is used. These default strings are the base translation.

Make sure you translate the entire model in a given language to avoid partially translated user interfaces.

Translation Bundles

Strings translation is performed using Translation Bundles, attached to model editions. A translation bundle is a properties file that contains a list of key/value pairs corresponding to the strings localized in a given locale. The translation bundle file is named translations_<locale>.properties, where <locale> is the locale of the translation.

The following example is a sample of a translation bundle file for the English language (translations_en.properties). In this file, the label for the Employee entity is the string Staff Member, and its description is A person who works for our company.

...
Entity.Employee.Label=Staff Member
Entity.Employee.Description=A person who works for our company.
Attribute.Employee.FirstName.Label=First Name
Attribute.Employee.FirstName.Description=First name of the employee
Attribute.Employee.Picture.Label=<New Key TODO>
...

Translating a Model

To translate a model:

  1. The translation bundles are exported for the language(s) requiring translation in a single zip file.

  2. Each translation bundle is translated by a translator using his translation tooling.

  3. The translated bundles are re-imported into the model edition (either each file at a time, or as a single zip file).

To export translation bundles:

  1. In the Model Editions view (Model Administration perspective), expand the model edition you want to localize.

  2. Right-click and then select Export Translation Bundles…. The Export Translation Bundles wizard opens.

  3. Select the languages you want to translate.

  4. Select Export Base Bundle if you also want to export the base bundle for reference. The base bundle contains the default strings, and cannot be translated.

  5. Select the Encoding for the exported bundles. Note that the encoding should be UTF-8 unless the language you want to translate or the translation tooling has other encoding requirements.

  6. Select in Values to Export the export type:

    • All exports all the keys with their current translated value. If a key is not translated yet, the value exported is the one specified by the Default Values option.

    • New keys only exports only the keys not translated yet.

    • All except removed keys exports all keys available, excluding those with no corresponding objects in the model. For example, the key for the description of an attribute that was deleted from a previous model edition will not be exported.

  7. Select in Default Values the value to set for new keys (keys with no translation in the language).

    • Use values for base bundle sets the value to the base bundle value.

    • Use the defined tag sets the value to the tag specified in the field at the right of the selection (defaults to <New Key TODO>).

    • Leave Empty set the value to an empty string.

  8. Click OK to download the translation bundles in a zip format and then Close to close the wizard.

To import translation bundles:

  1. In the Model Editions view (Model Administration perspective), expand the model edition you want to localize.

  2. Right-click and then select Import Translation Bundles…. The Import Translation Bundles wizard opens.

  3. Click the Open button and select the translation file to import. This file should be either a properties file named translations_<locale>.properties or a zip file containing several of these properties files.

  4. In the Language to Import table, select the language translations you want to import.

  5. Select the Encoding for the import. Note that this encoding should correspond to the encoding of the properties files you import.

  6. Select Cleanup Removed Keys During Import if you want to remove the translations for the keys that are no longer used in the model. This cleanup removes translations no longer used by the model.

  7. Click Finish to run the import.

The translations for the model edition in the selected languages are updated with those contained in the translation bundles. If the Cleanup Removed Keys During Import was selected, translations in these languages no longer used in the model are removed.

Translation and Model Edition Lifecycles

The lifecycle of the translations is entirely decoupled from the model edition and deployment lifecycle:

  • It is possible to modify the translations of open or closed model editions, including deployed model editions in production data locations.

  • Translation changes on deployed model editions are taken into account dynamically when a user accesses an application defined in this model edition.

Decoupling the translation lifecycle from the model edition avoids binding the critical model development and release process to the translation process, as the latter frequently is managed by a separate team. This also allows adding new translations or fixing translations without having to re-deploy a new model edition.
When creating a new model edition, the translations from the previous model edition are not copied to the next edition. It is necessary to export and import translations between editions.

Deployment

This process is the deployment in a run-time environment (for production or development) of a model designed with its certification process.

Introduction to Deployment

Deployment consists in deploying a Model Edition in a Data Location (a database schema accessed via a JDBC datasource).

Once this model edition is deployed, it is possible to load, access and manage data in the data location using the applications defined in the model.

In this process, the following components are involved:

  • A Data Location is a database schema into which successive Model Editions will be deployed. This data location is declared in Semarchy xDM, and uses a JDBC datasource defined in the application server.

  • In a data location, a Deployed Model Edition is a model version deployed at a given time in a data location. As an MDM Hub model evolves over time, for example to include new entities or functional areas, new model editions are created then deployed. The Deployment History tracks the successive model editions deployed in the data location.

In the deployment process, you can maintain as many data locations as you want in Semarchy xDM, but a data location is always attached to one repository. You can deploy successive model editions into a data location, but only the latest model edition deployed is active in the data location.

Data Location Types

There are two types of data locations. The type is selected when the data location is created and cannot be changed afterwards:

The data location types are:

  • Development Data Location: A data location of this type supports deploying open or closed model editions. This type of data location is suitable for testing models in development and quality assurance environments.

  • Production Data Location: A data location of this type supports deploying only closed model editions. This type of data location is suitable for deploying MDM hubs in production environments.

Be cautious when choosing the data location type, as it will determine the type of deployment operations that can be done. It is recommended to use only Production Data locations for Production and User Acceptance Test environments.

Data Location Contents

A Data Location contains the hub data, stored in the schema accessed using the data location’s datasource. This schema contains database tables and other objects generated from the model edition.

The data location also refers three type of jobs (stored in the repository):

  • Installation Jobs: The jobs for creating or modifying, in a non-destructive way, the data structures in the schema.

  • Integration Jobs: The jobs for certifying data in these data structures, according to the model job definitions.

  • Purge Jobs: The jobs for purging the logs and data history according to the retention policies.

Creating a Data Location

A data location is a connection to a database schema via a JDBC datasource defined in the application server running Semarchy xDM. Make sure the administrator of this application server creates this datasource, and the database administrator creates the schema for you before creating the data location in Semarchy Workbench.

To create a new data location:

  1. In the menu, select File > New > New Data Location. The Create New Data Location wizard opens.

  2. In the Create New Data Location wizard, check the Auto Fill option and then enter the following values:

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • JNDI Datasource Name: Select the JDBC datasource pointing to the schema that will host the data location.

    • In the Description field, optionally enter a description for the Data Location.

    • Select the Location Type for this data location.

    • Select the Deployed Model Edition: This model edition is the first one deployed in the data location.

  3. Click Finish to close the wizard. The data location is created and the first model edition deploys. The Data Location opens on the new data location.

To delete a data location:

  1. In the Data Locations view, right-click the data location node and select Delete. The Delete Data Location wizard opens. In this wizard, you only delete the data location definition in Semarchy xDM. Deleting the data stored in the data location is optional.

    • If you do not want to delete all the data in the data location schema, click Finish. The data location is deleted but the data is preserved.

    • If you want to delete all the data in the data location schema:

      1. Select the Drop the content of the database schema to delete the content of the schema. Note that with this option, you choose to delete all the data stored in the hub, which cannot be undone.

      2. Click Next. The wizard lists the objects that will be dropped from the schema.

      3. In the next wizard step, enter DROP to confirm the choice, and then click Finish. The data location as well as the schema content are deleted.

Deleting a data location is an operation that cannot be undone.
Deleting the data location as well as the schema content is a convenient mechanism to reset the hub content at the early stages of the model design.

Deploying a Model Edition

After the initial model edition is deployed, it is possible to deploy other model editions. This single mechanism is used for example to:

  • Update the deployed model edition with the latest changes performed in an open model edition.

  • Deploy a new closed model version to a production or test environment.

  • Revert a deployed model edition to a previous edition.

Deploying open model editions is only possible in a Development Data Location.

To deploy a model edition:

  1. In the Data Locations view, right-click the data location node and select Deploy Model Edition….

  2. If you have unsaved editors, select those to save when prompted.

  3. In the The Deploy Model Edition wizard*, select in the Deployed Model Edition the model edition you want to deploy.

  4. Leave the Generate Job Definition option checked to generate new integration jobs.

  5. Click Next. The changes to perform on the data location, to support this new model edition, are computed. A second page shows the SQL script to run on the schema to deploy this model edition.

  6. Click Finish to run the script and close the wizard.
    The model edition deploys the jobs first and then runs the SQL code to create or modify the database objects. You can follow this second operation in the Console view at the bottom of the Workbench.

The new model edition is deployed and the previous model deployment appears under the Deployment History node in the data location.

Deploying a model edition does not modify data already in place in the data location.
Although it is recommended to update both the jobs and schemas at the same time, you may want to update the data structure first then the jobs later. For example, if the data you have in the data editions using this model edition is not fit for the new version of the jobs. In that case, you may want to run some transformation on the data with the updated data structures before updating the jobs.
Another use case for not deploying the job definition is when you know that the new and old job definitions are similar and you want to preserve the existing job logs.

After multiple deployments, you may decide to remove old elements in the Deployment History.

To remove historized deployments:

  1. In the Data Locations view, expand the Deployment History node under the data location.

  2. Select one or many historized deployments (hold Shift for multiple selection).

  3. Right-click and select Delete.

The selected historized deployments are deleted.

Advanced Deployment Techniques

Moving Models at Design-Time

At design-time, it is possible to move models from one repository to another design repository using Export/Import:

  • Export is allowed from a design repository, but also from a deployment repository.

  • Import is possible in a design repository, either:

    • to create a new model from the import

    • or to overwrite an existing open model edition.

Exporting a Model Edition

To export a model edition:

  1. Open the Model Editions perspective.

  2. Expand the model branch containing the model edition you want to export.

  3. Select the model edition you want to export, right click and select Export Model Edition.

  4. In the Model Edition Export dialog, select an Encoding for the export file.

  5. Click OK to download the export file on your local file system.

  6. Click Close.

Importing to a New Model

To import and create a new model:

  1. In the menu, select File > New > New Model from import... The Import to a New Model wizard opens.

  2. Click the Open button and select the export file.

  3. Click Finish to perform the import.

Click the Refresh button in the Model Editions view. The new model appears in the list.

When importing a model from a different repository, make sure that the correct Target Technology is set for this model. For example, a model developed for the Oracle target technology will not work if you attempt to deploy it on a PostgreSQL data location.
Importing on an Existing Model

To import and replace an existing model:

  1. Open the Model Editions perspective.

  2. Expand the model branch containing the model edition you want to replace.

  3. Select the open model edition you want to replace, right click and select Import Replace…. The Import-Replace Model Edition wizard opens.

  4. Click the Open button and select the export file.

  5. Click Finish to perform the import.

  6. Click OK to confirm the deletion of the existing model.

The existing model edition is replaced by the content of the export file.

Deploying to a Remote Location

Frequently, the deployment environment is separated from the development environment. For example, the development/QA and production sites are located on different networks or locations. In such cases, it is necessary to use export and import to transfer the model edition before performing the deployment in production.

A remote deployment consists in moving a closed model edition:

  • From a design repository or a deployment repository used for Testing/UAT purposes;

  • To a deployment repository.

Remote Deployment Architecture

In this configuration, two repositories are created instead of one:

  • A Design repository for the development and QA site, with Development data locations attached to this repository.

  • A Deployment repository for the production site. Production data locations are attached to this repository.

The process for deploying a model edition in this configuration is given below:

  1. The model edition is closed in the design repository.

  2. The closed model edition is exported from the design repository to an export file.

  3. The closed model edition is imported from the export file into the deployment repository.

  4. The closed model edition is deployed from the deployment repository into a production data location.

Exporting a Model Edition

To export a model edition:

  1. Open the Model Editions perspective.

  2. Expand the model branch containing the model edition you want to export.

  3. Select the closed model edition you want to export, right click and select Export Model Edition.

  4. In the Model Edition Export dialog, select an Encoding for the export file.

  5. Click OK to download the export file on your local file system.

  6. Click Close.

Importing a Model Edition in a Deployment Repository

To import a model edition in a deployment repository:

  1. Open the Model Editions perspective.

  2. Select File > New > Import Model Edition. The Import Model Editions wizard opens.

  3. Click the Open button and select the export file.

  4. Click Next.

  5. Review the content of the Import Preview page and then click Finish.

When importing a model edition, the root model and the branches containing this model edition are automatically created as needed.
When importing successive versions of model editions in a deployment repository, it is not recommended to skip intermediate versions, as it is not possible to import these intermediate versions later. For example, if importing version 0.1 of a model, then importing version 0.4, the intermediate versions - 0.2, 0.3 - can no longer be imported into this repository.
Elements not Exported with the Model Edition

When exporting then importing a model to a different repository, note that some elements are not included in the model and need to be reconfigured in the remote environment. These elements are listed below.

Elements used by the model and applications:

  • Applications Common Configuration

  • Image Libraries

  • Installed Plug-ins

  • Role definitions

  • Variable Value Providers

Elements not used by the model, but required for operations:

  • Notification Servers

  • Notification Policies

  • Continuous Loads

  • Purge Schedules

  • Integration Batch Poller Configuration

Data Location Status

A data location status is:

  • Ready: A data location in this status can be accessed in read/write mode, accepts incoming batches and processes its current batches.

  • Maintenance: A data location in this status cannot be accessed. It does not accept incoming batches but completes its current batches. New loads cannot be initialized and existing loads cannot be submitted.

When moving a data location to a Maintenance status, the currently processed batches continue until completion. Loads submitted after the data location is moved to Maintenance will fail. They can be kept open and submitted later, when the data location is restored to a ready status.

When a data location is in maintenance mode, it appears with a disabled avatar in the welcome page.
Changing a Data Location Status

To set a data location status to maintenance:

  1. In the Data Locations view, expand the data location node.

  2. Right-click the data location and select Set Status to Maintenance.

  3. Click OK in the confirm dialog.

The data location is moved to Maintenance mode.

To set a data location status to ready:

  1. In the Data Locations view, expand the data location node.

  2. Right-click the data location and select Set Status to Ready.

The data location is moved to a ready state.

Using the Maintenance Mode

The Maintenance status can be used to perform maintenance tasks on the data locations.
For example, if you want to move the data location to a model edition with data structure changes that mandate manual DML commands to be issued on the hub tables, you may perform the following sequence:

  1. Move the data location to Maintenance mode.

  2. Let the currently running batches complete. No batch can be submitted to this edition.

  3. Deploy the new model edition.

  4. Perform your DML commands.

  5. Move the data location from Maintenance to the Ready status. Batches can now be submitted to this data location.

Using this sequence, you prevent batches being submitted while the hub is in Maintenance.

The data location is automatically set to maintenance when deploying a model edition and then automatically reverted to ready state.

Securing Data

Securing Data Access

This section describes how to define the access policies for the data model. They are implemented in the form of model and entity privileges.
They are enforced when the user accesses the data through the graphical user interface and integration points.

Introduction to Security in Semarchy xDM

There are two levels of security in Semarchy xDM:

  • Platform-Level Security defines access to features in the platform. This type of security is covered in the Semarchy xDM Administration Guide.

  • Model Level Security defines security privileges to access and modify data in the data location into which the model is deployed.

Both levels of security are role-based:

  • Privileges are granted to Roles defined in Semarchy xDM. These roles map to roles defined at the application server level.

  • Roles are given to Users defined in the application server.

  • Semarchy xDM does not store users and roles associations. These are provided by the application server which delegates users and roles management to a security provider (SSO, LDAP, etc.)

For more information about role creation and management, refer to the Semarchy xDM Administration Guide.

Model Privileges Grants

Model Privileges Grants define the access privileges for users via their roles.
A Model Privilege Grant is associated to a role and contains a set of Entity Privileges granted to this role. You can only define a single Privilege Grant for each role in a model.

Entity Privileges

An Entity Privilege defines the privilege of a role for an entity (or a subset of the entity records) and its attributes.

Entity-Level and Attribute-Level Privileges

An Entity Privileges includes privileges defined at:

  • Entity-level: Such a privilege is the default privilege for all the attributes of the entity.

  • Attribute-level: Such a privilege overrides the default entity-level privilege, allowing specific privileges per attribute.

Examples:

  • Entity-Level Privileges: Users with the Finance role can edit (read/write) cost centers; other users can only see them (read). Users with the Contractor role cannot see (none) this entity.

  • Attribute-Level Privileges: All users can see (read) general information from the employee entity, but they cannot see (none) sensitive information (Personal Email, SSID, Salary, etc. attributes). Only users with the HR role can see these attributes.

Privileges Types

The types of privilege are listed in the table below:

Privilege Type Entity-Level Attribute-Level Description

None

Yes

Yes

No privilege is granted to the user for the entity or attribute.

Read

Yes

Yes

Allows the user to view the entity or attribute.

Read/Write

Yes

Yes

Allows the user to view the entity or attribute. In addition, the user can edit the values in data authoring. Note that this privilege only allows modifying data. To create new records in a stepper, the Creation privilege is needed.

Allow Export

Yes

N/A

Allows the user to mass-export records from this entity in Excel format. Note that exported columns are filtered using the privileges granted on attributes. If a user can Export an entity, but does not have read or read/write privileges on certain attributes, these attributes will not appear in the export.

Allow Creation

Yes

N/A

Allows the user to create new records for this entity in steppers.

Allow Checkout

Yes

N/A

Allows the user to edit records for this entity in steppers.

Allow Removal

Yes

N/A

Allows the user to remove records for this entity in steppers.

Allow Delete

Yes

N/A

Allows the user to remove records for this entity. This entity must also have Delete Enabled selected.

If a user has read or read/write privileges on one attribute of an entity, then he also has read privileges to see the built-in attributes managed by the platform (for example: publisher, update date, etc.).
Privilege Precedence

Privileges apply in order of precedence: Read/Write then Read then None. As a consequence, a user always has the best privileges associated to his roles

For example: John has several roles given to him:

  • Finance has Read privileges on the Customer entity.

  • HR has None privileges on the Customer entity.

  • Sales has Read/Write and Creation privileges on the Customer entity.
    The resulting privilege for John is Read/Write and Creation on the Customer entity.

Make sure you take into account all the roles of a user when reviewing his privileges. Any given role may grant this user additional privileges.
Row-Level Filtering

Entity privileges support Row-Level Filtering, to apply privileges only to a subset of the records of the entity. The subsets are defined using SemQL filters.

To define different privileges for different subsets of records in the same entity, it is possible to create several entity privileges for the same entity. Each privilege will address a subset of the records, defined by a Filter.

Row-level security filters can use Model Variables to customize the data access privileges for the connected user’s session.

For example, the Privilege Grant for the GenericUser role contains:

  • A first Entity Privilege allowing Read access to the employee entity.

  • A second privilege grant for the same employee entity, allowing Read/Write for certain attributes. This second privilege grant contains the following SemQL filter: employeeName=:V_USERNAME.
    Both these grants apply to the same employee entity and the same GenericUser role. The second one only applies to the filtered subset of records in this entity passing this filter. This subset corresponds to the connected user’s employee record.

Row-Level Filtering is not supported for duplicate management. A user who performs duplicate management operations for an entity must be granted privileges on all the records of the entity. Read privilege is required on all records of a duplicate manager, and write privilege is required to manipulate these duplicates.
Built-in Roles

The following roles are built in within the platform:

  • semarchyConnect: This role must be granted for a user to log in. It should be granted by default to all users connecting to Semarchy xDM. This role is hard-coded and cannot be modified or used in the model.

  • semarchyAdmin: This role has full access to all the features of the platform. This role is hard-coded and cannot be modified. It can be used in the model, for example in privilege grants.

When creating a new model, a model-level privilege grant is automatically created for the semarchyAdmin role with Grant full access to the model selected. By modifying this privilege grant, the model designer can reduce the privileges of the semarchyAdmin role on the data, and prevent platform administrators from accessing data in the hub.
How are Privileges Applied?

The Semarchy Workbench Data Management user interface layout and actions are automatically modified depending on the user privileges:

  • Entities/Attributes/Records with no read privilege do not appear.

  • Entities/Attributes/Records with no read/write privilege cannot be modified in authoring.

  • Records cannot be added in steppers without the creation privileges.

  • Records cannot be removed from steppers without the removal privileges.

  • Exporting Data (Excel) is not possible without the export privilege.

Setting up Model Privileges

In this section, we assume that roles and users are already defined at the application server level, and that Roles are already defined in Semarchy xDM by the administrator. For more information about role creation and management, refer to the Semarchy xDM Administration Guide.

To add a model privilege grant:

  1. Connect to an open model edition.

  2. In the Model Edition view, right-click the Model Privilege Grants node and select Add Model Privilege Grant…. The Create New Model Privilege Grant wizard opens.

  3. In the Create New Model Privilege Grant wizard, check the Auto Fill option and then enter the following values:

    • Role Name: Select a role defined in Semarchy xDM by the administrator.

    • Name: Internal name of the object.

    • Label: User-friendly label for this object. Note that as the Auto Fill box is checked, the Label is automatically filled in. Modifying this label is optional.

    • In the Description field, optionally enter a description for the Model Privilege Grant.

    • Check the Grant full access to the model option if you want to give full privileges to this model. Checking this box overrides all privileges granted at the entity or attribute level.

    • Check the Grant access to integration web services option to allow this role to connect to the REST API interface for this model. Note that entity/attribute level privileges are also needed. This option only allows connecting to the web service interface.

  4. Click Next.

  5. In the Entities Privileges Grants page, select the Entities for which you want grant privileges and click the Add >> button to add them to the Selected Entities.

  6. Click Next.

  7. In the next page, select the default privileges for the selected entities:

    • Default Access Type: Select None, Read or Read/Write

    • Allow Creation: Select this option to allow this role to create new entity records in a stepper.

    • Allow Checkout: Select this option to allow this role to edit entity records into a stepper.

    • Allow Export: Select this option to allow this role to export entity records.

    • Allow Removal: Select this option to allow this role to remove entity records from a stepper.

    • Allow Delete: Select this option to allow this role to delete entity records.

  8. Click Finish to close the wizard. The Model Privilege Grant editor opens.

  9. Press CTRL+S to save the editor.

In the Model Privilege Grant editor, you can refine the default privileges in the Entity Privileges table:

  • You can modify the Access Type for each entity.

  • You can expand the entities and modify the Access Type for specific attributes.

  • You can change the Creation, Checkout, Removal, Delete and Export privileges on entities.

  • You can add privilege grants for more entities, or new privilege grants for entities already present in the grant.

  • You can set a filter for each privilege grant to enable row-level security.

Be cautious when checking the Grant full access to the model option in a privilege grant, as it overrides all privileges granted at the entity or attribute level.

Reviewing the Privileges

It is recommended to review the privileges of the users before releasing a model.

To test the security for a given user:

  1. Login to the Semarchy xDM Workbench using this user’s credentials.

  2. Connect to the Data Location using an application.

  3. In the workbench toolbar, select the user name in the upper right corner and then select User Information.

The platform, entity-level and attribute-level privileges for the connected user, computed from his various roles, are listed in the user information window.

Defining Data Retention

This section describes how to define the retention policies for historical and lineage data.

Introduction to Data Retention

The MDM hub stores the lineage and history of the certified golden data, that is the data that led to the current state of the golden data:

  • The built-in Lineage traces the whole certification process from the golden data back to the sources. It traces source data changes pushed in the hub - through external loads - or performed into the hub - for example using steppers.

  • Historization traces the changes made to the golden and master data.

Preserving the lineage and history is a master data governance requirement. It is key in a regulatory compliance focus. However, keeping this information may also create a large volume of data in the hub storage.

To keep a reasonable volume of information, administrators will schedule purges for this data. To make sure lineage and history are preserved according to the data governance and compliance requirements, model designers will want to define Data Retention Policy for the model.

Data Retention Policies

Data Retention Policies are defined with:

  • A Model Retention Policy, defining the retention duration for history and lineage data in the hub. This policy applies by default to all entities with no specific policy.

  • Entity Retention Policies that override the model retention policies for specific entities.

For example:

  • The hub is configured to retain no history at all. This is the general policy.

  • Employee data is retained for 10 years.

  • Product data is retained forever.

Data Purge

Depending on the retention policy defined for the model, data purge takes place in the deployed hub.

The purge deletes the following elements of the lineage and history:

  • Source data published to the hub via external loads

  • Data authored (created, modified or overridden) in the hub

  • Traces of deleted data

  • Golden and master data history (if historization is configured)

  • Errors detected on the source and authored data by the integration job

  • Duplicate choices made by users in duplicate managers.
    Note that the duplicate management decisions still apply after a purge, but information about the time of the decision and the decision maker is deleted.

The purges only impact the history and lineage of the data the data location. They do not delete actual golden and master data.

Optionally, the following repository artifacts can also be deleted as part of the purge process:

  • Job logs, batches and loads for which all the processed data has been purged.

  • Direct authoring, duplicate manager and workflow instances for which all the changed data has been purged.

Job logs, batches, loads, direct authoring, duplicate manager and workflow instances are purged when all their data have been purged. Therefore they are purged based on the longest retention policy of all the entities that they manage.
Deploying a Purge

When a model is deployed, a purge job is automatically created in the deployment data location. This job purges data and artifacts according to the retention policy defined in the deployed model.

Purge jobs are scheduled by the hub administrator as part of the data location configuration. For more information, refer to the Semarchy xDM Administration Guide.

Regardless of the frequency of the purges scheduled by the administrator, the data history is retained is as defined by the model designer in the data retention policies.

Defining the Data Retention Policies

The model retention policy applies to all the entities with no specific retention policy.

To define the model data retention policy:

  1. Connect to an open model edition.

  2. In the Model Edition view, double-click the Retention Policies node. The Data Retention Policy editor opens.

  3. In the Data Retention Policy editor, set the Default Retention Policy for the model:

    • Retention Period: Select the default retention type: Forever, No Retention or Period.

    • If you have selected Period, set a Time Unit and Time Duration for the retention period.

    • In the Description field, optionally enter a description for the retention policy.

  4. Press CTRL+S to save the editor.

In addition to the model retention policy, you can also define in this editor the entity-specific retention policies.

To define an entity retention policy:

  1. In the Data Retention Policy editor, click the Add Entity Retention Policy. The Create New Entity Retention Policy wizard opens.

  2. In the Create New Entity Retention Policy wizard:

    • Select in the Entity field the entity for which you want to define a specific policy.

    • Retention Period: Select the retention type: Forever, No Retention or Period.

    • If you have selected Period, set a Time Unit and Time Duration for the retention period.

    • In the Description field, optionally enter a description for the entity retention policy.

  3. Click Finish to close the wizard.

  4. Press CTRL+S to save the editor.

You can only have one entity retention policy per entity of the model.
The retention policy has no effect unless the model is (re)deployed and the administrator of the hub schedules the purge job for the data location. For more information, refer to the Semarchy xDM Administration Guide.

Appendices

Appendix A: Component Types and Properties

A component type is a UI component used to display or author a form field or table column.

Component Types Reference

The following table lists the component types available, their supported datatypes and behavior in the browsing and authoring experience.

Component

Supported Datatypes

Browsing Behavior

Authoring Behavior

Chips

List of Values (Multiple) [1]

Displays the values as a list of chips

Allows selecting multiple values using an auto-complete picker.

Checkbox

Boolean [1]

Displays a disabled checkbox or switch

Checkbox or switch for authoring.

Date Picker

Date [1], Timestamp [1]

Displays the value formatted per the user preferences.

Supports specific date and time picker component.

Hyperlink

String, LongText, Binary [1]

Displays a hyperlink to the URL or binary content, opened in a given target.

Allows authoring the URL or uploading binary content.

ID

ID [1]

Displays the record ID in text format.

Accepts formatted or unformatted ID values.

Image

String, LongText, Binary

Embeds binary content or URL as an image.

Supports image content upload or URL authoring.

Markdown

String, LongText

Renders a text containing Markdown content.

Support authoring markdown text content with a rendering preview.

Menu

String, List of Values [1]

Displays the value (LOV Label) as unformatted text

Allows selecting a value from a drop down menu.

Object

String, LongText, Binary

Embeds binary content or URL in an iFrame.

Supports binary content upload or URL authoring.

Reference

Reference (FDN) [1]

Displays the referenced record as a hyperlink text or chip

Allows selecting the reference using an auto-complete picker.

Text

UUID [1], String [1], LongText [1], Boolean, Decimal [1], ByteInteger [1], Integer [1], LongInt [1], Date, TimeStamp

Displays the value formatted per the user preferences.

Supports multi-line text and editing format for numbers and dates.

[1] The component type is used by default for this data type.

Common Properties Reference

The following properties are available for fields and/or columns regardless of the component type selected.

Component Tab Property Value Default Value Description

<common>

General

Read-Only [1]

Boolean

false

Defines when the current component should be set to read-only in an authoring form.

<common>

Label

Helper Text [1]

String

Helper text displayed under the editing component in an authoring form.

<common>

Label

Icon [2]

Image URL

Icon used instead of the label in a form.

<common>

Label

Label Position [2]

left, right, top, bottom

left

Position of the label in a browsing form.

<common>

Label

Label Visible [3]

Boolean

true

Defines whether the label is visible.

<common>

Label

Use Floating Label [1]

Boolean

false

Whether to use the floating label in an authoring form, if the component supports it.

<common>

Style

Background Color

Color

For tables, background color of the table cell. For forms, background color of the component.

<common>

Style

Column Alignment [3]

left, center, right

Depends

For tables only: Column header and content alignment. Default alignment depends on the column content.

<common>

Style

Header Color [3]

Color

For tables only: Column header text color.

<common>

Style

Icon Color [2]

Color

Icon color when an icon is used.

<common>

Style

Label Color [2]

Color

Text color for the label in a form.

<common>

Style

Label Text Alignment [2]

left, center, right

right

Label text alignment in a browsing form.

<common>

Style

Label Typography [2]

display-4, display-3, display-2, display-1, headline, title, subhead, body-2, body-1, caption, button

caption

Typography of the label a form.

<common>

Style

Text Alignment [2]

left, center, right

left

Component’s text alignment in a form.

<common>

Style

Text Color

Color

Component’s text color in a form or table.

<common>

Style

Text Typography [2]

display-4, display-3, display-2, display-1, headline, title, subhead, body-2, body-1, caption, button

Body-1

Text typography in forms. Tables do not allow typography.

[1] This property only applies to fields in forms in authoring mode.
[2] This property only applies to fields in forms.
[3] This property only applies to table columns.

Component Specific Properties Reference

The following properties are specific to component types.

Component Tab Property Value Default Value Description

Checkbox

Format

Use Switch [1]

Boolean

false

Displays a switch instead of a regular check-box.

Chips

Format

Display Avatar

Boolean

false

Displays the avatar in the chip.

Chips

Format

Display Secondary Text

Boolean

false

Displays the secondary text in the chip, between brackets.

Chips

Format

Max Width

String

Maximum horizontal size in pixels of each chip in the list.

Date Picker

Format

Display Format

String

Date and time formatting pattern. See Display and Editing Formats for more information.

Date Picker

Format

Editing Format [1]

String

Date and time data entry pattern. See Display and Editing Formats for more information.

Date Picker

Format

Use Time Picker [1]

Boolean

false

Whether the date picker component should also display a time picker, for timestamps only. For dates, the component never displays the time picker.

Date Picker

Data

Min. Date

Date

Minimum date for the date picker (inclusive)

Date Picker

Data

Max. Date

Date

Maximum date for the date picker (inclusive)

Hyperlink

Data

Display Text

String

<value>

Text displayed for the hyperlink value. The <display-string>|<url> syntax is also supported in the URL when Link Source is set to url. If no display test is specified the URL is displayed.

Hyperlink

Data

Link Source

content, url

Depends.

Defines the type of the value, whether it is a binary content or an URL in a string. Defaults to content for binary value, and to URL otherwise.

Hyperlink

Format

Link Target

same-window, new-window, download, popup_object, popup_image

new-window

Navigation target for the hyperlink.

ID

Format

Display Format

String

Display format pattern that can be used to format the number if ID is a number. See Display and Editing Formats for more information.

Image

Data

Fallback Image

Binary or String. See Images for more information.

Fallback image when the image value is empty. The image type is declared in the Fallback Image Source.

Image

Data

Fallback Image Source

content, url

url

Defines the type of the fallback-image expression.

Image

Data

Source

content, url

Depends

Defines the type for this image value. Defaults to content the data type is binary, URL otherwise.

Image

Format

Full Size on Click

Boolean

true

Open the image in full size when it is clicked.

Image

Format

Height

String

200px

Height of the image container in pixels.

Image

Style

Image Alignment

top-left, top-center, top-right, middle-left, middle-center, middle-right, bottom-left, bottom-center, bottom-right

middle-center

Defines how the image is aligned in its container.

Image

Style

Image Resize Mode

fill, fit, stretch, no-scale, contain

fit

Defines how the image is resized in its container.

Markdown

Format

Char Counter [1]

Boolean

false

Defines whether a character counter is displayed under the markdown component when authoring.

Markdown

Format

Height

String

200px

Height of the markdown component in pixels.

Markdown

Format

Max Width

String

Maximum width of the markdown component, in pixels. If no value is specified, all the horizontal space is used.

Menu

Data

List Items [1]

String

Semicolon-separated list of list items, used when the List Type is set to manual-list. This list contains only codes. The value displayed and selected is stored.

Menu

Data

List Type

list-of-values, manual-list

Depends

Source of the values in the menu: Either the underlying list of value, or a list of values provided in the List Items property. It defaults to list-of-values in the datatype is a list of values.

Menu

Format

Display Format

String

Display format pattern used to format the text displayed for a list of values. See Display and Editing Formats for more information.

Menu

Format

List Sort

code, label

label

Defines how the list is sorted. Only when List Type is list-of-values

Menu

Format

Display Avatar

Boolean

false

Displays the LOV values' images as avatars for the menu items.

Object

Data

Source Type

content, url

content

Defines whether the value contains the content to display or an URL to the content.

Object

Format

Height

String

200px

Height of the object component in pixels.

Object

Format

Max Width

String

Maximum width of the object component, in pixels. If no value is specified, all the horizontal space is used.

Reference

Format

Display Card

Display Card

Display card to use to display this reference as a chip in a form or in the auto-complete component when authoring.

Reference

Format

Max Width

String

Maximum horizontal size in pixels when this reference is displayed as a chip. Ignored if the Display Type is hyperlink.

Reference

Style

Display Avatar

Boolean

false

Display the avatar in the chip. Ignored if the Display Type is hyperlink.

Reference

Style

Display Secondary Text

Boolean

false

Display the secondary text in the chip, between brackets. Ignored if the Display Type is hyperlink.

Reference

Style

Display Type

chip, hyperlink

hyperlink

Defines how to display the reference, as a hyperlink showing the display card primary text, or a chip.

Text

Format

Char Counter [1]

Boolean

false

Defines whether a character counter is displayed under the text component when authoring.

Text

Format

Display Format

String

Display format pattern for numbers and dates. See Display and Editing Formats for more information.

Text

Format

Editing Format [1]

String

Data entry format pattern used as a helper when authoring. See Display and Editing Formats for more information.

Text

Format

Max Lines [1]

Number

3

Maximum number of lines for a multi-line component when authoring.

Text

Format

Min Lines [1]

Number

1

Initial number of lines for a multi-line component when authoring.

Text

Format

Multi-Line [1]

Boolean

false

Defines whether to use a multi-line component when authoring.

[1] This property only applies for forms in authoring.
[2] This property is not used when authoring.

Display and Editing Formats

Display Format
Dates

For dates, the display format uses the Simple Date Format patterns, with the following exceptions:

  • The following Simple Date Format pattern letters are not supported (E, F, K, W, z, Z).

Texts and IDs

For texts and IDs, the display format uses the following patterns elements for numeric values:

  • 0: A digit that must be displayed, after the decimal separator, the number of zeros indicates the number of decimals.

  • +: '+' or '-' sign

  • ,: The thousand separator

  • .: The decimal separator

The thousand and decimal separators used in the pattern (that is , and .) will appear in the application for each user according to the user settings. Although users may choose in their settings different separator characters to suit their locale and regional preferences, the patterns are always defined at design-time with the , and . characters.
Example 18. Number display format examples

In these examples, the results displayed are for user who has a thousand separator set to , and a decimal separator set to . in his preferences.

Number Format Displayed Result

10000

0,0.0000

10,000.0000

10000.25

0,0

10,000

10000.25

+0,0

+10,000

-10000

0,0.0

-10,000.0

10000.1254

0.000

10000.125

10000.1254

0[.]00000

10000.12540

-10000

(0,0.0000)

(10,000.0000)

0.25

0.00000

0.25000

0.25

0.0[0000]

0.25

List of Values

For list of values, the display format supports the code and label tokens, which are replaced by the code and label of the list of value item. For example, if you specify the code // label pattern for a gender list of value, the values will appear as F // Female, M // Male, etc.

Editing Format
Dates

For dates, the editing format property supports the same pattern as the display format, and this format is required strictly when inputing the date.

Texts and IDs

For texts and IDs, the editing format property supports the same pattern as the display format, but this pattern is only used to display the original number value. When the user enters values, only the decimal separator (as specified in the user preferences) and sign are taken into account and other characters are ignored.

List of Values

For list of values in menus, the display format supports the code and label tokens, which are replaced by the code and label of the list of value items in the selection menu.

Appendix B: Stepper Events Reference

This appendix lists the events triggered into steppers, grouped by User Action and ordered by their execution position. The validations running between those events are also indicated.

User opens a stepper

  1. Stepper - Start, as defined in the stepper.

  2. Step - Enter (Once), as defined in the first step displayed.

  3. Step - Enter (Each), as defined in the first step displayed.

User finishes a stepper by clicking FINISH

  1. Step - Exit (Pre validation - Once), as defined in the current step.

  2. Step - Exit (Pre validation - Each), as defined in the current step.

  3. Validation - Step Exit Validations, as defined in the current step.

  4. Step - Exit (Post validation - Each), as defined in the current step.

  5. Step - Exit (Post validation - Once), as defined in the current step.

  6. Step - Close (Once), as defined in the current step.

  7. Step - Close (Each), as defined in the current step.

  8. Collection Step - Close Child (Once), as defined in the parent collection step. (if any)

  9. Collection Step - Close Child (Each), as defined in the parent collection step. (if any)

  10. Step - Exit (Pre validation - Once), as defined in the parent collection step. (if any)

  11. Step - Exit (Pre validation - Each), as defined in the parent collection step. (if any)

  12. Validation - Step Exit Validations, as defined in the parent collection step. (if any)

  13. Step - Exit (Post validation - Each), as defined in the parent collection step. (if any)

  14. Step - Exit (Post validation - Once), as defined in the parent collection step. (if any)

  15. Stepper - Finish (Pre Validation), as defined in the stepper.

  16. Validation - Stepper Finish Validations, as defined in the stepper.

  17. Stepper - Finish (Post Validation), as defined in the stepper.

User closes a stepper with CLOSE then DISCARD ALL

  1. Stepper - Close (Discard), as defined in the stepper.

User closes a stepper with CLOSE then SAVE FOR LATER

  1. Stepper - Close (Saved), as defined in the stepper.

User clicks BACK on any step

  1. Step - Back (Once), as defined in the current step.

  2. Step - Back (Each), as defined in the current step.

  3. Step - Enter (Once), as defined in the previous step.

  4. Step - Enter (Each), as defined in the previous step.

User clicks PREVIOUS, CONTINUE or a step in the header, from any step

  1. Step - Exit (Pre validation - Once), as defined in the current step.

  2. Step - Exit (Pre validation - Each), as defined in the current step.

  3. Validation - Step Exit Validations, as defined in the current step.

  4. Step - Exit (Post validation - Each), as defined in the current step.

  5. Step - Exit (Post validation - Once), as defined in the current step.

  6. Step - Enter (Once), as defined in the previous or next step.

  7. Step - Enter (Each), as defined in the previous or next step.

User clicks CONTINUE on any step

  1. Step - Exit (Pre validation - Once), as defined in the current step.

  2. Step - Exit (Pre validation - Each), as defined in the current step.

  3. Validation - Step Exit Validations, as defined in the current step.

  4. Step - Exit (Post validation - Each), as defined in the current step.

  5. Step - Exit (Post validation - Once), as defined in the current step.

  6. Step - Continue (Once), as defined in the current step.

  7. Step - Continue (Each), as defined in the current step.

  8. Step - Enter (Once), as defined in the next step.

  9. Step - Enter (Each), as defined in the next step.

User clicks CLOSE on a step within a parent collection step

  1. Step - Exit (Pre validation - Once), as defined in the current step.

  2. Step - Exit (Pre validation - Each), as defined in the current step.

  3. Validation - Step Exit Validations, as defined in the current step.

  4. Step - Exit (Post validation - Each), as defined in the current step.

  5. Step - Exit (Post validation - Once), as defined in the current step.

  6. Step - Close (Once), as defined in the current step.

  7. Step - Close (Each), as defined in the current step.

  8. Collection Step - Close Child (Once), as defined in the parent collection step.

  9. Collection Step - Close Child (Each), as defined in the parent collection step.

User clicks ADD ANOTHER on a step within a parent collection step

  1. Step - Exit (Pre validation - Once), as defined in the current step.

  2. Step - Exit (Pre validation - Each), as defined in the current step.

  3. Validation - Step Exit Validations, as defined in the current step.

  4. Step - Exit (Post validation - Each), as defined in the current step.

  5. Step - Exit (Post validation - Once), as defined in the current step.

  6. Collection Step - Add Child, as defined in the parent collection step.

  7. Step - Enter (Once), as defined in the first step displayed of the parent collection step.

  8. Step - Enter (Each), as defined in the first step displayed of the parent collection step.

User clicks EDIT NEXT on a step within a parent collection step

  1. Step - Exit (Pre validation - Once), as defined in the current step.

  2. Step - Exit (Pre validation - Each), as defined in the current step.

  3. Validation - Step Exit Validations, as defined in the current step.

  4. Step - Exit (Post validation - Each), as defined in the current step.

  5. Step - Exit (Post validation - Once), as defined in the current step.

  6. Collection Step - Edit Child (Once), as defined in the parent collection step.

  7. Collection Step - Edit Child (Each), as defined in the parent collection step.

  8. Step - Enter (Once), as defined in the first step displayed of the parent collection step.

  9. Step - Enter (Each), as defined in the first step displayed of the parent collection step.

User clicks CREATE on a collection step

  1. Collection Step - Add Child, as defined in the current collection step.

  2. Step - Enter (Once), as defined in the first child step displayed.

  3. Step - Enter (Each), as defined in the first child step displayed.

User clicks EDIT on a collection step

  1. Collection Step - Edit Child (Once), as defined in the current collection step.

  2. Collection Step - Edit Child (Each), as defined in the current collection step.

  3. Step - Enter (Once), as defined in the first child step displayed.

  4. Step - Enter (Each), as defined in the first child step displayed.

User clicks REMOVE on a collection step

  1. Collection Step - Remove Child (Once), as defined in the current collection step.

  2. Collection Step - Remove Child (Each), as defined in the current collection step.

User clicks DELETE on a collection step

  1. Collection Step - Remove Child (Once), as defined in the current collection step.

  2. Collection Step - Remove Child (Each), as defined in the current collection step.

User applies a MASS-UPDATE on a collection step

  1. Collection Step - Edit Child (Once), as defined in the current collection step.

  2. Collection Step - Edit Child (Each), as defined in the current collection step.

User clicks COPY on a collection step

  1. Collection Step - Add Child, as defined in the current collection step.

  2. Collection Step - Copy Child (Once), as defined in the current collection step.

  3. Collection Step - Copy Child (Each), as defined in the current collection step.

  4. Collection Step - Edit Child (Once), as defined in the current collection step.

  5. Collection Step - Edit Child (Each), as defined in the current collection step.

  6. Step - Enter (Once), as defined in the first child step displayed.

  7. Step - Enter (Each), as defined in the first child step displayed.

User applies an IMPORT on a collection step

  1. Collection Step - Import (Once), as defined in the current collection step.

  2. Collection Step - Import (Each), as defined in the current collection step.

Appendix C: Workflow Notifications Variables

The following table lists the variables available in the workflow notifications for tasks and transitions.

Variable Syntax Description Example Tasks Transitions

${LoadId}

ID of the load (workflow instance) stored in the repository

23442

Yes

Yes

${JobName}

Name of the job executed at the end of the workflow

PRODUCT_ALL

Yes

Yes

${WorkflowName}

Name of workflow defined at design-time

CreateProductWorkflow

Yes

Yes

${ActivityLabel}

Label of workflow instance (either the workflow design-time label or the label entered at runtime)

Create Products Workflow

Yes

Yes

${AdminRoleName}

Name of the administrator role defined for the workflow

SemarchyAdministrator

Yes

Yes

${AdminRoleEmail}

Email of the administrator role, if any

administrators@acme.com

Yes

Yes

${CurrentTaskName}

Name of current task as defined at design-time

CreateItems

Yes

Yes

${CurrentTaskLabel}

Label of current task as defined at design-time

Create Items

Yes

Yes

${CurrentTaskEntityName}

Name of the entity managed in the current task

Product

Yes

Yes

${CurrentTaskEntityLabel}

Label of the entity managed in the current task

Product

Yes

Yes

${CurrentTaskRootRecordCount}

Number of root records for current task.
Note that this record count is related to the performer (connected user). For scheduled notifications (tasks durations reached), there is no connected user, and this variable is not computed (value set to "-").

2

Yes

Yes

${CurrentTaskAssignedRoleName}

Name of the role the current task is assigned to

SupplyChain

Yes

Yes

${CurrentTaskAssignedRoleEmail}

Email of role the current task is assigned to

suppliers@acme.com

Yes

Yes

${CurrentTaskAssignerRoleName}

Name of role who can assign/reassign the current task

SupplyChainRouter

Yes

Yes

${CurrentTaskAssignerRoleEmail}

Email of role who can assign/reassign the current task

spr@acme.com

Yes

Yes

${CurrentTaskAssigneeName}

Name of current task assignee (the performer, when task is running)

Joe Celko

Yes

Yes

${CurrentTaskAssigneeEmail}

Email of current task assignee

joe.celko@acme.com

Yes

Yes

${CurrentTaskUnassigneeName}

Name of current task assignee before an un-assign or re-assign operation

-

Yes

Yes

${CurrentTaskUnassigneeEmail}

Email of current task assignee before un-assign or re-assign operation

-

Yes

Yes

${CurrentTaskPendingMaxDuration}

Max pending duration in minutes for current task

-

Yes

Yes

${CurrentTaskPendingActualDuration}

Actual pending duration in minutes for current task

1929

Yes

Yes

${CurrentTaskRunningMaxDuration}

Max running duration in minutes for current task

-

Yes

Yes

${CurrentTaskRunningActualDuration}

Actual running duration in minutes for current task

12

Yes

Yes

${CurrentTaskTotalMaxDuration}

Max total duration in minutes for current task

-

Yes

Yes

${CurrentTaskTotalActualDuration}

Actual total duration in minutes for current task

1941

Yes

Yes

${CurrentTaskStartDate}

First start timestamp of current task

2016-12-21 21:23:11 PT

Yes

Yes

${CurrentTaskEndDate}

Timestamp at which task was terminated

2016-12-21 21:23:23 PT

Yes

Yes

${CurrentTaskExecCount}

Number of executions of current task

1

Yes

Yes

${CurrentEvent}

Label of the event that generated current message. Can be one of the following: Task Started, Task Suspended, Task Finished, Task Assigned, Task Unassigned, Task Reassigned, Task Max Pending Duration Reached, Task Max Running Duration Reached, Task Total Duration Reached, Transition

Transition

Yes

Yes

${TransitionName}

Name of the transition

RequestMoreInfo

No

Yes

${TransitionLabel}

Label of the transition

Request More Information

No

Yes

${TransitionComments}

Comments entered by the user during the transition

Please approve.

No

Yes

${PreviousTaskName}

Name of the previous task as defined at design-time

CreateProductData

No

Yes

${PreviousTaskLabel}

Label of previous task as defined at design-time

Create Product Data

No

Yes

${PreviousTaskEntityName}

Name of the entity for the previous task

Product

No

Yes

${PreviousTaskEntityLabel}

Label of the entity for the previous task

Product

No

Yes

${PreviousTaskRootRecordCount}

Number of root records for the previous task.
Note that this record count is related to the performer (connected user). For scheduled notifications (tasks durations reached), there is no connected user, and this variable is not computed (value set to "-").

2

No

Yes

${PreviousTaskAssignedRoleName}

Name of role the previous task is assigned to

Corp Marketing

No

Yes

${PreviousTaskAssignedRoleEmail}

Email of role the previous task is assigned to

-

No

Yes

${PreviousTaskAssignerRoleName}

Name of role who can assign/reassign the previous task

Corp Marketing

No

Yes

${PreviousTaskAssignerRoleEmail}

Email of role who can assign/reassign the previous task

-

No

Yes

${PreviousTaskAssigneeName}

Name of the previous task’s assignee

Bob Marley

No

Yes

${PreviousTaskAssigneeEmail}

Email of the previous task assignee (the performer, when task is running)

robert.nesta@acme.com

No

Yes

${PreviousTaskTotalMaxDuration}

Max total duration in minutes for the previous task

-

No

Yes

${PreviousTaskTotalActualDuration}

Actual total duration in minutes for the previous task

1941

No

Yes

${PreviousTaskStartDate}

First start timestamp of previous task

2016-12-21 21:23:11 PT

No

Yes

${PreviousTaskEndDate}

Timestamp at which previous task was terminated

2016-12-21 21:23:23 PT

No

Yes

${PreviousTaskExecCount}

Number of executions of previous task

1

No

Yes

${NextTaskName}

Name of the next task as defined at design-time

CreateProductData

No

Yes

${NextTaskLabel}

Label of next task as defined at design-time

Create Product Data

No

Yes

${NextTaskEntityName}

Name of the entity for the next task

Product

No

Yes

${NextTaskEntityLabel}

Label of the entity for the next task

Product

No

Yes

${NextTaskAssignedRoleName}

Name of role the next task is assigned to

Corp Marketing

No

Yes

${NextTaskAssignedRoleEmail}

Email of role the next task is assigned to

-

No

Yes

${NextTaskAssignerRoleName}

Name of role who can assign/reassign the next task

Corp Marketing

No

Yes

${NextTaskAssignerRoleEmail}

Email of role who can assign/reassign the next task

-

No

Yes

${NextTaskAssigneeName}

Name of the next task’s assignee

Bob Marley

No

Yes

${NextTaskAssigneeEmail}

Email of the next task assignee (the performer, when task is running)

robert.nesta@acme.com

No

Yes

${WorkflowUrl}

URL allowing to view and interact with the workflow

http://host/semarchy/mdm-apps/dloc/app/workflow/42

Yes

Yes

${ServerBaseUrl}

Base URL for the server

http://host/semarchy

Yes

Yes

${ApplicationBaseUrl}

Base URL for the current application

http://host/semarchy/mdm-apps/dloc/app

Yes

Yes

${ViewWorkflowAction}

Action button with URLs to interact with the task directly.

<div><a href=…..>View Workflow</a>…​</div>

Yes

Yes