Deployment
This process is the deployment in a run-time environment (for production or development) of a model designed with its integration process.
Introduction to Deployment
Deployment consist in creating or updating in a
Data Location (a database schema accessed via a JDBC datasource) a
Model Edition. Once this model edition is deployed, it is possible to create a
Data Edition (version of the data) using this model edition (version of the metadata).
In this process, the following components are involved:
- A
Data Location is a database schema into which several
Model Editions of the same model will be deployed and
Data Editions will be created. This data location is declared in Semarchy Convergence for MDM, and uses a JDBC datasource defined in the application server.
- In a data location, a
Deployed Model Edition is a model version deployed at a given time in a data location. As an MDM Hub model evolves over time, for example to include new entities or functional areas, new model editions are created then deployed.
Deployed Model Editions reflect this evolution in the
structure of the MDM Hub.
- Similarly, a
Data Edition reflects the evolution of the data stored in the hub over time. You can perform snapshots (editions) of the master data at given points in time.
Data Editions reflect the evolution in the
content of the MDM Hub.
In the deployment process:
- You can maintain as many data locations as you want in Semarchy Convergence for MDM, but a data location is always attached to one repository.
- A data location can contains editions from the one single model .
- You can deploy several model editions into the same data location.
- You can only have one open data edition in a data location. You can have old (closed) data editions in the same location. These closed editions are read-only, and can use old deployed model editions.
Data Location Types
There are two types of data location. This type is selected when the data location is created and cannot be changed afterwards:
The data location types are:
-
Development Data Locations: A data location of this type supports
installing and updating open or closed model editions. In short, it allows also install/update of models being edited.
-
Production Data Location: A data location of this type supports
installing only closed model editions. Updating is not allowed, and deploying an editable model edition (opened) is not allowed.
Important: Be cautious when choosing the data location type, at is will determine the type of deployment operations that can be done. It is recommended to use only Production Data locations for Production and User Acceptance Test environments.
Data Location Structures
A Data Edition is based on one of the Model Editions deployed in the location. Many data editions may coexist in the same data location.
The various data editions are stored in the same data structures. The data structures (tables) in a given Data Location are compatible with any of the Model Editions deployed in this data location. When a new model edition is installed, the structures are modified to support this new model edition in addition to the ones previously deployed.
In addition, data duplication is avoided as much as possible. For example: If a Golden Record exists but remains the same for 5 editions, it is only stored once golden data table, and not replicated five times.
Creating a Data Location
A Data Location is a database schema into which several Model Editions will be deployed and Data Editions will be created.
Note: A data location is a connection to a database schema via a JDBC datasource defined in the application server running Semarchy Convergence for MDM. Make sure to have the administrator of this application server create this datasource, and to have the database administrator create the schema for you before creating the data location in Semarchy Workbench.
To create a new data location:
- In the menu, select
File > New > New Data Location. The
Create New Data Location wizard opens.
- In the
Create New Data Location wizard, check the
Auto Fill option and then enter the following values:
-
Name: Internal name of the object.
-
Label: User-friendly label for this object. Note that as the
Auto Fill box is checked, the
Label is automatically filled in. Modifying this label is optional.
-
JNDI Datasource Name: Select the JDBC datasource pointing to the schema that will host the data location.
- In the
Description field, optionally enter a description for the Data Location.
- Select the
Location Type for this data location.
- Select the
Root Model: This model is the model for which editions can be deployed into the data location.
- Click
Finish to close the wizard. The
Data Locations perspective opens and the
Data Editions view displays the new data location.
To delete a data location:
- In the
Data Editions view, right-click the data location node and select
Delete. The
Delete Data Location wizard opens. In this wizard, you only delete the data location definition in Convergence for MDM. Deleting the data stored in the data location is optional.
- If you do not want to delete all the data in the data location schema, click
Finish. The data location is deleted but the data is preserved.
- If you want to delete all the data in the data location schema:
- Select the
Drop the content of the database schema to delete the content of the schema. Note that with this option, you choose to delete all the data stored in the hub, which cannot be undone.
- Click
Next. The wizard lists the objects that will be dropped from the schema.
- In the next wizard step, enter
DROP
to validate the choice, and then click
Finish. The data location as well as the schema content are deleted.
Warning! Deleting a data location is an operation that cannot be undone.
Tip: Deleting the data location as well as the schema content is convenient mechanism to reset the hub content at the early stages of the model design.
Deploying a Model Edition
There are two ways of deploying a model edition:
-
Install a Model Edition: This operation creates a new deployed model edition based on a given edition of the data location model. Note that if the data location is a production data location, the model edition must be closed.
-
Update a Model Edition: In a development data location, this option allows updating a deployed model edition that was previously deployed with the possible changes performed since the deployment or latest update operation. This method can be used to deploy and update for testing purposes model being developed at design-time.
Installing a Model Edition
To install a model edition:
- In the
Data Editions view, right-click the data location node and select
Install Model Edition.... The
Install Model Edition wizard opens.
- In the
Install Model Edition wizard, enter the following values:
-
Model Edition: Select the model edition to deploy into this location. Note that only the editions of the model selected for this data location are displayed.
- In the
Description field, optionally enter a description for the Model Edition.
- Click
Next. The changes to perform in the data location to support this new model edition are computed. A second page shows the SQL script to run on the schema to deploy this model edition. This script can be executed by Semarchy Convergence for MDM or you can download, customize and execute it separately.
- To download the script:
- Click the download link.
- Click
Cancel to quit the wizard without running the script.
- Run the script manually.
- Restart the wizard from Step 1. The wizard should detect no DDL changes.
- Click
Finish to create the integration jobs and register the new deployed model edition.
- To run this script:
- Click
Finish to run the script and close the wizard.
- When the deployment is complete, a
Successful Deployment dialog appears. Click
OK to close this dialog.
In the future, when you perform changes to this model’s structure, you will be able to create new editions of this model (For example [0.2], [0.3], ...) and will be able to deploy these new editions in the same data location. These new deployments will generate patching scripts to bring the physical database objects to the level of the new model edition.
Note: Deploying a model edition does not modify data in the data editions already in place in the data location.
Updating a Model Edition
To update a model edition:
- In the
Data Editions view, expand the data location node and the
Deployed Model Edition node.
- Right-click the deployed model edition that you want to update and select
Update Deployed Model Edition (Design-Time). The
Update Deployed Model Edition wizard opens.
- In the
Update Deployed Model Edition wizard, select the items that you want to update: Integration Jobs, Purge Jobs and/or Database schema.
- Click
Next. The list of DDL instructions needed to update the data structures is displayed.
- Click
Finish to run the script and close the wizard.
Warning: Updating the job definitions erases all the logs attached to the previous job definitions.
Note: Although it is recommended to update both the jobs and schemas at the same time, you may want to update the data structure first then the jobs later. For example, if the data that you have in the data editions using this model edition is not fit for the new version of the jobs. In that case, you may want to run some transformation on the data with the updated data structures before updating the jobs.
Another use case for not deploying the job definition is when you know that the new and old job definition are similar and you want to preserve the existing job logs.
Creating a Data Edition
Creating a Root Data Edition
To create a data edition:
- In the
Data Editions view, select the data location node, right-click and select
Create Root Data Edition. The
Create New Root Data Edition wizard opens.
- In the
Create New Root Data Edition wizard, check the
Auto Fill option and then enter the following values:
-
Name: Internal name of the object.
-
Label: User-friendly label for this object. Note that as the
Auto Fill box is checked, the
Label is automatically filled in. Modifying this label is optional.
- In the
Description field, optionally enter a description for the Root Data Branch.
- Click
Next.
- In the
Data Edition screen, select the
Deployed Model Edition that will support the data edition.
- In the
Description field, optionally enter a description for the Root Data Edition.
- Click
Finish to close the wizard. The
RootBranch data branch is created and a first data edition is created under this root branch.
This data edition is open. It means that we can perform loads and changes to this edition.
Later on, you will be able to close this edition and open a new one. Closing an edition freezes its content, creating a snapshot of the MDM Hub content.
Closing and Creating a New Data Edition
To close and create a new data edition:
- In the
Data Editions view, expand the data location node.
- Expand the
Data Branches > RootBranch node.
- Right-click the latest data edition (open) and select
Close and Create New Edition. The
Close and Create New Data Edition wizard opens.
- In the
Close and Create New Data Edition wizard, enter a
Description for the new data edition.
- Select the
Deployed Model Edition that will support the data edition.
- Click
Finish to close the wizard. The new data edition under the
RootBranch data branch is created. The previous data edition is closed.
Note: It is possible to close and create a new edition while data entry or duplicate management workflows are in progress in the data edition. Long running workflows can execute across several data editions. A warning is issued workflows are in progress.
Warning: When closing and opening a new data edition with a different model edition, there is no check performed on the new model to guarantee that the ongoing workflows are supported with the new model. The administrator should check that all ongoing workflows will be able to continue to completion with the new model edition.
Note: If integration jobs are currently running for this data edition, you may be prompted to move it to a Maintenance mode. This status allows current jobs to complete while rejecting new submitted jobs.
Note: You can automate this operation in production environements using the Administrative Web Service. See the
Web Services chapter in the
Semarchy Convergence for MDM Integration Guide for more information.
Advanced Deployment Techniques
Switching Model Edition
This operation allows you to change the deployed model edition supporting a given data edition without closing the data edition.
This prevent you from creating data snapshots for model changes.
To perform the switch, you must first
move the data edition to Maintenance mode.
To swich a data edition to a different model edition:
- In the
Data Editions view, expand the data location node.
- Expand the
Data Branches > RootBranch node.
- Right-click the latest data edition (open) and select
Switch Model Edition. The
Switch Model Edition wizard opens.
- Select the
Deployed Model Edition that will support the data edition from now on.
- Check the option if you want to automatically re-open the data edition after the switch.
- Click
Finish to close the wizard. The data edition is now switched to the new model edition.
If you have not selected the option to automatically re-open the data edition, you must now
move it back to Open.
Moving Models at Design-Time
At design-time, it is possible to move models from one repository to another design repository using Export/Import:
- Export is allowed from a design repository, but also from a deployment repository.
- Import is possible in a design repository, either:
- to create a new model from the import
- or to overwrite an existing
open model edition.
Exporting a Model Edition
To export a model edition:
- Open the
Model Editions perspective.
- Expand the model branch containing the model edition that you want to export.
- Select the model edition that you want to export, right click and select
Export Model Edition.
- In the
Model Edition Export dialog, select an
Encoding for the export file.
- Click the
Download link to download the export file on your local file system.
- Click
Close.
Importing to a New Model
To import and create a new model:
- In the menu, select
File > New > New Model from import... The
Import to a New Model wizard opens.
- Click the
Open button and select the export file.
- Click
Finish to perform the import.
Click the
Refresh button in the
Model Editions view. The new model appears in the list.
Importing on an Existing Model
To import and replace an existing model:
- Open the
Model Editions perspective.
- Expand the model branch containing the model edition that you want to replace.
- Select the
open model edition that you want to replace, right click and select
Import Replace.... The
Import-Replace Model Edition wizard opens.
- Click the
Open button and select the export file.
- Click
Finish to perform the import.
- Click
OK to confirm the deletion of the existing model.
The existing model edition is replaced by the content of the export file.
Deploying to a Remote Location
Frequently, the deployment environment is separated from the development environment. For example, the development/QA and production sites are located on different networks or locations. In such cases, it is necessary to use export and import to transfer the model edition before performing the deployment in production.
A remote deployment consists in moving a
closed model edition:
- From a design repository or a deployment repository used for Testing/UAT purposes;
- To a deployment repository.
Remote Deployment Architecture
In this configuration, two repositories are created instead of one:
- A
Design repository for the development and QA site, with
Development data locations attached to this repository.
- A
Deployment repository for the production site.
Production data locations are attached to this repository.
The process for deploying a model edition in this configuration is given below:
- The model edition is closed in the design repository.
- The closed model edition is exported from the design repository to an export file.
- The closed model edition is imported from the export file into the deployment repository.
- The closed model edition is deployed from the deployment repository into a production data location.
Exporting a Model Edition
To export a model edition:
- Open the
Model Editions perspective.
- Expand the model branch containing the model edition that you want to export.
- Select the
closed model edition that you want to export, right click and select
Export Model Edition.
- In the
Model Edition Export dialog, select an
Encoding for the export file.
- Click the
Download link to download the export file on your local file system.
- Click
Close.
Importing a Model Edition in a Deployment Repository
To import a model edition in a deployment repository:
- Open the
Model Editions perspective.
- Select
File > New > Import Model Edition. The
Import Model Editions wizard opens.
- Click the
Open button and select the export file.
- Click
Next.
- Review the content of the Import Preview page and then click
Finish.
Note: When importing a model edition, the root model and the branches containing this model edition are automatically created as needed.
Warning: When importing successive versions of model editions in a deployment repository, it is not recommended to skip intermediate versions, as it is not possible later to import these intermediate versions. For example, if importing version 0.1 of a model, then importing version 0.4, the intermediate versions – 0.2, 0.3 – can longer be imported into this repository.
Data Edition Statuses
Understanding Data Edition Statuses
This section provided detailed information about the processes involved when closing a data edition.
A data edition can be in three statuses:
-
Open: A data edition in this status can be accessed in read/write mode, accepts incoming batches and processes its current batches.
-
Closed: A closed data edition can be accessed in read mode, does not accept incoming batch (a submit load on this data edition will fail) and no batch is in processing. This status cannot be modified.
-
Maintenance: A data edition in this status can be accessed in read-only mode. It does not accept incoming batches but completes its current batches. New loads cannot be initialized and existing loads cannot be submitted. Unlike the closed status, this status can be reverted to the open status.
When moving a data edition to a closed status, no batch must be in progress. You must first wait for batches to complete before moving the data edition to a closed state. Loads submitted after the data edition is closed will fail as the closed data edition is read only.
When moving a data edition to a Maintenance status, the currently processed batches continue until completion. Loads submitted after the data edition is moved to Maintenance will fail. They can be kept open and submitted later, when the data edition is restored to an open status.
Moving a Data Edition to Maintenance Mode
To set a data edition status to maintenance mode:
- In the
Data Editions view, expand the data location node.
- Expand the
Data Branches > RootBranch node.
- Right-click the latest data edition (open) and select
Set Data Edition Status to Maintenance.
- Click
OK in the confirm dialog.
The data edition is moved to Maintenance mode.
To set a data edition status to open mode:
- In the
Data Editions view, expand the data location node.
- Expand the
Data Branches > RootBranch node.
- Right-click the latest data edition (open) and select
Set Data Edition Status Back to Open.
The data edition is moved to an open state.
Using the Maintenance Mode
The Maintenance status can be used to perform maintenance tasks on the data edition.
For example, if you want to close the data edition and open a new data edition on a new model edition with data structure changes that mandate manual DML commands to be issued on the hub tables, you may perform the following sequence:
- Move the data edition to Maintenance mode.
- Let the currently running batches complete. No batch can be submitted to this edition.
- Close and open a new data edition, using the updated model edition.
- Move the new data open edition to a Maintenance mode.
- Perform your DML commands.
- Move the data edition from Maintenance to the Open status. Batches can now be submitted to this edition.
Using this sequence, you prevent batches be submitted while the hub is in Maintenance.
Maintenance status is also mandatory when switching a data edition to a different deployed model edition.