Welcome to Semarchy xDM Integrator.
This document is intended for users interested in optimizing the usage of the Semarchy xDM Integrator components in production environments.
Preface
Audience
Document Conventions
This document uses the following formatting conventions:
Convention | Meaning |
---|---|
boldface |
Boldface type indicates graphical user interface elements associated with an action, or a product specific term or concept. |
italic |
Italic type indicates special emphasis or placeholder variable that you need to provide. |
|
Monospace type indicates code example, text or commands that you enter. |
Other Semarchy Resources
In addition to the product manuals, Semarchy provides other resources available on its web site: http://www.semarchy.com.
Obtaining Help
There are many ways to access the Semarchy Technical Support. You can call or email our global Technical Support Center (support@semarchy.com). For more information, see http://www.semarchy.com.
Feedback
We welcome your comments and suggestions on the quality and usefulness
of this documentation.
If you find any error or have any suggestion for improvement, please
mail support@semarchy.com and indicate the title of the documentation
along with the chapter, section, and page number, if available. Please
let us know if you want a reply.
Overview
Using this guide, you will:
-
Learn how to deploy and schedule deliveries for Semarchy xDM Integrator in production
This guide contains information about using the product to manage production tasks, such as:
-
deployment
-
delivery scheduling
Product Flow
When working with Semarchy xDM Integrator Analytics the final goal is to generate and deploy a Delivery, which is a file containing the complete sequence of tasks to be done, with ready to use connections (server, ports, passwords, etc.), and which will be executed by a Runtime Engine.
Producing those deliveries is performed through the following process, from the developer developing the Mappings and Processes to the final deployment of the delivery to be executed by the Runtimes.
-
Development of the mappings and processes by the development team.
-
Generation of packages for the production team to configure the elements required for the production environment
-
Generation of a delivery from the previously created package
-
Publishing of the delivery on Runtime(s)
The delivery can then be executed or scheduled.
This guide details the following tasks of this process:
-
Package Generation
-
Delivery Generation
-
Delivery Deployment
-
Delivery Execution
-
Delivery Scheduling
Generating Packages
A package is an archive containing all the needed files and folders to generate a delivery. Most of the time, a package is generated by the development team when they have finished working on a Mapping / Process. The package can then be used to generate the delivery that will be executed by the Runtimes.
There are two options to generate a package:
-
Generate the package directly from xDM Integrator Designer.
-
Generate the package from a command line.
Generating a package from xDM Integrator Designer
To generate a package from the Designer:
-
Right-click on the Mapping or Process
-
Select Build > Package [with documentation]
The package is generated in the /runtime/build/packages/
sub-directory of the xDM Integrator installation folder.
Generating a package from a command line
First, you must generate a file listing the packages to build, using the following syntax
build package "<PROCESS_PATH_WITHIN_WORKSPACE>" -target "<TARGET_DIRECTORY>"
Where:
-
<PATH_WITHIN_WORKSPACE>
is the relative path within the workspace to the Process to build the package from. -
<TARGET_DIRECTORY>
is the directory into which the generated package will be created.
For example:
build package "Tutorial - Fundamentals/Processes/Load All Datamart.proc.proc" -target "D:/Packages/" build package "My Project - 01/Process01.proc" -target "D:/Packages/" build package "My Project - 01/Process02.proc" -target "D:/Packages/"
To generate a package from a mapping, specify the path of the Mapping’s built Process file and not the .map file.
With the generated script file, you can generate the packages with the command line tool.
Open a command line in the Designer installation root folder, and use the following command:
java -jar plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar -application com.indy.shell.application -data "WORKSPACE_PATH" -script "SCRIPT_PATH" [-logFile "LOG_FILE_PATH"] -console –noSplash
This command uses the following parameters:
Parameter | Mandatory | Description |
---|---|---|
`-data "WORKSPACE_PATH" |
Yes |
Path to the xDM Integrator Designer workspace where the Processes are stored |
|
Yes |
Path to the script file containing the list of packages to build created earlier |
|
No |
Path to a log file where information about the Packages generation will be logged |
java -jar plugins/org.eclipse.equinox.launcher_1.3.0.v20120522-1813.jar -application com.indy.shell.application -data "D:/workspace/development" -script "D:/resources/scripts/packagesToGenerate.txt" -console –noSplash
Generating Deliveries
A delivery is what will be executed or scheduled in the final stage by the Runtimes.
It is a file which contains the complete sequence of tasks, with ready to use connections (server, ports, passwords, etc.). Everything in a delivery is already configured and it will only need a Runtime to run.
There are three possibilities to generate a delivery:
-
From the Designer
-
From Semarchy xDM Integrator Analytics
-
From the command line
From the Designer
With this method, the Designer takes care of everything. This means that the configuration (connection properties, parameters, …) of the delivery will be the one selected in the Designer at generation time.
To generate a delivery:
-
Right-click on the mapping or process
-
Select Build > Delivery.
-
-
The delivery file (with a
.deliv
extension) is generated in the/runtime/build/deliveries/
sub-directory of the xDM Integrator installation folder.
The delivery is generated and you must next deploy it to a Runtime to execute it.
From Semarchy xDM Integrator Analytics
For this operation, you must import into Semarchy xDM Integrator Analytics the package generated by the development team, configure it, and then generate (build) the delivery from it.
This method is recommended for production teams, as it offers the possibility to create configurations for each environment (development, test, production, …) and easily build and deploy deliveries from the web UI.
Refer to the {dia-full-product-name documentation for more information.
From the Command Line
For this operation, you use the package that has been generated by the development team to build the delivery.
The Runtime ships with command line build scripts to generate a delivery from a package: builddelivery.bat
and builddelivery.sh
.
To generate a delivery from a package:
-
Open an operating system command line.
-
Go to the
/runtime/
directory of the runtime engine into which the package is deployed (in the/runtime/build/packages/
sub-directory). -
Use the following command to extract the configuration:
buildDelivery.bat <package_file_name> [-buildmode=substitution|generation] [-listprocessnames] [-extract] [-conf <name>] [-conffile <file_path>] [-processname <name>] [-deliveryfolder <folder_path>] buildDelivery.sh <package_file_name>.pck [-buildmode=substitution|generation] [-listprocessnames] [-extract] [-conf <name>] [-conffile <file_path>] [-processname <name>] [-deliveryfolder <folder_path>]
This command uses the following parameters:
Parameter | Description |
---|---|
|
File name containing the package to build the delivery from. |
|
Specifies how the delivery should be generated. this parameter supports two modes:
|
|
Lists all the processes contained in a package |
|
Extracts the package configuration file, containing the externalized metadata parameters which are used in the processes and which can be customized. |
|
When used with the
|
|
Specifies which process contained in the package should be built. |
|
Location where the deliveries are generated. |
Build Modes
Each build mode has its advantages:
-
Generation: This mode is supported by all Runtime versions.
-
This mode has been used by default since the build scripts exist, which launches a generation mechanism to build the delivery.
-
The main advantage of this mode is that as a new generation step is performed, the user does not have to set all the parameters in the extracted configuration file or to use a configuration file. The default value with which the package has been generated is used for parameters that have not been set in the configuration file.
-
Another advantage is that it allows customizing metadata parameters which were not defined as externalized when generating the package. This offers the possibility to add extra parameters on the extracted configuration file used to build the delivery, to replace parameters which were not anticipated when generating the package.
-
This mode may be time-consuming for large processes.
-
-
Substitution: This mode requires a runtime engine version S17.6.0 and higher. This mode consists in replacing the externalized attributes with the values provided inside the extracted configuration file. The builder retrieves the pre-built delivery which is inside the package and replaces the parameters in it.
-
This is the mode used by Semarchy xDM Integrator Analytics for generating deliveries.
-
It provides better performances as no generation is performed to build the delivery.
-
This mode also offers the possibility to generate deliveries for all the processes contained in a multi-process package, which is not possible with generation mode.
-
In this mode, it is mandatory to build the delivery using a configuration with all the externalized metadata parameters set.
-
For most common usage, the substitution mode can be used as it offers better performances and allows to build all the processes contained in a package.
The generation mode, which is still the default mode for backward compatibility, is useful when you need to override values for parameters which were not originally externalized when the package was generated.
Configuration Files
When using the -extract
option, the configuration file of the package is extracted and re-named as specified with the -conf
(or -conffile
) options.
This file contains all the Metadata parameters which are defined as externalized in the Metadata.
Some are externalized by default, such as the JDBC URL or JDBC User and password, and others can be externalized manually by the user from the Designer in the Metadata.
Those parameters are commented with a default value in the extracted configuration file, for the user to be able to customize them when generating deliveries.
################################################################# ### Name: super/Rdbms MetaData/Hypersonic SQL/HSQL - localhost_62210 ### Type: com.stambia.rdbms.server #_-1vK4CVLEeWjxY2_6aCFbA/url=jdbc:hsqldb:hsql://localhost:62210 #_-1vK4CVLEeWjxY2_6aCFbA/user=sa ################################################################# ### Name: super/Rdbms MetaData/Hypersonic SQL/HSQL - localhost_62210/HOTEL_MANAGEMENT ### Type: com.stambia.rdbms.schema #_aUU9YE26Eeay9ZeykqAlHA/TABLE_SCHEM=HOTEL_MANAGEMENT #################################################################
To specify a value for a parameter, uncomment the corresponding line (remove the ‹#› which is at its start) and specify the wanted value.
################################################################# ### Name: super/Rdbms MetaData/Hypersonic SQL/HSQL - localhost_62210 ### Type: com.stambia.rdbms.server _-1vK4CVLEeWjxY2_6aCFbA/url=jdbc:hsqldb:hsql://productionhost:62210 _-1vK4CVLEeWjxY2_6aCFbA/user=productionuser ################################################################# ### Name: super/Rdbms MetaData/Hypersonic SQL/HSQL - localhost_62210/HOTEL_MANAGEMENT ### Type: com.stambia.rdbms.schema _aUU9YE26Eeay9ZeykqAlHA/TABLE_SCHEM=HOTEL_PRODUCTION_SCHEMA #################################################################
When the parameters correspond to the expected values, the delivery can be built using the configuration.
Passwords properties are waiting for passwords encrypted using the encrypt <password> command on the Runtime engine console (engine command line tool). |
Examples
Below are examples for the builddelivery script.
builddelivery.sh mypackage.pck -listprocessnames
builddelivery.sh mypackage.pck -conf myconf -extract
builddelivery.sh mypackage.pck -conf myconf
builddelivery.sh mymultipackage.pck -conf myconf -processname "Load All Datamart" -deliveryfolder D:/deliveries/
builddelivery.sh mymultipackage.pck -conf myconf -deliveryfolder D:/deliveries/ -buildmode substitution
In this case, the substitution mode is mandatory. The generated delivery will be found in the build/deliveries
folder if the -deliveryfolder
option is not specified.
Once the delivery is generated, you must next deploy it on the Runtime you want it to be executable. |
Deploying Deliveries
After generating a delivery, you can publish it on a Runtime.
When the delivery is published, the Runtime will then be able to execute it.
You can deploy a delivery:
-
From xDM Integrator Designer
-
From Semarchy xDM Integrator Analytics
-
Manually
From xDM Integrator Designer
With this method, the Designer takes care of everything. This means that the configuration (connection properties, parameters, …) of the delivery will be the one selected in the Designer at generation time.
To generate a delivery:
-
Right-click on the mapping or process
-
Select Publish > Delivery.
-
The delivery is automatically built locally and then published to the currently connected runtime engine. It can now be executed and scheduled.
From Semarchy xDM Integrator Analytics
After having imported a Package, configured it, and built the associated delivery, Semarchy xDM Integrator Analytics offers the possibility to publish it to runtime(s).
This method is recommended for production teams, as it offers the possibility to create configurations for each environment (development, test, production, …) as well as build and deploy deliveries from the web UI.
Refer to the {dia-full-product-name documentation for more information.
Manually
If the delivery was built from the command line, you can deploy it manually in the Runtime on which you want to execute it.
Copy the file inside the location where are stored the deliveries of this Runtime. By default it is the /runtime/build/deliveries/
sub-directory of the xDM Integrator installation folder
Executing a delivery
You can execute a delivery from the Designer, from Analytics or from a command line.
Two command line tools are available for this purpose:
-
startdelivery.sh
: Creates a new Java process and executes the delivery as standalone outside a running Runtime. -
startcommand.sh
: Connects to a Runtime. The execute delivery command can then be performed.
To optimize memory management, prefer the startcommand tool, as the deliveries are executed by a single Runtime, as multiple threads within the same Java process.
|
To execute a delivery, use the following command line:
./startdelivery.sh –name [DELIVERY_NAME]
Or
./startdelivery.sh –file [DELIVERY_FILE]
This script is synchronous and waits for the execution to complete. The return code is either:
-
0 if the execution was successful
-
-1 if the execution failed
-
-2 if the session was killed
Using Variables
Certain deliveries can be parameterized with variables that you provide on the command line as shown below:
./startdelivery.sh –name [DELIVERY_NAME] –var [VARPATH1] [VALUE1] … -var [VARPATHn] [VALUEn]
For example:
./startdelivery.sh -name LOAD_CUSTOMERS -var ~/COUNTRYNAME USA
Defining the Configuration
When working with multi-Configuration deliveries, the configuration to use must be specified in the command line:
./startdelivery.sh -name [DELIVERY_NAME] -configuration [CONFIGURATION_NAME]
Defining the Session Name
By default, the session name is the same as the delivery name, but you can override this value using the sessionName parameter:
./startdelivery.sh -name [DELIVERY_NAME] -sessionName [SESSION_NAME]
Defining the Repository
The repository, into which the deliveries are stored can also be defined with a parameter. Note that the repository must be configured in the Runtime.
./startdelivery.sh -name [DELIVERY_NAME] -repository [REPOSITORY_NAME]
Configuring the Log Level
The log information that must be kept after execution can be specified in the command line:
./startdelivery.sh -name [DELIVERY_NAME] -loglevel [LOG_LEVEL]
The LOG_LEVEL values are provided below:
Value |
Description |
-1 |
Do not log anything (even errors are not logged) |
0 |
Logs everything during execution and deletes everything at the end. |
100 |
Logs everything during execution and deletes everything but the session and its stats. |
200 |
Logs everything during execution and deletes everything but the session/processes and their stats. |
300 |
Logs everything during execution and deletes everything but the session/processes/actions and their stats. |
400 |
Everything is logged. |
Scheduling Deliveries
Third-Party Scheduling
It is possible to use third-party schedulers to start the startdelivery.sh
script.
The return code is for this command is:
-
0 if the execution was successful
-
-1 in case of errors
-
-2 if the session was stopped manually (session was stopped before the end)
In addition to the return code, the command provides additional information on the standard output, as shown in the example below:
##### BEGIN ##### 04/05/2011 17:22:11,718 - SESSION: e5b70658db3117952ad056f12fbb9a21e08000 is started -- DURATION = 00:00:11,907 ##### STATISTICS ##### SQL_NB_ROWS=177051 SQL_STAT_INSERT=37969 SQL_STAT_UPDATE=0 SQL_STAT_DELETE=37972 04/05/2011 17:22:23,671 - SESSION: e5b70658db3117952ad056f12fbb9a21e08000 is ended ##### END #####
Built-In Scheduling
Scheduling Basics
You can schedule deliveries using the built-in scheduler. Deliveries scheduled this way appear as jobs, named after the delivery. However, you can schedule a delivery several times as jobs with a different name, and attach one or more triggers to one job. A named job is unique.
Scheduling From the Command Line
Once you are in the command line (using startcommand.bat
or startcommand.sh
), connect to the runtime into which you wish to schedule a job.
To connect to the local runtime, use the command:
>connect
You can also use this command to connect to a remote runtime. Use the help
command for more details about all commands.
To schedule a job with a single schedule:
>schedule delivery MY_DELIVERY cron "0 15 10 * * ? *"
To schedule a job with different names and schedules
>schedule delivery MY_DELIVERY with name NAME1 cron "0 15 10 * * ? *" >schedule delivery MY_DELIVERY with name NAME1 cron "0 20 10 * * ? *" >schedule delivery MY_DELIVERY with name NAME2 cron "0 15 11 * * ? *"
To schedule a job with a starting date and/or an ending date:
>schedule delivery MY_DELIVERY start "2009/12/10 12:55:22" cron "0 15 10 * * ? *" >schedule delivery MY_DELIVERY start "2009/12/10 12:55:22" end "2009/12/25 12:55:22" cron "0 15 10 * * ? *"
Scheduling with variables
>schedule delivery MY_DELIVERY var ~/VAR1 VALUE1 var ~/VAR2 VALUE2 cron "0 15 10 * * ? *"
Scheduling with a configuration (for multi-Configuration deliveries)
>schedule delivery MY_DELIVERY configuration MY_CONFIGURATION cron "0 15 10 * * ? *"
To retrieve the list of schedules for a delivery
> get delivery schedules MY_DELIVERY Getting schedules for MY_DELIVERY -- Trigger Name: CRON_MY_DELIVERY-0 -- Job Name: MY_DELIVERY [...]
Using the Trigger Name returned in the list of schedules, you can remove a schedule.
> remove trigger CRON_MY_DELIVERY-0
If you need more information about the command line:
>help
Cron Trigger Tutorial
General syntax
A Cron expression is a character string containing 6 or 7 fields separated by spaces.
These fields can contain the characters listed in this table or a combination of them.
Field name |
Mandatory |
Authorized values |
Authorized special characters |
Seconds |
YES |
0-59 |
, - * / |
Minutes |
YES |
0-59 |
, - * / |
Hours |
YES |
0-23 |
, - * / |
Day of the month |
YES |
1-31 |
, - * ? / L W |
Month |
YES |
1-12 or JAN-DEC |
, - * / |
Weekday |
YES |
1-7 or SUN-SAT |
, - * ? / L # |
Year |
NO |
empty, 1970-2099 |
, - * / |
Special Characters:
-
("all values") – used to select all the values for this field. For example,
in the minutes field means ‘every minute’.
-
?
("no specific value") – useful if you need to specify something in one of the two fields relevant to this special character, but not in the other one. For example, if you wish to set a trigger for a specific day in the month (let’s say the 10th), whatever the weekday. In this case, you will put ‘10’ in the ‘day of the month’ field, and ‘?’ in the ‘weekday’ field. For further understanding, check the examples. -
-
used to specify an interval. For example,10-12
in the ‘hour’ field means “hours 10, 11 and 12”. -
,
– used to add more values. For example, "MON,WED,FRI" in the ‘weekday’ field means “Mondays, Wednesdays and Fridays”. -
/
– used to specify repetition increments. For example, "0/15" in the ‘seconds’ field means “seconds 0, 15, 30 and 45”, in other words, every 15 seconds, starting at 0 included. And "5/15" in the same field means "seconds 5, 20, 35, et 50". If you put ‘/’ with no number before and a number behind (for example ‘/5’) is equivalent to putting a 0 before the ‘/’. (i.e. ‘0/5’). Another example: '1/3' in the ‘day of the month’ field means “trigger every 3 days starting on the 1st of the month”. -
L
(Last) – this character has different meanings depending on the field it is used in. For example, "L" in the ‘day of the month’ field means “the last day of the month”, i.e. the 31st for January, the 28th for February in non-leap years. If ‘L’ is used in the ‘weekday’ field, it means the 7th day, i.e. Saturday (SAT). However, if ‘L’ is used in the ‘weekday’ field following a number, it will mean “the last X day in the month”; for example “6L” means “the last Friday in the month”. So as to have no ambiguity, it is advised not to use the ‘L’ character in value lists. -
W
(weekday) – Used to specify the working weekday (Monday to Friday) that is nearest to a given date. For example, “15W” in the ‘day of the month’ field means “the working weekday the closest to the 15th”. So, if the 15th happens to be a Saturday, the trigger will position itself to Friday the 14th. And if the 15th happens to be a Sunday, the job will trigger on Monday the 16th. Take care, though, if you specify “1W” and the 1st happens to be a Saturday, the job will only be triggered on Monday the 3rd since you cannot change the month. Also, ‘W’ will only work with unique values, not with intervals. Characters ‘L’ and ‘W’ can be combined in the ‘day of the month’ field: “LW” will then mean “the last working weekday in the month”. -
#
– used to specify “the n-th day XXX in the month”. For example, the value "6#3" in the ‘weekday’ field means “the 3rd Friday of the month” (Day 6 = Friday, and #3 = the 3rd of the month).
Day names are not case-sensitive. This means ‘MON’ and ‘mon’ are identical.
Examples
Expression |
Meaning for the trigger |
0 0 12 * * ? |
At 12 o’clock every day |
0 15 10 ? * * |
at 10:15 every day |
0 15 10 * * ? |
at 10:15 every day |
0 15 10 * * ? * |
at 10:15 every day |
0 15 10 * * ? 2005 |
at 10:15 every day of year 2005 |
0 * 14 * * ? |
Every minute, between 14:00 and 14:59, every day |
0 0/5 14 * * ? |
Every 5 minutes from 14:00 to 14:55, every day |
0 0/5 14,18 * * ? |
Every 5 minutes from 14:00 to 14:55, every day, and every 5 minutes from 18:00 to 18:55, every day |
0 0-5 14 * * ? |
Every minute from 14:00 to 14:05, every day |
0 10,44 14 ? 3 WED |
at 14:10 and 14:44 every Wednesday of the month of March |
0 15 10 ? * MON-FRI |
at 10:15 every Monday, Tuesday, Wednesday, Thursday and Friday |
0 15 10 15 * ? |
at 10:15 on the 15th of each month |
0 15 10 L * ? |
at 10h15 every last day of the month |
0 15 10 ? * 6L |
at 10:15 the last Friday of each month |
0 15 10 ? * 6L 2002-2005 |
at 10:15 the last Friday of each month for years 2002 to 2005 |
0 15 10 ? * 6#3 |
at 10:15 the third Friday of each month |
0 0 12 1/5 * ? |
at 12:00 every 5 days, each month, starting on the 1st |
0 11 11 11 11 ? |
Every November 11th at 11:11 |
Runtime Engine Monitoring
You can use the Designer thick client to monitor the sessions running in a runtime engine, as well as Semarchy xDM Integrator Analytics to monitor these from a web UI.
It is also possible from the runtime engine console to monitor runtime engine activity.
Connecting the Runtime
Once you are in the command line (using startcommand.bat
or startcommand.sh
), connect to the runtime running the session.
To connect to a remote runtime, use the command:
>connect to <host_name> port <port_number>
where <host_name>
is the name of the host running the runtime engine and <port_number>
is the port of the runtime engine.
To connect a runtime from its server’s comment line, just run the following:
>connect
Managing Runtime Services
The runtime runs several services which can be started and stopped from the console.
To retrieve the status of a given service (or all services) in a given format:
>get services [name <name>] [format <format>]
To stop/start/restart services:
><start|stop|restart> <name> service
Managing Sessions
The following commands allow monitoring and configuring of the list of sessions managed by the runtime.
To retrieve a list of sessions by name, id, status or duration:
>get sessions [name <name>] [id <id1,id2,idn>] [status <running,error,done,stopped>] [duration <min> [to <max>]] [limit <limit>] [format <format>]]
To stop, restart or wait for the end of a given session identified by its ID:
><stop|restart|wait> session <id>
Stopping the Runtime
To stop the runtime, optionally waiting for all sessions to complete (recommended):
>stop runtime [wait sessions]
To kill the runtime process (not recommended):
>kill runtime [wait sessions]
You can stop a runtime remotely, but you need to connect to this runtime’s server (using SSH for example) to restart it. |
Batch Commands
You can create batches of commands and run them using the startcommand
script.
This command supports the following syntaxes.
startcommand[.sh|.bat] [-separator <separator>] [<command1>;<command2>;...;<commandx>]
startcommand[.sh|.bat][-file <commands file>]
The first syntax allows you to start a sequence of commands separated by a separator (defaults to ";"). The second syntax allows you to specify a file containing a sequence of commands.
Note that the first command in the sequence should be a connect
command.