Working with Processes

Processes define organized sequences of Actions executed at run-time on the IT systems.
Processes are organized using Sub-processes, which are themselves composed of actions, sub-processes or References to other processes.

Creating a Process

Creating a new Process

To create a new process:

  1. Click on the New Process button in the Project Explorer toolbar. The New Process Diagram wizard opens.
  2. Select the parent folder or project for your new resource.
  3. Enter a File Name and then click Finish. The process file is created and the editor for this file opens.
  4. Press CTRL+S to save the process.

Adding a Mapping

To add a mapping to a process:

  1. Drag and drop first the mapping from the Project Explorer into the process editor. This mapping is added as a reference in the current process.

Adding an Action

To add an action:

  1. In the Palette, select the action that you want to add to the process. You can expand the accordions in the palette to access the appropriate action.
  2. Click in the process diagram. A block representing your action appears in the diagram.
  3. Right-Click this action and then select Show Properties View.
  4. In the Properties views, set the following values in the Standard section:
  5. In the Parameters section, set the action’s parameters. Each action has its own set of parameters.
  6. Press CTRL+S to save the process.

Adding a Sub-Process

To add a sub-process:

  1. In the Palette, select the Process tool in the Component accordion.
  2. Click in the process diagram. A block representing your sub-process appears in the diagram.
  3. The focus is on this sub-process. Type in a name for this process and then press ENTER.
  4. A new editor opens to edit this sub-process.
You can now proceed and edit both the main process and sub-process.

Note: You can nest sub-processes within sub-processes.

Referencing Another Process

For modular development, you can use another process within your process as a reference.

To reference another process:

  1. Drag and drop first the process that you want to reference from the Project Explorer into the process editor. This process is added as a reference in the current process.
  2. Select the referenced process step in the process diagram, right-click and then select Show Properties View.
  3. In the Parameters section, set the action’s parameters. Each process has its own set of parameters.
  4. Press CTRL+S to save the process.

Execution Flow

Steps Features

At a step level, a number of features impact the execution flow. These include Conditional Generation, Step Repetition and Startup Steps.

Conditional Generation

It is possible to make conditional the generation of given steps in a process.
For example, you can generate only certain steps in the process depending on the startup parameters of the process

To make a step’s generation conditional:

  1. Select the step in the process diagram, right-click and then select Show Properties View.
  2. In the Generation section, optionally set the following options:
  3. Press CTRL+S to save the process.

Step Repetition

Repetition allows you to generate a step several times to run in parallel or sequence. This repetition can be performed in parallel or sequence, and is done using the result of an XPath Query.

To enable step repetition:

  1. Select the step in the process diagram, right-click and then select Show Properties View.
  2. Press CTRL+S to save the process.

Note: This repetition applies to the generation. It can be used for example to create one step for each table in a model.

Startup Steps

By default, all the steps with no incoming links are considered as startup steps. As a consequence, a process can start with several simultaneous steps.

In certain cases, (for example in the case of a process that is a loop) there is no step that can be identified as a startup step. In these cases, you must explicitly indicate the startup steps by checking the Is Begin Action option for these steps.

A link sequences two steps. There are three type of links to define the execution flow:

To add a link:

  1. In the Palette, select a Successfull Link, Unsuccessfull Link or Anyway Link tool in the Link accordion.
  2. Select a first step in the package diagram. While keeping the mouse button pressed, move the cursor onto another step and then release it. A link appears between the two steps.
  3. Right-Click this link and then select Show Properties View.
  4. In the link properties, set the following options:

At execution time:

  1. When the first steps completes, depending on its completion status, the Generation Type and the Generation Condition, the link is created or not.
  2. Then the link is triggered depending on the first step’s completion status, the Execution Type and the Execution Condition.
  3. Finally, the second step runs depending on all the incoming links' Triggering Behavior value. For example:

Note: A default link is always generated (Generation Type=Anyway), triggered only if the first step is successful (Execution type=Successful) and Mandatory for the second step to execute.

Note: Links with conditions appear as dotted links.

Advanced Process Development

Process Parameters

A parameter is used to parameterize a process' behavior. The value of this parameter is passed when starting the process or when using it as a referenced process in another one.

To create a process parameter:

  1. In the Palette, select the Parameter tool in the Component accordion.
  2. Click in the process diagram. A block representing your parameter appears in the diagram.
  3. Right-Click this block and then select Show Properties View.
  4. In the Properties views, set the following values in the Core section:
  5. Press CTRL+S to save the parameter.

To use a process parameter, refer to it as a variable in your code, using the ${<parameter_name>}$ syntax.

The direct bind link is a specific type of link that allows you to run an action once for each record returned by a database query or by a FileWait action.

With such a link:

For example, the first action performs the following SELECT query:

SELECT CUST_NAME, CUST_COUNTRY, CUST_CODE FROM CUSTOMER

The second action can generate a file with the following parameter

One file is written per customer returned by the select statement, named after the customer code (CUST_CODE).

Scripting

Scripting a feature helps you to customize the behavior of your processes.

You can use scripting in various locations in Semarchy Convergence for DI:

Scripting Language

It is possible to specify the scripting language using the %e(<language>){...}e(<language>)% syntax.
In addition to JavaScript (Rhino), Groovy and Python (Jython) are supported.

For example:

Using Session Variables

Semarchy Convergence for DI stores information about each action and process. This information is available as session variables.
These variables can be used in the code/text of an action, in its parameters, and in its metadata. Variables can also be used in scripts, for example in conditions.

The syntax to use a variable is ${variable_path}$ where variable_path is the path to the variable.

A variable belongs to an action or a process. This action or process may be contained in another process, and so on.
This organization is similar to a file system organization, with the main process at the root. You can access a variable with an absolute or relative path, depending on the location from which you request this variable.

For example if a process LOAD_DW contains a sub-process LOAD_DIMENSION which includes a sub-process LOAD_TIME which includes an action called WRITE_FILE, then the return code of this action is the following variable:

${LOAD_DW/LOAD_DIMENSION/LOAD_TIME/WRITE_FILE/CORE_RET_CODE}$

If you use this variable from the READ_FILE process within the LOAD_TIME sub-process, then the relative path is:

${../WRITE_FILE/CORE_RET_CODE}$

To use the return code of the current action, you can use:

${./CORE_RET_CODE}$

If the parent process' name is unknown, you can use the ~/ syntax:

${~/WRITE_FILE/CORE_RET_CODE}$

In the previous case, ~/ is replaced with LOAD_DW/LOAD_DIMENSION/LOAD_TIME/

Using Scripts in Conditions

Conditions may define whether:

A condition:

The default language used for the conditions is JavaScript (Rhino implementation) and the interpreter adds the %e(rhino){...}e(rhino)% around the code in conditions.

Example of a condition: ${./AUTOMATION/CORE_NB_EXECUTIONS}$==1. The CORE_NB_EXECUTION variable (number of executions) for the AUTOMATION step should be equal to 1 for this condition to be true.

When a script is more complex than a simple expression, the context variable __ctx__.retvalue can be used to return the Boolean value from the script to the condition, as shown below.

%e(rhino){
/* Retrieves a session variable value */
myVarValue = '${~/MYVARIABLE}$';
if (myVarValue.substring(0,1).equals('R')) {
/* Returns true */
	__ctx__.retValue = "true";
}
else {
/* Returns false */
	__ctx__.retValue = "false";
	}
}e(rhino)%

Note that this script also shows on link #3 the retrieval of the value for session variable MYVARIABLE.

Using the Scripting Context

When a script is interpreted, an object is passed to this script to provide access to the Runtime Engine features. This Context object is accessed using __ctx___.

This object provides a list of methods for manipulating variables and values as well as a return value used for code substitution and condition evaluation.
The following sections describe the various elements available with this object.

retValue Variable

The scripting context provides the retValue ( __ctx__.retValue) variable.

This String variable is used to:

publishVariable Method
public void publishVariable(String name, String value) {}
public void publishVariable(String name, String value, String type) {}

This method is used to publish a session variable, and takes the following parameters:

Example: Publish a string variable called INCREMENTAL_MODE with value ACTIVE on the parent element of the current action.

__ctx__.publishVariable("../INCREMENTAL_MODE","ACTIVE");
getCurrentBindIteration Method
public long getCurrentBindIteration() {}

This method returns the current bind iteration number. It takes no input parameter. See Bind Links for more information about bind iterations.

sumVariable, averageVariable, countVariable, minVariable and maxVariable Methods
public String sumVariable(String name) {}
public String sumVariable(String name, String startingPath) {}
public String averageVariable(String name) {}
public String averageVariable(String name, String startingPath) {}
public String countVariable(String name) {}
public String countVariable(String name, String startingPath) {}
public String minVariable(String name) {}
public String minVariable(String name, String startingPath) {}
public String maxVariable(String name) {}
public String maxVariable(String name, String startingPath) {}

These methods return the aggregated values for a variable. This aggregate is either a Sum, Average, Count, Min or Max.
The aggregate is performed within a given path.

This method takes the following parameters:

Example: Aggregate all the numbers of rows processed for LOAD_DIMENSION process and all its sub-processes/actions.

__ctx__.sumVariable("SQL_NB_ROWS","../LOAD_DIMENSION");
getVariableCumulativeValue Method
public Object getVariableCumulativeValue(String name) {}

When an action iterates due to a bind or a loop, the variables store their value for the last iteration.
In addition, numerical values contain a cumulated value which can be retrieved.

This method takes the following parameters:

getVariableTreeByName Method
public Map<String, IVariable> getVariableTreeByName(String name) {}
public Map<String, IVariable> getVariableTreeByName(String name, String startingPath) {}
public Map<String, IVariable> getVariableTreeByName(String name, boolean withErrors) {}
public Map<String, IVariable> getVariableTreeByName(String name, String startingPath, boolean withErrors)

This method returns a treeMap object containing the variables corresponding to certain criteria. It takes the following parameters:

The returned Java Map object has the name of the action as the key and its value is a variable object with the following methods.

public Object getCumulativeValue(); // variable cumulated value (numbers only)
public String getName(); // variable name.
public String getSource(); // variable source.
public String getType(); // variable type.
public String getValue(); // variable value.

Usage example in Groovy: Retrieve the stack trace for all the steps in error.

%e(groovy){
def a = ""
def tree = __ctx__.getVariableTreeByName("CORE_STACK_TRACE","~/",true)
if (tree.size() != 0) {
	def es=tree.entrySet()
	es.each{
	  	a = a+ "-- ACTION --> " + it.key + "\n"
	  	a = a+ it.value.getValue() +"\n\n"
	}
	__ctx__.retValue = a
}
}e(groovy)%

Same example in JavaScript (Rhino):

%e(rhino){
importPackage(java.util);
a = "";
tree = __ctx__.getVariableTreeByName("CORE_STACK_TRACE","~/",true);if (tree.size() != 0) {
	for (i= tree.keySet().iterator() ; i.hasNext() ; ){
	    action = i.next();
	    maVar = tree.get(action);
	  	a = a+ "-- ACTION --> " + action + "\n";
	  	a = a+ maVar.getValue() +"\n\n";
	}
	__ctx__.retValue = a
}
}e(rhino)%
getLstVariablesByName Method
public List<IVariable> getLstVariablesByName(String name) {}
public List<IVariable> getLstVariablesByName(String name, boolean withErrors) {}
public List<IVariable> getLstVariablesByName(String name, String startingPath) {}
public List<IVariable> getLstVariablesByName(String name, String startingPath, boolean withErrors) {}

This method works like getVariableTreeByName, but returns a list of variables instead of a Java Map.

Example:

%e(groovy){
def a = ""
def lst = __ctx__.getLstVariablesByName("V1","~/")
if (lst.size() != 0) {
	for (var in lst) {
  		a =a + var.getValue() + "\n"
	}
	__ctx__.retValue = a
}
}e(groovy)%
executeCommand and executeRemoteCommand Methods
public String executeCommand(String command) {}
public String executeRemoteCommand(String host, int port, String command) {}

The executeCommand method enables you to execute a Semarchy Convergence for DI command on the current runtime.
The executeRemoteCommand method enables you to execute a Semarchy Convergence for DI command on a remote runtime.

The Semarchy Convergence for DI commands are those available when you use the startCommand (.bat or .sh) shell program.

These two methods return the standard output produced by the command’s execution.

The parameters are :

Example :

%e(rhino){
__ctx__.executeCommand("versions");
}e(rhino)%

Link Metadata To Actions

It is possible to link metadata to actions and processes.
A link allows you to use metadata information in actions without having to hardcode it. Instead, references to the metadata information are used in the actions.
When linking metadata to action, these metadata references are converted to metadata information at generation time, and the action code remains generic. If a link is made to another metadata file, or if the metadata is modified, then the action will use the new metadata information.

Note: Metadata Linking is used frequently to use within processes the connection information and other structural information (table names, list of columns, etc.) stored in metadata files.

Linking Metadata to a Process/Action

To link metadata to an action:

  1. Open the process that you want to modify.
  2. In the Project Explorer view, select the metadata that you want to link. For example a data store or a data server.
  3. Drag and drop on the action in the diagram the metadata. It is added to the Metadata Link zone on the action element.
  4. Select this metadata link, right-click and then select Show Properties View.
  5. In the properties view, set the following parameters:
  6. Press CTRL+S to save the process.

Using the Metadata Connection

By linking actions (or processes containing actions) to a data server or schema defined in metadata file, you automatically use this data server or schema for the action. If the action required a SQL Connection, the SQL connection of the linked metadata is automatically used.

Using Metadata Specific Information

Through the metadata link, you can access specific information within the metadata. This information is accessed using a XPath query.
To use a value in a metadata link, in the code or any text field of the action, specify the XPath to the value.
Note that:

For example, a metadata link to a dataserver was named Local_XE. It is possible to use the following XPath to retrieve the JDBC URL of this data server in the action:

%x{ $Local_XE/tech:jdbcUrl() }x%

Working with Restart Points

When a process failed (with an error or a killed status), it can be restarted.

By defaut, it will restart from the steps that had an error status.

In a process you can also open and close " Restart Points" to define other ways to restart.

To open a restart point :

  1. Right-Click on the step in the process
  2. Choose “Restart Point” and "open"
  3. This will add the following icon :

To close a restart point :

  1. Right-Click on the step in the process
  2. Choose “Restart Point” and "close"
  3. This will add the following icon :

In case you have defined Restart Points, if the process fails without reaching the step on which the Restart Point is closed, the process will restart from the Restart Point above the failed step.

Note: You can have several Restart Points. If the process fails, it will restart, for each failed step, on the last Restart Point above the failed step.