Working with Processes | ||
---|---|---|
Previous | Next | |
Working with Mappings | Working with Variables |
Processes
define organized sequences of
Actions executed at run-time on the IT systems.
Processes are organized using
Sub-processes, which are themselves composed of actions, sub-processes or
References to other processes.
To create a new process:
To add a mapping to a process:
To add an action:
To add a sub-process:
Note: You can nest sub-processes within sub-processes.
For modular development, you can use another process within your process as a reference.
To reference another process:
At a step level, a number of features impact the execution flow. These include Conditional Generation, Step Repetition and Startup Steps.
It is possible to make conditional the generation of given steps in a process.
For example, you can generate only certain steps in the process depending on the startup parameters of the process
To make a step’s generation conditional:
Repetition allows you to generate a step several times to run in parallel or sequence. This repetition can be performed in parallel or sequence, and is done using the result of an XPath Query.
To enable step repetition:
Note: This repetition applies to the generation. It can be used for example to create one step for each table in a model.
By default, all the steps with no incoming links are considered as startup steps. As a consequence, a process can start with several simultaneous steps.
In certain cases, (for example in the case of a process that is a loop) there is no step that can be identified as a startup step. In these cases, you must explicitly indicate the startup steps by checking the Is Begin Action option for these steps.
A link sequences two steps. There are three type of links to define the execution flow:
To add a link:
At execution time:
Note: A default link is always generated (Generation Type=Anyway), triggered only if the first step is successful (Execution type=Successful) and Mandatory for the second step to execute.
Note: Links with conditions appear as dotted links.
A parameter is used to parameterize a process' behavior. The value of this parameter is passed when starting the process or when using it as a referenced process in another one.
To create a process parameter:
To use a process parameter, refer to it as a variable in your code, using the
${<parameter_name>}$
syntax.
The direct bind link is a specific type of link that allows you to run an action once for each record returned by a database query or by a FileWait action.
With such a link:
:{column_name}:
. where
column_name
is the name of one of the columns selected.
:{FILE_NAME}:
. FILE_NAME is a fixed variable name.
For example, the first action performs the following SELECT query:
SELECT CUST_NAME, CUST_COUNTRY, CUST_CODE FROM CUSTOMER
The second action can generate a file with the following parameter
/cust/docs/customer_:{CUST_CODE}:_content.txt
One file is written per customer returned by the select statement, named after the customer code (CUST_CODE).
Scripting a feature helps you to customize the behavior of your processes.
You can use scripting in various locations in Semarchy Convergence for DI:
%e(<language>){<script>}e(<language>)%
syntax. In this context, the script is interpreted and the result of the script execution (return code) replaces the script in the field.
It is possible to specify the scripting language using the
%e(<language>){...}e(<language>)%
syntax.
In addition to JavaScript (Rhino), Groovy and Python (Jython) are supported.
For example:
%e(rhino){...}e(rhino)%
%e(groovy){...}e(groovy)%
%e(jython){...}e(jython)%
Semarchy Convergence for DI stores information about each action and process. This information is available as session variables.
These variables can be used in the code/text of an action, in its parameters, and in its metadata. Variables can also be used in scripts, for example in conditions.
The syntax to use a variable is
${variable_path}$
where
variable_path is the path to the variable.
A variable belongs to an action or a process. This action or process may be contained in another process, and so on.
This organization is similar to a file system organization, with the main process at the root. You can access a variable with an absolute or relative path, depending on the location from which you request this variable.
For example if a process LOAD_DW contains a sub-process LOAD_DIMENSION which includes a sub-process LOAD_TIME which includes an action called WRITE_FILE, then the return code of this action is the following variable:
${LOAD_DW/LOAD_DIMENSION/LOAD_TIME/WRITE_FILE/CORE_RET_CODE}$
If you use this variable from the READ_FILE process within the LOAD_TIME sub-process, then the relative path is:
${../WRITE_FILE/CORE_RET_CODE}$
To use the return code of the current action, you can use:
${./CORE_RET_CODE}$
If the parent process' name is unknown, you can use the
~/
syntax:
${~/WRITE_FILE/CORE_RET_CODE}$
In the previous case,
~/
is replaced with
LOAD_DW/LOAD_DIMENSION/LOAD_TIME/
Conditions may define whether:
A condition:
${<variable>}$
syntax.
The default language used for the conditions is JavaScript (Rhino implementation) and the interpreter adds the
%e(rhino){...}e(rhino)%
around the code in conditions.
Example of a condition:
${./AUTOMATION/CORE_NB_EXECUTIONS}$==1
. The
CORE_NB_EXECUTION
variable (number of executions) for the
AUTOMATION step should be equal to 1 for this condition to be true.
When a script is more complex than a simple expression, the context variable
__ctx__.retvalue
can be used to return the Boolean value from the script to the condition, as shown below.
%e(rhino){
/* Retrieves a session variable value */
myVarValue = '${~/MYVARIABLE}$';
if (myVarValue.substring(0,1).equals('R')) {
/* Returns true */
__ctx__.retValue = "true";
}
else {
/* Returns false */
__ctx__.retValue = "false";
}
}e(rhino)%
Note that this script also shows on link #3 the retrieval of the value for session variable
MYVARIABLE
.
When a script is interpreted, an object is passed to this script to provide access to the Runtime Engine features. This
Context object is accessed using
__ctx___
.
This object provides a list of methods for manipulating variables and values as well as a return value used for code substitution and condition evaluation.
The following sections describe the various elements available with this object.
The scripting context provides the retValue (
__ctx__.retValue
) variable.
This String variable is used to:
public void publishVariable(String name, String value) {}
public void publishVariable(String name, String value, String type) {}
This method is used to publish a session variable, and takes the following parameters:
Example: Publish a string variable called INCREMENTAL_MODE with value ACTIVE on the parent element of the current action.
__ctx__.publishVariable("../INCREMENTAL_MODE","ACTIVE");
public long getCurrentBindIteration() {}
This method returns the current bind iteration number. It takes no input parameter. See Bind Links for more information about bind iterations.
public String sumVariable(String name) {}
public String sumVariable(String name, String startingPath) {}
public String averageVariable(String name) {}
public String averageVariable(String name, String startingPath) {}
public String countVariable(String name) {}
public String countVariable(String name, String startingPath) {}
public String minVariable(String name) {}
public String minVariable(String name, String startingPath) {}
public String maxVariable(String name) {}
public String maxVariable(String name, String startingPath) {}
These methods return the aggregated values for a variable. This aggregate is either a Sum, Average, Count, Min or Max.
The aggregate is performed within a given path.
This method takes the following parameters:
Example: Aggregate all the numbers of rows processed for LOAD_DIMENSION process and all its sub-processes/actions.
__ctx__.sumVariable("SQL_NB_ROWS","../LOAD_DIMENSION");
public Object getVariableCumulativeValue(String name) {}
When an action iterates due to a bind or a loop, the variables store their value for the last iteration.
In addition, numerical values contain a cumulated value which can be retrieved.
This method takes the following parameters:
public Map<String, IVariable> getVariableTreeByName(String name) {}
public Map<String, IVariable> getVariableTreeByName(String name, String startingPath) {}
public Map<String, IVariable> getVariableTreeByName(String name, boolean withErrors) {}
public Map<String, IVariable> getVariableTreeByName(String name, String startingPath, boolean withErrors)
This method returns a treeMap object containing the variables corresponding to certain criteria. It takes the following parameters:
~/
)
The returned Java Map object has the name of the action as the key and its value is a variable object with the following methods.
public Object getCumulativeValue(); // variable cumulated value (numbers only)
public String getName(); // variable name.
public String getSource(); // variable source.
public String getType(); // variable type.
public String getValue(); // variable value.
Usage example in Groovy: Retrieve the stack trace for all the steps in error.
%e(groovy){
def a = ""
def tree = __ctx__.getVariableTreeByName("CORE_STACK_TRACE","~/",true)
if (tree.size() != 0) {
def es=tree.entrySet()
es.each{
a = a+ "-- ACTION --> " + it.key + "\n"
a = a+ it.value.getValue() +"\n\n"
}
__ctx__.retValue = a
}
}e(groovy)%
Same example in JavaScript (Rhino):
%e(rhino){
importPackage(java.util);
a = "";
tree = __ctx__.getVariableTreeByName("CORE_STACK_TRACE","~/",true);if (tree.size() != 0) {
for (i= tree.keySet().iterator() ; i.hasNext() ; ){
action = i.next();
maVar = tree.get(action);
a = a+ "-- ACTION --> " + action + "\n";
a = a+ maVar.getValue() +"\n\n";
}
__ctx__.retValue = a
}
}e(rhino)%
public List<IVariable> getLstVariablesByName(String name) {}
public List<IVariable> getLstVariablesByName(String name, boolean withErrors) {}
public List<IVariable> getLstVariablesByName(String name, String startingPath) {}
public List<IVariable> getLstVariablesByName(String name, String startingPath, boolean withErrors) {}
This method works like getVariableTreeByName, but returns a list of variables instead of a Java Map.
Example:
%e(groovy){
def a = ""
def lst = __ctx__.getLstVariablesByName("V1","~/")
if (lst.size() != 0) {
for (var in lst) {
a =a + var.getValue() + "\n"
}
__ctx__.retValue = a
}
}e(groovy)%
public String executeCommand(String command) {}
public String executeRemoteCommand(String host, int port, String command) {}
The
executeCommand
method enables you to execute a Semarchy Convergence for DI command on the current runtime.
The
executeRemoteCommand
method enables you to execute a Semarchy Convergence for DI command on a remote runtime.
The Semarchy Convergence for DI commands are those available when you use the startCommand (.bat or .sh) shell program.
These two methods return the standard output produced by the command’s execution.
The parameters are :
Example :
%e(rhino){
__ctx__.executeCommand("versions");
}e(rhino)%
It is possible to link metadata to actions and processes.
A link allows you to use metadata
information in actions without having to hardcode it. Instead,
references to the metadata information are used in the actions.
When linking metadata to action, these metadata references are converted to metadata information at generation time, and the action code remains generic. If a link is made to another metadata file, or if the metadata is modified, then the action will use the new metadata information.
Note: Metadata Linking is used frequently to use within processes the connection information and other structural information (table names, list of columns, etc.) stored in metadata files.
To link metadata to an action:
By linking actions (or processes containing actions) to a data server or schema defined in metadata file, you automatically use this data server or schema for the action. If the action required a SQL Connection, the SQL connection of the linked metadata is automatically used.
Through the metadata link, you can access specific information within the metadata. This information is accessed using a XPath query.
To use a value in a metadata link, in the code or any text field of the action, specify the XPath to the value.
Note that:
%x{...}x%
in order to be interpreted accordingly and replaced by its value at run-time.
For example, a metadata link to a dataserver was named
Local_XE
. It is possible to use the following XPath to retrieve the JDBC URL of this data server in the action:
%x{ $Local_XE/tech:jdbcUrl() }x%
When a process failed (with an error or a killed status), it can be restarted.
By defaut, it will restart from the steps that had an error status.
In a process you can also open and close " Restart Points" to define other ways to restart.
To open a restart point :
To close a restart point :
In case you have defined Restart Points, if the process fails without reaching the step on which the Restart Point is closed, the process will restart from the Restart Point above the failed step.
Note: You can have several Restart Points. If the process fails, it will restart, for each failed step, on the last Restart Point above the failed step.
Previous | Top | Next |
Working with Mappings | Working with Variables |