Configure Runtime logging

When using Semarchy xDI Runtime, you can log two kinds of information:

Session information

Data and messages that a runtime generates when executing deliveries.

Debug information

Supplementary technical data and messages that a runtime generates at any time while it is running.

A runtime stores these logs in two places.

First, a runtime always stores session information to a log database, which you can monitor from xDI Designer, xDI Analytics, or the command line. You can change which database to use. The default log database is an HSQL database built into xDI Runtime which provides a ready-to-use configuration.

xDI Runtime also uses the Log4j library to store both session and debug information to multiple locations. Configure Log4j settings to change storage locations, as well as how much information you want to log. You can monitor these logs with external monitoring tools

The built-in HSQL database is, by default, only accessible from the runtime. You should not use it for production use. Use a certified database server instead.

Configure session logging to a database

The main session logging settings are in the main xDI Runtime configuration file engineParameters.xml. This file is in the properties subdirectory of the Runtime installation directory by default.

The main configuration sections are in a <parameters> element, inside the engineParameters and logs nodes.

Here is an overview of how you configure runtime session logging to a database:

  1. Open the engineParameters.xml configuration file.

  2. Under <parameters>, find the engineParameters and the logs elements.

  3. Configure the initial logging engine parameters.

  4. Set the log database parameters.

  5. Restart the runtime.

Configure initial logging engine parameters

In the engineParameters node of the engineParameters.xml file, set two parameters to configure logging:

Refer to the runtime parameters reference for detailed information parameters.

Log levels

The defaultSessionLogLevel parameter defines how much information the runtime saves in a session. It applies only if the session does not set its log level elsewhere. Possible values are as follows:

Table 1. Logged information by level
logLevel Running session Successful session Session with errors

-1

Not logged

Not logged

Not logged

0

Full details

Not logged

Full details

100

Full details

Session header and statistics

Full details

200

Full details

[logLevel 100] + Process information and statistics

Full details

300

Full details

[logLevel 200] + Actions information and statistics

Full details

400

Full details

Full details

Full details

Example 1. Sample logging configuration
  <engineParameters>
    <...>
    <parameter name="userLogDefaultName" value="logDatabase"/>
    <parameter name="defaultSessionLogLevel" value="400"/>
    <...>
  </engineParameters>

Set an external logging engine database

If you want to use the default HSQL database instead of an external database, see the section about HSQL.

A log database configuration contains the connection information to a database that hosts the log. It must be inside the <logs> element, in its own <log> node.

To set a log database used by the runtime, follow these steps:

  1. Copy a <log> element and its contents, or find a <log> entry to modify.

  2. Set the userLogName attribute of the <log> node to match the initial userLogDefaultName engine parameter.

  3. Set the JDBC URL and driver in userLogRdbmsUrl and userLogRdbmsDriver parameters.

  4. Set the database username and password in userLogRdbmsUser and userLogRdbmsPassword parameters.

  5. Set the userLogSchemaName parameter to the database schema you want to use.

  6. Set the userLogRdbmsModule parameter to the name of an xDI module containing the required JDBC database drivers.

You do not need to configure a database schema ahead of time to prepare it for logging. The runtime creates the necessary tables the first time it connects to the database.

Example configuration for an external database

The following configuration example uses a log database named oracleLogDatabase using an Oracle storage.

Adapt the following parameters to your database.

  • userLogRdbmsUrl

  • userLogRdbmsUser

  • userLogRdbmsPassword

  • userLogSchemaName

  • userLogRdbmsModule

Refer to the runtime parameters reference for detailed information about each parameter.
<parameters>
  <...>
  <engineParameters>
  <...>
    <parameter name="userLogDefaultName"
      value="oracleLogDatabase"/>
    <parameter name="defaultSessionLogLevel"
      value="400"/>
    <...>
  </engineParameters>
  <...>
  <logs>
    <...>
    <log userLogName="oracleLogDatabase"
      autoUpdate="true"
      userLogClass="com.semarchy.xdi.runtime.sessionlog.RdbmsLogger">
      <parameter name="userLogRdbmsDriver"
        value="oracle.jdbc.driver.OracleDriver"/>
      <parameter name="userLogRdbmsUrl"
        value="jdbc:oracle:thin:@[host]:[port]:[sid]"/>
      <parameter name="userLogRdbmsUser"
        value="[USER]"/>
      <parameter name="userLogRdbmsPassword"
        value="[PASSWORD]"/>
      <parameter name="userLogRdbmsModule"
        value="[MODULE_NAME]"/>
      <parameter name="userLogRdbmsSchemaName"
        value="[SCHEMA_NAME]"/>
      <parameter name="userLogRdbmsVarcharType"
        value="varchar2"/>
      <parameter name="userLogRdbmsVarcharMaxSize"
        value="4000"/>
      <parameter name="userLogRdbmsClobType"
        value="clob"/>
      <parameter name="userLogRdbmsBlobType"
        value="blob"/>
      <parameter name="userLogRdbmsNumericType"
        value="number"/>
      <parameter name="userLogRdbmsDeleteSyntaxe"
        value="Delete from"/>
      <parameter name="userLogRdbmsDeliveryFormat"
        value="text"/>
      <parameter name="userLogRdbmsPropertyMaxVarcharSize"
        value="1000"/>
      <parameter name="userLogRdbmsPropertyMaxClobSize"
        value="10000"/>
      <parameter name="userLogRdbmsPropertyBinaryFormat"
        value="compressed"/>
      <parameter name="userLogRdbmsPoolEnabled"
        value="true"/>
      <parameter name="userLogRdbmsPoolConnectionTimeout"
        value="30000"/>
      <parameter name="userLogRdbmsPoolIdleTimeout"
        value="600000"/>
      <parameter name="userLogRdbmsPoolKeepAliveTime"
        value="0"/>
      <parameter name="userLogRdbmsPoolMaxLifetime"
        value="1800000"/>
      <parameter name="userLogRdbmsPoolMinimumIdle"
        value="0"/>
      <parameter name="userLogRdbmsPoolMaximumSize"
        value="20"/>
      <parameter name="userLogRdbmsPoolValidationTimeout"
        value="5000"/>
    </log>
    <...>
  </logs>
  <...>
</parameters>

The default engineParameters.xml installed with the runtime contains commented log database configuration samples. Uncomment and adapt these samples to configure to your log storage. Make sure to set the userLogDefaultName engine parameter to the appropriate log database configuration.

Set or configure the built-in HSQL database

If you want to use an external database instead of the default HSQL database, see the section about external databases.

The runtime ships a built-in HSQL quick start database. The default configuration uses this database to store logs for delivery execution sessions and the scheduler.

The built-in HSQL database is intended for development environments. It is only accessible from the runtime itself by default. You should use a certified database server where possible.

On runtime versions prior to 2023.4.0, the built-in database was an H2 database. See Migrate the built-in database from H2 to HSQL to migrate your data to HSQL.

Set the HSQL directory

In its default configuration, the runtime creates an HSQL database inside a sessions directory to hold logging information. The runtime tries to create the directory if it does not exist already. If an HSQL database already exists, make sure that the user and password in the configuration file match the ones expected by the database. This is most important when upgrading from a previous runtime version.

The directory name and location are set by the userLogRdbmsUrl engine parameter. The following example is from a runtime that writes to the default sessions/internalDb/sessionLogs location:

<parameter name="userLogRdbmsUrl" value="jdbc:hsqldb:file:internalDb/sessionLogs/sessionLogs"/>

The parameter can take a different file path, including absolute filesystem paths. Make sure that the directories you specify have write permissions enabled.

Open external access

The built-in HSQL database is only accessible from the runtime itself by default. The runtime logs to the database in-memory, and does not expose the database as an external service.

To open access to the built-in HSQL database from the outside, you must manually start it as a server. This can be useful to access its logs from xDI Analytics, for example. To do this:

  1. Stop the runtime.

  2. Update the runtime configuration

    1. Log Database:

      1. Change the userLogRdbmsUrl parameter value from jdbc:hsqldb:file:internalDb/sessionLogs/sessionLogs to jdbc:hsqldb:tcp://localhost:42100/sessions/internalDb/sessionLogs

    2. Scheduler:

      1. Change the org.quartz.dataSource.internal.URL parameter value from jdbc:hsqldb:file:internalDb/scheduler/scheduler to jdbc:hsql:tcp://localhost:42100/internalDb/scheduler/scheduler

  3. Start the built-in HSQL database as a server.

    1. According to your operating system, run the start-database.bat or start-database.sh from the runtime’s installation root folder.

  4. Start the runtime.

After this change, you must always start the HSQL database server before starting the runtime itself.

Disable session logging

You can disable delivery session logging in the runtime in two ways:

  • Explicitly define a special, non-logging database entry using the EmptyLogger class as shown in the further example. Use this method first.

  • Remove or comment out all log database configurations under the <logs> node. This stops the runtime from logging delivery sessions.

Example 2. Configure the runtime for no logging
<parameters>
  <...>
  <engineParameters>
    <...>
    <parameter name="userLogDefaultName" value="noLogs"/>
    <...>
  </engineParameters>
  <...>
  <logs>
    <log
    userLogName="noLogs"
    autoUpdate="false"
    userLogClass="com.semarchy.xdi.runtime.sessionlog.EmptyLogger">
    </log>
  </logs>
  <...>
</parameters>

Schedule log purges

You can schedule automatic purging of logs in Semarchy xDI Analytics, in the Purge tab of the Runtime Editor. You can also use the command line:

  1. Run the startCommand.bat|sh script.

  2. Run the schedule purge command described as follows:

schedule purge keep <number> <minute|hour|day|session>  cron <cronExpression> [sessionname <name,name2,...>] [status <done,error,killed>] [on host <hostname>] [port <hostport>]

For example:

schedule purge keep 90 day cron "0 0 23 * * ?"

Configure session and debug logging with Log4j

When xDI Runtime saves logs using Log4j, it reads settings from the XML file log4j.xml. This file is in the properties subdirectory of the Runtime installation directory by default.

The configuration file starts with a Configuration element. Inside it, you can find Appenders and Loggers.

Appenders send logging information to its final destination, like a log file or a console. Loggers set which elements log4j records, and need at least one reference to an appender.

Example 3. Example configuration
<Configuration shutdownHook="disable">
  <Appenders>
    <RollingFile fileName="log/xdi-runtime.log" filePattern="log/xdi-runtime.log.%i" immediateFlush="true" name="runtimeAppender">
          <PatternLayout>
              <Pattern>%d{dd/MM/yyyy HH:mm:ss,SSS} [%t] %c{10} - %-5p - %m%n%throwable</Pattern>
          </PatternLayout>
          <SizeBasedTriggeringPolicy size="20 MB"/>
          <DefaultRolloverStrategy fileIndex="min" max="6"/>
    </RollingFile>
  </Appenders>
  <Logger name="com.indy.engine.user.console" level="all">
    <AppenderRef ref="consoleAppender"/>
  </Logger>
  <Logger name="com.semarchy.xdi.runtime.httpserver" level="warn">
    <AppenderRef ref="consoleAppender"/>
    <AppenderRef ref="runtimeAppender"/>
  </Logger>
</Configuration>

Aside from the loggers built into the Log4j library, xDI Runtime implements some of its own, among which:

  • Session start

  • Session end

  • Action start

  • Action end

  • Java compiler

  • Module service

  • HTTP server

The default configuration disables some of these loggers. You can re-enable them.

Change the log location

By default, the Log4j library writes to the xdi-runtime.log file in a log subdirectory of the runtime installation path. If this subdirectory does not exist, the runtime tries to create it. The parameters that set the directory are in the log4j.xml file, as attributes of the RollingFile appender:

<RollingFile fileName="log/xdi-runtime.log" filePattern="log/xdi-runtime.log.%i" immediateFlush="true" name="runtimeAppender">

The attributes can take a different folder and file path, including absolute filesystem paths. If the path points to external directories, make sure those directories have write permissions enabled.

Logs can also be sent to other places, such as the runtime console using the Console appender.

Set appender patterns

When writing to disk, appenders determine how to write the log to disk from a Layout. One such layout is the PatternLayout, which sets string patterns to enrich log messages with additional information. The pattern strings have their own internal syntax and pattern parameters.

xDI has pattern parameters of its own. To use them, add them with the syntax %X{parameterName}.

You can also use %X on its own to get all the parameters.

Example 4. Example pattern for file appender
<PatternLayout>
     <Pattern>%d{dd/MM/yyyy HH:mm:ss,SSS} [%t] %c{10} - %-5p - %m%n%throwable</Pattern>
</PatternLayout>

List of xDI patterns

Table 2. All events
Parameter Name Description

version

Version of the logging system

type

Type of event

message

event message

sessionId

Id of the session

sessionIteration

Iteration of the session

sessionName

Name of the session

Table 3. Session start
Parameter Name Description

sessionStatus

Status of the session (always RUNNING for this event)

sessionParentId

Id of the parent session

sessionBeginDate

Start date of the session

processId

Id of the process

processName

Name of the process

deliveryId

Id of the delivery

deliveryName

Name of the delivery

deliveryBuildDate

Build date of the delivery

deliveryBuildUser

Build user of the delivery

configuration

Configuration used for the execution

runtimeHost

Hostname of the runtime used for the execution

runtimePort

Port of the runtime used for the execution

executionMode

Mode of execution (MEMORY/AUTONOMOUS)

launchMode

Launch mode (RESTART/COMMAND_LINE/SCHEDULE/DESIGNER/ACTION/WEB_INTERACTIVE/WEB_SERVICE/ENGINE_COMMAND)

sessionLogLevel

Log level of the session

maxparallelExecutions

Max parallel executions setting

autoRestartInterval

Auto restart interval setting

autoRestartMaxTries

Auto restart max tries setting

launchUserName

Name of the user who launched the session

Table 4. Session end
Parameter Name Description

sessionStatus

Status of the session

sessionErrorMessage

Error message of the session

sessionEndDate

End date of the session

sessionDuration

Total duration of the session

var.<varName>

Variables

stat.<aggregateName>

Statistics

Table 5. All action events
Parameter Name Description

actionId

ID of the action

actionIteration

Iteration of the action

actionType

Type of the action (Code/Process)

actionPath

Path of the action

actionPathWithIteration

Unique identifier of the action (<path>_<iteration>)

actionStatus

Status of the action (RUNNING for the Action Start event)

Table 6. Action start
Parameter Name Description

actionBeginDate

Start date of the action

actionParentID

ID of the parent session

actionParentIteration

Iteration of the parent session

actionIsBegin

Is the action begin`

actionNumber

Unique number of the action

Table 7. Action end
Parameter Name Description

actionErrorMessage

Error message of the Action

[NOTE] This is not for errors that occur during action executions. It is for all other action errors, such as if actions did not generate.

actionEndDate

End date of the action

actionDuration

Total duration of the action

actionBindIteration

Final bind iteration of the action

actionExecutionCount

Execution count of the action (when used in a loop)

var.<varName>

Variables

stat.<aggregateName>

Statistics

For more on patterns and layouts, see the official Log4j documentation.

Further information