Platform logging

Platform logging is used to monitor and troubleshoot issues in the Semarchy xDM platform when it is fully started.

For daily monitoring of job executions, use preferably the job logs and the job notifications.

View the logs

Important log events appear by default in the Semarchy Error Log. This error log shows events sent to the PDE appender.

You can access the error log:

  • In the Application Builder, select Error Log in the user menu.
    The Error Log view opens.

  • In Semarchy Configuration: Select View Logs in the navigation drawer.
    The Error Log editor opens.

Configure the logging

The default logging configuration of Semarchy works in most cases. However, you might want to change this configuration to troubleshoot complex issues.

Logging is configured through the Logging Configuration editor.

Starting from the 5.3.9 release, Semarchy xDM uses Apache Log4j 2 to log its activity. This section provides key concepts and configuration samples to work with Log4j. Refer to the Log4j Manual for reference information.

Instructions to upgrade the logging configuration from Log4j 1 to Log4j 2 are provided in the Upgrade Guide.

To configure the logging:

  1. In Semarchy Configuration, select Logging Configuration in the navigation drawer. The Logging Configuration editor opens.

  2. Change the configuration in the editor. See Understand the logging below for explanations about the configuration.

  3. Press Control+S (or Command+S on macOS) to save this editor.
    The new logging configuration is immediately taken into account.

You can reset the logging configuration to the default values by clicking the Restore Logging Configuration to Default button in the Logging Configuration editor toolbar.

Understand the logging

Log4j works with three key concepts: Appenders, Loggers, and Log Levels.

  • The Log Levels define at the same time the granularity and importance of log events.

  • The Loggers capture log events emitted by libraries and classes of the Semarchy xDM platform.

  • The Appenders deliver log events to a destination. For example, they write events to a file, a message queue, or the Semarchy Errors Log.

Log events are emitted by Semarchy xDM with a certain log level. They are captured by a logger and delivered to a destination via an appender.

Log levels

The Log Level defines the granularity and importance of a log event. The log level is one of the following: TRACE, DEBUG, INFO, WARN, ERROR, FATAL.

These values are ordered from highest to lowest granularity, but also in order of importance: The DEBUG level is very granular and most messages are not relevant in the normal course of operations, whereas the ERROR and FATAL messages will appear less frequently but are more important.

When you capture events, via a logger, at a certain level, all events of higher importance are also captured. For example, when you set a logger to capture INFO events, then the WARN, ERROR, and FATAL events are also captured.

When parameterizing a logger, it is possible to set the level to ALL to capture all log events, and to OFF to disable log capture.

INFO is typically a good level to start with, as it captures information messages as well as possible warnings and errors.

Appenders

Appenders deliver log events to a destination. Appenders are defined with a name and an appender class (that is, a Java class that writes log events to a file, to a JMS message queue, to an email, etc).

Appender classes may have one or more properties to configure their behavior. They also support Layout properties to define the format of the logged event.

Semarchy xDM provides a built-in com.semarchy.commons.log4j.appender.PDEAppender appender class to deliver logged events to the Semarchy Error Log.

Configure appenders

In the logging configuration, appenders are declared in the <Appenders> element.

An appender is defined with the following syntax:

<Appenders>
    <appender_class (1)
        name="appender_name" (2)
        ...
    </appender_class>

    ...
<Appenders>
1 The appender’s class (see Log4j 2 Appenders).
2 The name used by loggers to refer to this appender.

The parameters for each appender are defined using an XML structure composed of elements and attributes that depend on the appender’s class. Configuration samples for the most common appender classes are provided in the next section.

Configuration samples

This section provides configuration samples for appenders.

Example 1. FILE appender writing log events to a daily rolling log file.
<!-- Rolling file appender -->
<RollingFile (1)
  name="appender_name"
  fileName="${sys:user.home}/.semarchy/logfile.log" (2)
  filePattern="${sys:user.home}/.semarchy/logfile.%d{yyyy-MM-dd}.log">
  <PatternLayout pattern="%d{yyyy-MM-dd HH:mm s S}-%X{authenticatedUser}-%t-%-5p-%-10c:%m%n"/> (3)
  <Policies>
    <TimeBasedTriggeringPolicy modulate="true"/> (4)
  </Policies>
  <DefaultRolloverStrategy max="2147483647"/> (5)
</RollingFile>
1 The DailyRollingFileAppender appender class creates a daily logging file. You can also use the FileAppender to write to a simple file.
2 The daily log file is located in the .semarchy/ sub-directory of the user home directory. Its name is logfile.log to which the date, formatted with the pattern, is concatenated.
3 An appender uses a Layout to format log events. For a RollingFile appender, logged events are usually formatted using a PatternLayout pattern string, which supports the date, priority, etc, of the event to diagnose. This pattern also supports a specific %X{authenticatedUser} context key corresponding to the authenticated user operating in Semarchy xDM, if any.
4 The triggering policy determines if a rollover should be performed.
5 The RolloverStrategy defines how the rollover should be done. If no RolloverStrategy is configured, RollingFileAppender will use the DefaultRolloverStrategy.
Example 2. PDE appender sending log events to the Semarchy Error Log.
<!-- PDE Appender -->
<PDE name="appender_name" />
Example 3. SMTP appender sending log events by email
<!-- The SMTP class sends logs in emails via an SMTP host.         -->
<!-- This appender should be preferably used for important events. -->
<SMTP  (1)
        name="appender_name"
        smtpHost="smtp_host_name"  (2)
        smtpPort="smtp_host_port"  (2)
        from="email@my-domain.com" (2)
        to="admin1@my-domain.com,admin2@my-domain.com"  (3)
        subject="Log Event">                            (3)
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>  (4)
    </layout>
</SMTP>
1 This appender class sends log events by email.
2 Mail Server configuration. Note that additional properties may be set for the SMTP user, password, protocol, etc.
3 Email recipients and subject.
4 Log events are formatted using a Pattern Layout pattern string. If no layout is supplied, the HTML Layout will be used.

Loggers

Loggers are named and organized as a hierarchy of classes emitting log events. A logger captures events for all sub-classes in the hierarchy.

For example, the com.semarchy logger capture events for all the classes starting with com.semarchy, including com.semarchy.commons.sql and com.semarchy.mdm.

The highest logger in the hierarchy is named root logger.

Configure loggers

Loggers are declared in the <Loggers> element.

You configure loggers with a log level, and with one or more appenders. Any event emitted by the logger class that is more important than (or as important as) the given log level is captured by the logger and sent to the appender for delivery.

The loggers configuration is composed of:

  • The root logger.

  • A list of loggers.

Example 4. Loggers sample configuration
<Loggers>

    <!-- Root Logger -->
    <Root level="root_log_level">             (1)
        <AppenderRef ref="appender_name"/>    (2)
        ...
    </Root>

    <Logger level="log_level"  (3)
            name="logger_name">
        <AppenderRef ref="appender_name_1"/>  (4)
        <AppenderRef ref="appender_name_2"/>
        ...
    </Logger>
    ...
</Loggers>
1 Threshold level for the root logger. Accepted values are: TRACE, DEBUG, INFO, WARN, ERROR, and FATAL.
2 Appender to use for the root logger.
3 Threshold level for this logger.
4 Appender to use for this logger.
Example 5. Send DEBUG (and more important) events from the IPlatformManager class to the PDE appender
<Logger level="DEBUG" name="com.semarchy.platform.setup.IPlatformManager">
    <AppenderRef ref="PDE"/>
</Logger>

Logger inheritance

Loggers inherit their appenders additively from their parents in the hierarchy. In addition, they can override the level provided by their parent.

In the following configuration, the com.semarchy.platform.setup.IPlatformManager logger inherits the PDE appender from its parent com.semarchy logger and sends:

  • INFO events to the PDE appender

  • ERROR events to the PDE appender and the SMTP appender.

<Logger level="INFO"
        name="com.semarchy">
    <AppenderRef ref="PDE"/>
</Logger>
<Logger level="ERROR"
        name="com.semarchy.platform.setup.IPlatformManager">
    <AppenderRef ref="SMTP"/>
</Logger>

To prevent such inheritance, use the following syntax:

<Logger additivity="false"
        name="logger_name">
    ...
</Logger>

For example, in the following configuration, com.semarchy.platform.setup.IPlatformManager will no longer log into the PDE appender in addition to SMTP.

<Logger level="INFO"
        name="com.semarchy">
    <AppenderRef ref="PDE"/>
</Logger>
<Logger level="ERROR"
        additivity="false"
        name="com.semarchy.platform.setup.IPlatformManager">
    <AppenderRef ref="SMTP"/>
</Logger>

Configuration samples

The following configuration is the default one for Semarchy xDM.

Example 6. Sample logger configuration for Semarchy xDM.
<!-- External libs shutdown -->
<Logger level="ERROR" name="org.apache.directory.shared.asn1.ber.Asn1Decoder"/>
<Logger level="WARN" name="org.apache.commons.beanutils.converters"/>
<Logger level="WARN" name="org.quartz"/>
<Logger level="WARN" name="org.ops4j.pax.logging"/>
<Logger level="WARN" name="org.apache.cxf"/>
<Logger level="WARN" name="org.apache.cxf.services"/>
<Logger level="ERROR" name="org.apache.cxf.jaxrs.impl.WebApplicationExceptionMapper"/>
<Logger level="WARN" name="org.apache.cxf.services.WebClient"/>
<Logger level="WARN" name="org.apache.aries.blueprint"/>
<Logger level="OFF" name="org.apache.aries.blueprint.container.ServiceRecipe"/>
<Logger level="OFF" name="org.apache.aries.blueprint.container.BlueprintContainerImpl"/>
<Logger level="INFO" name="org.apache.jasper.servlet"/>
<Logger level="WARN" name="org.apache.aries.spifly"/>
<Logger level="INFO" name="com.sun.xml"/>
<Logger level="INFO" name="com.zaxxer.hikari"/>
<Logger level="INFO" name="javax.xml.bind"/>
<!-- Remove verbose WARN on Tika Parser see: https://issues.apache.org/jira/browse/TIKA-2518 -->
<Logger level="ERROR" name="org.apache.tika.parser.SQLite3Parser"/>
<Logger level="INFO" name="org"/>

<!-- Semarchy Loggers -->
<Logger level="INFO" name="com.semarchy"/>
<Logger level="INFO" name="com.semarchy.commons.sql"/>
<Logger level="WARN" name="org.springframework.jdbc.core.JdbcTemplate"/>
<Logger level="WARN" name="org.springframework.jdbc.core.StatementCreatorUtils"/>
<Logger level="WARN" name="org.apache.aries.blueprint.container">
    <AppenderRef ref="CONSOLE"/>
    <AppenderRef ref="PDE"/>
</Logger>
<Logger level="INFO" name="com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor"/>
<Logger level="INFO" name="com.semarchy.platform.engine"/>
<Logger level="INFO" name="com.semarchy.platform.setup.IPlatformManager"/>
<Logger level="INFO" name="com.semarchy.mdm.datahub.services.query.datamgr.IDataManager"/>
<!-- Plug-in Execution logging. Set to Debug to troubleshoot plugins. -->
<Logger level="INFO" name="com.semarchy.platform.engine.PluginExecution"/>
<Logger level="INFO" name="com.semarchy.platform.repository.domain.restclient.RestClientInstance"/>
<!-- Integration Batch Notification logger to be notified when notification errors occurs -->
<Logger level="INFO" name="com.semarchy.platform.product.notification.PlatformNotificationHandler"/>
<!-- Import / Export -->
<Logger level="INFO" name="com.semarchy.mdm.dataui.domain.dataimport"/>
<Logger level="INFO" name="com.semarchy.mdm.dataui.domain.dataexport"/>
<!-- Consistency Checker -->
<Logger level="INFO" name="com.semarchy.ui.platform.console.actions.CheckConsistencyAction">
    <AppenderRef ref="PDE"/>
</Logger>
<!-- Public REST API logger -->
<Logger level="INFO" name="com.semarchy.xdm.rest.accesslog"/>

Logger reference

The following table lists the various loggers available in Semarchy xDM:

Logger Name Description

com.semarchy

Root logger for the Semarchy platform.

com.semarchy.commons.sql

TRACE logs SQL queries made by the Application Builder and Configuration user interfaces.

com.semarchy.mdm.datahub.services.query.datamgr.IDataManager

Logs database operations performed by the platform for the MDM applications while browsing data. DEBUG logs all queries and execution times.

com.semarchy.platform.engine.core.impl.DefaultStandaloneEngineImpl

Logs execution engine events. DEBUG logs all submitted jobs, and ERROR can be used to troubleshoot engine issues.

com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor

Logs execution engine job processing. Use DEBUG to see execution engine details.

com.semarchy.platform.engine.PluginExecution

Logs enricher and validation plugins execution. DEBUG logs plugins feedback, TRACE logs the processing of every row.

com.semarchy.platform.integration.polling.IntegrationLoadDequeuer

Logs the Batch Poller activity. DEBUG logs every polled interval.

com.semarchy.platform.product.notification.JobNotificationHandler

Logs job notifications. ERROR traces notification failures.

com.semarchy.platform.setup.IPlatformManager

Logs platform status changes. INFO traces normal status changes. ERROR traces abnormal statuses.

org.apache.cxf.services

INFO traces REST requests, responses, and faults between the MDM applications and the server. This includes also REST requests performed by plugins on external services (Melissa Data, for example).

org.springframework.jdbc.core.JdbcTemplate

DEBUG traces the SQL queries made by the MDM applications and REST API calls as well as Dashboards-related SQL queries.

org.springframework.security.ldap

DEBUG traces the authentication attempts made with an LDAP or Active Directory identity provider.

org.springframework.security.oauth2
org.springframework.security.oauth2.jwt

Set oauth2 to DEBUG and oauth2.jwt to TRACE to log the authentication attempts made with an OpenID Connect identity provider.

waffle.spring

DEBUG traces the authentication attempts made with a Windows Authentication identity provider.

org.springframework.security.saml2
lorg.opensaml

DEBUG traces the authentication attempts made with a SAML identity provider.

com.semarchy.xdm.rest.accesslog

DEBUG traces REST API calls. This log is in JSON format and contains: username, API key name (or null for basic authentication), origin IP address, date and time, endpoint (relative to base server URL), response code and execution duration.

com.semarchy.mdm.discovery.core.impl.DefaultProfilingService

Logs the profiling activity for xDM Discovery. DEBUG traces the details of the profiling processes and queues.

com.semarchy.platform.repository.infrastructure.services.HttpClientRequestExecutor

DEBUG traces the REST requests and responses sent and received by Data Notifications.