Platform Logging

Platform logging is used to monitor and troubleshoot issues in the Semarchy xDM platform when it is fully started.

For daily monitoring of job executions, use preferably the job logs and the job notifications.

View the Logs

Important log events appear by default in the Semarchy Error Log. This error log shows events sent to the PDE appender.

You can access the error log:

  • In Semarchy Application Builder: In the user menu (upper-left corner), select Error Log.
    The Error Log view opens.

  • In Semarchy Configuration: Select View Logs in the navigation drawer.
    The Error Log editor opens.

Configure the Logging

The default logging configuration of Semarchy works in most cases. However, you might want to change this configuration to troubleshoot complex issues.

Logging is configured through the Logging Configuration editor.

To configure the logging:

  1. In Semarchy Configuration, select Logging Configuration in the navigation drawer. The Logging Configuration editor opens.

  2. Change the configuration in the editor. See [Understand Logging] below for explanations about the configuration.

  3. Press CTRL-S to save this editor.
    The new logging configuration is immediately taken into account.

You can reset the logging configuration to the default values by clicking the Restore Logging Configuration to Default button in the Logging Configuration editor toolbar.

Understand the Logging

Semarchy xDM uses Apache Log4J to log its activity. This section provides key concepts and configuration samples to work with Log4J. Refer to the Log4J Manual for reference information.

Log4J works with three key concepts: Appenders, Loggers, and Log Levels.

  • The Log Levels define at the same time the granularity and importance of log events.

  • The Loggers capture log events emitted by libraries and classes of the Semarchy xDM platform.

  • The Appenders deliver log events to a destination. For example, they write events to a file, a message queue, or the Semarchy Errors Log.

Log events are emitted by Semarchy xDM with a certain log level. They are captured by a logger and delivered to a destination via an appender.

Log Levels

The Log Level defines the granularity and importance of a log event. The log level is one of the following: TRACE, DEBUG, INFO, WARN, ERROR, FATAL.

These values are ordered from highest to lowest granularity, but also in order of importance: The DEBUG level is very granular and most messages are not relevant in the normal course of operations, whereas the ERROR and FATAL messages will appear less frequently but are more important.

When you capture events, via a logger, at a certain level, all events of higher importance are also captured. For example, when you set a logger to capture INFO events, then the WARN, ERROR, and FATAL events are also captured.

When parameterizing a logger, it is possible to set the level to ALL to capture all log events, and to OFF to disable log capture.

INFO is typically a good level to start with, as it captures information messages as well as possible warnings and errors.

Appenders

Appenders deliver log events to a destination. Appenders are defined with a name and an appender class (that is, a Java class that writes log events to a file, to a JMS message queue, to an email, etc).

Appender classes may have one or more properties to configure their behavior. They also support Layout properties to define the format of the logged event.

Semarchy xDM provides a built-in com.semarchy.commons.log4j.appender.pde.PDEAppender appender class to deliver logged events to the Semarchy Error Log.

Configure Appenders

In the logging configuration, an appender is defined with the following syntax:

log4J.appender.<appender_name>=<appender_class>

Properties for this appender are defined with the following syntax:

log4J.appender.<appender_name>.<property_name>=<property_value>

Configuration Samples

This section provides configuration samples for appenders.

Example 1. FILE appender writing log events to a daily rolling log file.
# Appender name and class
log4j.appender.FILE=org.apache.log4j.DailyRollingFileAppender   (1)

# Appender properties
log4j.appender.FILE.DatePattern='.'yyyy-MM-dd'.log'     (2)
log4j.appender.FILE.file=${user.home}/.semarchy/logfile.log     (2)

log4j.appender.FILE.append=true                 (3)
log4j.appender.FILE.encoding=UTF-8              (3)

# Appender layout configuration
log4j.appender.FILE.layout=org.apache.log4j.PatternLayout   (4)
log4j.appender.FILE.layout.ConversionPattern=%d{yyyy-MM-dd HH\:mm s S}-%X{authenticatedUser}-%t-%x-%-5p-%-10c\:%m%n (4)
1 The DailyRollingFileAppender appender class creates a daily logging file. You can also use the FileAppender to write to a simple file.
2 The daily log file is located in the .semarchy/ sub-directory of the user home directory. Its name is logfile.log to which the date, formatted with the pattern, is concatenated.
3 These properties configure the appender behavior: Add log events at the end of the daily file, and use UTF-8 encoding.
4 Logged events are formatted using a PatternLayout pattern string, which supports the date, priority, etc, of the event to diagnose. This pattern also supports a specific %X{authenticatedUser} context key corresponding to the authenticated user operating in Semarchy xDM, if any.
Example 2. PDE appender sending log events to the Semarchy Error Log.
# Appender name and class
# The PDEAppender class sends logs to the {mdm-short-product-name} Error Log.
log4j.appender.PDE=com.semarchy.commons.log4j.appender.pde.PDEAppender  (1)

# Appender properties
log4j.appender.PDE.Threshold=INFO                   (2)
1 This appender class logs events into the Semarchy Error Log.
2 Only log events more important than this threshold will be considered by the appender. Make sure to align this value with the loggers that send events to this appender.
Example 3. SMTP appender sending log events by email
# Appender name and class
# The PDEAppender class sends logs in emails via an SMTP host.
# This appender should be preferably used for important events.
log4j.appender.SMTP = org.apache.log4j.net.SMTPAppender         (1)

# Appender properties
log4j.appender.SMTP.SMTPHost = <smtp_host_name>             (2)
log4j.appender.SMTP.SMTPPort = <smtp_host_port>             (2)
log4j.appender.SMTP.From = email@my-domain.com              (2)
log4j.appender.SMTP.To = admin1@my-domain.com,admin2@my-domain.com  (3)
log4j.appender.SMTP.Subject = Log Event                 (3)
log4j.appender.SMTP.layout =  org.apache.log4j.TTCCLayout       (4)
1 This appender class sends log events by email.
2 Mail Server configuration. Note that additional properties may be set for the SMTP user, password, protocol, etc.
3 Email recipients and subject.
4 Log events are formatted using the TTCC Layout, a detailed format containing the time, thread, category, and nested diagnostic context.

Loggers

Loggers are named and organized as a hierarchy of classes emitting log events. A logger captures events for all sub-classes in the hierarchy.

For example, the com.semarchy logger capture events for all the classes starting with com.semarchy, including com.semarchy.commons.sql and com.semarchy.mdm.

The highest logger in the hierarchy is named rootLogger.

Configure Loggers

You configure loggers with a log level, and with one or more appenders. Any event emitted by the logger class that is more important than (or as important as) the given log level is captured by the logger and sent to the appender for delivery.

Use the following syntax to attach a logger to appenders with a given log level:

log4j.logger.<loggerName>=<log_level>, <appender_name> [,<appender_name>, …]
Example 4. Send DEBUG (and more important) events from the IPlatformManager class to the PDE appender
log4j.logger.com.semarchy.platform.setup.IPlatformManager=DEBUG, PDE

Logger Inheritance

Loggers inherit their appenders additively from their parents in the hierarchy. In addition, they can override the level provided by their parent.

In the following configuration, the com.semarchy.platform.setup.IPlatformManager logger inherits from the PDE appender of the com.semarchy logger and sends:

  • INFO events to the PDE appender

  • ERROR events to the PDE appender and the SMTP appender.

log4j.logger.com.semarchy=INFO, PDE
log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP

To prevent such inheritance, use the following syntax:

log4j.additivity.<logger_name> = false

For example, in the the following configuration, com.semarchy.platform.setup.IPlatformManager will no longer log into the PDE appender in addition to SMTP.

log4j.logger.com.semarchy=INFO, PDE
log4j.logger.com.semarchy.platform.setup.IPlatformManager=ERROR, SMTP
log4j.additivity.com.semarchy.platform.setup.IPlatformManager=false

Configuration Samples

The following configuration is the default one for Semarchy xDM.

Example 5. Sample logger configuration for Semarchy xDM.
log4j.logger.com.semarchy=INFO
log4j.logger.com.semarchy.commons.sql=INFO
log4j.logger.com.semarchy.mdm.datahub.services.query.datamgr.IDataManager=INFO
log4j.logger.com.semarchy.mdm.dataui.domain.dataexport=INFO
log4j.logger.com.semarchy.mdm.dataui.domain.dataimport=INFO
log4j.logger.com.semarchy.platform.engine=INFO
log4j.logger.com.semarchy.platform.engine.PluginExecution=INFO
log4j.logger.com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor=INFO
log4j.logger.com.semarchy.platform.product.notification.PlatformNotificationHandler=INFO
log4j.logger.com.semarchy.platform.repository.domain.restclient.RestClientInstance=INFO
log4j.logger.com.semarchy.platform.setup.IPlatformManager=INFO
log4j.logger.com.semarchy.ui.platform.console.actions.CheckConsistencyAction=INFO, PDE
log4j.logger.com.semarchy.xdm.rest.accesslog=INFO
log4j.logger.org=INFO
log4j.logger.org.apache.aries.blueprint=WARN
log4j.logger.org.apache.aries.blueprint.container=INFO, CONSOLE, PDE
log4j.logger.org.apache.commons.beanutils.converters=WARN
log4j.logger.org.apache.cxf=WARN
log4j.logger.org.apache.cxf.jaxrs.impl.WebApplicationExceptionMapper=ERROR
log4j.logger.org.apache.cxf.services=WARN
log4j.logger.org.apache.cxf.services.WebClient=WARN
log4j.logger.org.apache.directory.shared.asn1.ber.Asn1Decoder=ERROR
log4j.logger.org.ops4j.pax.logging=WARN
log4j.logger.org.quartz=WARN
log4j.logger.org.springframework.jdbc.core.JdbcTemplate=INFO
log4j.logger.org.springframework.jdbc.core.StatementCreatorUtils=INFO
log4j.rootLogger=INFO, PDE

Logger Reference

The following table lists the various loggers available in Semarchy xDM:

Logger Name Description

com.semarchy

Root logger for the Semarchy platform.

com.semarchy.commons.sql

TRACE logs SQL queries made by the Semarchy Application Builder and Configuration user interfaces.

com.semarchy.mdm.datahub.services.query.datamgr.IDataManager

Logs database operations performed by the platform for the MDM applications while browsing data. DEBUG logs all queries and execution times.

com.semarchy.platform.engine.core.impl.DefaultStandaloneEngineImpl

Logs execution engine events. DEBUG logs all submitted jobs, and ERROR can be used to troubleshoot engine issues.

com.semarchy.platform.engine.core.impl.product.SL4JExecutionMonitor

Logs execution engine job processing. Use DEBUG to see execution engine details.

com.semarchy.platform.engine.PluginExecution

Logs enricher and validation plugins execution. DEBUG logs plugins feedback, TRACE logs the processing of every row.

com.semarchy.platform.integration.polling.IntegrationLoadDequeuer

Logs the Batch Poller activity. DEBUG logs every polled interval.

com.semarchy.platform.product.notification.JobNotificationHandler

Logs job notifications. ERROR traces notification failures.

com.semarchy.platform.setup.IPlatformManager

Logs platform status changes. INFO traces normal status changes. ERROR traces abnormal statuses.

org.apache.cxf.services

INFO traces REST requests, responses, and faults between the MDM applications and the server. This includes also REST requests performed by plugins on external services (Melissa Data, for example).

org.springframework.jdbc.core.JdbcTemplate

DEBUG traces the SQL queries made by the MDM applications and REST API calls as well as Dashboards-related SQL queries.

log4j.logger.org.springframework.security.ldap

DEBUG traces the authentication attempts made with an LDAP or Active Directory identity provider.

log4j.logger.org.springframework.security.oauth2
log4j.logger.org.springframework.security.oauth2.jwt

Set oauth2 to DEBUG and oauth2.jwt to TRACE to log the authentication attempts made with an OpenID Connect identity provider.

log4j.logger.waffle.spring

DEBUG traces the authentication attempts made with a Windows Authentication identity provider.

log4j.logger.org.springframework.security.saml2
log4j.logger.org.opensaml

DEBUG traces the authentication attempts made with a SAML identity provider.

com.semarchy.xdm.rest.accesslog

DEBUG traces REST API calls. This log is in JSON format and contains: username, API key name (or null for basic authentication), origin IP address, date and time, endpoint (relative to base server URL), response code and execution duration.

com.semarchy.mdm.discovery.core.impl.DefaultProfilingService

Logs the profiling activity for xDM Discovery. DEBUG traces the details of the profiling processes and queues.

com.semarchy.platform.repository.infrastructure.services.HttpClientRequestExecutor

DEBUG traces the REST requests and responses sent and received by Data Notifications.