Getting started with Apache Kafka Raw metadata

Overview

This article describes how to read and write raw data with Apache Kafka. The Semarchy xDI raw metadata allows exchanging data with no specific format.

To exchange data in a specific format (AVRO or JSON), you need to use the Semarchy xDI structured metadata. Refer to Getting started with Kafka Structured metadata for more information.

Create metadata

To create the metadata:

  1. Right-click the project folder, then select New > Metadata.

  2. In the New Metadata window, select Kafka Server - raw and click Next.

  3. Name the data model and click Next.

  4. Select the installed Apache Kafka module and click Finish.
    The metadata is created.

Define the server properties

Select the root node of the metadata to define the server properties listed below:

Table 1. Apache Kafka Server Properties
Property Description

Name

A label/alias for this metadata.

Module

Apache Kafka module to use.

Bootstrap Servers

Corresponds to the boostrap.servers Kafka property. It is a comma-separated list of bootstrap servers in the <host>:<port> format.

SASL Mechanism

SASL mechanism to connect to Apache Kafka. Leave blank to use the Apache Kafka client’s default settings.

The following options are available:

  • GSSAPI

  • PLAIN

  • SCRAM-SHA-256

  • SCRAM-SHA-512

Enable SSL/TLS

Enables a secured connection between the client and the broker. Leave blank to use the Apache Kafka client’s default settings.

Default Write Mode

Defines the default value for the write mode in the integration template.
This parameter determines how the component writes messages to the Apache Kafka topic:

  • asynchronous: The component sends all the messages and does not wait for a response. This mode can optimize the performance but may fail to report errors and statistics.

  • synchronous: The component waits for a response after each message is written and stops if an error occurs.

The following image illustrates an example of a Apache Kafka host configuration:
getting started kafka host

Define topics

Topics are used to store data in Apache Kafka in the form of messages.

To define a topic:

  1. Right-click the server node, then click New > Topic.

    getting started kafka topic

  2. Enter a Name and a Physical Name for the topic. The physical name corresponds to the actual topic name in Kafka.

  3. Right-click the topic, click New > Value, to create the value and set its type. The Value node corresponds to the Kafka message payload (Record), which can be used in a mapping.

    You must define a single Value node for each topic.

    getting started kafka value

  4. Optionally add Key, Header, or Property nodes.
    These nodes correspond to the Kafka message headers, properties, and keys. They can be used in mappings that produce messages on the Kafka topic.

    • Key: A Key can be attached to the message produced on a topic. When this node is not defined, or not mapped in a mapping, no key is written.

      A topic may contain a single Key node.
    • Header: Headers are attached to the message when writing into the topic. You can add as many header nodes as required.

    • Property: An additional property is attached to the messages in the topic.

Define consumers

Consumers subscribe to topics to read messages.

To define a consumer:

  1. Right-click the host, then click New > Consumer Group.

    getting started kafka consumer

  2. Enter a name and a physical name for the consumer.

  3. In the Topics subscribed field, add the topics to which the consumer should subscribe.

    getting started kafka consumer subscribe

  4. Right-click the consumer, click New > Value, and set its type. The Value node corresponds to the Kafka message payload (Record) consumed from the topic.

    You must define a single Value node for each consumer.

    getting started kafka consumer value

  5. Add Key, Header, and Property nodes similarly to a topic definition. These fields are retrieved when consuming from the subscribed topics and can be used in mappings.

Create mappings

Mappings can be created for sending to (producer) and receiving (consumer) data from Kafka.

Producer mapping

The following example illustrates a producer mapping in which a topic is used as a target. This mapping loads the Key and Value (message payload) from the database table and produces messages to the topic.

getting started kafka mapping produce

Consumer mapping

The following example illustrates a consumer mapping in which a consumer is used as a source and mapped to a database table. This mapping uses the Key, Value (message payload) as well as a PAYMENT_NAME Header defined in the consumer metadata.

getting started kafka mapping consume

Kafka server timeout

If Apache Kafka is not returning messages to a consuming mapping, the mapping will run infinitely.
To prevent this and allow the consuming mapping to proceed when no message is returned, set a Kafka Timeout duration (for example, 5000ms) in the Kafka To Rdbms template.
getting started kafka timeout

Sample project

The Apache Kafka component is distributed with sample projects that contain various examples and files. Use these projects to better understand how the component works, and to get a head start on implementing it in your projects.

Refer to Install components to learn about importing sample projects.