Getting started with Apache Kafka Structured metadata
Overview
This article describes how to read and write structured data with Apache Kafka. The Semarchy xDI structured metadata allows exchanging data in a specific format (for example, AVRO or JSON).
To exchange data with no specific format, use the Semarchy xDI raw metadata. Refer to Getting started with Kafka Raw metadata for more information.
Create metadata
To create the metadata:
-
Right-click the project folder, then select New > Metadata.
-
In the New Metadata window, select Kafka Server - structured and click Next.
-
Name the data model and click Next.
-
Select the installed Apache Kafka module and click Finish.
The metadata is created.
Define a host
Select the root node of the metadata to define the server properties listed below:
Property | Description |
---|---|
Name |
A label/alias for this metadata. |
Module |
Apache Kafka module to use. |
Connection Properties |
Properties to be used for connection to kafka server, defined with the format: |
The following image illustrates an example of a host:
Create topics
Topics are used to store data in Apache Kafka in the form of messages.
To create a topic:
-
Right-click the host node, then click New > Topic.
-
Enter a name and a physical name for the topic.
-
Right-click the topic, click New > Key.
Do not create more than one key on a topic. -
In the Standard tab of the key properties, in the Schema field, drag and drop the JSON or the AVRO schema that the topic should represent. You can also use the browse button to browse through the available schemas.
Drag and drop or select only the root node of the JSON or AVRO schema. -
Right-click the topic again, click New > Value.
Do not create more than one value on a topic. -
In the Standard tab of the value properties, in the Schema field, drag and drop the JSON or the AVRO schema that the topic should represent. You can also use the browse button to browse through the available schemas.
Drag and drop or select only the root node of the JSON or AVRO schema.