Select a Streaming Connector: Kafka or HTTP

Stream layers are implemented as Kafka clusters.

To read from or write to a stream layer, your application must use one of the following connector types:

  • Direct Kafka is the preferred connector type since it directly communicates with the underlying Kafka cluster. It is the default connector.
  • HTTP Connector is an HTTP wrapper on top of Kafka, and therefore implies a communication overhead.

If required, you can provide a different connector type for the producer and the consumer. For example, your application can use Direct Kafka to read from a stream layer and the HTTP connector to write to a different stream layer.

Note

If your application is using the Data Client Library to access data in the Marketplace, it must use the HTTP connector so that your data consumption can be calculated.

Direct Kafka

To use the Direct Kafka connector type, you must provide the following dependency:

Maven
sbt
<dependencies>
    <dependency>
        <groupId>com.here.platform.data.client</groupId>
        <artifactId>kafka-support_2.11</artifactId>
        <version>0.16.72</version>
    </dependency>
</dependencies>
libraryDependencies += "com.here.platform.data.client" %% "kafka-support" % "0.16.72"

Note that, if your application uses flink-support, you do not need to add this dependency as the dependency is already included in flink-support. For more details, refer to the Dependencies for Stream Pipelines section.

To enable Direct Kafka support, you must provide the following configuration:

here.platform.data-client.stream.connector {
  producer = "kafka-connector"
  consumer = "kafka-connector"
}

Producers and consumers can be configured independently. For more information, see the Data Client configuration file in the Configuration section.

HTTP Connector

If you cannot use Kafka Direct, for example because your application needs to access data in the Marketplace or is running behind a proxy, use the HTTP connector.

If you cannot use Kafka Direct, for example because your application is running behind a proxy, use the HTTP connector. To use it, the following configuration must be set:

here.platform.data-client.stream.connector {
  producer = "http-connector"
  consumer = "http-connector"
}

Producers and consumers can be configured independently. For more information, see the Data Client configuration file in the Configuration section.

Note: Kafka Consumer Group Name

If your application uses the HTTP Connector type, the connector will define a unique Kafka consumer group name for a given combination of credentials, catalog HRN and layer ID. As a result, if you run two applications that use the same credentials, and consume data from the same stream layer, any given message will be consumed by only one application. Specify a unique kafka consumer group in each application to avoid this behaviour.

 

results matching ""

    No results matching ""