Streaming Pipelines

Streaming pipelines allow you to transform data in stream layers. Examples of these pipelines include pipelines that process vehicle sensor data for information of immediate use or to archive for historical analysis. For more on stream layers, see the Data User Guide.

The figure below illustrates the streaming pipeline creation process.

Streaming Pipeline Creation Process
Figure 1. Streaming Pipeline Creation Process

When you create a streaming pipeline, you need to go through the following steps.

  1. Identify one or more stream layers to serve as your data source. These may be versioned, volatile or streaming layers.
    • For more information on working with data catalogs, see the Data User Guide.
    • For example, HERE provides sensor data catalogs based on the Sensor Data Ingestion Interface (SDII). For more details on the SDII catalog data structures, see SDII Data Specification.
  2. Create an output catalog.
    • The portal provides a UI for managing data catalogs. For more information on creating catalogs, see the Data User Guide.
    • The OLP CLI offers you command line options for managing data catalogs. For more information on OLP CLI data commands, see the OLP CLI User Guide.
    • If you prefer to use REST requests, the Data API provides a series of endpoints for managing data catalogs. For more information, see Data API Developer's Guide.
  3. Create your pipeline and write to your output catalog.
  4. Build your pipeline as a fat JAR file.

    • To build your pipeline as a fat JAR, use the command below.
    mvn -Pplatform clean package
    
    • Note that you can only use this command if you use the environment POM as the parent POM.
    • For examples that show how to run your pipeline locally and how to build your pipeline, see the Code Examples page.
  5. Deploy
    • The portal provides a UI for managing pipelines For more information on deploying pipelines, see the Pipelines User Guide.
    • The OLP CLI provides you commands tool to deploy and manage your JAR file in the Workspace. For more information on OLP CLI pipeline commands, see the OLP CLI User Guide.
    • If you prefer to use REST requests, the Pipeline API provides a series of endpoints. For more information, see Pipeline API Developer's Guide.

The SDK provides streaming pipeline Maven Archetypes for setting up a streaming pipeline project. For more information on streaming pipeline Archetypes, see Create Streaming Pipelines.

results matching ""

    No results matching ""