Pipeline CLI User Guide


This guide provides instructions for using the Pipeline CLI.

Background


The Pipeline CLI allows for command line invocation of Pipeline deployment actions.

The Pipeline CLI package has the following contents:

pipeline-cli.jar
pipeline.py

These contents are included by default in the SDK and can be installed and run from any location.

Getting Started


The pipeline.py script runs all the Pipeline CLI actions. Each command follows this format:

pipeline.py <object> <action>

The object for the Pipeline CLI to act on is either 'pipeline', 'package', or 'pipeline-version'

The action for the Pipeline CLI is dependent upon the object. The most common actions are 'create', 'list', and 'get.'

  • Each object-action command combination has its own set of required additional parameters. The full list of possible commands are below in the Command Index section.

  • For every call, the required parameters must be passed in the expected order.

  • Optional parameters can be passed in any order after the required parameters.

  • Parameter values with spaces in them require the parameter value to be surrounded by quotes.

To get help, the following commands are available:

Command Description
pipeline.py -h Overall help
pipeline.py <object> -h Object-specific help
pipeline.py <object> <action> -h Action-specific help

Authentication - ~/.here/credentials.properties


In order to access the Pipeline Management features, the user must obtain Open Location Platform credentials from the Portal site at https://platform.here.com/profile/access-credentials.

Download the credentials.properties file and place it at .here/credentials.properties in your home directory.

Some Pipeline CLI commands require that you specify a Group ID. You administrator must set up the relevant group IDs for your account and assign your application to the group. If your app is not assigned to a group, then the command produces an error. For more information on Group IDs, see the Teams and Permissions User Guide.

The .here/credentials.properties file should be formatted like this:

here.user.id = HERE-...
here.client.id = AI9t...
here.access.key.id = _LIH9...
here.access.key.secret = fzb-M...

The credentials file can also include the endpoint for the HERE token provider in the here.token.endpoint.url setting. If the setting is missing, the CLI will default to https://account.api.here.com/oauth2/token

Using an alternate endpoint is rare, but can be done as in the following example:

here.token.endpoint.url=https://alternate.endpoint.here.com/oauth2/token
here.user.id = HERE-...
here.client.id = AI9t...
here.access.key.id = cxSiA....
here.access.key.secret = sAx59Vq_45y0....................

Proxy Settings


If you're using the Pipeline CLI on a network that uses a proxy, it will look for the http_proxy or https_proxy System environment variable and use that value to set the proxy. If both are found, it will use https_proxy

The standard format for this System environment variable is:

# protocol = http / https
protocol://user:password@proxy-server:port

If no port is specified, it will default to 443 for https, 80 for http

If authentication is not required, the System environment variable format is:

protocol://proxy-server:port

Common User Scenarios


The main user scenarios with the Pipeline CLI involve pipeline creation and deployment.

These are the steps to deploy a streaming pipeline to the Flink cluster using the CLI:

  1. Create the Pipeline object:

     pipeline.py pipeline create <pipeline-name> <runtime-model> <cluster-type> <group-id>
    

    Optional parameters:

    • --pipeline-description <description>

      Example:

      pipeline.py pipeline create "My pipeline name" \
      STREAMING \
      FLINK \
      GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
      

      This will return a Pipeline ID. Save this for the steps below.

  2. Create the Package object:

     pipeline.py package create <package-name> <package-jar> <group-id>
    

    Example:

     pipeline.py package create "My Streaming Pipeline JAR" \
     ../../pipeline.jar \
     GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
    

    This will return a Package ID. Save this for the steps below.

  3. Create the Pipeline Version:

     pipeline.py pipeline-version create <pipeline-version-name> <pipeline-id> <package-id> <main-class>
    

    Optional parameters:

    • --pipeline-config <path-to-config-file>
    • --container-memory <value>
    • --pipeline-memory <value>
    • --cpu-parallelism <value>
    • --memory-parallelism <value>

      Example:

      pipeline.py pipeline-version create "My Pipeline Version 1.0" \
      717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
      0e492565-4963-4169-87e8-8c5ade20d72d \
      com.some.package.MyPipeline
      

      This will return a Pipeline Version ID. Save this for the steps below.

  4. Activate the Pipeline Version:

     pipeline.py pipeline-version activate <pipeline-id> <pipeline-version-id>
    

    Optional parameters:

    • --scheduled
    • --pipeline-jobs-config <path-to-config-file>

      Example:

      pipeline.py pipeline-version activate \
      717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
      5f65244d-cde2-45dc-8c5c-ba678233da93
      

      This will run the Pipeline Version and perform 'Get Pipeline' actions until the pipeline enters the RUNNING state and returns a logging URL.

Deploying a Scheduled Batch pipeline with Pipeline configuration files to a Spark cluster

These are the steps to deploy a scheduled batch pipeline to the Spark cluster using the CLI:

In this case, the pipeline has both a pipeline-config.conf and pipeline-jobs.conf configuration files required for the pipeline to run.

  1. Create the Pipeline object:

     pipeline.py pipeline create <pipeline-name> <runtime-model> <cluster-type> <group-id>
    

    Optional parameters:

    • --pipeline-description <description>

      Example:

      pipeline.py pipeline create "My pipeline name" \
      BATCH \
      SPARK \
      GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
      

      This will return a Pipeline ID. Save this for the steps below.

  2. Create the Package object:

     pipeline.py package create <package-name> <package-jar> <group-id>
    

    Example:

     pipeline.py package create "My Batch Pipeline JAR" \
     ../../pipeline.jar \
     GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
    

    This will return a Package ID. Save this for the steps below.

  3. Create the Pipeline Version:

     pipeline.py pipeline-version create <pipeline-version-name> <pipeline-id> <package-id> <main-class>
    

    Optional parameters:

    • --pipeline-config <path-to-config-file>
    • --container-memory <value>
    • --pipeline-memory <value>
    • --cpu-parallelism <value>
    • --memory-parallelism <value>

      Example:

      pipeline.py pipeline-version create "My Pipeline Version 1.0" \
      717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
      0e492565-4963-4169-87e8-8c5ade20d72d \
      com.some.package.MyPipeline \
      --pipeline-config ../pipeline-config.conf
      

      This will return a Pipeline Version ID. Save this for the steps below.

  4. Activate the Pipeline Version:

     pipeline.py pipeline-version activate <pipeline-id> <pipeline-version-id>
    

    Optional parameters:

    • --scheduled
    • --pipeline-jobs-config <path-to-config-file>

      Example:

      pipeline.py pipeline-version activate \
      717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
      5f65244d-cde2-45dc-8c5c-ba678233da93 \
      --scheduled \
      --pipeline-jobs-config \
      ../pipeline-jobs.conf
      

      This will schedule the Pipeline Version to run when the next catalog update occurs.

  5. Get Pipeline status:

     pipeline.py pipeline get <pipeline-id>
    

    Example:

     pipeline.py pipeline get 717570b3-29d3-4fc3-9bc4-04c9cd1641e8
    

    This will return the pipeline state and the logging URL.

    Until the scheduled Pipeline runs, the logging URL will remain null.

Command Index


Activate Pipeline Version

pipeline.py pipeline-version activate <pipeline-id> <pipeline-version-id>

Optional parameters:

  • --scheduled
  • --pipeline-jobs-config <path-to-config-file>

Example:

pipeline.py pipeline-version activate \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93 \
--scheduled

Cancel Pipeline Version

pipeline.py pipeline-version cancel <pipeline-id>

Example:

pipeline.py pipeline-version cancel \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8

Create Package

pipeline.py package create <package-name> <package-jar> <group-id>

Example:

pipeline.py package create "My Pipeline JAR" \
../../pipeline.jar \
GROUP-fcf333b7-794d-4b94-b1ba-1d3116996c89

Create Pipeline

pipeline.py pipeline create <pipeline-name> <runtime-model> <cluster-type> <group-id>

Optional parameters:

  • -pipeline-description <description>

Example:

pipeline.py pipeline create "My Awesome Pipeline" \
STREAMING \
FLINK \
GROUP-fcf333b7-794d-4b94-b1ba-1d3116996c89 \
--pipeline-description "This is an amazing pipeline"

Create Pipeline Version

pipeline.py pipeline-version create <pipeline-version-name> <pipeline-id> <package-id> <main-class>

Optional parameters:

  • --pipeline-config <path-to-config-file>
  • --container-memory <value>
  • --pipeline-memory <value>
  • --cpu-parallelism <value>
  • --memory-parallelism <value>

Example:

pipeline.py pipeline-version create "My Pipeline Version 1.0" \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
0e492565-4963-4169-87e8-8c5ade20d72d \
com.some.package.MyPipeline \
--pipeline-config ../../pipeline-config.conf \
--container-memory 2048.0 \
--cpu-parallelism 2

Get Package

pipeline.py package get <package-id>

Example:

pipeline.py package get 0e492565-4963-4169-87e8-8c5ade20d72d

Get Pipeline

pipeline.py pipeline get <pipeline-id>

Example:

pipeline.py pipeline get 717570b3-29d3-4fc3-9bc4-04c9cd1641e8

Get Pipeline Version

pipeline.py pipeline-version get <pipeline-id> <pipeline-version-id>

Example:

pipeline.py pipeline-version get \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93

List Jobs for Pipeline Version

pipeline.py pipeline-version list-jobs <pipeline-id> <pipeline-version-id>

Example:

pipeline.py pipeline-version list-jobs \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93

List Packages

pipeline.py package list

List Pipeline Versions

pipeline.py pipeline-version list <pipeline-id>

Example:

pipeline.py pipeline-version list \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8

List Pipelines

pipeline.py pipeline list

Pause Pipeline Version

pipeline.py pipeline-version pause <pipeline-id>

Example:

pipeline.py pipeline-version pause \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8

Resume Pipeline Version

pipeline.py pipeline-version resume <pipeline-id>

Example:

pipeline.py pipeline-version resume \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8

Update Pipeline

pipeline.py pipeline update <pipeline-id> <pipeline-name> <runtime-model> <cluster-type> <group-id>

Optional parameters:

  • --pipeline-description <description>

Example:

pipeline.py pipeline update \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
"My Updated Pipeline" \
STREAMING \
FLINK \
GROUP-fcf333b7-794d-4b94-b1ba-1d3116996c89 \
--pipeline-description "This is an amazing pipeline"

Upgrade Pipeline Version

pipeline.py pipeline-version upgrade <pipeline-id> <pipeline-version-id>

Example:

pipeline.py pipeline-version upgrade \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
d76160c8-1a50-418f-be8d-67732e72c8db

Optional Parameters


In addition to the call-specific optional parameters, there are universal optional parameters that can be used for any call. All optional parameters must appear at the end of the call.

  • --cred <path-to-credentials-file> - Specifies an alternate credentials file
  • --pipeline-url <pipeline-url> - Specifies an alternate Pipeline Service URL
  • --profile <profile> - Specifies an alternate profile in the credentials file

results matching ""

    No results matching ""