Pipeline CLI User Guide
This guide provides instructions for using the Pipeline CLI.
Background
The Pipeline CLI allows for command line invocation of Pipeline deployment actions.
The Pipeline CLI package has the following contents:
pipeline-cli.jar
pipeline.py
These contents are included by default in the SDK and can be installed and run from any location.
Getting Started
The pipeline.py script runs all the Pipeline CLI actions. Each command follows this format:
pipeline.py <object> <action>
The object for the Pipeline CLI to act on is either 'pipeline', 'package', or 'pipeline-version'
The action for the Pipeline CLI is dependent upon the object. The most common actions are 'create', 'list', and 'get.'
-
Each object-action command combination has its own set of required additional parameters. The full list of possible commands are below in the Command Index section.
-
For every call, the required parameters must be passed in the expected order.
-
Optional parameters can be passed in any order after the required parameters.
-
Parameter values with spaces in them require the parameter value to be surrounded by quotes.
To get help, the following commands are available:
Command | Description |
pipeline.py -h | Overall help |
pipeline.py <object> -h | Object-specific help |
pipeline.py <object> <action> -h | Action-specific help |
Authentication - ~/.here/credentials.properties
In order to access the Pipeline Management features, the user must obtain Open Location Platform credentials from the Portal site at https://platform.here.com/profile/access-credentials.
Download the credentials.properties
file and place it at .here/credentials.properties
in your home directory.
Some Pipeline CLI commands require that you specify a Group ID. You administrator must set up the relevant group IDs for your account and assign your application to the group. If your app is not assigned to a group, then the command produces an error. For more information on Group IDs, see the Teams and Permissions User Guide.
The .here/credentials.properties
file should be formatted like this:
here.user.id = HERE-...
here.client.id = AI9t...
here.access.key.id = _LIH9...
here.access.key.secret = fzb-M...
The credentials file can also include the endpoint for the HERE token provider in the here.token.endpoint.url setting. If the setting is missing, the CLI will default to https://account.api.here.com/oauth2/token
Using an alternate endpoint is rare, but can be done as in the following example:
here.token.endpoint.url=https://alternate.endpoint.here.com/oauth2/token
here.user.id = HERE-...
here.client.id = AI9t...
here.access.key.id = cxSiA....
here.access.key.secret = sAx59Vq_45y0....................
Proxy Settings
If you're using the Pipeline CLI on a network that uses a proxy, it will look for the http_proxy or https_proxy System environment variable and use that value to set the proxy. If both are found, it will use https_proxy
The standard format for this System environment variable is:
protocol://user:password@proxy-server:port
If no port is specified, it will default to 443 for https, 80 for http
If authentication is not required, the System environment variable format is:
protocol://proxy-server:port
Common User Scenarios
The main user scenarios with the Pipeline CLI involve pipeline creation and deployment.
Deploying a Streaming pipeline to a Flink cluster
These are the steps to deploy a streaming pipeline to the Flink cluster using the CLI:
-
Create the Pipeline object:
pipeline.py pipeline create <pipeline-name> <runtime-model> <cluster-type> <group-id>
Optional parameters:
-
--pipeline-description <description>
Example:
pipeline.py pipeline create "My pipeline name" \
STREAMING \
FLINK \
GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
This will return a Pipeline ID. Save this for the steps below.
-
Create the Package object:
pipeline.py package create <package-name> <package-jar> <group-id>
Example:
pipeline.py package create "My Streaming Pipeline JAR" \
../../pipeline.jar \
GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
This will return a Package ID. Save this for the steps below.
-
Create the Pipeline Version:
pipeline.py pipeline-version create <pipeline-version-name> <pipeline-id> <package-id> <main-class>
Optional parameters:
--pipeline-config <path-to-config-file>
--container-memory <value>
--pipeline-memory <value>
--cpu-parallelism <value>
-
--memory-parallelism <value>
Example:
pipeline.py pipeline-version create "My Pipeline Version 1.0" \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
0e492565-4963-4169-87e8-8c5ade20d72d \
com.some.package.MyPipeline
This will return a Pipeline Version ID. Save this for the steps below.
-
Activate the Pipeline Version:
pipeline.py pipeline-version activate <pipeline-id> <pipeline-version-id>
Optional parameters:
--scheduled
-
--pipeline-jobs-config <path-to-config-file>
Example:
pipeline.py pipeline-version activate \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93
This will run the Pipeline Version and perform 'Get Pipeline' actions until the pipeline enters the RUNNING state and returns a logging URL.
Deploying a Scheduled Batch pipeline with Pipeline configuration files to a Spark cluster
These are the steps to deploy a scheduled batch pipeline to the Spark cluster using the CLI:
In this case, the pipeline has both a pipeline-config.conf
and pipeline-jobs.conf
configuration files required for the pipeline to run.
-
Create the Pipeline object:
pipeline.py pipeline create <pipeline-name> <runtime-model> <cluster-type> <group-id>
Optional parameters:
-
--pipeline-description <description>
Example:
pipeline.py pipeline create "My pipeline name" \
BATCH \
SPARK \
GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
This will return a Pipeline ID. Save this for the steps below.
-
Create the Package object:
pipeline.py package create <package-name> <package-jar> <group-id>
Example:
pipeline.py package create "My Batch Pipeline JAR" \
../../pipeline.jar \
GROUP-266917f2-072e-4e1b-bbff-5c60fef6ac7d
This will return a Package ID. Save this for the steps below.
-
Create the Pipeline Version:
pipeline.py pipeline-version create <pipeline-version-name> <pipeline-id> <package-id> <main-class>
Optional parameters:
--pipeline-config <path-to-config-file>
--container-memory <value>
--pipeline-memory <value>
--cpu-parallelism <value>
-
--memory-parallelism <value>
Example:
pipeline.py pipeline-version create "My Pipeline Version 1.0" \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
0e492565-4963-4169-87e8-8c5ade20d72d \
com.some.package.MyPipeline \
--pipeline-config ../pipeline-config.conf
This will return a Pipeline Version ID. Save this for the steps below.
-
Activate the Pipeline Version:
pipeline.py pipeline-version activate <pipeline-id> <pipeline-version-id>
Optional parameters:
--scheduled
-
--pipeline-jobs-config <path-to-config-file>
Example:
pipeline.py pipeline-version activate \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93 \
--scheduled \
--pipeline-jobs-config \
../pipeline-jobs.conf
This will schedule the Pipeline Version to run when the next catalog update occurs.
-
Get Pipeline status:
pipeline.py pipeline get <pipeline-id>
Example:
pipeline.py pipeline get 717570b3-29d3-4fc3-9bc4-04c9cd1641e8
This will return the pipeline state and the logging URL.
Until the scheduled Pipeline runs, the logging URL will remain null
.
Command Index
Activate Pipeline Version
pipeline.py pipeline-version activate <pipeline-id> <pipeline-version-id>
Optional parameters:
--scheduled
--pipeline-jobs-config <path-to-config-file>
Example:
pipeline.py pipeline-version activate \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93 \
--scheduled
Cancel Pipeline Version
pipeline.py pipeline-version cancel <pipeline-id>
Example:
pipeline.py pipeline-version cancel \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8
Create Package
pipeline.py package create <package-name> <package-jar> <group-id>
Example:
pipeline.py package create "My Pipeline JAR" \
../../pipeline.jar \
GROUP-fcf333b7-794d-4b94-b1ba-1d3116996c89
Create Pipeline
pipeline.py pipeline create <pipeline-name> <runtime-model> <cluster-type> <group-id>
Optional parameters:
-pipeline-description <description>
Example:
pipeline.py pipeline create "My Awesome Pipeline" \
STREAMING \
FLINK \
GROUP-fcf333b7-794d-4b94-b1ba-1d3116996c89 \
--pipeline-description "This is an amazing pipeline"
Create Pipeline Version
pipeline.py pipeline-version create <pipeline-version-name> <pipeline-id> <package-id> <main-class>
Optional parameters:
--pipeline-config <path-to-config-file>
--container-memory <value>
--pipeline-memory <value>
--cpu-parallelism <value>
--memory-parallelism <value>
Example:
pipeline.py pipeline-version create "My Pipeline Version 1.0" \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
0e492565-4963-4169-87e8-8c5ade20d72d \
com.some.package.MyPipeline \
--pipeline-config ../../pipeline-config.conf \
--container-memory 2048.0 \
--cpu-parallelism 2
Get Package
pipeline.py package get <package-id>
Example:
pipeline.py package get 0e492565-4963-4169-87e8-8c5ade20d72d
Get Pipeline
pipeline.py pipeline get <pipeline-id>
Example:
pipeline.py pipeline get 717570b3-29d3-4fc3-9bc4-04c9cd1641e8
Get Pipeline Version
pipeline.py pipeline-version get <pipeline-id> <pipeline-version-id>
Example:
pipeline.py pipeline-version get \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93
List Jobs for Pipeline Version
pipeline.py pipeline-version list-jobs <pipeline-id> <pipeline-version-id>
Example:
pipeline.py pipeline-version list-jobs \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
5f65244d-cde2-45dc-8c5c-ba678233da93
List Packages
pipeline.py package list
List Pipeline Versions
pipeline.py pipeline-version list <pipeline-id>
Example:
pipeline.py pipeline-version list \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8
List Pipelines
pipeline.py pipeline list
Pause Pipeline Version
pipeline.py pipeline-version pause <pipeline-id>
Example:
pipeline.py pipeline-version pause \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8
Resume Pipeline Version
pipeline.py pipeline-version resume <pipeline-id>
Example:
pipeline.py pipeline-version resume \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8
Update Pipeline
pipeline.py pipeline update <pipeline-id> <pipeline-name> <runtime-model> <cluster-type> <group-id>
Optional parameters:
--pipeline-description <description>
Example:
pipeline.py pipeline update \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
"My Updated Pipeline" \
STREAMING \
FLINK \
GROUP-fcf333b7-794d-4b94-b1ba-1d3116996c89 \
--pipeline-description "This is an amazing pipeline"
Upgrade Pipeline Version
pipeline.py pipeline-version upgrade <pipeline-id> <pipeline-version-id>
Example:
pipeline.py pipeline-version upgrade \
717570b3-29d3-4fc3-9bc4-04c9cd1641e8 \
d76160c8-1a50-418f-be8d-67732e72c8db
Optional Parameters
In addition to the call-specific optional parameters, there are universal optional parameters that can be used for any call. All optional parameters must appear at the end of the call.
-
--cred <path-to-credentials-file>
- Specifies an alternate credentials file -
--pipeline-url <pipeline-url>
- Specifies an alternate Pipeline Service URL -
--profile <profile>
- Specifies an alternate profile in the credentials file