Workflow Steps

Build Artifacts and Execute Tests Locally

To build artifacts and execute tests locally, follow the steps below. Pre Submit and Post Submit Verification require the first four steps below, but only Post-Submit verification requires Steps 5 and 6.

  1. Initialize. Cleans the workspace to start from a clean build.

  2. Clone your source code Git repository. Retrieves the code you want to test into the clean workspace.

  3. Build the package. Compiles the source code and creates the artifact. All the following testing steps use this artifact. The artifact created in this step must be immutable during the entire process.

  4. Run local tests. Tests the artifact as defined for those steps that do not require HERE platform resources.

  5. If the package passes the local tests, publish the JAR to your Artifact Repository. Publishes the artifact to an Artifact Repository Manager together with the tests. This example uses Artifactory as the Artifact Repository Manager. This step is executed only in Post Submit Verification.

  6. If the package passes the local tests, trigger verification in test environment. Deploys the artifact to the test environment. This step is executed only in Post Submit Verification.

jenkins/step1.groovy contains a reference implementation for the steps above. To examine the code, see the Build and Test Implementation.

Run Test Job on HERE platform

To run the test job in the platform, follow the steps below. This step uses the App-Y (Group-Y) credentials pair.

All artifacts created by this job have their job number included as a suffix in their IDs to prevent resources created by different jobs from conflicting with each other.

  1. Initialize. Downloads artifacts, parses deployment file and downloads OLP CLI.

  2. Prepare test catalogs. Creates the input and output catalogs used by the pipeline, shares them with Group-Y and generates the file pipeline-config.conf to configure the pipeline version. The Input catalog is filled with test partitions.

  3. Create pipeline. Creates a pipeline, pipeline template and pipeline version necessary to run test in pipelines.

  4. Activate pipeline. Activates a pipeline and waits for the pipeline to start running.

  5. Wait for pipeline completion. Waits until the pipeline job completes. Fails the test if the pipeline does not complete successfully.

  6. Run product acceptance tests. Downloads output partitions from the output catalog and executes product acceptance tests.

  7. Deploy to production environment. Triggers the production deployment job.

  8. Clean up. Removes test artifacts.

jenkins/step2.groovy contains a reference implementation for the steps above. To examine the code, see the Validation Implementation.

Deploy to Production

To deploy to production on the platform, follow the steps below.

  1. Initialize. Downloads artifacts, parses deployment file and downloads OLP CLI.

  2. Create new pipeline version. Creates a new pipeline version.

  3. Activate/upgrade pipeline. Upgrades the pipeline to the new pipeline version.

  4. Clean up. Removes created artifacts in case of deployment failure.

jenkins/step3.groovy contains a reference implementation for the steps above. To examine the code, see the Deployment Implementation.

Deployment Descriptions

When a job deploys a pipeline to the platform, the deployment is associated with a set of parameters, with most properties following the parameters of the corresponding CLI commands and their parameters. The example below is for a single deployment.


sdk_version=1.6.1.2

group_id=GROUP-9a42655d-ed42-4a0a-95ae-6f5a428aff4b ////please, specify here your group id

pipeline_name="CICD Auto Sensor Learning Processor P2"
pipeline_description="Reads from Archive layer and writes learnings to output catalog"
pipeline_type=batch-2.0.0

# Reflects the ids used for catalogs in your DPL application separated by spaces
pipeline_template_name="Auto Sensor Learning Processor"
pipeline_template_description="Auto Sensor Learning Processor"
input_catalog_ids=archive-catalog
class_name=com.here.platform.examples.p2.Main

# Method is either use or create, the latter means the catalog will be
# created and deleted at each run
input-catalogs.archive-catalog.method=use
input-catalogs.archive-catalog.hrn=hrn:here-cn:data:::auto-sensor-1-archive

output_catalog.method=use
output_catalog.hrn=hrn:here-cn:data:::auto-sensor-2-learnings

The table below provides the descriptions for these properties.

property description
sdk_version Version of the SDK used to download OLP CLI
group_id Group ID used to create pipeline templates and pipelines. Your Jenkins App ID must belong to this group.
pipeline_name Name of the pipeline created
pipeline_description Description of the pipeline created
pipeline_type The type of pipeline - batch-2.0.0 or stream-2.0.0
pipeline_template_name Name of the pipeline template created
pipeline_template_description Description of the pipeline template created
input_catalog_ids IDs of input catalogs as used in the Data Processing Library
class_name Pipeline template entry point
input_catalogs.*.method "use" - use existing catalog or "create" - create a catalog inside the job
input_catalogs.*.hrn HRN of the catalog when method is "use"
input_catalogs.*.id Hint for creating the catalog HRN when method is "create"
input_catalogs.*.name Name of the catalog to create
input_catalogs.*.config Name of the file with catalog configuration located in deployments/ directory
output_catalogs.*.method "use" - use existing catalog or "create" - create a catalog inside the job
output_catalogs.*.hrn HRN of the catalog when method is "use"
output_catalogs.*.id Hint for creating the catalog HRN when method is "create"
output_catalogs.*.name Name of the catalog to create
output_catalogs.*.config Name of the file with catalog configuration located in deployments/ directory

results matching ""

    No results matching ""