21.12.0

Load Klio Config from Main Session

Klio now uses Apache Beam’s main session to load the Klio config file onto workers. Klio config fields are defined in the klio-job.yaml and can also be overridden 1 when running a pipeline. The final config fields are serialized as the KlioConfig object, saved into the main session, and loaded by Dataflow workers upon starting up. As a result, Klio jobs no longer need to include .effective-klio-job.yaml and klio-job-run-effective.yaml in the job’s docker image.

This change is dependent on a bug fix available in apache-beam>=2.31.0. Klio jobs using a version lower than this should upgrade apache-beam.

Metrics

The Stackdriver log-based metric client, which has been deprecated since March 2021, has been removed in this release.

Verifying GCP Roles

The release of 21.12.0 also includes minor improvements to the user experience when verifying a Klio job and adding GCP roles to a job’s service account.

Changes

klio-cli

Added

  • Include more GCP roles when verifying a job’s service account in klio job verify.

Fixed

  • Correctly validate existence of Dataflow-related Klio config when running on Dataflow (and not just “not –direct-runner”).

  • Print out gcloud commands to add necessary GCP roles when running klio job verify --create-resources.

Changed

  • When running a job, effective config is no longer written to klio-job-run-effective.yaml, but instead to a temp file. This file no longer needs to be included in setup.py projects (See PR 233).

  • Error out when a user tries to run a Dataflow-based job with Stackdriver log-based metrics client configured.

klio

Fixed

  • Fixed bug in metrics parsing when metrics configuration was set to a bool instead of a dict.

Removed

  • Removed deprecated Stackdriver log-based metrics client.

Changes

  • Updated documentation related to the native metrics client.

  • KlioConfig is now loaded on workers from pickled main session instead of a bundled config file (See PR 220).

klio-core

No changes - bump version to sync with 21.12.0 release.

klio-exec

No changes - bump version to sync with 21.12.0 release.

Footnotes

1

https://docs.klio.io/en/stable/userguide/config/index.html?highlight=override#overriding-values