Klio now uses Apache Beam’s main session to load the Klio config file onto workers. Klio config fields are defined in the klio-job.yaml and can also be overridden 1 when running a pipeline. The final config fields are serialized as the KlioConfig object, saved into the main session, and loaded by Dataflow workers upon starting up. As a result, Klio jobs no longer need to include .effective-klio-job.yaml and klio-job-run-effective.yaml in the job’s docker image.
klio-job.yaml
KlioConfig
.effective-klio-job.yaml
klio-job-run-effective.yaml
This change is dependent on a bug fix available in apache-beam>=2.31.0. Klio jobs using a version lower than this should upgrade apache-beam.
apache-beam>=2.31.0
apache-beam
The Stackdriver log-based metric client, which has been deprecated since March 2021, has been removed in this release.
The release of 21.12.0 also includes minor improvements to the user experience when verifying a Klio job and adding GCP roles to a job’s service account.
21.12.0
Include more GCP roles when verifying a job’s service account in klio job verify.
klio job verify
Correctly validate existence of Dataflow-related Klio config when running on Dataflow (and not just “not –direct-runner”).
Print out gcloud commands to add necessary GCP roles when running klio job verify --create-resources.
gcloud
klio job verify --create-resources
When running a job, effective config is no longer written to klio-job-run-effective.yaml, but instead to a temp file. This file no longer needs to be included in setup.py projects (See PR 233).
setup.py
Error out when a user tries to run a Dataflow-based job with Stackdriver log-based metrics client configured.
Fixed bug in metrics parsing when metrics configuration was set to a bool instead of a dict.
Removed deprecated Stackdriver log-based metrics client.
Updated documentation related to the native metrics client.
KlioConfig is now loaded on workers from pickled main session instead of a bundled config file (See PR 220).
No changes - bump version to sync with 21.12.0 release.
Footnotes
https://docs.klio.io/en/stable/userguide/config/index.html?highlight=override#overriding-values