21.2.0

New Versioning Scheme & Synchronization

The maintainers of Klio have decided to synchronize the versions of the packages that make up the Klio ecosystems. From now on, every package in each release will have matching versions.

As well, we are switching to calendar versioning 1 and will target making monthly releases. The new scheme for all Klio packages will be YY.MM.MICRO with an optional modifier suffix (e.g. rc1, beta1, etc).

Since we’re moving off a semantic versioning scheme, going forward there should not be any general expectation of backwards compatibility between scheduled releases, although we will be clear about what changes are made in release notes and changelogs. When needed, we’ll publish off-schedule releases by bumping the MICRO part. While we’ll aim to keep those backwards compatible, it is not a guarantee.

Batch Mode, New Event IOs, Templating Configs

As of version 21.2.0, Klio now has full support for running jobs in batch mode 🎉. And with that, a lot of goodies that can be used in streaming & batch.

New features include:

  • Run a job in batch mode

    • Read how to easily convert (or otherwise simply toggle between) a current streaming job to batch here

  • Event IO support for:

    • reading/writing KlioMessages to:

      • files (both locally and on GCS) (docs)

      • BigQuery (docs)

      • Avro (both locally and on GCS) (docs)

    • Note: reading from non-Pub/Sub event input is only supported for Batch

  • Read multiple event inputs of KlioMessages (docs)

  • Support for templating configuration keys & values, and overriding configuration values at runtime (docs)

Changes

klio-cli

Fixed

  • The --config-file flag can now be used in setup.py projects.

Changed

  • Moved IndentListDumper to klio_core config utils.

  • Runtime config file for klioexec now written to klio-job-run-effective.yaml in the job’s directory.

klio

Changed

  • Changed thread limiter logging to debug.

  • Workers will look for klio-job-run-effective.yaml before dropping back to .effective-klio-job.yaml.

klio-core

Changed

  • Moved IndentListDumper to klio_core config utils.

klio-exec

Fixed

  • klioexec now writes runtime config to include in setup.py distribution.

Footnotes

1

https://calver.org