Fixed bug in metrics parsing when metrics configuration was set to a bool instead of a dict.
Removed deprecated Stackdriver log-based metrics client.
Updated documentation related to the native metrics client.
KlioConfig is now loaded on workers from pickled main session instead of a bundled config file (See PR 220).
KlioConfig
max_thread_count parameter in the handle_klio decorator now correctly works for DoFn process methods (See PR 209).
max_thread_count
handle_klio
DoFn
process
Klio’s config file is now read from disk once per worker instead of once per thread (See PR 218).
Add KlioReadFromPubSub and KlioWriteToPubSub IO transforms.
KlioReadFromPubSub
KlioWriteToPubSub
Add default metrics to be collected in Klio’s IO transforms.
Add default metrics to be collected in Klio’s helper transforms.
Add default metrics to be collected in Klio’s decorators.
Add support for using Beam’s metrics API directly.
Add DirectGKERunner, runs direct runner on GKE with added logic to only ack pub/sub message when: * the pipeline successfully ran, but before any write to event output happens if any, * the message is skipped because output data already exists, * the message is dropped because input data does not exist, or * the message is dropped because it was not the intended recipient.
Add metrics interface for shumway
KlioTriggerUpstream no longer raises a pickling error when trying to log.
KlioTriggerUpstream
The Beam metrics client will always be used, no matter the configured runner.
Marked Klio’s support for Stackdriver log-based metrics for deprecation and eventual removal.
Changed thread limiter logging to debug.
Workers will look for klio-job-run-effective.yaml before dropping back to .effective-klio-job.yaml.
klio-job-run-effective.yaml
.effective-klio-job.yaml
Add thread limiting context manager utility (See KEP 2: Thread Management).
Add default thread management to @handle_klio decorator (See KEP 2: Thread Management).
@handle_klio
Partially reverted back to reading config from effective-klio-job.yaml (See PR 147).
effective-klio-job.yaml
Fixed non-Klio -> Klio message parsing.
Fixed calling of to_klio_message in helper transforms.
to_klio_message
Added support for writing an avro file via KlioWriteToAvro
KlioWriteToAvro
Allow for support of empty job_config.data values for the built-in helper filter transforms.
job_config.data
Requires klio-core>=0.2.1 now that dependent changes have been released in new core version
Requires klio-core<0.2.1,>=0.2.0 to prevent usage of 0.2.1 until dependent code is released
Klio lib requires changes not yet released in klio-core
Handling of exceptions yielded by functions/methods decorated with @handle_klio
KlioReadFromBigQuery rewritten as reader + map transform
Limited Apache beam dependency to <2.25.0 due to a breaking change
Initial public release!