Klio Core

latest release: 22.3.0 (What’s new?)

Current Status

Klio is currently under rapid development. This means that APIs and features will evolve. It is recommended that teams who adopt Klio today upgrade their installation as new releases become available, as backwards compatibility is not yet guaranteed.

A library of common utilities, including the Klio protobuf definitions and configuration parsing.

As the klio-core package is not meant to be installed directly, check out the installation guide for how to setup installation. There is also the user guide and the API documentation for more information.

---

klio_core.config

Klio configuration handling.

KlioConfig	Klio config object representation of klio-job.yaml.
KlioJobConfig	Job-specific config representing the job_config key of klio-job.yaml.
KlioPipelineConfig	Pipeline-specific config representing the pipeline_options key of klio-job.yaml.

klio_core.options

image_tag(*args, **kwargs)	Docker image tag to use.
direct_runner(func)	Run the job locally via the DirectRunner.
update(func)	[Experimental] Update an existing streaming Cloud Dataflow job.
show_logs(func)	Show a job’s logs while profiling.
interval(func)	Sampling period (in seconds).
include_children(func)	Monitor forked processes as well (sums up all process memory).
multiprocess(func)	Monitor forked processes creating individual plots for each child.
plot_graph(func)	Plot memory profile using matplotlib.
maximum(func)	Print maximum memory usage per line in aggregate of all input elements processed
per_element(*args, **kwargs)	Print memory usage per line for each input element processed
iterations(func)	Number of times to execute each entity ID provided.

klio_core.utils

Utility functions for use within the Klio ecosystem.

get_publisher(topic)	Get a publisher client for a given topic.
get_or_initialize_global(name, initializer)	Get a global variable, initializing if does not exist.
set_global(name, value)	Set a variable in the global namespace.
get_global(name)	Get a variable from the global namespace.
delete_global(name)	Delete a variable from the global namespace.
get_config_by_path(config_filepath[, parse_yaml])	Read in the file given at specified config path
get_config_job_dir(job_dir, config_file)	Read in the file given at specified config path
with_klio_config(func)	Decorator for commands to automatically handle a number of options regarding config, and provide a properly constructed KlioConfig as an argument named klio_config.

klio_core.dataflow

Module for interacting with the Dataflow REST API.

DataflowClient	Client to interact with Dataflow REST API.
get_dataflow_client	Get an initialized DataflowClient.

Common Variables

Shared variables for use within the Klio ecosystem.

DATAFLOW_REGIONS

Default tuple of regions/locations for which to query.

klio_core.exceptions

Common exceptions for use within the Klio ecosystem.

KlioConfigTemplatingException(key)

Exception for missing keys when overriding Klio config parameters