klio_core.utils

Utility functions for use within the Klio ecosystem.

klio_core.utils.get_publisher(topic)

Get a publisher client for a given topic.

Will first check if there is an already initialized client in the global namespace. Otherwise, initialize one then set it in the global namespace to avoid redundant initialization.

Parameters

topic (str) – Pub/Sub topic for the client with which to be initialized.

Returns

an initialized client for publishing to Google Pub/Sub.

Return type

google.cloud.pubsub_v1.publisher.client.Client

klio_core.utils.get_or_initialize_global(name, initializer)

Get a global variable, initializing if does not exist.

Caution

Global variables may cause problems for jobs in Dataflow, particularly where jobs use more than one thread.

Parameters

  • name (str) – name of global variable.

  • initializer (Any) – initial value if name does not exist in the global namespace.

Returns

Value of the global variable.

Return type

Any

klio_core.utils.get_global(name)

Get a variable from the global namespace.

Parameters

name (str) – name of global variable.

Returns

value of global variable, or None if not set.

Return type

Any

klio_core.utils.set_global(name, value)

Set a variable in the global namespace.

Parameters

  • name (str) – name of global variable.

  • value (Any) – value of global variable.

klio_core.utils.delete_global(name)

Delete a variable from the global namespace.

Parameters

name (str) – name of global variable.

klio_core.utils.get_config_by_path(config_filepath, parse_yaml=True)

Read in the file given at specified config path

Parameters

  • config_filepath (str) – File path to klio config file

  • parse_yaml (bool) – Whether to parse the given file path as yaml

Returns

python object of yaml config file or bytes read from file

klio_core.utils.get_config_job_dir(job_dir, config_file)

Read in the file given at specified config path

Parameters

  • job_dir (str) – File directory path to klio config file

  • config_file (str) – File name of config file, if not provided klio-job.yaml will be used

Returns

job_dir - Absolute path to config file directory config_file - Path to config file

klio_core.utils.with_klio_config(func)

Decorator for commands to automatically handle a number of options regarding config, and provide a properly constructed KlioConfig as an argument named klio_config.

Be aware this is a function wrapper and must come after any other decorators for click options, arguments, etc.