klio_core.utils
Utility functions for use within the Klio ecosystem.
klio_core.utils.
get_publisher
Get a publisher client for a given topic.
Will first check if there is an already initialized client in the global namespace. Otherwise, initialize one then set it in the global namespace to avoid redundant initialization.
topic (str) – Pub/Sub topic for the client with which to be initialized.
an initialized client for publishing to Google Pub/Sub.
google.cloud.pubsub_v1.publisher.client.Client
get_or_initialize_global
Get a global variable, initializing if does not exist.
Caution
Global variables may cause problems for jobs in Dataflow, particularly where jobs use more than one thread.
name (str) – name of global variable.
initializer (Any) – initial value if name does not exist in the global namespace.
name
Value of the global variable.
Any
get_global
Get a variable from the global namespace.
value of global variable, or None if not set.
None
set_global
Set a variable in the global namespace.
value (Any) – value of global variable.
delete_global
Delete a variable from the global namespace.
get_config_by_path
Read in the file given at specified config path
config_filepath (str) – File path to klio config file
parse_yaml (bool) – Whether to parse the given file path as yaml
python object of yaml config file or bytes read from file
get_config_job_dir
job_dir (str) – File directory path to klio config file
config_file (str) – File name of config file, if not provided klio-job.yaml will be used
klio-job.yaml
job_dir - Absolute path to config file directory config_file - Path to config file
with_klio_config
Decorator for commands to automatically handle a number of options regarding config, and provide a properly constructed KlioConfig as an argument named klio_config.
Be aware this is a function wrapper and must come after any other decorators for click options, arguments, etc.