klio_core.dataflow
Module for interacting with the Dataflow REST API.
klio_core.dataflow.
DataflowClient
Client to interact with Dataflow REST API.
api_version (str) – Version of Dataflow REST API. Defaults to v1b3.
v1b3
find_job_by_name
Search Dataflow for a job given its name and GCP project.
job_name (str) – Name of Dataflow job.
gcp_project (str) – GCP project in which to search.
region (str) – Region in which to search. Defaults to searching all regions in klio_core.variables.DATAFLOW_REGIONS.
klio_core.variables.DATAFLOW_REGIONS
If found, dict of job summary results. Otherwise, None.
dict
None
dict or None
get_job_detail
Get verbose job detail given a job name.
If found, dict of detailed job results. Otherwise, None.
get_job_input_topic
Get input topic of a particular job.
If found, input topic of job. Otherwise, None.
str or None
get_dataflow_client
Get an initialized DataflowClient.
This function will first check if there is an already initialized client in the global namespace. Otherwise, initialize one then set it in the global namespace to avoid redundant initialization.
A client to interact with the Dataflow REST API.