klio_core.dataflow

Module for interacting with the Dataflow REST API.

class klio_core.dataflow.DataflowClient(api_version=None)

Client to interact with Dataflow REST API.

Parameters

api_version (str) – Version of Dataflow REST API. Defaults to v1b3.

find_job_by_name(job_name, gcp_project, region=None)

Search Dataflow for a job given its name and GCP project.

Parameters

• job_name (str) – Name of Dataflow job.

• gcp_project (str) – GCP project in which to search.

• region (str) – Region in which to search. Defaults to searching all regions in klio_core.variables.DATAFLOW_REGIONS.

Returns

If found, dict of job summary results. Otherwise, None.

Return type

dict or None

get_job_detail(job_name, gcp_project, region=None)

Get verbose job detail given a job name.

Parameters

• job_name (str) – Name of Dataflow job.

• gcp_project (str) – GCP project in which to search.

• region (str) – Region in which to search. Defaults to searching all regions in klio_core.variables.DATAFLOW_REGIONS.

Returns

If found, dict of detailed job results. Otherwise, None.

Return type

dict or None

get_job_input_topic(job_name, gcp_project, region=None)

Get input topic of a particular job.

Parameters

• job_name (str) – Name of Dataflow job.

• gcp_project (str) – GCP project in which to search.

• region (str) – Region in which to search. Defaults to searching all regions in klio_core.variables.DATAFLOW_REGIONS.

Returns

If found, input topic of job. Otherwise, None.

Return type

str or None

klio_core.dataflow.get_dataflow_client(api_version=None)

Get an initialized DataflowClient.

This function will first check if there is an already initialized client in the global namespace. Otherwise, initialize one then set it in the global namespace to avoid redundant initialization.

Parameters

api_version (str) – Version of Dataflow REST API. Defaults to v1b3.

Returns

A client to interact with the Dataflow REST API.

Return type

DataflowClient