`klio_core.config`¶

Klio configuration handling.

class klio_core.config.KlioConfig(config_dict)¶

Klio config object representation of klio-job.yaml.

Parameters: config_dict (dict) – dictionary representation of configuration as parsed from klio-job.yaml.

job_name¶

Name of Klio job.

Type: str

version¶

Version of Klio job.

Type: int

pipeline_options¶

Apache Beam pipeline-related configuration.

Type: KlioPipelineConfig

job_config¶

Job-related configuration.

Type: KlioJobConfig

as_dict()¶: Return a dictionary representation of the KlioConfig object.

class klio_core.config.KlioJobConfig(config_dict)¶

Job-specific config representing the job_config key of klio-job.yaml.

job_config is both for any user-specific job configuration needed, as well as klio-related configuration (i.e. timeouts, metrics).

See documentation for information on available configuration.

job_name¶

Name of Klio job.

Type: str

version¶

Version of Klio job.

Type: int

allow_non_klio_messages¶

Allow this job to process free-form, non-KlioMessage messages.

Type: bool

blocking¶

Wait for Dataflow job to finish before exiting.

Type: bool

metrics¶

Dictionary representing desired metrics configuration.

Type: dict

events¶

Job event I/O configuration.

Type: KlioIOConfigContainer

data¶

Job data I/O configuration.

Type: KlioIOConfigContainer

Parameters: config_dict (dict) – dictionary representation of job_config as parsed from klio-job.yaml.

as_dict()¶: Return a dictionary representation of the KlioJobConfig object.

Tip

Use this method to access any custom config key/value pairs defined under klio-job.yaml::job_config.

class klio_core.config.KlioPipelineConfig(config_dict)¶

Pipeline-specific config representing the pipeline_options key of klio-job.yaml.

Note

pipeline_options map 1:1 to options supported in Apache Beam and its runners (i.e. Dataflow). See all supported pipeline options for available options.

Any instance attribute not defined in this class but is available in Apache Beam or its runners will still be passed through when running the pipeline.

See documentation for information on available configuration.

Parameters: config_dict (dict) – dictionary representation of pipeline_options as parsed from klio-job.yaml.

job_name¶

Name of Klio job.

Type: str

version¶

Version of Klio job.

Type: int

(remaining attributes): See all supported pipeline options for all available remaining attributes.

as_dict()¶: Return a dictionary representation of the KlioPipelineConfig object.

klio_core.config¶

`klio_core.config`¶