On this page

`klio.transforms` Subpackage¶

Collection of commonly used transforms and transform-utilities.

Decorators ¶

`handle_klio`	Serialize & deserialize incoming PCollections as a KlioMessage.
`timeout`	Run the decorated method/function with a timeout in a separate process.
`retry`	Retry a decorated method/function on failure.
`set_klio_context`	Set `KlioContext` to the class instance.
`inject_klio_context`	Provide `KlioContext` as the first argument to a decorated method/func.
`serialize_klio_message`	Serialize/deserialize incoming PCollections as a KlioMessage.
`profile`	Decorator to mark a function/method for profiling.

Helpers ¶

`KlioMessageCounter`	Helper transform to count elements.
`KlioGcsCheckInputExists`	Klio transform to check input exists in GCS.
`KlioGcsCheckOutputExists`	Klio transform to check output exists in GCS.
`KlioFilterPing`	Klio transform to tag outputs if in ping mode or not.
`KlioFilterForce`	Klio transform to tag outputs if in force mode or not.
`KlioWriteToEventOutput`	Klio composite transform to write to the configured event output.
`KlioDrop`	Klio DoFn to log & drop a KlioMessage.
`KlioCheckRecipients`	Check if current job should handle a received v2 message.
`KlioUpdateAuditLog`	Update a KlioMessage’s audit log to include current job.
`KlioDebugMessage`	Log KlioMessage.
`KlioSetTrace`	Insert a Python debugger trace point.
`KlioTriggerUpstream`	Trigger upstream job from current job with a given `KlioMessage`.

I/O ¶

`KlioReadFromText`	Read from a local or GCS file with each new line as a `KlioMessage.data.element`.
`KlioReadFromBigQuery`	Read data from BigQuery.
`KlioReadFromAvro`	Read avro from a local directory or GCS bucket.
`KlioReadFromPubSub`	Read from a Google Pub/Sub topic or subscription.
`KlioWriteToText`	Write to a local or GCS file with each new line as `KlioMessage.data.element`.
`KlioWriteToBigQuery`	Writes to BigQuery table with each row as `KlioMessage.data.element`.
`KlioWriteToAvro`	Write avro to a local directory or GCS bucket.
`KlioWriteToPubSub`	Write to a Google Pub/Sub topic.
`KlioMissingConfiguration`	Required configuration is missing.

Core ¶

Context related to the currently running job.