Can I publish KlioMessages from a non-Klio job to a Klio job?

The publisher of KlioMessages does not need to be a Klio job, nor in Python. It just needs to have the KlioMessage proto definition compiled for whatever language in which you plan on publishing messages.

Step 1

Construct the KlioMessage:

from import pubsub
from klio_core.proto import klio_pb2

klio_message = klio_pb2.KlioMessage() = b"s0me-f1le-1D"

Step 2

Add the intended recipients to the KlioMessage depending if this message should be executed in Top-Down Execution or Bottom-Up Execution mode.

If the desired execution mode is top-down, update the recipients to anyone:


Or if the desired execution mode is bottom-up, update the recipients to include the receiving job:

this_klio_job = klio_pb2.KlioJob()

# Needs to match `klio-job.yaml::job_name`
this_klio_job.job_name = "my-job-name"

# If running on Dataflow, this needs to
# match `klio-job.yaml::pipeline_options.project`
this_klio_job.gcp_project = "my-gcp-project"


Step 3

Finally, serialize the KlioMessage into bytes and publish:

# serialize to a bytestring
to_pubsub = klio_message.SerializeToString()

# this would be what is in
# `[].inputs[].topic`
client = pubsub.PublisherClient()
client.publish(topic=JOB_INPUT_TOPIC, data=to_pubsub)