Files of a Klio Job

There are a handful of files that are required for a Klio job, which the klio job create command automatically generates for you.

$ tree

.
├── Dockerfile
├── MANIFEST.in
├── README.md
├── __init__.py
├── job-requirements.txt
├── klio-job.yaml
├── run.py
├── setup.py
├── test_transforms.py
└── transforms.py

0 directories, 8 files

Python Files

run.py and transforms.py jointly define the logic of your job (the pink box of “your code” in the job overview diagram).

test_transforms.py gives you a place to test the logic of your job.

__init__.py helps the Python executable find the path where the other Python files are.

setup.py file defines how the job should be packaged so the configured runner can appropriately install it. This is also where job-specific system-level dependencies should be declared. See setup.py for more information.

Dependency Declaration

job-requirements.txt is a standard pip requirements file where you declare the Python packages needed in your Klio job.

Dockerfile describes a Docker image that will be used for launching a Klio job, regardless of the configured runner. It is setup to install the packages you specify in job-requirements.txt.

Configuration

klio-job.yaml contains configuration that tells a runner how to run your job. It also stores Klio job-level information like event and data I/O configuration, metrics, etc. Read more about a job’s configuration here.

Other Files

MANIFEST.in declares files needed for job installation (i.e. job-requirements.txt) but not needed at runtime. See MANIFEST.in for more information.

README.md is initially populated with setup instructions on how to get a job running, but should be used however needed.