Installation

Current Status

Klio is currently under rapid development. This means that APIs and features will evolve. It is recommended that teams who adopt Klio today upgrade their installation as new releases become available, as backwards compatibility is not yet guaranteed.

Requirements

Google Cloud SDK Setup

Attention

A Klio streaming job is dependent on Google Cloud Pub/Sub and Google Cloud Storage (GCS).

To create the job’s relevant dependencies, you need to set up an account with Google Cloud Platform, create a project and enable those services for said project.

You can begin that process here.

Once you have set up your account on Google Cloud Platform (GCP), you will need to install and setup the Google Cloud SDK (a.k.a. gcloud).

Follow the recommended install instructions. Once it has installed, set up the Google Cloud SDK by running:

$ gcloud init

Next you will be asked to choose the account and project ID you would like to use. Enter in the account you used to set up GCP and the project you created which is enabled with Google Cloud Pub/Sub and Google Cloud Storage.

Lastly, authorize Google Cloud for application and Docker access by running:

$ gcloud auth application-default login
$ gcloud auth configure-docker

Install klio-cli

Option 1: pipx

Note

pipx is the recommended approach to installation the klio-cli.

Install via pipx:

# create a project directory
$ mkdir klio_quickstart && cd klio_quickstart

# install klio-cli
$ pipx install klio-cli

# confirm installation
$ klio --version

See an error while installing?

When running pipx install klio-cli, you may come across a dependency mismatch error, something like this:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

google-api-python-client 1.12.5 requires google-api-core<2dev,>=1.21.0, but you'll have google-api-core 1.20.1 which is incompatible.
klio-core 0.2.0 requires google-api-python-client<1.12,>=1.10.0, but you'll have google-api-python-client 1.12.5 which is incompatible.
Could not find package klio-cli. Is the name correct?

or like this:

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

google-api-python-client 1.12.5 requires google-api-core<2dev,>=1.21.0, but you'll have google-api-core 1.20.1 which is incompatible.
klio-core 0.2.0 requires google-api-python-client<1.12,>=1.10.0, but you'll have google-api-python-client 1.12.5 which is incompatible.
Internal error with venv metadata inspection.

Unable to install klio-cli.
Check the name or spec for errors, and verify that it can be installed with pip.

Fix: Try re-running the command with the added argument:

pipx install klio-cli --pip-args="--use-feature=2020-resolver"

Still failing to install? Please let us know by filing an issue.

pipx installs klio-cli into the local user space within its own virtualenv, so it’s accessible no matter the current working directory or if a virtualenv is activated.

Option 2: virtualenv + pip

Attention

If you use pyenv, it’s highly recommended to then use pyenv-virtualenv. Once installed, follow option 3 instead.

First, setup a new virtualenv:

# create a project directory
$ mkdir klio_quickstart && cd klio_quickstart

# create a new virtualenv within new project directory
$ virtualenv klio-cli

# activate the new virtualenv
$ source klio-cli/bin/activate
(klio-cli) $

# update pip & setuptools within the virtualenv
(klio-cli) $ pip install --upgrade pip setuptools

Then install klio-cli:

# within the activate virtualenv
(klio-cli) $ pip install klio-cli --use-feature=2020-resolver
# confirm installation
(klio-cli) $ klio --version

Why the flag “--feature=2020-resolver”?

The klio-cli libraries on which it depends have their own dependencies that can sometimes have conflicting versions. The Klio dev team has found that the new pip resolver (as of version 20.2) can help fix those conflicts.

Option 3: pyenv-virtualenv + pip

First, setup a new virtualenv with pyenv-virtualenv:

# create a project directory
$ mkdir klio_quickstart && cd klio_quickstart

# create a new virtualenv called klio-cli
$ pyenv virtualenv klio-cli

# activate the new virtualenv
$ pyenv activate klio-cli
(klio-cli) $

# update pip & setuptools within the virtualenv
(klio-cli) $ pip install --upgrade pip setuptools

Then install klio-cli:

# within the activate virtualenv
(klio-cli) $ pip install klio-cli --use-feature=2020-resolver
# confirm installation
(klio-cli) $ klio --version

Why the flag “--feature=2020-resolver”?

The klio-cli libraries on which it depends have their own dependencies that can sometimes have conflicting versions. The Klio dev team has found that the new pip resolver (as of version 20.2) can help fix those conflicts.