Current Status
Klio is currently under rapid development. This means that APIs and features will evolve. It is recommended that teams who adopt Klio today upgrade their installation as new releases become available, as backwards compatibility is not yet guaranteed.
git
Python version 3.6, 3.7, or 3.8 (recommended installation via pyenv)
Docker
Google Cloud SDK (setup instructions below)
pipx (suggested) or virtualenv (pyenv-virtualenv when using pyenv).
Attention
A Klio streaming job is dependent on Google Cloud Pub/Sub and Google Cloud Storage (GCS).
To create the job’s relevant dependencies, you need to set up an account with Google Cloud Platform, create a project and enable those services for said project.
You can begin that process here.
Once you have set up your account on Google Cloud Platform (GCP), you will need to install and setup the Google Cloud SDK (a.k.a. gcloud).
gcloud
Follow the recommended install instructions. Once it has installed, set up the Google Cloud SDK by running:
$ gcloud init
Next you will be asked to choose the account and project ID you would like to use. Enter in the account you used to set up GCP and the project you created which is enabled with Google Cloud Pub/Sub and Google Cloud Storage.
Lastly, authorize Google Cloud for application and Docker access by running:
$ gcloud auth application-default login $ gcloud auth configure-docker
klio-cli
pipx
Note
pipx is the recommended approach to installation the klio-cli.
Install via pipx:
# create a project directory $ mkdir klio_quickstart && cd klio_quickstart # install klio-cli $ pipx install klio-cli # confirm installation $ klio --version
See an error while installing?
When running pipx install klio-cli, you may come across a dependency mismatch error, something like this:
pipx install klio-cli
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts. We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default. google-api-python-client 1.12.5 requires google-api-core<2dev,>=1.21.0, but you'll have google-api-core 1.20.1 which is incompatible. klio-core 0.2.0 requires google-api-python-client<1.12,>=1.10.0, but you'll have google-api-python-client 1.12.5 which is incompatible. Could not find package klio-cli. Is the name correct?
or like this:
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts. We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default. google-api-python-client 1.12.5 requires google-api-core<2dev,>=1.21.0, but you'll have google-api-core 1.20.1 which is incompatible. klio-core 0.2.0 requires google-api-python-client<1.12,>=1.10.0, but you'll have google-api-python-client 1.12.5 which is incompatible. Internal error with venv metadata inspection. Unable to install klio-cli. Check the name or spec for errors, and verify that it can be installed with pip.
Fix: Try re-running the command with the added argument:
pipx install klio-cli --pip-args="--use-feature=2020-resolver"
Still failing to install? Please let us know by filing an issue.
pipx installs klio-cli into the local user space within its own virtualenv, so it’s accessible no matter the current working directory or if a virtualenv is activated.
virtualenv
pip
If you use pyenv, it’s highly recommended to then use pyenv-virtualenv. Once installed, follow option 3 instead.
First, setup a new virtualenv:
# create a project directory $ mkdir klio_quickstart && cd klio_quickstart # create a new virtualenv within new project directory $ virtualenv klio-cli # activate the new virtualenv $ source klio-cli/bin/activate (klio-cli) $ # update pip & setuptools within the virtualenv (klio-cli) $ pip install --upgrade pip setuptools
Then install klio-cli:
# within the activate virtualenv (klio-cli) $ pip install klio-cli --use-feature=2020-resolver # confirm installation (klio-cli) $ klio --version
Why the flag “--feature=2020-resolver”?
--feature=2020-resolver
The klio-cli libraries on which it depends have their own dependencies that can sometimes have conflicting versions. The Klio dev team has found that the new pip resolver (as of version 20.2) can help fix those conflicts.
20.2
pyenv-virtualenv
First, setup a new virtualenv with pyenv-virtualenv:
# create a project directory $ mkdir klio_quickstart && cd klio_quickstart # create a new virtualenv called klio-cli $ pyenv virtualenv klio-cli # activate the new virtualenv $ pyenv activate klio-cli (klio-cli) $ # update pip & setuptools within the virtualenv (klio-cli) $ pip install --upgrade pip setuptools