Data science projects need a real-time test environment to see if they are useful in real-life situations. Caliban solves this hurdle for a data scientist. It acts as a testing tool that tracks the environmental properties of your project during execution, and reproduces a specific running environment.
data science is an emerging field of advanced data analysis comprising collection, preparation, exploration, analysis, testing, visualisation, management and preservation of large data sets. In some cases, even a well-executed data science project may fail due to limitations in the test environment. Testing the project in all scenarios is mandatory to make it ready-to-use in reality. Caliban is a great open source tool developed by Google. It helps develop research workflows in isolated and reproducible learning environments. Caliban makes it easy to develop any machine learning prototype locally and run the code on your device. A similar code can be tried in a cloud environment for execution on giant machines. It is best suited with JAX, TensorFlow and PyTorch for machine learning workflows.
Caliban requires Python 3.6 or above as a prerequisite. Docker is another important component. Install nvidia-docker2 version, if you have a CPU and want to use Caliban.
Installing Caliban
Once the prerequisites are in place, Caliban can be installed using the following commands:
pipx install caliban pip install caliban
The same commands can also be used to upgrade Caliban. After it’s installed, you can check Caliban on the device by giving the following command:
caliban – help
This open source tool contains several useful sub-commands, which are essential for the execution of any project.
- shell: This generates a Docker image with the required dependencies. This environment is similar to what is available with the code.
- notebook: This starts a Jupyter notebook within a Docker image containing dependencies.
- run: This command packages directory code into a Docker image and executes it locally.
- cloud: This command allows submitting jobs to the cloud and runs inside the same Docker image that is used with the run command.
- build: This builds a Docker image used in the cloud and runs without submitting any code.
- cluster: This command creates Google Kubernetes Engine (GKE) clusters and submits jobs to them.
- status: This displays information about all the jobs submitted by Caliban. It is used to cancel pending jobs, re-build containers, or resubmit a batch after fixing errors.
- stop: This command stops running jobs.
- resubmit: This resubmits the batch of experiments.
Getting started
First, create an empty directory and write a script in Python to train a basic model. Next, create a file called requirements.txt and declare dependency. Caliban automatically makes an entry in the ‘requirements.txt’ file when the code is running. Train your model with the command:
caliban run –nogpu script.py
Check the accuracy of the model and if it is not up to the mark, improve it by setting the default learning rate with the following command:
caliban run --nogpu script.py -- --learning_rate 0.01
You can experiment with broadcasting, which allows many jobs with different sets of parameters to run at the same time:
caliban run --experiment_config experiment.json --nogpu script.py
The job can be submitted to the cloud by giving the following command:
caliban cloud --nogpu script.py -- --learning_rate 0.01
With the cloud you can use on-demand GPUs and TPUs to train models on huge data sets at very high speeds.
As can be seen, Caliban is very helpful for data scientists as it provides a robust testing environment.
