Skip to main content

Speed Up Conda

Further Reading


Managing dependencies with Metaflow


Single flow, multiple developers

Question

Why is conda "bootstrapping environment" for so long? Can I speed this up?

Solution

When you run a flow for the first time, each conda environment you define needs to gather and install the dependencies which can take a long time. This will be faster the second time you run the flow.

To speed up conda, you can install mamba which is a reimplementation of conda's package manager in C++.

To install mamba:

  1. Install conda. Here is the command to download on MacOS or Linux. You can find Windows installation instructions here.
bash Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
  1. Install mamba.
conda install mamba -n base -c conda-forge
  1. Set Metaflow's dependency resolver environment variable.
export METAFLOW_CONDA_DEPENDENCY_RESOLVER=mamba

You can also set the METAFLOW_CONDA_DEPENDENCY_RESOLVER environment variable in your Metaflow config which by default lives at $HOME/.metaflowconfig/config.json.