Skip to main content

Specify Conda Channels

Question

How can I specify conda channels in my Metaflow flows and steps?

Solution

You can change default conda channels by specifying the channel in the Metaflow @conda or @conda_base decorator, or by using the CONDA_CHANNELS environment variable.

1Set the Channel Explicitly in a Decorator

For example, this flow shows how to install a specific version of PyTorch from the pytorch conda channel using an argument to the @conda step-level decorator.

This is specified like:

@conda(libraries={
"conda-channel::python-module": "module-version",
"another-conda-channel::another-python-module": "module-version",
...
})
specify_conda_channel_step.py
from metaflow import FlowSpec, step, conda

class SpecifyChannelsStep(FlowSpec):

@step
def start(self):
self.next(self.make_pytorch_model)

@conda(libraries={"pytorch::pytorch": "1.11.0"})
@step
def make_pytorch_model(self):
import torch
self.next(self.end)

@step
def end(self):
pass

if __name__ == "__main__":
SpecifyChannelsStep()
python specify_conda_channel_step.py --environment=conda run
Workflow starting (run-id 671):
[671/start/3418 (pid 88727)] Task is starting.
[671/start/3418 (pid 88727)] Task finished successfully.
[671/make_pytorch_model/3419 (pid 88800)] Task is starting.
[671/make_pytorch_model/3419 (pid 88800)] Task finished successfully.
[671/end/3420 (pid 88930)] Task is starting.
[671/end/3420 (pid 88930)] Task finished successfully.
Done!

The above example uses the pytorch conda channel, but you can use any conda channel you'd like including a private one:

@conda(libraries={"my-private-channel::pandas": "0.22.0"})

2Set Global Channels with Environment Variables

You can also set an environment variable called CONDA_CHANNELS. For example you can run a very similar flow to the previous one with the pytorch module from the pytorch channel and use the CONDA_CHANNELS environment variable to tell Metaflow to install the rest of the packages from a different channel, conda-forge in this case. Also notice that instead of using the @conda step-level decorator, this example uses the @conda_base decorator that locks dependencies for the entire flow.

specify_conda_channel_flow.py
from metaflow import FlowSpec, step, conda_base

@conda_base(libraries={"pytorch::pytorch": "1.11.0",
"boto3": "1.24.4"},
python="3.8.0")
class SpecifyChannelsFlow(FlowSpec):

@step
def start(self):
self.next(self.make_pytorch_model)

@step
def make_pytorch_model(self):
import torch
import boto3
self.next(self.end)

@step
def end(self):
pass

if __name__ == "__main__":
SpecifyChannelsFlow()
CONDA_CHANNELS=conda-forge \
python specify_conda_channel_flow.py --environment=conda run
Workflow starting (run-id 673):
[673/start/3426 (pid 89382)] Task is starting.
[673/start/3426 (pid 89382)] Task finished successfully.
[673/make_pytorch_model/3427 (pid 89390)] Task is starting.
[673/make_pytorch_model/3427 (pid 89390)] Task finished successfully.
[673/end/3428 (pid 89398)] Task is starting.
[673/end/3428 (pid 89398)] Task finished successfully.
Done!

Further Reading

For help or feedback, please join Metaflow Slack. To suggest an article, you may open an issue on GitHub.

Join Slack