Skip to main content

Specify Conda Channels

Question​

How can I specify conda channels in my Metaflow flows and steps?

Solution​

You can change default conda channels by specifying the channel in the Metaflow @conda or @conda_base decorator, or by using the CONDA_CHANNELS environment variable.

1Set the Channel Explicitly in a Decorator​

For example, this flow shows how to install a specific version of PyTorch from the pytorch conda channel using an argument to the @conda step-level decorator.

This is specified like: 

@conda(libraries={
    "conda-channel::python-module": "module-version",
    "another-conda-channel::another-python-module": "module-version",
    ...
})
specify_conda_channel_step.py
from metaflow import FlowSpec, step, conda

class SpecifyChannelsStep(FlowSpec):
    
    @step
    def start(self):
        self.next(self.make_pytorch_model)
    
    @conda(libraries={"pytorch::pytorch": "1.11.0"})
    @step 
    def make_pytorch_model(self):
        import torch
        self.next(self.end)
        
    @step
    def end(self):
        pass
        
if __name__ == "__main__":
    SpecifyChannelsStep()
python specify_conda_channel_step.py --environment=conda run
     Workflow starting (run-id 671):
     [671/start/3418 (pid 88727)] Task is starting.
     [671/start/3418 (pid 88727)] Task finished successfully.
     [671/make_pytorch_model/3419 (pid 88800)] Task is starting.
     [671/make_pytorch_model/3419 (pid 88800)] Task finished successfully.
     [671/end/3420 (pid 88930)] Task is starting.
     [671/end/3420 (pid 88930)] Task finished successfully.
     Done!

The above example uses the pytorch conda channel, but you can use any conda channel you'd like including a private one: 

@conda(libraries={"my-private-channel::pandas": "0.22.0"})

2Set Global Channels with Environment Variables​

You can also set an environment variable called CONDA_CHANNELS. For example you can run a very similar flow to the previous one with the pytorch module from the pytorch channel and use the CONDA_CHANNELS environment variable to tell Metaflow to install the rest of the packages from a different channel, conda-forge in this case. Also notice that instead of using the @conda step-level decorator, this example uses the @conda_base decorator that locks dependencies for the entire flow.

specify_conda_channel_flow.py
from metaflow import FlowSpec, step, conda_base

@conda_base(libraries={"pytorch::pytorch": "1.11.0", 
                       "boto3": "1.24.4"},
            python="3.8.0")
class SpecifyChannelsFlow(FlowSpec):
    
    @step
    def start(self):
        self.next(self.make_pytorch_model)
    
    @step 
    def make_pytorch_model(self):
        import torch
        import boto3
        self.next(self.end)
        
    @step
    def end(self):
        pass
        
if __name__ == "__main__":
    SpecifyChannelsFlow()
CONDA_CHANNELS=conda-forge \
    python specify_conda_channel_flow.py --environment=conda run
     Workflow starting (run-id 673):
     [673/start/3426 (pid 89382)] Task is starting.
     [673/start/3426 (pid 89382)] Task finished successfully.
     [673/make_pytorch_model/3427 (pid 89390)] Task is starting.
     [673/make_pytorch_model/3427 (pid 89390)] Task finished successfully.
     [673/end/3428 (pid 89398)] Task is starting.
     [673/end/3428 (pid 89398)] Task finished successfully.
     Done!

Further Reading​