Skip to main content

Use Multiple Metaflow Configuration Files

Question​

How can I use multiple Metaflow configuration files?

Solution​

Metaflow has many configuration options. You can manage them with environment variables or through a configuration file that by default lives at ~/.metaflowconfig/config_default.json. When you run a flow, Metaflow will first try to pull these values from environment variables, and then from the Metaflow config JSON file.

This page describes the more portable config files approach through the simplest possible example. If you want to go through the variables common to both settings you can see their definitions in the Metaflow config file.

1The Default Mode​

After you install Metaflow you can run metaflow configure to see a list of configuration options.

Your default configuration file will reside in ~/.metaflowconfig/config_default.json. The METAFLOW_PROFILE variable determines which configuration file is looked at. For example, if you set 

export METAFLOW_PROFILE=local

and try to run a flow, it will produce an error saying Unable to locate METAFLOW_PROFILE 'local' in '~/.metaflowconfig').

2Your Custom Config​

To address this error, you can create a file ~/.metaflowconfig/config_local.json with an empty JSON object in it.

Now you will be able to run flows with the local profile. After setting this file you can either use the environment variable for the remainder of your terminal session (as described before) or you can do flow run commands like this:

METAFLOW_PROFILE=local python <YOUR FLOW SCRIPT NAME>.py run

3Extending Custom Config​

Later you can extend this JSON object to configure Metaflow to:

  • connect to the cloud and/or Kubernetes resources.
  • read and write data to the cloud.
  • connect to Argo, AWS Step Functions, or Airflow for orchestration.
  • tell conda which dependencies are needed and ship the environments.

Alternatively, you can use the commands that appear after metaflow configure like metaflow configure aws to go through a guided question-answer for how to set these variables in the JSON file.

Further Reading​