Use Multiple Metaflow Configuration Files
How can I use multiple Metaflow configuration files?
Metaflow has many configuration options. You can manage them with environment variables or through a configuration file that by default lives at
~/.metaflowconfig/config_default.json. When you run a flow, Metaflow will first try to pull these values from environment variables, and then from the Metaflow config JSON file.
This page describes the more portable config files approach through the simplest possible example. If you want to go through the variables common to both settings you can see their definitions in the Metaflow config file.
1The Default Mode
After you install Metaflow you can run
to see a list of configuration options.
Your default configuration file will reside in
METAFLOW_PROFILE variable determines which configuration file is looked at. For example, if you set
and try to run a flow, it will produce an error saying
Unable to locate METAFLOW_PROFILE 'local' in '~/.metaflowconfig').
2Your Custom Config
To address this error, you can create a file
~/.metaflowconfig/config_local.json with an empty JSON object in it.
Now you will be able to run flows with the
local profile. After setting this file you can either use the environment variable for the remainder of your terminal session (as described before) or you can do flow run commands like this:
METAFLOW_PROFILE=local python <YOUR FLOW SCRIPT NAME>.py run
3Extending Custom Config
Later you can extend this JSON object to configure Metaflow to:
- connect to the cloud and/or Kubernetes resources.
- read and write data to the cloud.
- connect to Argo, AWS Step Functions, or Airflow for orchestration.
- tell conda which dependencies are needed and ship the environments.
Alternatively, you can use the commands that appear after
metaflow configure like
metaflow configure aws to go through a guided question-answer for how to set these variables in the JSON file.