Skip to main content

Set Environment Variables with .env File


How can I use environment variables in a step of my Metaflow flow with .env file?


One way you can read environment variables in Python is using the PyPi module python_dotenv to load a .env file. You can customize your environment variables as desired by modifying your .env file. If you don't need to use a .env file to manage environment variables, you may consider using Metaflow's environment decorator.

1Create .env File

This .env file will be used in this example:


2Run Flow

The flow shows how to:

  • Use python_dotenv to load contents of a .env file in a Python script.
  • Use a secret from the .env file in the read_locally step.
  • Use a secret from the .env file in the read_in_container step that runs with Metaflow's @batch decorator.
    • Note you can also use remote compute using the @kubernetes decorator.
    • The Dockerfile used to build the image used in the batch step is shown below the flow.
from metaflow import FlowSpec, step, batch
from dotenv import load_dotenv
import os

IMAGE = ""

class EnvVarFlow(FlowSpec):

def start(self):, self.read_in_container)

def read_locally(self):
secret = os.getenv("SUPER_DUPER_SECRET")
print(f"secret message: {secret} " + \
"from local environment")

@batch(image=IMAGE, cpu=1)
def read_in_container(self):
secret = os.getenv('SUPER_DUPER_SECRET')
print(f"secret message: {secret} " + \
"from a container")

def join(self, inputs):

def end(self):

if __name__ == "__main__":
FROM --platform=linux/amd64 python:3

WORKDIR /usr/src/app

COPY .env .env

RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir python_dotenv
python run
[111/read_locally/499 (pid 18752)] Task is starting.
[111/read_locally/499 (pid 18752)] secret message: hi from local environment
[111/read_locally/499 (pid 18752)] Task finished successfully.
[111/read_in_container/500 (pid 18755)] Task is starting.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status SUBMITTED)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status RUNNABLE)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status STARTING)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status RUNNING)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Setting up task environment.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Downloading code package...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Code package downloaded.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] secret message: hi from a container
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task finished with exit code 0.
[111/read_in_container/500 (pid 18755)] Task finished successfully.

Further Reading