Skip to main content

Set Environment Variables with .env File

Question

How can I use environment variables in a step of my Metaflow flow with .env file?

Solution

One way you can read environment variables in Python is using the PyPi module python_dotenv to load a .env file. You can customize your environment variables as desired by modifying your .env file. If you don't need to use a .env file to manage environment variables, you may consider using Metaflow's environment decorator.

1Create .env File

This .env file will be used in this example:

.env
SUPER_DUPER_SECRET=hi

2Run Flow

The flow shows how to:

  • Use python_dotenv to load contents of a .env file in a Python script.
  • Use a secret from the .env file in the read_locally step.
  • Use a secret from the .env file in the read_in_container step that runs with Metaflow's @batch decorator.
    • Note you can also use remote compute using the @kubernetes decorator.
    • The Dockerfile used to build the image used in the batch step is shown below the flow.
set_environment_variables_file.py
from metaflow import FlowSpec, step, batch
from dotenv import load_dotenv
import os

IMAGE = "public.ecr.aws/outerbounds/dotenv:latest"

class EnvVarFlow(FlowSpec):

@step
def start(self):
self.next(self.read_locally, self.read_in_container)

@step
def read_locally(self):
secret = os.getenv("SUPER_DUPER_SECRET")
print(f"secret message: {secret} " + \
"from local environment")
self.next(self.join)

@batch(image=IMAGE, cpu=1)
@step
def read_in_container(self):
secret = os.getenv('SUPER_DUPER_SECRET')
print(f"secret message: {secret} " + \
"from a container")
self.next(self.join)

@step
def join(self, inputs):
self.next(self.end)

@step
def end(self):
pass

if __name__ == "__main__":
load_dotenv()
EnvVarFlow()
Dockerfile
FROM --platform=linux/amd64 python:3

WORKDIR /usr/src/app

COPY .env .env

RUN pip install --no-cache-dir --upgrade pip && \
pip install --no-cache-dir python_dotenv
python set_environment_variables_file.py run
    ...
[111/read_locally/499 (pid 18752)] Task is starting.
[111/read_locally/499 (pid 18752)] secret message: hi from local environment
[111/read_locally/499 (pid 18752)] Task finished successfully.
...
[111/read_in_container/500 (pid 18755)] Task is starting.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status SUBMITTED)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status RUNNABLE)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status STARTING)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting (status RUNNING)...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Setting up task environment.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Downloading code package...
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Code package downloaded.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task is starting.
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] secret message: hi from a container
[111/read_in_container/500 (pid 18755)] [34a6742a-5f14-402d-9871-2a2e6d18c4be] Task finished with exit code 0.
[111/read_in_container/500 (pid 18755)] Task finished successfully.
...

Further Reading