Flow Parameters

1Why Parameters?

Sometimes you need to pass a value to a flow at runtime. For example, in a machine learning system with automated aspects, you may not know how to select an input to the model training flow, such as a hyperparameter search space, until it is time to run the flow.

To address these cases you can pass values to a metaflow.Parameter in your flow. When you write a flow you can define which parameters the flow will take. Then you can pass corresponding values to the command that runs your flow:

python <FLOW SCRIPT> run --<PARAM NAME> <PARAM VALUE>

2Write a Flow with Parameters

Using parameters is a convenient way to quickly iterate in prototyping. For example, you might want to change a hyperparameter like a model's learning rate. This flow adds a parameter called learning_rate to the MinimumFlow example from episode 1.

parameter_flow.py

from metaflow import FlowSpec, step, Parameter

class ParameterizedFlow(FlowSpec):
    
    learning_rate = Parameter('lr', default=.01)
    
    @step
    def start(self):
        self.next(self.end)
    
    @step
    def end(self):
        print("Learning rate value is {}".format(self.learning_rate))

if __name__ == "__main__":
    ParameterizedFlow()

3Run the Flow

You can run the flow using the generic command and the default value will be used:

python parameter_flow.py run

     Workflow starting (run-id 1666720680013279):
     [1666720680013279/start/1 (pid 52610)] Task is starting.
     [1666720680013279/start/1 (pid 52610)] Task finished successfully.
     [1666720680013279/end/2 (pid 52613)] Task is starting.
     [1666720680013279/end/2 (pid 52613)] Learning rate value is 0.01
     [1666720680013279/end/2 (pid 52613)] Task finished successfully.
     Done!

Or you can pass in the parameter's value at run time:

python parameter_flow.py run --lr .001

     Workflow starting (run-id 1666720681347916):
     [1666720681347916/start/1 (pid 52631)] Task is starting.
     [1666720681347916/start/1 (pid 52631)] Task finished successfully.
     [1666720681347916/end/2 (pid 52634)] Task is starting.
     [1666720681347916/end/2 (pid 52634)] Learning rate value is 0.001
     [1666720681347916/end/2 (pid 52634)] Task finished successfully.
     Done!

Congratulations on finishing the first season of the tutorial!

In this season you have seen:

• In Metaflow, a flow defines the structure of your code.
• You can create flows by wrapping a FlowSpec object in a python script.
• Flows can store data using the self keyword. These objects are referred to as artifacts.
• Flow data can be accessed from previous steps, or after the flow using the client API.
• You can pass values to your flows at run time using parameters.

In the next season, you will see how to apply many of these concepts in machine learning workflows.