Decorating Flows

1Why Decorators?

Using Metaflow requires the use of decorators. In Python code, a decorator is a function that takes another function and extends its behavior without the need to modify it directly. You can find an in-depth introduction to decorators here and the documentation here. For now, you just need to know that they make it easier to work with your steps.

In season 1 episode 1 you saw Metaflow's @step decorators in action. This is just the beginning. There are many decorators built into Metaflow and plugins built by community members.

For example, there are function- or step-level decorators including:

• You can use @conda to handle dependency management for a step. This decorator helps you organize dependencies across environments in your machine learning system.
• You can use @batch or @kubernetes to run a step remotely on AWS Batch or a Kubernetes cluster, respectively. A powerful feature of Metaflow is that you can also run many replications of this step in parallel with minimal code changes.

There are also class- or flow-level decorators including:

• You can use @conda_base to handle dependency management for each step in a flow. This is similar to @conda but instead of creating different conda environments for each task, @conda_base will create one environment, package it, and reuse it across each task in the run.
• You can use @schedule to trigger flows automatically when they are deployed on a production orchestrator such as Argo or AWS Step Functions. This decorator helps you run flows at a specific time without needing to intervene. You can also trigger flows at any time using the command line functionality that comes with Metaflow.

2Decorate Your Flow

In this flow, we have a step my_decorated_func to which the @card decorator is applied.

Using @card creates a canvas where you can compose data visualizations using any data produced in that step or prior to it. After running the flow, you will see how to access the contents of the card from the command line.

decorator_flow.py

from metaflow import FlowSpec, step, card


class DecoratorFlow(FlowSpec):
    
    @step
    def start(self):
        self.next(self.my_decorated_func)
        
    @card
    @step
    def my_decorated_func(self):
        self.data = [1, 2, 3]
        self.next(self.end)
    
    @step
    def end(self):
        print("Flow is done!")

if __name__ == "__main__":
    DecoratorFlow()

3Run the Flow

python decorator_flow.py run

     Workflow starting (run-id 1666720670441830):
     [1666720670441830/start/1 (pid 52568)] Task is starting.
     [1666720670441830/start/1 (pid 52568)] Task finished successfully.
     [1666720670441830/my_decorated_func/2 (pid 52571)] Task is starting.
     [1666720670441830/my_decorated_func/2 (pid 52571)] Task finished successfully.
     [1666720670441830/end/3 (pid 52578)] Task is starting.
     [1666720670441830/end/3 (pid 52578)] Flow is done!
     [1666720670441830/end/3 (pid 52578)] Task finished successfully.
     Done!

4View the Card

Now that we have run @card for the my_decorated_func step, we can use the following command to visualize our flow:

python decorator_flow.py card view my_decorated_func

    Resolving card: DecoratorFlow/1666720670441830/my_decorated_func/2

You should now see a browser tab open where you can inspect flow results.

Using @card in this way is one example of a decorator extending the functionality of a step. There are many other decorators you can use to extend your steps in Metaflow. You can view a list of all step decorators here and all flow decorators here.

In the next episode, you will learn how to store and analyze the results of your flows.