Skip to main content

Schedule Flows on AWS Step Functions

Question​

How can I schedule flows to run at a specific time on AWS?

Solution​

There is a Metaflow decorator for that!

Outerbounds user note

If you run Metaflow on Outerbounds, click here to follow the twin of this guide which uses Argo workflows and does not depend on AWS.

1Scheduling Flows​

You can use Metaflow's @schedule flow-level decorator to run on AWS Step Functions where Metaflow automatically maps a FlowSpec onto an AWS Step Functions state machine. Alternatively, you can schedule flows with Argo Workflows.

After deploying the script containing your flow to AWS Step Functions the execution of a FlowSpec can happen on any event-based trigger you setup or a time-based trigger defined with Metaflow's @schedule decorator.

2Run Flow​

This flow is scheduled to run daily. Notice Metaflow's @schedule decorator has arguments that determine when the flow is run. Time based triggers you can use include:

  • @schedule(weekly=True) runs the workflow on Sundays at midnight.
  • @schedule(daily=True) runs the workflow every day at midnight.
  • @schedule(hourly=True) runs the workflow every hour.
  • @schedule(cron='0 10 * * ? *') runs the workflow at the given Cron schedule, in this case at 10am UTC every day. You can use the rules defined here to define the schedule for the cron option.
schedule_flow_aws.py
from metaflow import FlowSpec, schedule, step
from datetime import datetime

@schedule(daily=True)
class DailyFlowAWS(FlowSpec):

    @step
    def start(self):
        now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        print('time is %s' % now)
        self.next(self.end)

    @step
    def end(self):
        pass

if __name__ == '__main__':
    DailyFlowAWS()
python schedule_flow_aws.py --with retry step-functions create
    Deploying DailyFlowAWS to AWS Step Functions...
    It seems this is the first time you are deploying DailyFlowAWS to AWS Step Functions.
    
    A new production token generated.
    
    The namespace of this production flow is
        production:dailyflowaws-0-liek
    To analyze results of this production flow add this line in your notebooks:
        namespace("production:dailyflowaws-0-liek")
    If you want to authorize other people to deploy new versions of this flow to AWS Step Functions, they need to call
        step-functions create --authorize dailyflowaws-0-liek
    when deploying this flow to AWS Step Functions for the first time.
    See "Organizing Results" at https://docs.metaflow.org/ for more information about production tokens.
    
    State Machine DailyFlowAWS for flow DailyFlowAWS pushed to AWS Step Functions successfully.
    
    What will trigger execution of the workflow:
        This workflow triggers automatically via a cron schedule DailyFlowAWS defined in AWS EventBridge.

After running the above command your flow will be triggered daily!

3Manually Trigger Flow​

You can manually trigger the flow at any time:

python schedule_flow_aws.py step-functions trigger
    Workflow DailyFlowAWS triggered on AWS Step Functions (run-id sfn-ef136a09-7082-4826-ac41-4704e145ebe2).

You can also interact with the flow through your AWS Step Functions console:

Further Reading​