Skip to main content

Schedule Flows on AWS Step Functions

Question

How can I schedule flows to run at a specific time on AWS?

Solution

There is a Metaflow decorator for that!

1Scheduling Flows

You can use Metaflow's @schedule flow-level decorator to run on AWS Step Functions where Metaflow automatically maps a FlowSpec onto an AWS Step Functions state machine. Alternatively, you can schedule flows with Argo Workflows.

After deploying the script containing your flow to AWS Step Functions the execution of a FlowSpec can happen on any event-based trigger you setup or a time-based trigger defined with Metaflow's @schedule decorator.

2Run Flow

This flow is scheduled to run daily. Notice Metaflow's @schedule decorator has arguments that determine when the flow is run. Time based triggers you can use include:

  • @schedule(weekly=True) runs the workflow on Sundays at midnight.
  • @schedule(daily=True) runs the workflow every day at midnight.
  • @schedule(hourly=True) runs the workflow every hour.
  • @schedule(cron='0 10 * * ? *') runs the workflow at the given Cron schedule, in this case at 10am UTC every day. You can use the rules defined here to define the schedule for the cron option.
schedule_flow_aws.py
from metaflow import FlowSpec, schedule, step
from datetime import datetime

@schedule(daily=True)
class DailyFlowAWS(FlowSpec):

@step
def start(self):
now = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print('time is %s' % now)
self.next(self.end)

@step
def end(self):
pass

if __name__ == '__main__':
DailyFlowAWS()
python daily_flow_aws.py --with retry step-functions create
    Deploying DailyFlowAWS to AWS Step Functions...
It seems this is the first time you are deploying DailyFlowAWS to AWS Step Functions.

A new production token generated.

The namespace of this production flow is
production:dailyflowaws-0-liek
To analyze results of this production flow add this line in your notebooks:
namespace("production:dailyflowaws-0-liek")
If you want to authorize other people to deploy new versions of this flow to AWS Step Functions, they need to call
step-functions create --authorize dailyflowaws-0-liek
when deploying this flow to AWS Step Functions for the first time.
See "Organizing Results" at https://docs.metaflow.org/ for more information about production tokens.

State Machine DailyFlowAWS for flow DailyFlowAWS pushed to AWS Step Functions successfully.

What will trigger execution of the workflow:
This workflow triggers automatically via a cron schedule DailyFlowAWS defined in AWS EventBridge.

After running the above command your flow will be triggered daily!

3Manually Trigger Flow

You can manually trigger the flow at any time:

python daily_flow_aws.py step-functions trigger
    Workflow DailyFlowAWS triggered on AWS Step Functions (run-id sfn-ef136a09-7082-4826-ac41-4704e145ebe2).

You can also interact with the flow through your AWS Step Functions console:

Further Reading