Skip to main content

Load Local Data with IncludeFile


How do I load data from a local directory structure on AWS Batch using Metaflow's IncludeFile?


When using Metaflow's @batch decorator as a compute environment for a step, there are several options for accessing data. This page will show how to use metaflow.IncludeFile to access a file on AWS Batch or Kubernetes.

1Acquire Data

The example will access this CSV file from a step the runs on AWS Batch in several ways including:

1, 2, 3
4, 5, 6

2Run Flow

This flow shows how to:

  • Include flow artifacts with self.little_data.
  • Use artifacts to access the contents of a local file on AWS Batch.
from metaflow import FlowSpec, step, IncludeFile, batch

class IncludeFileFlow(FlowSpec):
data = IncludeFile('data',

def start(self):

def end(self):
print('Finished reading the data!')

if __name__ == '__main__':
python run
[468/end/2406 (pid 46569)] Task is starting.
[468/end/2406 (pid 46569)] Finished reading the data!
[468/end/2406 (pid 46569)] Task finished successfully.

Further Reading