How do I load data from a local directory structure on AWS Batch using Metaflow's
When using Metaflow's
@batch decorator as a compute environment for a step, there are several options for accessing data. This page will show how to use
metaflow.IncludeFile to access a file on AWS Batch or Kubernetes.
The example will access this CSV file from a step the runs on AWS Batch in several ways including:
1, 2, 3
4, 5, 6
This flow shows how to:
- Include flow artifacts with
- Use artifacts to access the contents of a local file on AWS Batch.
from metaflow import FlowSpec, step, IncludeFile, batch
data = IncludeFile('data',
print('Finished reading the data!')
if __name__ == '__main__':
python local_data_on_batch_include.py run
[468/end/2406 (pid 46569)] Task is starting.
[468/end/2406 (pid 46569)] Finished reading the data!
[468/end/2406 (pid 46569)] Task finished successfully.
- Loading and storing data with Metaflow