Skip to main content

Kubernetes on AWS - Details

Here are key technical details about the Metaflow deployment on AWS Kubernetes.

AWS Services List

The terraform template will deploy these services in your AWS account:

  • Amazon S3 - A dedicated private bucket to serve as a centralized storage backend.
  • AWS Fargate and Amazon Relational Database Service - A Metadata service running on AWS Fargate with a PostgresSQL DB on Amazon Relational Database Service to log flow execution metadata
  • AWS EKS cluster - an AWS-managed Kubernetes cluster to execute Metaflow tasks

Argo Workflows

In addition to this, you'll want to install Argo Workflows for scheduling of production runs. The quickstart Kubernetes manifest published by Argo Workflows spins up the following services inside the EKS cluster:

kubectl get services -n argo
argo-server ClusterIP <none> 2746/TCP 32m
httpbin ClusterIP <none> 9100/TCP 32m
minio ClusterIP <none> 9000/TCP,9001/TCP 32m
postgres ClusterIP <none> 5432/TCP 32m
workflow-controller-metrics ClusterIP <none> 9090/TCP 32m