Skip to main content

Kubernetes on AWS - FAQ

I want to use an existing storage account/container, an existing EKS, or DB. What to do?​

The terraform templates provide an all-in experience, spinning up all fresh resources. These templates can be adapted to accommodate existing resources. Please talk to us if you would like assistance in modifying these templates.

How do I use GPU nodes in my EKS cluster?​

Our quickstart terraform templates do support that out of box but can be easily adapted to accommodate GPU-enabled EC2 instances.

How do I change the VM instance types for Metaflow task runner nodes?​

The quickstart terraform templates provided use EKS's autoscaler out of the box. Node auto provisioning should be aware of the resource requirements of incoming pods, which means it will spin up appropriately sized instances as needed. You can override the instance types by modifying the template.

I want finer grain auth on running flows. What to do?​

See here.

Why is my Metaflow flow stuck on k8s pending state forever?​

When Metaflow submits tasks to Kubernetes for execution, there are two scenarios:

  • There are sufficient spare resources to immediately run the task.
  • There are NOT sufficient spare resources right now. EKS autoscaler provisions additional compute nodes to satisfy the requirements of the new task. Once provisioning is complete, the task is run.

If EKS autoscaler can never satisfy the new task and the task will be stuck in pending forever. E.g. if the aggregate hard CPU or memory limit has been reached on the EKSE cluster. To resolve the aggregate limit issue, up the limit in the templates.

How do I access Argo Workflows UI?​

Argo Workflows UI is not accessible from outside the cluster in these templates, but you can use port forwarding to access it.

kubectl port-forward -n argo service/argo-argo-workflows-server 2746:2746

How do I deploy the Metaflow UI?​

Please talk to us for more information about this topic.

Need help?​

The quickest way to get help is our public Slack channel #ask-metaflow. We look forward to your questions and feedback.