Skip to main content

Add and Remove Tags

Question

How can I add and remove tags to flow runs programmatically?

Solution

You can do this within a flow or using the Client API.

1Run Flow

This flow shows how to:

  • Load a dataset.
  • Train a Scikit-learn model.
  • Evaluate the model on a test set.
    • Tag the model as a production_candidate if the model score is greater than accuracy_threshold.
add_remove_tags_programmatically.py
from metaflow import FlowSpec, step, Flow, current, Parameter

class ModelTaggingFlow(FlowSpec):

max_depth = Parameter('max-depth', default=2)
tag_msg = 'Tagging run {} as a promising model'
accuracy_threshold = 0.85

@step
def start(self):
from sklearn import datasets
from sklearn.model_selection import train_test_split
data = datasets.load_wine()
data = train_test_split(data['data'],
data['target'],
random_state = 42)
self.X_train = data[0]
self.X_test = data[1]
self.y_train = data[2]
self.y_test = data[3]
self.next(self.train)

@step
def train(self):
from sklearn.tree import DecisionTreeClassifier
self.params = {
'max_leaf_nodes': None,
'max_depth': self.max_depth,
'max_features' : 'sqrt',
'random_state': 0
}
self.model = DecisionTreeClassifier(**self.params)
self.model.fit(self.X_train, self.y_train)
self.next(self.eval_and_tag)

@step
def eval_and_tag(self):
from sklearn.metrics import (accuracy_score,
classification_report)
self.pred = self.model.predict(self.X_test)
self.accuracy = float(
accuracy_score(self.y_test, self.pred))
print(self.accuracy)
if self.accuracy > self.accuracy_threshold:
print(self.tag_msg.format(current.run_id))
run = Flow(current.flow_name)[current.run_id]
run.add_tag('promising model')
self.next(self.end)

@step
def end(self):
pass

if __name__ == '__main__':
ModelTaggingFlow()
python add_remove_tags_programmatically.py run
     Workflow starting (run-id 1659643565956220):
[1659643565956220/start/1 (pid 20307)] Task is starting.
[1659643565956220/start/1 (pid 20307)] Task finished successfully.
[1659643565956220/train/2 (pid 20310)] Task is starting.
[1659643565956220/train/2 (pid 20310)] Task finished successfully.
[1659643565956220/eval_and_tag/3 (pid 20313)] Task is starting.
[1659643565956220/eval_and_tag/3 (pid 20313)] 0.8666666666666667
[1659643565956220/eval_and_tag/3 (pid 20313)] Tagging run 1659643565956220 as a promising model
[1659643565956220/eval_and_tag/3 (pid 20313)] Task finished successfully.
[1659643565956220/end/4 (pid 20316)] Task is starting.
[1659643565956220/end/4 (pid 20316)] Task finished successfully.
Done!
python add_remove_tags_programmatically.py run --max-depth 6
     Workflow starting (run-id 1659643570153989):
[1659643570153989/start/1 (pid 20322)] Task is starting.
[1659643570153989/start/1 (pid 20322)] Task finished successfully.
[1659643570153989/train/2 (pid 20325)] Task is starting.
[1659643570153989/train/2 (pid 20325)] Task finished successfully.
[1659643570153989/eval_and_tag/3 (pid 20328)] Task is starting.
[1659643570153989/eval_and_tag/3 (pid 20328)] 0.8888888888888888
[1659643570153989/eval_and_tag/3 (pid 20328)] Tagging run 1659643570153989 as a promising model
[1659643570153989/eval_and_tag/3 (pid 20328)] Task finished successfully.
[1659643570153989/end/4 (pid 20331)] Task is starting.
[1659643570153989/end/4 (pid 20331)] Task finished successfully.
Done!

2Observe Model Scores

You can use the client API to get the latest flow runs. Here is a way to list the accuracy value of each Run.

from metaflow import Flow
flow = Flow('ModelTaggingFlow')
tag = 'promising model'
runs = list(flow.runs(tag))
print("All models tagged with `{}`:".format(tag))
for run in runs:
acc = round(100 * run.data.accuracy, 2)
print("\tRun {}: {}% Accuracy".format(run.id, acc))
    All models tagged with `promising model`:
Run 1659643570153989: 88.89% Accuracy
Run 1659643565956220: 86.67% Accuracy
Run 1659643254562464: 86.67% Accuracy
Run 1659643511759883: 88.89% Accuracy
Run 1659643507482620: 86.67% Accuracy
Run 1659643258636203: 88.89% Accuracy

3Update Tags Using the Client API

You can use the run.add_tag, run.remove_tag or run.replace_tag functions to change a Run tag.

These lines will add the production candidate tag for each promising model with an 87% accuracy score.

flow = Flow('ModelTaggingFlow')
runs = list(flow.runs('promising model'))
for run in runs:
if run.data.accuracy > .87:
run.add_tag('production candidate')

Now you can see the model accuracy only for these models. This can be a useful pattern when reviewing models or testing and promoting them to production.

flow = Flow('ModelTaggingFlow')
tag = 'production candidate'
runs = list(flow.runs(tag))
print("All models tagged `{}`:".format(tag))
for run in runs:
acc = round(100 * run.data.accuracy, 2)
print("\tRun {}: {}% Accuracy".format(run.id, acc))
    All models tagged `production candidate`:
Run 1659643570153989: 88.89% Accuracy
Run 1659643511759883: 88.89% Accuracy
Run 1659643258636203: 88.89% Accuracy

Further Reading

For help or feedback, please join Metaflow Slack. To suggest an article, you may open an issue on GitHub.

Join Slack