Skip to main content

Natural Language Processing - Episode 3

This episode references the Python script

In the previous episode, you saw how we constructed a model in preparation for Metaflow.
In this lesson, we will construct a basic flow that reads our data and reports a baseline. At the end of this lesson, you will be able to:

  • Operationalize the tasks of loading data and computing a baseline.
  • Run and view tasks with Metaflow.

1Best Practice: Create a Baseline

When creating flows, we recommend starting simple: create a flow that reads your data and reports a baseline metric. This way, you can ensure you have the right foundation to incorporate your model. Furthermore, starting simple helps with debugging.

2Write a Flow

For our baseline flow, we will just have three steps including

  • a start step where we read the data,
  • a baseline step, and
  • an end step that will be a placeholder for now.

Below is a detailed explanation of each step:

  1. Read data from a parquet file in the start step.
    • We use pandas to read train.parquet.
    • Notice how we are assigning the training data to self.df and the validation data to self.valdf this stores the data as an artifact in Metaflow, which means it will be versioned and saved in the artifact store for later retrieval. Furthermore, this allows you to pass data to another step. The prerequisite for being able to do this is that the data you are trying to store must be pickleable.
    • We log the number of rows in the data. It is always a good idea to log information about your dataset for debugging.
  2. Compute the baseline in the baseline step.
    • The baseline step records the performance metrics (accuracy and ROC AUC score) that result from classifying all examples with the majority class. This will be our baseline against which we evaluate our model.
  3. Print the baseline metrics in the end step.
    • This is just a placeholder for now, but also serves to illustrate how you can retrieve artifacts from any step.
from metaflow import FlowSpec, step, Flow, current

class BaselineNLPFlow(FlowSpec):

def start(self):
"Read the data"
import pandas as pd
self.df = pd.read_parquet('train.parquet')
self.valdf = pd.read_parquet('valid.parquet')
print(f'num of rows: {self.df.shape[0]}')

def baseline(self):
"Compute the baseline"
from sklearn.metrics import accuracy_score, roc_auc_score
baseline_predictions = [1] * self.valdf.shape[0]
self.base_acc = accuracy_score(
self.valdf.labels, baseline_predictions)
self.base_rocauc = roc_auc_score(
self.valdf.labels, baseline_predictions)

def end(self):
msg = 'Baseline Accuracy: {}\nBaseline AUC: {}'
round(self.base_acc,3), round(self.base_rocauc,3)

if __name__ == '__main__':

3Run the Flow

python run
     Workflow starting (run-id 1666721128115595):
[1666721128115595/start/1 (pid 53035)] Task is starting.
[1666721128115595/start/1 (pid 53035)] num of rows: 20377
[1666721128115595/start/1 (pid 53035)] Task finished successfully.
[1666721128115595/baseline/2 (pid 53038)] Task is starting.
[1666721128115595/baseline/2 (pid 53038)] Task finished successfully.
[1666721128115595/end/3 (pid 53041)] Task is starting.
[1666721128115595/end/3 (pid 53041)] Baseline Accuracy: 0.773
[1666721128115595/end/3 (pid 53041)] Baseline AUC: 0.5
[1666721128115595/end/3 (pid 53041)] Task finished successfully.

In the next lesson, you will learn how to incorporate your model into the flow as well as deal with branching for parallel runs.