05 Workflows

The MLOps loop

A model is built, checked, released in stages and watched in production. When the live data drifts away from what it was trained on, an alert starts the loop again.

Contributed by

Ruchita Singla

The problem

Without a loop, a model is trained once, shipped and forgotten. Nobody can reproduce it, nobody checked who it works badly for, and nobody notices when real-world data changes and its predictions quietly get worse.

Step by step

  1. Prepare the features

    Raw data is cleaned and turned into features. The feature definitions are versioned and shared, so the model sees the same inputs in training and in production.

    What can go wrong: If training and serving compute a feature differently, the model looks good offline and misbehaves live.

  2. Shrink the model

    A large model is used as a teacher, and a smaller student learns to imitate its outputs. The student is cheaper to run and fast enough for a tight response-time budget.

    What can go wrong: The student can lose accuracy on rare cases the teacher handled well.

  3. Search for good settings

    Instead of guessing, an automated search tries many combinations of training settings and keeps the ones that score best on validation data.

    What can go wrong: Searching too long against one validation set can overfit to it.

  4. Record every experiment

    Each run stores its code version, data version, settings, metrics and output. Anyone can later explain why a model behaves the way it does and reproduce it.

    What can go wrong: Untracked runs make a good result impossible to repeat.

  5. Evaluate on held-out data

    The candidate is scored on data it never saw, using accuracy and F1, and compared with the model currently in production. It only moves on if it is at least as good.

    What can go wrong: A single average score can hide weak spots, so evaluation alone is not enough.

  6. Check fairness

    Results are split by group to see whether errors fall unevenly on some people. A large gap between groups blocks the release and sends the work back to the data.

    What can go wrong: Groups with very little data give noisy numbers, so small gaps can be missed or exaggerated.

  7. Register and sign the release

    The approved model is stored in a registry with its metrics and history, then signed. Deployment only accepts signed, registered versions, so every model in production has a known origin.

    What can go wrong: Skipping the signature lets an unverified or tampered file reach production.

  8. Test it on a slice of real traffic

    A small share of live requests goes to the new model while the rest stays on the old one. If the new model wins without making responses slower, it takes over.

    What can go wrong: Ending the test too early can crown a model that only got lucky.

  9. Serve predictions and watch them

    The model answers requests while monitoring tracks how the incoming data and the predictions compare with the training data, and how long each response takes.

    What can go wrong: Latency can be fine while the model's quality silently decays, so both must be watched.

  10. Raise the drift alert

    When the live data moves past an agreed limit for long enough, an alert fires and starts retraining on fresh data. This returns the loop to the first step.

    What can go wrong: A threshold set too low causes alert fatigue; set too high, drift is noticed late.

Trade-offs

  • Distillation lowers cost and latency, but the smaller model gives up some accuracy.
  • Strict fairness and evaluation gates slow releases, but they catch problems before users do.
  • Automatic retraining on drift keeps the model current, but it needs a human check so a bad data change does not train a bad model.

Stack

  • PyTorch
  • Optuna
  • MLflow
  • Prometheus

All workflows