05 Workflows

The resource-aware MLOps loop

Large teacher models are distilled into one small student, trained for INT8 and served on CPU, shipped as signed bundles into each tenant's own cloud, and watched in production until drift starts the loop again.

The problem

A large model is accurate but costly: served on GPUs in every tenant's cloud, its cost and latency grow with each new client, and a model file copied between clouds can be swapped or corrupted on the way without anyone noticing.

Step by step

  1. Prepare the features

    Raw events and content are turned into features from one shared set of definitions, so training and inference compute every input the same way.

    What can go wrong: A feature computed one way in training and another way in serving makes a model that scores well offline and misbehaves live.

  2. Run the teachers once

    Large video, audio and text models label the training data with their outputs. Those outputs are computed once on a GPU and cached, so every training run after that reads the cache instead of reloading the teachers.

    What can go wrong: A cache built from an old version of the data quietly trains the student on stale targets; the cache is keyed by the data version so it is rebuilt when the data changes.

  3. Distil many teachers into one student

    A single compact student learns to match all the teachers at once. One small model replaces several large ones, which is what makes CPU serving possible.

    What can go wrong: Teachers can disagree; weighting one too heavily makes the student inherit its blind spots.

  4. Search the settings, track every run

    An automated search tries combinations of learning rate, loss weights and model size. Each run records its code, data version, settings and scores, so any result can be explained and reproduced.

    What can go wrong: Searching too long against one validation set overfits to it; a final check on data the search never saw keeps it honest.

  5. Train for INT8

    Quantization-aware training simulates 8-bit arithmetic while the student trains, so it learns weights that keep their accuracy once they are stored as INT8: a model a quarter of the size, fast on an ordinary CPU.

    What can go wrong: Quantizing after training instead can cost accuracy the model never had the chance to recover.

  6. Gate on held-out data

    The INT8 candidate is scored on data it never saw, for accuracy and F1, overall and per group, against the model in production. Scores are read beside the balance of the data, since an underrepresented class caps the accuracy any model can reach on it. A candidate that is worse, or unfair to a group, goes back to the data.

    What can go wrong: An average score hides a group the model fails; without per-group results that failure ships.

  7. Sign the bundle and publish it

    The approved weights are packed into a bundle with their metrics and signed with an Ed25519 key, then registered in the model hub, which keeps every version and its history.

    What can go wrong: A leaked signing key lets anyone publish a model; the key stays offline and the hub accepts only bundles signed by it.

  8. Deliver into each tenant's cloud

    Each tenant's cloud pulls new bundles from the hub and checks the signature before loading anything, so a model that was altered on the way, or never published, is refused.

    What can go wrong: Loading a bundle before its signature is checked turns the hub into a way to run any file in every tenant.

  9. Test on a share of traffic

    The new model answers a small share of requests while the current one answers the rest. It takes over only if it wins on quality without breaking the latency budget.

    What can go wrong: Ending the test early can crown a model that only got lucky.

  10. Serve on CPU

    One service runs all the models on CPU, with no GPU runtime, inside a fixed latency budget. The weights are memory-mapped, so a new model loads without copying them.

    What can go wrong: A model that fits the budget on average can still miss it under load; latency is measured at the slow tail, not the mean.

  11. Watch drift and latency

    Monitoring compares live inputs and predictions with the training data and tracks how long each response takes. When drift stays past its limit, an alert fires and starts the loop again from the features.

    What can go wrong: A limit set too low causes alert fatigue; set too high, drift is caught late.

Weak points and compensations

  • Weak point A small student keeps less of what the large teachers know, and loses most on rare cases.

    Compensation Several teachers train it at once through soft targets, which carry how sure each teacher is and not just its answer; the student is scored class by class on held-out data, and a class that falls behind gets more of its examples in the next run.

  • Weak point INT8 weights round away detail that a full-precision model keeps.

    Compensation Quantization-aware training lets the student learn around the rounding while it trains, and the release gate compares the INT8 model with its full-precision copy on the same held-out data, failing a bundle that loses more than an agreed margin.

  • Weak point A class with little training data caps the accuracy any model can reach on it.

    Compensation Evaluation reports each class's accuracy beside its share of the data, so a weak class is traced to its data rather than to the model, and the fix is more or better examples of it.

  • Weak point A latency budget on CPU limits how large the model can grow.

    Compensation Truncatable embeddings let one model serve a short, fast vector where speed matters and its full length where precision does, so accuracy is spent only where it pays.

  • Weak point Accuracy measured offline fades as live data drifts away from the training data.

    Compensation An A/B test confirms the gain on real traffic before a model takes over, and drift monitoring starts retraining before the loss shows in results.

Trade-offs

  • Distillation and INT8 make CPU serving possible and cut cost per tenant, but the student gives up some of the teachers' accuracy on rare cases.
  • Caching teacher outputs makes training cheap to repeat, but the cache has to be rebuilt whenever the data or a teacher changes.
  • Signed bundles in each tenant's cloud keep every tenant's data in its own cloud, but every release has to reach many clouds instead of one.

Stack

  • PyTorch
  • Burn
  • Rust
  • MLflow
  • Ed25519
  • Prometheus

All workflows