A machine learning model creates value not on the day it's built but for as long as it runs reliably in production. Some organisations deploy a new model within hours and monitor it continuously; for others it takes months, and once live the model is forgotten. The difference is MLOps maturity.
Two common models
Google's MLOps model has three levels:
- Level 0 – Manual process: training and deployment are done by hand.
- Level 1 – ML pipeline automation: the model is retrained automatically on new data.
- Level 2 – CI/CD pipeline automation: the pipeline itself is automatically tested and deployed.
Microsoft's model describes the steps in between in more detail and has five levels. We use it as the basis below.
The five levels
Level 0 – No MLOps
- Models are built in notebooks on data scientists' machines.
- Experiments aren't tracked; nobody knows which data a model was trained on.
- Deployment means handing a file over to the software team.
- Production models aren't monitored.
Next step: experiment tracking and a model registry (e.g. MLflow), versioning code and data.
Level 1 – DevOps but no MLOps
- Application code has CI/CD and automated tests.
- But model training is still manual and not reproducible.
- The model itself is added to the application as a "static file".
Next step: move training into a reproducible pipeline and automate data validation.
Level 2 – Automated training
- Training pipelines are defined and reproducible.
- Experiments, parameters and data versions are logged automatically.
- But deployment is still manual, with a long approval chain.
Next step: CI/CD for model deployment, automated tests and one-step rollback.
Level 3 – Automated model deployment
- Validated models are deployed automatically.
- If something goes wrong, you can roll back quickly.
- Monitoring exists, but the retraining decision is still manual.
Next step: monitor data drift and performance, and trigger retraining automatically.
Level 4 – Full MLOps automated operations
- Model performance and data drift are monitored continuously.
- When thresholds are breached, retraining, validation and deployment run automatically.
- The whole process is traceable and auditable.
Focus: cost optimisation, governance and extending the platform across the organisation.
Summary of the levels
| Area | Level 0 | Level 2 | Level 4 |
|---|---|---|---|
| Experiment tracking | None | Automated | Automated and auditable |
| Training | In notebooks, manual | Automated pipeline | Event-triggered pipeline |
| Deployment | Months, manual | Manual, days | Automated, hours |
| Monitoring | None | Reports | Alerts and automated retraining |
| Rollback | Hard | Manual | One step or automatic |
LLM applications and LLMOps
Generative AI applications bring new needs:
- Prompt versioning: prompts should be versioned and tested like code.
- Evaluation sets: output quality should be measured with every change; methods such as LLM-as-a-judge can help.
- Tracing: you should be able to see which steps a request went through and which documents were used.
- Cost and safety: token usage, sensitive data leakage and misuse attempts should be monitored.
These needs should be assessed alongside the classic MLOps levels; in many organisations LLM applications change faster than classic models.
Which level is right for you?
Not every organisation needs Level 4. When deciding, consider:
- Number of models: a heavy platform may be unnecessary for a handful of models.
- Update frequency: if data changes fast, automated retraining is critical.
- Risk level: for high-risk models such as credit scoring, traceability and auditability become essential.
Measure your MLOps level
MLTeam's free MLOps Maturity Assessment reviews experiment tracking, training pipelines, deployment, monitoring, LLMOps and team processes in 12 questions, and shows your level (0–4) with recommendations for your weakest areas.
Sources