Six iterative stages take every project from business question to production system — and back again. Mastering each stage is what separates teams that ship from those that don't.
Teams regularly revisit earlier phases — data exploration may reframe the problem, deployment may expose gaps, monitoring may trigger retraining.
Monitoring feeds back into earlier stages — the lifecycle loops rather than ending.
Align stakeholders on the business outcome, success metrics, and operational constraints before a single line of code is written.
Identify, assess, and ingest every relevant source — involve domain experts early to surface non-obvious signals.
EDA surfaces structure, quality issues, and predictive signals; preparation transforms raw data into model-ready features.
Select algorithms, train, tune, and evaluate — always starting with baselines to confirm learnable signal exists.
Transform notebook prototypes into reliable production systems with proper engineering practices at their core.
Deployment is the beginning, not the end — models degrade silently as real-world data drifts from training distributions.
Articulate the business outcome, success criteria, and constraints before touching data. Vague goals — "improve customer experience" — provide insufficient guidance for any technical team. Rushing past this stage is the most common cause of project failure.
Practical questions to answer upfront: which decision does this improve and who currently makes it? Are false positives or false negatives more costly? What are the latency requirements — real-time scoring or overnight batch? Does regulation or customer expectation demand explainability? Is deployment on-premises or cloud?
Concrete success metrics guard against scope drift: lift Net Promoter Score by 5 points, reduce support ticket volume by 20%, double customer retention rate — each measured against an agreed baseline.
Key Insight — Time invested in problem definition pays compound returns: 1–2 weeks of stakeholder alignment prevents months of misdirected effort downstream.
Identify, assess, and ingest every relevant source. Timelines in large organisations regularly exceed estimates — siloed systems and competing team priorities mean data access alone can consume months before a single model is trained.
Brainstorm sources with domain experts, not just engineers. For a customer churn model the obvious sources are customer demographics and transaction histories; the non-obvious ones — support ticket text, community forum engagement — often prove most predictive. Other common sources include website interactions, product usage logs, and external market data.
Examine a real sample early. A data quality assessment should flag: missing values percentage, inconsistencies across systems, update lag, and structural breaks caused by historical regime changes. Discovering these after months of modelling work is expensive.
Exploratory data analysis (EDA) serves multiple purposes simultaneously: building familiarity with the data, uncovering quality issues before they corrupt a model, generating hypotheses about predictive relationships, identifying promising features, and spotting outliers that need investigation.
Preparation steps — handling missing values, categorical encoding, numeric scaling, and creating train/validation/test splits — are mechanical. Feature engineering is not. Transforming raw transactions into average order value, purchase frequency, and days since last purchase for an e-commerce churn model requires domain understanding and creativity. This stage rewards practitioners who combine technical skill with business intuition.
Select algorithms, train on historical data, tune hyperparameters, and evaluate rigorously. Strong foundations in stages 1–3 cannot be replaced by sophisticated modelling — a brilliant algorithm applied to the wrong problem or dirty data will still fail.
The problem type shapes algorithm selection: classification (is this customer churning?), regression (what will this customer's spend be?), clustering (which segments exist?), ranking (which offer to show first?), generation (what response fits this query?). Start with a baseline model — a simple rule or logistic regression — to confirm that learnable signal actually exists before investing in complexity.
Model selection balances competing constraints: interpretability for regulated domains, training time for fast iteration cycles, inference latency for real-time applications, memory footprint for edge deployment, and maintenance requirements over the model's production life. Involve decision-makers in this selection — the highest-scoring technical metric rarely maps directly to business value.
A model running in a Jupyter notebook delivers no business value. Deployment transforms prototype code into a reliable system serving predictions at scale — and this transformation is far harder than it looks.
Production requirements that notebooks lack: modular code structure, configuration management, comprehensive error handling, structured logging, automated testing, and documentation. Each is non-negotiable at scale.
Deployment is the beginning of the production lifecycle, not the end. Real-world data distributions shift — customer behaviour changes, product catalogues evolve, economic conditions move — and models degrade silently in response.
Two monitoring tracks run in parallel: prediction distribution monitoring detects when current inputs diverge from the training distribution; performance monitoring tracks model accuracy over time. The second is harder — ground truth labels often arrive with significant delay, making immediate performance assessment impossible.
The solution is automated retraining pipelines: drift detection triggers retraining, a new model version is evaluated, then promoted to production or rolled back if it underperforms. Building and operating this MLOps infrastructure typically requires more engineering effort than the initial model development.
Modelling consumes a smaller share of project effort than most practitioners expect. The surrounding stages — defining, acquiring, cleaning, deploying, and monitoring — dominate real-world timelines.