Many companies building an AI team face the same problem: the ads are live, applications are coming in, but choosing the right person is hard. CVs repeat the same buzzwords, interviews depend on the interviewer's intuition, and a few months later someone says "this isn't who we expected".
This article offers a structured approach to make data scientist and machine learning (ML) engineer interviews more reliable.
Clarify the role first
Most interview mistakes are made before the interview: if the role is vague, you can't ask the right questions. The title "data scientist" can cover very different jobs:
| Role | Main work | Interview priority |
|---|---|---|
| Data analyst | Reporting, analysis, answering business questions | SQL, statistics, communication |
| Data scientist | Modelling, experiment design, solving business problems | Statistics, modelling, problem framing |
| Machine learning engineer | Taking models to production, scaling | Software engineering, system design, MLOps |
| MLOps / platform engineer | Infrastructure, automation, monitoring | DevOps, cloud, CI/CD |
| AI product owner | Use cases, prioritisation, team coordination | Product thinking, business knowledge, communication |
The role description should state the problems to be solved in the first six months, the data and technologies involved, who the person will work with and the seniority level. We cover which roles you need and in what order in how to build an AI team.
Why a structured interview?
In a structured interview, all candidates get the same questions and answers are scored against predefined criteria. Hiring research has repeatedly shown its strength: in Sackett and colleagues' comprehensive 2022 reanalysis, structured interviews had the highest mean validity for predicting job performance. The researchers also note this isn't a mandate to prefer one method in every circumstance, but the case for replacing intuition-driven conversation with a systematic structure is strong.
A recommended interview process
1. Screening call (30 min)
To understand the candidate's experience and fit with the role. "Tell me about the hardest project on your CV: what was the problem, what did you do, what was the result?" is a good opener to see whether the candidate can separate their own contribution from the team's.
2. Technical foundations (45–60 min)
Measures the core knowledge the role requires. "Why" and "when" questions rather than memorised definitions:
- Statistics and evaluation: "Why can accuracy be misleading on an imbalanced dataset? What would you look at instead?"
- Modelling: "A model performs very well in training and poorly in testing. What do you check?"
- Data: "What is data leakage? How would you spot it in a project?"
- Engineering (for ML engineers): "How would you know a model's performance has dropped in production? What monitoring would you set up?"
- LLM applications (if relevant): "How would you measure answer quality for an assistant that works with internal documents?"
3. Case study (2–4 hours, time-boxed)
The stage closest to real work. A good case:
- Uses an anonymised or synthetic dataset similar to your real problem but with no confidential data.
- Clearly limits the time and doesn't take several days of the candidate's life.
- Measures approach more than a "perfect model": how they framed the problem, explored the data, what assumptions they made, how they explained the result.
4. Case discussion (45 min)
The candidate presents their solution and the panel asks questions. This confirms the solution is theirs and shows how they think about reasoning, alternatives and limitations. "What would you do with one more week?" reveals a lot.
5. Team and communication conversation (30–45 min)
AI projects require constant contact with business units. Questions like "How would you explain to a non-technical manager why your model made a wrong prediction?" measure this skill.
Scoring criteria
Write the criteria and score definitions for each stage before the interviews. An example scorecard:
| Criterion | 1 – Weak | 3 – Adequate | 5 – Strong |
|---|---|---|---|
| Problem framing | Jumps straight to a model | Asks the business question | Clarifies the decision, success criteria and constraints |
| Data exploration | Builds a model without looking at data | Does basic checks | Systematically checks for leakage, missing data and bias |
| Evaluation | Looks at a single metric | Chooses the right metric | Explains the metric's business meaning and limits |
| Communication | Buries the listener in jargon | Explains clearly | Speaks the decision-maker's language, including uncertainties |
| Engineering (if required) | Messy code | Works and is readable | Thinks testable, reproducible, production-ready |
Tips: - Each panellist scores independently; discussion happens after scores are in. - Apply the same case and the same criteria to every candidate. - Base notes on concrete observations ("spotted the data leakage on their own"), not general impressions ("good energy").
Common mistakes
- Brain teasers: Puzzle questions unrelated to the job don't measure the skills the role needs.
- Tool trivia: "What does this parameter of that library do?" measures knowledge anyone can look up.
- Endless take-homes: Case studies that take days drive good candidates away.
- Single-person decisions: Leaving the technical stage to one person's intuition.
- Discovering the role in the interview: "Let's see what they can do" makes comparison impossible.
Conclusion
Hiring well for AI roles takes a clear role definition, a case study close to real work and a structured interview with criteria written in advance. It's fairer to candidates and protects the organisation from the cost of a wrong hire.
In our Talent & Hiring Support module we prepare role definitions, interview question sets, the case study and scoring criteria with you, and join the panel as a technical expert. We don't offer candidate sourcing; you find the candidates, and we run the technical evaluation together.
Sources
- Sackett, Zhang, Berry and Lievens (2022) – Revisiting meta-analytic estimates of validity in personnel selection, Journal of Applied Psychology
- SIOP – Is cognitive ability the best predictor of job performance?
- Chip Huyen – Introduction to Machine Learning Interviews
- Google – Rules of Machine Learning