Your questions are part of the assessment. Nobody writes "asked weak questions" on the feedback form, but the panel notices, and it lands in the debrief as a vague comment about depth or seniority. At the same time, the last ten minutes of a round is the only part of the process where you are the one collecting information. Both things are true at once, which means a good question has to do two jobs: show the panel that you know what actually matters in an AI system, and get you an answer you genuinely need.
There is a specific reason this page exists for AI roles rather than engineering roles in general. This field currently has an unusual number of jobs that are, underneath the job description, an executive's enthusiasm with a headcount attached. Someone senior read something, or sat through a vendor demo, or got asked by the board what the AI strategy is. A team was funded. A job description was written by someone who has never shipped an AI feature. The role is real in the sense that the salary clears, and unreal in the sense that it has no sponsor who will defend it, no definition of success, and no budget line for inference. These roles get reorganised. The questions below are how you detect one before you accept it.
Each group opens with what it signals about you. Each question comes with what a bad answer tells you, because the point of asking is what you learn from the reply, not the asking itself.
One thing to be clear about before you start. Every quoted answer on this page is illustrative: a composite of the kinds of replies these questions tend to produce, written to show you the shape of a good signal and a bad one. None of them is a quote from a real interview, a real company, or a real person, and any name, number, or timeframe in them is invented for the example. Treat them as a calibration aid, not as evidence about any employer you are about to meet.
What this group signals: that you have been near an AI system long enough to know it does not have a green build. There is no compiler telling you the output is wrong. If a team cannot describe how they know their system works, they do not know it works. Asking about evals separates candidates who have shipped from candidates who have read.
-
"What does your eval suite look like today, and who owns it?"
Signals that you treat evaluation as infrastructure rather than a QA afterthought, and that you know ownership is the thing that decides whether it survives contact with a deadline.
Bad answer: "We review outputs manually before each release." That is a team with no regression safety net. Every change is a coin flip and someone is spending their evenings reading model output.
-
"How do you know when an AI feature is good enough to ship?"
Signals that you understand quality here is a threshold decision under uncertainty, not a binary pass. You are asking whether they have a bar or a vibe.
Bad answer: "The team just knows." That means one person's taste is the release gate. When that person leaves or gets tired, quality drifts and nobody can name the day it started.
-
"What is your golden dataset, and where did it come from?"
Signals that you know evaluation depends on curated examples with known-good answers, and that provenance matters more than size.
Bad answer: a blank look. If the phrase does not land, they are testing against whatever inputs the last engineer happened to try. That means the eval set drifts toward cases the system already handles.
-
"What happened the last time a model change regressed quality?"
Signals that you know provider updates and prompt edits break things silently, and that you are interested in their detection time rather than their intentions.
Bad answer: "That has not happened." It has happened. They did not notice. That is worse than a story about an ugly week.
-
"How do you measure whether this feature is working for users, separately from whether the model is behaving?"
Signals that you can hold two layers at once: offline model quality and downstream product outcome. Senior candidates know these can move in opposite directions.
Bad answer: only one number, usually thumbs-up rate. Thumbs data is sparse and biased toward the angry and the delighted. If it is the only signal, the loop is not closed.
For the underlying material, see Evaluation fundamentals and Evaluation-driven development.
What this group signals: that you have carried the pager for something non-deterministic. Production AI has failure modes that do not appear in a demo, and these questions are recognisable to anyone who has lived through them. They are also unbluffable in the other direction: a team that has not shipped will struggle to answer them.
-
"What is in production today, and how long did it take to get there?"
Signals that you measure organisations by what shipped, not what is planned. The duration question is the sharp half.
Bad answer: nothing in production after eighteen months of effort. That is not a hard problem, that is an organisation that cannot make a decision. You would inherit the same blockers.
-
"What does on-call look like for AI failures specifically?"
Signals that you know an AI incident is not a normal incident. There is often no stack trace, no error rate spike, just answers quietly getting worse.
Bad answer: "Same as the rest of the platform." Then nobody is paged when quality degrades, only when the service returns a 500. Silent degradation runs for weeks.
-
"What does the user see when the model has no good answer?"
Signals that you think about the unhappy path first, which is the single clearest marker of production experience in this domain.
Bad answer: hesitation, or "it always returns something." A system with no abstention path is a system that confabulates under pressure and calls it a feature.
-
"Walk me through how you would trace one specific bad answer a customer complained about."
Signals that you understand observability for AI means logging prompts, retrieved context, model version, and output together, or you have nothing to investigate with.
Bad answer: they cannot reconstruct it. Then every customer complaint is unfalsifiable, and support becomes an apology desk.
-
"What is your cost per request, and who watches it?"
Signals that you treat inference as a unit-economics problem, which is where AI features go to die at scale.
Bad answer: nobody owns the bill. That means the bill is fine until it is a crisis, and the crisis arrives as a mandate to cut quality on a Friday.
See Observability and monitoring and Cost management.
What this group signals: that you can be trusted with something that talks to customers in your company's voice. Panels for AI leadership roles are alert to candidates who see only upside, because that candidate becomes the incident.
-
"What is the worst thing this system could realistically do to a user?"
Signals that you have thought about harm concretely rather than as a compliance checkbox. Good teams answer this immediately because they have argued about it.
Bad answer: "We have not really thought about that." Then the first serious harm is a discovery, made in public, by a journalist or a regulator.
-
"What did your last red team session find?"
Signals that you know adversarial testing is a practice with outputs, not a policy statement. The word "last" does the work.
Bad answer: there has not been one. Their prompt injection surface is untested, and their guardrails are a string of instructions nobody has attacked.
-
"How do you handle PII in prompts and traces?"
Signals that you know traces are a data store, and that the useful thing about them (full context) is exactly the risky thing.
Bad answer: vagueness. Trace logs are a common place for personal data to end up outside its intended retention path, and nobody notices because they are debug output.
-
"Who can block a launch?"
Signals that you understand governance is only real when a named individual can say no and survive saying it.
Bad answer: a committee with no name attached, or "legal" as the only answer. A committee that has never blocked anything is a rubber stamp. Legal alone means engineering has outsourced judgement to people without the context to exercise it.
See Guardrails and safety and Governance and compliance.
What this group signals: that you are interviewing for a management job, not a staff engineer job with a different title. These are the questions that show you think about people systems as carefully as technical ones, and they are the ones most candidates skip.
-
"How did the team learn this, and who is the person everyone asks?"
Signals that you know AI capability arrives through a small number of people and then spreads, or does not.
Bad answer: one hero holding everything. That is a bus factor of one wearing a cape. Your first year would be reducing the dependency, and the hero may not want that.
-
"How do you level and promote AI engineers here?"
Signals that you have thought about career structure for a role most ladders were not written for, and that you intend to defend your people at calibration.
Bad answer: no differentiation from the general ladder. Then eval work, dataset curation, and prompt iteration are all invisible at promotion time, and your best people leave.
-
"What happened the last time an AI project was killed?"
Signals that you value organisations that can stop. Killing things is the strongest evidence of a working evidence loop.
Bad answer: no project has ever been killed. That is not discipline, that is POC purgatory. Everything is alive because nothing is measured, and headcount is spread across zombie initiatives.
-
"Who owns the eval suite, and is that their job or their hobby?"
Signals that you know unowned infrastructure decays, and the second half of the question is the one that gets an honest answer.
Bad answer: evals owned by whoever cares. Whoever cares is doing it after hours, will burn out, and the suite will rot the month they stop.
-
"What is the split between the platform team and the product teams?"
Signals that you understand the central structural choice in AI organisations, and that the wrong answer creates a permanent tax.
Bad answer: no clear boundary. Then either the platform builds what nobody wants, or every product team rebuilds retrieval and evals separately.
See Team structures and Career ladders for AI engineers.
What this group signals: that you can hold a view above the current sprint. Directors and skip-level interviewers are listening for whether you would defend the roadmap in a room they are not in.
-
"What is the AI capability you think is genuinely yours rather than the provider's?"
Signals that you can distinguish a moat (data, distribution, workflow depth, evaluation discipline) from an API call.
Bad answer: a list of model features. If everything defensible belongs to the provider, the product is a thin wrapper waiting for a pricing change.
-
"What happens to this product if the frontier model ships this feature natively next year?"
Signals that you have priced in the fastest-moving variable in the field. It is the question that most reliably changes the temperature of a room.
Bad answer: no answer at all, or nervous laughter. They have not war-gamed it. You would be building on ground they have not surveyed.
-
"How did you choose your current provider, and what would make you switch?"
Signals that you think about vendor selection as a reversible decision with an exit cost, not a loyalty.
Bad answer: a provider chosen because someone knew a salesperson, or because it was first. Then switching cost was never measured and the lock-in is real but unacknowledged.
-
"What are you deliberately not building?"
Signals that you know strategy is subtraction. Teams with a real strategy answer fast.
Bad answer: nothing is off the table. Then the roadmap is a wish list and priorities are set by whoever escalated most recently.
See Build vs buy vs fine-tune and Vendor and model selection.
What this group signals: honestly, less about you and more for you. These are self-protective. Ask them warmly and late, and ask them anyway.
-
"Who asked for this team, and what happens to it if that person leaves?"
Signals that you understand organisational physics. Every AI team has a sponsor, and the sponsor's tenure is the team's tenure.
Bad answer: no named person, or a name three levels up who has never met the team. The team exists because of a slide. Slides get replaced.
-
"What does success look like in twelve months, and who decides?"
Signals that you want a target you can actually be held to, which reads as confidence rather than caution.
Bad answer: adjectives. "Transformative impact." "AI-first." If nobody can name the decider, then at review time everyone gets to invent their own bar.
-
"What is the budget for inference next year, and who owns that line?"
Signals that you know an AI team without a cost owner is a team whose scope is set by an accountant they have never met.
Bad answer: it comes out of a general cloud budget nobody has modelled. The first quarterly review with a surprising bill will produce a mandate, and you will not be in the room where it is written.
Put plainly: a role with no named sponsor, no defined success, and no budget owner is a role that gets reorganised, usually sooner than anyone expects and always for reasons that have nothing to do with you. You will have done good work and it will not matter, because the thing that ends these teams is never the code. This group is how you find out before you sign, and there is no other point in the process where you get to ask.
Some rounds end with two minutes and an apologetic interviewer. Use these.
| Question | Why this one |
|---|---|
| "What does your eval suite look like, and who owns it?" | It reveals engineering maturity faster than anything else you can ask. The answer tells you whether they can measure, whether anyone is accountable, and whether quality is a system or a person. |
| "What happened the last time an AI project was killed?" | It reveals the organisation's relationship with evidence. Teams that can stop are teams that can measure. Teams that cannot stop are teams whose roadmap is decided by status. |
| "Who asked for this team, and what happens if they leave?" | It reveals whether the job survives. Everything else is downstream of this one. |
If you get three answers you like, the rest of the process is about fit. If you get three answers you do not like, you have learned more in two minutes than most candidates learn in the whole loop.
You are listening for texture, not enthusiasm. Everyone is enthusiastic about AI right now, and enthusiasm costs nothing to fake.
Specificity beats enthusiasm. "We really care about quality" is worth less than an answer that names a rough size for the eval set and then names what it still misses, the way an engineer describes their own work when they are not selling it.
A named person beats a team. "Priya owns it" is a real answer. "The platform team owns it" means nobody does.
A number beats an adjective. Any number. A rough number given with a caveat is stronger evidence than a confident adjective.
A story about a failure beats a story about a launch. Launch stories are rehearsed. Failure stories reveal what they actually do on a bad Tuesday, and how they talk about each other when things go wrong.
Watch for hesitation on the ownership questions. A pause before "who owns the eval suite" tells you more than any polished answer to "what is the culture like here", because culture answers are prepared and ownership answers are not.
| Their answer | What it probably means |
|---|---|
| "We're still figuring that out." | Honest. Fine at month six, worrying at month thirty. Ask which month they are in. |
| "That's a great question." (long pause) | Nobody has asked before. Nobody internally has asked either. |
| "It depends on the team." | There is no standard, and you would be inventing it without authority. |
| A specific number, offered with a caveat | Strong signal. They measure, and they know the limits of the measurement. |
| An unhappy story, told plainly | The best answer on this list. They learn, and they are not managing you. |
| "Legal handles that." | Risk has been outsourced away from the people with the context. |
A few reliably cost you. Anything answered on the careers page or in the job description signals you did not read either, and it is a cheap loss. Compensation before an offer stage is a mistake in most processes: it is a fair thing to want and the wrong room to want it in, and recruiters are the right channel. Anything phrased as a challenge ("so how do you justify shipping that without evals?") reads as a person who will be exhausting in a design review, however right you are. And the fake-interested question, asked to seem thorough, about something you do not care about, is visible from across the table, because you do not follow up on the answer. Interviewers notice when a question dies on delivery. Ask four questions you actually want answered instead of eight you rehearsed.
One caveat on all of this: processes change, and every panel is a person having a day. Treat these as a well-stocked shelf rather than a script. Pick the ones that fit the round you are in, the level of the person in front of you, and the thing you actually still do not know. Build your own list from How AI EM interviews work and the company guides, which describe commonly reported themes rather than any inside knowledge of a given company's loop.
Next: Company guides
Related: How AI EM interviews work | Hiring guide | Common failure modes