AI Engineering Interview Questions and Answers

AI Engineering Interview Questions and Answers - Your Cheat Sheet For AI Engineering Interviews

These interview questions and answers are helpful for roles such as:

• AI Engineer
• Gen AI Engineer
• LLM Engineer
• Agentic AI Engineer
• AI Agent Engineer
• Forward Deployed Engineer
• AI Solutions Architect
• AI Platform Engineer
• Applied AI Engineer
• MLOps Engineer
• LLMOps Engineer

Table of Contents

• Must Know
• LLM Fundamentals
• Prompt Engineering
• Retrieval-Augmented Generation (RAG)
• AI Agents and Agentic Systems
• Fine-Tuning and Model Adaptation
• Vector Databases and Embeddings
• AI System Design
• LLMOps and Production AI
• Evaluation and Testing
• AI Safety, Ethics, and Responsible AI
• Multimodal AI
• AI Infrastructure and Scalability
• Coding and Practical Implementation
• Behavioral and Scenario-Based Questions

Prepared and maintained by the Founder of Outcome School: Amit Shekhar

Follow Amit Shekhar

• X/Twitter
• LinkedIn
• GitHub

Follow Outcome School

• YouTube
• X/Twitter
• LinkedIn
• GitHub

I teach at Outcome School

• AI and Machine Learning

Note: We will keep updating this with new questions and answers.

Must Know

• LLM
• RAG
• MCP
• Agent
• Fine-tuning
• Quantization

Learn about the LLM, RAG, MCP, Agent, Fine-tuning & Quantization: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization

LLM Fundamentals

• What are foundation models, and how have they changed AI engineering?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• What is a Large Language Model (LLM), and how does it work?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• Inside ChatGPT: What Happens After You Hit Enter?
  • Answer: Inside ChatGPT: What Happens After You Hit Enter
• What is the Transformer architecture and how does it work?
  • Answer: Decoding Transformer Architecture
• What are the key components of the Transformer architecture?
  • Answer: Decoding Transformer Architecture
• What is tokenization in LLMs?
  • Answer: Tokenization in Large Language Models (LLMs)
• Explain BPE (Byte Pair Encoding).
  • Answer: Byte Pair Encoding
• Explain WordPiece and SentencePiece.
• What is positional encoding, and why is it needed in Transformers?
  • Answer: Positional Embeddings in LLMs
• What are embeddings?
  • Answer: Embeddings in Machine Learning
• Explain the Query(Q), Key(K), and Value(V) in attention.
  • Answer: Math behind Attention - Q, K, and V
• What is self-attention, and how does it work in Transformers?
  • Answer: Self Attention in Transformers
• What is Cross Attention in Transformers?
  • Answer: Cross Attention in Transformers
• Why do we scale the dot product attention by √dₖ in the Transformer architecture?
  • Answer: Math behind √dₖ Scaling Factor in Attention
• What is causal masking?
  • Answer: Causal Masking in Attention
• What are multi-head attention mechanisms? Why use multiple attention heads?
  • Answer: Multi-Head Attention in Transformers
• What are Feed-Forward Networks in LLMs?
  • Answer: Feed-Forward Networks in LLMs
• What is the context window in LLMs, and why does it matter?
  • Answer: Context Window in LLMs
• Why is the context window limited in LLMs?
  • Answer: Why is the context window limited in LLMs?
• What is temperature in the context of LLMs, and how does it affect output?
  • Answer: What is temperature in the context of LLMs?
• Why is the first token slower than the rest in an LLM?
  • Answer: The First-Token Latency Problem in LLMs
• Explain Top-p (nucleus) sampling and Top-k sampling. How do they differ?
• What are logits, and how are they used in text generation?
  • Answer: Understanding Logits in Machine Learning
• What are skip connections (residual connections) in Transformers?
  • Answer: Skip connections (residual connections) in Transformers
• What is the difference between open-source and closed-source LLMs? When would you choose one over the other?
• What is the difference between encoder-only, decoder-only, and encoder-decoder Transformer architectures?
  • Answer: Encoder vs Decoder in Transformers
• What is KV cache, and how does it speed up inference?
  • Answer: What is KV Cache in LLMs?
• What is model distillation, and how is it used with LLMs?
  • Answer: How does Knowledge Distillation work?
• What is Mixture of Experts (MoE), and how does it work in models like Mixtral?
  • Answer: Mixture of Experts Explained
• What is the difference between dense and sparse models?
  • Answer: Mixture of Experts Explained
• What is Flash Attention?
  • Answer: Decoding Flash Attention in LLMs
• What is Cross-Entropy Loss?
  • Answer: Math Behind Cross-Entropy Loss
• What is Grouped-Query Attention (GQA), and how does it differ from Multi-Head Attention (MHA)?
  • Answer: Grouped Query Attention
• How does Rotary Position Embedding (RoPE) work, and why is it preferred over learned positional embeddings?
  • Answer: Math Behind RoPE (Rotary Position Embedding)
• Explain Layer Normalization
  • Answer: Batch Normalization vs Layer Normalization
• Explain RMSNorm (Root Mean Square Layer Normalization)
  • Answer: RMSNorm (Root Mean Square Layer Normalization)
• Your LLM keeps ignoring your instructions. How do you make it follow structured output formats?
• Your LLM-powered tool hits the context window limit on long documents. How do you handle it?
• Your LLM does not admit when it does not know the answer. How do you make it say "I don't know"?
• Your LLM generates responses that are too verbose. How do you control response length?
• Your LLM memorized proprietary training data and leaks it in responses. How do you prevent this?
• Your LLM coding assistant generates outdated code using deprecated libraries. How do you fix it?
• Your tokenizer splits important domain terms into meaningless subword pieces. How do you fix it?
• Your Transformer's KV cache grows too large during long sequence generation. How do you manage memory?
  • Answer: Paged Attention in LLMs
• Your Transformer runs out of memory on long documents due to quadratic self-attention. How do you scale it?
• Your distilled student model fails on the complex reasoning that the teacher model handled. How do you close the gap?
• After RLHF alignment, your LLM became safer but lost capability on hard tasks. How do you manage the alignment tax?
• Your RLHF-trained LLM is gaming the reward model instead of being genuinely helpful. How do you fix reward hacking?
  • Answer: Reinforcement Learning from Human Feedback (RLHF)
• Your chatbot loses context after 10 turns in a conversation. How do you maintain a long conversation context?
  • Answer: AI Agent Memory
• Your chatbot fails when users switch topics mid-conversation. How do you handle topic switches?
• Your QA system always generates an answer even when no answer exists in the context. How do you detect unanswerable questions?
• Your summarization system hallucinated facts not in the original article. How do you fix it?
• Your text generation repeats phrases in long outputs. How do you fix repetition?
• Transformers work on text, so can they also understand images?
  • Answer: Decoding Vision Transformer (ViT)
• Small Language Models (SLMs)
  • Answer: Small Language Models (SLMs)
• Large Reasoning Models (LRMs)
  • Answer: Large Reasoning Models (LRMs)
• What are Autoregressive Models?
  • Answer: Autoregressive Models
• Explain the difference between autoregressive and masked language modeling.
• Proximal Policy Optimization (PPO)
  • Answer: Proximal Policy Optimization (PPO)
• Direct Preference Optimization (DPO)
  • Answer: Direct Preference Optimization (DPO)
• Group Relative Policy Optimization (GRPO)
  • Answer: Group Relative Policy Optimization (GRPO)
• Recursive Language Models (RLMs)
  • Answer: Recursive Language Models (RLMs)
• Continual Learning in LLMs
  • Answer: Continual Learning in LLMs
• How do Diffusion Language Models (DLMs) work?
  • Answer: How do Diffusion Language Models (DLMs) work?
• How Does LLM Watermarking Work?
  • Answer: How Does LLM Watermarking Work?
• How do RNNs and Transformers differ?
  • Answer: How do RNNs and Transformers differ?

Prompt Engineering

• What is prompt engineering, and why is it critical for AI applications?
• Explain zero-shot, one-shot, and few-shot prompting with examples.
  • Answer: Explain zero-shot, one-shot, and few-shot prompting with examples
• What is chain-of-thought (CoT) prompting, and when should you use it?
  • Answer: How does Chain-of-Thought (CoT) Prompting work?
• Explain self-consistency prompting and how it improves reasoning.
• What is tree-of-thought prompting?
• What is ReAct (Reasoning + Acting) prompting, and how does it work?
  • Answer: ReAct Agent
• What is a system prompt, and how does it influence model behavior?
• How do you structure prompts for consistent structured output (JSON, XML)?
• What is prompt injection, and how do you defend against it?
  • Answer: Prompt Injection in LLMs
• What is jailbreaking in LLMs, and what are common jailbreak techniques?
• How do you optimize prompts for cost and latency?
• What is the difference between prompt engineering and prompt tuning?
• What is a prompt template, and how do you design one for production use?
• How do you handle multi-turn conversations with LLMs?
• What is role prompting, and when is it effective?
• What is prompt chaining, and how do you design a chain of prompts for complex tasks?
  • Answer: How does Prompt Chaining work?
• How do you evaluate and iterate on prompt quality?
• What are meta-prompts, and how can they be used to generate prompts?
• What are the common failure modes in prompting, and how do you debug them?
• How do you handle edge cases and adversarial inputs in prompt design?
• What is the "lost in the middle" problem in long-context prompting?
  • Answer: The Lost in the Middle Problem in LLMs
• What are output parsers, and why are they needed for production applications?
• How do you handle multi-language prompting effectively?
• Your few-shot prompting gives inconsistent results across similar inputs. How do you stabilize it?
• Your LLM classification system is too sensitive to prompt wording changes. How do you reduce prompt sensitivity?
• Your chatbot's system prompt containing proprietary business logic is being leaked by users. How do you prevent it?
• Your LLM agent is vulnerable to prompt injection that reveals the system prompt. How do you defend it?
  • Answer: Prompt Injection in LLMs
• Your chain-of-thought prompting is not improving LLM accuracy on reasoning tasks. What do you fix?
• Your AI system works in English but fails for other languages. How do you add multilingual support?
• Your zero-shot cross-lingual transfer from English fails on other languages. How do you fix it?

Retrieval-Augmented Generation (RAG)

• What is Retrieval-Augmented Generation (RAG), and why is it important?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• Explain the architecture of a basic RAG system.
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• What are the key components of a RAG pipeline?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• What are chunking strategies, and how do you choose the right chunk size?
  • Answer: Chunking Strategies for RAG
• Compare fixed-size chunking, semantic chunking, and recursive chunking.
  • Answer: Chunking Strategies for RAG
• What are embedding models, and how do they convert text to vectors?
  • Answer: What are Embeddings?
• How do you choose an embedding model for your RAG system?
• Explain Agentic RAG.
  • Answer: Agentic RAG
• What is hybrid search, and why is it better than pure vector search?
  • Answer: How does Hybrid Search work?
• What is re-ranking, and how does it improve RAG retrieval quality?
  • Answer: How does a Reranker work?
• How do you handle multi-document and multi-hop questions in RAG?
• What is the "lost in the middle" problem in RAG systems?
  • Answer: The Lost in the Middle Problem in LLMs
• How do you evaluate a RAG system? Explain faithfulness, relevance, and context precision/recall.
• Explain Self-RAG. How does the model decide when to retrieve?
• What is GraphRAG, and when would you use it over traditional RAG?
  • Answer: GraphRAG
• Vectorless RAG
  • Answer: Vectorless RAG
• How do you handle structured data (tables, SQL databases) in a RAG pipeline?
• What are the common failure modes of RAG systems, and how do you debug them?
• How do you handle document updates and maintain freshness in a RAG system?
• How do you optimize RAG for latency in production?
• What is the role of metadata filtering in RAG systems?
• Compare RAG vs fine-tuning. When would you use each?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• What is query transformation in RAG (HyDE, query decomposition, step-back prompting)?
  • Answer: How does HyDE work in RAG?
• How do you implement citation and source attribution in RAG?
• How do you scale a RAG system to millions of documents?
• What is parent-child chunking, and how does it improve retrieval?
  • Answer: Chunking Strategies for RAG
• Your RAG system is hallucinating despite having the right context. How do you fix it?
• Your RAG chunk overlap causes redundant results. How do you reduce redundancy?
• Your RAG retrieval is too slow with a large knowledge base. How do you speed it up?
• Your RAG system returns duplicate results. How do you deduplicate?
• Your RAG system needs per-user access control on internal documents. How do you implement it?
• Your RAG system fails on domain-specific jargon. How do you fix it?
• Your text-only RAG system now needs to handle images and tables. How do you extend it?
• Your RAG knowledge base gets updated frequently and needs versioning. How do you manage it?
• Your RAG system fails on multi-hop questions that require combining multiple facts. How do you fix it?
• Your enterprise RAG system returns contradictory answers from different source documents. How do you resolve conflicts?
• Your RAG system returns outdated answers from an evolving knowledge base. How do you keep it current?
• Your RAG system struggles with PDF documents containing tables and layouts. How do you fix PDF parsing?

AI Agents and Agentic Systems

• What is an AI agent, and how does it differ from a simple LLM call?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization and AI Agent Explained
• AI Agent Memory
  • Answer: AI Agent Memory
• Harness Engineering in AI
  • Answer: Harness Engineering in AI
• Explain the ReAct (Reasoning + Acting) agent architecture.
  • Answer: ReAct Agent
• What is the Plan-and-Execute agent pattern?
  • Answer: Plan-and-Execute Agent
• What is tool use (function calling) in LLMs, and how does it enable agents?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• How do you design and define tools for an AI agent?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• What is the difference between single-agent and multi-agent systems?
  • Answer: Multi-Agent Systems
• What is Model Context Protocol (MCP), and how does it standardize tool integration?
  • Answer: Explained in this video: AI Engineering Explained: LLM, RAG, MCP, Agent, Fine-Tuning, Quantization
• What are AI SubAgents?
  • Answer: AI SubAgents
• What are the different types of agent memory (short-term, long-term, episodic)?
  • Answer: AI Agent Memory
• How do you handle agent failures and implement error recovery?
• What is an agent loop, and how does it decide when to stop?
  • Answer: AI Agent Loop
• Context Engineering
  • Answer: Context Engineering
• How does context compaction work?
  • Answer: How does context compaction work?
• Loop Engineering
  • Answer: Loop Engineering
• Graph Engineering
  • Answer: Graph Engineering
• How AI Agents Communicate?
  • Answer: How AI Agents Communicate
• What are Agent Skills?
  • Answer: What are Agent Skills?
• How do you evaluate and test AI agents?
  • Answer: AI Agent Evaluation
• What are the security risks of agentic systems, and how do you mitigate them?
• What is the difference between reactive and proactive agents?
• How do you manage token consumption and cost in long-running agent workflows?
• What is the human-in-the-loop pattern for agents, and when is it needed?
• How do you implement guardrails for AI agents to prevent harmful actions?
  • Answer: How do LLM guardrails work?
• What is agent reflection, and how does it improve agent performance?
  • Answer: Reflection Agent
• What is the difference between code-generating agents and tool-calling agents?
• How do you handle multi-modal inputs and outputs in agentic systems?
• How do you implement state management in complex agent workflows?
  • Answer: [How does LangGraph work?](https://o