# How Microsoft Built Ask Learn, the RAG System Behind Copilot for Azure
*Microsoft turned its docs into an Advanced RAG service that answers questions, grounds Copilot's actions, and steps in when Copilot can't help.*
By [Rohit Lakhotia](https://scaleengineer.com/authors/rohit-lakhotia)
Published: 2026-11-16
Canonical: https://scaleengineer.com/blog/how-microsoft-built-ask-learn-the-rag-system-behind-copilot-for-azure
---
Imagine you're working in the Azure portal and something breaks\. Normally, you'd open a new tab, search the docs, go through several articles, and then work out how the answer applies to your setup\. It works, but it's slow, and every switch between tabs costs you focus\. Now imagine asking the portal directly: "Why is my VM failing to start?" You get an accurate answer based on Microsoft's official documentation without leaving the page\.

That's the idea behind Microsoft Copilot for Azure, a generative AI chat experience in the Azure portal\. Customers can ask questions about Azure services, ask about their own deployments, and even ask it to take actions, like deploying a new service or shutting down all running VMs in a subscription\.

But making an AI assistant like this reliable is hard\. A chatbot that sounds confident and is wrong is worse than no chatbot at all\. Microsoft needed answers that were accurate, relevant, reliable, and verifiable\.

Much of that reliability comes from a service called Ask Learn, an API built by Microsoft's Skilling organization on top of Microsoft Learn documentation\. Microsoft describes it as one of the first retrieval\-augmented generation \(RAG\) systems built at this scale\. Let's understand how they built it in this blog today\.

## It Started with a Different Problem

Ask Learn wasn't originally built for Copilot\. It started at a hackathon\. Every February, the Skilling organization holds an internal hackathon called "Fix\. Hack\. Learn\." In February 2023, almost every participant experimented with generative AI, and some engineers focused on Microsoft Q&A, Microsoft's community question\-and\-answer platform\.

Q&A had a clear problem\. After someone posted a question, it could take hours for the community to respond\. And the first replies were often clarifying questions, like "Which version are you using?" or "What error do you see?", before anyone could actually help\.

The team tackled this in two steps:

1. Help users ask better questions\. A new feature gave AI feedback on how a question was worded and offered a "rewrite for me" option\.
2. Give users an answer while they wait\. The next step was to generate an interim answer from Microsoft Learn documentation\.

The second step needed a system that could find the right pieces of documentation for any question and feed them to a large language model \(LLM\)\. That's where RAG comes in\.

## What is RAG?

An LLM on its own only knows what it learned during training\. It doesn't know your latest documentation, and when it's unsure, it may confidently make things up\.

Retrieval\-augmented generation \(RAG\) fixes this by giving the LLM the right context from a trusted source before it answers\. Instead of asking the model to remember facts, you hand it the relevant facts and ask it to answer from them\.

Here's how basic RAG works:

1. Chunk the documents\. Large documents are split into smaller pieces called chunks\.
2. Create embeddings\. Each chunk is converted into an embedding, a vector of numbers that represents its meaning, including topic, keywords, language, and tone\.
3. Store them\. The embeddings, chunks, and metadata are saved in a vector database\.
4. Embed the question\. When a user asks something, the question is also converted into an embedding\.
5. Search for similar chunks\. The database finds chunks whose embeddings are closest to the question's, using techniques like cosine similarity or nearest\-neighbor search\.
6. Generate the answer\. The question and the matching chunks are sent to the LLM, which writes a response\.

Even this "simple" version takes plenty of tuning: choosing the right chunk size, deciding how much chunks should overlap, cleaning up chunks after splitting, and organizing them so retrieval works well\.

## Naive RAG vs Advanced RAG

The pipeline above is what one research paper calls naive RAG\. It gives decent results, but decent isn't enough for Microsoft's customers\. To get reliable, accurate, verifiable answers, you need what the same paper calls Advanced RAG\. It adds processing before and after almost every step, often with extra LLM calls\.

Before retrieval, the system improves the question itself:

- Query rewriting: rephrasing the question so it matches the documentation better\.
- Query expansion: adding related terms to find more relevant chunks\.
- Query clarification: resolving vague or ambiguous questions\.

After retrieval, the system improves the chunks it found:

- Re\-ranking: reordering chunks so the most relevant come first\.
- Expanding: adding related neighboring chunks for more context\.
- Filtering: removing irrelevant, redundant, or low\-quality chunks\.
- Compressing: condensing the results into a more compact form\.

After generation, there's often one more round of checks to make sure the answer meets the quality bar and doesn't break any safety, ethical, or operational guidelines\.

Each of these steps adds a little quality\. Together, they're the difference between a demo and a product\.

## The Architecture: Orchestration Plus Services

The team had only a few months to build many of these Advanced RAG features\.

They chose a service\-oriented architecture\. At the top sits an orchestration layer, which calls separate services, each responsible for one major job\. Some services handle pre\-processing and post\-processing, while others run evaluations to check how accurate and reliable an answer is\.

The most essential of these is the Knowledge Service\.

### The Knowledge Service

The Knowledge Service handles the data side of RAG\. It:

- Breaks Microsoft Learn's technical documentation into chunks\.
- Creates embedding vectors for each chunk\.
- Saves the vectors and chunks in a vector database\.
- Exposes that database through a web API for the orchestration layer\.

That sounds simple until you look at the scale\. Microsoft Learn has hundreds of thousands of documents, and hundreds of technical writers update them constantly\. Every time a document changes, its chunks and embeddings must be refreshed, or the AI starts answering from outdated information\.

Keeping a vector store continuously updated was a big data engineering challenge\. Here, the team didn't have to start from scratch: they got guidance from Microsoft's Discovery and Search teams and the Bing teams, who had solved similar problems of keeping search indexes fresh years earlier\.

## The Real Challenge: Non\-Determinism

Traditional software is deterministic: the same input gives the same output\. Generative AI is not\. Ask the same question twice and you may get two different answers\.

This makes a basic engineering question very hard: how do you know whether a change made the system better or worse?

If you tweak a prompt or change how documents are chunked, a few good answers prove nothing\. You need a consistent way to measure quality across many questions\. Microsoft's answer was the golden dataset\.

## Building the "Golden Dataset"

A golden dataset is a carefully curated, annotated set of examples used as a benchmark for evaluating AI models\.

For a RAG system, it usually includes:

- Questions and their approved answers\.
- Metadata, such as the topic and type of question\.
- References to the source documents that should be used as ground truth\.
- Different phrasings of the same question, to reflect how real users ask things\.

A team of technical writers, engineers, and data scientists from Skilling and the CX Data & AI team worked with experts across many technologies to build this dataset, and they keep growing it\.

Its value is in before\-and\-after testing\. Tom FitzMacken, a member of the team, gave an example: when engineers considered a major change to what gets indexed in the Knowledge Service, they ran tests on the golden dataset before and after the change to see whether quality improved or declined\.

Without a benchmark like this, every change to a generative AI system is a guess\.

## Being First Meant Building Their Own Tools

In early 2023, the generative AI tooling ecosystem barely existed\. The team couldn't just pick off\-the\-shelf components, so they built what they needed and swapped in standard tools as those became available\.

Two examples:

- Calling the models\. The team first used \.NET to call Azure OpenAI's Embedding and Completion APIs directly\. Later, they moved to the Azure SDK's preview package for Azure OpenAI once it was released\.
- Evaluation\. Engineers initially built custom Python notebooks to measure three things:
  
    - Retrieval quality: were the retrieved documents relevant to the question?
    - Groundedness: was the answer actually based on facts in those documents?
    - Answer relevance: did the answer address what the user asked?
  
  They also built a custom harms evaluation tool to flag potentially harmful answers\. Eventually, all of these were replaced with evaluation flows in Prompt flow\.

Being first wasn't only a technical challenge\. The team also needed input from legal, AI ethics, privacy, and security teams, which sometimes had to create new policies and procedures on the spot\.

## From Q&A Assist to Ask Learn

In May 2023, after a few intense months of development, the team launched Microsoft Q&A Assist\.

Soon after, the Azure portal team wanted to use the same RAG system for Copilot for Azure\. Because the service was well designed, it needed only a few minor changes to serve both Microsoft Q&A and Copilot\. The repurposed service was named Ask Learn\.

### How Copilot for Azure is Built

The Azure portal supports hundreds of thousands, possibly millions, of distinct operations\. No single team could build AI support for all of them\.

So Copilot for Azure uses an extensibility model\. Partner teams across Azure build chat handlers, which Microsoft describes as super\-powered Copilot plugins\. Copilot orchestrates these handlers to complete customer tasks, and Ask Learn is one of them\.

## Ask Learn's Three Roles

Ask Learn does more than answer questions\.

### 1\. Answering Questions Directly

The most visible role is letting users get answers inside the Azure portal instead of searching the docs in another tab\. Internally, the team called this "ask, don't search\."

Searching has hidden costs: going through results, finding the right article, and working out how it applies to your situation\. Ask Learn aims to remove that effort\.

### 2\. Grounding Data for Other Components

Less visibly, Ask Learn supplies grounding data, meaning reliable facts and context that other chat handlers use to guide their actions\.

When Copilot needs to coordinate several chat handlers to complete a task, it may call Ask Learn first so the final output is accurate and based on documented facts\. In this role, Ask Learn doesn't answer the user directly but makes everyone else's answers better\.

### 3\. A Fallback When Copilot Can't Help

Sometimes Copilot can't do what the user asked, or isn't sure how\. Instead of replying "I'm sorry, I can't help with that," it can call Ask Learn to find relevant resources so the user can solve the problem themselves\.

## Capturing Feedback Without Breaking Privacy

To improve a generative AI system, you need feedback from real users\. But Microsoft's privacy policies state that customer data belongs to the customer, and by default, Microsoft employees cannot see customer AI inputs and outputs\.

The team approached this in stages:

1. Thumbs up / thumbs down\. At launch, customers could rate responses\. These ratings were combined into quality metrics, which helped the team spot broad issues\.
2. Consent\-based chat history\. After working closely with Microsoft's privacy team, Copilot for Azure added an option for customers to consent to sharing their chat history along with written feedback\.

### Root Cause Analysis is Expensive

With detailed feedback, the team analyzes each response, positive or negative, to find the root cause\. This can take up to 30 minutes per response\. Engineers investigate:

- How the query produced the result\.
- Whether the right chunks from Learn documentation were used\.
- How the source article was chunked\.
- Which pre\- or post\-processing steps could have improved the answer\.

Sometimes they find a content gap: no Learn documentation existed to answer the question at all\.

This manual investigation doesn't scale\. So the team is building an assessment pipeline with custom tooling and metrics that approximate answer quality\. It aims to answer questions like: "Why did we give this answer? Which Learn articles did we send to Azure OpenAI? What happened at each step of the pipeline?"

The goal is to improve both the inference pipeline and, eventually, the documentation itself, by finding gaps and better ways to explain product facts\.

## Lessons Learned

The Ask Learn team shared practical advice for anyone building a "chat with your data" system\.

### Start With Your Content

A RAG system is only as good as the content behind it\. Before building anything, ask:

- Does your content actually answer the questions users will ask, with enough context for an LLM to rephrase it?
- Is it well structured, with useful metadata?
- How should it be chunked so each chunk fits the model's context window and still carries useful information?
- How will you handle versioning, where some facts apply to one product version but not another?
- Is it accurate and up to date?

Also involve your content writers early\. RAG projects often lead to content changes, and writers need to understand how their content shapes what the LLM generates\.

### Decide How You'll Measure Success

Define your metrics up front so you can track improvements and regressions over time\. Answer relevance matters because it's closely tied to accuracy\. Build a golden dataset that represents your whole content corpus, and decide early how you'll collect and review customer feedback\.

Don't underestimate the people\-hours needed to review answers and trace them back through the pipeline\.

### Set the Right Expectations

Generative AI is good at summarization, which makes it a strong fit for a service like Ask Learn\. It's less reliable at logic and discriminative tasks\.

It's also non\-deterministic\. Small changes in how a question is worded can produce very different answers, and wording can even affect whether the answer is true\. A system might be right 999 times out of 1,000 and still get one wrong, so you need some form of ongoing oversight\.

### Balance Quality, Latency, and Cost

- More inference steps improve quality, but at a price\. Every pre\- or post\-processing step adds latency and cost, so measure both as you experiment\.
- Plan for scale\. Understand your throughput limits and how they affect capacity\.
- Build safety and privacy into the release plan\. This includes privacy and safety requirements, risk assessments, data protection at every customer touchpoint, external safety reviews, and red\-teaming\.

Red\-teaming means simulating an attacker to find weaknesses\. For LLMs, one of the biggest risks is jailbreaking: techniques users try in order to get the model to bypass its safety, ethical, or operational rules\.

## The Bigger Picture

Ask Learn shows that the hard part of RAG isn't the basic pipeline\. Embedding documents and running a similarity search is easy to get working\.

The real work is everything around it: keeping hundreds of thousands of documents fresh, adding pre\- and post\-processing to improve answer quality, building a golden dataset to measure non\-deterministic output, and gathering feedback without compromising customer privacy\.

And because Microsoft built Ask Learn as a separate service behind an API, it became more than a Q&A feature\. It became a shared source of truth that Copilot for Azure uses to answer questions, ground its actions, and help users when nothing else can\.

## Key Takeaways

- Origins: Ask Learn started as a hackathon project to give instant answers on Microsoft Q&A and later became a core service for Copilot for Azure\.
- RAG basics: documents are chunked, converted into embeddings, stored in a vector database, and retrieved by similarity to the user's question\.
- Advanced RAG: query rewriting, re\-ranking, filtering, compression, and output checks improve on basic RAG\.
- Architecture: an orchestration layer calls separate services, with the Knowledge Service managing chunking, embeddings, and the vector database\.
- Freshness at scale: embeddings must stay in sync as hundreds of writers update hundreds of thousands of documents\.
- Golden dataset: a curated benchmark of questions, approved answers, and source articles makes before\-and\-after testing possible despite non\-determinism\.
- Three roles: Ask Learn answers questions, supplies grounding data to other chat handlers, and acts as a fallback when Copilot can't help\.
- Privacy\-first feedback: thumbs\-up/down ratings came first, followed by consent\-based sharing of chat history\.
- Trade\-offs: extra inference steps improve quality but add latency and cost\.

Official blog from Microsoft: **[How we built “Ask Learn”, the RAG\-based knowledge service](https://devblogs.microsoft.com/engineering-at-microsoft/how-we-built-ask-learn-the-rag-based-knowledge-service/)**

By now, you must have had a clear idea of,** How Microsoft Built Ask Learn, the RAG System Behind Copilot for Azure? **In a nutshell, Microsoft built Ask Learn as an Advanced RAG service over Microsoft Learn documentation, using a golden dataset, layered pre\- and post\-processing, and privacy\-safe feedback to power reliable answers in Copilot for Azure\.

**Congratulations\! You've just advanced another step in your tech journey\. Keep progressing\!**
