Cloudflare has launched a new feature that changes how websites can respond to AI bots. The feature is called Markdown for Agents, and it allows websites to automatically convert HTML pages into Markdown format when AI systems request it.
This update is currently in beta. It is available at no extra cost for users on Pro, Business, and Enterprise plans, as well as SSL for SaaS customers.
Let’s break this down in simple terms and understand what it means for website owners, developers, and publishers.
What Exactly Did Cloudflare Launch?
Most websites today are built in HTML. When someone opens a webpage, their browser loads HTML code, along with CSS, JavaScript, images, and other elements.
AI bots, however, do not need design elements. They mainly need clean text.
With Markdown for Agents, when an AI crawler sends a request asking specifically for markdown format, Cloudflare converts the original HTML page into markdown and sends that instead.
This happens automatically at Cloudflare’s edge network. The original server does not need to create a separate markdown page.
In simple words:
- The website still has only one version (HTML).
- If an AI bot asks for markdown, Cloudflare converts it instantly.
- If a normal browser asks for HTML, it gets the regular webpage.
How Does It Work?
This feature works using something called HTTP content negotiation.
Here’s how the process works step by step:
- An AI bot sends a request.
- In the request header, it includes: Accept: text/markdown
- Cloudflare sees this request.
- Cloudflare fetches the original HTML page.
- Cloudflare converts the HTML into markdown.
- The markdown version is sent back to the AI bot.
The same URL is used. The difference is only in the format requested.
This is important because it means the content stays the same. Only the format changes.
Why Convert HTML to Markdown?
Cloudflare explained that HTML pages contain many extra elements that AI systems do not need. These include:
- Navigation menus
- Styling code
- Scripts
- Tracking codes
- Layout structures
All of this increases token usage when AI models process the page.
Cloudflare gave an example using its own blog post:
- HTML version: 16,180 tokens
- Markdown version: 3,150 tokens
That is a big difference.
In simple terms, AI systems can process markdown faster and more efficiently because there is less unnecessary code.
Cloudflare compared it to paying by the word to read packaging material instead of reading the actual letter inside.
Google’s John Mueller’s Reaction
Shortly before this launch, John Mueller from Google criticized the idea of serving markdown pages to AI bots.
He called it “a stupid idea” and questioned whether bots can properly understand markdown links.
However, it is important to understand that Mueller was referring to a different setup.
Some website owners were creating separate markdown pages and serving them only to AI bots using user-agent detection. That method can create issues such as:
- Cloaking concerns
- Broken links
- Different content versions
Cloudflare’s method is different.
Instead of detecting user agents, it relies on content negotiation. The client (AI bot) requests a format, and the server responds accordingly.
The content remains the same. Only the format changes.
This reduces the risk of cloaking because the information is not different. It is simply formatted differently.
Understanding the Cloaking Concern
Google defines cloaking as showing different content to users and search engines to manipulate rankings.
With user-agent detection, the server decides what to show based on who is visiting.
With content negotiation, the client asks for a specific format.
That difference matters.
Still, from a practical view:
- Googlebot requesting HTML will see the full webpage.
- An AI bot requesting markdown will see a simplified version.
The information remains the same, but the presentation differs.
Google has not officially said whether markdown via content negotiation would be considered cloaking.
Content-Signal Headers: Another Important Part
When Markdown for Agents is enabled, Cloudflare automatically adds a Content-Signal header.
By default, it sets:
- ai-train=yes
- search=yes
- ai-input=yes
This means the content is allowed to be used for:
- AI training
- Search indexing
- AI agent inputs
This default setting is important.
Publishers who are careful about how AI systems use their content should review these settings before turning the feature on.
Cloudflare introduced the Content Signals framework during its Birthday Week 2025 event. It allows website owners to send signals about how their content can be used.
However, not all bots are required to respect these signals. It depends on the bot operator.
Cloudflare has said that custom Content-Signal options will be available in the future.
Token Tracking with x-markdown-tokens
Each converted markdown response includes an x-markdown-tokens header.
This header estimates how many tokens are in the markdown version.
Developers can use this information to:
- Manage context limits
- Plan chunking strategies
- Optimize AI input size
This is especially useful for developers working with large language models that have token limits.
AI Tools Already Requesting Markdown
Cloudflare mentioned that some AI coding tools already send the Accept: text/markdown header.
Two examples given were:
- Claude Code
- OpenCode
These tools benefit from receiving clean markdown instead of heavy HTML.
New Tracking in Cloudflare Radar
Cloudflare has also added new tracking features in Cloudflare Radar.
Radar now shows:
- Content types served to AI bots
- Distribution by MIME type
- Breakdown by specific crawler
For example, Cloudflare showed tracking for OAI-SearchBot, displaying how much markdown content it receives.
This data is available via:
- Public APIs
- Data Explorer
This gives publishers better visibility into how AI bots interact with their content.
Why This Feature Matters
This update is significant for several reasons.
1. No Need to Build Separate Pages
Before this feature, site owners had to manually create markdown versions if they wanted to serve them.
Now, a single toggle inside Cloudflare’s dashboard can enable it.
2. Faster AI Processing
AI bots get lighter, cleaner content.
This may improve:
- Processing speed
- Cost efficiency
- Data clarity
3. Edge-Level Conversion
The conversion happens at Cloudflare’s network edge, not at the origin server.
This reduces load on the main server.
4. Better Control Signals
With Content-Signal headers, publishers can communicate content usage preferences.
Things Publishers Should Consider
Before enabling this feature, website owners should review:
- Default Content-Signal settings
- Their AI usage policy
- Their monetization strategy
If a publisher does not want AI systems to train on their content, they must check these headers carefully.
It is not just a format change. It also signals permission for AI-related use.
The Bigger Question
The internet is changing.
AI bots are crawling more websites. They are not only indexing content but also using it to generate answers.
Cloudflare’s update treats AI agents as important users, similar to human visitors.
But there are still open questions:
- Will Google officially comment on content negotiation for markdown?
- Will more AI systems request markdown format?
- Will publishers widely adopt this feature?
At the moment, Google has only criticized separate markdown pages served via user-agent detection.
It has not directly addressed this specific implementation.
Looking Ahead
Cloudflare has said it will add custom Content-Signal options soon.
This will allow more detailed control over how content is used.
For now, Markdown for Agents is:
- Optional
- Limited to paid plans
- In beta
- Easy to enable
It does not replace HTML pages. It simply provides an alternative format when requested.
You can also read: Google’s Core Update Could End the Era of Promotional Listicles
Final Thoughts
Cloudflare’s Markdown for Agents is a technical but important update.
It allows AI bots to receive clean markdown instead of heavy HTML. This reduces token usage and improves efficiency for AI processing.
However, publishers must pay attention to default AI usage signals before enabling it.
The feature does not create separate pages. It uses standard content negotiation, which makes it different from the method criticized by John Mueller.
As AI systems become more common in content discovery and search, tools like this may become standard.
For now, it is a useful option for websites already running on Cloudflare’s network.
The decision to enable it should depend on your goals, your content policy, and how you want AI systems to interact with your website.
