The answer to the question of what makes a good product is changing rapidly.
Nearly every developer today is leveraging AI Agents to get work done. Cursor, Claude Code, Windsurf, Gemini CLI — all kinds of Coding Agents have been deeply embedded into daily workflows. A common experience is that when you ask an Agent to call certain APIs or integrate a specific service, some processes run so smoothly they feel like cheating, while others throw repeated errors, or even fabricate a completely non-existent interface out of thin air.
Most people attribute this to:
“The models are not smart enough yet“.
But if we look at it the other way around, it’s not that Agents are not smart enough, but that some products are naturally Agent-friendly, while others are inherently hostile to Agents.
This is extremely similar to SEO more than a decade ago. You think your website has great content, but Google just won’t index it. The problem is not with Google’s algorithm, but that your website has not been optimized for search engines.
The exact same thing is happening to Agents right now, except this time, most products don’t even know they are being “judged”.
Addy Osmani, Director of Engineering for Google Cloud AI, gave this phenomenon a name back in April this year: AEO, short for Agentic Engine Optimization.
“Agent Engine Optimization“.
If SEO is optimization for Google crawlers, AEO is optimization for AI Agents. This is not a distant concept, but a set of fully implementable practical practices. Today, when Agents are becoming the most important “users” of software products, what kind of design can make it easier for Agents to use products may determine the real direction of the future software industry.
How do Agents read documentation?
The following is a very common daily scenario for users.
An engineer opens Cursor and asks the Agent to help him connect to a payment API. The Agent sends an HTTP request, retrieves the documentation page, and makes a judgment 400 milliseconds later. But this document has nearly 200,000 Tokens, which far exceeds the context window capacity of the Agent. The Agent does not throw an error or give a prompt, but silently abandons the document and makes up an integration solution based on the memory in its training data.
What’s the result? The user gets a piece of seemingly reasonable code that never runs properly. It takes two hours of debugging to find out that the Agent provided a long-deprecated endpoint.
Meanwhile, in the Google Analytics backend of this payment company, this visit only leaves one record: zero scroll depth, 400 milliseconds of page stay time, no clicks at all. In the traditional analysis framework, this is a “low-quality visitor”, an anonymous IP with a 100% bounce rate. No one knows this is actually an AI Agent, let alone that this Agent just “judged” their product and sentenced it to death.
This is the fundamental difference between how Agents and humans read documentation. Humans “browse”, while Agents “judge”.
Osmani cited a research paper on the HTTP behavior of nine mainstream Coding Agents in his article, with very intuitive data. The way humans read documentation is: open the homepage, navigate to a certain section, scan a few titles, read a few paragraphs, try the code examples, click two or three links, and spend 4 to 8 minutes. During the whole process, your carefully designed progressive guidance, sidebar navigation, and interactive tutorials are all playing their role.
Agents are completely different. They send a GET request, get the full text, and decide whether to use it or not within 400 milliseconds. The human “user journey” is compressed into a single HTTP request for Agents.
The documentation navigation system, breadcrumb paths, and progressive display that you spent months polishing are all noise in the eyes of Agents — not only useless, but also wasting Tokens.
There are three key differences here. Understanding them means you understand the underlying logic of AEO.
An Agent’s “patience” can be accurately quantified.
Osmani gave a specific number: the first 500 Tokens of your page must answer three questions: what this is, what it can do, and how to get started. If the answers are buried in the middle or at the end of the page, the Agent will most likely give up before reading that far. Humans can tolerate lengthy prefaces, but Agents cannot.
Agents have a strict upper limit for their “appetite”.
Osmani cited a real case: the REST API Quick Start Guide for Cisco Secure Firewall Management Center has 193,217 Tokens, nearly 720,000 characters. This single document can eat up or even burst the entire context window of most Agents. When encountering such documents, the Agent will either truncate the content (losing key information), skip it (treating the document as non-existent), or fall back to its own memory (making up an answer on its own). No matter which outcome it is, the result the user gets is wrong.
What is the reasonable magnitude of the Token budget? Osmani suggests that the quick start guide should be less than 15,000 Tokens, a single API reference page less than 25,000 Tokens, and the conceptual guide less than 20,000 Tokens. Beyond this range, a clear chunking strategy is required.
Agents completely ignore your UI design.
Sidebars, breadcrumb navigation, footer links, interactive code sandboxes — all these elements in the rendered HTML are pure noise for Agents. Worse still, for the same content, the HTML format consumes far more Tokens than the Markdown format, because div tags, CSS class names, ARIA attributes, and inline styles are all counted in the total.
Agents read text, not interfaces.
Understanding these three points leads to a disturbing conclusion. Your product documentation is very likely being silently “sentenced to death” by Agents, while your Analytics backend will only tell you that you got a few more low-quality visitors with a 100% bounce rate today.
Unique characteristics left by different AI Agents on the server | Image
How to “please” Agents?
Since Agents choose tools in a completely different way from humans, how can you make your products be prioritized by Agents?
Osmani proposed a six-layer framework. If you have SEO experience, you will find that this set of things is surprisingly similar in structure, except that the optimization target has changed from Google crawlers to AI Agents.
Level 1: Check robots.txt.
This is the first stop when an Agent visits your website. Many companies modified their robots.txt in 2024 and 2025 out of fear of AI crawlers, blocking the crawler User-Agents of AI companies such as Anthropic, OpenAI, and Google. The logic at that time was to prevent content from being used for training.
But the side effect of this approach is that your documentation completely disappears from the Agent’s world. Before crawling the page, the Agent will check robots.txt. If it finds itself blocked, it will skip directly, no error will be reported, no notification will be sent, and no trace will be left in any log. Your team won’t even know this ever happened.
The good news is that this can be fixed with just ten minutes of auditing. Check if your robots.txt mistakenly blocks the User-Agent of AI Agents, which is probably the AEO work with the highest ROI.
A properly structured llms.txt file looks like this | Image
Level 2: Publish an llms.txt, the Agent version of Sitemap.
In the SEO era, you needed a sitemap.xml to tell Google crawlers the structure of your website. In the AEO era, the corresponding file is called llms.txt.
This is a Markdown file placed in the root directory of your domain, listing all the pages of your documentation, what each page is about, and the approximate Token count. After reading this file, the Agent can accurately locate the exact page it needs, instead of blindly crawling your entire website.
A good llms.txt should meet several conditions: the description should tell the Agent “what can be found” on each page, not just the name of the page. The Token count should be marked so that the Agent can judge in advance whether its context window is large enough to hold the page content. Organize the content by task rather than by product hierarchy. The length of llms.txt itself should be controlled within 5,000 Tokens — an index file cannot be longer than the actual content.
The Lighthouse team of Google Chrome has already started to experimentally check the existence of llms.txt, which indicates that this standard is evolving from a “geek community experiment” to the “default configuration for engineering practices”.
Level 3: Write a skill.md, the “self-recommendation letter” of your product for Agents.
llms.txt tells the Agent where your documentation is. skill.md tells the Agent what your product can do.
The difference between these two things is very critical. llms.txt is a directory that lets the Agent know where to find information. skill.md is a capability statement that allows the Agent to directly judge “whether this API can meet the user’s needs” without spending Tokens reading the full document only to find out the answer is “no”.
The structure of a skill.md usually includes several parts: “What it can do” lists the core capabilities of the product; “Required inputs” lists the necessary parameters and credentials; “Restrictions” lists constraints such as rate limits and Token expiration time; “Key documentation” lists the most relevant links.
A good skill.md allows the Agent to judge “whether this tool is suitable for the current task” in a few seconds, saving a lot of invalid documentation reading. For Agents, the value of this file is equivalent to giving a precise 30-second self-introduction in an interview, instead of letting the interviewer go through all your resume by themselves.
Level 4: Make the Markdown version of the documentation directly accessible.
Many documentation platforms support getting the pure Markdown version of the page by adding the .md suffix after the URL. If your platform supports this feature, make sure it is turned on and discoverable.
The reason is very simple. For the same piece of content, the HTML version consumes far more Tokens than the Markdown version because it contains wrapper divs, CSS class names, ARIA attributes, inline styles, and navigation elements. For Agents, HTML is like a dish wrapped in three layers of plastic wrap: it only wants the food, but has to spend a lot of energy removing the packaging first. Markdown is pure content with all the packaging removed.
In addition, there is an easily overlooked problem: if your documentation requires JavaScript execution to load content (such as a documentation site rendered by a single-page application), the Agent will most likely directly get an empty page. Many Agents use lightweight HTTP clients that do not execute JS. Your content “exists” technically, but it is equivalent to “non-existent” in the Agent’s world.
Level 5: Expose the Token count as metadata.
This level is the easiest to implement, but has very high leverage. Mark the Token count of the page in the page’s meta tag or HTTP response header.
This gives the Agent a basis for pre-judgment: for a page with 8,000 Tokens, the Agent can safely load the full text; for a page with 150,000 Tokens, the Agent knows it only needs to extract relevant paragraphs; for pages exceeding the context window, the Agent can directly use the summary in llms.txt instead.
The implementation is also very straightforward: count the number of characters on the server side, divide by 4 to get a rough Token estimate, and output it as a meta tag.
Level 6: Add a “Copy for AI” button.
This is a very small UX feature, but it solves a real pain point.
When developers collaborate with AI assistants in the IDE, they often need to paste documentation content into the context. Most people directly copy and paste from the rendered web page, but the copied content contains a lot of noise text from navigation bars, footers, and sidebars. A “Copy for AI” button can copy the clean Markdown version with one click, greatly improving the quality of the context received by the Agent.
The documentation sites of Anthropic and Cloudflare have already launched this feature. The engineering cost is very low, but the improvement to the developer experience is very significant.
You don’t need to complete all six layers at once. Osmani’s priority order is: audit robots.txt first (10 minutes), then add llms.txt (a few hours), then measure and expose the Token count (one weekend), then write skill.md for your three most important APIs, and finally add the “Copy for AI” button. Most of the work can be completed in one weekend, but the changes it brings are structural.
It benefits Agents, and it benefits you too
There is a common misunderstanding about AEO: some people think it is sacrificing the human reading experience to accommodate machines.
The opposite is true. One of the most interesting insights in Osmani’s framework is that AEO highly overlaps with good human documentation design.
Put the answer at the top of the page instead of burying it after three paragraphs of background introduction. Control the length of a single page so that each page focuses on one thing. Organize the content with a clear hierarchical structure instead of writing it as a single long article. Remove unnecessary navigation noise. These practices also bring a better experience for human readers.
The only difference is fault tolerance. Human readers can tolerate imperfect documentation: if the answer is buried deep, they can always find it after flipping a few more pages; if the format is messy, they can still understand it based on experience; if the page is too long, they can use Ctrl+F to search for keywords.
Agents won’t give you these chances. Their judgment is fast, ruthless, and irreversible. Skip if you can’t find the answer within 500 Tokens. Skip if the Token count exceeds the limit. Skip if JavaScript execution is required to load the content. Every “skip” means your product disappears from that Agent’s tool list, and you don’t even know it ever happened.
You can track traffic from AI Agents in analytics tools right now | Image
Osmani was surprised after checking his website server logs: there is already a huge amount of Agent traffic in them, but they are classified as “junk visitors” with a 100% bounce rate in traditional analysis tools. These silent visitors are determining the position of your product in the Agent ecosystem, while most companies are completely unaware of it.
From this perspective, Agents are actually evaluating your product in a more honest way. They won’t give you a few more seconds of patience just because of your high brand awareness, won’t prioritize you just because your sales team has treated them to a meal, and won’t follow the crowd just because “everyone is using it”. They only care about one thing: whether you can let them understand who you are and what you can do in the shortest possible time.
In the latest episode of the 20VC podcast, investor Jason Calacanis said a very vivid sentence: roughly meaning you can’t treat Agents to a steak dinner to make them buy your product, they only care about whether your product works well
