Developer Tool Showdown: Which Devtool Site Is the Most AI-Ready?

I compared 10 of the biggest developer-tool sites on AI readiness: GitHub, GitLab, Bitbucket, Stackoverflow, dev.to, Vercel, Netlify, Cloudflare, DigitalOcean, Heroku. The results will surprise you. One of them actively blocked the scanner. Another has a 27 KB llms.txt with 112 links and still only gets a C.

Why This Matters

When we published the State of AI Crawlers 2026 report, the most common pushback was: “Those are content sites. Show us how the tools we actually use every day look.” Fair. So I ran the same scanner against 10 of the most widely used developer-tool and platform sites.

This is not a content–farm ranking. These are the sites where developers host code, deploy apps, read documentation, and publish blogs. If any category of the open web should be setting a good example for AI readiness, it is this one. They are talking to crawlers constantly — Googlebot, GPTBot, Bingbot, every code–search indexer that has ever existed. So what does the data look like when you run a multi-signal scanner against all of them?

The Ranking

Every site was scanned live via the free ZeroKit AI Readiness API on April 10, 2026 at 20:46 UTC. The scanner checks five categories: robots.txt for AI bots (30 points), llms.txt (20), structured data (25), content citability (15), and AI meta directives (10). Full methodology is at the bottom of the page.

Some immediate observations before we dig into each category:

• The top three are all platform-as-a-service companies (Vercel, Heroku, GitHub), and they are the only three that cleared a 60 point score.
• The drop from #3 (GitHub, 67) to #4 (DigitalOcean, 48) is 19 points. There is a real cliff in the middle of the rankings.
• The bottom five are within 7 points of each other. Essentially interchangeable.
• Netlify is the only site in the bottom half with a valid llms.txt — and its score is being dragged down by a 0/30 in robots.txt. We will explain that below.

llms.txt: The Devtool Surprise

In our top-100 report, only 31% of the largest content sites had an llms.txt file. In this devtool sample, 4 of 9 scanned sites (44%) have one: Vercel, Heroku, GitHub, and Netlify.

GitHub’s llms.txt is the most impressive by raw content: roughly 27 KB, 112 links, full h1 + h2 sections, summary blockquote. It scores 19 out of 20 in that category — the highest in the sample. Vercel has 1,733 links in its llms.txt, by far the largest, which scores 18/20 (the extra volume does not buy extra points beyond a certain threshold).

But here is the twist: having a great llms.txt does not save your overall grade if you neglect robots.txt. GitHub has the best llms.txt in the sample and still only lands at a C. Why? Because its robots.txt score is 8 out of 30 — the scanner could not find explicit AI-bot rules. Every AI bot on GitHub is in the “allowed by default” state, which loses points in the rubric because it is not a deliberate decision.

robots.txt: Where Everyone Fails

The highest robots.txt score in the sample is 12/30, shared by Vercel, Heroku, and Cloudflare. The average is roughly 8/30. Two sites scored 5 (GitLab, Bitbucket) and one scored 0 (Netlify).

Why is Netlify a zero? Because their robots.txt at the time of the scan returned no content the scanner could parse into explicit AI-bot rules. Netlify users have their own robots.txt per site, but the corporate netlify.com marketing domain — the one we scanned — did not contain recognized AI-bot directives. For a company whose business is letting developers configure HTTP headers and rewrites, that is a surprising oversight.

GitLab and Bitbucket both scored 5/30 — low. You would expect the two largest Git hosts after GitHub to at least match GitHub’s posture, but GitHub scored 8/30 and GitLab scored 5/30. The gap is small in absolute terms but directionally meaningful: GitHub is explicitly engaging with AI crawlers, GitLab is not.

The real story: not one of these sites has a robots.txt score above 12 out of 30. There is no winner here, only less-bad losers. Every single one of them allows every major AI bot by default because they have not written explicit rules. If you are a devtool startup and you want a competitive differentiator on AI governance, this is the cheapest one you will ever find.

Structured Data and Citability: The Quiet Leaders

Structured data (JSON-LD, microdata, Open Graph depth) is the one category where the top of the field is actually strong. Vercel and GitLab both hit 22/25, Heroku 22/25, GitHub 21/25, DigitalOcean 21/25. The laggards are Netlify and dev.to, both at 12/25 — they ship basic OpenGraph but not much else.

Content citability (whether an LLM can cleanly quote a sentence from the page) is also uniformly decent. Most sites score 12–13 out of 15. The only outlier below that is DigitalOcean at 11/15, and that is a small enough margin that it is not a real signal.

The implication: when these sites lose points, they lose them in the AI-specific categories (robots.txt and llms.txt), not in the general-SEO categories they have been investing in for a decade. Structured data is a solved problem. AI rules are not.

The Stackoverflow Story

Stackoverflow returned HTTP 403 to our scanner. The scanner fetches the homepage with a generic user-agent and checks robots.txt, llms.txt, and structured data on the fetched HTML. Stackoverflow refused to serve the homepage. Every category scored 0.

This is not a bug in the scanner. Multiple large sites now block third-party crawlers aggressively — Cloudflare Bot Management, Kasada, PerimeterX, and similar products are everywhere. Stackoverflow is not the only site on the web that does this, and it is entirely within its rights to do so. But for the purpose of measuring “AI readiness” as a public–web concept, a site that refuses to be audited by a basic open–source scanner is effectively opaque.

The practical consequence: if you build an AI agent that fetches Stackoverflow pages with a generic user-agent, you will get a 403 too. If you run an llms.txt-style governance check in your pipeline, you will not get any data from Stackoverflow. The only way to read Stackoverflow programmatically is through their official API, which has its own licensing terms. That is a meaningful, underreported detail about the state of the developer web in 2026.

Why Is GitHub Only a C?

Several people will read this ranking and do a double-take at GitHub getting a C. GitHub is the largest code host on earth, it has an llms.txt with 112 links, it serves canonical structured data on every repo page, and it is on literally every crawl list that exists. How can it only score 67?

The answer is narrow and specific: the rubric weights explicit bot rules heavily, and GitHub does not write explicit rules for AI bots. GitHub’s robots.txt is long and sophisticated in the SEO sense — it controls which directories classic search engines crawl — but it does not contain User-agent: GPTBot followed by an allow or disallow. Every AI bot is in the “allowed by default” state, which the scanner flags as an undeliberate policy.

That is a reasonable product choice for GitHub. Their business model is open source code, and defaulting open is consistent with that. But it does mean their AI readiness score in any rubric that rewards explicit policy will be capped. The fix is a one-line addition to robots.txt that makes their position official. Until then, they sit at a C, not because they are unprepared, but because they have not said so out loud.

Is Vercel Really the Most AI-Ready Devtool Site?

Yes — on this rubric, with this sample, today. Vercel wins on structured data (22/25), wins on content citability (13/15), has an llms.txt with 1,733 links, ties for the best robots.txt score (12/30), and has the highest overall score at 71. It is the only site in the top three that wins three of five categories.

The caveat is that 71/100 is still a B. Vercel has 29 points of headroom. The biggest gap is robots.txt, where the whole field is failing. If Vercel added 10 lines of explicit AI-bot rules tomorrow, they could likely push into the 80s, and they would be the only developer-tool site in the A range. At the moment, nobody has claimed that position.

Methodology

Run the Same Scan Yourself

Everything in this post is reproducible. The scanner is a public API with no auth, no rate limiting beyond basic abuse protection, and no hidden weights. You can run it on your own site in 5 seconds, on any competitor, or on the same 10 domains we used.

Related