What this test checks
AI search and answer crawlers
robots.txt rules for the 8 crawlers that fetch pages to answer users: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot and Bingbot.
Home page and inner pages
Rules are checked for your home page and up to 3 inner pages linked from it, to catch sites that open only the home page to crawlers.
AI training crawlers
Rules for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent and Amazonbot, reported as a content-licensing choice and not scored.
Firewall and bot protection
Whether requests with OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot user agents get the same page as a browser, or an error, a challenge page or a cut-down page.
llms.txt and llms-full.txt
Whether /llms.txt exists and follows the llmstxt.org layout (title, summary, sections with links), and whether an optional llms-full.txt is offered.
Content without JavaScript
Whether the main text is in the HTML itself, because most AI crawlers do not run JavaScript.
Quoting allowed
nosnippet or max-snippet:0 directives that tell Google, including AI Overviews, not to quote the page.
Sample result
A real result for devteam.co.kr, tested 1 hour ago.
AI crawler access
8 of 8 AI search crawlers allowed in robots.txt, 4 of 4 served by your firewall, llms.txt present.
AI search crawlers allowed (robots.txt) (Good)
All 8 AI search and answer crawlers may read the site.
8 / 8 allowed
AI crawlers not blocked by firewall (Good)
Requests with ChatGPT, Claude and Perplexity crawler user agents received the same page as a browser.
4 / 4 served
llms.txt (Good)
llms.txt follows the llmstxt.org format: 19 links in 6 sections.
4KB
Content readable without JavaScript (Good)
The page text is in the HTML itself, so crawlers that do not run JavaScript can read it.
4,756 chars of text
robots.txt rules for AI and search crawlers
Real requests with AI crawler user agents
llms.txt
DevTeam Korea(주식회사 뎁팀)는 한국의 소프트웨어 개발사입니다. 스타트업 MVP, 정부지원사업 과제 개발, AI 연동·사내 AI 도입, ERP·그룹웨어·WMS 시스템 연동, Flutter 앱, Electron 데스크톱 앱, 다국어 웹사이트를 기획부터 배포·운영까지 한 팀이 맡고, 운영 중인 사이트의 장애(502·500 에러, 해킹, 인증서 만료)를 긴급 복구합니다. 소스코드와 서버 권한은 모두 고객에게 이관합니다.
How it works
- Our test server in Virginia, USA, fetches /robots.txt and applies each crawler's rules as RFC 9309 describes; a robots.txt that returns a server error counts as blocking everything.
- Your page is then requested at the same moment with a regular Chrome user agent and with the published user agents of OAI-SearchBot, GPTBot, ClaudeBot and PerplexityBot, and the responses are compared.
- These requests come from our server, not from the AI companies' IP ranges, so a firewall that verifies crawler IP addresses may treat the real bots differently.
- We also request /llms.txt and /llms-full.txt and read your page HTML without running JavaScript.
- Every request goes only to public IP addresses on ports 80 and 443, and the check usually takes about 10 seconds.
How it is scored
The score uses our item-weighted formula: a pass earns full credit, a warning half and a fail none. AI search crawler access, the firewall check and readable content without JavaScript count 3×; llms.txt and permission to quote 2×; llms-full.txt 1× when present. Training-crawler rules are reported but not scored, and a missing llms-full.txt is treated as optional.
| Check | Good | Needs work | Poor |
|---|---|---|---|
| AI search crawlers allowed in robots.txt | All 8 | 6–7 | 5 or fewer |
| AI crawler user agents at your firewall | Same page as a browser | Other error, no response or a much shorter page | HTTP 401, 403, 406, 429, 451 or 503, or a challenge page |
| Text in the HTML without JavaScript | Readable text present | — | Under 250 characters on a script-built page |
| llms.txt | Title, links and a summary or sections | Missing or incomplete | — |
| nosnippet or max-snippet:0 | Not set | — | Set |
Grades: A+ 95–100 · A 90–94 · B 80–89 · C 70–79 · D 60–69 · F below 60.