Robots.txt and Sitemap Checker: Crawlability Test

What this test checks

robots.txt availability

Whether /robots.txt returns 200, is missing (404) or fails with a server error, which makes Google pause crawling of the entire site.

Search engine access

Whether the rules let Googlebot, Bingbot, Naver's Yeti and Daum's crawler reach your homepage.

AI search crawlers

Whether OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Applebot may crawl the site; blocked AI training crawlers are listed but not scored.

Blocked CSS and JavaScript

Disallow rules on asset folders such as /wp-content/, /assets/ or /_next/static/ that stop Google from rendering your pages correctly.

Syntax and file size

Lines crawlers cannot parse, and files over 500 KB, beyond which Google ignores the remaining rules.

Sitemap discovery and validity

Sitemap: lines in robots.txt and common sitemap paths, well-formed XML in the sitemaps.org format, gzipped files and sitemap indexes.

Sitemap contents

How many URLs are listed, whether lastmod dates are present and recent, and whether every URL uses your final host and https.

Sampled URL health

A spread of sitemap URLs requested directly to confirm they return 200 without redirects or errors.

Sample result

A real result for devteam.co.kr, tested 16 hours ago.

Crawlability: robots.txt & sitemap

Parsed the robots.txt rules. Problems: 0, improvements: 0. URLs in the sitemap: 184; most recent update 2026-09-27. Naver search readiness: 6/8 items ready.

96/100A+

AI training crawler policy

AI training crawlers (GPTBot and others) are not blocked. If you do not want your content used for AI training, you can block them selectively.

None blocked

robots.txt file

robots.txt is served correctly.

HTTP 200 · 1.1KB

Target: /robots.txt returns 200 · text/plain

Search engine crawling

Google, Bing, Naver (Yeti) and Daum can all crawl the homepage.

All allowed

Target: Google, Bing, Naver and Daum allowed

Sitemap directive

Tells crawlers where your sitemap is.

Target: Sitemap: https://.../sitemap.xml

AI search crawlers

AI search services such as ChatGPT search, Claude and Perplexity can read and cite your site.

All allowed

Target: ChatGPT, Claude and Perplexity search bots allowed

How we test this →

Sitemap file

Sitemap found (URL list).

https://www.devteam.co.kr/sitemap.xml

Target: sitemap.xml served · listed in robots.txt

URL

https://www.devteam.co.kr/sitemap.xml

Search engine crawlers

GoogleGooglebot

Allowed

BingBingbot

Allowed

NaverYeti

Allowed

Daum (Kakao)Daumoa

Allowed

Sitemap

184URLs listed

179with lastmod

2026-09-27Newest update

URL listhttps://www.devteam.co.kr/sitemap.xml

Show robots.txt

# DevTeam Korea (주식회사 뎁팀) — https://www.devteam.co.kr
# AI guidance: https://www.devteam.co.kr/llms.txt
# Full context:  https://www.devteam.co.kr/llms-full.txt

# 기본: 모든 크롤러 허용 (비공개/시스템 영역만 차단)
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /client/
Disallow: /google/
Disallow: /github/
Disallow: /logout
Disallow: /estimate/
Disallow: /pay/
Disallow: /partner/
Disallow: /login
Disallow: /cdn-cgi/
Disallow: /images/contact/

# AI · 답변 엔진(GEO/AEO) 크롤러 — 공개 콘텐츠 학습/인용 환영
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bingbot
User-agent: CCBot
Allow: /
Disallow: /admin/
Disallow: /client/
Disallow: /google/
Disallow: /github/
Disallow: /logout
Disallow: /estimate/
Disallow: /pay/
Disallow: /partner/
Disallow: /login
Disallow: /cdn-cgi/
Disallow: /images/contact/

Sitemap: https://www.devteam.co.kr/sitemap.xml

Suggested robots.txt

User-agent: *
Allow: /
Disallow: /admin/

# Allow AI search and answer engines (GEO)
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

Sitemap: https://www.devteam.co.kr/sitemap.xml

How it works

Our test server in Virginia, USA, requests /robots.txt as WebPSQCBot/3.0 and applies its rules as Google does (RFC 9309): the most specific user-agent group and the longest matching rule win.
Sitemaps are looked for in the robots.txt Sitemap: lines and at /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml, reading up to 12 MB per file.
For a sitemap index, the first 3 child sitemaps are read, and 6 URLs spread across the list are requested to check their status.
Korean sites, detected by the html lang attribute, a .kr domain or mostly Korean text, also get Naver Search Advisor readiness items.
Every request is limited to public IP addresses on ports 80 and 443, and the check usually takes about 8 seconds.

How it is scored

The score uses our item-weighted formula: a pass earns full credit, a warning half and a fail none. Search engine access, the sitemap file and its XML validity count 3×, and so does a robots.txt that returns a server error; robots.txt presence, the Sitemap: line, blocked CSS/JS, AI search crawler access, URL count, URL consistency and sampled URL health 2×; syntax, file size and lastmod 1×. AI training crawler rules are reported but not scored, and on Korean sites the Naver items join the same score.

Check	Good	Needs work	Poor
robots.txt response	200 with readable rules	Missing (404) or an HTML page	Server error or no response
Google, Bing, Naver and Daum crawlers	All allowed	—	Any blocked
AI search crawlers	All allowed	Any blocked	—
XML sitemap	Found and well-formed	—	Missing, empty or broken XML
URLs with lastmod (newest within a year)	80% or more	Fewer, or newest over a year old	—
URLs on another host or http://	None	Up to 10%	Over 10%
Sampled URLs with errors or redirects	None	Up to 20%	Over 20%

Grades: A+ 95–100 · A 90–94 · B 80–89 · C 70–79 · D 60–69 · F below 60.

FAQ

What should a robots.txt file contain?

At minimum, a User-agent: * group with Disallow rules for private areas such as /admin/, and a Sitemap: line with the full URL of your sitemap. Do not block CSS, JavaScript or image folders, because Google needs them to render your pages. The report includes a suggested starter file.

Does blocking a page in robots.txt keep it out of Google?

Not reliably. robots.txt controls crawling, not indexing: a blocked URL can still be indexed without its content if other pages link to it. To keep a page out of search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header instead.

What happens if robots.txt returns a server error?

Google temporarily treats a 5xx response for robots.txt as if the whole site were disallowed, so crawling stops until the file loads again. This test fails the check and treats every crawler as blocked. A missing robots.txt (404) is safe: crawlers assume everything is allowed.

Where should my sitemap be and how do I submit it?

Most sites serve it at /sitemap.xml, but you can use another location if you list it in robots.txt with a Sitemap: line. Also submit it in Google Search Console and Bing Webmaster Tools. One sitemap file may hold up to 50,000 URLs or 50 MB uncompressed; larger sites split it and list the parts in a sitemap index.

Does lastmod in a sitemap matter?

Google uses lastmod as a recrawl hint when it is consistently accurate, and may ignore it when it is not, for example when every URL shows the current date. This test passes when at least 80% of URLs have a lastmod and the newest is less than a year old. Set lastmod to the date the page content actually changed.

robots.txt and XML sitemap checker

What this test checks

robots.txt availability

Search engine access

AI search crawlers

Blocked CSS and JavaScript

Syntax and file size

Sitemap discovery and validity

Sitemap contents

Sampled URL health

Sample result

Crawlability: robots.txt & sitemap

Search engine crawlers

Sitemap

How it works

How it is scored

FAQ