robots.txt Size Checker (500 KiB Limit)

Counts the bytes of a pasted robots.txt, compares them with the 500 KiB limit and, if it is too big, shows which line is cut.

Use the tool

Why this tool exists

RFC 9309 tells crawlers to parse at least 500 kibibytes, and Google ignores content past that limit. A file that grows through generated rules can silently lose its last lines, which might hold the rules that matter.

The checker counts real bytes (UTF-8, not characters), shows the share of the limit used and marks the first line a crawler would not read. It also suggests ways to shrink the file.

How to use it

  1. Paste the whole robots.txt.
  2. Read the size and percentage of the limit.
  3. If over, note the cut-off line and what comes after it.
  4. Shrink with wildcard patterns or the minifier.

Frequently asked questions

Why 500 KiB?

RFC 9309 says crawlers must parse at least 500 kibibytes, and Google documents the same limit.

Are bytes different from characters?

Yes. Non-ASCII characters take several bytes in UTF-8, so the byte count is what matters.

Can I change the limit?

Yes, the option lets you test another limit, for example for a stricter crawler.

What happens past the limit?

Google ignores the rest of the file. Other crawlers may behave differently.