Free previewFiles $4.97No sign-upObeys your robots.txt
Site Files pack
Type your address. We read up to 50 pages and write six files from their own titles and text: llms.txt, llms-full.txt, sitemap.xml, robots.txt, a business schema block and a list of what to fix, with the install steps for your host.
- See your files before you pay: the llms.txt opening, the robots.txt changes line by line, the first ten sitemap entries and the top three fixes.
- sitemap.xml lists only pages that answer 200, are not noindex and are their own canonical, checked with our free sitemap checker's reader before you get it.
- robots.txt keeps every rule your file has for Googlebot, Bingbot and other crawlers. We test it: they get the same answer for every page we read.
- Install steps for WordPress, Squarespace, Wix, Shopify, Cloudflare and ordinary hosting, including which files your host can't take.
Purchases are paused. You can still run the free preview and see what the files would say.
How it works
- Preview, free Type your domain. We read robots.txt, your sitemap and up to 50 pages, six at a time. Most sites take under a minute. You see what the files will say.
- Pay once $4.97 for up to 50 pages, or $9.97 for up to 250 pages with 90 days of re-runs. Pay by card, Apple Pay, Google Pay or Link without leaving the page. Card details are handled by Stripe; we never see your card.
- Upload and check Download the ZIP or each file, follow the steps for your host, then re-run the crawl to check. Re-runs are included: 10 in 30 days, or 10 in 90.
What each file is for
| File | What it does | Who reads it | Where it goes |
|---|---|---|---|
| llms.txt | A Markdown map of your site: its name, a one-line summary, then your pages by section, each with its description. | AI tools and agents that look for it. No AI company has said its crawler reads it. | /llms.txt |
| llms-full.txt | The same map followed by the text of each page (up to 6 KB a page, 12 KB on the $9.97 pack). | A convention from documentation hosts such as Mintlify, not part of the llms.txt proposal. | /llms-full.txt |
| sitemap.xml | The pages you want found, with a date only when your server gave a real one. | Google, Bing and other search engines. Up to 50,000 addresses a file. | /sitemap.xml |
| robots.txt | Your current rules for Googlebot, Bingbot and other crawlers, kept, plus a full Sitemap: line and your choice for AI crawlers. | Every well-behaved crawler. Google reads the first 500 KiB. | /robots.txt |
| schema.jsonld | Organization or LocalBusiness structured data, filled from your site, with TODO where we found nothing. | Search engines, when it is on your home page. | inside the home page's head |
| FIXES.md | Every problem we found, most important first, with the pages and what to change. | You, or whoever runs your site. | nowhere: it is for you |
What we know about llms.txt, and what we don't
llms.txt is a proposal Jeremy Howard published on 3 September 2024 (llmstxt.org). Many sites and documentation hosts publish one, and OpenAI, Anthropic and Google publish one for their developer docs. That is adoption by publishers. We found no statement from an AI company that its crawler reads the file, and Google says no special file is needed to appear in AI Overviews. In April 2025 Google's John Mueller compared it to the keywords meta tag. So we don't promise that it gets you into AI answers. It costs nothing to have, it does no harm, and a correct one gives any tool that does look a clean map of your site. That is all we claim for it.
What robots.txt can and can't do about AI
robots.txt is a request: well-behaved crawlers follow it, and Google and Cloudflare both say following it is voluntary. It also can't take back pages already collected. Some AI crawlers collect pages for training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot), others decide whether your site shows up in AI search answers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). OpenAI says blocked sites "will not be shown in ChatGPT search answers", so for most small businesses we suggest leaving those open. Two fetchers act when a person asks about a page, and their operators say robots.txt "may not apply" (ChatGPT-User) or is "generally" ignored (Perplexity-User). Googlebot and Bingbot are never touched by any of our three choices.
On WordPress, Squarespace, Wix or Shopify?
On WordPress, Yoast and Rank Math already make llms.txt and a sitemap for free; our pack adds the robots.txt choices, the fix list and the schema, so decide if that is worth $4.97 to you. Squarespace doesn't let you edit robots.txt or upload files, but 7.1 sites have an LLMS.txt field you can paste ours into. Shopify takes robots.txt and llms.txt as theme templates. Hosted builders make their own sitemap. The preview tells you which files your host can take before you pay.
Questions
Will these files improve my rankings?
We can't promise that, and nobody honest can. A clean sitemap helps search engines find your pages, and a robots.txt with a Sitemap: line helps them find the sitemap. Google decides what to index and how to rank it. The fix list is where most sites have real work to do.
What if my site has more than 50 pages?
The preview says so. The $4.97 pack covers the first 50 pages we reach (the home page, then your sitemap, then links). The $9.97 pack reads up to 250. For bigger sites, ask us for a quote.
My site is built with JavaScript. Will it work?
We read the HTML your server sends, without running scripts. If your pages are empty without JavaScript, the preview warns you before you pay, and the files would be thin. That is also how many crawlers see your site.
Can I change my mind about the AI crawlers later?
Yes. After paying, you can switch between the three choices and download robots.txt again as often as you like while your order lasts.
What is a re-run?
A fresh crawl of the same site, so the files match your pages after you add or change some, or after you install them. 10 re-runs are included: within 30 days on the $4.97 pack, 90 days on the $9.97 pack.
What if the files are wrong?
Files are delivered instantly, so sales are final. If the pack doesn't work for your site, tell us within 7 days and we'll fix it or refund you. A refund ends the download link.
Will the crawl slow my site down?
Unlikely. We fetch at most six pages at a time, 150 ms apart, read the first 128 KB of each, and skip every page your robots.txt closes to our crawler. A site can be crawled by the pack about six times an hour in all, and a preview of the same site within 24 hours reuses the last crawl.
What do you keep?
The crawl of public pages, 24 hours for a preview and for as long as your order lasts when you buy. Stripe has your email and card; we keep a one-way hash of the email and the Stripe ids. Our privacy policy has the details.
Related
- Install guide: where each file goes on WordPress, Squarespace, Wix, Shopify, Cloudflare and ordinary hosting.
- Sitemap checker: check the sitemap you have now, free.
- SEO checker: check a page's title, description, canonical and links.
- About our checker's requests: what it fetches, and how to opt out.