Where the files go
Every file in the pack goes at the root of your site, so it answers at yoursite.com/robots.txt, not in a folder. How you put it there depends on your host. Below: the four steps, then the details for WordPress, Squarespace, Wix, Shopify, Cloudflare and ordinary hosting. Your pack's INSTALL.md has the same steps with your own addresses filled in.
1. Get the files
After paying you land on your order page. Its address is your download link for 30 days (90 on the $9.97 pack), and we email it to you too.
- Download all eight files as one ZIP.
- Or view and download each file on its own.
- Change your robots.txt choice for AI crawlers at any time; download robots.txt again afterwards.
- Re-run the crawl after you change pages or install the files: 10 re-runs are included.
2. The root of your site
The root is the folder your home page lives in. A file put there answers at your domain followed by its name. Crawlers only look there: a robots.txt in /files/robots.txt is never read.
3. Your host
| Host | llms.txt | robots.txt | sitemap.xml | schema |
|---|---|---|---|---|
| WordPress | Yes | Yes | Yes, or your SEO plugin's | Yes |
| Squarespace | Pasted in SEO settings (7.1) | Not editable | Made by the host | Code Injection |
| Wix | Not confirmed | Its editor | Made by the host | Yes |
| Shopify | Theme template | Theme template | Made by the host | theme.liquid |
| Cloudflare | Yes | Yes | Yes | Yes |
| Ordinary hosting | Yes | Yes | Yes | Yes |
WordPress
WordPress has made its own sitemap at /wp-sitemap.xml since version 5.5, and it generates a robots.txt by itself. Yoast SEO and Rank Math can also make llms.txt: in Yoast it is Yoast SEO > Settings > Site Features > AI tools > LLMS.txt, free; in Rank Math, the LLMS Txt module in its Dashboard. Keep theirs if you like; then use only our robots.txt, schema and fix list, and delete the line "Sitemap: https://yoursite.com/sitemap.xml" from our robots.txt. Yoast · Rank Math · WordPress 5.5
- Open your hosting account's file manager, or connect with SFTP.
- Go to the folder with wp-config.php and wp-content in it.
- Upload robots.txt, llms.txt and llms-full.txt, and sitemap.xml unless you keep your plugin's. Download any file you replace first, as a backup.
- Open yoursite.com/robots.txt. Most hosts serve a real file before WordPress sees the request; if you still see WordPress's version, a plugin is answering for it, so change robots.txt in that plugin's settings instead.
Squarespace
- llms.txt: on Squarespace 7.1, open the SEO panel, then SEO Settings > LLMS.txt, paste the contents of our llms.txt and save. It is off until you do. There is one field, so llms-full.txt has nowhere to go. support.squarespace.com
- robots.txt: every Squarespace site uses the same file and it can't be edited, so ours can't be installed. For AI crawlers, use Settings > Crawlers > "Block known artificial intelligence crawlers". Its list is longer than ours and not the same. support.squarespace.com
- sitemap.xml: Squarespace makes its own at /sitemap.xml and it can't be edited. Ours lets you check that theirs lists the right pages.
- The schema: Settings > Website Tools > Code Injection, the Header field, inside a script tag (Core, Plus and Advanced plans). support.squarespace.com
Wix
- robots.txt: in your dashboard, SEO & GEO > Tools and settings > Robots.txt Editor > View File. Replace the text with ours, keep any line Wix says it needs, and save. Wix Customer Care doesn't help with changes made here. support.wix.com
- sitemap.xml: Wix makes its own and lists it in robots.txt. Keep it.
- llms.txt: we found no Wix help page about llms.txt. Check the SEO & GEO section of your dashboard; if it isn't offered, it can't go at the root.
- The schema: in the home page's SEO settings, Structured Data Markup > + Add New Markup, paste the JSON (without the script tag) and Apply. Wix takes up to 5 blocks a page, each under 7,000 characters. support.wix.com
Shopify
All three changes are in your theme's code: Online Store > Themes > … > Edit code. Duplicate the theme first so you can go back. help.shopify.com
- robots.txt: Add a new template > robots > Create template. Keep Shopify's default rules and add our extra lines (the Sitemap: line and the AI crawler group) below them. Shopify calls this an unsupported customization; deleting the template brings the default back. help.shopify.com
- llms.txt: Shopify already answers /llms.txt for every store. To serve ours, add a templates/llms.txt.liquid template and paste our llms.txt into it. shopify.dev
- sitemap.xml: Shopify makes its own. Keep it.
- The schema: in layout/theme.liquid, inside the head. Many themes already output Organization data; check first, so you don't add a second block.
Cloudflare (Pages, Workers)
- Put robots.txt, sitemap.xml, llms.txt and llms-full.txt in the folder that ends up at the root of your build (often public/ or static/).
- Deploy as usual.
- If Cloudflare's managed robots.txt is on (Security Settings > Bot traffic > Configure AI bot policies), Cloudflare adds its own lines in front of your file. Keep one source of AI rules: either that setting or our AI crawler group. developers.cloudflare.com
Ordinary hosting (cPanel, FTP, Netlify, GitHub Pages, S3)
- Open the file manager or connect with FTP or SFTP.
- Find the folder with your home page (index.html or index.php) and upload the four files there.
- Netlify, GitHub Pages and Vercel: put them in the folder that gets published and deploy. S3: upload them to the bucket's root with Content-Type text/plain for the .txt files and application/xml for the sitemap.
4. The schema
schema.jsonld describes your business to search engines. Open it, replace every TODO with the real value or delete that line, then paste it into your home page's head section like this:
<head>
…
<script type="application/ld+json">
{ "@context": "https://schema.org", "@type": "LocalBusiness", "name": "…" }
</script>
</head>
Then test the page with Google's Rich Results Test. search.google.com/test/rich-results
5. Check it worked
Open each address in your browser. You should see the plain text of the file. If you see your home page or an error page, the file isn't at the root yet.
Then go back to your order page and press Re-run: the new crawl should find your robots.txt and sitemap. Submit the sitemap to Google in Search Console. support.google.com
Before you buy: what the preview shows
- How many pages we read and how many can go in the sitemap and llms.txt.
- Which files your host can take, before you pay.
- Pick what AI crawlers are asked to do.
- See exactly which lines are added (+) to your current robots.txt.
- $4.97 for up to 50 pages, or $9.97 for up to 250 pages and 90 days.
- Your host, so INSTALL.md has the right steps (you can change it later).
- Buy opens the payment window on the same page.
- The email the download link goes to.
- Confirm you own or manage the site.
- Card, Apple Pay, Google Pay or Link appear here once the box is ticked. Card details are handled by Stripe; we never see your card.