The three mistakes that cost the most
Disallow: / left behind after launch — that single line blocks the whole site. It is the robots.txt equivalent of a stray noindex tag, and it routinely goes unnoticed for months. Open yoursite.com/robots.txt right now and read the first lines: it takes five seconds and it is the highest-value check on this page.- Blocking CSS and JavaScript — Google needs to load your layout to judge the page, on mobile above all. Block them and you hand it a broken page.
- Confusing robots.txt with noindex — the first says « do not visit », the second says « do not display ». A page blocked in robots.txt can stay in the results, because Google was never able to read its noindex.
A robots.txt file does not push a site up the results: it stops the site from blocking itself. That is the whole job, and it is worth doing right once.
WordPress: the virtual robots.txt
WordPress serves a virtual robots.txt: it answers correctly at yoursite.com/robots.txt, but no such file exists on your hosting. You will not find it by digging through your folders, and that is normal.
Telling a virtual file from a real one
Open yoursite.com/robots.txt. If all you see is three or four lines around /wp-admin/, that is WordPress's virtual file. The moment you drop a real file at the root, WordPress steps aside and yours is served instead.
Where to edit it without touching the server
- Yoast SEO: Yoast SEO → Tools → File editor.
- Rank Math: Rank Math → General Settings → Edit robots.txt.
- No plugin: create a text file named
robots.txt and upload it to the root, next to the wp-content folder.
The WordPress setting that makes a site invisible
Under Settings → Reading there is a checkbox, « Discourage search engines from indexing this site ». Ticked, it does not just alter the robots.txt: it adds a noindex tag across the entire site. It is the number one cause of invisibility after a redesign, and no amount of robots.txt tuning will fix it while the box stays ticked.
Should you block /wp-json/?
Plenty of guides recommend it. We do not: Google does not index those technical responses anyway, while blocking them can break blocks and plugins that rely on them. The risk outweighs the benefit.
Frequently asked questions
What is a robots.txt file for?
It is a text file at the root of your site telling search engines which areas they may crawl. It protects nothing and hides nothing — it is a traffic sign, not a lock. Its single most useful job is pointing to your sitemap.
Where do I put the robots.txt file?
Always at the root of the domain, reachable at yoursite.com/robots.txt. Anywhere else and it is ignored. On WordPress, an SEO plugin (Yoast, Rank Math) lets you edit it without touching the server.
Can robots.txt make my site invisible on Google?
Yes, and it happens often. A single line — Disallow: / — blocks the entire site, and it is regularly left behind after a launch. Together with a stray noindex tag, it is one of the two most common causes of a site vanishing.
What's the difference between robots.txt and noindex?
robots.txt asks crawlers not to VISIT a page; noindex asks them not to SHOW it in results. The classic trap: a page blocked in robots.txt can stay in the results, because Google was never allowed to read its noindex. To remove a page from search, use noindex and do NOT block it in robots.txt.
Should I block images, CSS or JavaScript?
No. Google needs your stylesheets and scripts to see the page the way a visitor does. Blocking them degrades how it understands your site, mobile first of all.
Does this work for Wix, Squarespace or Shopify?
On Shopify you can edit the file through your theme code. On Wix it is editable from the SEO tools. On Squarespace the platform owns it and does not allow edits — there the template is a reference to check nothing essential is blocked.
Do I need a robots.txt file at all?
Not strictly. A site with no robots.txt is crawled entirely, which is usually what a small site wants. It becomes useful when you have areas worth keeping out of the crawl, and it is always useful as the place to declare your sitemap.
