DashboardSite Health
Robots.txt
Generate a robots.txt file that tells search engines and AI crawlers which parts of your site to read.
Paid plan feature. Without a paid plan, you'll see an option to start your free trial or upgrade. See what's included in each plan.
Your robots.txt file tells search engines and AI crawlers which parts of your site they may read. The Robots.txt page builds one for you, so you don't have to write the rules by hand.
Create your robots.txt
- Open Site Health → Robots.txt in the dashboard.
- Pick the options you want, such as Block AI Crawlers or Disallow /admin directories.
- Press Generate Rules.
- Copy or Download the file, then add it to your site yourself.
A banner at the top tells you what the latest site audit found, for example a missing file or one that blocks every crawler.