Should You Let AI Crawl Your Therapy Website? How to Block or Allow AI Bots on WordPress
A practical guide to help therapists decide whether to block AI bots, plus step-by-step instructions for robots.txt and Cloudflare.
This is one of the questions I’m getting asked more than almost anything else right now: “Should I let AI crawl my website?” And honestly, it’s a fair question. With ChatGPT, Perplexity, Google’s AI Overviews, and a growing list of AI tools all sending bots to scrape website content, it makes sense that therapists want to know what’s happening and whether they should do anything about it.
The short answer: it depends on what you want from your website. The slightly longer answer is what this guide is for. We’ll look at what AI crawling actually is, the genuine pros and cons for therapy websites, and then I’ll walk you through exactly how to block AI bots on WordPress using robots.txt and Cloudflare.
What is AI Crawling and Why Are Bots Visiting Your Website?
Before we get into the pros and cons, let’s quickly cover what’s actually happening when AI bots crawl your therapy website.
Web crawlers (also called bots or spiders) are automated programs that visit websites and read the content on them. You’re already familiar with some of these: Googlebot has been crawling your site for years so that Google can index your pages and show them in search results. That’s how SEO works.
AI crawlers are a newer type of bot. Instead of (or as well as) indexing your site for search results, they’re reading your content to either train AI models like ChatGPT and Claude, or to fetch information in real time when someone asks an AI tool a question.
The Main AI Bots You Need to Know About
There are quite a few AI crawlers now active on the web. Here are the ones most relevant to therapist websites:
- GPTBot (OpenAI): Crawls content to train OpenAI’s language models. Separate from ChatGPT’s search feature.
- OAI-SearchBot (OpenAI): Fetches content in real time to power ChatGPT search results. This one can send you referral traffic.
- ChatGPT-User (OpenAI): Fetches pages when a ChatGPT user asks it to browse a specific URL.
- ClaudeBot (Anthropic): Crawls content for training Anthropic’s Claude models.
- PerplexityBot (Perplexity AI): Crawls content to power Perplexity’s AI search engine.
- Google-Extended (Google): Crawls content specifically for Google’s AI features like Gemini. Separate from regular Googlebot.
- Applebot-Extended (Apple): Crawls content for Apple’s AI features including Apple Intelligence.
- CCBot (Common Crawl): Crawls the web to build open datasets used by various AI companies.
- Bytespider (ByteDance/TikTok): Crawls content for ByteDance’s AI products.
Important: These are different from regular search engine bots. Blocking GPTBot will NOT affect your Google search rankings. Blocking Googlebot would, so never do that.
The scale of AI crawling is significant. By late 2025, Cloudflare’s network was seeing around 50 billion AI crawler requests per day. AI-oriented bots now account for over 4% of all page requests on the web, and that number is climbing rapidly.
The Case for Allowing AI Bots to Crawl Your Therapy Website
Let’s start with the reasons you might want to keep your doors open to AI crawlers. There are some genuine advantages, particularly for smaller therapy practices looking to increase visibility.
1. Your Content Can Appear in AI Search Results
This is the biggest argument in favour of allowing AI crawlers. When someone asks ChatGPT, Perplexity, or Google’s AI Overview a question like “What should I look for in a therapist for anxiety?” the AI pulls its answer from websites it has crawled. If your site is blocked, your content simply cannot appear in those answers.
AI search is growing fast. User-driven AI crawling (bots fetching content in real time to answer user questions) grew over 15 times in 2025 alone. For therapists, this represents a new channel for potential clients to discover your practice.
2. Competitive Advantage While Others Block
Here’s something a lot of people miss: if your competitors block AI crawlers and you don’t, your content becomes the source that AI tools draw from. For smaller therapy practices competing against larger directories and clinics, this can actually be an advantage. Your expertise fills the gap that others have left.
3. Brand Visibility and Authority
When an AI tool references your content or recommends your approach, it builds authority for your practice. Even if the user doesn’t click through to your website directly, being cited as a source in AI answers reinforces your reputation as a credible expert in your field.
4. Referral Traffic from AI Platforms
AI search tools are starting to send referral traffic back to source websites. It’s not as much as traditional Google search (yet), but it’s growing. If your content is well-structured and genuinely helpful, AI tools are more likely to cite you and link back to your site.
5. Future-Proofing Your Online Presence
The way people search for information is changing. AI-powered search is not a passing trend. Staying visible across both traditional search and AI platforms means your practice is positioned for however search evolves over the next few years.
The Case for Blocking AI Bots on Your Therapy Website
Now let’s look at the other side. There are legitimate reasons to block some or all AI crawlers, and for therapy websites in particular, some of these carry real weight.
1. Your Content is Being Used Without Compensation
This is the core concern for most website owners. AI training crawlers take your content and bake it into their models permanently. They use your expertise to generate answers for their own customers, within their own apps, and they send very little traffic back to you in return.
Cloudflare’s data paints a stark picture: Google crawls about 14 times for every one referral it sends back. OpenAI’s ratio is around 1,700 to 1. Anthropic’s peaked at 500,000 to 1. In other words, these companies are taking a lot and giving very little back.
2. Your Content Could Be Reproduced Without Credit
When someone asks an AI tool a therapy-related question, the answer might be based heavily on your blog post or service page. But the user sees the AI’s answer, not your website. Your carefully written content about anxiety management or CBT techniques gets rephrased and presented as the AI’s own response, with no mention of you.
3. It Can Affect Your Website Performance
AI bots can be aggressive crawlers. Unlike search engine bots that typically crawl at a measured pace, some AI bots can hit your site with large bursts of requests. For therapy websites on shared hosting, this can slow things down for your actual human visitors, the people you’re trying to attract as clients.
4. Privacy and Ethical Concerns
Therapist websites sometimes contain sensitive content about mental health conditions, therapeutic approaches, and client experiences (even anonymised ones). Some therapists feel uncomfortable with AI companies ingesting this content to train commercial products, particularly given the lack of transparency about exactly how the data is used.
5. You Have No Control Over How Your Content is Used
Once an AI model has been trained on your content, you cannot take it back. Unlike a search engine listing (which you can remove), content absorbed into an AI model’s training data becomes part of the model permanently. If the AI then generates inaccurate advice that was loosely based on your content, you have no control over that.
The Smart Approach: Block Training Bots, Allow Search Bots
For most therapy websites, the best approach is not an all-or-nothing decision. It’s a selective strategy:
- Allow AI search and retrieval bots (like OAI-SearchBot, ChatGPT-User, PerplexityBot) so your content can appear in AI-generated answers and send you referral traffic.
- Block AI training bots (like GPTBot, CCBot, Google-Extended) so your content isn’t used to train future AI models without your consent or compensation.
- Always keep Googlebot and Bingbot allowed, as these are your traditional search engine crawlers and blocking them will destroy your Google rankings.
This gives you the best of both worlds: you stay visible in AI search results while protecting your content from being absorbed into training datasets. Let me show you exactly how to set this up.
How to Block AI Bots on WordPress Using robots.txt
The robots.txt file is a simple text file that sits in the root of your website. It tells bots what they are and aren’t allowed to crawl. Every well-behaved bot checks this file before crawling your site.
Important caveat: robots.txt is a request, not a wall. Legitimate bots from companies like OpenAI and Google respect it. Malicious scrapers ignore it entirely. Think of it as a polite “please don’t” sign rather than a locked door. For actual enforcement, you’ll need Cloudflare (covered in the next section).
Step 1: Find Your robots.txt File
If you’re using WordPress with Rank Math (which I recommend for therapist websites), your robots.txt file is managed through the plugin:
- Go to your WordPress dashboard
- Navigate to Rank Math > General Settings > Edit robots.txt
- You’ll see your current robots.txt content, which you can edit directly
If you’re not using Rank Math, you can also edit robots.txt via Yoast SEO (under Tools > File editor) or by accessing the file directly through your hosting file manager or FTP. The file lives at yourdomain.com/robots.txt.
Step 2: The Recommended robots.txt for Therapist Websites
Here’s the robots.txt configuration I recommend for most therapy websites. This blocks AI training bots while keeping your site visible in both traditional search and AI search results:
# Search engines – ALWAYS ALLOW
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# AI search/retrieval bots – ALLOW for visibility
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
# AI training bots – BLOCK
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: FacebookBot
Disallow: /
User-agent: Omgilibot
Disallow: /
# Default – allow everything else
User-agent: *
Allow: /
# Sitemap
Sitemap: https://yourdomain.com/sitemap_index.xml
Remember to replace yourdomain.com with your actual website URL in the Sitemap line.
Step 3: If You Want to Block All AI Bots
If you’d rather take the simpler approach and block all AI bots entirely, add these directives for each AI bot user-agent with Disallow: / for every one listed above, plus OAI-SearchBot, ChatGPT-User, and PerplexityBot. This will prevent your content from appearing in AI search answers entirely. You will still appear in regular Google and Bing results.
Step 4: Test Your robots.txt
After saving your changes, verify your robots.txt is working:
- Visit yourdomain.com/robots.txt in your browser to check it looks correct
- Use Google Search Console > Settings > robots.txt to check Google can read it
- Check that Googlebot and Bingbot are NOT blocked (this is critical for your SEO)
How to Block AI Bots Using Cloudflare
If your WordPress site is behind Cloudflare (and if it isn’t, it probably should be for speed and security), you have much more powerful options for managing AI crawlers. Unlike robots.txt, Cloudflare actually enforces blocks at the network level, meaning bots cannot ignore them.
Good to know: Since mid-2025, Cloudflare now blocks AI bots by default on newly created domains. If your site has been on Cloudflare for a while, your existing settings were preserved, so you may need to check and update them.
Option 1: The One-Click Block (All AI Bots)
If you want to block all AI crawlers with one click:
- Log into your Cloudflare dashboard
- Select your domain
- Go to Security > Bots
- Find the “Block AI bots” toggle
- Choose “Block on all pages” to block AI bots across your entire site
This blocks verified AI crawlers and unverified bots that behave like AI scrapers. It’s the simplest option but it’s all-or-nothing: you lose AI search visibility entirely.
Option 2: Granular Control with AI Crawl Control (Recommended)
For the balanced approach I recommended earlier (block training, allow search), use Cloudflare’s AI Crawl Control:
- Log into your Cloudflare dashboard
- Go to Security > Bots > AI Crawl Control
- Review the Crawlers tab to see which bots are visiting your site
- Set individual allow/block rules for each bot
With this tool, you can see exactly which AI crawlers are visiting your site, how often they make requests, and whether they’re respecting your robots.txt directives. You can then make crawler-by-crawler decisions.
Option 3: Managed robots.txt via Cloudflare
Cloudflare can also manage your robots.txt for you, automatically adding AI bot blocking directives:
- Go to Security > Bots in your Cloudflare dashboard
- Enable “Instruct AI bot traffic with robots.txt”
- Cloudflare will prepend its managed AI bot directives before your existing robots.txt content
This is useful if you’re not comfortable editing robots.txt manually, but it gives you less fine-grained control than doing it yourself.
Option 4: AI Labyrinth (for Misbehaving Bots)
Cloudflare also offers a clever feature called AI Labyrinth. Instead of simply blocking bad bots, it traps them in an endless maze of fake pages. Human visitors can’t see these pages, and compliant bots following robots.txt ignore them. But scrapers that ignore your rules get caught in a loop that wastes their resources. The fake pages use nofollow tags, so they won’t affect your SEO.
What I Recommend for Most Therapist Websites
Having set up and managed hundreds of therapy websites over the past decade, here’s my practical recommendation:
- Start with robots.txt. Add the recommended configuration from earlier in this guide to block AI training bots while allowing AI search bots. This takes five minutes and covers the basics.
- Use Cloudflare for enforcement. If you’re on Cloudflare (which I recommend for all therapy websites for speed and security), enable AI Crawl Control and set individual bot policies. This gives you real enforcement, not just polite requests.
- Keep Googlebot and Bingbot completely untouched. Never, ever block these. Your entire SEO strategy depends on search engines being able to crawl your site.
- Monitor and review regularly. The AI crawler landscape is changing fast. New bots appear regularly. Check your Cloudflare analytics or server logs every few months to see what’s visiting your site.
- Focus on what you can control. Whether you allow or block AI bots, the best thing you can do for your therapy practice online is create genuinely helpful, expert content that demonstrates your qualifications and experience. That’s what drives both SEO results and AI visibility.
Frequently Asked Questions
Will blocking AI bots affect my Google rankings?
No. AI bots like GPTBot, ClaudeBot, and CCBot are completely separate from Google’s search crawler (Googlebot). Blocking AI training bots has zero impact on your Google search rankings. Just make sure you never block Googlebot or Bingbot.
Can AI bots ignore my robots.txt file?
Technically, yes. robots.txt is a voluntary protocol. Reputable companies like OpenAI and Google respect it, but not all bots do. For actual enforcement, you need a tool like Cloudflare that blocks bots at the network level before they even reach your website.
Should I block all AI bots or just some of them?
For most therapist websites, I recommend a selective approach: block AI training bots (GPTBot, CCBot, ClaudeBot, Google-Extended) while allowing AI search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot). This protects your content from being used for training while keeping you visible in AI search results.
What happens if I do nothing?
If you don’t update your robots.txt or Cloudflare settings, AI bots will continue to crawl your site freely. Your content will be used for both AI model training and AI search answers. If you’re on Cloudflare and your domain was created recently, AI bots may already be blocked by default.
Is Cloudflare free to use?
Yes, Cloudflare has a free plan that includes the basic AI bot blocking toggle. The more granular AI Crawl Control features are available on paid plans, but the free tier gives you the one-click block and managed robots.txt options.
Do I need to worry about this if my website is small?
Even small therapy websites get crawled by AI bots. The decision to block or allow isn’t about the size of your site. It’s about whether you’re comfortable with AI companies using your content for training, and whether you want to appear in AI search results.
Will this affect how my site appears in Google’s AI Overviews?
Google’s AI Overviews use a bot called Google-Extended for AI-specific features. If you block Google-Extended, your content may not appear in AI Overviews, but your regular search listings will not be affected. Blocking Googlebot would affect everything, which is why you should never do that.
The Bottom Line
The AI crawling question doesn’t have a single right answer for every therapist. What matters is that you make an informed, active decision rather than leaving it to chance.
If visibility in AI search is important to you, allow the search bots and block the training bots. If you’d rather keep your content entirely out of AI systems, block everything. Either way, take the 10 minutes to update your robots.txt and check your Cloudflare settings. It’s one of those small tasks that gives you genuine control over how your content is used online.
And whatever you decide about AI bots, keep doing what matters most: creating genuinely helpful, expert content that serves your potential clients. That’s what drives results in both traditional SEO and the emerging world of AI search.
Need Help Setting This Up on Your Therapy Website?
If you’d rather not wrestle with robots.txt files and Cloudflare dashboards yourself, I can help. I set up and manage WordPress websites exclusively for therapists, counsellors, and mental health professionals, and AI bot management is now part of every website build and SEO package I offer.
Book a free 30-minute consultation and I’ll take a look at your current setup, tell you what’s crawling your site, and help you make the right decision for your practice.
Need Help With Your Hypnotherapy Website?
I specialise in websites for therapists and counsellors across the UK. Whether you need a complete website build or just some guidance, I’m happy to chat—no pressure, no sales pitch.
Get in touch: www.theoruby.com/contact
Email: [email protected] | Phone: 07709 852 364
Share this guide to help others grow their business online!
Bitesize Marketing Guides
Hand-picked training & resources to cover all areas of business including, digital marketing, SEO, website design, social media, networking & lead generation. Members can also get a free consultation and online marketing action plan.
Written by Theo Ruby, a digital marketing consultant with over 10 years of experience helping therapists and mental health professionals grow their practices online. Theo has worked with 300+ therapy professionals across the UK on website design, SEO, and digital marketing strategy.
Last updated: March 2026