Protecting your data from scraping

Your public club site is meant to be read. Search engines index it, members share it, and people who have never heard of your club find you through it — that is the whole point of having one. Scrape protection is about the narrower problem underneath: somebody copying the parts of your site that are about people rather than about the club, usually to sell to you or to someone else. You'll find these controls in the Protecting your data card on the Settings page, under System in the admin sidebar.

Read this first, because it shapes everything below. A determined scraper running a real browser from ordinary home internet addresses cannot be stopped — only made expensive and visible. Nothing on this page is a lock. The one control here that a determined copier genuinely cannot ignore is the one that takes the information out of an anonymous request in the first place, and that is deliberately where the default sits.

Who can use this

Admins only. Moderators can open Settings for other things, but the Protecting your data card and its activity list are admin-only — both the settings and what we've noticed are readable only by an admin at your organisation, so a moderator won't see them.

There's no module to switch on and no plan requirement. Every organisation has these settings, and every organisation starts on the private default whether or not anyone has ever opened the page.

What's on the page

• Show team rosters to visitors who are not signed in — off by default. It controls whether the squad list on a team page is visible to someone who hasn't signed in.
• AI crawlers — allow them (the default), or ask them to stay away. There's a link to your organisation's robots.txt so you can see exactly what your choice records.
• Alert me above this many requests a day — how many page requests one source can make in a day before we look at the shape of its traffic and, if it looks like an automated copy, tell your admins.
• Unusual activity we have noticed — what our nightly sweep found, with the date, what tripped it and how many pages that source walked. You can mark one as dealt with.

Your member directory is a separate setting with its own per-member consent, and it lives under Members → Directory. It's off by default too, and no member appears in it without opting in themselves — see Members.

Step by step

Decide what an anonymous visitor may see

1. Open Settings and find Protecting your data.
2. Leave Show team rosters to visitors who are not signed in off unless you actively want your squad lists public. This is the strongest thing on the page: with it off, the names are never in the response an anonymous visitor receives, so there is nothing for a copier to take, however clever it is. Signed-in visitors still see the roster.
3. Turn it on only if a public squad list is genuinely something your club wants — some clubs do, and that's a fair choice to make deliberately.
4. Save. The change applies on the next page load; there's no cache to wait out. If our own check ever fails to run, the page falls back to hiding the roster rather than showing it.

Choose your AI-crawler policy

1. Allow AI crawlers (recommended) lets them read your site the way a search engine does. People increasingly find clubs by asking an assistant rather than typing into a search box, so this is a real way for new members to reach you.
2. Ask AI crawlers to stay away records a request aimed at crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Read the FAQ before choosing it — it has a cost, and it is a request rather than an instruction.
3. Where your choice actually goes. A crawler only ever looks for robots.txt at the root of the web address it is visiting, so that is where we publish it. On your own custom domain the whole site is yours, and asking AI crawlers away is a blanket request across it. On a clubhelix.com address the file at the root is shared by every organisation, so your choice appears there as your own lines within it. The link beside the setting is a preview of what your organisation's part says — it isn't a file any crawler fetches.
4. Whichever you pick, Google and Bing are untouched. Those are separate crawlers and we never restrict them. Your site's search visibility is governed by Allow search engines to index this site, higher up the same page.

Set your alert level and read what we notice

1. Alert me above this many requests a day counts requests from a single source. The default suits most club sites. A busy site during finals week will legitimately produce big numbers — raise it rather than train yourself to ignore the alerts.
2. Each night we look over the previous day's traffic for harvest-shaped patterns: a lot of requests, a lot of different pages, pages walked in order, no referring site, no sign of a real browser finishing the page.
3. When something trips, it appears under Unusual activity we have noticed and your admins get a notification. It says what tripped, how many requests and how many distinct pages — never an IP address, and never a visitor's identity.
4. Read it and decide. Most of them need nothing: a badly written but harmless bot, or a genuinely busy day. A repeated, deliberate walk of your team pages is the point at which turning the roster off is worth what it costs.
5. Mark it dealt with to clear it from the list.

Field reference

Tips & FAQ

• Can this stop someone copying our site? No, and anyone who tells you otherwise is selling something. A person running a real browser from an ordinary home connection looks exactly like a member reading your site, because that is what they are doing — just faster and more of it. What these controls do is make bulk copying slow and conspicuous, and take the genuinely sensitive part out of reach. That is worth having. It is not a lock.
• What actually works, then? Not publishing the information. Leaving team rosters signed-in-only, and leaving your member directory off, are the controls a determined copier cannot ignore, because the names are never in the response. Everything else here is friction or detection.
• Is robots.txt enforced? No. It is a file that asks crawlers to behave, and well-run crawlers do — search engines honour it, and so do we when we read your old site during a migration. A crawler that ignores it faces nothing. So asking AI crawlers to stay away stops the ones that follow the rules and does nothing at all to the ones that don't. Several of these crawlers have a patchy record.
• What does asking AI crawlers away cost us? Visibility, in a channel that is growing. When someone asks an assistant "which walking football clubs are near Adelaide", the answer is drawn from sites those crawlers were allowed to read. Ask them away, and you are not in that answer. Clubs whose pages are mostly public information — fixtures, training times, how to join — usually gain more from being findable than they lose from being read. Clubs whose pages carry a lot about identifiable people may weigh it differently. It is a genuine trade-off, which is why we don't decide it for you.
• Does any of this affect our Google ranking? The AI-crawler setting does not — search crawlers are never included in it, and are never rate-limited by us for any reason. Hiding team rosters removes those names from the public page, so they leave the index like any other content you stop publishing. Your ranking for everything else is untouched.
• "Activity we have noticed" — why the careful wording? Because that is all it can honestly claim. Traffic we failed to recognise leaves no record, so an empty list means we saw nothing, not that nothing happened. We'd rather tell you that than let you read a quiet week as proof of safety.
• Will a normal member set this off? They shouldn't. The sweep is deliberately biased towards calling traffic human, because an alert that cries wolf is an alert you stop reading. It looks for the combination — volume plus breadth plus no referring site plus no sign of a real browser — rather than volume alone. A clubroom, a school or a stadium where forty people share one internet connection is exactly the case that biases it that way. If you do get a false alarm, raise the alert level and tell us.
• We limit how fast anyone can request pages, too. Every public page request passes a ceiling a person browsing will never touch and a fast bulk copier will. A visitor claiming to be a search engine isn't taken at its word — we look up the address the request came from and confirm it really belongs to that search engine, so borrowing Googlebot's name buys nothing. Genuine search crawlers are exempt, always, and if we can't complete that lookup we give the benefit of the doubt rather than risk turning a real search engine away. What the ceiling cannot do is slow somebody patient: a copier walking your site at reading speed from a handful of home connections stays under it, which is why the roster switch matters more than the limit does.
• Where do the alerts go? To the list on the Settings page, and as a notification to your organisation's admins. They follow your notification preferences like any other non-urgent notice, so if nobody is seeing them, check that first.
• What about email addresses on our pages? Anything you publish, anyone can read — a committee email address printed on a public page is collected within days. There's no switch for that here yet, so treat it as a decision you're making when you type the address in. A contact form is the safer way to be reachable.
• Someone is copying us anyway. What now? Turn the roster switch off and check what else you're publishing about people, so there is less to take. Then tell us — repeated, deliberate harvesting of one club is something we want to see, and our terms of service prohibit automated bulk access, which gives you something to point at with a business that ought to know better. It will not deter anyone acting in bad faith.

Related: Settings for site visibility and your web address, Members for the member directory and its per-member consent, Teams for the squad lists this affects, and Legal & policies for your published terms.