User-agent: * Allow: / Disallow: /admin Disallow: /planner # Matchup Lab deep links are a combinatorial space: every head-to-head row on # every athlete page renders a "Compare" link, which is 261,053 reachable # /matchup/-vs- URLs against ~10,300 real content URLs in the sitemap. # None of them are declared, so Googlebot was spending its crawl budget on a # 25:1 majority of pages we never asked it to index -- and the sitemap's own # athlete pages sat in "Discovered - currently not indexed" behind them. # The pattern requires "-vs-", so /matchup itself (the pair-picker) stays # crawlable. Human visitors are unaffected; the links still work. Disallow: /matchup/*-vs-* # /embed/athletes/ is a framable rendering of a profile that is ALREADY in # the sitemap -- ~9,100 duplicate URLs carrying no fact the canonical page lacks. # Each one serves noindex + rel=canonical, but Googlebot has to spend a crawl to # READ that noindex, which is the same crawl-budget problem the matchup rule # above exists to solve (8,215 athlete pages sat "Discovered - currently not # indexed" behind exactly that kind of majority). Disallowing costs nothing here: # nothing links to these but the copy-paste snippet on the profile itself, and # they hold no content of their own to lose. Framing is unaffected -- robots.txt # governs crawlers, not browsers. Disallow: /embed/ # The three parameterised families below were 699 URLs in "Alternate page with # proper canonical tag" on 2026-08-17, and that bucket is not a defect: they DO # self-canonicalise to their bare path, correctly. Canonicalising correctly and # wasting crawl budget are separate questions, and the 2026-08-08 read only # answered the first -- so the leak survived the matchup fix and put the ratio of # undeclared to real URLs back above 1:1 (~13,000 vs ~10,300). None of the three # is in the sitemap, and none renders a fact its bare path doesn't already carry, # which is the test each had to pass before being blocked. # reportLink() (functions/_shared.js) renders a crawlable "Report an error" link # on EVERY athlete and EVERY contest page: 9,128 + 1,177 = ~10,305 URLs, 1:1 with # the entire real content set, all canonicalising to /report. 584 of the 699 above # (84%) were these, and only 584 of the 10,305 had been crawled -- so the bucket # was still growing toward five figures while 7,630 real athlete pages sat in # "Discovered - currently not indexed", never fetched. # The trailing "?" is load-bearing: bare /report is the actual form page and stays # crawlable, exactly as /matchup survives the -vs- rule above. Reporting is # unaffected -- every link still works, and /report remains the route for a # visitor to send us a correction, which the project requires. Disallow: /report? # ?division= on a contest page selects which division shows an "In focus" chip. # It reveals NOTHING: renderContestPage maps over every division regardless, so # the bare /contests/ already carries every division's full field. ~2,601 # contest-division links means ~2,601 of these against 1,177 real contest URLs. # Same shape as the old GROUP_VISIBLE bug -- a variant that changed a page's # appearance and nothing a crawler can see. Disallow: /contests/*?division= # ?cell=- opens a drill-down panel on the /scorecards coverage # matrix, ~154 cells. The panel only LISTS shows, and every contest it lists is # already in the sitemap at its own /contests/, so blocking this costs no # reachability. Deliberately narrow: /scorecards?contest=&division= is a real # server-rendered card and one of ~300 genuine content URLs, so it must stay # crawlable, and this pattern cannot match it. Disallow: /scorecards?cell= # CSV exports, blocked 2026-09-26. This is the SAME leak the /report? rule above # documents, missed at the time: renderCiteTools() (functions/_shared.js) emits a # real crawlable anchor -- -- inside the # "Cite / download" panel on EVERY athlete page and every contest page that has # results. That is 39,925 + 1,404 = ~41,300 URLs against ~10,300 real content URLs, # a 4:1 majority, none of them in the sitemap. Each one is a Pages Function run # plus Supabase queries that return, in CSV form, exactly the table the canonical # page already rendered -- no fact a crawler cannot already read at /athletes/ # or /contests/. # Downloading is UNAFFECTED. robots.txt governs crawlers, not people: the button # still works, the file still generates, and /api/v1/* (the other machine-readable # route) is untouched and stays open. This only stops crawlers spending fetches on # a second copy of pages they have already read. Disallow: /athletes/*/career.csv Disallow: /contests/*/results.csv # Presentation view, blocked 2026-09-26. Same shape as /embed/ above, but weaker # on its own behalf: every contest page with a readable card renders an "Open # presentation view" link (the present-cta block in functions/contests/[slug].js), # giving 2,980 reachable /present// URLs -- one per # contest_division holding two or more scored placings. It is a full-screen, # large-type re-rendering of the scorecard already shown on /contests/, built # for screen-sharing and screenshots. # Unlike /embed/, which serves noindex + rel=canonical via _shared.embedDoc and was # STILL disallowed because reading that noindex costs a crawl, /present/ declares # NEITHER -- so to a crawler it looks like original content rather than a duplicate. # That makes it the stronger candidate of the two, not the weaker one. # Human use is unaffected: the link still opens, the view still works. Disallow: /present/ # SEO-tool crawlers, blocked outright 2026-09-21. They index the site for # third-party backlink/keyword products and send no readers back, but every page # here is server-rendered with ~0% edge cache, so each hit is a Pages Function # run plus Supabase queries. On 2026-09-20 AhrefsBot (9,761), SemrushBot (4,332) # and DotBot (829) were ~40% of all requests, against ~230 real browser page # loads. Search engines and AI assistants (Googlebot, bingbot, GPTBot, etc.) are # NOT in this list -- they do send readers, and they stay under the * rules above. # A bot named here ignores the * group entirely, which is fine: it's a full block. User-agent: AhrefsBot User-agent: AhrefsSiteAudit User-agent: SemrushBot User-agent: SiteAuditBot User-agent: DotBot User-agent: rogerbot User-agent: MJ12bot User-agent: BLEXBot User-agent: DataForSeoBot User-agent: serpstatbot Disallow: / Sitemap: https://calloutiq.com/sitemap.xml