User-agent: * Sitemap: https://www.computercablestore.com/sitemap.xml Disallow: /bin/ Disallow: /content/files/ Disallow: /content/files/exportimport/ Disallow: /country/getstatesbycountryid Disallow: /install Disallow: /setproductreviewhelpfulness Disallow: /addproducttocart/catalog/ Disallow: /addproducttocart/details/ Disallow: /backinstocksubscriptions/manage Disallow: /boards/forumsubscriptions Disallow: /boards/forumwatch Disallow: /boards/postedit Disallow: /boards/postdelete Disallow: /boards/postcreate Disallow: /boards/topicedit Disallow: /boards/topicdelete Disallow: /boards/topiccreate Disallow: /boards/topicmove Disallow: /boards/topicwatch Disallow: /cart Disallow: /checkout Disallow: /checkout/billingaddress Disallow: /checkout/completed Disallow: /checkout/confirm Disallow: /checkout/shippingaddress Disallow: /checkout/shippingmethod Disallow: /checkout/paymentinfo Disallow: /checkout/paymentmethod Disallow: /clearcomparelist Disallow: /compareproducts Disallow: /compareproducts/add/* Disallow: /customer/avatar Disallow: /customer/activation Disallow: /customer/addresses Disallow: /customer/changepassword Disallow: /customer/checkusernameavailability Disallow: /customer/downloadableproducts Disallow: /customer/info Disallow: /deletepm Disallow: /emailwishlist Disallow: /inboxupdate Disallow: /newsletter/subscriptionactivation Disallow: /onepagecheckout Disallow: /order/history Disallow: /orderdetails Disallow: /passwordrecovery/confirm Disallow: /poll/vote Disallow: /privatemessages Disallow: /returnrequest Disallow: /returnrequest/history Disallow: /rewardpoints/history Disallow: /sendpm Disallow: /sentupdate Disallow: /shoppingcart/* Disallow: /subscribenewsletter Disallow: /topic/authenticate Disallow: /viewpm Disallow: /uploadfileproductattribute Disallow: /uploadfilecheckoutattribute Disallow: /wishlist # --------------------------------------------------------------------------- # IMPORTANT - read before editing # # robots.txt groups do NOT inherit from one another. A crawler obeys only the # most specific User-agent group that matches it, and ignores every other group # including "User-agent: *". # # So a named crawler that has its own group ONLY to set Crawl-delay is silently # exempted from every Disallow rule below it. That is not hypothetical: on # 2026-09-21 Baiduspider crawled the faceted filter URLs on computercablestore.com # at ~17 requests/second, ~30 seconds each, and pinned 4.5 CPU cores - while this # file appeared to disallow exactly those URLs. # # The Disallow block is therefore REPEATED in full inside every group. The # repetition is deliberate. Do not "tidy" it by moving the rules to a single # group - that reintroduces the bug. # # Agents are grouped by the crawl-delay they share, which is the only reason # they need a named group at all. # --------------------------------------------------------------------------- # --- Everyone not named below (includes Googlebot and Bingbot) ------------- User-agent: * Disallow: /*?*specs= Disallow: /*&specs= Disallow: /*?*price= Disallow: /*?*manufacturerid= Disallow: /*?*orderby= Disallow: /*?*viewmode= Disallow: /*?*pagesize= Disallow: /productreviews/ Disallow: /review/ Disallow: /search Disallow: /producttag/ # --- Crawl-delay 15 -------------------------------------------------------- User-agent: Baiduspider Crawl-delay: 15 Disallow: /*?*specs= Disallow: /*&specs= Disallow: /*?*price= Disallow: /*?*manufacturerid= Disallow: /*?*orderby= Disallow: /*?*viewmode= Disallow: /*?*pagesize= Disallow: /productreviews/ Disallow: /review/ Disallow: /search Disallow: /producttag/ # --- Crawl-delay 20 -------------------------------------------------------- User-agent: AhrefsBot User-agent: SemrushBot Crawl-delay: 20 Disallow: /*?*specs= Disallow: /*&specs= Disallow: /*?*price= Disallow: /*?*manufacturerid= Disallow: /*?*orderby= Disallow: /*?*viewmode= Disallow: /*?*pagesize= Disallow: /productreviews/ Disallow: /review/ Disallow: /search Disallow: /producttag/ # --- Crawl-delay 30 -------------------------------------------------------- User-agent: DotBot User-agent: Amazonbot User-agent: meta-externalagent User-agent: Bytespider User-agent: YandexBot User-agent: AwarioRssBot User-agent: AwarioSmartBot Crawl-delay: 30 Disallow: /*?*specs= Disallow: /*&specs= Disallow: /*?*price= Disallow: /*?*manufacturerid= Disallow: /*?*orderby= Disallow: /*?*viewmode= Disallow: /*?*pagesize= Disallow: /productreviews/ Disallow: /review/ Disallow: /search Disallow: /producttag/ # --- Crawl-delay 60 (AI / LLM crawlers) ------------------------------------ User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot Crawl-delay: 60 Disallow: /*?*specs= Disallow: /*&specs= Disallow: /*?*price= Disallow: /*?*manufacturerid= Disallow: /*?*orderby= Disallow: /*?*viewmode= Disallow: /*?*pagesize= Disallow: /productreviews/ Disallow: /review/ Disallow: /search Disallow: /producttag/ # Content sitemap (Misc.Cms plugin) - this line is maintained PER LIVE SITE with that # site's own host (robots.txt Sitemap directives must be absolute URLs), e.g.: # Sitemap: https://www.securecableties.com/content/sitemap.xml # It is intentionally commented here so this shared repo copy does not emit a wrong-host # line; add/uncomment it in each site's deployed robots.additions.txt.