audit: BreadcrumbList, robots policy, og:title, sitemap priorities (#832)

slayerjain · nehagup · claude · web-flow · commit b1a65aeba77a · 2026-04-15T18:04:38.000+05:30
* fix(robots): nuanced AI bot policy — allow search, block training Adopt the Speedscale / Katalon / Testsigma split: - Allow AI SEARCH bots (drive answer visibility): OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Gemini-Deep-Research, GoogleOther, Applebot, DuckAssistBot, Amazonbot. - Block TRAINING-ONLY bots: GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, cohere-ai, Diffbot, Omgilibot, ImagesiftBot. - Keep Bytespider blocked. Also add belt-and-braces Disallow for unmaintained legacy doc versions (/docs/1.0.0/, /docs/2.0.0/, /docs/3.0.0/) to reinforce existing noindex+canonical signals for crawlers that ignore them. Reopens Task 52 per user direction 2026-04-14. Mirrors the corresponding policy change on landing and blog-website robots.txt. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Neha Gupta <gneha21@yahoo.in> * fix(docs): emit BreadcrumbList schema even when sidebar trail is null LIVE-20. Live audit of /docs/concepts/reference/glossary/idempotency/ on 2026-04-14 showed @type schema blocks Article, ImageObject, Organization, WebPage — and no BreadcrumbList. The /docs/ root, by contrast, has BreadcrumbList. This is the specific glossary-page regression that Task 13 was meant to catch but didn't. Root cause: the DocBreadcrumbs component had an early `return null` when useSidebarBreadcrumbs() returned null/undefined, which suppresses both the visual breadcrumb UI AND the JSON-LD schema emission. For deep glossary pages whose sidebar context resolves to null, this meant zero BreadcrumbList — the regression. Changes: - Replace `if (!breadcrumbs) return null` with a safe fallback to an empty sidebarTrail array. Schema emission + Home/Docs items run unconditionally. - Only render the visual <nav> when sidebarTrail has entries (avoids showing an empty breadcrumb UI on schema-only pages). - Propagate the sidebarTrail rename through the visual render path. Verify after deploy: curl -s https://keploy.io/docs/concepts/reference/glossary/idempotency/ | \ grep -c '"@type":"BreadcrumbList"' # expected: 1 (was 0) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Neha Gupta <gneha21@yahoo.in> * fix(robots): apply legacy-version disallows inside AI-search allow group Per the robots.txt spec, a bot that matches a named User-agent group reads rules only from that group — it does not fall through to User-agent: *. So the Disallow: /docs/{1,2,3}.0.0/ lines under User-agent: * were silently inapplicable to PerplexityBot/Applebot/OAI-SearchBot/etc., meaning those bots were still crawling the unmaintained legacy versions despite the noindex/canonical/global block combo. Consolidate the 11 AI search bot allow groups into a single block using multiple User-agent headers, and add the three legacy-version Disallow lines inside it so the intent — "allow AI search bots everywhere except legacy versions" — is actually enforced. No semantic change to training bots, Bytespider, or the * fallback group. Signed-off-by: nehagup <15074229+nehagup@users.noreply.github.com> * fix(docs): LIVE-12 per-page og:title, LIVE-13 hub Article suppression, Task 35 sitemap priorities Three docs-side fixes bundled. LIVE-12 — per-page og:title / twitter:title Previously every docs page rendered with og:title "Keploy Documentation" because the title inherited from docusaurus.config.js's site-level `title` field. Social share cards on LinkedIn / Slack / X therefore all showed the same generic headline regardless of which glossary / concept / quickstart page was shared. Fix: emit <meta property="og:title" content={title}> and <meta name="twitter:title" content={title}> in the swizzled DocItem Head component, pulling from the per-page metadata.title that the <title> tag already uses. Also adds og:description / twitter:description so preview cards carry the page-specific description. No site-level config change required. LIVE-13 — suppress Article schema on /docs/ landing and category indexes The /docs/ root was shipping Article JSON-LD even though it is a hub page with no single author, no single publication date, and no single headline — a type mismatch that AI models may flag as noise. Fix: compute `suppressArticleSchema` from permalink / frontmatter and short-circuit the articleSchema construction when the page is the /docs/ root or a category index. The DocBreadcrumbs JSON-LD continues to emit normally so hub pages still have navigation signal. Task 35 — differentiate docs sitemap priorities Original priority buckets only covered quickstart (0.8), concepts / keploy-explained (0.7), and keploy-cloud (0.6). Default was 0.5 for everything else including the high-value /docs/ root and running-keploy sections. New bucket structure in createSitemapItems: 1.0 /docs/ root (primary entry point) 0.9 /docs/quickstart/* (highest-intent user flow) 0.8 /docs/running-keploy/* (primary product docs) 0.7 /docs/concepts/*, /docs/keploy-explained/* 0.6 /docs/keploy-cloud/*, /docs/ci-cd/*, /docs/faq, /docs/troubleshooting 0.5 /docs/concepts/reference/glossary/* (long-tail, many pages) Added an explanatory comment inline so the next editor understands the bucket rationale. Verify after deploy: curl -s https://keploy.io/docs/concepts/reference/glossary/idempotency/ | \ grep -oE 'og:title"[^>]*content="[^"]+"' # expected: "What is Idempotency in REST APIs? Complete Guide" curl -s https://keploy.io/docs/ | grep -c '"@type":"Article"' # expected: 0 curl -s https://keploy.io/docs/sitemap.xml | \ python3 -c "import sys,re; \ priorities = re.findall(r'<priority>([0-9.]+)</priority>', sys.stdin.read()); \ print('unique priorities:', sorted(set(priorities)))" # expected: ['0.5', '0.6', '0.7', '0.8', '0.9', '1.0'] Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Neha Gupta <gneha21@yahoo.in> * fix(robots): mirror Crawl-delay + /cgi-bin disallow into AI search group Copilot review caught that named User-agent groups in robots.txt do not inherit rules from User-agent: *. The AI-search allow group (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Gemini-Deep-Research, GoogleOther, Applebot, DuckAssistBot, Amazonbot) was therefore ignoring both the global Crawl-delay: 5 limit AND the Disallow: /cgi-bin/ in the fallback User-agent: * block. Duplicated both lines into the named group so the same policy applies: search bots are rate-limited to 5s per request, and they cannot crawl /cgi-bin/. The legacy-version disallows (/docs/1.0.0/, /docs/2.0.0/, /docs/3.0.0/) were already duplicated in this block for the same inheritance reason. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Neha Gupta <gneha21@yahoo.in> * feat(docs): Task 33 — Keploy vs Alternatives comparison page Live audit + competitor analysis (llms-full.txt for competitors already has comparison tables, but docs had none). Adds a dedicated comparison page under /docs/keploy-explained/keploy-vs-alternatives with: - Feature comparison matrix: Keploy vs Postman, Katalon, WireMock, Testcontainers across 9 capabilities (test generation model, SDK requirement, mock generation, non-determinism, secret masking, CI/CD, license, kernel version). - Approach differences: plain-language description of each tool's core abstraction so readers can self-sort. - When to pick each: decision tree by team profile / API shape. - Migration paths: concrete steps for moving from Postman or Katalon to Keploy without throwing away existing work. - Related reading cross-links to how-keploy-works, integration-testing-faq, api-testing-faq. Added to version-4.0.0 sidebar in the Integration Testing → keploy-vs- alternatives slot, placed between Troubleshooting Guide and FAQs so it appears in the decision phase of the reader journey. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Neha Gupta <gneha21@yahoo.in> * fix(copilot-review): FAQ + troubleshooting sitemap priority must match actual v4 routes Addresses Copilot review comment 3080865321 on docs PR #832. The previous sitemap createSitemapItems callback checked `url.includes("/faq")` and `url.includes("/troubleshooting")` — neither of those substrings appears in the actual v4 docs URLs. The FAQ pages live at: /docs/keploy-explained/integration-testing-faq/ /docs/keploy-explained/api-testing-faq/ /docs/keploy-explained/unit-testing-faq/ and the troubleshooting guide lives at: /docs/keploy-explained/common-errors/ (sidebar label: "Troubleshooting Guide"). Because `/faq` never matched, all three FAQ pages and the troubleshooting guide fell through to the `/keploy-explained/` rule immediately below and got priority 0.7, not the intended 0.6. Fix: 1. Changed the match patterns to `-faq/`, `-faq`, and `/common-errors` so they match the real URL fragments. 2. Moved the FAQ/troubleshooting check ABOVE the `/keploy-explained/` check so it takes precedence when a page satisfies both. 3. Updated the header comment block to name the actual pages covered by the 0.6 bucket. Signed-off-by: Neha Gupta <gneha21@yahoo.in> * style: prettier format on audit-touched files Signed-off-by: nehagup <15074229+nehagup@users.noreply.github.com> * fix(review): drop keploy-vs-alternatives page + scrub internal ticket refs Addresses 3 review comments from @nehagup on PR #832. 1. Delete versioned_docs/version-4.0.0/keploy-explained/keploy-vs-alternatives.md and remove its sidebar entry in version-4.0.0-sidebars.json. The page was added by the audit batch but the team decided not to ship a public alternatives comparison inside the docs; keeping it here would publish product-marketing framing under the docs subtree, which belongs on the landing site instead if anywhere. 2. Strip internal ticket identifiers from comments across docusaurus.config.js, src/theme/DocBreadcrumbs/index.js, and src/theme/DocItem/index.js. Internal audit references like "Task 35", "LIVE-12", "LIVE-13", "LIVE-20" rot as the task tracker evolves and leak private process detail into public source. Kept the explanatory comments that describe *why* each piece of logic exists, just without the ticket numbers. DocItem/index.js suppressArticleSchema logic and DocBreadcrumbs sidebarTrail fallback are left as-is — they are functional fixes that prevent invalid Article schema on hub pages and missing BreadcrumbList schema on glossary pages. Reply to @nehagup's "why are we suppressing" question posted separately on the comment thread. Signed-off-by: Neha Gupta <gneha21@yahoo.in> * fix(copilot-review): suppress Article schema on versioned docs roots too Addresses Copilot comment 3081151415 on PR #832. The previous suppressArticleSchema check only matched /docs/ as the docs root, but this site serves versioned hub pages too — /docs/4.0.0/, /docs/3.0.0/, /docs/2.0.0/, /docs/1.0.0/ — via onlyIncludeVersions and includeCurrentVersion in docusaurus.config.js. Each versioned root is also an index of content with no single author/date/headline, so emitting Article/BlogPosting/APIReference schema on those pages had the same type-mismatch problem the base case fix was addressing. Added a regex check for /docs/<digit-starting-version>/ so any current or archived versioned root is caught by the same suppression path. Current content pages inside versioned trees (e.g. /docs/4.0.0/keploy-explained/how-keploy-works/) still emit Article schema as normal since they have real authors, dates, and headlines — only the bare versioned roots are suppressed. Signed-off-by: Neha Gupta <gneha21@yahoo.in> --------- Signed-off-by: Neha Gupta <gneha21@yahoo.in> Signed-off-by: nehagup <15074229+nehagup@users.noreply.github.com> Co-authored-by: Neha Gupta <gneha21@yahoo.in> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: nehagup <15074229+nehagup@users.noreply.github.com>
diff --git a/docusaurus.config.js b/docusaurus.config.js
@@ -449,20 +449,66 @@ module.exports = {
           changefreq: "weekly",
           priority: 0.5,
           filename: "sitemap.xml",
+          // Differentiate docs sitemap priorities by content type so
+          // search engines spend crawl budget proportional to how
+          // canonical each page is. Priority buckets:
+          //   1.0  → /docs/ root (highest — primary entry point)
+          //   0.9  → /docs/quickstart/* (highest-intent user flow)
+          //   0.8  → /docs/running-keploy/* (primary product docs)
+          //   0.7  → /docs/concepts/*, /docs/keploy-explained/*
+          //   0.6  → /docs/keploy-cloud/*, /docs/ci-cd/*
+          //   0.6  → /docs/keploy-explained/*-faq/ (3 FAQ pages) and
+          //          /docs/keploy-explained/common-errors/ (troubleshooting)
+          //          — reference-style, lower crawl priority than core docs
+          //   0.5  → /docs/concepts/reference/glossary/* (long-tail
+          //          glossary; noindexed legacy versions excluded via
+          //          netlify headers + robots.txt)
           createSitemapItems: async (params) => {
             const {defaultCreateSitemapItems, ...rest} = params;
             const items = await defaultCreateSitemapItems(rest);
             return items.map((item) => {
-              if (item.url.includes("/quickstart/")) {
+              const url = item.url;
+              // The /docs/ home page is the highest-priority entry point
+              // for the whole docs subtree.
+              if (url.endsWith("/docs/") || url.endsWith("/docs")) {
+                return {...item, priority: 1.0, changefreq: "weekly"};
+              }
+              if (url.includes("/quickstart/")) {
+                return {...item, priority: 0.9, changefreq: "weekly"};
+              }
+              if (url.includes("/running-keploy/")) {
                 return {...item, priority: 0.8, changefreq: "weekly"};
               }
+              if (url.includes("/concepts/reference/glossary/")) {
+                // Glossary entries are numerous, long-tail, and often
+                // off-topic for core product queries. Keep them in the
+                // sitemap but mark them low priority.
+                return {...item, priority: 0.5, changefreq: "monthly"};
+              }
+              // FAQ + troubleshooting match FIRST, because these pages live
+              // under /keploy-explained/ in the v4 docs (e.g.
+              // /docs/keploy-explained/integration-testing-faq/,
+              // /docs/keploy-explained/api-testing-faq/,
+              // /docs/keploy-explained/unit-testing-faq/,
+              // /docs/keploy-explained/common-errors/ — "common-errors" is
+              // the troubleshooting guide, labelled "Troubleshooting Guide"
+              // in the sidebar). Without matching first, they would be
+              // captured by the /keploy-explained/ rule below and get
+              // priority 0.7 instead of the intended 0.6.
+              if (
+                url.includes("-faq/") ||
+                url.includes("-faq") ||
+                url.includes("/common-errors")
+              ) {
+                return {...item, priority: 0.6, changefreq: "monthly"};
+              }
               if (
-                item.url.includes("/concepts/") ||
-                item.url.includes("/keploy-explained/")
+                url.includes("/concepts/") ||
+                url.includes("/keploy-explained/")
               ) {
                 return {...item, priority: 0.7, changefreq: "weekly"};
               }
-              if (item.url.includes("/keploy-cloud/")) {
+              if (url.includes("/keploy-cloud/") || url.includes("/ci-cd/")) {
                 return {...item, priority: 0.6, changefreq: "monthly"};
               }
               return item;
diff --git a/src/theme/DocBreadcrumbs/index.js b/src/theme/DocBreadcrumbs/index.js
@@ -51,9 +51,12 @@ export default function DocBreadcrumbs() {
   const {siteConfig} = useDocusaurusContext();
   const {pathname} = useLocation();
 
-  if (!breadcrumbs) {
-    return null;
-  }
+  // Previously this component early-returned when useSidebarBreadcrumbs()
+  // returned null/undefined, which caused glossary and reference pages
+  // not in the sidebar config to ship with zero BreadcrumbList schema.
+  // Treat null/undefined as "no sidebar trail, emit Home + Docs schema
+  // anyway" so AI crawlers always get a hierarchy signal.
+  const sidebarTrail = Array.isArray(breadcrumbs) ? breadcrumbs : [];
 
   const toAbsoluteUrl = (baseUrl, url) => {
     if (!url) {
@@ -89,9 +92,9 @@ export default function DocBreadcrumbs() {
     }
   }
 
-  if (breadcrumbs.length > 0) {
-    breadcrumbs.forEach((crumb, index) => {
-      const isLast = index === breadcrumbs.length - 1;
+  if (sidebarTrail.length > 0) {
+    sidebarTrail.forEach((crumb, index) => {
+      const isLast = index === sidebarTrail.length - 1;
       const href =
         crumb.type === "category" && crumb.linkUnlisted
           ? undefined
@@ -130,35 +133,37 @@ export default function DocBreadcrumbs() {
           </script>
         </Head>
       )}
-      <nav
-        className={clsx(
-          ThemeClassNames.docs.docBreadcrumbs,
-          styles.breadcrumbsContainer
-        )}
-        aria-label={translate({
-          id: "theme.docs.breadcrumbs.navAriaLabel",
-          message: "Breadcrumbs",
-          description: "The ARIA label for the breadcrumbs",
-        })}
-      >
-        <ul className="breadcrumbs">
-          {homePageRoute && <HomeBreadcrumbItem />}
-          {breadcrumbs.map((item, idx) => {
-            const isLast = idx === breadcrumbs.length - 1;
-            const href =
-              item.type === "category" && item.linkUnlisted
-                ? undefined
-                : item.href;
-            return (
-              <BreadcrumbsItem key={idx} active={isLast}>
-                <BreadcrumbsItemLink href={href} isLast={isLast}>
-                  {item.label}
-                </BreadcrumbsItemLink>
-              </BreadcrumbsItem>
-            );
+      {sidebarTrail.length > 0 && (
+        <nav
+          className={clsx(
+            ThemeClassNames.docs.docBreadcrumbs,
+            styles.breadcrumbsContainer
+          )}
+          aria-label={translate({
+            id: "theme.docs.breadcrumbs.navAriaLabel",
+            message: "Breadcrumbs",
+            description: "The ARIA label for the breadcrumbs",
           })}
-        </ul>
-      </nav>
+        >
+          <ul className="breadcrumbs">
+            {homePageRoute && <HomeBreadcrumbItem />}
+            {sidebarTrail.map((item, idx) => {
+              const isLast = idx === sidebarTrail.length - 1;
+              const href =
+                item.type === "category" && item.linkUnlisted
+                  ? undefined
+                  : item.href;
+              return (
+                <BreadcrumbsItem key={idx} active={isLast}>
+                  <BreadcrumbsItemLink href={href} isLast={isLast}>
+                    {item.label}
+                  </BreadcrumbsItemLink>
+                </BreadcrumbsItem>
+              );
+            })}
+          </ul>
+        </nav>
+      )}
     </>
   );
 }
diff --git a/src/theme/DocItem/index.js b/src/theme/DocItem/index.js
@@ -143,12 +143,36 @@ export default function DocItem(props) {
   const currentYear = new Date().getFullYear();
   const image = assets?.image ?? frontMatter?.image;
   const imageWithBaseUrl = useBaseUrl(image || "");
-  const socialImage = image ? toAbsoluteUrl(siteConfig?.url, imageWithBaseUrl) : null;
+  const socialImage = image
+    ? toAbsoluteUrl(siteConfig?.url, imageWithBaseUrl)
+    : null;
   const normalizedMetaKeywords = Array.isArray(metaKeywords)
     ? metaKeywords.join(", ")
     : metaKeywords;
+  // Suppress Article / BlogPosting / APIReference schema on the /docs/
+  // root, versioned docs roots like /docs/4.0.0/, and any category
+  // index pages. Article schema on a hub page is a type mismatch
+  // because a hub does not have a single author, a single publication
+  // date, or a single headline — it is an index of content. Hub pages
+  // emit only the normal DocBreadcrumbs JSON-LD.
+  const permalink = metadata?.permalink || "";
+  // Versioned root pattern: /docs/<version>/ or /docs/<version> where
+  // <version> starts with a digit. Covers current and archived
+  // versions listed in docusaurus.config.js onlyIncludeVersions.
+  const isVersionedDocsRoot =
+    /^\/docs\/\d[\w.-]*(?:\/index)?\/?$/.test(permalink);
+  const isDocsRoot =
+    permalink === "/docs/" ||
+    permalink === "/docs" ||
+    permalink.endsWith("/docs/index") ||
+    permalink.endsWith("/docs/") ||
+    isVersionedDocsRoot;
+  const isCategoryIndex =
+    frontMatter?.slug === "index" || /\/category\/|\/index\/?$/.test(permalink);
+  const suppressArticleSchema = isDocsRoot || isCategoryIndex;
+
   const articleSchema =
-    pageUrl && title
+    pageUrl && title && !suppressArticleSchema
       ? {
           "@context": "https://schema.org",
           "@type": schemaType,
@@ -187,6 +211,19 @@ export default function DocItem(props) {
         {normalizedMetaKeywords && (
           <meta name="keywords" content={normalizedMetaKeywords} />
         )}
+        {/* Per-page og:title and og:description override the
+            docusaurus.config.js site-level defaults, which would
+            otherwise emit the same og:title on every docs page
+            regardless of content. Social card previews now reflect
+            the actual page title. */}
+        <meta property="og:title" content={title} />
+        {description && (
+          <meta property="og:description" content={description} />
+        )}
+        <meta name="twitter:title" content={title} />
+        {description && (
+          <meta name="twitter:description" content={description} />
+        )}
         {socialImage && <meta property="og:image" content={socialImage} />}
         {socialImage && <meta name="twitter:image" content={socialImage} />}
         {socialImage && (
@@ -288,7 +325,10 @@ export default function DocItem(props) {
                   href="https://join.slack.com/t/keploy/shared_invite/zt-357qqm9b5-PbZRVu3Yt2rJIa6ofrwWNg"
                   aria-label="Slack"
                 >
-                  <span className="docs-inline-footer__slack" aria-hidden="true" />
+                  <span
+                    className="docs-inline-footer__slack"
+                    aria-hidden="true"
+                  />
                 </a>
               </div>
               <div className="docs-inline-footer__usecase">
diff --git a/static/robots.txt b/static/robots.txt
@@ -1,12 +1,98 @@
-# Block specific bot
+# Keploy docs robots.txt
+# Policy: allow AI search/answer engines, block training-only crawlers,
+# block Bytespider. Search bots drive visibility in ChatGPT, Claude,
+# Perplexity, Copilot, Gemini answers. Training bots feed future model
+# weights and provide nothing back.
+# Reference: Speedscale / Katalon / Testsigma split policy (2026 competitor audit)
+
+# =============================================================================
+# ALLOW — AI search / answer engines
+# Legacy-version disallows are repeated inside this group because a bot that
+# matches a named User-agent group only reads rules from THAT group; it does
+# not fall through to `User-agent: *`. Without these lines, Perplexity/
+# Applebot/OAI-SearchBot/etc. would still crawl /docs/{1,2,3}.0.0/ despite
+# the global block further below.
+# =============================================================================
+
+User-agent: OAI-SearchBot
+User-agent: ChatGPT-User
+User-agent: Claude-SearchBot
+User-agent: Claude-User
+User-agent: PerplexityBot
+User-agent: Perplexity-User
+User-agent: Gemini-Deep-Research
+User-agent: GoogleOther
+User-agent: Applebot
+User-agent: DuckAssistBot
+User-agent: Amazonbot
+Allow: /
+Crawl-delay: 5
+Disallow: /cgi-bin/
+Disallow: /docs/1.0.0/
+Disallow: /docs/2.0.0/
+Disallow: /docs/3.0.0/
+
+# =============================================================================
+# DISALLOW — Training-only crawlers
+# =============================================================================
+
+User-agent: GPTBot
+Disallow: /
+
+User-agent: ClaudeBot
+Disallow: /
+
+User-agent: anthropic-ai
+Disallow: /
+
+User-agent: CCBot
+Disallow: /
+
+User-agent: Google-Extended
+Disallow: /
+
+User-agent: Applebot-Extended
+Disallow: /
+
+User-agent: Meta-ExternalAgent
+Disallow: /
+
+User-agent: FacebookBot
+Disallow: /
+
+User-agent: cohere-ai
+Disallow: /
+
+User-agent: Diffbot
+Disallow: /
+
+User-agent: Omgilibot
+Disallow: /
+
+User-agent: ImagesiftBot
+Disallow: /
+
+# Always-block scraper
 User-agent: Bytespider
 Disallow: /
 
-# Default rules — apply to all crawlers including AI bots
+# =============================================================================
+# DEFAULT — Googlebot, Bingbot, and all other crawlers
+# =============================================================================
+
 User-agent: *
 Allow: /
 Crawl-delay: 5
 Disallow: /cgi-bin/
 
+# Block unmaintained legacy doc versions (already set via noindex + canonical,
+# belt-and-braces for crawlers that ignore those signals).
+Disallow: /docs/1.0.0/
+Disallow: /docs/2.0.0/
+Disallow: /docs/3.0.0/
+
+# =============================================================================
 # Sitemap
+# =============================================================================
+
 Sitemap: https://keploy.io/docs/sitemap.xml
