Update robots.txt to blacklist AI training crawlers

Signed-off-by: Y. Meyer-Norwood <[email protected]>
jirastopwatch · Jan 5, 2025 · 7d38fe2 · 7d38fe2
1 parent ec714ae
commit 7d38fe2
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml
@@ -75,6 +75,10 @@ jobs:
       - name: "Generate a robots.txt file"
         run: |
           tee robots.txt << EOF
+          # Block AI Crawlers (see: https://github.com/ai-robots-txt)
+          $(gh release download --repo ai-robots-txt/ai.robots.txt --pattern 'robots.txt' --output -)
+
+          # List of pages and files
           Sitemap: ${BASE_URL}/sitemap.txt
           EOF