sitemap index / robots.txt parsers

[ghost]

Hi,

Hope you are all well ! And merry Christmas first of all !

I was playing to today with site-audit-seo and I was missing some features like a robots.txt parser to find available sitemaps for a website and the related sitemap extractor (handling sitemap indexes also).

Do you think it is possible to add these 2 components easily ?

Please fin below some references that I found:

• https://github.com/seantomburke/sitemapper
• https://github.com/evanderkoogh/node-sitemap-stream-parser#readme
• https://github.com/b4dnewz/robots-parse
• https://github.com/ChristopherAkroyd/robots-txt-parser

Thanks for any insights or inputs on that.

Ps. Do you have a telegram account as I have some questions for you and do not want to pollute this thread ?
My handle is "deepocrates"

Cheers,
Luc Michalski

[popstas]

site-audit-seo based on https://github.com/yujiosaka/headless-chrome-crawler that using https://www.npmjs.com/package/robots-parser

In our cases this parser working correct for robots.txt.