New features & updated documentation (#78)

* New features & updated documentation

* Ability to report on sitemap crawl errors in returned results. Added a new "errors" property in the `SitesData` object

* Added an option to set a concurrency limit to rate limit sitemap crawling. Useful when crawling sitemaps with multiple children to avoid getting blocked by firewalls. #77

* Added an option to have retry requests upon failure and to set the number of maximum retries per crawl.

* Updated documentation to include all the new features described above.

Co-Authored-By: Panagiotis Tzamtzis <[email protected]>
Co-Authored-By: PanagiotisTzamtzis <[email protected]>

* Fix for error on the main sitemap

In this case the errors object in the results was not an ErrorsDataArray but a single ErrorsData

* Bug fixes

* Error logging improvements with more details for `UnknownStateErrors` & errors when parsing the parent sitemap

* Retries option was not working when `debug` was set to false

* Bug fix

* Console.log statement was getting triggered when `debug` option was set to false

* Update src/examples/index.js

* 3.2.0

* Cleaning up, changing error to errors, updating Typescript, removing returnErrors option

* Removing returnErrors option

* quotes fix

* Updates

* Fixing errors array

* updating tests

Co-authored-by: PanagiotisTzamtzis <[email protected]>
Co-authored-by: Sean Thomas Burke <[email protected]>
Co-authored-by: Sean Thomas Burke <[email protected]>

README.md

@@ -68,7 +68,6 @@ You can add options on the initial Sitemapper object when instantiating it.
 + `debug`: (Boolean) - Enables/Disables debug console logging. Default: False
 + `concurrency`: (Number) - Sets the maximum number of concurrent sitemap crawling threads. Default: 10
 + `retries`: (Number) - Sets the maximum number of retries to attempt in case of an error response (e.g. 404 or Timeout). Default: 0
-+ `requestHeaders`: (Object) - Additional Request Headers
 + `insecure`: (Boolean) - Ignore invalid certificates
 
 ```javascript

src/assets/sitemapper.js

@@ -41,6 +41,8 @@ export default class Sitemapper {
     this.concurrency = settings.concurrency || 10;
     this.retries = settings.retries || 0;
     this.insecure = settings.insecure || false;
+    this.concurrency = settings.concurrency || 10;
+    this.retries = settings.retries || 0;
   }
 
   /**