New features & updated documentation #78
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Hi @seantomburke , First of all congrats on the work you've done on this library. |
|
@bsq-panagiotis Thanks for submitting a PR with all the great additions! I'll be reviewing thoroughly to make sure there are no breaking changes with the existing package. |
seantomburke
approved these changes
Mar 5, 2021
seantomburke
reviewed
Mar 5, 2021
seantomburke
reviewed
Mar 5, 2021
seantomburke
reviewed
Mar 5, 2021
seantomburke
reviewed
Mar 6, 2021
# New features added * Ability to report on sitemap crawl errors in returned results. Added a new "errors" property in the `SitesData` object * Added an option to set a concurrency limit to rate limit sitemap crawling. Useful when crawling sitemaps with multiple children to avoid getting blocked by firewalls. seantomburke#77 * Added an option to have retry requests upon failure and to set the number of maximum retries per crawl. # Documentation changes * Updated documentation to include all the new features described above. Co-Authored-By: Panagiotis Tzamtzis <[email protected]> Co-Authored-By: PanagiotisTzamtzis <[email protected]>
In this case the errors object in the results was not an ErrorsDataArray but a single ErrorsData
…returnErrors option
seantomburke
added a commit
that referenced
this pull request
Dec 24, 2021
* New features & updated documentation * Ability to report on sitemap crawl errors in returned results. Added a new "errors" property in the `SitesData` object * Added an option to set a concurrency limit to rate limit sitemap crawling. Useful when crawling sitemaps with multiple children to avoid getting blocked by firewalls. #77 * Added an option to have retry requests upon failure and to set the number of maximum retries per crawl. * Updated documentation to include all the new features described above. Co-Authored-By: Panagiotis Tzamtzis <[email protected]> Co-Authored-By: PanagiotisTzamtzis <[email protected]> * Fix for error on the main sitemap In this case the errors object in the results was not an ErrorsDataArray but a single ErrorsData * Bug fixes * Error logging improvements with more details for `UnknownStateErrors` & errors when parsing the parent sitemap * Retries option was not working when `debug` was set to false * Bug fix * Console.log statement was getting triggered when `debug` option was set to false * Update src/examples/index.js * 3.2.0 * Cleaning up, changing error to errors, updating Typescript, removing returnErrors option * Removing returnErrors option * quotes fix * Updates * Fixing errors array * updating tests Co-authored-by: PanagiotisTzamtzis <[email protected]> Co-authored-by: Sean Thomas Burke <[email protected]> Co-authored-by: Sean Thomas Burke <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New features added
Ability to report on sitemap crawl errors in returned results. Added a new "errors" property in the
SitesDataobjectAdded an option to set a concurrency limit using the
p-limitlibrary, to rate limit sitemap crawling. Useful when crawling sitemaps with multiple children to avoid getting blocked by firewalls. Default max concurrency limit is set to10Throttling when parsing multiple sitemaps #77Added an option to have retry requests upon failure and to set the number of maximum retries per crawl.
Documentation changes
Co-Authored-By: Panagiotis Tzamtzis [email protected]
Co-Authored-By: Panagiotis Tzamtzis [email protected]