Add MaxDocumentLength and custom UserAgent support #13
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
August 7, 2020 13:18
- added support for og:type - fixed incorrect handling of relative paths - changed to "silent" checks Content-Length in Head requests
…o 2 different functions `GetDocument` and `ParseDocument` (#2)
Due to the reason that we "re-create" new link - only by scheme, host, and path - is present risk to lose some other data from the original link. Previously `/some/path.png?param=value`, was transformed into `http://mydomain.com/some/path.png` Now this issue should be fixed, and the output should be `http://mydomain.com/some/path.png?param=value`
For some ULRs was found that we can't get for strange reason the `og:type` data. One of this ULRs - was youtube links. Was detected that in YouTube they keep metadata in body (and not in head as other normal services). And because previously the criteria for breaking loop of procession of tokens was "we have Title + description + ogImage and we passed head" - we were not able to process all other optional meta after that we pass head. Now we are able to control how much tokens we can process before breaking loop (or if we found required optional fields already)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Due to the reason that sometimes URLs can be to some pages/files that are a pretty big one - we want to be able to abort the reading body for those pages.