[Feature Request] ESI Support #19
Labels
Comments
|
I'd like to have ESI implemented |
jiangwenyuan
pushed a commit
that referenced
this issue
Mar 10, 2019
…own target srv_list There was a bug reported in issue #19 regarding the fact that haproxy could mis-route requests to the wrong server. It turns out that when switching to another server, the old connection was put back into the srv_list corresponding to the stream's target instead of this connection's target. Thus if this connection was later picked, it was pointing to the wrong server. The patch fixes this and also clarifies the assignment to srv_conn->target so that it's clear we don't change it when picking it from the srv_list. This must be backported to 1.9 only. (cherry picked from commit 3c4e19f) Signed-off-by: Willy Tarreau <[email protected]>
jiangwenyuan
pushed a commit
that referenced
this issue
Dec 9, 2019
…own target srv_list There was a bug reported in issue #19 regarding the fact that haproxy could mis-route requests to the wrong server. It turns out that when switching to another server, the old connection was put back into the srv_list corresponding to the stream's target instead of this connection's target. Thus if this connection was later picked, it was pointing to the wrong server. The patch fixes this and also clarifies the assignment to srv_conn->target so that it's clear we don't change it when picking it from the srv_list. This must be backported to 1.9 only.
jiangwenyuan
pushed a commit
that referenced
this issue
Feb 13, 2020
Squashed commit of the following: commit 59c0cf32d8e7f7f53a78a332120ddda77912b786 Author: Willy Tarreau <[email protected]> Date: Wed Feb 6 14:51:03 2019 +0100 [RELEASE] Released version 1.9.4 Released version 1.9.4 with the following main changes : - BUG/MEDIUM: mux-h1: Don't add "transfer-encoding" if message-body is forbidden - BUG/MEDIUM: compression: Rewrite strong ETags - DOC: compression: Update the reasons for disabled compression - BUG/MINOR: deinit: tcp_rep.inspect_rules not deinit, add to deinit - DOC: add a missing space in the documentation for bc_http_major - SCRIPTS: add the issue tracker URL to the announce script - BUG/MEDIUM: connections: Don't forget to remove CO_FL_SESS_IDLE. - BUG/MINOR: server: fix logic flaw in idle connection list management - BUG/MINOR: stream: don't close the front connection when facing a backend error - MINOR: xref: Add missing barriers. - BUG/MEDIUM: peers: Handle mux creation failure. - BUG/MEDIUM: checks: Check that conn_install_mux succeeded. - BUG/MEDIUM: servers: Only destroy a conn_stream we just allocated. - BUG/MEDIUM: servers: Don't add an incomplete conn to the server idle list. - BUG/MEDIUM: checks: Don't try to set ALPN if connection failed. - BUG/MEDIUM: h2: In h2_send(), stop the loop if we failed to alloc a buf. - BUG/MEDIUM: servers: Close the connection if we failed to install the mux. - BUG/MEDIUM: buffer: Make sure b_is_null handles buffers waiting for allocation. - DOC: htx: make it clear that htxbuf() and htx_from_buf() always return valid pointers - MINOR: htx: never check for null htx pointer in htx_is_{,not_}empty() - MEDIUM: stream-int: always mark pending outgoing SI_ST_CON - MINOR: stream: don't wait before retrying after a failed connection reuse - MEDIUM: h2: always parse and deduplicate the content-length header - BUG/MINOR: mux-h2: always compare content-length to the sum of DATA frames - BUG/MEDIUM: mux-h2: only close connection on request frames on closed streams - BUG/MEDIUM: mux-h2: wake up flow-controlled streams on initial window update - BUG/MEDIUM: mux-h2: fix two half-closed to closed transitions - BUG/MEDIUM: mux-h2: make sure never to send GOAWAY on too old streams - BUG/MEDIUM: mux-h2: do not abort HEADERS frame before decoding them - BUG/MINOR: mux-h2: make sure response HEADERS are not received in other states than OPEN and HLOC - MINOR: h2: add a generic frame checker - MEDIUM: mux-h2: check the frame validity before considering the stream state - CLEANUP: mux-h2: remove misleading leftover test on h2s' nullity - CLEANUP: mux-h2: clean the stream error path on HEADERS frame processing - CLEANUP: mux-h2: remove stream ID and frame length checks from the frame parsers - BUG/MINOR: mux-h2: make sure request trailers on aborted streams don't break the connection - MINOR: mux-h2: consistently rely on the htx variable to detect the mode - BUG/MEDIUM: mux-h2: wait for the mux buffer to be empty before closing the connection - MINOR: stream-int: add a new flag to mention that we want the connection to be killed - MINOR: connstream: have a new flag CS_FL_KILL_CONN to kill a connection - BUG/MEDIUM: mux-h2: do not close the connection on aborted streams - MINOR: mux-h2: max-concurrent-streams should be unsigned - MINOR: mux-h2: make sure to only check concurrency limit on the frontend - MINOR: mux-h2: learn and store the peer's advertised MAX_CONCURRENT_STREAMS setting - BUG/MEDIUM: mux-h2: properly consider the peer's advertised max-concurrent-streams - BUG/MEDIUM: backend: always release the previous connection into its own target srv_list - BUG/MEDIUM: htx: check the HTX compatibility in dynamic use-backend rules - BUG/MINOR: backend: check srv_conn before dereferencing it - BUG/MEDIUM: mux-h2: always omit :scheme and :path for the CONNECT method - BUG/MEDIUM: mux-h2: always set :authority on request output - BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free(). - BUG/MINOR: config: fix bind line thread mask validation - BUG/MINOR: compression: properly report compression stats in HTX mode - BUG/MINOR: task: close a tiny race in the inter-thread wakeup - BUG/MAJOR: config: verify that targets of track-sc and stick rules are present - BUG/MAJOR: spoe: verify that backends used by SPOE cover all their callers' processes - MINOR: backend: move url_param_name/len to lbprm.arg_str/len - MINOR: backend: make headers and RDP cookie also use arg_str/len - MINOR: backend: add new fields in lbprm to store more LB options - MINOR: backend: make the header hash use arg_opt1 for use_domain_only - MINOR: backend: remap the balance uri settings to lbprm.arg_opt{1,2,3} - MINOR: backend: move hash_balance_factor out of chash - MEDIUM: backend: move all LB algo parameters into an union - BUG/MAJOR: htx/backend: Make all tests on HTTP messages compatible with HTX - BUG/MINOR: config: make sure to count the error on incorrect track-sc/stick rules commit 6981c190b6aea3d11eadadfd48b2a1fde05e232e Author: Willy Tarreau <[email protected]> Date: Wed Feb 6 10:25:07 2019 +0100 BUG/MINOR: config: make sure to count the error on incorrect track-sc/stick rules When commit 151e1ca98 ("BUG/MAJOR: config: verify that targets of track-sc and stick rules are present") added a check for some process inconsistencies between rules and their stick tables, some errors resulted in a "return 0" statement, which is taken as "no error" in some cases. Let's fix this. This must be backported to all versions using the above commit. (cherry picked from commit 1a0fe3becd99d7860b4eeaccec407325d5a8b8c2) Signed-off-by: Willy Tarreau <[email protected]> commit b5f9915dd56fedec8499161e51931d63816d83d5 Author: Christopher Faulet <[email protected]> Date: Mon Feb 4 12:02:18 2019 +0100 BUG/MAJOR: htx/backend: Make all tests on HTTP messages compatible with HTX A piece of code about the HTX was lost this summer, after the "big merge" (htx/http2/connection layer refactoring). I forgot to keep HTX changes in the functions connect_server() and assign_server(). So, this patch fixes "uri", "url_param" and "hdr" LB algorithms when the HTX is enabled. It also fixes evaluation of the "sni" expression on server lines. This issue was reported on github. See issue #32. This patch must be backported in 1.9. (cherry picked from commit f7679ad4db0ddbd68e37e3d6e8dc9ed30f8a034f) Signed-off-by: Willy Tarreau <[email protected]> commit 3af0a9ecc5016f1bd40b4211f927e351bdc89b9c Author: Willy Tarreau <[email protected]> Date: Mon Jan 14 16:55:42 2019 +0100 MEDIUM: backend: move all LB algo parameters into an union Since all of them are exclusive, let's move them to an union instead of eating memory with the sum of all of them. We're using a transparent union to limit the code changes. Doing so reduces the struct lbprm from 392 bytes to 372, and thanks to these changes, the struct proxy is now down to 6480 bytes vs 6624 before the changes (144 bytes saved per proxy). (cherry picked from commit 0cac26cd8841491ab205357d4a7248a6e2d44e08) [wt: only for ease of merging upcoming fix] Signed-off-by: Willy Tarreau <[email protected]> commit 7f0b4fd34fdae8ac389f66e16b10db9d16e3e5a5 Author: Willy Tarreau <[email protected]> Date: Mon Jan 14 16:50:58 2019 +0100 MINOR: backend: move hash_balance_factor out of chash This one is a proxy option which can be inherited from defaults even if the LB algo changes. Move it out of the lb_chash struct so that we don't need to keep anything separate between these structs. This will allow us to merge them into an union later. It even takes less room now as it fills a hole and removes another one. (cherry picked from commit 76e84f50919be89d8c7c0af4e9dd2be95cf45036) [wt: only for ease of merging upcoming fix] Signed-off-by: Willy Tarreau <[email protected]> commit 743a2a83fcadf737fc6bb80739dfd1b5827ba978 Author: Willy Tarreau <[email protected]> Date: Mon Jan 14 16:14:15 2019 +0100 MINOR: backend: remap the balance uri settings to lbprm.arg_opt{1,2,3} The algo-specific settings move from the proxy to the LB algo this way : - uri_whole => arg_opt1 - uri_len_limit => arg_opt2 - uri_dirs_depth1 => arg_opt3 (cherry picked from commit a9a7249966683eeb0e474c80589cbc1fc5c29f8a) [wt: only for ease of merging upcoming fix] Signed-off-by: Willy Tarreau <[email protected]> commit d8af262911c7c8ffcbf219df380d0e188b70c5b8 Author: Willy Tarreau <[email protected]> Date: Mon Jan 14 16:04:54 2019 +0100 MINOR: backend: make the header hash use arg_opt1 for use_domain_only This is only a boolean extra arg. Let's map it to arg_opt1 and remove hh_match_domain from struct proxy. (cherry picked from commit 9fed8586b5a96efae97207d0a7f290ef5361b2bf) [wt: only for ease of merging upcoming fix] Signed-off-by: Willy Tarreau <[email protected]> commit 8e2851ce128b80bd24dd327723aef74f7ab6bbab Author: Willy Tarreau <[email protected]> Date: Mon Jan 14 16:04:01 2019 +0100 MINOR: backend: add new fields in lbprm to store more LB options Some algorithms require a few extra options (up to 3). Let's provide some room in lbprm to store them, and make sure they're passed from defaults to backends. (cherry picked from commit 20e68378f1e20112458bbee4c9c3646a74261b5a) [wt: only for ease of merging upcoming fix] Signed-off-by: Willy Tarreau <[email protected]> commit c6f844cedee64d71471b53df69c15c7c98385165 Author: Willy Tarreau <[email protected]> Date: Mon Jan 14 15:28:53 2019 +0100 MINOR: backend: make headers and RDP cookie also use arg_str/len These ones used to rely on separate variables called hh_name/hh_len but they are exclusive with the former. Let's use the same variable which becomes a generic argument name and length for the LB algorithm. (cherry picked from commit 484ff076912c0ce43234758fd14216b3d7ece6ca) [wt: only for ease of merging upcoming fix] Signed-off-by: Willy Tarreau <[email protected]> commit 64976fbc0fa538202138a3b8d4d02cef448d3996 Author: Willy Tarreau <[email protected]> Date: Mon Jan 14 15:23:54 2019 +0100 MINOR: backend: move url_param_name/len to lbprm.arg_str/len This one is exclusively used by LB parameters, when using URL param hashing. Let's move it to the lbprm struct under a more generic name. (cherry picked from commit 4c03d1c9b68ec07932a5bedeb25bf5936a83b351) [wt: only for ease of merging upcoming fix] Signed-off-by: Willy Tarreau <[email protected]> commit b54a86df6f12ac82614921a8c0d4078d781b1ea7 Author: Willy Tarreau <[email protected]> Date: Tue Feb 5 13:37:19 2019 +0100 BUG/MAJOR: spoe: verify that backends used by SPOE cover all their callers' processes When a filter is installed on a proxy and references spoe, we must be absolutely certain that the whole chain is valid on a given process when running in multi-process mode. The problem here is that if a proxy 1 runs on process 1, referencing an SPOE agent itself based on a backend running on process 2, this last one will be completely deinited on process 1, and will thus cause random crashes when it gets messages from this proess. This patch makes sure that the whole chain is valid on all of the caller's processes. This fix must be backported to all spoe-enabled maintained versions. It may potentially disrupt configurations which already randomly crash. There hardly is any intermediary solution though, such configurations need to be fixed. (cherry picked from commit 2bdcfde4260ac9115b8a0b7aa916975799273ea9) Signed-off-by: Willy Tarreau <[email protected]> commit c294490c4c977e9c1202ef76b2e0449a038caaa1 Author: Willy Tarreau <[email protected]> Date: Tue Feb 5 11:38:38 2019 +0100 BUG/MAJOR: config: verify that targets of track-sc and stick rules are present Stick and track-sc rules may optionally designate a table in a different proxy. In this case, a number of verifications are made such as validating that this proxy actually exists. However, in multi-process mode, the target table might indeed exist but not be bound to the set of processes the rules will execute on. This will definitely result in a random behaviour especially if these tables do require peer synchronization, because some tasks will be started to try to synchronize form uninitialized areas. The typical issue looks like this : peers my-peers peer foo ... listen proxy bind-process 1 stick on src table ip ... backend ip bind-process 2 stick-table type ip size 1k peers my-peers While it appears obvious that the example above will not work, there are less obvious situations, such as having bind-process in a defaults section and having a larger set of processes for the referencing proxy than the referenced one. The present patch adds checks for such situations by verifying that all processes from the referencing proxy are present on the other one in all track-sc* and stick-* rules, and in sample fetch / converters referencing another table so that sc_inc_gpc0() and similar are safe as well. This fix must be backported to all maintained versions. It may potentially disrupt configurations which already randomly crash. There hardly is any intermediary solution though, such configurations need to be fixed. (cherry picked from commit 151e1ca989968f5092baa593efd9f485e4947d17) Signed-off-by: Willy Tarreau <[email protected]> commit 5b1efaddb90d6140de44c2ed5f79aee1c708c705 Author: Willy Tarreau <[email protected]> Date: Mon Feb 4 10:26:53 2019 +0100 BUG/MINOR: task: close a tiny race in the inter-thread wakeup __task_wakeup() takes care of a small race that exists between threads, but it uses a store barrier that is not sufficient since apparently the state read after clearing the leaf_p pointer sometimes is incorrect. This results in missed wakeups between threads competing at a high rate. Let's use a full barrier instead to serialize the operations. This may be backported to 1.9 though it's extremely unlikely that this bug will ever manifest itself there. (cherry picked from commit 155acffc13d4972f5642a84f5040eec4e9203c11) Signed-off-by: Willy Tarreau <[email protected]> commit c98758f0f46e07178df28218528a1827edf441b9 Author: Willy Tarreau <[email protected]> Date: Mon Feb 4 11:48:03 2019 +0100 BUG/MINOR: compression: properly report compression stats in HTX mode When HTX support was added to HTTP compression, a set of counters was missed, namely comp_in and comp_byp, resulting in no stats being available for compression. This must be backported to 1.9. (cherry picked from commit ef6fd85623ba0137593e60d30cba40c7489d9231) Signed-off-by: Willy Tarreau <[email protected]> commit 2116028dd441165058fc1e89c2914c5c181c5723 Author: Willy Tarreau <[email protected]> Date: Sat Feb 2 17:46:24 2019 +0100 BUG/MINOR: config: fix bind line thread mask validation When no nbproc is specified, a computation leads to reading bind_thread[-1] before checking if the thread mask is valid for a bind conf. It may either report a false warning and compute a wrong mask, or miss some incorrect configs. This must be backported to 1.9 and possibly 1.8. (cherry picked from commit 6b4a39adc4f9f21dec00a118128f179ade698b17) Signed-off-by: Willy Tarreau <[email protected]> commit f49cc4bfbcd052e6dd448a564e3cd9505ae3fbae Author: Olivier Houchard <[email protected]> Date: Fri Feb 1 18:10:46 2019 +0100 BUG/MEDIUM: stream: Don't forget to free s->unique_id in stream_free(). In stream_free(), free s->unique_id. We may still have one, because it's allocated in log.c::strm_log() no matter what, even if it's a TCP connection and thus it won't get free'd by http_end_txn(). Failure to do so leads to a memory leak. This should probably be backported to all maintained branches. (cherry picked from commit 32211a17eb1f1a18d960ec2a451992a928aaaf95) Signed-off-by: Willy Tarreau <[email protected]> commit 4e62738680764a7955267b6869da0d223a702844 Author: Willy Tarreau <[email protected]> Date: Fri Feb 1 16:13:59 2019 +0100 BUG/MEDIUM: mux-h2: always set :authority on request output PiBa-NL reported that some servers don't fall back to the Host header when :authority is absent. After studying all the combinations of Host and :authority, it appears that we always have to send the latter, hence we never need the former. In case of CONNECT method, the authority is retrieved from the URI part, otherwise it's extracted from the Host field. The tricky part is that we have to scan all headers for the Host header before dumping other headers. This is due to the fact that we must emit pseudo headers before other ones. One improvement could possibly be made later in the request parser to search and emit the Host header immediately if authority was not found. This would cost nothing on the vast marjority of requests and make the lookup faster on output since Host would appear first. This fix must be backported to 1.9. (cherry picked from commit 053c15750bc7207979e8ef8317c1aa125bfe9ea2) Signed-off-by: Willy Tarreau <[email protected]> commit 76388565d52bc963738b69ce5bfbad0bd2b0507e Author: Willy Tarreau <[email protected]> Date: Fri Feb 1 15:51:59 2019 +0100 BUG/MEDIUM: mux-h2: always omit :scheme and :path for the CONNECT method This is mandated by RFC7540 #8.3, these pseudo-headers must not be emitted with the CONNECT method. This must be backported to 1.9. (cherry picked from commit 5be92ff23f7494902091b1bf76d7787ec3fbf706) Signed-off-by: Willy Tarreau <[email protected]> commit 67d66eddd0f7bc96e50bab0671883d675696bee6 Author: Willy Tarreau <[email protected]> Date: Fri Feb 1 16:38:48 2019 +0100 BUG/MINOR: backend: check srv_conn before dereferencing it Commit 3c4e19f42 ("BUG/MEDIUM: backend: always release the previous connection into its own target srv_list") introduced a valid warning about a null-deref risk since we didn't check conn_new()'s return value. This patch must be backported to 1.9 with the patch above. (cherry picked from commit 1da41ecf5bcf7cfd19baaa521cea23af6b4c8adb) Signed-off-by: Willy Tarreau <[email protected]> commit 9f6dbc6ac317ee4b118776f6c7a5813a9d2484a0 Author: Willy Tarreau <[email protected]> Date: Fri Feb 1 15:06:09 2019 +0100 BUG/MEDIUM: htx: check the HTX compatibility in dynamic use-backend rules I would have sworn it was done, probably we lost it during the refactoring. If a frontend is in HTX and the backend not (and conersely), this is normally detected at config parsing time unless the rule is dynamic. In this case we must abort with an error 500. The logs will report "RR" (resource issue while processing request) with the frontend and the backend assigned, so that it's possible to figure what was attempted. This must be backported to 1.9. (cherry picked from commit 3e451842dcd5f2d6f063156e2fa4b78195f097e6) Signed-off-by: Willy Tarreau <[email protected]> commit a4d49a4bb9c49e1cf969b8c4e7cddad1693fef44 Author: Willy Tarreau <[email protected]> Date: Fri Feb 1 11:54:23 2019 +0100 BUG/MEDIUM: backend: always release the previous connection into its own target srv_list There was a bug reported in issue #19 regarding the fact that haproxy could mis-route requests to the wrong server. It turns out that when switching to another server, the old connection was put back into the srv_list corresponding to the stream's target instead of this connection's target. Thus if this connection was later picked, it was pointing to the wrong server. The patch fixes this and also clarifies the assignment to srv_conn->target so that it's clear we don't change it when picking it from the srv_list. This must be backported to 1.9 only. (cherry picked from commit 3c4e19f4297dd426d221859904d3a785bfcf330a) Signed-off-by: Willy Tarreau <[email protected]> commit 13ac3eafaa088a6961f101bb4dc56f3556b87fa5 Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 10:42:05 2019 +0100 BUG/MEDIUM: mux-h2: properly consider the peer's advertised max-concurrent-streams Till now we used to only rely on tune.h2.max-concurrent-streams but if a peer advertises a lower limit this can cause streams to be reset or even the conection to be killed. Let's respect the peer's value for outgoing streams. This patch should be backported to 1.9, though it depends on the following ones : BUG/MINOR: server: fix logic flaw in idle connection list management MINOR: mux-h2: max-concurrent-streams should be unsigned MINOR: mux-h2: make sure to only check concurrency limit on the frontend MINOR: mux-h2: learn and store the peer's advertised MAX_CONCURRENT_STREAMS setting (cherry picked from commit 8694978892587b92125946a5b6f051976c3675a4) Signed-off-by: Willy Tarreau <[email protected]> commit 82e55df49822e9cba2f8351d5b0267b46cb76c1c Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 10:34:07 2019 +0100 MINOR: mux-h2: learn and store the peer's advertised MAX_CONCURRENT_STREAMS setting We used not to take it into account because we only used the configured parameter everywhere. This patch makes sure we can actually learn the value advertised by the peer. We still enforce our own limit on top of it however, to make sure we can actually limit resources or stream concurrency in case of suboptimal server settings. (cherry picked from commit 2e2083ae5b2aafcb05eac79aba92d74803b56a52) Signed-off-by: Willy Tarreau <[email protected]> commit ea3d4abc67af4819e2b6f7a9f94c0976ac3b34fc Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 10:31:51 2019 +0100 MINOR: mux-h2: make sure to only check concurrency limit on the frontend h2_has_too_many_cs() was renamed to h2_frt_has_too_many_cs() to make it clear it's only used to throttle the frontend connection, and the call places were adjusted to only call this code on a front connection. In practice it was already the case since the H2_CF_DEM_TOOMANY flag is only set there. But now the ambiguity is removed. (cherry picked from commit fa1d357f05872a4485677d6bb4c47b2942b8cd51) Signed-off-by: Willy Tarreau <[email protected]> commit 88d31617eec2626fc592393cbf46ebe3c9e6521e Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 10:39:51 2019 +0100 MINOR: mux-h2: max-concurrent-streams should be unsigned We compare it to other unsigned values, let's make it unsigned as well. (cherry picked from commit 5a490b669e6501c66d64b19c80de7c7ea57da7aa) Signed-off-by: Willy Tarreau <[email protected]> commit b147fb2fdcfef423e545cb43afdbe1268874f796 Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 19:12:48 2019 +0100 BUG/MEDIUM: mux-h2: do not close the connection on aborted streams We used to rely on a hint that a shutw() or shutr() without data is an indication that the upper layer had performed a tcp-request content reject and really wanted to kill the connection, but sadly there is another situation where this happens, which is failed keep-alive request to a server. In this case the upper layer stream silently closes to let the client retry. In our case this had the side effect of killing all the connection. Instead of relying on such hints, let's address the problem differently and rely on information passed by the upper layers about the intent to kill the connection. During shutr/shutw, this is detected because the flag CS_FL_KILL_CONN is set on the connstream. Then only in this case we send a GOAWAY(ENHANCE_YOUR_CALM), otherwise we only send the reset. This makes sure that failed backend requests only fail frontend requests and not the whole connections anymore. This fix relies on the two previous patches adding SI_FL_KILL_CONN and CS_FL_KILL_CONN as well as the fix for the connection close, and it must be backported to 1.9 and 1.8, though the code in 1.8 could slightly differ (cs is always valid) : BUG/MEDIUM: mux-h2: wait for the mux buffer to be empty before closing the connection MINOR: stream-int: add a new flag to mention that we want the connection to be killed MINOR: connstream: have a new flag CS_FL_KILL_CONN to kill a connection (cherry picked from commit 180590409ffd34d4032f89839482ab098aae6f04) Signed-off-by: Willy Tarreau <[email protected]> commit d3ee7806176c4f5ae666ae1d5d197a364e191033 Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 19:09:59 2019 +0100 MINOR: connstream: have a new flag CS_FL_KILL_CONN to kill a connection This is the equivalent of SI_FL_KILL_CONN but for the connstreams. It will be set by the stream-interface during the various shutdown operations. (cherry picked from commit 51d0a7e54c4d2b1c90cc182a022f3635ac0ebf1c) Signed-off-by: Willy Tarreau <[email protected]> commit 3c297e2b0d5f7b88ce7b5b476649e7b2c3bd839c Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 19:02:43 2019 +0100 MINOR: stream-int: add a new flag to mention that we want the connection to be killed The new flag SI_FL_KILL_CONN is now set by the rare actions which deliberately want the whole connection (and not just the stream) to be killed. This is only used for "tcp-request content reject", "tcp-response content reject", "tcp-response content close" and "http-request reject". The purpose is to desambiguate the close from a regular shutdown. This will be used by the next patches. (cherry picked from commit 0f9cd7b196073f6d3a3826049b985edcd20c18be) Signed-off-by: Willy Tarreau <[email protected]> commit ddc68d3cc6a6ce8b4b4ac1853252701c1d984059 Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 18:48:20 2019 +0100 BUG/MEDIUM: mux-h2: wait for the mux buffer to be empty before closing the connection When finishing to respond on a stream, a shutw() is called (resulting in either an end of stream or RST), then h2_detach() is called, and may decide to kill the connection is a number of conditions are satisfied. Actually one of these conditions is that a GOAWAY frame was already sent or attempted to be sent. This one is wrong, because it can happen in at least these two situations : - a shutw() sends a GOAWAY to obey tcp-request content reject - a graceful shutdown is pending In both cases, the connection will be aborted with the mux buffer holding some data. In case of a strong abort the client will not see the GOAWAY or RST and might want to try again, which is counter-productive. In case of the graceful shutdown, it could result in truncated data. It looks like a valid candidate for the issue reported here : https://www.mail-archive.com/[email protected]/msg32433.html A backport to 1.9 and 1.8 is necessary. (cherry picked from commit 4dbda620f2872b33aefc8c87ac34f7c71dbd1701) Signed-off-by: Willy Tarreau <[email protected]> commit a80a8142d4db589fa3653f29f846ec6df62db76a Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 07:23:00 2019 +0100 MINOR: mux-h2: consistently rely on the htx variable to detect the mode In h2_frt_transfer_data(), we support both HTX and legacy modes. The HTX mode is detected from the proxy option and sets a valid pointer into the htx variable. Better rely on this variable in all the function rather than testing the option again. This way the code is clearer and even the compiler knows this pointer is valid when it's dereferenced. This should be backported to 1.9 if the b_is_null() patch is backported. (cherry picked from commit a9b7796862e2ab61745d142dde82e615407fee46) Signed-off-by: Willy Tarreau <[email protected]> commit dc881ce93642e3e0f10bfeb5ef5d2191bc756f77 Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 11:44:07 2019 +0100 BUG/MINOR: mux-h2: make sure request trailers on aborted streams don't break the connection We used to respond a connection error in case we received a trailers frame on a closed stream, but it's a problem to do this if the error was caused by a reset because the sender has not yet received it and is just a victim of the timing. Thus we must not close the connection in this case. This patch may be backported to 1.9 but then it requires the following previous ones : MINOR: h2: add a generic frame checker MEDIUM: mux-h2: check the frame validity before considering the stream state CLEANUP: mux-h2: remove stream ID and frame length checks from the frame parsers (cherry picked from commit 1f035507af2bb5dfba51b844f2c57e5698e5c2e9) Signed-off-by: Willy Tarreau <[email protected]> commit e2f72031ceeab41af85a290303b0e017b3d20033 Author: Willy Tarreau <[email protected]> Date: Fri Feb 1 08:41:23 2019 +0100 CLEANUP: mux-h2: remove stream ID and frame length checks from the frame parsers It's not convenient to have such structural checks mixed with the ones related to the stream state. Let's remove all these basic tests that are already covered once for all when reading the frame header. (cherry picked from commit b860c7375655495939424f29137f4e339fa21db4) Signed-off-by: Willy Tarreau <[email protected]> commit f059e793b303bef484ee3ac97f3445d4dcc937a4 Author: Willy Tarreau <[email protected]> Date: Thu Jan 3 11:48:45 2019 +0100 CLEANUP: mux-h2: clean the stream error path on HEADERS frame processing In h2c_frt_handle_headers() and h2c_bck_handle_headers() we have an unused error path made of the strm_err label, while send_rst is used to emit an RST upon stream error after forcing the stream to h2_refused_stream. Let's remove this unused strm_err block now. (cherry picked from commit b8c4dd332040b4af97827e3c887b985e92d3af1d) Signed-off-by: Willy Tarreau <[email protected]> commit d68b0d328ff1e5c214477c7a9599e996d5753013 Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 15:42:44 2019 +0100 CLEANUP: mux-h2: remove misleading leftover test on h2s' nullity The WINDOW_UPDATE and DATA frame handlers used to still have a check on h2s to return either h2s_error() or h2c_error(). This is a leftover from the early code. The h2s cannot be null there anymore as it has already been dereferenced before reaching these locations. (cherry picked from commit 6432dc87837c95f5364365bf499f4848f8a1e36e) Signed-off-by: Willy Tarreau <[email protected]> commit ecf434bfa711fc4bd7a94588aee792a06cebaf9d Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 15:11:03 2019 +0100 MEDIUM: mux-h2: check the frame validity before considering the stream state There are some uneasy situation where it's difficult to validate a frame's format without being in an appropriate state. This patch makes sure that each frame passes through h2_frame_check() before being checked in the context of the stream's state. This makes sure we can always return a GOAWAY for protocol violations even if we can't process the frame. (cherry picked from commit 54f46e53dddf465713218fdbe342a4cb7d806ae1) Signed-off-by: Willy Tarreau <[email protected]> commit a33a55e61613ac94b65b33ff00dc7ca5022bd644 Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 15:09:21 2019 +0100 MINOR: h2: add a generic frame checker The new function h2_frame_check() checks the protocol limits for the received frame (length, ID, direction) and returns a verdict made of a connection error code. The purpose is to be able to validate any frame regardless of the state and the ability to call the frame handler, and to emit a GOAWAY early in this case. (cherry picked from commit 9c84d8299a733bf9bb979cd147e9c566f9e1ae67) Signed-off-by: Willy Tarreau <[email protected]> commit eccbfe33a92300892dbe75acafd81c12d1c7a392 Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 16:55:48 2019 +0100 BUG/MINOR: mux-h2: make sure response HEADERS are not received in other states than OPEN and HLOC RFC7540#5.1 states that these are the only states allowing any frame type. For response HEADERS frames, we cannot accept that they are delivered on idle streams of course, so we're left with these two states only. It is important to test this so that we can remove the generic CLOSE_STREAM test for such frames in the main loop. This must be backported to 1.9 (1.8 doesn't have response HEADERS). (cherry picked from commit 08bb1d61099e7f12aea17a03c608e51002442ed7) Signed-off-by: Willy Tarreau <[email protected]> commit dfdcc7dc86f7d98223b73f7c3a239399b354b6ba Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 16:58:30 2019 +0100 BUG/MEDIUM: mux-h2: do not abort HEADERS frame before decoding them If a response HEADERS frame arrives on a closed connection (due to a client abort sending an RST_STREAM), it's currently immediately rejected with an RST_STREAM, like any other frame. This is incorrect, as HEADERS frames must first be decoded to keep the HPACK decoder synchronized, possibly breaking subsequent responses. This patch excludes HEADERS/CONTINUATION/PUSH_PROMISE frames from the central closed state test and leaves to the respective frame parsers the responsibility to decode the frame then send RST_STREAM. This fix must be backported to 1.9. 1.8 is not directly impacted since it doesn't have response HEADERS nor trailers thus cannot recover from such situations anyway. (cherry picked from commit 8d9ac3ed8b02c98c42f87a21a387c5d6446d834e) Signed-off-by: Willy Tarreau <[email protected]> commit 27a97e1ac0aadf453fde1811b4a84ca91e3a5847 Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 19:20:09 2019 +0100 BUG/MEDIUM: mux-h2: make sure never to send GOAWAY on too old streams The H2 spec requires to send GOAWAY when the client sends a frame after it has already closed using END_STREAM. Here the corresponding case was the fallback of a series of tests on the stream state, but it unfortunately also catches old closed streams which we don't know anymore. Thus any late packet after we've sent an RST_STREAM will trigger this GOAWAY and break other streams on the connection. This can happen when launching two tabs in a browser targetting the same slow page through an H2-to-H2 proxy, and pressing Escape to stop one of them. The other one gets an error when the page finally responds (and it generally retries), and the logs in the middle indicate SD-- flags since the late response was cancelled. This patch takes care to only send GOAWAY on streams we still know. It must be backported to 1.9 and 1.8. (cherry picked from commit 24ff1f834151727cb107995b72a72e9992fd8159) Signed-off-by: Willy Tarreau <[email protected]> commit 29922e39eae8e13d3eb7db64b34e6ea1abd90284 Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 19:28:32 2019 +0100 BUG/MEDIUM: mux-h2: fix two half-closed to closed transitions When receiving a HEADERS or DATA frame with END_STREAM set, we would inconditionally switch to half-closed(remote). This is wrong because we could already have been in half-closed(local) and need to switch to closed. This happens in the following situations : - receipt of the end of a client upload after we've already responded (e.g. redirects to POST requests) - receipt of a response on the backend side after we've already finished sending the request (most common case). This may possibly have caused some streams to stay longer than needed at the end of a transfer, though this is not apparent in tests. This must be backported to 1.9 and 1.8. (cherry picked from commit fc10f599cc5e5606c15be4828848e04ed2c70f9c) Signed-off-by: Willy Tarreau <[email protected]> commit d195a9c7e877423106c315d23184b47f4d30971c Author: Willy Tarreau <[email protected]> Date: Wed Jan 30 16:11:20 2019 +0100 BUG/MEDIUM: mux-h2: wake up flow-controlled streams on initial window update When a settings frame updates the initial window, all affected streams's window is updated as well. However the streams are not put back into the send list if they were already blocked on flow control. The effect is that such a stream will only be woken up by a WINDOW_UPDATE message but not by a SETTINGS changing the initial window size. This can be verified with h2spec's test http2/6.9.2/1 which occasionally fails without this patch. It is unclear whether this situation is really met in field, but the fix is trivial, it consists in adding each unblocked streams to the wait list as is done for the window updates. This fix must be backported to 1.9. For 1.8 the patch needs quite a few adaptations. It's better to copy-paste the code block from h2c_handle_window_update() adding the stream to the send_list when its mws is > 0. (cherry picked from commit b1c9edc579aedd608107c4693c17160474b5ae62) Signed-off-by: Willy Tarreau <[email protected]> commit c98550ff5bcac097f12682e2a097e84dd6d2e766 Author: Willy Tarreau <[email protected]> Date: Tue Jan 29 18:33:26 2019 +0100 BUG/MEDIUM: mux-h2: only close connection on request frames on closed streams A subtle bug was introduced with H2 on the backend. RFC7540 states that an attempt to create a stream on an ID not higher than the max known is a connection error. This was translated into rejecting HEADERS frames for closed streams. But with H2 on the backend, if the client aborts and causes an RST_STREAM to be emitted, the stream is effectively closed, and if/once the server responds, it starts by emitting a HEADERS frame with this ID thus it is interpreted as a connection error. This test must of course consider the side the mux is installed on and not take this for a connection error on responses. The effect is that an aborted stream on an outgoing H2 connection, for example due to a client stopping a transfer with option abortonclose set, would lead to an abort of all other streams. In the logs, this appears as one or several CD-- line(s) followed by one or several SD-- lines which are victims. Thanks to Luke Seelenbinder for reporting this problem and providing enough elements to help understanding how to reproduce it. This fix must be backported to 1.9. (cherry picked from commit 3ad5d31bdf66c3a9449bb4af4cb131ff8e2ca662) Signed-off-by: Willy Tarreau <[email protected]> commit a6bab26c8235c2480ba3032469022f84691a576e Author: Willy Tarreau <[email protected]> Date: Thu Jan 24 11:49:37 2019 +0100 BUG/MINOR: mux-h2: always compare content-length to the sum of DATA frames This is mandated by RFC7541#8.1.2.6. Till now we didn't have a copy of the content-length header field. But now that it's already parsed, it's easy to add the check. The reg-test was updated to match the new behaviour as the previous one expected unadvertised data to be silently discarded. This should be backported to 1.9 along with previous patch (MEDIUM: h2: always parse and deduplicate the content-length header) after it has got a bit more exposure. (cherry picked from commit 1915ca273832ba542d72eb0645dd7ccb6d5b945f) [wt: removed the accidental debugging printf] Signed-off-by: Willy Tarreau <[email protected]> commit 825dc34f7d28365508aa5efe44a02f74a81e6d85 Author: Willy Tarreau <[email protected]> Date: Thu Jan 24 11:33:02 2019 +0100 MEDIUM: h2: always parse and deduplicate the content-length header The header used to be parsed only in HTX but not in legacy. And even in HTX mode, the value was dropped. Let's always parse it and report the parsed value back so that we'll be able to store it in the streams. (cherry picked from commit 4790f7c90743ce52021373c5567d78c32c42d972) Signed-off-by: Willy Tarreau <[email protected]> commit 0f39e3ff6e45e69d2516621134ed97423ffac72b Author: Willy Tarreau <[email protected]> Date: Wed Jan 23 16:29:53 2019 +0100 MINOR: stream: don't wait before retrying after a failed connection reuse When a connection reuse fails, we must not wait before retrying, as most likely the issue is related to the reused connection and not to the server itself. This should be backported to 1.9, though it depends on previous patches dealing with SI_ST_CON for connection reuse. (cherry picked from commit f7a259d46f84ade7222debbce937d5d94bc568f4) Signed-off-by: Willy Tarreau <[email protected]> commit b3e932e3153cc8ce89db37daf72463efbfbad91b Author: Willy Tarreau <[email protected]> Date: Wed Jan 23 15:15:09 2019 +0100 MEDIUM: stream-int: always mark pending outgoing SI_ST_CON Before the first send() attempt, we should be in SI_ST_CON, not SI_ST_EST, since we have not yet attempted to send and we are allowed to retry. This is particularly important with complex outgoing muxes which can fail during the first send attempt (e.g. failed stream ID allocation). It only requires that sess_update_st_con_tcp() knows about this possibility, as we must not forcefully close a reused connection when facing an error in this case, this will be handled later. This may be backported to 1.9 with care after some observation period. (cherry picked from commit bf66bd1b8baf4e35d9be3655fe61eb6d017b0cfd) Signed-off-by: Willy Tarreau <[email protected]> commit 532b885507e45ffb0cee9fad8cf96ed1813b8e56 Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 07:31:18 2019 +0100 MINOR: htx: never check for null htx pointer in htx_is_{,not_}empty() The previous patch clarifies the fact that the htx pointer is never null along all the code. This test for a null will never match, didn't catch the pointer 1 before the fix for b_is_null(), but it confuses the compiler letting it think that any dereferences made to this pointer after this test could actually mean we're dereferencing a null. Let's now drop this test. This saves us from having to add impossible tests everywhere to avoid the warning. This should be backported to 1.9 if the b_is_null() patch is backported. (cherry picked from commit e5fcfbed5cbeba61a7de8cbe36f68f0138b95062) Signed-off-by: Willy Tarreau <[email protected]> commit 137ffbad27052d20b5b7ffb2d8084bac171a01d6 Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 07:21:42 2019 +0100 DOC: htx: make it clear that htxbuf() and htx_from_buf() always return valid pointers Update the comments above htxbuf() and htx_from_buf() to make it clear that they always return valid htx pointers so that callers know they do not have to test them. This is only true after the fix on b_is_null() which was the only known corner case. This should be backported to 1.9 if the b_is_null() patch is backported. (cherry picked from commit 245d189cce724967d4709ab41365d909870be829) Signed-off-by: Willy Tarreau <[email protected]> commit eaee767b7f5a7ac4d354560721bc71a161cfa5d6 Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 19:10:02 2019 +0100 BUG/MEDIUM: buffer: Make sure b_is_null handles buffers waiting for allocation. In b_is_null(), make sure we return 1 if the buffer is waiting for its allocation, as users assume there's memory allocated if b_is_null() returns 0. The indirect impact of not having this was that htxbuf() would not match b_is_null() for a buffer waiting for an allocation, and would thus return the value 1 for the htx pointer, causing various crashes under low memory condition. Note that this patch makes gcc versions 6 and above report two null-deref warnings in proto_htx.c since htx_is_empty() continues to check for a null pointer without knowing that this is protected by the test on b_is_null(). This is addressed by the following patches. This should be backported to 1.9. (cherry picked from commit 203d735cac55a46326c10ee36bad241cab38ee73) Signed-off-by: Willy Tarreau <[email protected]> commit 1fd1a6993c1d5fd152508a484dd2daadf7e89bc7 Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 19:11:16 2019 +0100 BUG/MEDIUM: servers: Close the connection if we failed to install the mux. If we failed to install the mux, just close the connection, or conn_fd_handler() will be called for the FD, and crash as soon as it attempts to access the mux' wake method. This should be backported to 1.9. (cherry picked from commit 74931147292699384e42343b2fac59003ee73b0c) Signed-off-by: Willy Tarreau <[email protected]> commit 20d655642b2bf7973be201bf766bd13c09b27fe3 Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 18:28:36 2019 +0100 BUG/MEDIUM: h2: In h2_send(), stop the loop if we failed to alloc a buf. In h2_send(), make sure we break the loop if we failed to alloc a buffer, or we'd end up looping endlessly. This should be backported to 1.9. (cherry picked from commit 2b09443e043bb58b8d3855b21a6b1183f3e1690a) Signed-off-by: Willy Tarreau <[email protected]> commit 1a835db027eb8bc77a5742ac703969a62dc5d4ca Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 16:37:52 2019 +0100 BUG/MEDIUM: checks: Don't try to set ALPN if connection failed. If we failed to connect, don't attempt to set the ALPN, as we don't have a SSL context, anyway. This should be backported to 1.9. (cherry picked from commit a48437bb5e64e04159e320b99ae983f42c2abcb6) Signed-off-by: Willy Tarreau <[email protected]> commit fab7530b6c33ffc8f21fe8c7ff61f5ed28a37c39 Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 16:05:02 2019 +0100 BUG/MEDIUM: servers: Don't add an incomplete conn to the server idle list. If we failed to add the connection to the session, only try to add it back to the server idle list if it has a mux, otherwise the connection is incomplete and unusable. This should be backported to 1.9. (cherry picked from commit 26da323cb9f72c34062a3f023950da100f4fef55) Signed-off-by: Willy Tarreau <[email protected]> commit 9dd9014dbe20632b237b50302793d40fa651e8df Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 15:50:38 2019 +0100 BUG/MEDIUM: servers: Only destroy a conn_stream we just allocated. In connect_server(), if we failed to add the connection to the session, only destroy the conn_stream if we just allocated it, otherwise it may have been allocated outside connect_server(), along with a connection which has its destination address set. Also use si_release_endpoint() instead of cs_destroy(), to make sure the stream_interface doesn't reference it anymore. This should be backported to 1.9. (cherry picked from commit 4dc85538ba6333ef493ff6c81edf6772db1556b2) Signed-off-by: Willy Tarreau <[email protected]> commit aa9dc7136ac84c834ff960dad2f63355bf685ef7 Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 15:47:43 2019 +0100 BUG/MEDIUM: checks: Check that conn_install_mux succeeded. If conn_install_mux failed, then the connection has no mux and won't be usable, so just give up is on failure instead of ignoring it. This should be backported to 1.9. (cherry picked from commit f67be93ae0caa8de538b5e2ac6fa1eef90e3f9cd) Signed-off-by: Willy Tarreau <[email protected]> commit 7f04dd631287c5fcb9e631df670239fc0bf3a3eb Author: Olivier Houchard <[email protected]> Date: Tue Jan 29 19:00:33 2019 +0100 BUG/MEDIUM: peers: Handle mux creation failure. If the mux fails to properly be created by conn_install_mux, fail, instead of silently ignoring it. This should be backported to 1.9. (cherry picked from commit ef60ff38fb55d0a2a0eaaadd23945755258cf609) [wt: adjusted context] Signed-off-by: Willy Tarreau <[email protected]> commit 7fb71841f7b07f3079ee98fd204b23587939c1d5 Author: Olivier Houchard <[email protected]> Date: Fri Jan 18 17:21:32 2019 +0100 MINOR: xref: Add missing barriers. Add a few missing barriers in the xref code, it's unlikely to be a problem for x86, but may be on architectures with weak memory ordering. (cherry picked from commit ff5dd74e25e1069d74635dba9e8215a6093c481e) Signed-off-by: Willy Tarreau <[email protected]> commit 9fae3fc9ba6612180b84a5deb27c5ddee7ac366e Author: Willy Tarreau <[email protected]> Date: Thu Jan 31 18:58:06 2019 +0100 BUG/MINOR: stream: don't close the front connection when facing a backend error In 1.5-dev13, a bug was introduced by commit e3224e870 ("BUG/MINOR: session: ensure that we don't retry connection if some data were sent"). If a connection error is reported after some data were sent (and lost), we used to accidently mark the front connection as being in error instead of only the back one because the two direction flags were applied to the same channel. This case is extremely rare with raw connections but can happen a bit more often with multiplexed streams. This will result in the error not being correctly reported to the client. This patch can be backported to all supported versions. (cherry picked from commit 28e581b21c8229aa50b7e45148dd46fa6f43da5e) Signed-off-by: Willy Tarreau <[email protected]> commit 580eaac20812f065669bd55006e74aaf6e012f05 Author: Willy Tarreau <[email protected]> Date: Sat Jan 26 12:19:01 2019 +0100 BUG/MINOR: server: fix logic flaw in idle connection list management With variable connection limits, it's not possible to accurately determine whether the mux is still in use by comparing usage and max to be equal due to the fact that one determines the capacity and the other one takes care of the context. This can cause some connections to be dropped before they reach their stream ID limit. It seems it could also cause some connections to be terminated with streams still alive if the limit was reduced to match the newly computed avail_streams() value, though this cannot yet happen with existing muxes. Instead let's switch to usage reports and simply check whether connections are both unused and available before adding them to the idle list. This should be backported to 1.9. (cherry picked from commit 00f18a36b6403d9694946bb3377d0b941bc6fe73) Signed-off-by: Willy Tarreau <[email protected]> commit b1b593037d88b6b7d57654783f8ac7d9b9f2c8b0 Author: Olivier Houchard <[email protected]> Date: Thu Jan 31 19:31:19 2019 +0100 BUG/MEDIUM: connections: Don't forget to remove CO_FL_SESS_IDLE. If we're adding a connection to the server orphan idle list, don't forget to remove the CO_FL_SESS_IDLE flag, or we will assume later it's still attached to a session. This should be backported to 1.9. (cherry picked from commit 8788b4111c78be8397155949f245e6b34caa6acd) Signed-off-by: Willy Tarreau <[email protected]> commit e68c1ed442d7294968af061ddac8caf8b6601547 Author: Willy Tarreau <[email protected]> Date: Tue Jan 29 06:51:16 2019 +0100 SCRIPTS: add the issue tracker URL to the announce script This way it's easier for users to follow the status of pending issues with each release. (cherry picked from commit 9589c3bce78a28fc75978b26168e912455d2b525) Signed-off-by: Willy Tarreau <[email protected]> commit 6254c6601e87fd2e7b6e8d374e3b526b8cfb1c4c Author: Jérôme Magnin <[email protected]> Date: Wed Jan 16 14:38:37 2019 +0100 DOC: add a missing space in the documentation for bc_http_major add a missing space between sample fetch name and colon, and make haproxy-dconv happier. (cherry picked from commit 35e53a6a0821cdd3065736de53b03a770441343a) Signed-off-by: Willy Tarreau <[email protected]> commit c6358e374dd040c6b4bc5e6d6933f2b9a1b213d2 Author: Kevin Zhu <[email protected]> Date: Wed Jan 30 16:01:21 2019 +0800 BUG/MINOR: deinit: tcp_rep.inspect_rules not deinit, add to deinit It seems like this can be backported as far as 1.5. (cherry picked from commit 13ebef7ecb94168039241bce66c2bdc0a3789c16) Signed-off-by: Willy Tarreau <[email protected]> commit c3a02aef057cf9947ded458935e3298c0828f3cc Author: Tim Duesterhus <[email protected]> Date: Wed Jan 30 23:46:04 2019 +0100 DOC: compression: Update the reasons for disabled compression - Update the list of status codes to include 201 - 203. - Remove the fact about the temporary workaround for chunked responses (this is verified using reg-test compression/h00000.vtc). - Add malformed ETags see b229f018eedef4d18571ce6da23d8e153249a836 This commit should be backported together with b229f018eedef4d18571ce6da23d8e153249a836 the changes should be correct until 1.7 at the very least, possibly older. (cherry picked from commit bb48c9a0eec653147be0e39ff85e624f7a4d5553) Signed-off-by: Willy Tarreau <[email protected]> commit e90aa0fd59b72c2be43bd4d5286540659eb1b8c2 Author: Tim Duesterhus <[email protected]> Date: Tue Jan 29 16:38:56 2019 +0100 BUG/MEDIUM: compression: Rewrite strong ETags RFC 7232 section 2.3.3 states: > Note: Content codings are a property of the representation data, > so a strong entity-tag for a content-encoded representation has to > be distinct from the entity tag of an unencoded representation to > prevent potential conflicts during cache updates and range > requests. In contrast, transfer codings (Section 4 of [RFC7230]) > apply only during message transfer and do not result in distinct > entity-tags. Thus a strong ETag must be changed when compressing. Usually this is done by converting it into a weak ETag, which represents a semantically, but not byte-by-byte identical response. A conversion to a weak ETag still allows If-None-Match to work. This should be backported to 1.9 and might be backported to every supported branch with compression. (cherry picked from commit b229f018eedef4d18571ce6da23d8e153249a836) Signed-off-by: Willy Tarreau <[email protected]> commit 97a5d75ac2ccd15f80aae449f66d3dd04fb16d61 Author: Christopher Faulet <[email protected]> Date: Wed Jan 30 21:55:21 2019 +0100 BUG/MEDIUM: mux-h1: Don't add "transfer-encoding" if message-body is forbidden When a HTTP/1.1 or above response is emitted to a client, if the flag H1_MF_XFER_LEN is set whereas H1_MF_CLEN and H1_MF_CHNK are not, the header "transfer-encoding" is added. It is a way to make HTX chunking consistent with H2. But we must exclude all cases where the message-body is explicitly forbidden by the RFC: * for all 1XX, 204 and 304 responses * for any responses to HEAD requests * for 200 responses to CONNECT requests For these 3 cases, the flag H1_MF_XFER_LEN is set but H1_MF_CLEN and H1_MF_CHNK not. And the header "transfer-encoding" must not be added. See issue #27 on github for details about the bug. This patch must be backported in 1.9. (cherry picked from commit 3949c9d90d0328bbe34578e9ac6768a4f2c5615a) Signed-off-by: Willy Tarreau <[email protected]> commit fcf3d4d043b0eb652e6d48168b592399ca1ff68f Author: Willy Tarreau <[email protected]> Date: Tue Jan 29 06:09:34 2019 +0100 [RELEASE] Released version 1.9.3 Released version 1.9.3 with the following main changes : - REGTEST: checks basic stats webpage functionality - BUG/MEDIUM: checks: fix recent regression on agent-check making it crash - BUG/MEDIUM: servers: Make assign_tproxy_address work when ALPN is set. - BUG/MEDIUM: connections: Add the CO_FL_CONNECTED flag if a send succeeded. - BUG/MINOR: startup: certain goto paths in init_pollers fail to free - BUG/MINOR: server: don't always trust srv_check_health when loading a server state - BUG/MINOR: check: Wake the check task if the check is finished in wake_srv_chk() - BUG/MEDIUM: ssl: Fix handling of TLS 1.3 KeyUpdate messages - DOC: mention the effect of nf_conntrack_tcp_loose on src/dst - BUG/MINOR: proto-htx: Return an error if all headers cannot be received at once - BUG/MEDIUM: mux-h2/htx: Respect the channel's reserve - BUG/MINOR: mux-h1: Apply the reserve on the channel's buffer only - BUG/MINOR: mux-h1: avoid copying output over itself in zero-copy - BUG/MAJOR: mux-h2: don't destroy the stream on failed allocation in h2_snd_buf() - BUG/MEDIUM: backend: also remove from idle list muxes that have no more room - BUG/MEDIUM: mux-h2: properly abort on trailers decoding errors - MINOR: h2: declare new sets of frame types - BUG/MINOR: mux-h2: CONTINUATION in closed state must always return GOAWAY - BUG/MINOR: mux-h2: headers-type frames in HREM are always a connection error - BUG/MINOR: mux-h2: make it possible to set the error code on an already closed stream - BUG/MINOR: hpack: return a compression error on invalid table size updates - MINOR: server: make sure pool-max-conn is >= -1 - BUG/MINOR: stream: take care of synchronous errors when trying to send - BUG/MINOR: mux-h2: always check the stream ID limit in h2_avail_streams() - BUG/MINOR: mux-h2: refuse to allocate a stream with too high an ID - BUG/MEDIUM: backend: never try to attach to a mux having no more stream available - MINOR: server: add a max-reuse parameter - MINOR: mux-h2: always consider a server's max-reuse parameter - DOC: nbthread is no longer experimental. - BUG/MINOR: listener: always fill the source address for accepted socketpairs - BUG/MINOR: mux-h2: do not report available outgoing streams after GOAWAY - BUG/MINOR: spoe: corrected fragmentation string size - BUG/MINOR: task: fix possibly missed event in inter-thread wakeups - BUG/MEDIUM: servers: Attempt to reuse an unfinished connection on retry. - BUG/MEDIUM: backend: always call si_detach_endpoint() on async connection failure commit 9f61fc91f526d5f62940a9966a9c05e01cff7314 Author: Willy Tarreau <[email protected]> Date: Mon Jan 28 16:33:35 2019 +0100 BUG/MEDIUM: backend: always call si_detach_endpoint() on async connection failure In case an asynchronous connection (ALPN) succeeds but the mux fails to attach, we must release the stream interface's endpoint, otherwise we leave the stream interface with an endpoint pointing to a freed connection with si_ops == si_conn_ops, and sess_update_st_cer() calls si_shutw() on it, causing it to crash. This must be backported to 1.9 only. (cherry picked from commit d822013f45dea36d23a01a1d2f3680525fbc5e95) Signed-off-by: Willy Tarreau <[email protected]> commit 97a42489b426880f75624990f1d3f61eaa28ee37 Author: Olivier Houchard <[email protected]> Date: Mon Jan 28 15:33:15 2019 +0100 BUG/MEDIUM: servers: Attempt to reuse an unfinished connection on retry. In connect_server(), if the previous connection failed, but had an alpn, no mux was created, and thus the stream_interface's endpoint would be the connection. In this case, instead of forgetting about it, and overriding the stream_interface's endpoint later, try to reuse the connection, or the connection will still be in the session's connection list, and will reference to a stream that was probably destroyed. This should be backported to 1.9. (cherry picked from commit 9ef5155ba6b7455d3c8c88bf661aa35dd0096003) Signed-off-by: Willy Tarreau <[email protected]> commit 467994d2dd072fc531562d734827f7ade7655a21 Author: Willy Tarreau <[email protected]> Date: Sun Jan 27 17:41:27 2019 +0100 BUG/MINOR: task: fix possibly missed event in inter-thread wakeups There's a very small but existing uncertainty window when waking another thread up where it is possible for task_wakeup() not to wake the other task up because it's still running while this once is in the process of finishing and loses its TASK_RUNNING flag. In this case the wakeup will be missed. The problem is that we have a single flag to store 3 states, since the transition from running to sleeping isn't atomic. Thus we need to have another flag to cover this part. This patch introduces TASK_QUEUED to mention that the task is already in the run queue, running or not. This bit will be removed while TASK_RUNNING is kept once dequeued, and will be used when removing TASK_RUNNING to check if the task has been requeued. It might be possible to slightly improve this but the occurrence rate is quite low and we don't really need to complexify the scheduler to optimize for a rare case. The impact with the current code is very low since we have few inter- thread wakeups. Most of them are caused by checks killing sessions. This must be backported to 1.9. (cherry picked from commit 13afcb7ab31a8f4e82ea943128706b5bd8df9e51) Signed-off-by: Willy Tarreau <[email protected]> commit eac5bea548dd07d82af12ef91f47a2e908e52bd7 Author: Miroslav Zagorac <[email protected]> Date: Sun Jan 13 16:55:01 2019 +0100 BUG/MINOR: spoe: corrected fragmentation string size This patch must be backported to 1.9 and 1.8. (cherry picked from commit 6b3690bc6ae44c60677d55c6a82b459f76b91e30) Signed-off-by: Willy Tarreau <[email protected]> commit 0fd47aaeffd6a74615522ec61349197d68ee1c7e Author: Willy Tarreau <[email protected]> Date: Mon Jan 28 06:40:19 2019 +0100 BUG/MINOR: mux-h2: do not report available outgoing streams after GOAWAY The calculation of available outgoing H2 streams was improved by commit d64a3ebe6 ("BUG/MINOR: mux-h2: always check the stream ID limit in h2_avail_streams()"), but it still is incorrect because RFC7540#6.8 specifically forbids the creation of new streams after a GOAWAY frame was received. Thus we must not mark the connection as available anymore in order to be able to handle a graceful shutdown. This needs to be backported to 1.9. (cherry picked from commit 6afec46ba327ef4de202271397f61ef4cf8a0d7d) Signed-off-by: Willy Tarreau <[email protected]> commit 8762e251bea780ff55a843f8e5b8c45efe690624 Author: Willy Tarreau <[email protected]> Date: Sun Jan 27 18:34:12 2019 +0100 BUG/MINOR: listener: always fill the source address for accepted socketpairs The source address was not set but passed down the chain to the upper layer's accept() calls. Let's initialize it like other UNIX sockets in this case. At the moment it should not have any impact since socketpairs are only usable for the master CLI. This should be backported to 1.9. (cherry picked from commit 888d5678f7829e0685d0f692cdb159e655ac0e25) Signed-off-by: Willy Tarreau <[email protected]> commit 227f473d78d2cacdc01fdab80b9bc337753ec4c0 Author: Willy Tarreau <[email protected]> Date: Sat Jan 26 14:20:55 2019 +0100 DOC: nbthread is no longer experimental. It was mentioned when releasing 1.8 but early bugs have long been addressed and this comment discourages some users from using threads. This should be backported to 1.9 and 1.8 now. (cherry picked from commit 1f672a8162eda18c404c6784dd749b6e061e2e4d) Signed-off-by: Willy Tarreau <[email protected]> commit 5d39138956c8598b5b3a9bfad29bba047f8c9d4c Author: Willy Tarreau <[email protected]> Date: Wed Jan 23 10:25:10 2019 +0100 MINOR: mux-h2: always consider a server's max-reuse parameter This parameter allows to limit the number of successive requests sent on a connection. Let's compare it to the number of streams already sent on the connection to decide if the connection may still appear in the idle list or not. This may be used to help certain servers work around resource leaks, and also helps dealing with the issue of the GOAWAY in flight which requires to set a usage limit on the client to be reliable. This must be backported to 1.9. (cherry picked from commit e9634bdc224c9c56854766647207e4f1b000f713) Signed-off-by: Willy Tarreau <[email protected]> commit 51bf96b23eaa47e348e67cb2917ae71f31d5d8cc Author: Willy Tarreau <[email protected]> Date: Wed Jan 23 10:21:49 2019 +0100 MINOR: server: add a max-reuse parameter Some servers may wish to limit the total number of requests they execute over a connection because some of their components might leak resources. In HTTP/1 it was easy, they just had to emit a "connection: close" header field with the last response. In HTTP/2, it's less easy because the info is not always shared with the component dealing with the H2 protocol and it could be harder to advertise a GOAWAY with a stream limit. This patch provides a solution to this by adding a new "max-reuse" parameter to the server keyword. This parameter indicates how many times an idle connection may be reused for new request…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is it possible to implement ESI support?
http cache is not much powerful without it.
The text was updated successfully, but these errors were encountered: