wasmtime-wasi: Unexpected tcp stream break (Returning 0 for std::io::ErrorKind::WouldBlock)

[Heap-Hop]

wasmtime/crates/wasi/src/tcp.rs

Lines 719 to 723 in 6767488

	// Failing with `EWOULDBLOCK` is how we differentiate between a closed channel and no
	// data to read right now.
	Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => 0,

Returning 0 for std::io::ErrorKind::WouldBlock causes downstream to interpret it as a closed stream, see:
yoshuawuyts/wstd#25 (comment).
https://github.com/yoshuawuyts/wstd/blob/5ce367add5e7bcb569b6487453cb9ba94468dc99/src/io/copy.rs#L12

This is also found in:

wasmtime/crates/test-programs/src/bin/preview2_tcp_streams.rs

Lines 17 to 22 in 6767488

	// The input stream should immediately signal StreamError::Closed.
	// Notably, it should _not_ return an empty list (the wasi-io equivalent of EWOULDBLOCK)
	// See: https://github.com/bytecodealliance/wasmtime/pull/8968
	assert!(matches!(client.input.read(10), Err(StreamError::Closed)));

Given that the wit files already include many would-block errors, would it make sense to extend stream-error to include a would-block?

[pchickey]

WASI streams operations are nonblocking, unless they have blocking_ in their name, so read should always return an empty list instead of blocking. The blocking_ functions are equivalent to waiting for readiness on the stream's pollable, and then calling the nonblocking variant.

This is a significant departure from POSIX, which uses the file's mode to make the read and write syscalls blocking or nonblocking. So, while it makes sense for POSIX to use an error to indicate when a nonblocking socket would block, it doesn't make sense for WASI to follow that same convention.

[bjorn3]

How can you distinguish in WASI between an EOF stream (no data will be returned in the future because the connection was closed or the end of the file is reached) and a stream where you have to wait until the stream is ready before you can read more data? The distinction is very important for std::io::copy/tokio::io::copy which needs to continue reading until EOF and only returns when EOF is reached.

[pchickey]

When the stream is EOF, it returns Err(StreamError::Closed) as seen in the above snippet line 21. Maybe this isn't so much an "Error" as the "Err" constructor indicates, but the alternative was returning a custom enum there of Ok/Closed/Error, which we rejected on fairly arbitrary stylistic grounds.

[Heap-Hop]

Thanks for your kind explanation!

wasmtime/crates/wasi/wit/deps/io/streams.wit

Lines 47 to 54 in 6767488

	///
	/// This function returns a list of bytes containing the read data,
	/// when successful. The returned list will contain up to `len` bytes;
	/// it may return fewer than requested, but not more. The list is
	/// empty when no bytes are available for reading at this time. The
	/// pollable given by `subscribe` will be ready when more bytes are
	/// available.
	///

The pollable given by subscribe will be ready when more bytes are available.

I think it would be helpful to update the doc to explicitly note this situation for users.

[Heap-Hop]

WASI streams operations are nonblocking, unless they have blocking_ in their name

For the same reason, there's a potential bug in HostInputStream::blocking_read(also affects blocking_skip).
The doc states: it blocks until at least one byte can be read.

wasmtime/crates/wasi/src/stream.rs

Lines 23 to 28 in 6767488

	/// Similar to `read`, except that it blocks until at least one byte can be
	/// read.
	async fn blocking_read(&mut self, size: usize) -> StreamResult<Bytes> {
	self.ready().await;
	self.read(size)
	}

The ready() is implemented using readable from Tokio:

wasmtime/crates/wasi/src/tcp.rs

Lines 757 to 762 in 6767488

	#[async_trait::async_trait]
	impl Subscribe for TcpReadStream {
	async fn ready(&mut self) {
	self.0.lock().await.ready().await
	}
	}

wasmtime/crates/wasi/src/tcp.rs

Lines 739 to 745 in 6767488

	async fn ready(&mut self) {
	if self.closed {
	return;
	}
	self.stream.readable().await.unwrap();
	}

However, readable in Tokio does not await when io::ErrorKind::WouldBlock.

[Heap-Hop]

Maybe all we really need is to ensure that every ready() in WASI behaves as the explicit "ready" it's supposed to be.

[pchickey]

Thanks, it sounds like this is indeed a bug in wasmtime-wasi then. Can you create a minimal reproduction so I can work on a fix and use it as a test once we have the fix in? Something along the lines of https://github.com/bytecodealliance/wasmtime/blob/main/crates/test-programs/src/bin/preview2_tcp_streams.rs

[Heap-Hop]

@pchickey https://github.com/bytecodealliance/wasmtime/actions/runs/12070585313/job/33660435175?pr=9691#step:18:419

[pchickey]

Thanks! Its a holiday weekend here so I’ll get to this next week.