Occasional data loss with .write_all on io::stdout() without explicit .flush

[kpcyrd]

Version

│   │   ├── tokio v1.43.0
│   │   │   └── tokio-macros v2.5.0 (proc-macro)
│   │   └── tokio-util v0.7.13
│   │       └── tokio v1.43.0 (*)
│   ├── tokio v1.43.0 (*)
│   ├── tokio-rustls v0.24.1
│   │   └── tokio v1.43.0 (*)
│   ├── tokio-stream v0.1.17
│   │   └── tokio v1.43.0 (*)
│   ├── tokio-util v0.7.13 (*)
│   │   ├── tokio v1.43.0 (*)
│   │   │   ├── tokio v1.43.0 (*)
│   │   ├── tokio v1.43.0 (*)
│   │   ├── tokio-rustls v0.26.1
│   │   │   └── tokio v1.43.0 (*)
│   ├── tokio v1.43.0 (*)
│   ├── tokio-rustls v0.26.1 (*)
│   ├── tokio-socks v0.5.2
│   │   └── tokio v1.43.0 (*)
│   ├── tokio-util v0.7.13 (*)
│   │   ├── tokio v1.43.0 (*)
├── tokio v1.43.0 (*)
├── tokio-socks v0.5.2 (*)

Platform

Linux redacted 6.7.6-hardened1-1-hardened #1 SMP PREEMPT_DYNAMIC Sat, 24 Feb 2024 23:47:28 +0000 x86_64 GNU/Linux

Description

I had code along the lines of:

let value: Cow<'_, [u8]> = /* some stuff here */;
stdout.write_all(&value).await?;

The program exits almost immediately afterwards. Upon testing I noticed it would sometimes fail to print anything, I first assumed a bug in my implementation but adding stdout.flush().await?; made the problem go away, and ##rust on irc confirmed stdout is buffered in Rust:

let value: Cow<'_, [u8]> = /* some stuff here */;
stdout.write_all(&value).await?;
stdout.flush().await?;

Usually the Rust standard library flushes stdout on exit:

12:41 <Alexendoo> kpcyrd: The unwritten output is flushed at a normal exit though (return from main/std::process::exit)

It seems this doesn't work well with tokio, possibly because it would require an .await, although #[tokio::main] should be able to do this for me, so I don't have to sprinkle stdout.flush().await?; throughout my codebase to avoid silent data loss.

[Darksonn]

Thanks for this issue. I agree that stdout and stderr should use spawn_mandatory_blocking instead of spawn_blocking, just like tokio::fs does.

Of course, we cannot fully guarantee that the operation will really happen. If you call std::process::abort or use shutdown_timeout, then the printing may not happen.