Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download of large files fails #28

Open
SpiGAndromeda opened this issue Mar 3, 2025 · 9 comments
Open

Download of large files fails #28

SpiGAndromeda opened this issue Mar 3, 2025 · 9 comments
Labels
bug Something isn't working

Comments

@SpiGAndromeda
Copy link

I just uploaded a 7GB large file to my Quickdrop instance. It's shown in the admin panel, I can start the download and I can also fetch the file manually from the server filesystem.

The download breaks after about 1GB.

This is the cruacial part of the logile. As you can see, I already restarted the application (recreated the container)

2025-03-03T18:21:06.193Z ERROR 1 --- [quickdrop] [http-nio-8080-exec-4] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception

java.io.IOException: Broken pipe
	at java.base/sun.nio.ch.SocketDispatcher.write0(Native Method) ~[na:na]
	at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:62) ~[na:na]
	at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:137) ~[na:na]
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:102) ~[na:na]
	at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:58) ~[na:na]
	at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:542) ~[na:na]
	at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:122) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1378) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:764) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.tomcat.util.net.SocketWrapperBase.writeBlocking(SocketWrapperBase.java:589) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.tomcat.util.net.SocketWrapperBase.write(SocketWrapperBase.java:533) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.coyote.http11.Http11OutputBuffer$SocketOutputBuffer.doWrite(Http11OutputBuffer.java:548) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:73) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.coyote.http11.Http11OutputBuffer.doWrite(Http11OutputBuffer.java:193) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.coyote.Response.doWrite(Response.java:633) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:329) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.catalina.connector.OutputBuffer.flushByteBuffer(OutputBuffer.java:776) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.catalina.connector.OutputBuffer.append(OutputBuffer.java:673) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:376) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:354) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:103) ~[tomcat-embed-core-10.1.30.jar!/:na]
	at org.springframework.security.web.util.OnCommittedResponseWrapper$SaveContextServletOutputStream.write(OnCommittedResponseWrapper.java:639) ~[spring-security-web-6.3.3.jar!/:6.3.3]
	at org.springframework.web.context.request.async.StandardServletAsyncWebRequest$LifecycleServletOutputStream.write(StandardServletAsyncWebRequest.java:398) ~[spring-web-6.1.13.jar!/:6.1.13]
	at org.rostislav.quickdrop.service.FileService.lambda$getStreamingResponseBody$0(FileService.java:73) ~[!/:0.0.1-SNAPSHOT]
	at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:110) ~[spring-webmvc-6.1.13.jar!/:6.1.13]
	at org.springframework.web.servlet.mvc.method.annotation.StreamingResponseBodyReturnValueHandler$StreamingResponseBodyTask.call(StreamingResponseBodyReturnValueHandler.java:97) ~[spring-webmvc-6.1.13.jar!/:6.1.13]
	at org.springframework.web.context.request.async.WebAsyncManager.lambda$startCallableProcessing$4(WebAsyncManager.java:367) ~[spring-web-6.1.13.jar!/:6.1.13]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

2025-03-03T18:21:06.806Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-3] o.r.quickdrop.service.FileService        : Sending file: FileEntity{id=7, name='1-Off-24.rar', uuid='d3eb3a54-4154-47c5-ae96-f1b834955c99', description='Und so3', size=7764449100, keepIndefinitely=false, uploadDate=2025-03-03, passwordHash='null'}
2025-03-03T18:21:37.354Z  WARN 1 --- [quickdrop] [http-nio-8080-exec-7] .w.s.m.s.DefaultHandlerExceptionResolver : Ignoring exception, response committed already: org.springframework.web.context.request.async.AsyncRequestTimeoutException
2025-03-03T18:21:37.354Z  WARN 1 --- [quickdrop] [http-nio-8080-exec-7] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.context.request.async.AsyncRequestTimeoutException]
2025-03-03T18:43:56.697Z  INFO 1 --- [quickdrop] [SpringApplicationShutdownHook] j.LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
2025-03-03T18:43:56.699Z  INFO 1 --- [quickdrop] [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...
2025-03-03T18:43:56.706Z  INFO 1 --- [quickdrop] [SpringApplicationShutdownHook] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.
2025-03-03T18:43:59.024Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : Starting QuickdropApplication v0.0.1-SNAPSHOT using Java 21 with PID 1 (/app/quickdrop.jar started by root in /app)
2025-03-03T18:43:59.025Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : The following 1 profile is active: "default"
2025-03-03T18:44:00.286Z  INFO 1 --- [quickdrop] [main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2025-03-03T18:44:00.349Z  INFO 1 --- [quickdrop] [main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 51 ms. Found 4 JPA repository interfaces.
2025-03-03T18:44:00.632Z  INFO 1 --- [quickdrop] [main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=f962a57c-f974-38a2-99ae-75f41b452d2f
2025-03-03T18:44:01.267Z  INFO 1 --- [quickdrop] [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2025-03-03T18:44:01.280Z  INFO 1 --- [quickdrop] [main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2025-03-03T18:44:01.281Z  INFO 1 --- [quickdrop] [main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.30]
2025-03-03T18:44:01.313Z  INFO 1 --- [quickdrop] [main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2025-03-03T18:44:01.313Z  INFO 1 --- [quickdrop] [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2219 ms
2025-03-03T18:44:01.467Z  INFO 1 --- [quickdrop] [main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2025-03-03T18:44:01.577Z  INFO 1 --- [quickdrop] [main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 6.5.3.Final
2025-03-03T18:44:01.607Z  INFO 1 --- [quickdrop] [main] o.h.c.internal.RegionFactoryInitiator    : HHH000026: Second-level cache disabled
2025-03-03T18:44:01.869Z  INFO 1 --- [quickdrop] [main] o.s.o.j.p.SpringPersistenceUnitInfo      : No LoadTimeWeaver setup: ignoring JPA class transformer
2025-03-03T18:44:01.897Z  INFO 1 --- [quickdrop] [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2025-03-03T18:44:02.050Z  INFO 1 --- [quickdrop] [main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.sqlite.jdbc4.JDBC4Connection@6106dfb6
2025-03-03T18:44:02.052Z  INFO 1 --- [quickdrop] [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2025-03-03T18:44:02.799Z  INFO 1 --- [quickdrop] [main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration)
2025-03-03T18:44:02.831Z  INFO 1 --- [quickdrop] [main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2025-03-03T18:44:03.144Z  INFO 1 --- [quickdrop] [main] o.s.d.j.r.query.QueryEnhancerFactory     : Hibernate is in classpath; If applicable, HQL parser will be used.
2025-03-03T18:44:04.592Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : File save path created: files
2025-03-03T18:44:04.737Z  WARN 1 --- [quickdrop] [main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2025-03-03T18:44:04.781Z  INFO 1 --- [quickdrop] [main] eAuthenticationProviderManagerConfigurer : Global AuthenticationManager configured with AuthenticationProvider bean with name authenticationProvider
2025-03-03T18:44:05.186Z  INFO 1 --- [quickdrop] [main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint beneath base path '/actuator'
2025-03-03T18:44:05.882Z  INFO 1 --- [quickdrop] [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
2025-03-03T18:44:05.917Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : Started QuickdropApplication in 7.6 seconds (process running for 8.257)
2025-03-03T18:44:17.883Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2025-03-03T18:44:17.883Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2025-03-03T18:44:17.884Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 1 ms
2025-03-03T18:44:34.383Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-4] o.r.quickdrop.service.FileService        : Sending file: FileEntity{id=7, name='1-Off-24.rar', uuid='d3eb3a54-4154-47c5-ae96-f1b834955c99', description='Und so3', size=7764449100, keepIndefinitely=false, uploadDate=2025-03-03, passwordHash='null'}
2025-03-03T18:45:04.899Z  WARN 1 --- [quickdrop] [http-nio-8080-exec-5] .w.s.m.s.DefaultHandlerExceptionResolver : Ignoring exception, response committed already: org.springframework.web.context.request.async.AsyncRequestTimeoutException
2025-03-03T18:45:04.900Z  WARN 1 --- [quickdrop] [http-nio-8080-exec-5] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.context.request.async.AsyncRequestTimeoutException]
2025-03-03T18:47:49.684Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : Starting QuickdropApplication v0.0.1-SNAPSHOT using Java 21 with PID 1 (/app/quickdrop.jar started by root in /app)
2025-03-03T18:47:49.695Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : The following 1 profile is active: "default"
2025-03-03T18:47:50.940Z  INFO 1 --- [quickdrop] [main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data JPA repositories in DEFAULT mode.
2025-03-03T18:47:50.991Z  INFO 1 --- [quickdrop] [main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 44 ms. Found 4 JPA repository interfaces.
2025-03-03T18:47:51.215Z  INFO 1 --- [quickdrop] [main] o.s.cloud.context.scope.GenericScope     : BeanFactory id=f962a57c-f974-38a2-99ae-75f41b452d2f
2025-03-03T18:47:51.791Z  INFO 1 --- [quickdrop] [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port 8080 (http)
2025-03-03T18:47:51.804Z  INFO 1 --- [quickdrop] [main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
2025-03-03T18:47:51.805Z  INFO 1 --- [quickdrop] [main] o.apache.catalina.core.StandardEngine    : Starting Servlet engine: [Apache Tomcat/10.1.30]
2025-03-03T18:47:51.835Z  INFO 1 --- [quickdrop] [main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
2025-03-03T18:47:51.836Z  INFO 1 --- [quickdrop] [main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 2052 ms
2025-03-03T18:47:52.014Z  INFO 1 --- [quickdrop] [main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [name: default]
2025-03-03T18:47:52.115Z  INFO 1 --- [quickdrop] [main] org.hibernate.Version                    : HHH000412: Hibernate ORM core version 6.5.3.Final
2025-03-03T18:47:52.145Z  INFO 1 --- [quickdrop] [main] o.h.c.internal.RegionFactoryInitiator    : HHH000026: Second-level cache disabled
2025-03-03T18:47:52.408Z  INFO 1 --- [quickdrop] [main] o.s.o.j.p.SpringPersistenceUnitInfo      : No LoadTimeWeaver setup: ignoring JPA class transformer
2025-03-03T18:47:52.437Z  INFO 1 --- [quickdrop] [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...
2025-03-03T18:47:52.556Z  INFO 1 --- [quickdrop] [main] com.zaxxer.hikari.pool.HikariPool        : HikariPool-1 - Added connection org.sqlite.jdbc4.JDBC4Connection@6106dfb6
2025-03-03T18:47:52.557Z  INFO 1 --- [quickdrop] [main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.
2025-03-03T18:47:53.362Z  INFO 1 --- [quickdrop] [main] o.h.e.t.j.p.i.JtaPlatformInitiator       : HHH000489: No JTA platform available (set 'hibernate.transaction.jta.platform' to enable JTA platform integration)
2025-03-03T18:47:53.402Z  INFO 1 --- [quickdrop] [main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
2025-03-03T18:47:53.737Z  INFO 1 --- [quickdrop] [main] o.s.d.j.r.query.QueryEnhancerFactory     : Hibernate is in classpath; If applicable, HQL parser will be used.
2025-03-03T18:47:55.231Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : File save path created: files
2025-03-03T18:47:55.386Z  WARN 1 --- [quickdrop] [main] JpaBaseConfiguration$JpaWebConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
2025-03-03T18:47:55.430Z  INFO 1 --- [quickdrop] [main] eAuthenticationProviderManagerConfigurer : Global AuthenticationManager configured with AuthenticationProvider bean with name authenticationProvider
2025-03-03T18:47:55.966Z  INFO 1 --- [quickdrop] [main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint beneath base path '/actuator'
2025-03-03T18:47:56.660Z  INFO 1 --- [quickdrop] [main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port 8080 (http) with context path '/'
2025-03-03T18:47:56.702Z  INFO 1 --- [quickdrop] [main] o.r.quickdrop.QuickdropApplication       : Started QuickdropApplication in 7.587 seconds (process running for 8.217)
2025-03-03T18:48:09.482Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
2025-03-03T18:48:09.484Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
2025-03-03T18:48:09.486Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 2 ms
2025-03-03T18:48:32.962Z  INFO 1 --- [quickdrop] [http-nio-8080-exec-3] o.r.quickdrop.service.FileService        : Sending file: FileEntity{id=7, name='1-Off-24.rar', uuid='d3eb3a54-4154-47c5-ae96-f1b834955c99', description='Und so3', size=7764449100, keepIndefinitely=false, uploadDate=2025-03-03, passwordHash='null'}
2025-03-03T18:49:03.677Z  WARN 1 --- [quickdrop] [http-nio-8080-exec-4] .w.s.m.s.DefaultHandlerExceptionResolver : Ignoring exception, response committed already: org.springframework.web.context.request.async.AsyncRequestTimeoutException
2025-03-03T18:49:03.677Z  WARN 1 --- [quickdrop] [http-nio-8080-exec-4] .w.s.m.s.DefaultHandlerExceptionResolver : Resolved [org.springframework.web.context.request.async.AsyncRequestTimeoutException]

This is the docker compose configuration:

services:
  app:
    image: roastslav/quickdrop:latest
    restart: unless-stopped
    ports:
      - 8200:8080
    volumes:
      - database:/app/db
      - logs:/app/log
      - /mnt/cloud/quickdrop/files:/app/files
      - /opt/quickdrop/application.properties:/app/application.properties

volumes:
  database:
  logs:

I already created a application.properties file:

max-upload-file-size=10GB

Smaller files can be downloaded without problems. Other files larger 1GB result in the same problems.

@RoastSlav
Copy link
Owner

application.properties was used in older versions. You should set the limit from the settings in the admin dashboard

@SpiGAndromeda
Copy link
Author

The limit is at 100GB and I can easily upload files thar are larger than 1GB. Only the download stops after about 1GB without finishing.

Image

@RoastSlav RoastSlav added the bug Something isn't working label Mar 16, 2025
@RoastSlav
Copy link
Owner

After some more tests, I got a similar issue but with much larger files, much later in the download process and I don't know if it's connected but will update here if I manage to fix it so you can try again

@SpiGAndromeda
Copy link
Author

I played around with the reverse proxy settings (using Nginx Proxy Manager). Setting proxy_max_temp_file_size to the max. filesize did the job.

I am not that much into the file transfer mechanics, but is it normal that a reverse proxy has to buffer the whole response? I thought responses are split into chunks or something.

@RoastSlav
Copy link
Owner

Hey! Yeah, the app handles the file in chunks, but it looks like Nginx tries to buffer the whole file before sending it to the client, which is probably what's causing the error you're seeing. In the next update, I’ve added a header to tell Nginx not to buffer before sending the file, so that should fix the issue.

If it still happens, let me know and I’ll figure out another solution! In the meantime, you can also disable buffering in the Nginx config instead of increasing the limit.

@RoastSlav
Copy link
Owner

You can try 1.4.0 to see if it's fixed.

@SpiGAndromeda
Copy link
Author

Doesn't work: situation is the same with version 1.3.0.
The download stops at nearly the same point with these settings:
proxy_buffering off;
proxy_max_temp_file_size 0;

@SpiGAndromeda
Copy link
Author

Ok it's worse actually.

proxy_buffering on;
proxy_max_temp_file_size 20480m;

Isn't working with v1.4.0. It just stops after 1GB.

@RoastSlav
Copy link
Owner

RoastSlav commented Mar 21, 2025 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants