Stream too long when trying to download very large file

[fazleskhan]

I am attempting to host a very large file for HTTP download via Pode and am encountering the following exception

`PS E:\git\navision-docker-container\navlive-backup-copied> pode start
VERBOSE: Adding Route: [Static] /backup[/]{0,1}(?.*)
Listening on the following 1 endpoint(s) [1 thread(s)]:
- http://xxx.xx.x.xxx:xxxx/
Date: 2022-09-13 12:34:39
Level: Error
ThreadId: 1
Server: xxxxxxx
Category: NotSpecified: (:) [Set-PodeResponseAttachment], MethodInvocationException
Message: Exception calling "Write" with "3" argument(s): "Stream was too long."
StackTrace: at Set-PodeResponseAttachment, C:\Program Files\WindowsPowerShell\Modules\Pode\2.7.1\Public\Responses.ps1: line 115
at , : line 97

Date: 2022-09-13 12:34:39
Level: Error
ThreadId: 1
Server: xxxxxx
Category: System.Management.Automation
Message: Exception calling "Write" with "3" argument(s): "Stream was too long."
StackTrace: at System.Management.Automation.ExceptionHandlingOps.CheckActionPreference(FunctionContext funcContext, Exception exception)
at System.Management.Automation.Interpreter.ActionCallInstruction2.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.EnterTryCatchFinallyInstruction.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.Interpreter.Run(InterpretedFrame frame) at System.Management.Automation.Interpreter.LightLambda.RunVoid1[T0](T0 arg0) at System.Management.Automation.PSScriptCmdlet.RunClause(Action1 clause, Object dollarUnderbar, Object inputToProcess)
at System.Management.Automation.PSScriptCmdlet.DoEndProcessing()
at System.Management.Automation.CommandProcessorBase.Complete()

Date: 2022-09-13 12:34:39
Level: Error
ThreadId: 1
Server: xxxxx
Category: mscorlib
Message: Stream was too long.
StackTrace: at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)
at CallSite.Target(Closure , CallSite , Object , Object , Int32 , Object )

172.26.4.26 - - [13/Sep/2022:12:34:39 -04:00] "GET /backup/NavLive-Daily-Full.bak HTTP/1.1" 500 1148 "-" "-"`

The file I am trying to serve is huge +10GB. Based on the stacktrace this issue may be related to the open PowerShell issue

PowerShell/Microsoft.PowerShell.Archive#19

I've tried disabling compression but that does not seem to help. Any suggestions on what to try next?

server.ps1
`
$HostIPAddress = (Get-NetIPAddress -InterfaceIndex 10).IPAddress
Write-Host $HostIPAddress

$Port = 9898
Write-Host $Port

Start-PodeServer {

Add-PodeEndpoint -Address $HostIPAddress -Port $Port -Protocol Http

Add-PodeRoute -Method Get -Path '/' -ScriptBlock {
    Write-PodeJsonResponse -Value @{ 'value' = 'Hello, world!' }
}

Add-PodeStaticRoute -Path '/backup' -Source './backup' -DownloadOnly -Verbose

Use-PodeLogging -Path './log'

New-PodeLoggingMethod -Terminal | Enable-PodeRequestLogging

New-PodeLoggingMethod -Terminal | Enable-PodeErrorLogging

}
`

server.psd1
@{ Server = @{ Request = @{ Timeout = 86400 } } Web = @{ Compression = @{ Enable = $false } } }

[Badgerati]

Hi @fazleskhan,

The issue comes down to MemoryStream using a Integer, which limits the max size to ~2GB. I believe it was at the beginning of the year, I actually fixed the opposite issue, whereby file uploads to the server were capped at 2GB. I might be able to use similar logic that fixed that issue here 🤔

One other workaround for now could be to split the file up into multiple parts, and then reform the file on the client side?

[fazleskhan]

Thank you for letting me know. I've been pulled into another project for the next couple of weeks. When I get a chance I'll give the splitting up the file try.