Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bootstrap package upload failure over ~55MB. #25533

Closed
rfairburn opened this issue Jan 17, 2025 · 5 comments
Closed

Bootstrap package upload failure over ~55MB. #25533

rfairburn opened this issue Jan 17, 2025 · 5 comments
Assignees
@iansltx
Labels
~backend Backend-related issue. bug Something isn't working as documented customer-eponym #g-software Software product group P2 Prioritize as urgent :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.
Milestone
4.62.2

Comments

@rfairburn
Copy link
Contributor

Fleet version: v4.61.0

Web browser and operating system: Chrome 131.0.6778.265

💥  Actual behavior

When uploading a file that is over ~55 MB or so, the upload fails and the following log entry shows up on the server:

{"component":"http","err":"failed to parse multipart form","internal":"read tcp 10.10.1.170:8080->10.10.13.174:32394: i/o timeout","level":"info","path":"/api/latest/fleet/mdm/bootstrap","ts":"2025-01-16T22:04:52.553139173Z","uuid":"b57f021e-f203-457c-bc69-601a1c0e7592"}

🧑‍💻  Steps to reproduce

  1. Go to bootstrap package upload in the UI
  2. Attempt to upload a package larger than 55-60 MB.

🕯️ More info (optional)

It looks like we should be limited to 512 MB here as it is hard coded at

https://github.com/fleetdm/fleet/blob/main/server/service/apple_mdm.go#L2240-L2277

This behavior may be inconsistent as customer reported eventual success after multiple tries.
@rfairburn rfairburn added :incoming New issue in triage process. :reproduce Involves documenting reproduction steps in the issue bug Something isn't working as documented customer-eponym P2 Prioritize as urgent labels Jan 17, 2025
@rfairburn
Copy link
Contributor Author

FYI @lukeheath

@iansltx iansltx assigned iansltx and unassigned mostlikelee Jan 17, 2025
@iansltx
Copy link
Member

iansltx commented Jan 17, 2025

We're missing an extended read timeout on this endpoint...just like I originally missed this on software edits. Will throw in a PR now for a fix and test in the AM. Should be able to repro this by uploading a large-ish package over a throttled connection.

iansltx added a commit that referenced this issue Jan 17, 2025
@iansltx
Exempt bootstrap package uploads from server-side request timeout
a058ec5 
For #25533
@iansltx
Copy link
Member

iansltx commented Jan 17, 2025

jk, testing tonight.

To repro, grab the Chrome Enterprise pkg (~200MB) and try uploading as a bootstrap package on a connection throttled to 20 Mbps up. Time to upload will be ~80 sec., which is longer than our default request deadline.

With the fix, the file eventually uploads.

@iansltx iansltx added ~backend Backend-related issue. :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. and removed :reproduce Involves documenting reproduction steps in the issue :incoming New issue in triage process. labels Jan 17, 2025
@iansltx iansltx mentioned this issue Jan 17, 2025
Merged
3 tasks
@iansltx iansltx added the #g-software Software product group label Jan 17, 2025
@iansltx iansltx added this to the 4.63.0 milestone Jan 17, 2025
iansltx added a commit that referenced this issue Jan 17, 2025
@iansltx
Exempt bootstrap package uploads from server-side request timeout (#2…
9a5d74c 
…5536)

For #25533

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
iansltx added a commit that referenced this issue Jan 17, 2025
@iansltx
Exempt bootstrap package uploads from server-side request timeout (#2…
c897f3b 
…5536)

For #25533

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
@iansltx iansltx mentioned this issue Jan 17, 2025
Merged
iansltx added a commit that referenced this issue Jan 17, 2025
@iansltx
Cherry-Pick: Exempt bootstrap package uploads from server-side reques…
b6b7032 
…t timeout (#25552)

For #25533, merged into `main` in #25536
@lukeheath lukeheath modified the milestones: 4.63.0, 4.62.2 Jan 17, 2025
lukeheath pushed a commit that referenced this issue Jan 17, 2025
@iansltx @lukeheath
Exempt bootstrap package uploads from server-side request timeout (#2…
cbbf9ef 
…5536)

For #25533

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
lukeheath pushed a commit that referenced this issue Jan 17, 2025
@iansltx @lukeheath
Exempt bootstrap package uploads from server-side request timeout (#2…
f74ab14 
…5536)

For #25533

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
lukeheath pushed a commit that referenced this issue Jan 17, 2025
@iansltx @lukeheath
Exempt bootstrap package uploads from server-side request timeout (#2…
ad5560d 
…5536)

For #25533

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
@jmwatts
Copy link
Member

jmwatts commented Jan 17, 2025 •
edited
Loading

QA Notes

First reproduced on 4.62.0, got the following in the logs:
level=info ts=2025-01-17T18:29:52.957044Z component=http path=/api/latest/fleet/mdm/bootstrap internal="read tcp [::1]:8080->[::1]:64376: i/o timeout" uuid=4d9986ba-a46d-455d-9ff7-9ab4ae03b1cb err="failed to parse multipart form"
And saw:
Image

Tested on 4.62.2
And was able to successfully upload package:
level=debug ts=2025-01-17T19:41:00.307161Z component=http [email protected] method=POST uri=/api/latest/fleet/mdm/bootstrap took=1m29.521490209s

Tested on 4.63.0 as well, and was able to successfully upload a package.
level=debug ts=2025-01-20T14:46:22.109401Z component=http [email protected] method=POST uri=/api/latest/fleet/mdm/bootstrap took=1m47.018030875s

@lukeheath lukeheath added the ~released bug This bug was found in a stable release. label Jan 17, 2025
@fleet-release
Copy link
Contributor

Larger files in view,
Silent streams of data flow,
Fleet's reach expanding too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iansltx iansltx

Labels
~backend Backend-related issue. bug Something isn't working as documented customer-eponym #g-software Software product group P2 Prioritize as urgent :release Ready to write code. Scheduled in a release. See "Making changes" in handbook. ~released bug This bug was found in a stable release.
Projects
None yet
Milestone
4.62.2
Development

No branches or pull requests
6 participants
@iansltx @lukeheath @rfairburn @mostlikelee @fleet-release @jmwatts