support async load module info when click module tab

[hardfist]

in our application bundle-stats index.html contains module takes 70M size, which is too big too show, so I wonder whether we could lazy load module info when click module analyze tab

[vio]

Thanks for opening the issue @hardfist!

I am aware of the modules tab performance issue when there are > 5K entries, the list is processed for each run and then compared and rendered. I've been looking recently at possible ways to improve the processing or optimize the rendering (ex: show only the first 100 entries). I will try to pick it up next week and see if there is anything I can do short-term. I would appreciate it if you could share the webpack stats (here or send it to the email on my profile).

For reference, how many modules does your project have?

in our application bundle-stats index.html contains module takes 70M size

Do you mean the resulting bundle-stats report or the original webpack-stats JSON file?

[hardfist]

we have more then 10K+ modules and the original webpack-stats json file is bigger than 70M

[hardfist]

I tried to use zlib to compress stats.json which reduce size from 70M to 3M, which may be helpful

[vio]

@hardfist if the size of the webpack stats JSON file is a concern for you, you can try to export only the webpack stats that are needed for bundle-stats. Here is an example with webpack-stats-plugin, the plugin gives more control when there are other plugins or configs that are altering the default webpack stats options:

webpack-stats-plugin: Output webpack stats

It depends on many factors, but based on existing usage, the webpack-stats for a project with 1-2K modules is around 250KB, and for a project with 6K is ~ 1MB, I think that if you manage to update the webpack-stats generation, your JSON file should be less than 5MB.

[hardfist]

I send the stats.json to your email, I already tried to minimize the stats.json but it is still too large

[vio]

Thanks for sharing the webpack stats @hardfist!

I think that's the minimum size that you can generate using webpack stats options, most of the data is module entries:

# Size in bytes
cat stats.json | jq '.' | wc -c
> 83136327 # 83MB

# Modules data size in bytes
cat stats.json | jq '.modules' | wc -c  
> 79999569 # 80MB

## Module count
cat stats.json | jq '.modules | length'                                                                                            
> 16085

Filtering the module properties based on the ones extracted by bundle-stats, the result is ~4% of the total modules byte size:

cat stats.json | jq '.modules[] | .name, .size, .chunks, .modules' | wc -c
3485193 # 3.5MB

The largest data is held by .issuerPath and reasons:

cat stats.json | jq '.modules[] | .issuerPath' | wc -c 
> 34084164 # 34MB

cat stats.json | jq '.modules[] | .reasons' | wc -c 
> 27531760 # 27.5MB 

If you want to make the JSON file smaller, you can remove issuerPath and/or reasons data since they are not used on the current version of bundle-stats. No plans to use issuerPath, but I am POC-ing extracting reasons data.

To remove the data from the webpack stats you can use webpack-stats-plugin transform function or a tool like jq:

cat stats.json | jq '.modules[] |=  del(.issuerPath, .reasons)' > stats.filtered.json
cat stats.filtered.json | jq '.' | wc -c
> 13995308 # 14MB

[vio]

@hardfist, if you have the chance, please install 4.1.8-beta.0. It renders only the first 1000 rows by default and adds a toggler to show more / less.

Long term, I think react-virtualized might be a better solution for this case, but due to the embedded nature of the report, I avoided increasing the bundle size for now.

[hardfist]

thanks, i will give it a try

[hardfist]

@vio it did reduce bundle-stats.html to 11M, thanks a lot

[hardfist]

It's still large because bundle diff html is twice size of bundle-stats which is 20M, it seems we may need compression and lazy load to solve this problem

[vio]

@hardfist you mean that when you are generating the bundle-stats report with a baseline, the resulting html is 20MB and that is loading slow when trying to load from a web server?

Feel free to explain your full workflow in case I didn't get it right ;)

[hardfist]

@hardfist you mean that when you are generating the bundle-stats report with a baseline, the resulting html is 20MB and that is loading slow when trying to load from a web server?

Feel free to explain your full workflow in case I didn't get it right ;)

yes we use bundle-stats in our company to build things like relative-ci, which will send build stats json and build stats html to cdn for every Merge Request and generate bundle-stats diff for mr source branch and master branch, and it turns the diff page size is twice of webpack stats size, since it contains both source branch stats info and master branch stats info

[vio]

and it turns the diff page size is twice of webpack stats size, since it contains both source branch stats info and master branch stats info

Yes, the report is inlining the job data for the current report and baseline. The normalized data is usually larger than the webpack stats source and looks like, in your case, the modules size is almost 2x the source size. I think that's something that needs to be optimized, will try to have a look.

Also, the normalized data includes a copy of the webpack stats data (__INITIAL_DATA__[].rawData). Without breaking the current functionality, we can introduce a new option to skip the inlining of the source. Ran a test locally, and it decreased the size of the bundle-stats.html single report from 9.8MB to 6.3MB, for a baseline comparison, it will decrease from 20MB to 13MB. Is that something that will help you?

Since the document is html, i think is recommended to enable compression, the transfer size can decrease quite a lot:

zip bundle-stats.zip dist/bundle-stats.html                                                                                                                                                                   

> adding: dist/bundle-stats.html (deflated 92%)

[vio]

here is a way to optimize the webpack-stats JSON file to the minimum required (will not impact the size of bundle-stats html report): https://relative-ci.com/documentation/guides/webpack-stats/webpack-stats-plugin/#optimize-the-webpack-stats-json-file-size

[hardfist]

thank you for your suggestion, i will try it later

[vio]

@hardfist [email protected] is not embedding the raw data anymore. Based on my tests the file size of the report will decrease in your case from 20MB to 13MB.

Going to close the issue for now, but feel free to reopen it if you have any issues.