feat(web): add web generator #285

avivkeller · 2025-05-28T21:53:14Z

Fixes #7.

This PR adds the web generator.

Tasks / Issues

P1 – Must Complete Before Merge

Add more items (anyone can do this as they review¹)

P2 – Must complete before migration

ChangeHistory component doesn’t render Markdown correctly. (See entry.changes aren't converted to HTML #326)
Add more items (anyone can do this as they review¹)

P3 – Can Be Done in a Follow-up

Remove mustache dependency
Add more items (anyone can do this as they review¹)

Get a preview

node bin/cli.mjs generate -t orama-db -i "/node/doc/api/*.md" -o "./out" # If you want search functionality
node bin/cli.mjs generate -t web -i "/node/doc/api/*.md" -o "./out" --index "/node/doc/api/index.md" # Specifying `--index` is optional, but recommended
npx serve out # You can serve the output, or just open one of the files in your browser. Serving is required for using search.

Add things as they appear, or leave review comments. ↩ ↩² ↩³

codecov-commenter · 2025-05-28T21:53:59Z

Codecov Report

Attention: Patch coverage is 60.60831% with 531 lines in your changes missing coverage. Please review.

Project coverage is 70.69%. Comparing base (9e3ba2f) to head (4274e9e).
Report is 2 commits behind head on main.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...nerators/jsx-ast/utils/createSignatureElements.mjs	27.39%	167 Missing ⚠️
src/generators/web/index.mjs	42.40%	72 Missing ⚠️
src/generators/jsx-ast/utils/buildContent.mjs	46.61%	63 Missing ⚠️
src/generators/web/build/generate.mjs	23.61%	55 Missing ⚠️
src/generators/web/build/bundle.mjs	34.17%	52 Missing ⚠️
src/generators/jsx-ast/index.mjs	26.00%	37 Missing ⚠️
src/generators/jsx-ast/utils/buildBarProps.mjs	42.00%	28 Missing and 1 partial ⚠️
src/generators/orama-db/index.mjs	54.34%	21 Missing ⚠️
src/utils/queries/index.mjs	71.73%	10 Missing and 3 partials ⚠️
src/generators/web/build/plugins.mjs	89.15%	9 Missing ⚠️
... and 7 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #285      +/-   ##
==========================================
- Coverage   71.52%   70.69%   -0.84%     
==========================================
  Files         117      129      +12     
  Lines        9719    10756    +1037     
  Branches      590      647      +57     
==========================================
+ Hits         6952     7604     +652     
- Misses       2764     3145     +381     
- Partials        3        7       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

src/generators/web/components/SearchBox/constants.mjs

AugustinMauroy · 2025-05-29T18:42:20Z

src/generators/web/components/CodeBox.jsx

+  const language = matches?.groups?.language ?? '';
+  const [copyText, setCopyText] = useState('Copy to clipboard');
+
+  const handleCopy = async text => {


Why not including useCopyToClipord in ui lib ?

IIRC I didn't include it since it didn't fall under the "UI Component" category, it's a hook, however, if you disagree, I won't block.

src/generators/legacy-html/utils/tableOfContents.mjs

src/generators/web/components/NavBar.jsx

ovflowd · 2025-05-29T19:51:14Z

package.json

    "estree-util-value-to-estree": "^3.4.0",
    "estree-util-visit": "^2.0.0",
    "github-slugger": "^2.0.0",
    "glob": "^11.0.2",
    "hast-util-to-string": "^3.0.1",
    "hastscript": "^9.0.1",
    "html-minifier-terser": "^7.2.0",
+    "mustache": "^4.2.0",


I tried to use __<VARIABLE>__ like in the other generators, however, the JavaScript I was trying to add to the HTML broke the structure of the file. Using a library like Mustache properly escapes anything injected.

export const TEMPLATE = ({...props}) => dedent` <html> ${props.foo} </html> `;

this can work

Sorry if I wasn’t clear, the compiled JavaScript broke the HTML structure because it contained characters that need to be escaped to be in an HTML script.

For example, in your scenario, if props.foo contains a </html>, it’ll escape the sandbox.

Mustache performs all the needed escaping.

I don't like that we're using yet another layer of whatever freakery here. Mustache is great, but it feels like you're throing a nuke at an ant's sized problem. Not to mention adding yet another engine here will make things even more complex and costly-performance wise. Let's try to avoid throwing npm packages to any problem we encounter, shall we?

avivkeller · 2025-05-30T14:54:47Z

🎉 The code now dehydrates to the client so it can render without JavaScript!

AugustinMauroy · 2025-05-30T15:28:54Z

🎉 The code now dehydrates to the client so it run without JavaScript!

Wow 😵‍💫 and what about codetab

avivkeller · 2025-05-30T15:43:36Z

It rehydrates and runs with JS, but if you don't have JS, you can still view the docs. I used React's SSRing

src/constants.mjs

avivkeller · 2025-05-31T21:54:29Z

@AugustinMauroy and I got search to finally work 🎉

src/generators/web/client/components/MetaBar.jsx

src/generators/web/index.mjs

src/generators/web/client/components/MetaBar.jsx

src/generators/web/client/components/SearchBox/index.jsx

avivkeller · 2025-06-03T23:05:06Z

@nodejs/nodejs-website @nodejs/web-infra ChangeHistory and SideBar aren't implemented yet, so this is still a draft, but it's ready for you to take a look, so feel free to review :-)

avivkeller · 2025-06-29T20:52:07Z

On my machine, performance tanked, I need to investigate.

ovflowd · 2025-06-29T21:01:41Z

Noticed the same.

ovflowd · 2025-06-29T21:10:10Z

I wonder if for these sort of properties (such as an object), we can do something like

(Just a rough idea) but allows better understanding that these are pieces of options?

Also I noticed what's wrong, the text on the table is 14px but the <code> is 16px when it should also be 14px!

avivkeller · 2025-06-29T21:15:15Z

I actually think the perf regression is a bug in tailwind. Maybe the patch they just released contained a bug.

avivkeller · 2025-06-29T21:24:24Z

Indeed. Downgrading to @tailwind/[email protected] resolves the perf regression

avivkeller · 2025-06-29T21:47:28Z

Yes, but that's not the <code> styling, these elements are written with ffff we could sanitize something or change the font for small code blocks, or the antialising because it looks quite weird and contrasting the table/theme

I've adjusted the code tags to inherit their size.

Orama also supports tags and labels

We could add a "tags" in the YAML metadata in nodejs/node?

I wonder if for these sort of properties (such as an object), we can do something like

Done.

BTW, have we integrated i18n on the new web gen?

#280

ovflowd · 2025-06-29T22:00:53Z

We could add a "tags" in the YAML metadata in nodejs/node?

too much upstream work. -1

ovflowd · 2025-06-29T22:04:23Z

Is it still supposed to take this much?:

node bin/cli.mjs generate -t web -i "../node/doc/api/*.md" -o "./out" --index  106.07s user 34.23s system 293% cpu 47.827 total

ovflowd · 2025-06-29T22:05:25Z

I feel we need to fix styles upstream on the Table component?

(missing borders, and maybe making the paddings be equally correct top/bottom)

ovflowd · 2025-06-29T22:06:56Z

Also, OOC: Do we need line numbers on these that are from the definitions?

ovflowd · 2025-06-29T22:08:11Z

Also missing Description column for top level items?

I also think when there's no description can we either write N/A or just -

avivkeller · 2025-06-30T01:00:55Z

Is it still supposed to take this much?:

106.07s isn't great, we have several improvements planned, including:

Using a faster minifier
Precompiling ui-components (So no custom ESBuild resolver, PostCSS, or Tailwind)

I feel we need to fix styles upstream on the Table component?
(missing borders, and maybe making the paddings be equally correct top/bottom)

nodejs/nodejs.org#7911

Also, OOC: Do we need line numbers on these that are from the definitions?

No, I can update the CSS to remove them.

Also missing Description column for top level items?

It's not a bug, if there aren't any descriptions on any of the entries, the column is omitted. Ditto for name and type. However, perhaps we should give the subtable headings if it has different columns than the parent table.

avivkeller · 2025-06-30T01:07:38Z

We are also Promise.all-ing the generation, which, while faster, can be very intensive for the host machine, so I want make that use a for loop, like the legacy generator.

It prevents memory crashes from happening, but it exponentially slower, so I really need to improve performance.

(We can also p-limit the Promise.all, what what would the limit be? --threads is for the amount of workers, should it also dictate how many promises the worker can spawn?)

ovflowd · 2025-06-30T01:08:43Z

It's not a bug, if there aren't any descriptions on any of the entries, the column is omitted. Ditto for name and type. However, perhaps we should give the subtable headings if it has different columns than the parent table.

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

ovflowd · 2025-06-30T01:09:46Z

We are also Promise.all-ing the generation, which, while faster, can be very intensive for the host machine, so I want make that use a for loop, like the legacy generator.

It prevents memory crashes from happening, but it exponentially slower, so I really need to improve performance.

(We can also p-limit the Promise.all, what what would the limit be? --threads is for the amount of workers, should it also dictate how many promises the worker can spawn?)

Let's think of that if all other performance improvements aren't enough

avivkeller · 2025-06-30T01:10:53Z

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

I feel like there are cases where that's not beneficial, like

Let's think of that if all other performance improvements aren't enough

That (being using Promise.all) is what we are using right now... with the 100s+ execution time 😬

ovflowd · 2025-06-30T10:00:20Z

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

I feel like there are cases where that's not beneficial, like

Let's think of that if all other performance improvements aren't enough

That (being using Promise.all) is what we are using right now... with the 100s+ execution time 😬

I feel that we should always have the same columns, to make it consistent. Otherwise it might be confusing. Simply set the values to said empty values as N/A or - I'm 100% inclined into consistency here (please 🙇)

ovflowd · 2025-06-30T10:43:06Z

I think we should always render all columns .. But yeah, I also think suitable headings make sense. At the very least suitable headings and fixed paddings

I feel like there are cases where that's not beneficial, like

Let's think of that if all other performance improvements aren't enough

That (being using Promise.all) is what we are using right now... with the 100s+ execution time 😬

I feel that we should always have the same columns, to make it consistent. Otherwise it might be confusing. Simply set the values to said empty values as N/A or - I'm 100% inclined into consistency here (please 🙇)

Even more as just this Type , doesn't tell anything to the users. Is this a parameter? Return type? Argument of the method? Also where's the snippet on that example?

avivkeller · 2025-06-30T12:07:27Z

Even more as just this Type, doesn't tell anything to the users. Is this a parameter? Return type? Argument of the method? Also where's the snippet on that example?

That example was generated from the following markdown:

#### `dir.path`

<!-- YAML
added: v12.12.0
-->

* {string}

The read-only path of this directory as was provided to [`fs.opendir()`][],
[`fs.opendirSync()`][], or [`fsPromises.opendir()`][].

Maybe it's just me, but it feels weird to have the table below when only Type is specified:

Property	Type	Description
-	`<string>`	-

avivkeller · 2025-07-01T00:36:18Z

Moving to draft until the following performance improvements are completed (dynamic list, feel free to add more items):

PostCSS / Tailwind removed¹
ESBuild replaced with Rolldown¹² (Speed comparison: https://x.com/youyuxi/status/1869608132386922720)
Externalize Server-side dependencies¹
Hast compiled directly to Babel OR Babel replaced with ESTree alternative³
- Fix: Hast -> String -> Transforms -> Compile (Rather than Hast -> ESTree -> Babel -> Transforms -> String -> Compile)
OOM Crashes on less powerful machines resolved⁴

Depends on the precompilation of @node-core/ui-components. ↩ ↩² ↩³
Ideally, we'd use a pluginless implementation due to the removal of PostCSS, making everything even faster. ↩
Currently, we MDX -> ESTree -> Babel transforms -> Text. I would like to either MDX -> ESTree -> Transforms -> Text OR MDX -> ESTree -> Text -> String Transforms. Although, perhaps MDX -> String -> Transforms will be speedy? ↩
My A2179 crashes (OOM) when generating doc/api/*.md, but maybe it's just me? Our perf improvements should resolve this. ↩

allevo · 2025-07-01T07:39:04Z

src/generators/orama-db/constants.mjs

+  title: 'string',
+  description: 'string',
+  path: 'string',
+};


Hi!
This schema is the one used to display the results. You can add other properties that Orama will search for. This will help add metadata like tags and similar to all documents.

export const SCHEMA = { title: 'string', description: 'string', path: 'string', tags: 'string[]', // or 'string' };

@ovflowd WDYT about this enhancement?

NB: You can also play with the "boost" parameter. From my experience, boosting the title and similar fields is helpful because people commonly want to see first the title-matching result and the rest later.

cc @avivkeller; Thank you, @allevo!

Sounds good to me, but we'd need to add tags / whatnot to the upstream docs, or hardcode them here. I feel like, while more work, upstreaming the tags aligns more with #332.

Let's at least configure the boost parameter. We could add these "tags" as part of the YAML schema (for the time being)

How should we define what to "boost"? A hardcoded list?

avivkeller · 2025-07-03T20:47:48Z

Everything shown in this image is PostCSS / Tailwind. That's roughly ~90s (on my machine). Once @Harkunwar finishes that (thx btw), we should be able to get this to run at a fairly decent speed.

avivkeller marked this pull request as ready for review May 28, 2025 22:13

avivkeller requested a review from a team as a code owner May 28, 2025 22:13

avivkeller commented May 29, 2025

View reviewed changes

src/generators/web/components/SearchBox/constants.mjs Outdated Show resolved Hide resolved

avivkeller marked this pull request as draft May 29, 2025 16:36

AugustinMauroy reviewed May 29, 2025

View reviewed changes

ovflowd reviewed May 29, 2025

View reviewed changes

avivkeller force-pushed the feat/web/gen branch from 1eb144a to 492a696 Compare May 29, 2025 20:30

avivkeller force-pushed the feat/web/gen branch from d8ab24c to 9ebcb95 Compare May 30, 2025 14:57

avivkeller mentioned this pull request May 30, 2025

chore(ui): simplify CodeBox-like UI components nodejs/nodejs.org#7793

Merged

avivkeller added the enhancement New feature or request label May 30, 2025

avivkeller added this to Node.js API Documentation Tooling May 31, 2025

avivkeller moved this to In Progress in Node.js API Documentation Tooling May 31, 2025

avivkeller removed this from Node.js API Documentation Tooling May 31, 2025

AugustinMauroy reviewed May 31, 2025

View reviewed changes

src/constants.mjs Outdated Show resolved Hide resolved

flakey5 reviewed May 31, 2025

View reviewed changes

src/generators/web/client/components/MetaBar.jsx Outdated Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch from 705f9e2 to 76ef14a Compare May 31, 2025 22:18

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/index.mjs Show resolved Hide resolved

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/client/components/MetaBar.jsx Outdated Show resolved Hide resolved

AugustinMauroy reviewed Jun 1, 2025

View reviewed changes

src/generators/web/client/components/SearchBox/index.jsx Outdated Show resolved Hide resolved

avivkeller force-pushed the feat/web/gen branch 2 times, most recently from 024bbea to dbfe55d Compare June 3, 2025 21:40

avivkeller force-pushed the feat/web/gen branch from e814300 to ba07721 Compare June 5, 2025 20:27

avivkeller linked an issue Jun 6, 2025 that may be closed by this pull request

Write React Components #10

Closed

avivkeller force-pushed the feat/web/gen branch from ca72ed0 to 99176ba Compare June 6, 2025 20:33

avivkeller force-pushed the feat/web/gen branch from 0be0b84 to a38b047 Compare June 29, 2025 21:57

avivkeller force-pushed the feat/web/gen branch from a38b047 to f621054 Compare June 30, 2025 12:20

avivkeller marked this pull request as draft July 1, 2025 00:34

allevo reviewed Jul 1, 2025

View reviewed changes

feat(web): add web generator

76bbf42

avivkeller force-pushed the feat/web/gen branch from 6869a9f to 76bbf42 Compare July 3, 2025 15:15

avivkeller added 3 commits July 3, 2025 11:17

fix test

ebbdc6c

fixup!

ff28fe3

externalize preact

4274e9e

feat(web): add web generator #285

Are you sure you want to change the base?

feat(web): add web generator #285

Uh oh!

Conversation

avivkeller commented May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks / Issues

P1 – Must Complete Before Merge

P2 – Must complete before migration

P3 – Can Be Done in a Follow-up

Get a preview

Footnotes

Uh oh!

codecov-commenter commented May 28, 2025 • edited by codecov bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AugustinMauroy May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

avivkeller May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

avivkeller commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AugustinMauroy commented May 30, 2025

Uh oh!

avivkeller commented May 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

avivkeller commented May 31, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

avivkeller commented Jun 3, 2025

Uh oh!

avivkeller commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avivkeller commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avivkeller commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

avivkeller commented Jun 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025

Uh oh!

ovflowd commented Jun 29, 2025

avivkeller commented May 28, 2025 •

edited

Loading

codecov-commenter commented May 28, 2025 •

edited by codecov bot

Loading

AugustinMauroy May 29, 2025 •

edited

Loading

avivkeller May 30, 2025 •

edited

Loading

avivkeller commented May 30, 2025 •

edited

Loading

avivkeller commented May 30, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

ovflowd commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 29, 2025 •

edited

Loading

avivkeller commented Jun 30, 2025 •

edited

Loading

avivkeller commented Jun 30, 2025 •

edited

Loading

avivkeller commented Jul 1, 2025 •

edited

Loading