refactor: migrate to typescript, undici, and improve contributing #42

semoal · 2024-12-09T02:36:25Z

Hi guys, hope this pull-request suits you well

I've had a boring weekend and had time to give some love to this library, QuestDb is cool but db clients experience must be a requirement for you as a company, so since it's open-source I'm happy to collaborate with you 🤟

What's new?

From npm to pnpm for installing dependencies, much faster and stable, more common in JavaScript ecosystem lately.
Eslint 9.x with better defaults, also added Prettier to handle style-guide.
Migrate the entire source-code to Typescript, I've removed ALL anys and everything is type-safe.
Bunchee for compiling the library, this emits the dual-cjs approach that is complicated to generate, now we're compatible with esm and cjs without any problems, also the typings are auto-generated.
Migrated from Jest to Vitest, much more maintained, faster and has a lot of features that are super-useful for testing.
Upgraded JSDoc and regenerated the documentation site.
Added a CONTRIBUTING.md guide for making it easier to collaborate and introduce new features.

Breaking changes

Drop non-LTS version of Node, now we're compatible from Node 20 or greater, this is also tested on the matrix Github workflow testing suite.
http.Agent and http.Agents are cool, but latest Node version ship with a library called Undici which is by far better, easier to handle retry, faster, and more maintained and its core contributors are Node maintainers.

Performance improvements

Getting rid of http.agent and using Undici made that when I've installed the library in my project and run some benchmarks creating 2M rows in 64 iterations with a time limit of 5000ms of run we had a nice result:

┌─────────┬───────────┬────────────────────────┬───────────────────────────┬────────────────────────────┬───────────────────────────┬─────────┐
│ (index) │ Task name │ Latency average (ns)   │ Latency median (ns)       │ Throughput average (ops/s) │ Throughput median (ops/s) │ Samples │
├─────────┼───────────┼────────────────────────┼───────────────────────────┼────────────────────────────┼───────────────────────────┼─────────┤
│ 0       │ 'v4.0.0'  │ '54626798.72 ± 27.37%' │ '43224916.50 ± 17958.50'  │ '22 ± 3.08%'               │ '23'                      │ 106     │
│ 1       │ 'v3.0.0'  │ '63162652.05 ± 5.29%'  │ '59678416.50 ± 291166.50' │ '16 ± 2.81%'               │ '17'                      │ 80      │
└─────────┴───────────┴────────────────────────┴───────────────────────────┴────────────────────────────┴───────────────────────────┴─────────┘

Latency (average):
- v4.0.0 achieves a ~14% reduction in average latency, indicating faster performance.
Latency (median):
- v4.0.0 shows a significant improvement in median latency, reducing it by ~28%.
Throughput (average):
- Throughput increased by ~37.5% in v4.0.0.
Throughput (median):
- v4.0.0 delivers a ~35% improvement in median throughput.

Summary of Improvements

Substantial reduction in both average and median latency (~14% and ~28%, respectively).
Significant increase in both average and median throughput (~37.5% and ~35%, respectively).
Overall, v4.0.0 demonstrates superior performance and efficiency compared to v3.0.0.

These results confirm the improvements introduced in v4.0.0, making it a faster and more capable version.

CLAassistant · 2024-12-09T02:36:31Z

All committers have signed the CLA.

woss

Excellent PR!

I have left a few comments.

package.json

src/sender.ts

semoal · 2024-12-12T15:46:36Z

@puzpuzpuz @glasstiger a cool review here would be super nice:) sorry for pinging just want to move this forward

nwoolmer · 2024-12-13T13:06:45Z

Hi @semoal ,

Thanks for the contribution!

Do you know if this resolves your socket issue, or is this an ongoing problem?

#40

semoal · 2024-12-13T13:07:38Z

Hi @semoal ,

Thanks for the contribution!

Do you know if this resolves your socket issue, or is this an ongoing problem?

#40

Yes, it does. Undici handles way better the connection under the hood

puzpuzpuz · 2024-12-13T13:10:28Z

Wow, that's a biggish PR! We may need some time to review it, so please keep this in mind.

semoal · 2024-12-13T13:11:52Z

Wow, that's a biggish PR! We may need some time to review it, so please keep this in mind.

Yes, no hurry from my side, we forked internally and using it already in production, just wanted to ping to make sure that you're "subscribed" to this repo :)

semoal · 2024-12-18T15:00:37Z

To make you a real demo, I've reviewed some heavy load (it's bandwidth coming through our production QuestDB machine):

The initial peak of this graph generated the error socket: hang out without the new version, from 400K jobs easily 15k were failed because socket hang out
The latest peak, was handled with the new version (v4.0.0) , 0 errors, 90% cpu and handled without any problems the connection.

This demonstrates how well Undici manages connection pooling under the hood and how fast it is, easily managed a throughput of 2x or 3x, we were limited to run our asynchronous pipelines to 15 machines because this error, and with the new version we put 100 machines at the same time, and went amazingly good.

- Bumped versions of several devDependencies in package.json and pnpm-lock.yaml, including eslint, prettier, and testcontainers. - Enhanced the Sender class by introducing a flushPromiseChain to manage flush operations sequentially, ensuring data integrity during high load. - Refactored the HTTP request handling to improve error logging and response management. - Added new private methods for buffer management and data sending logic. - Updated tests to reflect changes in error messages and ensure proper functionality under high load conditions.

glasstiger · 2025-07-08T12:45:01Z

@semoal, sorry, it took a long long time to get to it, but I am reviewing your PR.

This PR could have been cut into 3-4 separate ones, tbh.
At least migrating to typescript, and the code reformatting could have been done on separate PRs, but it is ok.

glasstiger

In spite of the many comments I am approving this PR.
We were planning to port the codebase to typescript, and introducing undici as transport is a very good idea.

I will be raising multiple PRs to address the comments on this PR.

glasstiger · 2025-07-08T12:58:45Z

src/sender.ts

+    }
+
+    // Use toBufferView to get a reference, actual data copy for sending happens based on protocol needs
+    const dataView = this.toBufferView(dataAmountToSend);


Could have just used toBufferNew() which creates a copy of the buffer.
toBufferNew() originally compacted the buffer too, but that has been removed in this PR for some reason.
If we create a copy of the buffer for the actual sending task, we can just compact the buffer before we start sending.

This code also ignores the copy_buffer config option, which is kind of ok.
It is definitely not a popular one, so we could remove/deprecate it.

For more detailed explanation, please, see my comment on #44 (comment).

glasstiger · 2025-07-08T13:04:04Z

src/sender.ts

+   */
+  async flush(): Promise<boolean> {
+    // Add to the promise chain to ensure sequential execution
+    this.flushPromiseChain = this.flushPromiseChain


tbh, I am not convinced that promise chaining is needed.
all tests, including ingests all data without loss under high load with auto-flush, passes after removing it.

glasstiger · 2025-07-08T13:11:06Z

test/sender.test.ts

+
+  it("throws exception if the username or the token is missing when TCP transport is used", async function () {
+    try {
+      await Sender.fromConfig("tcp::addr=hostname;username=bobo;").close();


There was a fail(...) statement after this line, which has been removed.
This means that the test will pass no matter what.

There are lots of similar changes in this test suite, they need fixing.
Many fail() statements were removed/commented out.

glasstiger · 2025-07-08T13:13:00Z

test/sender.test.ts

+
+  it("uses default logger if log function is not set", async function () {
+    const sender = new Sender({ protocol: "http", host: "host" });
+    expect(JSON.stringify(sender.log)).toEqual(JSON.stringify(sender.log));


Another example of making tests always pass, regardless of breaking the code.
There are more occurrences of this type of change too.

glasstiger · 2025-07-08T13:14:15Z

test/sender.test.ts

+    expect(sender.agent[symbols[6]]).toEqual({ pipelining: 3 });
+
+    await sender.close();
+    agent.destroy();


missing await here?

glasstiger · 2025-07-08T13:40:09Z

src/sender.ts

+
+  /** @private */ http; // true if the protocol is HTTP/HTTPS, false if it is TCP/TCPS
+  /** @private */ secure; // true if the protocol is HTTPS or TCPS, false otherwise
+  /** @private */ host;


none of the Sender's fields have type, not sure why.

glasstiger · 2025-07-08T13:42:49Z

src/sender.ts

+import net from "node:net";
+import tls from "node:tls";
+import crypto from "node:crypto";
+import { Agent, RetryAgent } from "undici";


Instead of replacing the core http module with undici, it would be better to make it configurable which one to use.
Maybe we could even abstract the transport layer out of the sender, and create separate Transport interface implementations for TCP, HTTP core, and HTTP undici.

glasstiger · 2025-07-08T13:44:57Z

examples/basic.js

@@ -1,10 +1,14 @@
-const { Sender } = require('@questdb/nodejs-client')


We could add typescript examples too.

glasstiger · 2025-07-08T13:46:45Z

.github/workflows/build.yml


 jobs:
  test:
    name: Build with Node.js ${{ matrix.node-version }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        node-version: [16, latest]


It would be nice to keep compatibility with node 18 at least, but it is ok if we cannot.

glasstiger · 2025-07-08T13:49:19Z

.github/workflows/build.yml

@@ -2,16 +2,17 @@ name: build

 on:
  push:
-  schedule:
-    - cron: '15 2,10,18 * * *'


This build was scheduled to run 3 times a day to catch possible incompatibility issues with the server.
Would be good to keep it that way.
The server moves very fast, and it is good to see early on, if we broke compatibility with the clients.

refactor: migrate to ts, undici, and improve contributing

a07bdb4

semoal mentioned this pull request Dec 9, 2024

why is this library written in commonjs? #41

Closed

woss suggested changes Dec 9, 2024

View reviewed changes

package.json Show resolved Hide resolved

package.json Show resolved Hide resolved

src/sender.ts Show resolved Hide resolved

puzpuzpuz added the enhancement New feature or request label Dec 13, 2024

nwoolmer linked an issue Dec 13, 2024 that may be closed by this pull request

Error: socket hang up, with cpu at 90% #40

Closed

semoal mentioned this pull request May 9, 2025

Data loss if a lot of data is inserted simultaneously #44

Open

glasstiger approved these changes Jul 8, 2025

View reviewed changes

glasstiger merged commit 9018102 into questdb:main Jul 8, 2025
1 check passed

glasstiger mentioned this pull request Jul 9, 2025

chore(nodejs): refactor buffer handling #47

Open

		@@ -1,10 +1,14 @@
		const { Sender } = require('@questdb/nodejs-client')

refactor: migrate to typescript, undici, and improve contributing #42

refactor: migrate to typescript, undici, and improve contributing #42

Conversation

semoal commented Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's new?

Breaking changes

Performance improvements

Summary of Improvements

Uh oh!

CLAassistant commented Dec 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

woss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

semoal commented Dec 12, 2024

Uh oh!

nwoolmer commented Dec 13, 2024

Uh oh!

semoal commented Dec 13, 2024

Uh oh!

puzpuzpuz commented Dec 13, 2024

Uh oh!

semoal commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

semoal commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

glasstiger commented Jul 8, 2025

Uh oh!

glasstiger left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

glasstiger Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

semoal commented Dec 9, 2024 •

edited

Loading

CLAassistant commented Dec 9, 2024 •

edited

Loading

semoal commented Dec 13, 2024 •

edited

Loading

semoal commented Dec 18, 2024 •

edited

Loading

glasstiger Jul 8, 2025 •

edited

Loading