Merge pull request #3 from askorama/ci/adds-nodejs-tests

Adds nodejs tests

.github/workflows/nodejs.yml

@@ -0,0 +1,31 @@
+name: Node.js Tests
+
+on:
+  push:
+    branches: [ main ]
+  pull_request:
+    branches: [ main ]
+
+jobs:
+  build-and-test:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Set up Deno
+      uses: denoland/setup-deno@v1
+
+    - name: Build with Deno
+      run: deno run -A dnt.ts
+
+    - name: Set up Node.js
+      uses: actions/setup-node@v3
+      with:
+        node-version: '20'
+
+    - name: Install Node.js dependencies
+      run: cp npm/package.json . && npm install
+
+    - name: Run tests with Node.js
+      run: node tests/node.js/nodejs.test.js

README.md

@@ -1,6 +1,7 @@
 # Orama Chunker
 
-[![Tests](https://github.com/oramasearch/chunker/actions/workflows/deno.yml/badge.svg)](https://github.com/oramasearch/chunker/actions/workflows/deno.yml)
+[![Node.js Tests](https://github.com/askorama/chunker/actions/workflows/nodejs.yml/badge.svg)](https://github.com/askorama/chunker/actions/workflows/nodejs.yml)
+[![Deno Tests](https://github.com/oramasearch/chunker/actions/workflows/deno.yml/badge.svg)](https://github.com/oramasearch/chunker/actions/workflows/deno.yml)
 
 When engaging with ChatGPT or other Large Language Models (LLMs), breaking down your input into smaller chunks is a strategy that significantly enhances the interaction experience. This approach is not just about managing the technical constraints of these models, such as input length limitations, but also about improving the quality of the dialogue. By dividing a complex query or detailed discussion into more digestible parts, users can guide the model through the conversation in a step-by-step manner. This method allows for a more nuanced understanding of the context and the specifics of each query, leading to responses that are not only accurate but also highly relevant to the user's needs.
 

deno.json

@@ -15,5 +15,8 @@
   },
   "lint": {
     "include": ["./src/**/*.ts", "./tests/**/*.ts"]
+  },
+  "test": {
+    "exclude": ["./tests/node.js/**/*"]
   }
 }

tests/node.js/nodejs.test.js

@@ -0,0 +1,39 @@
+const test = require('node:test')
+const { strict: assert } = require('node:assert')
+const { NLPChunker } = require('../../npm/script/nlp')
+const { FixedChunker } = require('../../npm/script/fixed')
+
+const longInput = `Steven Paul Jobs (February 24, 1955 – October 5, 2011) was an American businessman, inventor, and investor best known for co-founding the technology giant Apple Inc. Jobs was also the founder of NeXT and chairman and majority shareholder of Pixar. He was a pioneer of the personal computer revolution of the 1970s and 1980s, along with his early business partner and fellow Apple co-founder Steve Wozniak.
+
+Jobs was born in San Francisco in 1955 and adopted shortly afterwards. He attended Reed College in 1972 before withdrawing that same year. In 1974, he traveled through India, seeking enlightenment before later studying Zen Buddhism. He and Wozniak co-founded Apple in 1976 to further develop and sell Wozniak's Apple I personal computer. Together, the duo gained fame and wealth a year later with production and sale of the Apple II, one of the first highly successful mass-produced microcomputers. Jobs saw the commercial potential of the Xerox Alto in 1979, which was mouse-driven and had a graphical user interface (GUI). This led to the development of the unsuccessful Apple Lisa in 1983, followed by the breakthrough Macintosh in 1984, the first mass-produced computer with a GUI. The Macintosh introduced the desktop publishing industry in 1985 with the addition of the Apple LaserWriter, the first laser printer to feature vector graphics.
+
+In 1985, Jobs departed Apple after a long power struggle with the company's board and its then-CEO, John Sculley. That same year, Jobs took some Apple employees with him to found NeXT, a computer platform development company that specialized in computers for higher-education and business markets, serving as its CEO. In 1986, he helped develop the visual effects industry by funding the computer graphics division of Lucasfilm that eventually spun off independently as Pixar, which produced the first 3D computer-animated feature film Toy Story (1995) and became a leading animation studio, producing over 27 films since.
+`
+
+test('NLP chunker', async () => {
+  const chunker = new NLPChunker()
+  const chunks = await chunker.chunk(longInput, 128)
+
+  assert.strictEqual(chunks.length, 4)
+})
+
+test('NLP chunker should combine multiple sentences when their total number of token is less than a given threshold', async () => {
+  const input = `This is an example.
+Every sentence here has just a few tokens.
+  `
+
+  const chunker = new NLPChunker()
+  const singleChunk = await chunker.chunk(input, 50)
+  const multipleChunks = await chunker.chunk(input, 10)
+
+  assert.strictEqual(singleChunk.length, 1)
+  assert.strictEqual(multipleChunks.length, 2)
+})
+
+test('fixed chunker', async () => {
+  const chunker = new FixedChunker()
+  const chunks = await chunker.chunk(longInput, 128)
+  const inputTokens = await chunker.getNumberOfTokens(longInput)
+
+  assert.strictEqual(chunks.length, Math.ceil(inputTokens / 128))
+})