Code

Handle zero chunk size#264

Code

Closed

moshaad7 wants to merge 74 commits intov15.xblevesearch/zapx:v15.xfrom handle_zero_chunk_size_BACKPORTblevesearch/zapx:handle_zero_chunk_size_BACKPORTCopy head branch name to clipboard

+4,561-1,234

Commits

Commits on Sep 12, 2024

• Index sections (#165)

  Show description for 7b13491
  

  3 peopleauthored and

  moshaad7

  committed
  7b13491

  Copy full SHA for 7b13491

• memory leak fixes (#176)

  Show description for 7b302a5
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  7b302a5

  Copy full SHA for 7b302a5

• bug fix: reconstruct failing due to key not found (#177)

  Show description for c7c5cc5
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  c7c5cc5

  Copy full SHA for c7c5cc5

• Use latest of blevesearch/go-faiss (#178)

  Show description for b8c2117
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  b8c2117

  Copy full SHA for b8c2117

• MB-59569: Upgrade to blevesearch/[email protected] (#183)

  Show description for badde32
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  badde32

  Copy full SHA for badde32

• MB-59692: Respect choice of similarity metric, [email protected] (#182)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  bbe6b87

  Copy full SHA for bbe6b87

• optimization: avoiding storing of bitmaps when vector is in a single document (#181)

  Show description for 2ff712a
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  2ff712a

  Copy full SHA for 2ff712a

• bug fix: accounting the vecIDs approriately while merging (#179)

  Show description for 24c8620
  

  2 peopleauthored and

  moshaad7

  committed
  24c8620

  Copy full SHA for 24c8620

• replacing the flat index with the IVF family index. (#180)

  Show description for 225d0b0
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  225d0b0

  Copy full SHA for 225d0b0

• command line tool support for vector search (#175)

  Show description for cd8af4c
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  cd8af4c

  Copy full SHA for cd8af4c

• choosing the index types more responsibly (#186)

  Show description for c182fcb
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  c182fcb

  Copy full SHA for c182fcb

• fixing IO stats computation (#188)

  

  Thejas-bhat

  authored and

  moshaad7

  committed
  9be5eb5

  Copy full SHA for 9be5eb5

• bug fix: fixing the logic of tracking vector ids during a merge (#187)

  Show description for d176983
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  d176983

  Copy full SHA for d176983

• Resetting the fields of opaque for resusability (#189)

  Show description for c86b701
  

  2 peopleauthored and

  moshaad7

  committed
  c86b701

  Copy full SHA for c86b701

• Separate functions to read and search vector indexes. (#184)

  Show description for 6e8258a
  

  metonymic-smokey

  authored and

  moshaad7

  committed
  6e8258a

  Copy full SHA for 6e8258a

• Improve search path for multi kNN and more necessarily for single kNN (#190)

  Show description for e11ed23
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  e11ed23

  Copy full SHA for e11ed23

• Graceful exit when min arguments not provided for vector (#193)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  b9a5731

  Copy full SHA for b9a5731

• Beautify & Fix cmd line tooling output for vector content (#194)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  8d58766

  Copy full SHA for 8d58766

• MB-60068: Case when dimensionality of index and query vector mismatches (#195)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  d8031a6

  Copy full SHA for d8031a6

• MB-60071 - using the appropriate error value when the closeCh is closed (#196)

  Show description for 972e54b
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  972e54b

  Copy full SHA for 972e54b

• MB-60084: Change index merge procedure to account for existing index type. (#197)

  Show description for 13ff83d
  

  metonymic-smokey

  authored and

  moshaad7

  committed
  13ff83d

  Copy full SHA for 13ff83d

• MB-59447: support nested/chunked vectors (#185)

  Show description for 4c4c02d
  

  moshaad7

  and

  abhinavdangeti

  committed
  4c4c02d

  Copy full SHA for 4c4c02d

• MB-60071: fixing out of bound slice access panic (#198)

  Show description for 335bab0
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  335bab0

  Copy full SHA for 335bab0

• MB-60152: Panic fix while trying to fetch metadata of a vector index (#199)

  Show description for b31d5e8
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  b31d5e8

  Copy full SHA for b31d5e8

• TestVecPostingsIterator unit test fix (#200)

  Show description for 7a95724
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  7a95724

  Copy full SHA for 7a95724

• MB-60269 - Merge path fixes (#202)

  Show description for 71c6f00
  

  metonymic-smokey

  authored and

  moshaad7

  committed
  71c6f00

  Copy full SHA for 71c6f00

• MB-60029 - Changing the defaults for centroid count and nprobe. (#203)

  Show description for d96321b
  

  2 peopleauthored and

  moshaad7

  committed
  d96321b

  Copy full SHA for d96321b

• MB-60202 - Filtering deleted documents during FAISS vector index search (#205)

  Show description for fb96e93
  

  2 peopleauthored and

  moshaad7

  committed
  fb96e93

  Copy full SHA for fb96e93

• MB-60029 - Persisting faiss index optimization setting (#204)

  Show description for ec40d20
  

  2 peopleauthored and

  moshaad7

  committed
  ec40d20

  Copy full SHA for ec40d20

• MB-60367: Handle memory allocations more responsibly (#206)

  Show description for 9cb449c
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  9cb449c

  Copy full SHA for 9cb449c

• Upgrade to [email protected] + additional commentary (#207)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  478f2b8

  Copy full SHA for 478f2b8

• MB-60400: Reduce Garbage generated (#208)

  Show description for a592edf
  

  CascadingRadium

  authored and

  moshaad7

  committed
  a592edf

  Copy full SHA for a592edf

• Defer mergeBufferPool.Put to cover error exits (#211)

  Show description for f7c6ff1
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  f7c6ff1

  Copy full SHA for f7c6ff1

• Introducing a size API for vector index on query path (#212)

  Show description for a0ebb93
  

  2 peopleauthored and

  moshaad7

  committed
  a0ebb93

  Copy full SHA for a0ebb93

• Early exit condition for vector index merging (#213)

  Show description for 84ca6fe
  

  metonymic-smokey

  authored and

  moshaad7

  committed
  84ca6fe

  Copy full SHA for 84ca6fe

• MB-60662: Revert merge buffer pool (#214)

  

  CascadingRadium

  authored and

  moshaad7

  committed
  03e71e1

  Copy full SHA for 03e71e1

• Upgrade go-faiss to address a possible overflow during slice access (#215)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  a9ccd23

  Copy full SHA for a9ccd23

• MB-60367 Restructuring the code for better memory cleanup (#217)

  Show description for 5515ef0
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  5515ef0

  Copy full SHA for 5515ef0

• MB-60739: Faiss UP (#218)

  Show description for 06f9883
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  06f9883

  Copy full SHA for 06f9883

• MB-60781: Writing the vector section only when there are valid number of vectors in section (#219)

  Show description for 9d28635
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  9d28635

  Copy full SHA for 9d28635

• MB-60816: Upgrade go-faiss for fix (#220)

  Show description for 18bb93e
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  18bb93e

  Copy full SHA for 18bb93e

• Upgrade blevesearch/go-faiss (#221)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  a4265eb

  Copy full SHA for a4265eb

• MB-60662: revert doc id cache (#222)

  Show description for 96431e1
  

  CascadingRadium

  authored and

  moshaad7

  committed
  96431e1

  Copy full SHA for 96431e1

• Update zap.md to reflect v16 file format (#224)

  Show description for 8cea0fc
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  8cea0fc

  Copy full SHA for 8cea0fc

• MB-60914: Zap field specific metrics (#223)

  Show description for 9ec1077
  

  2 peopleauthored and

  moshaad7

  committed
  9ec1077

  Copy full SHA for 9ec1077

• Upgrade to RoaringBitmap/[email protected] (#227)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  40d804f

  Copy full SHA for 40d804f

• MB-60791: Avoid compute of vectorIDsToExclude on every search (#229)

  Show description for 579fb96
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  579fb96

  Copy full SHA for 579fb96

• MB-60943 - Add a coarse quantiser to the IVF indexes (#225)

  Show description for f72c611
  

  metonymic-smokey

  authored and

  moshaad7

  committed
  f72c611

  Copy full SHA for f72c611

• Revert "MB-60943 - Add a coarse quantiser to the IVF indexes (#225)" (#232)

  Show description for c4f4234
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  c4f4234

  Copy full SHA for c4f4234

• MB-61029: Deferring the closing of vector index (#226)

  Show description for 1beccd1
  

  3 peopleauthored and

  moshaad7

  committed
  1beccd1

  Copy full SHA for 1beccd1

• minor optimizations and bug fixes (#233)

  Show description for 0aa74ef
  

  CascadingRadium

  authored and

  moshaad7

  committed
  0aa74ef

  Copy full SHA for 0aa74ef

• MB-61029: Caching Vec To DocID Map (#231)

  Show description for 08feb6f
  

  2 peopleauthored and

  moshaad7

  committed
  08feb6f

  Copy full SHA for 08feb6f

• MB-61650: Upgrade blevesearch/go-faiss (#236)

  Show description for 6a955fa
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  6a955fa

  Copy full SHA for 6a955fa

• MB-60697: Upgrade blevesearch/go-faiss for windows fix (#238)

  Show description for 3d28f8e
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  3d28f8e

  Copy full SHA for 3d28f8e

• add map capacity (#237)

  

  CascadingRadium

  authored and

  moshaad7

  committed
  a8febb7

  Copy full SHA for a8febb7

• MB-60943 - Reduce number of centroids for IVF indexes. (#234)

  

  metonymic-smokey

  authored and

  moshaad7

  committed
  52342b1

  Copy full SHA for 52342b1

• Stop the ticker in vector cache (#240)

  Show description for d16a7e9
  

  CascadingRadium

  authored and

  moshaad7

  committed
  d16a7e9

  Copy full SHA for d16a7e9

• Move up to [email protected] & [email protected] (#242)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  9d26375

  Copy full SHA for 9d26375

• MB-59575: Handling memory mapped content more responsibly (#241)

  Show description for 164c638
  

  2 peopleauthored and

  moshaad7

  committed
  164c638

  Copy full SHA for 164c638

• bug fix: de-duplicating fieldsInv value (#239)

  Show description for f3fb79e
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  f3fb79e

  Copy full SHA for f3fb79e

• MB-60943 - Change quantiser when vector index is tuned for memory (#244)

  Show description for 0ec3ad3
  

  2 peopleauthored and

  moshaad7

  committed
  0ec3ad3

  Copy full SHA for 0ec3ad3

• MB-60943: Index optimization refactor memory -> memory-efficient (#245)

  Show description for 7860ad1
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  7860ad1

  Copy full SHA for 7860ad1

• MB-62167: Fix windows crash (#246)

  Show description for b6b0428
  

  CascadingRadium

  authored and

  moshaad7

  committed
  b6b0428

  Copy full SHA for b6b0428

• MB-61930: Use the go-faiss OpenMP API for setting the default number of OMP Threads to be 1 (#247)

  Show description for 2d9329f
  

  2 peopleauthored and

  moshaad7

  committed
  2d9329f

  Copy full SHA for 2d9329f

• MB-62221: Upgrade go-faiss (#248)

  Show description for ab311ca
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  ab311ca

  Copy full SHA for ab311ca

• MB-62221: Upgrade go-faiss to v1.0.19 (#249)

  Show description for 75dad85
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  75dad85

  Copy full SHA for 75dad85

• MB-61889: support search with params (#250)

  Show description for 2108c55
  

  moshaad7

  and

  abhinavdangeti

  committed
  2108c55

  Copy full SHA for 2108c55

• MB-62427: seeking the pointer correctly while reading segmentBase data (#252)

  Show description for af88577
  

  Thejas-bhat

  authored and

  moshaad7

  committed
  af88577

  Copy full SHA for af88577

• MB-61889: fix test cases (#253)

  moshaad7

  committed
  1b549a1

  Copy full SHA for 1b549a1

• MB-62354: Support Cosine Similarity (#251)

  Show description for 90e0ee6
  

  2 peopleauthored and

  moshaad7

  committed
  90e0ee6

  Copy full SHA for 90e0ee6

• MB-61889: Upgrade go-faiss for corner case fix (#260)

  Show description for e2e3b77
  

  abhinavdangeti

  authored and

  moshaad7

  committed
  e2e3b77

  Copy full SHA for e2e3b77

• MB-62230 - Support for pre-filtering with kNN (#255)

  Show description for 9049151
  

  2 peopleauthored and

  moshaad7

  committed
  9049151

  Copy full SHA for 9049151

• Fix unit test breakage from previous commit (API change) (#262)

  

  abhinavdangeti

  authored and

  moshaad7

  committed
  4d44092

  Copy full SHA for 4d44092

• Handle zero chunk size

  Show description for af6bef4
  moshaad7

  committed
  af6bef4

  Copy full SHA for af6bef4