Add updateable random scorer interface for vector index building #14181

benwtrent · 2025-01-29T21:46:46Z

As stated by @ChrisHegarty and @msokolov the amount of garbage we create during vector index creation is pretty astounding.

This adjusts the interface to allow an "Updateable" random vector interface (@msokolov 's idea if I remember correctly) and refactors the usage to keep it threadsafe.

I still need to do some larger scale tests, I imagine the actual indexing times are not effected much.

I think I caught all the weird edge cases.

… building

lucene/codecs/src/java/org/apache/lucene/codecs/bitvectors/FlatBitVectorsScorer.java

msokolov

Thanks for tackling this! I have a few questions, possible suggestions for further improvement, but they shouldn't block anything.

msokolov · 2025-01-30T16:41:32Z

lucene/core/src/java/org/apache/lucene/codecs/hnsw/DefaultFlatVectorScorer.java

@@ -90,23 +91,29 @@ public String toString() {
  private static final class ByteScoringSupplier implements RandomVectorScorerSupplier {
    private final ByteVectorValues vectors;
    private final ByteVectorValues vectors1;
-    private final ByteVectorValues vectors2;
    private final VectorSimilarityFunction similarityFunction;

    private ByteScoringSupplier(
        ByteVectorValues vectors, VectorSimilarityFunction similarityFunction) throws IOException {
      this.vectors = vectors;
      vectors1 = vectors.copy();


since it is asymmetric now, instead of 1 vs 2, maybe rename to targetVectors or so?

I can happily rename everywhere. We use the same pattern many times.

msokolov · 2025-01-30T16:45:36Z

lucene/core/src/java/org/apache/lucene/util/hnsw/HnswConcurrentMergeBuilder.java

+      }
+      super.addGraphNode(node, scorer);
+    }
+
    @Override
    public void addGraphNode(int node) throws IOException {


curious if we still need this, and generally, if we are still using non-updateable RandomVectorScorer?

I was unsure about adjusting the full interface :/ There are things that inherit from HNSW graph, and I wanted it to "just work" without any fuss.

msokolov · 2025-01-30T18:34:32Z

lucene/core/src/java/org/apache/lucene/util/hnsw/HnswConcurrentMergeBuilder.java

    @Override
    public void addGraphNode(int node) throws IOException {
      if (initializedNodes != null && initializedNodes.get(node)) {
        return;
      }
-      super.addGraphNode(node);
+      if (scorer == null) {
+        scorer = scorerSupplier.scorer(node);


would it make sense to add this (maybe with node==0?) when creating the builder so it's never null

Let me see if I can adjust the interface a bit to ensure that we allow a "null" ordinal that simply initializes the buffer.

msokolov · 2025-01-30T18:35:11Z

lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java

    for (int node = minOrd; node < maxOrd; node++) {
-      addGraphNode(node);
+      if (scorer == null) {
+        scorer = scorerSupplier.scorer(node);


ditto here - these lazy inits are okay, but proactively initializing is better

msokolov · 2025-01-30T18:36:12Z

lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java

+       to the newly introduced levels (repeating step 2,3 for new levels) and again try to
+       promote the node to entry node.
+    */
+    UpdateableRandomVectorScorer scorer = scorerSupplier.scorer(node);


how did we get away with not doing this before? Is it that we pushed the supplier.scorer() call deeper in the call graph?

yeah, supplier got pushed all the way down everywhere. Creating new scorers during diversity checking, etc. So, a single node add would likely create many many short lived objects for no useful purpose.

msokolov · 2025-01-30T18:37:34Z

lucene/core/src/java/org/apache/lucene/util/hnsw/HnswGraphBuilder.java

@@ -463,7 +478,11 @@ private boolean connectComponents(int level) throws IOException {

          beam.clear();
          eps[0] = c0.start();
-          RandomVectorScorer scorer = scorerSupplier.scorer(c.start());
+          if (scorer == null) {


oh yes, here it is -- again could we initialize outside the loop, maybe even initialize to all zeros? Can ScorerSupplier accept null??

benwtrent · 2025-01-31T13:28:05Z

Ugh, I just tested recall over a larger data set with multiple indexing and merging threads. Recall is negatively effected by this change, indicating I have a silly bug somewhere.

msokolov · 2025-01-31T13:57:30Z

sorry, one more idea; please feel to go ahead with what you have, but I wonder, now that the RandomVectorScorer is updateable, do we even need to initialize it with an ord at all? Won't we generally speaking be updating it? If that were the case then we could change to scorer() and wouldn't need the null checks (it would always be the null case). Hmm but I guess this only applies to UpdateableRandomVectorScorer? But ... again -- do we have non-updateable instances? We shouldn't be afraid of making API changes on main at least

benwtrent · 2025-02-06T12:43:58Z

OK, benchmarks show pretty much no difference when I was testing on my machine. But, recall numbers all checkout and the API is nicer and makes more sense for vector merging.

So, I am gonna merge.

ChrisHegarty · 2025-02-06T16:16:54Z

Thank you @benwtrent - let's wait for the next lucene nightly. If no perf improvement, that's ok. There should be a lot less garbage created, and CPU devoted to cleaning young heap regions.

@ChrisHegarty

) As stated by @ChrisHegarty and @msokolov the amount of garbage we create during vector index creation is pretty astounding. This adjusts the interface to allow an "Updateable" random vector interface (@msokolov 's idea if I remember correctly) and refactors the usage to keep it threadsafe.

benwtrent added 2 commits January 29, 2025 16:28

Refactor the vector scorer interface to allow reuse during HNSW graph…

59dab18

… building

fixing scorers

5e2ae5f

benwtrent requested review from msokolov and ChrisHegarty January 29, 2025 21:46

benwtrent commented Jan 29, 2025

View reviewed changes

lucene/codecs/src/java/org/apache/lucene/codecs/bitvectors/FlatBitVectorsScorer.java Show resolved Hide resolved

iter

9b7fe53

msokolov approved these changes Jan 30, 2025

View reviewed changes

addressing pr comments

0d1de68

further refactoring

47efc99

benwtrent added this to the 10.2.0 milestone Feb 5, 2025

adding CHANGES

5d42a29

github-actions bot added module:core/codecs module:core/hnsw labels Feb 6, 2025

benwtrent merged commit 48884e7 into apache:main Feb 6, 2025
6 checks passed

benwtrent deleted the refactor/adj-scorer-reuse-hnsw-building branch February 6, 2025 16:12

benwtrent mentioned this pull request Feb 6, 2025

Evaluate adding a double addressing vector scorer #13966

Closed

benwtrent added a commit to elastic/elasticsearch that referenced this pull request Feb 13, 2025

Fixing RandomScorerSupplier after lucene change apache/lucene#14181

620a45f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add updateable random scorer interface for vector index building #14181

Add updateable random scorer interface for vector index building #14181

benwtrent commented Jan 29, 2025

msokolov left a comment

msokolov Jan 30, 2025

benwtrent Jan 30, 2025

msokolov Jan 30, 2025

benwtrent Jan 30, 2025

msokolov Jan 30, 2025

benwtrent Jan 30, 2025 •

edited

Loading

msokolov Jan 30, 2025

msokolov Jan 30, 2025

benwtrent Jan 30, 2025

msokolov Jan 30, 2025

benwtrent commented Jan 31, 2025

msokolov commented Jan 31, 2025 •

edited

Loading

benwtrent commented Feb 6, 2025

ChrisHegarty commented Feb 6, 2025

Add updateable random scorer interface for vector index building #14181

Add updateable random scorer interface for vector index building #14181

Conversation

benwtrent commented Jan 29, 2025

msokolov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benwtrent Jan 30, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benwtrent commented Jan 31, 2025

msokolov commented Jan 31, 2025 • edited Loading

benwtrent commented Feb 6, 2025

ChrisHegarty commented Feb 6, 2025

benwtrent Jan 30, 2025 •

edited

Loading

msokolov commented Jan 31, 2025 •

edited

Loading