Module org.apache.lucene.core
Class Lucene99HnswVectorsWriter
java.lang.Object
org.apache.lucene.codecs.KnnVectorsWriter
org.apache.lucene.codecs.lucene99.Lucene99HnswVectorsWriter
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Accountable
Writes vector values and knn graphs to index segments.
-
Nested Class Summary
Nested ClassesNested classes/interfaces inherited from class org.apache.lucene.codecs.KnnVectorsWriter
KnnVectorsWriter.MergedVectorValues
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final int
private final List
<Lucene99HnswVectorsWriter.FieldWriter<?>> private boolean
private final FlatVectorsWriter
private final int
private final TaskExecutor
private final IndexOutput
private final int
private final SegmentWriteState
private static final long
private final IndexOutput
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
Constructor Summary
ConstructorsConstructorDescriptionLucene99HnswVectorsWriter
(SegmentWriteState state, int M, int beamWidth, FlatVectorsWriter flatVectorWriter, int numMergeWorkers, TaskExecutor mergeExec) -
Method Summary
Modifier and TypeMethodDescriptionAdd new field for indexingvoid
close()
private HnswGraphMerger
createGraphMerger
(FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier, TaskExecutor parallelMergeTaskExecutor, int numParallelMergeWorkers) (package private) static int
void
finish()
Called once at the end before closevoid
flush
(int maxDoc, Sorter.DocMap sortMap) Flush all buffered data on disk *void
mergeOneField
(FieldInfo fieldInfo, MergeState mergeState) Write field for merginglong
Return the memory usage of this object in bytes.private HnswGraph
reconstructAndWriteGraph
(OnHeapHnswGraph graph, int[] newToOldMap, int[] oldToNewMap, int[][] levelNodeOffsets) Reconstructs the graph given the old and new node ids.private void
reconstructAndWriteNeighbours
(NeighborArray neighbors, int[] oldToNewMap, int maxOrd) private void
writeField
(Lucene99HnswVectorsWriter.FieldWriter<?> fieldData) private int[][]
writeGraph
(OnHeapHnswGraph graph) private void
writeMeta
(FieldInfo field, long vectorIndexOffset, long vectorIndexLength, int count, HnswGraph graph, int[][] graphLevelNodeOffsets) private void
writeSortingField
(Lucene99HnswVectorsWriter.FieldWriter<?> fieldData, Sorter.DocMap sortMap) Methods inherited from class org.apache.lucene.codecs.KnnVectorsWriter
merge
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
Field Details
-
SHALLOW_RAM_BYTES_USED
private static final long SHALLOW_RAM_BYTES_USED -
segmentWriteState
-
meta
-
vectorIndex
-
M
private final int M -
beamWidth
private final int beamWidth -
flatVectorWriter
-
numMergeWorkers
private final int numMergeWorkers -
mergeExec
-
fields
-
finished
private boolean finished
-
-
Constructor Details
-
Lucene99HnswVectorsWriter
public Lucene99HnswVectorsWriter(SegmentWriteState state, int M, int beamWidth, FlatVectorsWriter flatVectorWriter, int numMergeWorkers, TaskExecutor mergeExec) throws IOException - Throws:
IOException
-
-
Method Details
-
addField
Description copied from class:KnnVectorsWriter
Add new field for indexing- Specified by:
addField
in classKnnVectorsWriter
- Throws:
IOException
-
flush
Description copied from class:KnnVectorsWriter
Flush all buffered data on disk *- Specified by:
flush
in classKnnVectorsWriter
- Throws:
IOException
-
finish
Description copied from class:KnnVectorsWriter
Called once at the end before close- Specified by:
finish
in classKnnVectorsWriter
- Throws:
IOException
-
ramBytesUsed
public long ramBytesUsed()Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal. -
writeField
- Throws:
IOException
-
writeSortingField
private void writeSortingField(Lucene99HnswVectorsWriter.FieldWriter<?> fieldData, Sorter.DocMap sortMap) throws IOException - Throws:
IOException
-
reconstructAndWriteGraph
private HnswGraph reconstructAndWriteGraph(OnHeapHnswGraph graph, int[] newToOldMap, int[] oldToNewMap, int[][] levelNodeOffsets) throws IOException Reconstructs the graph given the old and new node ids.Additionally, the graph node connections are written to the vectorIndex.
- Parameters:
graph
- The current on heap graphnewToOldMap
- the new node ids indexed to the old node idsoldToNewMap
- the old node ids indexed to the new node idslevelNodeOffsets
- where to place the new offsets for the nodes in the vector index.- Returns:
- The graph
- Throws:
IOException
- if writing to vectorIndex fails
-
reconstructAndWriteNeighbours
private void reconstructAndWriteNeighbours(NeighborArray neighbors, int[] oldToNewMap, int maxOrd) throws IOException - Throws:
IOException
-
mergeOneField
Description copied from class:KnnVectorsWriter
Write field for merging- Overrides:
mergeOneField
in classKnnVectorsWriter
- Throws:
IOException
-
writeGraph
- Parameters:
graph
- Write the graph in a compressed format- Returns:
- The non-cumulative offsets for the nodes. Should be used to create cumulative offsets.
- Throws:
IOException
- if writing to vectorIndex fails
-
writeMeta
private void writeMeta(FieldInfo field, long vectorIndexOffset, long vectorIndexLength, int count, HnswGraph graph, int[][] graphLevelNodeOffsets) throws IOException - Throws:
IOException
-
createGraphMerger
private HnswGraphMerger createGraphMerger(FieldInfo fieldInfo, RandomVectorScorerSupplier scorerSupplier, TaskExecutor parallelMergeTaskExecutor, int numParallelMergeWorkers) -
close
- Throws:
IOException
-
distFuncToOrd
-