Module org.apache.lucene.core
Class Lucene99ScalarQuantizedVectorsFormat
java.lang.Object
org.apache.lucene.codecs.KnnVectorsFormat
org.apache.lucene.codecs.hnsw.FlatVectorsFormat
org.apache.lucene.codecs.lucene99.Lucene99ScalarQuantizedVectorsFormat
- All Implemented Interfaces:
NamedSPILoader.NamedSPI
Format supporting vector quantization, storage, and retrieval
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate static final int
(package private) final byte
(package private) final boolean
(package private) final Float
Controls the confidence interval used to scalar quantize the vectors the default value is calculated as `1-1/(vector_dimensions + 1)`static final float
Dynamic confidence interval(package private) final Lucene99ScalarQuantizedVectorScorer
private static final float
The maximum confidence interval(package private) static final String
(package private) static final String
private static final float
The minimum confidence intervalstatic final String
static final String
private static final FlatVectorsFormat
(package private) static final String
(package private) static final String
(package private) static final int
(package private) static final int
(package private) static final int
Fields inherited from class org.apache.lucene.codecs.KnnVectorsFormat
DEFAULT_MAX_DIMENSIONS, EMPTY
-
Constructor Summary
ConstructorsConstructorDescriptionConstructs a format using default graph construction parametersLucene99ScalarQuantizedVectorsFormat
(Float confidenceInterval, int bits, boolean compress) Constructs a format using the given graph construction parameters. -
Method Summary
Modifier and TypeMethodDescriptionstatic float
calculateDefaultConfidenceInterval
(int vectorDimension) fieldsReader
(SegmentReadState state) Returns aKnnVectorsReader
to read the vectors from the index.fieldsWriter
(SegmentWriteState state) Returns aFlatVectorsWriter
to write the vectors to the index.toString()
Methods inherited from class org.apache.lucene.codecs.hnsw.FlatVectorsFormat
getMaxDimensions
Methods inherited from class org.apache.lucene.codecs.KnnVectorsFormat
availableKnnVectorsFormats, forName, getName, reloadKnnVectorsFormat
-
Field Details
-
ALLOWED_BITS
private static final int ALLOWED_BITS- See Also:
-
QUANTIZED_VECTOR_COMPONENT
- See Also:
-
NAME
- See Also:
-
VERSION_START
static final int VERSION_START- See Also:
-
VERSION_ADD_BITS
static final int VERSION_ADD_BITS- See Also:
-
VERSION_CURRENT
static final int VERSION_CURRENT- See Also:
-
META_CODEC_NAME
- See Also:
-
VECTOR_DATA_CODEC_NAME
- See Also:
-
META_EXTENSION
- See Also:
-
VECTOR_DATA_EXTENSION
- See Also:
-
rawVectorFormat
-
MINIMUM_CONFIDENCE_INTERVAL
private static final float MINIMUM_CONFIDENCE_INTERVALThe minimum confidence interval- See Also:
-
MAXIMUM_CONFIDENCE_INTERVAL
private static final float MAXIMUM_CONFIDENCE_INTERVALThe maximum confidence interval- See Also:
-
DYNAMIC_CONFIDENCE_INTERVAL
public static final float DYNAMIC_CONFIDENCE_INTERVALDynamic confidence interval- See Also:
-
confidenceInterval
Controls the confidence interval used to scalar quantize the vectors the default value is calculated as `1-1/(vector_dimensions + 1)` -
bits
final byte bits -
compress
final boolean compress -
flatVectorScorer
-
-
Constructor Details
-
Lucene99ScalarQuantizedVectorsFormat
public Lucene99ScalarQuantizedVectorsFormat()Constructs a format using default graph construction parameters -
Lucene99ScalarQuantizedVectorsFormat
Constructs a format using the given graph construction parameters.- Parameters:
confidenceInterval
- the confidenceInterval for scalar quantizing the vectors, when `null` it is calculated based on the vector dimension. When `0`, the quantiles are dynamically determined by sampling many confidence intervals and determining the most accurate pair.bits
- the number of bits to use for scalar quantization (must be between 1 and 8, inclusive)compress
- whether to compress the quantized vectors by another 50% when bits=4. If `true`, pairs of (4 bit quantized) dimensions are packed into a single byte. This must be `false` when bits=7. This provides a trade-off of 50% reduction in hot vector memory usage during searching, at some decode speed penalty.
-
-
Method Details
-
calculateDefaultConfidenceInterval
public static float calculateDefaultConfidenceInterval(int vectorDimension) -
toString
-
fieldsWriter
Description copied from class:FlatVectorsFormat
Returns aFlatVectorsWriter
to write the vectors to the index.- Specified by:
fieldsWriter
in classFlatVectorsFormat
- Throws:
IOException
-
fieldsReader
Description copied from class:FlatVectorsFormat
Returns aKnnVectorsReader
to read the vectors from the index.- Specified by:
fieldsReader
in classFlatVectorsFormat
- Throws:
IOException
-