Module org.apache.lucene.core
Class Lucene90CompressingStoredFieldsWriter
java.lang.Object
org.apache.lucene.codecs.StoredFieldsWriter
org.apache.lucene.codecs.lucene90.compressing.Lucene90CompressingStoredFieldsWriter
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Accountable
-
Nested Class Summary
Nested ClassesModifier and TypeClassDescriptionprivate static class
private static enum
Nested classes/interfaces inherited from class org.apache.lucene.codecs.StoredFieldsWriter
StoredFieldsWriter.MergeVisitor
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate final ByteBuffersDataOutput
(package private) static final boolean
(package private) static final String
(package private) static final int
private final int
private final CompressionMode
private Compressor
(package private) static final long
(package private) static final int
private int
private int[]
static final String
Extension of stored fields fileprivate IndexOutput
(package private) static final long
(package private) static final int
static final String
Codec name for the index.static final String
Extension of stored fields indexprivate FieldsIndexWriter
private final int
static final String
Extension of stored fields meta(package private) static final int
private IndexOutput
(package private) static final long
(package private) static final int
private int
private long
private long
private long
(package private) static final int
(package private) static final int
(package private) static final int
(package private) static final int
private int[]
private int
(package private) static final long
(package private) static final int
private final String
(package private) static final int
(package private) static final int
(package private) static final int
(package private) static final int
(package private) static final int
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
Constructor Summary
ConstructorsConstructorDescriptionLucene90CompressingStoredFieldsWriter
(Directory directory, SegmentInfo si, String segmentSuffix, IOContext context, String formatName, CompressionMode compressionMode, int chunkSize, int maxDocsPerChunk, int blockShift) Sole constructor. -
Method Summary
Modifier and TypeMethodDescriptionvoid
close()
private void
copyChunks
(MergeState mergeState, Lucene90CompressingStoredFieldsWriter.CompressingStoredFieldsMergeSub sub, int fromDocID, int toDocID) private void
copyOneDoc
(Lucene90CompressingStoredFieldsReader reader, int docID) void
finish
(int numDocs) Called beforeStoredFieldsWriter.close()
, passing in the number of documents that were written.void
Called when a document and all its fields have been added.private void
flush
(boolean force) getMergeStrategy
(MergeState mergeState, MatchingReaders matchingReaders, int readerIndex) int
merge
(MergeState mergeState) Merges in the stored fields from the readers inmergeState
.long
Return the memory usage of this object in bytes.private static void
saveInts
(int[] values, int length, DataOutput out) void
Called before writing the stored fields of the document.(package private) boolean
tooDirty
(Lucene90CompressingStoredFieldsReader candidate) Returns true if we should recompress this reader, even though we could bulk merge compressed dataprivate boolean
void
writeField
(FieldInfo info, IndexableField field) Writes a single stored field.private void
writeHeader
(int docBase, int numBufferedDocs, int[] numStoredFields, int[] lengths, boolean sliced, boolean dirtyChunk) (package private) static void
writeTLong
(DataOutput out, long l) Writes a long in a variable-length format.(package private) static void
writeZDouble
(DataOutput out, double d) Writes a float in a variable-length format.(package private) static void
writeZFloat
(DataOutput out, float f) Writes a float in a variable-length format.Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
Field Details
-
FIELDS_EXTENSION
Extension of stored fields file- See Also:
-
INDEX_EXTENSION
Extension of stored fields index- See Also:
-
META_EXTENSION
Extension of stored fields meta- See Also:
-
INDEX_CODEC_NAME
Codec name for the index.- See Also:
-
STRING
static final int STRING- See Also:
-
BYTE_ARR
static final int BYTE_ARR- See Also:
-
NUMERIC_INT
static final int NUMERIC_INT- See Also:
-
NUMERIC_FLOAT
static final int NUMERIC_FLOAT- See Also:
-
NUMERIC_LONG
static final int NUMERIC_LONG- See Also:
-
NUMERIC_DOUBLE
static final int NUMERIC_DOUBLE- See Also:
-
TYPE_BITS
static final int TYPE_BITS -
TYPE_MASK
static final int TYPE_MASK -
VERSION_START
static final int VERSION_START- See Also:
-
VERSION_CURRENT
static final int VERSION_CURRENT- See Also:
-
META_VERSION_START
static final int META_VERSION_START- See Also:
-
segment
-
indexWriter
-
metaStream
-
fieldsStream
-
compressor
-
compressionMode
-
chunkSize
private final int chunkSize -
maxDocsPerChunk
private final int maxDocsPerChunk -
bufferedDocs
-
numStoredFields
private int[] numStoredFields -
endOffsets
private int[] endOffsets -
docBase
private int docBase -
numBufferedDocs
private int numBufferedDocs -
numChunks
private long numChunks -
numDirtyChunks
private long numDirtyChunks -
numDirtyDocs
private long numDirtyDocs -
numStoredFieldsInDoc
private int numStoredFieldsInDoc -
NEGATIVE_ZERO_FLOAT
static final int NEGATIVE_ZERO_FLOAT -
NEGATIVE_ZERO_DOUBLE
static final long NEGATIVE_ZERO_DOUBLE -
SECOND
static final long SECOND- See Also:
-
HOUR
static final long HOUR- See Also:
-
DAY
static final long DAY- See Also:
-
SECOND_ENCODING
static final int SECOND_ENCODING- See Also:
-
HOUR_ENCODING
static final int HOUR_ENCODING- See Also:
-
DAY_ENCODING
static final int DAY_ENCODING- See Also:
-
BULK_MERGE_ENABLED_SYSPROP
-
BULK_MERGE_ENABLED
static final boolean BULK_MERGE_ENABLED
-
-
Constructor Details
-
Lucene90CompressingStoredFieldsWriter
Lucene90CompressingStoredFieldsWriter(Directory directory, SegmentInfo si, String segmentSuffix, IOContext context, String formatName, CompressionMode compressionMode, int chunkSize, int maxDocsPerChunk, int blockShift) throws IOException Sole constructor.- Throws:
IOException
-
-
Method Details
-
close
- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
- Specified by:
close
in classStoredFieldsWriter
- Throws:
IOException
-
startDocument
Description copied from class:StoredFieldsWriter
Called before writing the stored fields of the document.StoredFieldsWriter.writeField(FieldInfo, IndexableField)
will be called for each stored field. Note that this is called even if the document has no stored fields.- Specified by:
startDocument
in classStoredFieldsWriter
- Throws:
IOException
-
finishDocument
Description copied from class:StoredFieldsWriter
Called when a document and all its fields have been added.- Overrides:
finishDocument
in classStoredFieldsWriter
- Throws:
IOException
-
saveInts
- Throws:
IOException
-
writeHeader
private void writeHeader(int docBase, int numBufferedDocs, int[] numStoredFields, int[] lengths, boolean sliced, boolean dirtyChunk) throws IOException - Throws:
IOException
-
triggerFlush
private boolean triggerFlush() -
flush
- Throws:
IOException
-
writeField
Description copied from class:StoredFieldsWriter
Writes a single stored field.- Specified by:
writeField
in classStoredFieldsWriter
- Throws:
IOException
-
writeZFloat
Writes a float in a variable-length format. Writes between one and five bytes. Small integral values typically take fewer bytes.ZFloat --> Header, Bytes*?
- Header -->
Uint8
. When it is equal to 0xFF then the value is negative and stored in the next 4 bytes. Otherwise if the first bit is set then the other bits in the header encode the value plus one and no other bytes are read. Otherwise, the value is a positive float value whose first byte is the header, and 3 bytes need to be read to complete it. - Bytes --> Potential additional bytes to read depending on the header.
- Throws:
IOException
- Header -->
-
writeZDouble
Writes a float in a variable-length format. Writes between one and five bytes. Small integral values typically take fewer bytes.ZFloat --> Header, Bytes*?
- Header -->
Uint8
. When it is equal to 0xFF then the value is negative and stored in the next 8 bytes. When it is equal to 0xFE then the value is stored as a float in the next 4 bytes. Otherwise if the first bit is set then the other bits in the header encode the value plus one and no other bytes are read. Otherwise, the value is a positive float value whose first byte is the header, and 7 bytes need to be read to complete it. - Bytes --> Potential additional bytes to read depending on the header.
- Throws:
IOException
- Header -->
-
writeTLong
Writes a long in a variable-length format. Writes between one and ten bytes. Small values or values representing timestamps with day, hour or second precision typically require fewer bytes.ZLong --> Header, Bytes*?
- Header --> The first two bits indicate the compression scheme:
- 00 - uncompressed
- 01 - multiple of 1000 (second)
- 10 - multiple of 3600000 (hour)
- 11 - multiple of 86400000 (day)
zigzag-decode
it and finally multiply by the multiple corresponding to the compression scheme. - Bytes --> Potential additional bytes to read depending on the header.
- Throws:
IOException
- Header --> The first two bits indicate the compression scheme:
-
finish
Description copied from class:StoredFieldsWriter
Called beforeStoredFieldsWriter.close()
, passing in the number of documents that were written. Note that this is intentionally redundant (equivalent to the number of calls toStoredFieldsWriter.startDocument()
, but a Codec should check that this is the case to detect the JRE bug described in LUCENE-1282.- Specified by:
finish
in classStoredFieldsWriter
- Throws:
IOException
-
copyOneDoc
- Throws:
IOException
-
copyChunks
private void copyChunks(MergeState mergeState, Lucene90CompressingStoredFieldsWriter.CompressingStoredFieldsMergeSub sub, int fromDocID, int toDocID) throws IOException - Throws:
IOException
-
merge
Description copied from class:StoredFieldsWriter
Merges in the stored fields from the readers inmergeState
. The default implementation skips over deleted documents, and usesStoredFieldsWriter.startDocument()
,StoredFieldsWriter.writeField(FieldInfo, IndexableField)
, andStoredFieldsWriter.finish(int)
, returning the number of documents that were written. Implementations can override this method for more sophisticated merging (bulk-byte copying, etc).- Overrides:
merge
in classStoredFieldsWriter
- Throws:
IOException
-
tooDirty
Returns true if we should recompress this reader, even though we could bulk merge compressed dataThe last chunk written for a segment is typically incomplete, so without recompressing, in some worst-case situations (e.g. frequent reopen with tiny flushes), over time the compression ratio can degrade. This is a safety switch.
-
getMergeStrategy
private Lucene90CompressingStoredFieldsWriter.MergeStrategy getMergeStrategy(MergeState mergeState, MatchingReaders matchingReaders, int readerIndex) -
ramBytesUsed
public long ramBytesUsed()Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.
-