java.lang.Object
org.apache.lucene.index.DocumentsWriterFlushControl
- All Implemented Interfaces:
Closeable
,AutoCloseable
,Accountable
This class controls
DocumentsWriterPerThread
flushing during indexing. It tracks the
memory consumption per DocumentsWriterPerThread
and uses a configured FlushPolicy
to decide if a DocumentsWriterPerThread
must flush.
In addition to the FlushPolicy
the flush control might set certain DocumentsWriterPerThread
as flush pending iff a DocumentsWriterPerThread
exceeds the
IndexWriterConfig.getRAMPerThreadHardLimitMB()
to prevent address space exhaustion.
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate long
private final Queue<DocumentsWriterPerThread>
private boolean
private final LiveIndexWriterConfig
private final DocumentsWriter
private boolean
private long
private final AtomicBoolean
private final List<DocumentsWriterPerThread>
private final FlushPolicy
private final Queue<DocumentsWriterPerThread>
private boolean
private boolean
private final long
private final InfoStream
private double
private int
private int
private long
private long
private long
private long
private final DocumentsWriterPerThreadPool
(package private) final DocumentsWriterStallControl
private long
Fields inherited from interface org.apache.lucene.util.Accountable
NULL_ACCOUNTABLE
-
Constructor Summary
ConstructorsConstructorDescriptionDocumentsWriterFlushControl
(DocumentsWriter documentsWriter, LiveIndexWriterConfig config) -
Method Summary
Modifier and TypeMethodDescription(package private) void
(package private) void
long
private void
addFlushingDWPT
(DocumentsWriterPerThread perThread) Returns an iterator that provides access to all currently activeDocumentsWriterPerThread
s(package private) boolean
Returnstrue
iff stalledprivate boolean
(package private) boolean
assertBlockedFlushes
(DocumentsWriterDeleteQueue flushingQueue) private boolean
private boolean
assertNumDocsSinceStalled
(boolean stalled) private DocumentsWriterPerThread
checkout
(DocumentsWriterPerThread perThread, boolean markPending) private void
checkoutAndBlock
(DocumentsWriterPerThread perThread) To be called only by the owner of this object's monitor lockprivate DocumentsWriterPerThread
checkOutForFlush
(DocumentsWriterPerThread perThread) (package private) final DocumentsWriterPerThread
Returns the largest non-pending flushable DWPT ornull
if there is none.void
close()
(package private) DocumentsWriterPerThread
doAfterDocument
(DocumentsWriterPerThread perThread, boolean isUpdate) (package private) void
(package private) void
doOnAbort
(DocumentsWriterPerThread perThread) (package private) void
(package private) DocumentsWriterPerThread
(package private) void
boolean
long
Returns heap bytes currently consumed by buffered deletes/updates that would be freed if we pushed all deletes.(package private) long
Returns theIndexWriter
InfoStream
(package private) long
(package private) long
(package private) boolean
Returnstrue
if a full flush is currently running(package private) long
(package private) long
netBytes()
(package private) DocumentsWriterPerThread
(package private) int
Returns the number of flushes that are checked out but not yet available for flushing.(package private) int
(package private) int
Returns the number of flushes that are already checked out but not yet actively flushing(package private) DocumentsWriterPerThread
private void
pruneBlockedQueue
(DocumentsWriterDeleteQueue flushingQueue) Prunes the blockedQueue by removing all DWPTs that are associated with the given flush queue.long
Return the memory usage of this object in bytes.void
void
setFlushPending
(DocumentsWriterPerThread perThread) Sets flush pending state on the givenDocumentsWriterPerThread
.private long
toString()
private boolean
updatePeaks
(long delta) private boolean
void
(package private) void
This method will block if too many DWPT are currently flushing and no checked out DWPT are availableMethods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
Field Details
-
hardMaxBytesPerDWPT
private final long hardMaxBytesPerDWPT -
activeBytes
private long activeBytes -
flushBytes
private volatile long flushBytes -
numPending
private volatile int numPending -
numDocsSinceStalled
private int numDocsSinceStalled -
flushDeletes
-
fullFlush
private boolean fullFlush -
fullFlushMarkDone
private boolean fullFlushMarkDone -
flushQueue
-
blockedFlushes
-
flushingWriters
-
maxConfiguredRamBuffer
private double maxConfiguredRamBuffer -
peakActiveBytes
private long peakActiveBytes -
peakFlushBytes
private long peakFlushBytes -
peakNetBytes
private long peakNetBytes -
peakDelta
private long peakDelta -
flushByRAMWasDisabled
private boolean flushByRAMWasDisabled -
stallControl
-
perThreadPool
-
flushPolicy
-
closed
private boolean closed -
documentsWriter
-
config
-
infoStream
-
stallStartNS
private long stallStartNS
-
-
Constructor Details
-
DocumentsWriterFlushControl
DocumentsWriterFlushControl(DocumentsWriter documentsWriter, LiveIndexWriterConfig config)
-
-
Method Details
-
activeBytes
public long activeBytes() -
getFlushingBytes
long getFlushingBytes() -
netBytes
long netBytes() -
stallLimitBytes
private long stallLimitBytes() -
assertMemory
private boolean assertMemory() -
updatePeaks
private boolean updatePeaks(long delta) -
doAfterDocument
-
checkout
-
assertNumDocsSinceStalled
private boolean assertNumDocsSinceStalled(boolean stalled) -
doAfterFlush
-
updateStallState
private boolean updateStallState() -
waitForFlush
public void waitForFlush() -
setFlushPending
Sets flush pending state on the givenDocumentsWriterPerThread
. TheDocumentsWriterPerThread
must have indexed at least on Document and must not be already pending. -
doOnAbort
-
checkoutAndBlock
To be called only by the owner of this object's monitor lock -
checkOutForFlush
-
addFlushingDWPT
-
toString
-
nextPendingFlush
DocumentsWriterPerThread nextPendingFlush() -
close
public void close()- Specified by:
close
in interfaceAutoCloseable
- Specified by:
close
in interfaceCloseable
-
allActiveWriters
Returns an iterator that provides access to all currently activeDocumentsWriterPerThread
s -
doOnDelete
void doOnDelete() -
getDeleteBytesUsed
public long getDeleteBytesUsed()Returns heap bytes currently consumed by buffered deletes/updates that would be freed if we pushed all deletes. This does not include bytes consumed by already pushed delete/update packets. -
ramBytesUsed
public long ramBytesUsed()Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-
numFlushingDWPT
int numFlushingDWPT() -
getAndResetApplyAllDeletes
public boolean getAndResetApplyAllDeletes() -
setApplyAllDeletes
public void setApplyAllDeletes() -
obtainAndLock
DocumentsWriterPerThread obtainAndLock() -
markForFullFlush
long markForFullFlush() -
assertActiveDeleteQueue
-
pruneBlockedQueue
Prunes the blockedQueue by removing all DWPTs that are associated with the given flush queue. -
finishFullFlush
void finishFullFlush() -
assertBlockedFlushes
-
abortFullFlushes
void abortFullFlushes() -
abortPendingFlushes
void abortPendingFlushes() -
isFullFlush
boolean isFullFlush()Returnstrue
if a full flush is currently running -
numQueuedFlushes
int numQueuedFlushes()Returns the number of flushes that are already checked out but not yet actively flushing -
numBlockedFlushes
int numBlockedFlushes()Returns the number of flushes that are checked out but not yet available for flushing. This only applies during a full flush if a DWPT needs flushing but must not be flushed until the full flush has finished. -
waitIfStalled
void waitIfStalled()This method will block if too many DWPT are currently flushing and no checked out DWPT are available -
anyStalledThreads
boolean anyStalledThreads()Returnstrue
iff stalled -
getInfoStream
Returns theIndexWriter
InfoStream
-
findLargestNonPendingWriter
DocumentsWriterPerThread findLargestNonPendingWriter() -
checkoutLargestNonPendingWriter
Returns the largest non-pending flushable DWPT ornull
if there is none. -
getPeakActiveBytes
long getPeakActiveBytes() -
getPeakNetBytes
long getPeakNetBytes()
-