Package edu.berkeley.nlp.lm.values
Class CountValueContainer
java.lang.Object
edu.berkeley.nlp.lm.values.CountValueContainer
- All Implemented Interfaces:
CompressibleValueContainer<LongRef>
,ValueContainer<LongRef>
,Serializable
- See Also:
-
Field Summary
FieldsModifier and TypeFieldDescriptionprotected final int
protected final long[]
protected final boolean
protected final int[]
protected boolean
protected final VariableLengthBitCompressor
protected final int
protected final CustomWidthArray[]
protected int
-
Constructor Summary
ConstructorsConstructorDescriptionCountValueContainer
(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder) -
Method Summary
Modifier and TypeMethodDescriptionboolean
add
(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, LongRef val_, long suffixOffset, boolean ngramIsNew) Adds a new value at the specified offset.void
clearStorageAfterCompression
(int ngramOrder) void
clearStorageForOrder
(int ngramOrder) createFreshValues
(long[] numNgramsForEachOrder_) Creates a fresh value container for copying purposes.void
decompress
(BitStream bits, int ngramOrder, boolean justConsume, LongRef outputVal) Reads and decompresses from the bit stream bits.getCompressed
(long offset, int ngramOrder) Compresses the value at the given offset into a list of bits.final long
getCount
(int ngramOrder, long index) protected long
getCountRank
(long val) protected LongRef
void
getFromOffset
(long index, int ngramOrder, LongRef outputVal) Gets the value living at a particular offset.protected void
getFromRank
(long rank, LongRef outputVal) protected long
getRank
(int ngramOrder, long offset) Creates a fresh value of object (useful for passing as an output parameter)long
getSuffixOffset
(long index, int ngramOrder) protected int
getSuffixOffsetFromInternalVal
(long internalVal) long
int
numValueBits
(int ngramOrder) void
Destructively sets internal storage from another object.void
Initializes a value container with the map that contains itvoid
setSizeAtLeast
(long size, int ngramOrder) Sets internal storage for size for a particular n-gram orderboolean
void
swap
(long a, long b, int ngramOrder) Swaps values at offsets a and b.void
trim()
Final clean up of storage.void
trimAfterNgram
(int ngramOrder, long size) Clear storage after an n-gram order is completeprotected boolean
-
Field Details
-
valueRanks
-
storeSuffixIndexes
protected final boolean storeSuffixIndexes -
valueCoder
-
valueRadix
protected final int valueRadix -
valueWidth
protected int valueWidth -
defaultValRank
protected final int defaultValRank- See Also:
-
numNgramsForEachOrder
protected final long[] numNgramsForEachOrder -
suffixBitsForOrder
protected final int[] suffixBitsForOrder -
useMapValueArray
protected boolean useMapValueArray
-
-
Constructor Details
-
CountValueContainer
public CountValueContainer(LongToIntHashMap countCounter, int valueRadix, boolean storePrefixes, long[] numNgramsForEachOrder)
-
-
Method Details
-
createFreshValues
Description copied from interface:ValueContainer
Creates a fresh value container for copying purposes.- Returns:
-
getFromOffset
Description copied from interface:ValueContainer
Gets the value living at a particular offset.- Parameters:
index
-ngramOrder
-
-
getFromRank
-
getCount
public final long getCount(int ngramOrder, long index) -
getDefaultVal
-
trimAfterNgram
public void trimAfterNgram(int ngramOrder, long size) Description copied from interface:ValueContainer
Clear storage after an n-gram order is complete- Specified by:
trimAfterNgram
in interfaceValueContainer<LongRef>
- Parameters:
ngramOrder
-size
-
-
getUnigramSum
public long getUnigramSum() -
getScratchValue
Description copied from interface:ValueContainer
Creates a fresh value of object (useful for passing as an output parameter)- Returns:
-
setFromOtherValues
Description copied from interface:ValueContainer
Destructively sets internal storage from another object.- Specified by:
setFromOtherValues
in interfaceValueContainer<LongRef>
- Parameters:
o
-
-
trim
public void trim()Description copied from interface:ValueContainer
Final clean up of storage.- Specified by:
trim
in interfaceValueContainer<LongRef>
-
getCountRank
protected long getCountRank(long val) -
setMap
Description copied from interface:ValueContainer
Initializes a value container with the map that contains it- Specified by:
setMap
in interfaceValueContainer<V extends LongRepresentable<V>>
-
useValueStoringArray
protected boolean useValueStoringArray() -
swap
public void swap(long a, long b, int ngramOrder) Description copied from interface:CompressibleValueContainer
Swaps values at offsets a and b.- Specified by:
swap
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
- Parameters:
a
-b
-ngramOrder
-
-
add
public boolean add(int[] ngram, int startPos, int endPos, int ngramOrder, long offset, long prefixOffset, int word, LongRef val_, long suffixOffset, boolean ngramIsNew) Description copied from interface:ValueContainer
Adds a new value at the specified offset.- Specified by:
add
in interfaceValueContainer<V extends LongRepresentable<V>>
- Parameters:
ngramOrder
- As always, ngramOrder is 0-based (0=unigram)offset
-prefixOffset
-word
-val_
-suffixOffset
-- Returns:
- Whether or not the add was successful
-
getCompressed
Description copied from interface:CompressibleValueContainer
Compresses the value at the given offset into a list of bits.- Specified by:
getCompressed
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
- Parameters:
offset
-ngramOrder
-- Returns:
-
decompress
Description copied from interface:CompressibleValueContainer
Reads and decompresses from the bit stream bits.- Specified by:
decompress
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
- Parameters:
bits
-ngramOrder
-justConsume
- If true, nothing is returned, and the function simply consumes the appropriate number of bits from the BitStream.
-
setSizeAtLeast
public void setSizeAtLeast(long size, int ngramOrder) Description copied from interface:ValueContainer
Sets internal storage for size for a particular n-gram order- Specified by:
setSizeAtLeast
in interfaceValueContainer<V extends LongRepresentable<V>>
- Parameters:
size
-ngramOrder
-
-
getSuffixOffset
public long getSuffixOffset(long index, int ngramOrder) -
getSuffixOffsetFromInternalVal
protected int getSuffixOffsetFromInternalVal(long internalVal) - Parameters:
internalVal
-- Returns:
-
getRank
protected long getRank(int ngramOrder, long offset) -
clearStorageAfterCompression
public void clearStorageAfterCompression(int ngramOrder) - Specified by:
clearStorageAfterCompression
in interfaceCompressibleValueContainer<V extends LongRepresentable<V>>
-
clearStorageForOrder
public void clearStorageForOrder(int ngramOrder) - Specified by:
clearStorageForOrder
in interfaceValueContainer<V extends LongRepresentable<V>>
-
storeSuffixoffsets
public boolean storeSuffixoffsets()- Specified by:
storeSuffixoffsets
in interfaceValueContainer<V extends LongRepresentable<V>>
-
numValueBits
public int numValueBits(int ngramOrder) - Specified by:
numValueBits
in interfaceValueContainer<V extends LongRepresentable<V>>
-