java.lang.Object
org.apache.lucene.analysis.Analyzer
org.apache.lucene.analysis.StopwordAnalyzerBase
org.apache.lucene.analysis.cjk.CJKAnalyzer
- All Implemented Interfaces:
Closeable
,AutoCloseable
An
Analyzer
that tokenizes text with StandardTokenizer
, normalizes content with
CJKWidthFilter
, folds case with LowerCaseFilter
, forms bigrams of CJK with CJKBigramFilter
, and filters stopwords with StopFilter
- Since:
- 3.1
-
Nested Class Summary
Nested ClassesNested classes/interfaces inherited from class org.apache.lucene.analysis.Analyzer
Analyzer.ReuseStrategy, Analyzer.TokenStreamComponents
-
Field Summary
FieldsFields inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase
stopwords
Fields inherited from class org.apache.lucene.analysis.Analyzer
GLOBAL_REUSE_STRATEGY, PER_FIELD_REUSE_STRATEGY
-
Constructor Summary
ConstructorsConstructorDescriptionBuilds an analyzer which removes words ingetDefaultStopSet()
.CJKAnalyzer
(CharArraySet stopwords) Builds an analyzer with the given stop words -
Method Summary
Modifier and TypeMethodDescriptionprotected Analyzer.TokenStreamComponents
createComponents
(String fieldName) Creates a newAnalyzer.TokenStreamComponents
instance for this analyzer.static CharArraySet
Returns an unmodifiable instance of the default stop-words set.protected TokenStream
normalize
(String fieldName, TokenStream in) Wrap the givenTokenStream
in order to apply normalization filters.Methods inherited from class org.apache.lucene.analysis.StopwordAnalyzerBase
getStopwordSet, loadStopwordSet, loadStopwordSet, loadStopwordSet
Methods inherited from class org.apache.lucene.analysis.Analyzer
attributeFactory, close, getOffsetGap, getPositionIncrementGap, getReuseStrategy, initReader, initReaderForNormalization, normalize, tokenStream, tokenStream
-
Field Details
-
DEFAULT_STOPWORD_FILE
File containing default CJK stopwords.Currently it contains some common English words that are not usually useful for searching and some double-byte interpunctions.
- See Also:
-
-
Constructor Details
-
CJKAnalyzer
public CJKAnalyzer()Builds an analyzer which removes words ingetDefaultStopSet()
. -
CJKAnalyzer
Builds an analyzer with the given stop words- Parameters:
stopwords
- a stopword set
-
-
Method Details
-
getDefaultStopSet
Returns an unmodifiable instance of the default stop-words set.- Returns:
- an unmodifiable instance of the default stop-words set.
-
createComponents
Description copied from class:Analyzer
Creates a newAnalyzer.TokenStreamComponents
instance for this analyzer.- Specified by:
createComponents
in classAnalyzer
- Parameters:
fieldName
- the name of the fields content passed to theAnalyzer.TokenStreamComponents
sink as a reader- Returns:
- the
Analyzer.TokenStreamComponents
for this analyzer.
-
normalize
Description copied from class:Analyzer
Wrap the givenTokenStream
in order to apply normalization filters. The default implementation returns theTokenStream
as-is. This is used byAnalyzer.normalize(String, String)
.
-