Skip navigation links
$ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ 

$

$colon$bslash(B, Function2<A, B, B>) - Static method in class org.apache.spark.sql.types.StructType
 
$colon$plus(B, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
$div$colon(B, Function2<B, A, B>) - Static method in class org.apache.spark.sql.types.StructType
 
$greater(A) - Static method in class org.apache.spark.sql.types.Decimal
 
$greater(A) - Static method in class org.apache.spark.storage.RDDInfo
 
$greater$eq(A) - Static method in class org.apache.spark.sql.types.Decimal
 
$greater$eq(A) - Static method in class org.apache.spark.storage.RDDInfo
 
$less(A) - Static method in class org.apache.spark.sql.types.Decimal
 
$less(A) - Static method in class org.apache.spark.storage.RDDInfo
 
$less$eq(A) - Static method in class org.apache.spark.sql.types.Decimal
 
$less$eq(A) - Static method in class org.apache.spark.storage.RDDInfo
 
$minus$greater(T) - Static method in class org.apache.spark.ml.param.DoubleParam
 
$minus$greater(T) - Static method in class org.apache.spark.ml.param.FloatParam
 
$plus$colon(B, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
$plus$eq(T) - Static method in class org.apache.spark.Accumulator
Deprecated.
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.api.r.RRDD
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
$plus$plus(RDD<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
$plus$plus(GenTraversableOnce<B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
$plus$plus$colon(TraversableOnce<B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
$plus$plus$colon(Traversable<B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
$plus$plus$eq(R) - Static method in class org.apache.spark.Accumulator
Deprecated.
 

A

abort(WriterCommitMessage[]) - Method in interface org.apache.spark.sql.sources.v2.writer.DataSourceWriter
Aborts this writing job because some data writers are failed and keep failing when retry, or the Spark job fails with some unknown reasons, or DataSourceWriter.commit(WriterCommitMessage[]) fails.
abort() - Method in interface org.apache.spark.sql.sources.v2.writer.DataWriter
Aborts this writer if it is failed.
abort(long, WriterCommitMessage[]) - Method in interface org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
Aborts this writing job because some data writers are failed and keep failing when retry, or the Spark job fails with some unknown reasons, or StreamWriter.commit(WriterCommitMessage[]) fails.
abort(WriterCommitMessage[]) - Method in interface org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
 
abortJob(JobContext) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Aborts a job after the writes fail.
abortJob(JobContext) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
abortTask(TaskAttemptContext) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Aborts a task after the writes have failed.
abortTask(TaskAttemptContext) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
abs(Column) - Static method in class org.apache.spark.sql.functions
Computes the absolute value.
abs() - Method in class org.apache.spark.sql.types.Decimal
 
absent() - Static method in class org.apache.spark.api.java.Optional
 
AbsoluteError - Class in org.apache.spark.mllib.tree.loss
Developer API Class for absolute error loss calculation (for regression).
AbsoluteError() - Constructor for class org.apache.spark.mllib.tree.loss.AbsoluteError
 
AbstractLauncher<T extends AbstractLauncher> - Class in org.apache.spark.launcher
Base class for launcher implementations.
accept(Parsers) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
accept(ES, Function1<ES, List<Object>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
accept(String, PartialFunction<Object, U>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
accept(Path) - Method in class org.apache.spark.ml.image.SamplePathFilter
 
acceptIf(Function1<Object, Object>, Function1<Object, String>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
acceptMatch(String, PartialFunction<Object, U>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
acceptSeq(ES, Function1<ES, Iterable<Object>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
acceptsType(DataType) - Method in class org.apache.spark.sql.types.ObjectType
 
accId() - Method in class org.apache.spark.CleanAccum
 
Accumulable<R,T> - Class in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
Accumulable(R, AccumulableParam<R, T>) - Constructor for class org.apache.spark.Accumulable
Deprecated.
 
accumulable(T, AccumulableParam<T, R>) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulable(T, String, AccumulableParam<T, R>) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulable(R, AccumulableParam<R, T>) - Method in class org.apache.spark.SparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulable(R, String, AccumulableParam<R, T>) - Method in class org.apache.spark.SparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulableCollection(R, Function1<R, Growable<T>>, ClassTag<R>) - Method in class org.apache.spark.SparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
AccumulableInfo - Class in org.apache.spark.scheduler
Developer API Information about an Accumulable modified during a task or stage.
AccumulableInfo - Class in org.apache.spark.status.api.v1
 
accumulableInfoFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
accumulableInfoToJson(AccumulableInfo) - Static method in class org.apache.spark.util.JsonProtocol
 
AccumulableParam<R,T> - Interface in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulables() - Method in class org.apache.spark.scheduler.StageInfo
Terminal values of accumulables updated during this stage, including all the user-defined accumulators.
accumulables() - Method in class org.apache.spark.scheduler.TaskInfo
Intermediate updates to accumulables during this task.
accumulablesToJson(Traversable<AccumulableInfo>) - Static method in class org.apache.spark.util.JsonProtocol
 
Accumulator<T> - Class in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulator(int) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().longAccumulator(). Since 2.0.0.
accumulator(int, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().longAccumulator(String). Since 2.0.0.
accumulator(double) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().doubleAccumulator(). Since 2.0.0.
accumulator(double, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().doubleAccumulator(String). Since 2.0.0.
accumulator(T, AccumulatorParam<T>) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulator(T, String, AccumulatorParam<T>) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulator(T, AccumulatorParam<T>) - Method in class org.apache.spark.SparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
accumulator(T, String, AccumulatorParam<T>) - Method in class org.apache.spark.SparkContext
Deprecated.
use AccumulatorV2. Since 2.0.0.
AccumulatorContext - Class in org.apache.spark.util
An internal class used to track accumulators by Spark itself.
AccumulatorContext() - Constructor for class org.apache.spark.util.AccumulatorContext
 
AccumulatorParam<T> - Interface in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
AccumulatorParam.DoubleAccumulatorParam$ - Class in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
AccumulatorParam.FloatAccumulatorParam$ - Class in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
AccumulatorParam.IntAccumulatorParam$ - Class in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
AccumulatorParam.LongAccumulatorParam$ - Class in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
AccumulatorParam.StringAccumulatorParam$ - Class in org.apache.spark
Deprecated.
use AccumulatorV2. Since 2.0.0.
ACCUMULATORS() - Static method in class org.apache.spark.status.TaskIndexNames
 
accumulatorUpdates() - Method in class org.apache.spark.status.api.v1.StageData
 
accumulatorUpdates() - Method in class org.apache.spark.status.api.v1.TaskData
 
AccumulatorV2<IN,OUT> - Class in org.apache.spark.util
The base class for accumulators, that can accumulate inputs of type IN, and produce output of type OUT.
AccumulatorV2() - Constructor for class org.apache.spark.util.AccumulatorV2
 
accumUpdates() - Method in class org.apache.spark.ExceptionFailure
 
accumUpdates() - Method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
accuracy() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns accuracy.
accuracy() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns accuracy (equals to the total number of correctly classified instances out of the total number of instances.)
accuracy() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns accuracy
acos(Column) - Static method in class org.apache.spark.sql.functions
 
acos(String) - Static method in class org.apache.spark.sql.functions
 
active() - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
Returns a list of active queries associated with this SQLContext
active() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
ACTIVE() - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
activeStages() - Method in class org.apache.spark.status.LiveJob
 
activeTasks() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
activeTasks() - Method in class org.apache.spark.status.LiveExecutor
 
activeTasks() - Method in class org.apache.spark.status.LiveJob
 
activeTasks() - Method in class org.apache.spark.status.LiveStage
 
add(T) - Method in class org.apache.spark.Accumulable
Deprecated.
Add more data to this accumulator / accumulable
add(T) - Static method in class org.apache.spark.Accumulator
Deprecated.
 
add(Vector) - Method in class org.apache.spark.ml.clustering.ExpectationAggregator
Add a new training instance to this ExpectationAggregator, update the weights, means and covariances for each distributions, and update the log likelihood.
add(AFTPoint) - Method in class org.apache.spark.ml.regression.AFTAggregator
Add a new training data to this AFTAggregator, and update the loss and gradient of the objective function.
add(double[], MultivariateGaussian[], ExpectationSum, Vector<Object>) - Static method in class org.apache.spark.mllib.clustering.ExpectationSum
 
add(Vector) - Method in class org.apache.spark.mllib.feature.IDF.DocumentFrequencyAggregator
Adds a new document.
add(BlockMatrix) - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Adds the given block matrix other to this block matrix: this + other.
add(Vector) - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Add a new sample to this summarizer, and update the statistical summary.
add(StructField) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new field.
add(String, DataType) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new nullable field with no metadata.
add(String, DataType, boolean) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new field with no metadata.
add(String, DataType, boolean, Metadata) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new field and specifying metadata.
add(String, DataType, boolean, String) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new field and specifying metadata.
add(String, String) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new nullable field with no metadata where the dataType is specified as a String.
add(String, String, boolean) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new field with no metadata where the dataType is specified as a String.
add(String, String, boolean, Metadata) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
add(String, String, boolean, String) - Method in class org.apache.spark.sql.types.StructType
Creates a new StructType by adding a new field and specifying metadata where the dataType is specified as a String.
add(long, long) - Static method in class org.apache.spark.streaming.util.RawTextHelper
 
add(IN) - Method in class org.apache.spark.util.AccumulatorV2
Takes the inputs and accumulates.
add(T) - Method in class org.apache.spark.util.CollectionAccumulator
 
add(Double) - Method in class org.apache.spark.util.DoubleAccumulator
Adds v to the accumulator, i.e.
add(double) - Method in class org.apache.spark.util.DoubleAccumulator
Adds v to the accumulator, i.e.
add(T) - Method in class org.apache.spark.util.LegacyAccumulatorWrapper
 
add(Long) - Method in class org.apache.spark.util.LongAccumulator
Adds v to the accumulator, i.e.
add(long) - Method in class org.apache.spark.util.LongAccumulator
Adds v to the accumulator, i.e.
add(Object) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by one.
add(Object, long) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by count.
add_months(Column, int) - Static method in class org.apache.spark.sql.functions
Returns the date that is numMonths after startDate.
addAccumulator(R, T) - Method in interface org.apache.spark.AccumulableParam
Deprecated.
Add additional data to the accumulator value.
addAccumulator(T, T) - Method in interface org.apache.spark.AccumulatorParam
Deprecated.
 
addAppArgs(String...) - Method in class org.apache.spark.launcher.AbstractLauncher
Adds command line arguments for the application.
addAppArgs(String...) - Method in class org.apache.spark.launcher.SparkLauncher
 
addBinary(byte[]) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by one.
addBinary(byte[], long) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by count.
addFile(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Add a file to be downloaded with this Spark job on every node.
addFile(String, boolean) - Method in class org.apache.spark.api.java.JavaSparkContext
Add a file to be downloaded with this Spark job on every node.
addFile(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Adds a file to be submitted with the application.
addFile(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
addFile(String) - Method in class org.apache.spark.SparkContext
Add a file to be downloaded with this Spark job on every node.
addFile(String, boolean) - Method in class org.apache.spark.SparkContext
Add a file to be downloaded with this Spark job on every node.
addFilters(Seq<ServletContextHandler>, SparkConf) - Static method in class org.apache.spark.ui.JettyUtils
Add filters, if any, to the given list of ServletContextHandlers
addGrid(Param<T>, Iterable<T>) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a param with multiple values (overwrites if the input param exists).
addGrid(DoubleParam, double[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a double param with multiple values.
addGrid(IntParam, int[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds an int param with multiple values.
addGrid(FloatParam, float[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a float param with multiple values.
addGrid(LongParam, long[]) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a long param with multiple values.
addGrid(BooleanParam) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Adds a boolean param with true and false.
addInPlace(R, R) - Method in interface org.apache.spark.AccumulableParam
Deprecated.
Merge two accumulated values together.
addInPlace(double, double) - Method in class org.apache.spark.AccumulatorParam.DoubleAccumulatorParam$
Deprecated.
 
addInPlace(float, float) - Method in class org.apache.spark.AccumulatorParam.FloatAccumulatorParam$
Deprecated.
 
addInPlace(int, int) - Method in class org.apache.spark.AccumulatorParam.IntAccumulatorParam$
Deprecated.
 
addInPlace(long, long) - Method in class org.apache.spark.AccumulatorParam.LongAccumulatorParam$
Deprecated.
 
addInPlace(String, String) - Method in class org.apache.spark.AccumulatorParam.StringAccumulatorParam$
Deprecated.
 
addJar(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
addJar(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Adds a jar file to be submitted with the application.
addJar(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
addJar(String) - Method in class org.apache.spark.SparkContext
Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
addJar(String) - Method in class org.apache.spark.sql.hive.HiveSessionResourceLoader
 
addListener(SparkAppHandle.Listener) - Method in interface org.apache.spark.launcher.SparkAppHandle
Adds a listener to be notified of changes to the handle's information.
addListener(L) - Static method in class org.apache.spark.scheduler.AsyncEventQueue
 
addListener(StreamingQueryListener) - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
Register a StreamingQueryListener to receive up-calls for life cycle events of StreamingQuery.
addLocalConfiguration(String, int, int, int, JobConf) - Static method in class org.apache.spark.rdd.HadoopRDD
Add Hadoop configuration specific to a single partition and attempt.
addLong(long) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by one.
addLong(long, long) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by count.
addMapOutput(int, MapStatus) - Method in class org.apache.spark.ShuffleStatus
Register a map output.
addMetrics(TaskMetrics, TaskMetrics) - Static method in class org.apache.spark.status.LiveEntityHelpers
Add m2 values to m1.
addPartition(LiveRDDPartition) - Method in class org.apache.spark.status.RDDPartitionSeq
 
addPartToPGroup(Partition, PartitionGroup) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
addPyFile(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Adds a python file / zip / egg to be submitted with the application.
addPyFile(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
address() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
addShutdownHook(Function0<BoxedUnit>) - Static method in class org.apache.spark.util.ShutdownHookManager
Adds a shutdown hook with default priority.
addShutdownHook(int, Function0<BoxedUnit>) - Static method in class org.apache.spark.util.ShutdownHookManager
Adds a shutdown hook with the given priority.
addSparkArg(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Adds a no-value argument to the Spark invocation.
addSparkArg(String, String) - Method in class org.apache.spark.launcher.AbstractLauncher
Adds an argument with a value to the Spark invocation.
addSparkArg(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
addSparkArg(String, String) - Method in class org.apache.spark.launcher.SparkLauncher
 
addSparkListener(SparkListenerInterface) - Method in class org.apache.spark.SparkContext
Developer API Register a listener to receive up-calls from events that happen during execution.
addStreamingListener(StreamingListener) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Add a StreamingListener object for receiving system events related to streaming.
addStreamingListener(StreamingListener) - Method in class org.apache.spark.streaming.StreamingContext
Add a StreamingListener object for receiving system events related to streaming.
addString(StringBuilder, String, String, String) - Static method in class org.apache.spark.sql.types.StructType
 
addString(StringBuilder, String) - Static method in class org.apache.spark.sql.types.StructType
 
addString(StringBuilder) - Static method in class org.apache.spark.sql.types.StructType
 
addString(String) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by one.
addString(String, long) - Method in class org.apache.spark.util.sketch.CountMinSketch
Increments item's count by count.
addSuppressed(Throwable) - Static method in exception org.apache.spark.sql.AnalysisException
 
addTaskCompletionListener(TaskCompletionListener) - Method in class org.apache.spark.TaskContext
Adds a (Java friendly) listener to be executed on task completion.
addTaskCompletionListener(Function1<TaskContext, BoxedUnit>) - Method in class org.apache.spark.TaskContext
Adds a listener in the form of a Scala closure to be executed on task completion.
addTaskFailureListener(TaskFailureListener) - Method in class org.apache.spark.TaskContext
Adds a listener to be executed on task failure.
addTaskFailureListener(Function2<TaskContext, Throwable, BoxedUnit>) - Method in class org.apache.spark.TaskContext
Adds a listener to be executed on task failure.
addTime() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
addTime() - Method in class org.apache.spark.status.LiveExecutor
 
addURL(URL) - Method in class org.apache.spark.util.MutableURLClassLoader
 
AddWebUIFilter(String, Map<String, String>, String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter
 
AddWebUIFilter$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter$
 
AFTAggregator - Class in org.apache.spark.ml.regression
AFTAggregator computes the gradient and loss for a AFT loss function, as used in AFT survival regression for samples in sparse or dense vector in an online fashion.
AFTAggregator(Broadcast<DenseVector<Object>>, boolean, Broadcast<double[]>) - Constructor for class org.apache.spark.ml.regression.AFTAggregator
 
AFTCostFun - Class in org.apache.spark.ml.regression
AFTCostFun implements Breeze's DiffFunction[T] for AFT cost.
AFTCostFun(RDD<AFTPoint>, boolean, Broadcast<double[]>, int) - Constructor for class org.apache.spark.ml.regression.AFTCostFun
 
AFTSurvivalRegression - Class in org.apache.spark.ml.regression
Experimental Fit a parametric survival regression model named accelerated failure time (AFT) model (see Accelerated failure time model (Wikipedia)) based on the Weibull distribution of the survival time.
AFTSurvivalRegression(String) - Constructor for class org.apache.spark.ml.regression.AFTSurvivalRegression
 
AFTSurvivalRegression() - Constructor for class org.apache.spark.ml.regression.AFTSurvivalRegression
 
AFTSurvivalRegressionModel - Class in org.apache.spark.ml.regression
Experimental Model produced by AFTSurvivalRegression.
agg(Column, Column...) - Method in class org.apache.spark.sql.Dataset
Aggregates on the entire Dataset without groups.
agg(Tuple2<String, String>, Seq<Tuple2<String, String>>) - Method in class org.apache.spark.sql.Dataset
(Scala-specific) Aggregates on the entire Dataset without groups.
agg(Map<String, String>) - Method in class org.apache.spark.sql.Dataset
(Scala-specific) Aggregates on the entire Dataset without groups.
agg(Map<String, String>) - Method in class org.apache.spark.sql.Dataset
(Java-specific) Aggregates on the entire Dataset without groups.
agg(Column, Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Aggregates on the entire Dataset without groups.
agg(TypedColumn<V, U1>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Computes the given aggregation, returning a Dataset of tuples for each unique key and the result of computing this aggregation over all elements in the group.
agg(TypedColumn<V, U1>, TypedColumn<V, U2>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Computes the given aggregations, returning a Dataset of tuples for each unique key and the result of computing these aggregations over all elements in the group.
agg(TypedColumn<V, U1>, TypedColumn<V, U2>, TypedColumn<V, U3>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Computes the given aggregations, returning a Dataset of tuples for each unique key and the result of computing these aggregations over all elements in the group.
agg(TypedColumn<V, U1>, TypedColumn<V, U2>, TypedColumn<V, U3>, TypedColumn<V, U4>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Computes the given aggregations, returning a Dataset of tuples for each unique key and the result of computing these aggregations over all elements in the group.
agg(Column, Column...) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute aggregates by specifying a series of aggregate columns.
agg(Tuple2<String, String>, Seq<Tuple2<String, String>>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
(Scala-specific) Compute aggregates by specifying the column names and aggregate methods.
agg(Map<String, String>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
(Scala-specific) Compute aggregates by specifying a map from column name to aggregate methods.
agg(Map<String, String>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
(Java-specific) Compute aggregates by specifying a map from column name to aggregate methods.
agg(Column, Seq<Column>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute aggregates by specifying a series of aggregate columns.
aggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Static method in class org.apache.spark.api.java.JavaRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Aggregate the elements of each partition, and then the results for all the partitions, using given combine functions and a neutral "zero value".
aggregate(U, Function2<U, T, U>, Function2<U, U, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
aggregate(Function0<B>, Function2<B, A, B>, Function2<B, B, B>) - Static method in class org.apache.spark.sql.types.StructType
 
aggregateByKey(U, Partitioner, Function2<U, V, U>, Function2<U, U, U>) - Method in class org.apache.spark.api.java.JavaPairRDD
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, int, Function2<U, V, U>, Function2<U, U, U>) - Method in class org.apache.spark.api.java.JavaPairRDD
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, Function2<U, V, U>, Function2<U, U, U>) - Method in class org.apache.spark.api.java.JavaPairRDD
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, Partitioner, Function2<U, V, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, int, Function2<U, V, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Aggregate the values of each key, using given combine functions and a neutral "zero value".
aggregateByKey(U, Function2<U, V, U>, Function2<U, U, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Aggregate the values of each key, using given combine functions and a neutral "zero value".
AggregatedDialect - Class in org.apache.spark.sql.jdbc
AggregatedDialect can unify multiple dialects into one virtual Dialect.
AggregatedDialect(List<JdbcDialect>) - Constructor for class org.apache.spark.sql.jdbc.AggregatedDialect
 
aggregateMessages(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, ClassTag<A>) - Method in class org.apache.spark.graphx.Graph
Aggregates values from the neighboring edges and vertices of each vertex.
aggregateMessages(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, ClassTag<A>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
aggregateMessages$default$3() - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
aggregateMessagesWithActiveSet(Function1<EdgeContext<VD, ED, A>, BoxedUnit>, Function2<A, A, A>, TripletFields, Option<Tuple2<VertexRDD<?>, EdgeDirection>>, ClassTag<A>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
aggregateUsingIndex(RDD<Tuple2<Object, VD2>>, Function2<VD2, VD2, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
aggregateUsingIndex(RDD<Tuple2<Object, VD2>>, Function2<VD2, VD2, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.VertexRDD
Aggregates vertices in messages that have the same ids using reduceFunc, returning a VertexRDD co-indexed with this.
AggregatingEdgeContext<VD,ED,A> - Class in org.apache.spark.graphx.impl
 
AggregatingEdgeContext(Function2<A, A, A>, Object, BitSet) - Constructor for class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
aggregationDepth() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
aggregationDepth() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
aggregationDepth() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
aggregationDepth() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
aggregationDepth() - Method in interface org.apache.spark.ml.param.shared.HasAggregationDepth
Param for suggested depth for treeAggregate (&gt;= 2).
aggregationDepth() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
aggregationDepth() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
aggregationDepth() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
aggregationDepth() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
Aggregator<K,V,C> - Class in org.apache.spark
Developer API A set of functions used to aggregate data.
Aggregator(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>) - Constructor for class org.apache.spark.Aggregator
 
aggregator() - Method in class org.apache.spark.ShuffleDependency
 
Aggregator<IN,BUF,OUT> - Class in org.apache.spark.sql.expressions
Experimental A base class for user-defined aggregations, which can be used in Dataset operations to take all of the elements of a group and reduce them to a single value.
Aggregator() - Constructor for class org.apache.spark.sql.expressions.Aggregator
 
aic(RDD<Tuple3<Object, Object, Object>>, double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
 
aic(RDD<Tuple3<Object, Object, Object>>, double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
 
aic(RDD<Tuple3<Object, Object, Object>>, double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
aic(RDD<Tuple3<Object, Object, Object>>, double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
 
aic() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
Akaike Information Criterion (AIC) for the fitted model.
Algo - Class in org.apache.spark.mllib.tree.configuration
Enum to select the algorithm for the decision tree
Algo() - Constructor for class org.apache.spark.mllib.tree.configuration.Algo
 
algo() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
algo() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
 
algo() - Method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
algo() - Method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
algorithm() - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
 
alias(String) - Method in class org.apache.spark.sql.Column
Gives the column an alias.
alias(String) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with an alias set.
alias(Symbol) - Method in class org.apache.spark.sql.Dataset
(Scala-specific) Returns a new Dataset with an alias set.
All - Static variable in class org.apache.spark.graphx.TripletFields
Expose all the fields (source, edge, and destination).
allAttributes() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
allAttributes() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
allAttributes() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
allAttributes() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
AllJobsCancelled - Class in org.apache.spark.scheduler
 
AllJobsCancelled() - Constructor for class org.apache.spark.scheduler.AllJobsCancelled
 
AllReceiverIds - Class in org.apache.spark.streaming.scheduler
A message used by ReceiverTracker to ask all receiver's ids still stored in ReceiverTrackerEndpoint.
AllReceiverIds() - Constructor for class org.apache.spark.streaming.scheduler.AllReceiverIds
 
allSources() - Static method in class org.apache.spark.metrics.source.StaticSources
The set of all static sources.
alpha() - Static method in class org.apache.spark.ml.recommendation.ALS
 
alpha() - Method in class org.apache.spark.mllib.random.WeibullGenerator
 
ALS - Class in org.apache.spark.ml.recommendation
Alternating Least Squares (ALS) matrix factorization.
ALS(String) - Constructor for class org.apache.spark.ml.recommendation.ALS
 
ALS() - Constructor for class org.apache.spark.ml.recommendation.ALS
 
ALS - Class in org.apache.spark.mllib.recommendation
Alternating Least Squares matrix factorization.
ALS() - Constructor for class org.apache.spark.mllib.recommendation.ALS
Constructs an ALS instance with default parameters: {numBlocks: -1, rank: 10, iterations: 10, lambda: 0.01, implicitPrefs: false, alpha: 1.0}.
ALS.InBlock$ - Class in org.apache.spark.ml.recommendation
 
ALS.Rating<ID> - Class in org.apache.spark.ml.recommendation
Developer API Rating class for better code readability.
ALS.Rating$ - Class in org.apache.spark.ml.recommendation
 
ALS.RatingBlock$ - Class in org.apache.spark.ml.recommendation
 
ALSModel - Class in org.apache.spark.ml.recommendation
Model fitted by ALS.
am() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterClusterManager
 
AnalysisException - Exception in org.apache.spark.sql
Thrown when a query fails to analyze, usually because the query itself is invalid.
and(Column) - Method in class org.apache.spark.sql.Column
Boolean AND.
And - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff both left or right evaluate to true.
And(Filter, Filter) - Constructor for class org.apache.spark.sql.sources.And
 
andThen(Function1<B, C>) - Static method in class org.apache.spark.sql.types.StructType
 
antecedent() - Method in class org.apache.spark.mllib.fpm.AssociationRules.Rule
 
ANY() - Static method in class org.apache.spark.scheduler.TaskLocality
 
AnyDataType - Class in org.apache.spark.sql.types
An AbstractDataType that matches any concrete data types.
AnyDataType() - Constructor for class org.apache.spark.sql.types.AnyDataType
 
anyNull() - Method in interface org.apache.spark.sql.Row
Returns true if there are any NULL values in this row.
anyNull() - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
ApiHelper - Class in org.apache.spark.ui.jobs
 
ApiHelper() - Constructor for class org.apache.spark.ui.jobs.ApiHelper
 
appAttemptId() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
Append() - Static method in class org.apache.spark.sql.streaming.OutputMode
OutputMode in which only the new rows in the streaming DataFrame/Dataset will be written to the sink.
appendBias(Vector) - Static method in class org.apache.spark.mllib.util.MLUtils
Returns a new vector with 1.0 (bias) appended to the input vector.
appendColumn(StructType, String, DataType, boolean) - Static method in class org.apache.spark.ml.util.SchemaUtils
Appends a new column to the input schema.
appendColumn(StructType, StructField) - Static method in class org.apache.spark.ml.util.SchemaUtils
Appends a new column to the input schema.
appendReadColumns(Configuration, Seq<Integer>, Seq<String>) - Static method in class org.apache.spark.sql.hive.HiveShim
 
appId() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
APPLICATION_EXECUTOR_LIMIT() - Static method in class org.apache.spark.ui.ToolTips
 
applicationAttemptId() - Method in class org.apache.spark.SparkContext
 
ApplicationAttemptInfo - Class in org.apache.spark.status.api.v1
 
applicationEndFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
applicationEndToJson(SparkListenerApplicationEnd) - Static method in class org.apache.spark.util.JsonProtocol
 
ApplicationEnvironmentInfo - Class in org.apache.spark.status.api.v1
 
applicationId() - Method in class org.apache.spark.SparkContext
A unique identifier for the Spark application.
ApplicationInfo - Class in org.apache.spark.status.api.v1
 
applicationStartFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
applicationStartToJson(SparkListenerApplicationStart) - Static method in class org.apache.spark.util.JsonProtocol
 
ApplicationStatus - Enum in org.apache.spark.status.api.v1
 
apply(RDD<Tuple2<Object, VD>>, RDD<Edge<ED>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.Graph
Construct a graph from a collection of vertices and edges with attributes.
apply(RDD<Edge<ED>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from edges, setting referenced vertices to defaultVertexAttr.
apply(RDD<Tuple2<Object, VD>>, RDD<Edge<ED>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from vertices and edges, setting missing vertices to defaultVertexAttr.
apply(VertexRDD<VD>, EdgeRDD<ED>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from a VertexRDD and an EdgeRDD with arbitrary replicated vertices.
apply(Graph<VD, ED>, A, int, EdgeDirection, Function3<Object, VD, A, VD>, Function1<EdgeTriplet<VD, ED>, Iterator<Tuple2<Object, A>>>, Function2<A, A, A>, ClassTag<VD>, ClassTag<ED>, ClassTag<A>) - Static method in class org.apache.spark.graphx.Pregel
Execute a Pregel-like iterative vertex-parallel abstraction.
apply(RDD<Tuple2<Object, VD>>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.VertexRDD
Constructs a standalone VertexRDD (one that is not set up for efficient joins with an EdgeRDD) from an RDD of vertex-attribute pairs.
apply(RDD<Tuple2<Object, VD>>, EdgeRDD<?>, VD, ClassTag<VD>) - Static method in class org.apache.spark.graphx.VertexRDD
Constructs a VertexRDD from an RDD of vertex-attribute pairs.
apply(RDD<Tuple2<Object, VD>>, EdgeRDD<?>, VD, Function2<VD, VD, VD>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.VertexRDD
Constructs a VertexRDD from an RDD of vertex-attribute pairs.
apply(DenseMatrix<Object>, DenseMatrix<Object>, Function1<Object, Object>) - Static method in class org.apache.spark.ml.ann.ApplyInPlace
 
apply(DenseMatrix<Object>, DenseMatrix<Object>, DenseMatrix<Object>, Function2<Object, Object, Object>) - Static method in class org.apache.spark.ml.ann.ApplyInPlace
 
apply(String) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Gets an attribute by its name.
apply(int) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Gets an attribute by its index.
apply(int, int) - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
apply(int) - Method in class org.apache.spark.ml.linalg.DenseVector
 
apply(int, int) - Method in interface org.apache.spark.ml.linalg.Matrix
Gets the (i, j)-th element.
apply(int, int) - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
apply(int) - Static method in class org.apache.spark.ml.linalg.SparseVector
 
apply(int) - Method in interface org.apache.spark.ml.linalg.Vector
Gets the value of the ith element.
apply(Param<T>) - Method in class org.apache.spark.ml.param.ParamMap
Gets the value of the input param or its default value if it does not exist.
apply(GeneralizedLinearRegressionBase) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.FamilyAndLink$
Constructs the FamilyAndLink object from a parameter map
apply(Split) - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData$
 
apply(BinaryConfusionMatrix) - Static method in class org.apache.spark.mllib.evaluation.binary.FalsePositiveRate
 
apply(BinaryConfusionMatrix) - Static method in class org.apache.spark.mllib.evaluation.binary.Precision
 
apply(BinaryConfusionMatrix) - Static method in class org.apache.spark.mllib.evaluation.binary.Recall
 
apply(int, int) - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
apply(int) - Method in class org.apache.spark.mllib.linalg.DenseVector
 
apply(int, int) - Method in interface org.apache.spark.mllib.linalg.Matrix
Gets the (i, j)-th element.
apply(int, int) - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
apply(int) - Static method in class org.apache.spark.mllib.linalg.SparseVector
 
apply(int) - Method in interface org.apache.spark.mllib.linalg.Vector
Gets the value of the ith element.
apply(int) - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
apply(int) - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
apply(int) - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
apply(int) - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
apply(int, Predict, double, boolean) - Static method in class org.apache.spark.mllib.tree.model.Node
Construct a node with nodeIndex, predict, impurity and isLeaf parameters.
apply(int) - Static method in class org.apache.spark.rdd.CheckpointState
 
apply(int) - Static method in class org.apache.spark.rdd.DeterministicLevel
 
apply(long, String, Option<String>, String, boolean) - Static method in class org.apache.spark.scheduler.AccumulableInfo
Deprecated.
do not create AccumulableInfo. Since 2.0.0.
apply(long, String, Option<String>, String) - Static method in class org.apache.spark.scheduler.AccumulableInfo
Deprecated.
do not create AccumulableInfo. Since 2.0.0.
apply(long, String, String) - Static method in class org.apache.spark.scheduler.AccumulableInfo
Deprecated.
do not create AccumulableInfo. Since 2.0.0.
apply(String, long, Enumeration.Value, ByteBuffer) - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate$
Alternate factory method that takes a ByteBuffer directly for the data field
apply(long, TaskMetrics) - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
apply(int) - Static method in class org.apache.spark.scheduler.SchedulingMode
 
apply(int) - Static method in class org.apache.spark.scheduler.TaskLocality
 
apply(Object) - Method in class org.apache.spark.sql.Column
Extracts a value or values from a complex type.
apply(String) - Method in class org.apache.spark.sql.Dataset
Selects column based on the column name and returns it as a Column.
apply(Column...) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Creates a Column for this UDAF using given Columns as input arguments.
apply(Seq<Column>) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Creates a Column for this UDAF using given Columns as input arguments.
apply(Column...) - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
Returns an expression that invokes the UDF, using the given arguments.
apply(Seq<Column>) - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
Returns an expression that invokes the UDF, using the given arguments.
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.DetermineTableStats
 
apply(int) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
apply(ScriptInputOutputSchema) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
apply(int) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
apply(int) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
apply(int) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
apply(LogicalPlan) - Static method in class org.apache.spark.sql.hive.HiveAnalysis
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.RelationConversions
 
apply(LogicalPlan) - Method in class org.apache.spark.sql.hive.ResolveHiveSerdeTable
 
apply(Dataset<Row>, Seq<Expression>, RelationalGroupedDataset.GroupType) - Static method in class org.apache.spark.sql.RelationalGroupedDataset
 
apply(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i.
apply(String) - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
use Trigger.ProcessingTime(interval)
apply(Duration) - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
use Trigger.ProcessingTime(interval)
apply(DataType) - Static method in class org.apache.spark.sql.types.ArrayType
Construct a ArrayType object with the given element type.
apply(double) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(long) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(int) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(BigDecimal) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(BigDecimal) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(BigInteger) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(BigInt) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(BigDecimal, int, int) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(BigDecimal, int, int) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(long, int, int) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(String) - Static method in class org.apache.spark.sql.types.Decimal
 
apply(DataType, DataType) - Static method in class org.apache.spark.sql.types.MapType
Construct a MapType object with the given key type and value type.
apply(String) - Method in class org.apache.spark.sql.types.StructType
Extracts the StructField with the given name.
apply(Set<String>) - Method in class org.apache.spark.sql.types.StructType
Returns a StructType containing StructFields of the given names, preserving the original order of fields.
apply(int) - Method in class org.apache.spark.sql.types.StructType
 
apply(int) - Method in class org.apache.spark.status.RDDPartitionSeq
 
apply(String) - Static method in class org.apache.spark.storage.BlockId
 
apply(String, String, int, Option<String>) - Static method in class org.apache.spark.storage.BlockManagerId
Returns a BlockManagerId for the given configuration.
apply(ObjectInput) - Static method in class org.apache.spark.storage.BlockManagerId
 
apply(boolean, boolean, boolean, boolean, int) - Static method in class org.apache.spark.storage.StorageLevel
Developer API Create a new StorageLevel object.
apply(boolean, boolean, boolean, int) - Static method in class org.apache.spark.storage.StorageLevel
Developer API Create a new StorageLevel object without setting useOffHeap.
apply(int, int) - Static method in class org.apache.spark.storage.StorageLevel
Developer API Create a new StorageLevel object from its integer representation.
apply(ObjectInput) - Static method in class org.apache.spark.storage.StorageLevel
Developer API Read StorageLevel object from ObjectInput stream.
apply(long) - Static method in class org.apache.spark.streaming.Milliseconds
 
apply(long) - Static method in class org.apache.spark.streaming.Minutes
 
apply(int) - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
apply(long) - Static method in class org.apache.spark.streaming.Seconds
 
apply(int) - Static method in class org.apache.spark.TaskState
 
apply(TraversableOnce<Object>) - Static method in class org.apache.spark.util.StatCounter
Build a StatCounter from a list of values.
apply(Seq<Object>) - Static method in class org.apache.spark.util.StatCounter
Build a StatCounter from a list of values passed as variable-length arguments.
ApplyInPlace - Class in org.apache.spark.ml.ann
Implements in-place application of functions in the arrays
ApplyInPlace() - Constructor for class org.apache.spark.ml.ann.ApplyInPlace
 
applyOrElse(A1, Function1<A1, B1>) - Static method in class org.apache.spark.sql.types.StructType
 
applySchema(RDD<Row>, StructType) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
Use createDataFrame instead. Since 1.3.0.
applySchema(JavaRDD<Row>, StructType) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
Use createDataFrame instead. Since 1.3.0.
applySchema(RDD<?>, Class<?>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
Use createDataFrame instead. Since 1.3.0.
applySchema(JavaRDD<?>, Class<?>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
Use createDataFrame instead. Since 1.3.0.
appName() - Method in class org.apache.spark.api.java.JavaSparkContext
 
appName() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
appName() - Method in class org.apache.spark.SparkContext
 
appName(String) - Method in class org.apache.spark.sql.SparkSession.Builder
Sets a name for the application, which will be shown in the Spark web UI.
approx_count_distinct(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the approximate number of distinct items in a group.
approx_count_distinct(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the approximate number of distinct items in a group.
approx_count_distinct(Column, double) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the approximate number of distinct items in a group.
approx_count_distinct(String, double) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the approximate number of distinct items in a group.
approxCountDistinct(Column) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use approx_count_distinct. Since 2.1.0.
approxCountDistinct(String) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use approx_count_distinct. Since 2.1.0.
approxCountDistinct(Column, double) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use approx_count_distinct. Since 2.1.0.
approxCountDistinct(String, double) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use approx_count_distinct. Since 2.1.0.
ApproxHist() - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
approxNearestNeighbors(Dataset<?>, Vector, int, String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
approxNearestNeighbors(Dataset<?>, Vector, int) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
approxNearestNeighbors(Dataset<?>, Vector, int, String) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
approxNearestNeighbors(Dataset<?>, Vector, int) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
approxQuantile(String, double[], double) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Calculates the approximate quantiles of a numerical column of a DataFrame.
approxQuantile(String[], double[], double) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Calculates the approximate quantiles of numerical columns of a DataFrame.
approxSimilarityJoin(Dataset<?>, Dataset<?>, double, String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
approxSimilarityJoin(Dataset<?>, Dataset<?>, double) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
approxSimilarityJoin(Dataset<?>, Dataset<?>, double, String) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
approxSimilarityJoin(Dataset<?>, Dataset<?>, double) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
appSparkVersion() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
AppStatusUtils - Class in org.apache.spark.status
 
AppStatusUtils() - Constructor for class org.apache.spark.status.AppStatusUtils
 
AreaUnderCurve - Class in org.apache.spark.mllib.evaluation
Computes the area under the curve (AUC) using the trapezoidal rule.
AreaUnderCurve() - Constructor for class org.apache.spark.mllib.evaluation.AreaUnderCurve
 
areaUnderPR() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Computes the area under the precision-recall curve.
areaUnderROC() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
Computes the area under the receiver operating characteristic (ROC) curve.
areaUnderROC() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Computes the area under the receiver operating characteristic (ROC) curve.
argmax() - Method in class org.apache.spark.ml.linalg.DenseVector
 
argmax() - Method in class org.apache.spark.ml.linalg.SparseVector
 
argmax() - Method in interface org.apache.spark.ml.linalg.Vector
Find the index of a maximal element.
argmax() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
argmax() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
argmax() - Method in interface org.apache.spark.mllib.linalg.Vector
Find the index of a maximal element.
argString() - Method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
argString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
argString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
argString() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
array(DataType) - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type array.
array(Column...) - Static method in class org.apache.spark.sql.functions
Creates a new array column.
array(String, String...) - Static method in class org.apache.spark.sql.functions
Creates a new array column.
array(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Creates a new array column.
array(String, Seq<String>) - Static method in class org.apache.spark.sql.functions
Creates a new array column.
array() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
array_contains(Column, Object) - Static method in class org.apache.spark.sql.functions
Returns null if the array is null, true if the array contains value, and false otherwise.
arrayLengthGt(double) - Static method in class org.apache.spark.ml.param.ParamValidators
Check that the array length is greater than lowerBound.
ArrayType - Class in org.apache.spark.sql.types
 
ArrayType(DataType, boolean) - Constructor for class org.apache.spark.sql.types.ArrayType
 
ArrowColumnVector - Class in org.apache.spark.sql.vectorized
A column vector backed by Apache Arrow.
ArrowColumnVector(ValueVector) - Constructor for class org.apache.spark.sql.vectorized.ArrowColumnVector
 
as(Encoder<U>) - Method in class org.apache.spark.sql.Column
Provides a type hint about the expected return value of this column.
as(String) - Method in class org.apache.spark.sql.Column
Gives the column an alias.
as(Seq<String>) - Method in class org.apache.spark.sql.Column
(Scala-specific) Assigns the given aliases to the results of a table generating function.
as(String[]) - Method in class org.apache.spark.sql.Column
Assigns the given aliases to the results of a table generating function.
as(Symbol) - Method in class org.apache.spark.sql.Column
Gives the column an alias.
as(String, Metadata) - Method in class org.apache.spark.sql.Column
Gives the column an alias with metadata.
as(Encoder<U>) - Method in class org.apache.spark.sql.Dataset
Experimental Returns a new Dataset where each record has been mapped on to the specified type.
as(String) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with an alias set.
as(Symbol) - Method in class org.apache.spark.sql.Dataset
(Scala-specific) Returns a new Dataset with an alias set.
asBinary() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Convenient method for casting to binary logistic regression summary.
asBreeze() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts to a breeze matrix.
asBreeze() - Method in interface org.apache.spark.ml.linalg.Vector
Converts the instance to a breeze vector.
asBreeze() - Method in interface org.apache.spark.mllib.linalg.Matrix
Converts to a breeze matrix.
asBreeze() - Method in interface org.apache.spark.mllib.linalg.Vector
Converts the instance to a breeze vector.
asc() - Method in class org.apache.spark.sql.Column
Returns a sort expression based on ascending order of the column.
asc(String) - Static method in class org.apache.spark.sql.functions
Returns a sort expression based on ascending order of the column.
asc_nulls_first() - Method in class org.apache.spark.sql.Column
Returns a sort expression based on ascending order of the column, and null values return before non-null values.
asc_nulls_first(String) - Static method in class org.apache.spark.sql.functions
Returns a sort expression based on ascending order of the column, and null values return before non-null values.
asc_nulls_last() - Method in class org.apache.spark.sql.Column
Returns a sort expression based on ascending order of the column, and null values appear after non-null values.
asc_nulls_last(String) - Static method in class org.apache.spark.sql.functions
Returns a sort expression based on ascending order of the column, and null values appear after non-null values.
ascii(Column) - Static method in class org.apache.spark.sql.functions
Computes the numeric value of the first character of the string column, and returns the result as an int column.
asCode() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
asCode() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
asCode() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
asCode() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
asin(Column) - Static method in class org.apache.spark.sql.functions
 
asin(String) - Static method in class org.apache.spark.sql.functions
 
asIterator() - Method in class org.apache.spark.serializer.DeserializationStream
Read the elements of this stream through an iterator.
asJavaPairRDD() - Method in class org.apache.spark.api.r.PairwiseRRDD
 
asJavaRDD() - Method in class org.apache.spark.api.r.RRDD
 
asJavaRDD() - Method in class org.apache.spark.api.r.StringRRDD
 
asKeyValueIterator() - Method in class org.apache.spark.serializer.DeserializationStream
Read the elements of this stream through an iterator over key-value pairs.
AskPermissionToCommitOutput - Class in org.apache.spark.scheduler
 
AskPermissionToCommitOutput(int, int, int, int) - Constructor for class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
askRpcTimeout(SparkConf) - Static method in class org.apache.spark.util.RpcUtils
Returns the default Spark timeout to use for RPC ask operations.
askSlaves() - Method in class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus
 
askSlaves() - Method in class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds
 
asMap() - Method in class org.apache.spark.sql.sources.v2.DataSourceOptions
 
asML() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
asML() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
asML() - Method in interface org.apache.spark.mllib.linalg.Matrix
Convert this matrix to the new mllib-local representation.
asML() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
asML() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
asML() - Method in interface org.apache.spark.mllib.linalg.Vector
Convert this vector to the new mllib-local representation.
asNondeterministic() - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
Updates UserDefinedFunction to nondeterministic.
asNonNullable() - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
Updates UserDefinedFunction to non-nullable.
asNullable() - Method in class org.apache.spark.sql.types.ObjectType
 
asRDDId() - Method in class org.apache.spark.storage.BlockId
 
asRDDId() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
asRDDId() - Static method in class org.apache.spark.storage.RDDBlockId
 
asRDDId() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
asRDDId() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
asRDDId() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
asRDDId() - Static method in class org.apache.spark.storage.StreamBlockId
 
asRDDId() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
assertConf(JobContext, SparkConf) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
assertNotSpilled(SparkContext, String, Function0<T>) - Static method in class org.apache.spark.TestUtils
Run some code involving jobs submitted to the given context and assert that the jobs did not spill.
assertSpilled(SparkContext, String, Function0<T>) - Static method in class org.apache.spark.TestUtils
Run some code involving jobs submitted to the given context and assert that the jobs spilled.
Assignment(long, int) - Constructor for class org.apache.spark.mllib.clustering.PowerIterationClustering.Assignment
 
Assignment$() - Constructor for class org.apache.spark.mllib.clustering.PowerIterationClustering.Assignment$
 
assignments() - Method in class org.apache.spark.mllib.clustering.PowerIterationClusteringModel
 
AssociationRules - Class in org.apache.spark.ml.fpm
 
AssociationRules() - Constructor for class org.apache.spark.ml.fpm.AssociationRules
 
associationRules() - Method in class org.apache.spark.ml.fpm.FPGrowthModel
Get association rules fitted using the minConfidence.
AssociationRules - Class in org.apache.spark.mllib.fpm
Generates association rules from a RDD[FreqItemset[Item}.
AssociationRules() - Constructor for class org.apache.spark.mllib.fpm.AssociationRules
Constructs a default instance with default parameters {minConfidence = 0.8}.
AssociationRules.Rule<Item> - Class in org.apache.spark.mllib.fpm
An association rule between sets of items.
ASYNC_TRACKING_ENABLED() - Static method in class org.apache.spark.status.config
 
AsyncEventQueue - Class in org.apache.spark.scheduler
An asynchronous queue for events.
AsyncEventQueue(String, SparkConf, LiveListenerBusMetrics, LiveListenerBus) - Constructor for class org.apache.spark.scheduler.AsyncEventQueue
 
AsyncRDDActions<T> - Class in org.apache.spark.rdd
A set of asynchronous RDD actions available through an implicit conversion.
AsyncRDDActions(RDD<T>, ClassTag<T>) - Constructor for class org.apache.spark.rdd.AsyncRDDActions
 
atan(Column) - Static method in class org.apache.spark.sql.functions
 
atan(String) - Static method in class org.apache.spark.sql.functions
 
atan2(Column, Column) - Static method in class org.apache.spark.sql.functions
 
atan2(Column, String) - Static method in class org.apache.spark.sql.functions
 
atan2(String, Column) - Static method in class org.apache.spark.sql.functions
 
atan2(String, String) - Static method in class org.apache.spark.sql.functions
 
atan2(Column, double) - Static method in class org.apache.spark.sql.functions
 
atan2(String, double) - Static method in class org.apache.spark.sql.functions
 
atan2(double, Column) - Static method in class org.apache.spark.sql.functions
 
atan2(double, String) - Static method in class org.apache.spark.sql.functions
 
attempt() - Method in class org.apache.spark.status.api.v1.TaskData
 
ATTEMPT() - Static method in class org.apache.spark.status.TaskIndexNames
 
attemptId() - Method in class org.apache.spark.scheduler.StageInfo
Deprecated.
Use attemptNumber instead. Since 2.3.0.
attemptId() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
attemptId() - Method in class org.apache.spark.status.api.v1.StageData
 
attemptNumber() - Method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
attemptNumber() - Method in class org.apache.spark.scheduler.StageInfo
 
attemptNumber() - Method in class org.apache.spark.scheduler.TaskInfo
 
attemptNumber() - Method in class org.apache.spark.TaskCommitDenied
 
attemptNumber() - Method in class org.apache.spark.TaskContext
How many times this task has been attempted.
attempts() - Method in class org.apache.spark.status.api.v1.ApplicationInfo
 
AtTimestamp(Date) - Constructor for class org.apache.spark.streaming.kinesis.KinesisInitialPositions.AtTimestamp
 
attr() - Method in class org.apache.spark.graphx.Edge
 
attr() - Method in class org.apache.spark.graphx.EdgeContext
The attribute associated with the edge.
attr() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
Attribute - Class in org.apache.spark.ml.attribute
Developer API Abstract class for ML attributes.
Attribute() - Constructor for class org.apache.spark.ml.attribute.Attribute
 
attribute() - Method in class org.apache.spark.sql.sources.EqualNullSafe
 
attribute() - Method in class org.apache.spark.sql.sources.EqualTo
 
attribute() - Method in class org.apache.spark.sql.sources.GreaterThan
 
attribute() - Method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
attribute() - Method in class org.apache.spark.sql.sources.In
 
attribute() - Method in class org.apache.spark.sql.sources.IsNotNull
 
attribute() - Method in class org.apache.spark.sql.sources.IsNull
 
attribute() - Method in class org.apache.spark.sql.sources.LessThan
 
attribute() - Method in class org.apache.spark.sql.sources.LessThanOrEqual
 
attribute() - Method in class org.apache.spark.sql.sources.StringContains
 
attribute() - Method in class org.apache.spark.sql.sources.StringEndsWith
 
attribute() - Method in class org.apache.spark.sql.sources.StringStartsWith
 
AttributeGroup - Class in org.apache.spark.ml.attribute
Developer API Attributes that describe a vector ML column.
AttributeGroup(String) - Constructor for class org.apache.spark.ml.attribute.AttributeGroup
Creates an attribute group without attribute info.
AttributeGroup(String, int) - Constructor for class org.apache.spark.ml.attribute.AttributeGroup
Creates an attribute group knowing only the number of attributes.
AttributeGroup(String, Attribute[]) - Constructor for class org.apache.spark.ml.attribute.AttributeGroup
Creates an attribute group with attributes.
AttributeKeys - Class in org.apache.spark.ml.attribute
Keys used to store attributes.
AttributeKeys() - Constructor for class org.apache.spark.ml.attribute.AttributeKeys
 
attributes() - Method in class org.apache.spark.ml.attribute.AttributeGroup
Optional array of attributes.
ATTRIBUTES() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
AttributeType - Class in org.apache.spark.ml.attribute
Developer API An enum-like type for attribute types: AttributeType$.Numeric, AttributeType$.Nominal, and AttributeType$.Binary.
AttributeType(String) - Constructor for class org.apache.spark.ml.attribute.AttributeType
 
attrType() - Method in class org.apache.spark.ml.attribute.Attribute
Attribute type.
attrType() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
attrType() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
attrType() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
attrType() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
available() - Method in class org.apache.spark.io.NioBufferedFileInputStream
 
available() - Method in class org.apache.spark.io.ReadAheadInputStream
 
available() - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
Average() - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
avg(MapFunction<T, Double>) - Static method in class org.apache.spark.sql.expressions.javalang.typed
Average aggregate function.
avg(Function1<IN, Object>) - Static method in class org.apache.spark.sql.expressions.scalalang.typed
Average aggregate function.
avg(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the average of the values in a group.
avg(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the average of the values in a group.
avg(String...) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the mean value for each numeric columns for each group.
avg(Seq<String>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the mean value for each numeric columns for each group.
avg() - Method in class org.apache.spark.util.DoubleAccumulator
Returns the average of elements added to the accumulator.
avg() - Method in class org.apache.spark.util.LongAccumulator
Returns the average of elements added to the accumulator.
avgEventRate() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
avgInputRate() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
avgMetrics() - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
avgProcessingTime() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
avgSchedulingDelay() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
avgTotalDelay() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
awaitAnyTermination() - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
Wait until any of the queries on the associated SQLContext has terminated since the creation of the context, or since resetTerminated() was called.
awaitAnyTermination(long) - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
Wait until any of the queries on the associated SQLContext has terminated since the creation of the context, or since resetTerminated() was called.
awaitReady(Awaitable<T>, Duration) - Static method in class org.apache.spark.util.ThreadUtils
Preferred alternative to Await.ready().
awaitResult(Awaitable<T>, Duration) - Static method in class org.apache.spark.util.ThreadUtils
Preferred alternative to Await.result().
awaitTermination() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Waits for the termination of this query, either by query.stop() or by an exception.
awaitTermination(long) - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Waits for the termination of this query, either by query.stop() or by an exception.
awaitTermination() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Wait for the execution to stop.
awaitTermination() - Method in class org.apache.spark.streaming.StreamingContext
Wait for the execution to stop.
awaitTerminationOrTimeout(long) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Wait for the execution to stop.
awaitTerminationOrTimeout(long) - Method in class org.apache.spark.streaming.StreamingContext
Wait for the execution to stop.
axpy(double, Vector, Vector) - Static method in class org.apache.spark.ml.linalg.BLAS
y += a * x
axpy(double, Vector, Vector) - Static method in class org.apache.spark.mllib.linalg.BLAS
y += a * x

B

BACKUP_STANDALONE_MASTER_PREFIX() - Static method in class org.apache.spark.util.Utils
An identifier that backup masters use in their responses.
balanceSlack() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
base64(Column) - Static method in class org.apache.spark.sql.functions
Computes the BASE64 encoding of a binary column and returns it as a string column.
baseOn(ParamPair<?>...) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Sets the given parameters in this grid to fixed values.
baseOn(ParamMap) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Sets the given parameters in this grid to fixed values.
baseOn(Seq<ParamPair<?>>) - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Sets the given parameters in this grid to fixed values.
BaseRelation - Class in org.apache.spark.sql.sources
Represents a collection of tuples with a known schema.
BaseRelation() - Constructor for class org.apache.spark.sql.sources.BaseRelation
 
baseRelationToDataFrame(BaseRelation) - Method in class org.apache.spark.sql.SparkSession
Convert a BaseRelation created for external data sources into a DataFrame.
baseRelationToDataFrame(BaseRelation) - Method in class org.apache.spark.sql.SQLContext
 
BaseRRDD<T,U> - Class in org.apache.spark.api.r
 
BaseRRDD(RDD<T>, int, byte[], String, String, byte[], Broadcast<Object>[], ClassTag<T>, ClassTag<U>) - Constructor for class org.apache.spark.api.r.BaseRRDD
 
BasicBlockReplicationPolicy - Class in org.apache.spark.storage
 
BasicBlockReplicationPolicy() - Constructor for class org.apache.spark.storage.BasicBlockReplicationPolicy
 
basicSparkPage(Function0<Seq<Node>>, String, boolean) - Static method in class org.apache.spark.ui.UIUtils
Returns a page with the spark css/js and a simple format.
basicWriteJobStatsTracker(Configuration) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
basicWriteJobStatsTracker(Configuration) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
basicWriteJobStatsTracker(Configuration) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
batchDuration() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
batchDuration() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
BATCHES() - Static method in class org.apache.spark.mllib.clustering.StreamingKMeans
 
batchId() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
batchId() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
BatchInfo - Class in org.apache.spark.status.api.v1.streaming
 
BatchInfo - Class in org.apache.spark.streaming.scheduler
Developer API Class having information on completed batches.
BatchInfo(Time, Map<Object, StreamInputInfo>, long, Option<Object>, Option<Object>, Map<Object, OutputOperationInfo>) - Constructor for class org.apache.spark.streaming.scheduler.BatchInfo
 
batchInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
batchInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
batchInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
batchInfos() - Method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
BatchStatus - Enum in org.apache.spark.status.api.v1.streaming
 
batchTime() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
batchTime() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
 
batchTime() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
bean(Class<T>) - Static method in class org.apache.spark.sql.Encoders
Creates an encoder for Java Bean of type T.
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
beforeFetch(Connection, Map<String, String>) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Override connection specific properties to run before a select is made.
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
beforeFetch(Connection, Map<String, String>) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
BernoulliCellSampler<T> - Class in org.apache.spark.util.random
Developer API A sampler based on Bernoulli trials for partitioning a data sequence.
BernoulliCellSampler(double, double, boolean) - Constructor for class org.apache.spark.util.random.BernoulliCellSampler
 
BernoulliSampler<T> - Class in org.apache.spark.util.random
Developer API A sampler based on Bernoulli trials.
BernoulliSampler(double, ClassTag<T>) - Constructor for class org.apache.spark.util.random.BernoulliSampler
 
bestModel() - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
bestModel() - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
beta() - Method in class org.apache.spark.mllib.random.WeibullGenerator
 
between(Object, Object) - Method in class org.apache.spark.sql.Column
True if the current column is between the lower bound and upper bound, inclusive.
bin(Column) - Static method in class org.apache.spark.sql.functions
An expression that returns the string representation of the binary value of the given long column.
bin(String) - Static method in class org.apache.spark.sql.functions
An expression that returns the string representation of the binary value of the given long column.
Binarizer - Class in org.apache.spark.ml.feature
Binarize a column of continuous features given a threshold.
Binarizer(String) - Constructor for class org.apache.spark.ml.feature.Binarizer
 
Binarizer() - Constructor for class org.apache.spark.ml.feature.Binarizer
 
Binary() - Static method in class org.apache.spark.ml.attribute.AttributeType
Binary type.
binary() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
binary() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
binary() - Method in class org.apache.spark.ml.feature.HashingTF
Binary toggle to control term frequency counts.
binary() - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type binary.
BINARY() - Static method in class org.apache.spark.sql.Encoders
An encoder for arrays of bytes.
BinaryAttribute - Class in org.apache.spark.ml.attribute
Developer API A binary attribute.
BinaryClassificationEvaluator - Class in org.apache.spark.ml.evaluation
Experimental Evaluator for binary classification, which expects two input columns: rawPrediction and label.
BinaryClassificationEvaluator(String) - Constructor for class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
BinaryClassificationEvaluator() - Constructor for class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
BinaryClassificationMetrics - Class in org.apache.spark.mllib.evaluation
Evaluator for binary classification.
BinaryClassificationMetrics(RDD<Tuple2<Object, Object>>, int) - Constructor for class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
 
BinaryClassificationMetrics(RDD<Tuple2<Object, Object>>) - Constructor for class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Defaults numBins to 0.
binaryFiles(String, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Read a directory of binary files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI as a byte array.
binaryFiles(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Read a directory of binary files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI as a byte array.
binaryFiles(String, int) - Method in class org.apache.spark.SparkContext
Get an RDD for a Hadoop-readable dataset as PortableDataStream for each file (useful for binary data)
binaryLabelValidator() - Static method in class org.apache.spark.mllib.util.DataValidators
Function to check if labels used for classification are either zero or one.
BinaryLogisticRegressionSummary - Interface in org.apache.spark.ml.classification
Experimental Abstraction for binary logistic regression results for a given model.
BinaryLogisticRegressionSummaryImpl - Class in org.apache.spark.ml.classification
Binary logistic regression results for a given model.
BinaryLogisticRegressionSummaryImpl(Dataset<Row>, String, String, String, String) - Constructor for class org.apache.spark.ml.classification.BinaryLogisticRegressionSummaryImpl
 
BinaryLogisticRegressionTrainingSummary - Interface in org.apache.spark.ml.classification
Experimental Abstraction for binary logistic regression training results.
BinaryLogisticRegressionTrainingSummaryImpl - Class in org.apache.spark.ml.classification
Binary logistic regression training results.
BinaryLogisticRegressionTrainingSummaryImpl(Dataset<Row>, String, String, String, String, double[]) - Constructor for class org.apache.spark.ml.classification.BinaryLogisticRegressionTrainingSummaryImpl
 
binaryMetrics() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
 
binaryRecords(String, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Load data from a flat binary file, assuming the length of each record is constant.
binaryRecords(String, int, Configuration) - Method in class org.apache.spark.SparkContext
Load data from a flat binary file, assuming the length of each record is constant.
binaryRecordsStream(String, int) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as flat binary files with fixed record lengths, yielding byte arrays
binaryRecordsStream(String, int) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as flat binary files, assuming a fixed length per record, generating one byte array per record.
BinarySample - Class in org.apache.spark.mllib.stat.test
Class that represents the group and value of a sample.
BinarySample(boolean, double) - Constructor for class org.apache.spark.mllib.stat.test.BinarySample
 
binarySummary() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
Gets summary of model on training set.
BinaryType - Class in org.apache.spark.sql.types
The data type representing Array[Byte] values.
BinaryType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the BinaryType object.
Binomial$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
 
BinomialBounds - Class in org.apache.spark.util.random
Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact sample size with high confidence when sampling without replacement.
BinomialBounds() - Constructor for class org.apache.spark.util.random.BinomialBounds
 
BisectingKMeans - Class in org.apache.spark.ml.clustering
A bisecting k-means algorithm based on the paper "A comparison of document clustering techniques" by Steinbach, Karypis, and Kumar, with modification to fit Spark.
BisectingKMeans(String) - Constructor for class org.apache.spark.ml.clustering.BisectingKMeans
 
BisectingKMeans() - Constructor for class org.apache.spark.ml.clustering.BisectingKMeans
 
BisectingKMeans - Class in org.apache.spark.mllib.clustering
A bisecting k-means algorithm based on the paper "A comparison of document clustering techniques" by Steinbach, Karypis, and Kumar, with modification to fit Spark.
BisectingKMeans() - Constructor for class org.apache.spark.mllib.clustering.BisectingKMeans
Constructs with the default configuration
BisectingKMeansModel - Class in org.apache.spark.ml.clustering
Model fitted by BisectingKMeans.
BisectingKMeansModel - Class in org.apache.spark.mllib.clustering
Clustering model produced by BisectingKMeans.
BisectingKMeansModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.clustering
 
BisectingKMeansSummary - Class in org.apache.spark.ml.clustering
Experimental Summary of BisectingKMeans.
bitSize() - Method in class org.apache.spark.util.sketch.BloomFilter
Returns the number of bits in the underlying bit array.
bitwiseAND(Object) - Method in class org.apache.spark.sql.Column
Compute bitwise AND of this expression with another expression.
bitwiseNOT(Column) - Static method in class org.apache.spark.sql.functions
Computes bitwise NOT.
bitwiseOR(Object) - Method in class org.apache.spark.sql.Column
Compute bitwise OR of this expression with another expression.
bitwiseXOR(Object) - Method in class org.apache.spark.sql.Column
Compute bitwise XOR of this expression with another expression.
BLACKLISTED() - Static method in class org.apache.spark.ui.ToolTips
 
BlacklistedExecutor - Class in org.apache.spark.scheduler
 
BlacklistedExecutor(String, long) - Constructor for class org.apache.spark.scheduler.BlacklistedExecutor
 
BLAS - Class in org.apache.spark.ml.linalg
BLAS routines for MLlib's vectors and matrices.
BLAS() - Constructor for class org.apache.spark.ml.linalg.BLAS
 
BLAS - Class in org.apache.spark.mllib.linalg
BLAS routines for MLlib's vectors and matrices.
BLAS() - Constructor for class org.apache.spark.mllib.linalg.BLAS
 
blockData() - Method in class org.apache.spark.storage.EncryptedManagedBuffer
 
BlockId - Class in org.apache.spark.storage
Developer API Identifies a particular Block of data, usually associated with a single file.
BlockId() - Constructor for class org.apache.spark.storage.BlockId
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetLocations
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetLocationsAndStatus
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveBlock
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.ReplicateBlock
 
blockId() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
blockId() - Method in class org.apache.spark.storage.BlockUpdatedInfo
 
blockIds() - Method in class org.apache.spark.storage.BlockManagerMessages.GetLocationsMultipleBlockIds
 
BlockLocationsAndStatus(Seq<BlockManagerId>, BlockStatus) - Constructor for class org.apache.spark.storage.BlockManagerMessages.BlockLocationsAndStatus
 
BlockLocationsAndStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.BlockLocationsAndStatus$
 
blockManager() - Method in class org.apache.spark.SparkEnv
 
blockManagerAddedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockManagerAddedToJson(SparkListenerBlockManagerAdded) - Static method in class org.apache.spark.util.JsonProtocol
 
BlockManagerHeartbeat(BlockManagerId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat
 
BlockManagerHeartbeat$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat$
 
blockManagerId() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
blockManagerId() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
BlockManagerId - Class in org.apache.spark.storage
Developer API This class represent an unique identifier for a BlockManager.
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetPeers
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager
 
blockManagerId() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
blockManagerId() - Method in class org.apache.spark.storage.BlockUpdatedInfo
 
blockManagerId() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
 
blockManagerIdCache() - Static method in class org.apache.spark.storage.BlockManagerId
The max cache size is hardcoded to 10000, since the size of a BlockManagerId object is about 48B, the total memory cost should be below 1MB which is feasible.
blockManagerIdFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockManagerIdToJson(BlockManagerId) - Static method in class org.apache.spark.util.JsonProtocol
 
BlockManagerMessages - Class in org.apache.spark.storage
 
BlockManagerMessages() - Constructor for class org.apache.spark.storage.BlockManagerMessages
 
BlockManagerMessages.BlockLocationsAndStatus - Class in org.apache.spark.storage
 
BlockManagerMessages.BlockLocationsAndStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.BlockManagerHeartbeat - Class in org.apache.spark.storage
 
BlockManagerMessages.BlockManagerHeartbeat$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetBlockStatus - Class in org.apache.spark.storage
 
BlockManagerMessages.GetBlockStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetExecutorEndpointRef - Class in org.apache.spark.storage
 
BlockManagerMessages.GetExecutorEndpointRef$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocations - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocations$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocationsAndStatus - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocationsAndStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocationsMultipleBlockIds - Class in org.apache.spark.storage
 
BlockManagerMessages.GetLocationsMultipleBlockIds$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetMatchingBlockIds - Class in org.apache.spark.storage
 
BlockManagerMessages.GetMatchingBlockIds$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetMemoryStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetPeers - Class in org.apache.spark.storage
 
BlockManagerMessages.GetPeers$ - Class in org.apache.spark.storage
 
BlockManagerMessages.GetStorageStatus$ - Class in org.apache.spark.storage
 
BlockManagerMessages.HasCachedBlocks - Class in org.apache.spark.storage
 
BlockManagerMessages.HasCachedBlocks$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RegisterBlockManager - Class in org.apache.spark.storage
 
BlockManagerMessages.RegisterBlockManager$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBlock - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBlock$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBroadcast - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveBroadcast$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveExecutor - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveExecutor$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveRdd - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveRdd$ - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveShuffle - Class in org.apache.spark.storage
 
BlockManagerMessages.RemoveShuffle$ - Class in org.apache.spark.storage
 
BlockManagerMessages.ReplicateBlock - Class in org.apache.spark.storage
 
BlockManagerMessages.ReplicateBlock$ - Class in org.apache.spark.storage
 
BlockManagerMessages.StopBlockManagerMaster$ - Class in org.apache.spark.storage
 
BlockManagerMessages.ToBlockManagerMaster - Interface in org.apache.spark.storage
 
BlockManagerMessages.ToBlockManagerSlave - Interface in org.apache.spark.storage
 
BlockManagerMessages.TriggerThreadDump$ - Class in org.apache.spark.storage
Driver to Executor message to trigger a thread dump.
BlockManagerMessages.UpdateBlockInfo - Class in org.apache.spark.storage
 
BlockManagerMessages.UpdateBlockInfo$ - Class in org.apache.spark.storage
 
blockManagerRemovedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockManagerRemovedToJson(SparkListenerBlockManagerRemoved) - Static method in class org.apache.spark.util.JsonProtocol
 
BlockMatrix - Class in org.apache.spark.mllib.linalg.distributed
Represents a distributed matrix in blocks of local matrices.
BlockMatrix(RDD<Tuple2<Tuple2<Object, Object>, Matrix>>, int, int, long, long) - Constructor for class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
BlockMatrix(RDD<Tuple2<Tuple2<Object, Object>, Matrix>>, int, int) - Constructor for class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Alternate constructor for BlockMatrix without the input of the number of rows and columns.
blockName() - Method in class org.apache.spark.status.api.v1.RDDPartitionInfo
 
blockName() - Method in class org.apache.spark.status.LiveRDDPartition
 
BlockNotFoundException - Exception in org.apache.spark.storage
 
BlockNotFoundException(String) - Constructor for exception org.apache.spark.storage.BlockNotFoundException
 
BlockReplicationPolicy - Interface in org.apache.spark.storage
::DeveloperApi:: BlockReplicationPrioritization provides logic for prioritizing a sequence of peers for replicating blocks.
BlockReplicationUtils - Class in org.apache.spark.storage
 
BlockReplicationUtils() - Constructor for class org.apache.spark.storage.BlockReplicationUtils
 
blocks() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
blocks() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the blocks stored in this block manager.
blockSize() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
BlockStatus - Class in org.apache.spark.storage
 
BlockStatus(StorageLevel, long, long) - Constructor for class org.apache.spark.storage.BlockStatus
 
blockStatusFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockStatusToJson(BlockStatus) - Static method in class org.apache.spark.util.JsonProtocol
 
blockUpdatedInfo() - Method in class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
BlockUpdatedInfo - Class in org.apache.spark.storage
Developer API Stores information about a block status in a block manager.
BlockUpdatedInfo(BlockManagerId, BlockId, StorageLevel, long, long) - Constructor for class org.apache.spark.storage.BlockUpdatedInfo
 
blockUpdatedInfoFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockUpdatedInfoToJson(BlockUpdatedInfo) - Static method in class org.apache.spark.util.JsonProtocol
 
blockUpdateFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
blockUpdateToJson(SparkListenerBlockUpdated) - Static method in class org.apache.spark.util.JsonProtocol
 
bloomFilter(String, long, double) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Bloom filter over a specified column.
bloomFilter(Column, long, double) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Bloom filter over a specified column.
bloomFilter(String, long, long) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Bloom filter over a specified column.
bloomFilter(Column, long, long) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Bloom filter over a specified column.
BloomFilter - Class in org.apache.spark.util.sketch
A Bloom filter is a space-efficient probabilistic data structure that offers an approximate containment test with one-sided error: if it claims that an item is contained in it, this might be in error, but if it claims that an item is not contained in it, then this is definitely true.
BloomFilter() - Constructor for class org.apache.spark.util.sketch.BloomFilter
 
BloomFilter.Version - Enum in org.apache.spark.util.sketch
 
bmAddress() - Method in class org.apache.spark.FetchFailed
 
BOOLEAN() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable boolean type.
BooleanParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Boolean] for Java.
BooleanParam(String, String, String) - Constructor for class org.apache.spark.ml.param.BooleanParam
 
BooleanParam(Identifiable, String, String) - Constructor for class org.apache.spark.ml.param.BooleanParam
 
BooleanType - Class in org.apache.spark.sql.types
The data type representing Boolean values.
BooleanType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the BooleanType object.
boost(RDD<LabeledPoint>, RDD<LabeledPoint>, BoostingStrategy, boolean, long, String) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Internal method for performing regression using trees as base learners.
BoostingStrategy - Class in org.apache.spark.mllib.tree.configuration
Configuration options for GradientBoostedTrees.
BoostingStrategy(Strategy, Loss, int, double, double) - Constructor for class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
Both() - Static method in class org.apache.spark.graphx.EdgeDirection
Edges originating from *and* arriving at a vertex of interest.
boundaries() - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
Boundaries in increasing order for which predictions are known.
boundaries() - Method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
 
BoundedDouble - Class in org.apache.spark.partial
A Double value with error bars and associated confidence.
BoundedDouble(double, double, double, double) - Constructor for class org.apache.spark.partial.BoundedDouble
 
BreezeUtil - Class in org.apache.spark.ml.ann
In-place DGEMM and DGEMV for Breeze
BreezeUtil() - Constructor for class org.apache.spark.ml.ann.BreezeUtil
 
broadcast(T) - Method in class org.apache.spark.api.java.JavaSparkContext
Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions.
Broadcast<T> - Class in org.apache.spark.broadcast
A broadcast variable.
Broadcast(long, ClassTag<T>) - Constructor for class org.apache.spark.broadcast.Broadcast
 
broadcast(T, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Broadcast a read-only variable to the cluster, returning a Broadcast object for reading it in distributed functions.
broadcast(Dataset<T>) - Static method in class org.apache.spark.sql.functions
Marks a DataFrame as small enough for use in broadcast joins.
BROADCAST() - Static method in class org.apache.spark.storage.BlockId
 
BroadcastBlockId - Class in org.apache.spark.storage
 
BroadcastBlockId(long, String) - Constructor for class org.apache.spark.storage.BroadcastBlockId
 
broadcastId() - Method in class org.apache.spark.CleanBroadcast
 
broadcastId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast
 
broadcastId() - Method in class org.apache.spark.storage.BroadcastBlockId
 
broadcastManager() - Method in class org.apache.spark.SparkEnv
 
bround(Column) - Static method in class org.apache.spark.sql.functions
Returns the value of the column e rounded to 0 decimal places with HALF_EVEN round mode.
bround(Column, int) - Static method in class org.apache.spark.sql.functions
Round the value of e to scale decimal places with HALF_EVEN round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
bucketBy(int, String, String...) - Method in class org.apache.spark.sql.DataFrameWriter
Buckets the output by the given columns.
bucketBy(int, String, Seq<String>) - Method in class org.apache.spark.sql.DataFrameWriter
Buckets the output by the given columns.
BucketedRandomProjectionLSH - Class in org.apache.spark.ml.feature
Experimental
BucketedRandomProjectionLSH(String) - Constructor for class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
BucketedRandomProjectionLSH() - Constructor for class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
BucketedRandomProjectionLSHModel - Class in org.apache.spark.ml.feature
Experimental
Bucketizer - Class in org.apache.spark.ml.feature
Bucketizer maps a column of continuous features to a column of feature buckets.
Bucketizer(String) - Constructor for class org.apache.spark.ml.feature.Bucketizer
 
Bucketizer() - Constructor for class org.apache.spark.ml.feature.Bucketizer
 
bucketLength() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
bucketLength() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
buf() - Method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
buffer() - Method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
bufferEncoder() - Method in class org.apache.spark.sql.expressions.Aggregator
Specifies the Encoder for the intermediate value type.
BufferReleasingInputStream - Class in org.apache.spark.storage
Helper class that ensures a ManagedBuffer is released upon InputStream.close()
BufferReleasingInputStream(InputStream, ShuffleBlockFetcherIterator) - Constructor for class org.apache.spark.storage.BufferReleasingInputStream
 
bufferSchema() - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
A StructType represents data types of values in the aggregation buffer.
build(Node, int) - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData$
Create DecisionTreeModelReadWrite.NodeData instances for this node and all children.
build(DecisionTreeModel, int) - Method in class org.apache.spark.ml.tree.EnsembleModelReadWrite.EnsembleNodeData$
Create EnsembleModelReadWrite.EnsembleNodeData instances for the given tree.
build() - Method in class org.apache.spark.ml.tuning.ParamGridBuilder
Builds and returns all combinations of parameters specified by the param grid.
build() - Static method in class org.apache.spark.sql.hive.HiveSessionStateBuilder
 
build() - Method in class org.apache.spark.sql.types.MetadataBuilder
Builds the Metadata instance.
builder() - Static method in class org.apache.spark.sql.SparkSession
Creates a SparkSession.Builder for constructing a SparkSession.
Builder() - Constructor for class org.apache.spark.sql.SparkSession.Builder
 
buildReader(SparkSession, StructType, StructType, StructType, Seq<Filter>, Map<String, String>, Configuration) - Method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
buildReaderWithPartitionValues(SparkSession, StructType, StructType, StructType, Seq<Filter>, Map<String, String>, Configuration) - Static method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
buildScan(Seq<Attribute>, Seq<Expression>) - Method in interface org.apache.spark.sql.sources.CatalystScan
 
buildScan(String[], Filter[]) - Method in interface org.apache.spark.sql.sources.PrunedFilteredScan
 
buildScan(String[]) - Method in interface org.apache.spark.sql.sources.PrunedScan
 
buildScan() - Method in interface org.apache.spark.sql.sources.TableScan
 
buildTreeFromNodes(DecisionTreeModelReadWrite.NodeData[], String) - Static method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite
Given all data for all nodes in a tree, rebuild the tree.
builtinHiveVersion() - Static method in class org.apache.spark.sql.hive.HiveUtils
The version of hive used internally by Spark SQL.
BYTE() - Static method in class org.apache.spark.api.r.SerializationFormats
 
BYTE() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable byte type.
BytecodeUtils - Class in org.apache.spark.graphx.util
Includes an utility function to test whether a function accesses a specific attribute of an object.
BytecodeUtils() - Constructor for class org.apache.spark.graphx.util.BytecodeUtils
 
byteFromString(String, ByteUnit) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
BYTES_READ() - Method in class org.apache.spark.InternalAccumulator.input$
 
BYTES_WRITTEN() - Method in class org.apache.spark.InternalAccumulator.output$
 
BYTES_WRITTEN() - Method in class org.apache.spark.InternalAccumulator.shuffleWrite$
 
bytesRead() - Method in class org.apache.spark.status.api.v1.InputMetricDistributions
 
bytesRead() - Method in class org.apache.spark.status.api.v1.InputMetrics
 
bytesToString(long) - Static method in class org.apache.spark.util.Utils
Convert a quantity in bytes to a human-readable string such as "4.0 MB".
bytesToString(BigInt) - Static method in class org.apache.spark.util.Utils
 
byteStringAsBytes(String) - Static method in class org.apache.spark.util.Utils
Convert a passed byte string (e.g.
byteStringAsGb(String) - Static method in class org.apache.spark.util.Utils
Convert a passed byte string (e.g.
byteStringAsKb(String) - Static method in class org.apache.spark.util.Utils
Convert a passed byte string (e.g.
byteStringAsMb(String) - Static method in class org.apache.spark.util.Utils
Convert a passed byte string (e.g.
bytesWritten() - Method in class org.apache.spark.status.api.v1.OutputMetricDistributions
 
bytesWritten() - Method in class org.apache.spark.status.api.v1.OutputMetrics
 
bytesWritten() - Method in class org.apache.spark.status.api.v1.ShuffleWriteMetrics
 
byteToString(long, ByteUnit) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
ByteType - Class in org.apache.spark.sql.types
The data type representing Byte values.
ByteType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the ByteType object.

C

cache() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Persist this RDD with the default storage level (MEMORY_ONLY).
cache() - Method in class org.apache.spark.api.java.JavaPairRDD
Persist this RDD with the default storage level (MEMORY_ONLY).
cache() - Method in class org.apache.spark.api.java.JavaRDD
Persist this RDD with the default storage level (MEMORY_ONLY).
cache() - Static method in class org.apache.spark.api.r.RRDD
 
cache() - Static method in class org.apache.spark.graphx.EdgeRDD
 
cache() - Method in class org.apache.spark.graphx.Graph
Caches the vertices and edges associated with this graph at the previously-specified target storage levels, which default to MEMORY_ONLY.
cache() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
Persists the edge partitions using targetStorageLevel, which defaults to MEMORY_ONLY.
cache() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
cache() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
Persists the vertex partitions at targetStorageLevel, which defaults to MEMORY_ONLY.
cache() - Static method in class org.apache.spark.graphx.VertexRDD
 
cache() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Caches the underlying RDD.
cache() - Static method in class org.apache.spark.rdd.HadoopRDD
 
cache() - Static method in class org.apache.spark.rdd.JdbcRDD
 
cache() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
cache() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
cache() - Method in class org.apache.spark.rdd.RDD
Persist this RDD with the default storage level (MEMORY_ONLY).
cache() - Static method in class org.apache.spark.rdd.UnionRDD
 
cache() - Method in class org.apache.spark.sql.Dataset
Persist this Dataset with the default storage level (MEMORY_AND_DISK).
cache() - Method in class org.apache.spark.streaming.api.java.JavaDStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
cache() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
cache() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
cache() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
cache() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
cache() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
cache() - Method in class org.apache.spark.streaming.dstream.DStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
cacheNodeIds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
cacheNodeIds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
cacheNodeIds() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
cacheNodeIds() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
cacheNodeIds() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
cacheNodeIds() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
cacheNodeIds() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
cacheNodeIds() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
cacheNodeIds() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
cacheNodeIds() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
cacheNodeIds() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
cacheNodeIds() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
cacheSize() - Method in interface org.apache.spark.SparkExecutorInfo
 
cacheSize() - Method in class org.apache.spark.SparkExecutorInfoImpl
 
cacheSize() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the memory used by caching RDDs
cacheTable(String) - Method in class org.apache.spark.sql.catalog.Catalog
Caches the specified table in-memory.
cacheTable(String, StorageLevel) - Method in class org.apache.spark.sql.catalog.Catalog
Caches the specified table with the given storage level.
cacheTable(String) - Method in class org.apache.spark.sql.SQLContext
Caches the specified table in-memory.
calculate(DenseVector<Object>) - Method in class org.apache.spark.ml.regression.AFTCostFun
 
calculate(double[], double) - Static method in class org.apache.spark.mllib.tree.impurity.Entropy
Developer API information calculation for multiclass classification
calculate(double, double, double) - Static method in class org.apache.spark.mllib.tree.impurity.Entropy
Developer API variance calculation
calculate(double[], double) - Static method in class org.apache.spark.mllib.tree.impurity.Gini
Developer API information calculation for multiclass classification
calculate(double, double, double) - Static method in class org.apache.spark.mllib.tree.impurity.Gini
Developer API variance calculation
calculate(double[], double) - Method in interface org.apache.spark.mllib.tree.impurity.Impurity
Developer API information calculation for multiclass classification
calculate(double, double, double) - Method in interface org.apache.spark.mllib.tree.impurity.Impurity
Developer API information calculation for regression
calculate(double[], double) - Static method in class org.apache.spark.mllib.tree.impurity.Variance
Developer API information calculation for multiclass classification
calculate(double, double, double) - Static method in class org.apache.spark.mllib.tree.impurity.Variance
Developer API variance calculation
calculateNumberOfPartitions(long, int, int) - Method in class org.apache.spark.ml.feature.Word2VecModel.Word2VecModelWriter$
Calculate the number of partitions to use in saving the model.
CalendarIntervalType - Class in org.apache.spark.sql.types
The data type representing calendar time intervals.
CalendarIntervalType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the CalendarIntervalType object.
call(K, Iterator<V1>, Iterator<V2>) - Method in interface org.apache.spark.api.java.function.CoGroupFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.DoubleFlatMapFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.DoubleFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.FilterFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.FlatMapFunction
 
call(T1, T2) - Method in interface org.apache.spark.api.java.function.FlatMapFunction2
 
call(K, Iterator<V>) - Method in interface org.apache.spark.api.java.function.FlatMapGroupsFunction
 
call(K, Iterator<V>, GroupState<S>) - Method in interface org.apache.spark.api.java.function.FlatMapGroupsWithStateFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.ForeachFunction
 
call(Iterator<T>) - Method in interface org.apache.spark.api.java.function.ForeachPartitionFunction
 
call(T1) - Method in interface org.apache.spark.api.java.function.Function
 
call() - Method in interface org.apache.spark.api.java.function.Function0
 
call(T1, T2) - Method in interface org.apache.spark.api.java.function.Function2
 
call(T1, T2, T3) - Method in interface org.apache.spark.api.java.function.Function3
 
call(T1, T2, T3, T4) - Method in interface org.apache.spark.api.java.function.Function4
 
call(T) - Method in interface org.apache.spark.api.java.function.MapFunction
 
call(K, Iterator<V>) - Method in interface org.apache.spark.api.java.function.MapGroupsFunction
 
call(K, Iterator<V>, GroupState<S>) - Method in interface org.apache.spark.api.java.function.MapGroupsWithStateFunction
 
call(Iterator<T>) - Method in interface org.apache.spark.api.java.function.MapPartitionsFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.PairFlatMapFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.PairFunction
 
call(T, T) - Method in interface org.apache.spark.api.java.function.ReduceFunction
 
call(T) - Method in interface org.apache.spark.api.java.function.VoidFunction
 
call(T1, T2) - Method in interface org.apache.spark.api.java.function.VoidFunction2
 
call() - Method in interface org.apache.spark.sql.api.java.UDF0
 
call(T1) - Method in interface org.apache.spark.sql.api.java.UDF1
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10) - Method in interface org.apache.spark.sql.api.java.UDF10
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11) - Method in interface org.apache.spark.sql.api.java.UDF11
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12) - Method in interface org.apache.spark.sql.api.java.UDF12
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13) - Method in interface org.apache.spark.sql.api.java.UDF13
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14) - Method in interface org.apache.spark.sql.api.java.UDF14
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15) - Method in interface org.apache.spark.sql.api.java.UDF15
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16) - Method in interface org.apache.spark.sql.api.java.UDF16
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17) - Method in interface org.apache.spark.sql.api.java.UDF17
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18) - Method in interface org.apache.spark.sql.api.java.UDF18
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19) - Method in interface org.apache.spark.sql.api.java.UDF19
 
call(T1, T2) - Method in interface org.apache.spark.sql.api.java.UDF2
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20) - Method in interface org.apache.spark.sql.api.java.UDF20
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20, T21) - Method in interface org.apache.spark.sql.api.java.UDF21
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15, T16, T17, T18, T19, T20, T21, T22) - Method in interface org.apache.spark.sql.api.java.UDF22
 
call(T1, T2, T3) - Method in interface org.apache.spark.sql.api.java.UDF3
 
call(T1, T2, T3, T4) - Method in interface org.apache.spark.sql.api.java.UDF4
 
call(T1, T2, T3, T4, T5) - Method in interface org.apache.spark.sql.api.java.UDF5
 
call(T1, T2, T3, T4, T5, T6) - Method in interface org.apache.spark.sql.api.java.UDF6
 
call(T1, T2, T3, T4, T5, T6, T7) - Method in interface org.apache.spark.sql.api.java.UDF7
 
call(T1, T2, T3, T4, T5, T6, T7, T8) - Method in interface org.apache.spark.sql.api.java.UDF8
 
call(T1, T2, T3, T4, T5, T6, T7, T8, T9) - Method in interface org.apache.spark.sql.api.java.UDF9
 
callSite() - Method in class org.apache.spark.storage.RDDInfo
 
callUDF(String, Column...) - Static method in class org.apache.spark.sql.functions
Call an user-defined function.
callUDF(String, Seq<Column>) - Static method in class org.apache.spark.sql.functions
Call an user-defined function.
cancel() - Method in class org.apache.spark.ComplexFutureAction
 
cancel() - Method in interface org.apache.spark.FutureAction
Cancels the execution of this action.
cancel() - Method in class org.apache.spark.SimpleFutureAction
 
cancelAllJobs() - Method in class org.apache.spark.api.java.JavaSparkContext
Cancel all jobs that have been scheduled or are running.
cancelAllJobs() - Method in class org.apache.spark.SparkContext
Cancel all jobs that have been scheduled or are running.
cancelJob(int, String) - Method in class org.apache.spark.SparkContext
Cancel a given job if it's scheduled or running.
cancelJob(int) - Method in class org.apache.spark.SparkContext
Cancel a given job if it's scheduled or running.
cancelJobGroup(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Cancel active jobs for the specified group.
cancelJobGroup(String) - Method in class org.apache.spark.SparkContext
Cancel active jobs for the specified group.
cancelStage(int, String) - Method in class org.apache.spark.SparkContext
Cancel a given stage and all jobs associated with it.
cancelStage(int) - Method in class org.apache.spark.SparkContext
Cancel a given stage and all jobs associated with it.
canDoMerge() - Method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
canEqual(Object) - Static method in class org.apache.spark.Aggregator
 
canEqual(Object) - Static method in class org.apache.spark.CleanAccum
 
canEqual(Object) - Static method in class org.apache.spark.CleanBroadcast
 
canEqual(Object) - Static method in class org.apache.spark.CleanCheckpoint
 
canEqual(Object) - Static method in class org.apache.spark.CleanRDD
 
canEqual(Object) - Static method in class org.apache.spark.CleanShuffle
 
canEqual(Object) - Static method in class org.apache.spark.ExceptionFailure
 
canEqual(Object) - Static method in class org.apache.spark.ExecutorLostFailure
 
canEqual(Object) - Static method in class org.apache.spark.ExecutorRegistered
 
canEqual(Object) - Static method in class org.apache.spark.ExecutorRemoved
 
canEqual(Object) - Static method in class org.apache.spark.ExpireDeadHosts
 
canEqual(Object) - Static method in class org.apache.spark.FetchFailed
 
canEqual(Object) - Static method in class org.apache.spark.graphx.Edge
 
canEqual(Object) - Static method in class org.apache.spark.ml.feature.Dot
 
canEqual(Object) - Static method in class org.apache.spark.ml.feature.LabeledPoint
 
canEqual(Object) - Static method in class org.apache.spark.ml.param.ParamPair
 
canEqual(Object) - Static method in class org.apache.spark.mllib.feature.VocabWord
 
canEqual(Object) - Static method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
canEqual(Object) - Static method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
canEqual(Object) - Static method in class org.apache.spark.mllib.linalg.QRDecomposition
 
canEqual(Object) - Static method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
canEqual(Object) - Static method in class org.apache.spark.mllib.recommendation.Rating
 
canEqual(Object) - Static method in class org.apache.spark.mllib.regression.LabeledPoint
 
canEqual(Object) - Static method in class org.apache.spark.mllib.stat.test.BinarySample
 
canEqual(Object) - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
canEqual(Object) - Static method in class org.apache.spark.mllib.tree.model.Split
 
canEqual(Object) - Static method in class org.apache.spark.Resubmitted
 
canEqual(Object) - Static method in class org.apache.spark.rpc.netty.OnStart
 
canEqual(Object) - Static method in class org.apache.spark.rpc.netty.OnStop
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.AllJobsCancelled
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.BlacklistedExecutor
 
canEqual(Object) - Method in class org.apache.spark.scheduler.cluster.ExecutorInfo
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.JobSucceeded
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.local.KillTask
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.local.ReviveOffers
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.local.StatusUpdate
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.local.StopExecutor
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.ResubmitFailedStages
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerJobStart
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerLogStart
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerStageCompleted
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
canEqual(Object) - Static method in class org.apache.spark.scheduler.StopCoordinator
 
canEqual(Object) - Static method in class org.apache.spark.sql.DatasetHolder
 
canEqual(Object) - Static method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
canEqual(Object) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
canEqual(Object) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
canEqual(Object) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
canEqual(Object) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
canEqual(Object) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
canEqual(Object) - Static method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
canEqual(Object) - Static method in class org.apache.spark.sql.hive.RelationConversions
 
canEqual(Object) - Static method in class org.apache.spark.sql.jdbc.JdbcType
 
canEqual(Object) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
canEqual(Object) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
canEqual(Object) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.And
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.EqualNullSafe
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.EqualTo
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.GreaterThan
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.In
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.IsNotNull
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.IsNull
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.LessThan
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.LessThanOrEqual
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.Not
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.Or
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.StringContains
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.StringEndsWith
 
canEqual(Object) - Static method in class org.apache.spark.sql.sources.StringStartsWith
 
canEqual(Object) - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.ArrayType
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.CharType
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.DecimalType
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.MapType
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.ObjectType
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.StructField
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.StructType
 
canEqual(Object) - Static method in class org.apache.spark.sql.types.VarcharType
 
canEqual(Object) - Static method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
canEqual(Object) - Static method in class org.apache.spark.status.api.v1.ApplicationInfo
 
canEqual(Object) - Static method in class org.apache.spark.StopMapOutputTracker
 
canEqual(Object) - Static method in class org.apache.spark.storage.BlockStatus
 
canEqual(Object) - Static method in class org.apache.spark.storage.BlockUpdatedInfo
 
canEqual(Object) - Static method in class org.apache.spark.storage.BroadcastBlockId
 
canEqual(Object) - Static method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
canEqual(Object) - Static method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
canEqual(Object) - Static method in class org.apache.spark.storage.RDDBlockId
 
canEqual(Object) - Static method in class org.apache.spark.storage.ShuffleBlockId
 
canEqual(Object) - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
canEqual(Object) - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
canEqual(Object) - Static method in class org.apache.spark.storage.StreamBlockId
 
canEqual(Object) - Static method in class org.apache.spark.storage.TaskResultBlockId
 
canEqual(Object) - Static method in class org.apache.spark.streaming.Duration
 
canEqual(Object) - Static method in class org.apache.spark.streaming.kinesis.DefaultCredentials
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.AllReceiverIds
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.BatchInfo
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.GetAllReceiverInfo
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StopAllReceivers
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
canEqual(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
canEqual(Object) - Static method in class org.apache.spark.streaming.Time
 
canEqual(Object) - Static method in class org.apache.spark.Success
 
canEqual(Object) - Static method in class org.apache.spark.TaskCommitDenied
 
canEqual(Object) - Static method in class org.apache.spark.TaskKilled
 
canEqual(Object) - Static method in class org.apache.spark.TaskResultLost
 
canEqual(Object) - Static method in class org.apache.spark.TaskSchedulerIsSet
 
canEqual(Object) - Static method in class org.apache.spark.UnknownReason
 
canEqual(Object) - Static method in class org.apache.spark.util.MethodIdentifier
 
canEqual(Object) - Method in class org.apache.spark.util.MutablePair
 
canHandle(String) - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
canHandle(String) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Check if this dialect instance can handle a certain jdbc url.
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
canHandle(String) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
canonicalized() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
canonicalized() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
canonicalized() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
canonicalized() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
CanonicalRandomVertexCut$() - Constructor for class org.apache.spark.graphx.PartitionStrategy.CanonicalRandomVertexCut$
 
cartesian(JavaRDDLike<U, ?>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
cartesian(JavaRDDLike<U, ?>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
cartesian(JavaRDDLike<U, ?>) - Static method in class org.apache.spark.api.java.JavaRDD
 
cartesian(JavaRDDLike<U, ?>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
cartesian(RDD<U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in this and b is in other.
cartesian(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
caseSensitive() - Method in class org.apache.spark.ml.feature.StopWordsRemover
Whether to do a case sensitive comparison over the stop words.
cast(DataType) - Method in class org.apache.spark.sql.Column
Casts the column to a different data type.
cast(String) - Method in class org.apache.spark.sql.Column
Casts the column to a different data type, using the canonical string representation of the type.
Catalog - Class in org.apache.spark.sql.catalog
Catalog interface for Spark.
Catalog() - Constructor for class org.apache.spark.sql.catalog.Catalog
 
catalog() - Method in class org.apache.spark.sql.SparkSession
Interface through which the user may create, drop, alter or query underlying databases, tables, functions etc.
catalogString() - Method in class org.apache.spark.sql.types.ArrayType
 
catalogString() - Static method in class org.apache.spark.sql.types.BinaryType
 
catalogString() - Static method in class org.apache.spark.sql.types.BooleanType
 
catalogString() - Static method in class org.apache.spark.sql.types.ByteType
 
catalogString() - Static method in class org.apache.spark.sql.types.CalendarIntervalType
 
catalogString() - Static method in class org.apache.spark.sql.types.CharType
 
catalogString() - Method in class org.apache.spark.sql.types.DataType
String representation for the type saved in external catalogs.
catalogString() - Static method in class org.apache.spark.sql.types.DateType
 
catalogString() - Static method in class org.apache.spark.sql.types.DecimalType
 
catalogString() - Static method in class org.apache.spark.sql.types.DoubleType
 
catalogString() - Static method in class org.apache.spark.sql.types.FloatType
 
catalogString() - Static method in class org.apache.spark.sql.types.HiveStringType
 
catalogString() - Static method in class org.apache.spark.sql.types.IntegerType
 
catalogString() - Static method in class org.apache.spark.sql.types.LongType
 
catalogString() - Method in class org.apache.spark.sql.types.MapType
 
catalogString() - Static method in class org.apache.spark.sql.types.NullType
 
catalogString() - Static method in class org.apache.spark.sql.types.NumericType
 
catalogString() - Static method in class org.apache.spark.sql.types.ObjectType
 
catalogString() - Static method in class org.apache.spark.sql.types.ShortType
 
catalogString() - Static method in class org.apache.spark.sql.types.StringType
 
catalogString() - Method in class org.apache.spark.sql.types.StructType
 
catalogString() - Static method in class org.apache.spark.sql.types.TimestampType
 
catalogString() - Static method in class org.apache.spark.sql.types.VarcharType
 
CatalystScan - Interface in org.apache.spark.sql.sources
::Experimental:: An interface for experimenting with a more direct connection to the query planner.
Categorical() - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
categoricalCols() - Method in class org.apache.spark.ml.feature.FeatureHasher
Numeric columns to treat as categorical features.
categoricalFeaturesInfo() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
CategoricalSplit - Class in org.apache.spark.ml.tree
Split which tests a categorical feature.
categories() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.SplitData
 
categories() - Method in class org.apache.spark.mllib.tree.model.Split
 
categoryMaps() - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
categorySizes() - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
cause() - Method in exception org.apache.spark.sql.AnalysisException
 
cause() - Method in exception org.apache.spark.sql.streaming.StreamingQueryException
 
CausedBy - Class in org.apache.spark.util
Extractor Object for pulling out the root cause of an error.
CausedBy() - Constructor for class org.apache.spark.util.CausedBy
 
cbrt(Column) - Static method in class org.apache.spark.sql.functions
Computes the cube-root of the given value.
cbrt(String) - Static method in class org.apache.spark.sql.functions
Computes the cube-root of the given column.
ceil(Column) - Static method in class org.apache.spark.sql.functions
Computes the ceiling of the given value.
ceil(String) - Static method in class org.apache.spark.sql.functions
Computes the ceiling of the given column.
ceil() - Method in class org.apache.spark.sql.types.Decimal
 
censorCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
censorCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
chainl1(Function0<Parsers.Parser<T>>, Function0<Parsers.Parser<Function2<T, T, T>>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
chainl1(Function0<Parsers.Parser<T>>, Function0<Parsers.Parser<U>>, Function0<Parsers.Parser<Function2<T, U, T>>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
chainr1(Function0<Parsers.Parser<T>>, Function0<Parsers.Parser<Function2<T, U, U>>>, Function2<T, U, U>, U) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
changePrecision(int, int) - Method in class org.apache.spark.sql.types.Decimal
Update precision and scale while keeping our value the same, and return true if successful.
channelRead0(ChannelHandlerContext, byte[]) - Method in class org.apache.spark.api.r.RBackendAuthHandler
 
CharType - Class in org.apache.spark.sql.types
Hive char type.
CharType(int) - Constructor for class org.apache.spark.sql.types.CharType
 
checkAndGetK8sMasterUrl(String) - Static method in class org.apache.spark.util.Utils
Check the validity of the given Kubernetes master URL and return the resolved URL.
checkColumnNameDuplication(Seq<String>, String, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.util.SchemaUtils
Checks if input column names have duplicate identifiers.
checkColumnNameDuplication(Seq<String>, String, boolean) - Static method in class org.apache.spark.sql.util.SchemaUtils
Checks if input column names have duplicate identifiers.
checkColumnType(StructType, String, DataType, String) - Static method in class org.apache.spark.ml.util.SchemaUtils
Check whether the given schema contains a column of the required data type.
checkColumnTypes(StructType, String, Seq<DataType>, String) - Static method in class org.apache.spark.ml.util.SchemaUtils
Check whether the given schema contains a column of one of the require data types.
checkDataColumns(RFormula, Dataset<?>) - Static method in class org.apache.spark.ml.r.RWrapperUtils
DataFrame column check.
checkFileExists(String, Configuration) - Static method in class org.apache.spark.streaming.util.HdfsUtils
Check if the file exists at the given path.
checkHost(String) - Static method in class org.apache.spark.util.Utils
 
checkHostPort(String) - Static method in class org.apache.spark.util.Utils
 
checkNumericType(StructType, String, String) - Static method in class org.apache.spark.ml.util.SchemaUtils
Check whether the given schema contains a column of the numeric data type.
checkpoint() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
checkpoint() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
checkpoint() - Static method in class org.apache.spark.api.java.JavaRDD
 
checkpoint() - Method in interface org.apache.spark.api.java.JavaRDDLike
Mark this RDD for checkpointing.
checkpoint() - Static method in class org.apache.spark.api.r.RRDD
 
checkpoint() - Static method in class org.apache.spark.graphx.EdgeRDD
 
checkpoint() - Method in class org.apache.spark.graphx.Graph
Mark this Graph for checkpointing.
checkpoint() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
checkpoint() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
checkpoint() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
checkpoint() - Static method in class org.apache.spark.graphx.VertexRDD
 
checkpoint() - Method in class org.apache.spark.rdd.HadoopRDD
 
checkpoint() - Static method in class org.apache.spark.rdd.JdbcRDD
 
checkpoint() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
checkpoint() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
checkpoint() - Method in class org.apache.spark.rdd.RDD
Mark this RDD for checkpointing.
checkpoint() - Static method in class org.apache.spark.rdd.UnionRDD
 
checkpoint() - Method in class org.apache.spark.sql.Dataset
Eagerly checkpoint a Dataset and return the new Dataset.
checkpoint(boolean) - Method in class org.apache.spark.sql.Dataset
Returns a checkpointed version of this Dataset.
checkpoint(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
checkpoint(Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Enable periodic checkpointing of RDDs of this DStream.
checkpoint(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
checkpoint(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
checkpoint(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
checkpoint(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
checkpoint(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
checkpoint(String) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Sets the context to periodically checkpoint the DStream operations for master fault-tolerance.
checkpoint(Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Enable periodic checkpointing of RDDs of this DStream
checkpoint(String) - Method in class org.apache.spark.streaming.StreamingContext
Set the context to periodically checkpoint the DStream operations for driver fault-tolerance.
Checkpointed() - Static method in class org.apache.spark.rdd.CheckpointState
 
CheckpointingInProgress() - Static method in class org.apache.spark.rdd.CheckpointState
 
checkpointInterval() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
checkpointInterval() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
checkpointInterval() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
checkpointInterval() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
checkpointInterval() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
checkpointInterval() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
checkpointInterval() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
checkpointInterval() - Static method in class org.apache.spark.ml.clustering.LDA
 
checkpointInterval() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
checkpointInterval() - Method in interface org.apache.spark.ml.param.shared.HasCheckpointInterval
Param for set checkpoint interval (&gt;= 1) or disable checkpoint (-1).
checkpointInterval() - Static method in class org.apache.spark.ml.recommendation.ALS
 
checkpointInterval() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
checkpointInterval() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
checkpointInterval() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
checkpointInterval() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
checkpointInterval() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
checkpointInterval() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
checkpointInterval() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
CheckpointReader - Class in org.apache.spark.streaming
 
CheckpointReader() - Constructor for class org.apache.spark.streaming.CheckpointReader
 
CheckpointState - Class in org.apache.spark.rdd
Enumeration to manage state transitions of an RDD through checkpointing
CheckpointState() - Constructor for class org.apache.spark.rdd.CheckpointState
 
checkSchemaColumnNameDuplication(StructType, String, boolean) - Static method in class org.apache.spark.sql.util.SchemaUtils
Checks if an input schema has duplicate column names.
checkSingleVsMultiColumnParams(Params, Seq<Param<?>>, Seq<Param<?>>) - Static method in class org.apache.spark.ml.param.ParamValidators
Utility for Param validity checks for Transformers which have both single- and multi-column support.
checkState(boolean, Function0<String>) - Static method in class org.apache.spark.streaming.util.HdfsUtils
 
child() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
child() - Method in class org.apache.spark.sql.sources.Not
 
CHILD_CONNECTION_TIMEOUT - Static variable in class org.apache.spark.launcher.SparkLauncher
Maximum time (in ms) to wait for a child process to connect back to the launcher server when using @link{#start()}.
CHILD_PROCESS_LOGGER_NAME - Static variable in class org.apache.spark.launcher.SparkLauncher
Logger name to use when launching a child process.
ChildFirstURLClassLoader - Class in org.apache.spark.util
A mutable class loader that gives preference to its own URLs over the parent class loader when loading classes and resources.
ChildFirstURLClassLoader(URL[], ClassLoader) - Constructor for class org.apache.spark.util.ChildFirstURLClassLoader
 
children() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
children() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
children() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
children() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
childrenResolved() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
childrenResolved() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
childrenResolved() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
chiSqFunc() - Method in class org.apache.spark.mllib.stat.test.ChiSqTest.Method
 
ChiSqSelector - Class in org.apache.spark.ml.feature
Chi-Squared feature selection, which selects categorical features to use for predicting a categorical label.
ChiSqSelector(String) - Constructor for class org.apache.spark.ml.feature.ChiSqSelector
 
ChiSqSelector() - Constructor for class org.apache.spark.ml.feature.ChiSqSelector
 
ChiSqSelector - Class in org.apache.spark.mllib.feature
Creates a ChiSquared feature selector.
ChiSqSelector() - Constructor for class org.apache.spark.mllib.feature.ChiSqSelector
 
ChiSqSelector(int) - Constructor for class org.apache.spark.mllib.feature.ChiSqSelector
The is the same to call this() and setNumTopFeatures(numTopFeatures)
ChiSqSelectorModel - Class in org.apache.spark.ml.feature
Model fitted by ChiSqSelector.
ChiSqSelectorModel - Class in org.apache.spark.mllib.feature
Chi Squared selector model.
ChiSqSelectorModel(int[]) - Constructor for class org.apache.spark.mllib.feature.ChiSqSelectorModel
 
ChiSqSelectorModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.feature
 
ChiSqSelectorModel.SaveLoadV1_0$.Data - Class in org.apache.spark.mllib.feature
Model data for import/export
chiSqTest(Vector, Vector) - Static method in class org.apache.spark.mllib.stat.Statistics
Conduct Pearson's chi-squared goodness of fit test of the observed data against the expected distribution.
chiSqTest(Vector) - Static method in class org.apache.spark.mllib.stat.Statistics
Conduct Pearson's chi-squared goodness of fit test of the observed data against the uniform distribution, with each category having an expected frequency of 1 / observed.size.
chiSqTest(Matrix) - Static method in class org.apache.spark.mllib.stat.Statistics
Conduct Pearson's independence test on the input contingency matrix, which cannot contain negative entries or columns or rows that sum up to 0.
chiSqTest(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.stat.Statistics
Conduct Pearson's independence test for every feature against the label across the input RDD.
chiSqTest(JavaRDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.stat.Statistics
Java-friendly version of chiSqTest()
ChiSqTest - Class in org.apache.spark.mllib.stat.test
Conduct the chi-squared test for the input RDDs using the specified method.
ChiSqTest() - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest
 
ChiSqTest.Method - Class in org.apache.spark.mllib.stat.test
param: name String name for the method.
ChiSqTest.Method$ - Class in org.apache.spark.mllib.stat.test
 
ChiSqTest.NullHypothesis$ - Class in org.apache.spark.mllib.stat.test
 
ChiSqTestResult - Class in org.apache.spark.mllib.stat.test
Object containing the test results for the chi-squared hypothesis test.
chiSquared(Vector, Vector, String) - Static method in class org.apache.spark.mllib.stat.test.ChiSqTest
 
chiSquaredFeatures(RDD<LabeledPoint>, String) - Static method in class org.apache.spark.mllib.stat.test.ChiSqTest
Conduct Pearson's independence test for each feature against the label across the input RDD.
chiSquaredMatrix(Matrix, String) - Static method in class org.apache.spark.mllib.stat.test.ChiSqTest
 
ChiSquareTest - Class in org.apache.spark.ml.stat
Experimental
ChiSquareTest() - Constructor for class org.apache.spark.ml.stat.ChiSquareTest
 
chmod700(File) - Static method in class org.apache.spark.util.Utils
JDK equivalent of chmod 700 file.
CholeskyDecomposition - Class in org.apache.spark.mllib.linalg
Compute Cholesky decomposition.
CholeskyDecomposition() - Constructor for class org.apache.spark.mllib.linalg.CholeskyDecomposition
 
classForName(String) - Static method in class org.apache.spark.util.Utils
Preferred alternative to Class.forName(className)
Classification() - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
ClassificationModel<FeaturesType,M extends ClassificationModel<FeaturesType,M>> - Class in org.apache.spark.ml.classification
Developer API
ClassificationModel() - Constructor for class org.apache.spark.ml.classification.ClassificationModel
 
ClassificationModel - Interface in org.apache.spark.mllib.classification
Represents a classification model that predicts to which of a set of categories an example belongs.
Classifier<FeaturesType,E extends Classifier<FeaturesType,E,M>,M extends ClassificationModel<FeaturesType,M>> - Class in org.apache.spark.ml.classification
Developer API
Classifier() - Constructor for class org.apache.spark.ml.classification.Classifier
 
classifier() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
classifier() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
classIsLoadable(String) - Static method in class org.apache.spark.util.Utils
Determines whether the provided class is loadable in the current thread.
className() - Method in class org.apache.spark.ExceptionFailure
 
className() - Static method in class org.apache.spark.ml.linalg.JsonMatrixConverter
Unique class name for identifying JSON object encoded by this class.
className() - Method in class org.apache.spark.sql.catalog.Function
 
classpathEntries() - Method in class org.apache.spark.status.api.v1.ApplicationEnvironmentInfo
 
classTag() - Method in class org.apache.spark.api.java.JavaDoubleRDD
 
classTag() - Method in class org.apache.spark.api.java.JavaPairRDD
 
classTag() - Method in class org.apache.spark.api.java.JavaRDD
 
classTag() - Method in interface org.apache.spark.api.java.JavaRDDLike
 
classTag() - Method in class org.apache.spark.sql.Dataset
 
classTag() - Method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
classTag() - Method in interface org.apache.spark.storage.memory.MemoryEntry
 
classTag() - Method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaDStream
 
classTag() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
classTag() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
classTag() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
classTag() - Method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
clean(long, boolean) - Method in class org.apache.spark.streaming.util.WriteAheadLog
Clean all the records that are older than the threshold time.
clean(Object, boolean, boolean) - Static method in class org.apache.spark.util.ClosureCleaner
Clean the given closure in place.
CleanAccum - Class in org.apache.spark
 
CleanAccum(long) - Constructor for class org.apache.spark.CleanAccum
 
CleanBroadcast - Class in org.apache.spark
 
CleanBroadcast(long) - Constructor for class org.apache.spark.CleanBroadcast
 
CleanCheckpoint - Class in org.apache.spark
 
CleanCheckpoint(int) - Constructor for class org.apache.spark.CleanCheckpoint
 
cleaning() - Method in class org.apache.spark.status.LiveStage
 
CleanRDD - Class in org.apache.spark
 
CleanRDD(int) - Constructor for class org.apache.spark.CleanRDD
 
CleanShuffle - Class in org.apache.spark
 
CleanShuffle(int) - Constructor for class org.apache.spark.CleanShuffle
 
CleanupTask - Interface in org.apache.spark
Classes that represent cleaning tasks.
CleanupTaskWeakReference - Class in org.apache.spark
A WeakReference associated with a CleanupTask.
CleanupTaskWeakReference(CleanupTask, Object, ReferenceQueue<Object>) - Constructor for class org.apache.spark.CleanupTaskWeakReference
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.LDA
 
clear(Param<?>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
clear(Param<?>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
clear(Param<?>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
clear(Param<?>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.DCT
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.IDF
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Imputer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Interaction
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.NGram
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.PCA
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormula
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
clear(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
clear(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
clear(Param<?>) - Method in interface org.apache.spark.ml.param.Params
Clears the user-supplied value for the input param.
clear(Param<?>) - Static method in class org.apache.spark.ml.Pipeline
 
clear(Param<?>) - Static method in class org.apache.spark.ml.PipelineModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
clear(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
clear(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
clear(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
clear(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
clear(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
clear() - Method in class org.apache.spark.sql.util.ExecutionListenerManager
Removes all the registered QueryExecutionListener.
clear() - Static method in class org.apache.spark.util.AccumulatorContext
Clears all registered AccumulatorV2s.
clearActive() - Static method in class org.apache.spark.sql.SQLContext
Deprecated.
Use SparkSession.clearActiveSession instead. Since 2.0.0.
clearActiveSession() - Static method in class org.apache.spark.sql.SparkSession
Clears the active SparkSession for current thread.
clearCache() - Method in class org.apache.spark.sql.catalog.Catalog
Removes all cached tables from the in-memory cache.
clearCache() - Method in class org.apache.spark.sql.SQLContext
Removes all cached tables from the in-memory cache.
clearCallSite() - Method in class org.apache.spark.api.java.JavaSparkContext
Pass-through to SparkContext.setCallSite.
clearCallSite() - Method in class org.apache.spark.SparkContext
Clear the thread-local property for overriding the call sites of actions and RDDs.
clearDefaultSession() - Static method in class org.apache.spark.sql.SparkSession
Clears the default SparkSession that is returned by the builder.
clearDependencies() - Method in class org.apache.spark.rdd.CoGroupedRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.ShuffledRDD
 
clearDependencies() - Method in class org.apache.spark.rdd.UnionRDD
 
clearJobGroup() - Method in class org.apache.spark.api.java.JavaSparkContext
Clear the current thread's job group ID and its description.
clearJobGroup() - Method in class org.apache.spark.SparkContext
Clear the current thread's job group ID and its description.
clearThreshold() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
Clears the threshold so that predict will output raw prediction scores.
clearThreshold() - Method in class org.apache.spark.mllib.classification.SVMModel
Clears the threshold so that predict will output raw prediction scores.
CLogLog$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.CLogLog$
 
clone() - Method in class org.apache.spark.SparkConf
Copy this object
clone() - Method in class org.apache.spark.sql.ExperimentalMethods
 
clone() - Method in class org.apache.spark.sql.types.Decimal
 
clone() - Method in class org.apache.spark.sql.util.ExecutionListenerManager
Get an identical copy of this listener manager.
clone() - Method in class org.apache.spark.storage.StorageLevel
 
clone() - Method in class org.apache.spark.util.random.BernoulliCellSampler
 
clone() - Method in class org.apache.spark.util.random.BernoulliSampler
 
clone() - Method in class org.apache.spark.util.random.PoissonSampler
 
clone() - Method in interface org.apache.spark.util.random.RandomSampler
return a copy of the RandomSampler object
clone(T, SerializerInstance, ClassTag<T>) - Static method in class org.apache.spark.util.Utils
Clone an object using a Spark serializer.
cloneComplement() - Method in class org.apache.spark.util.random.BernoulliCellSampler
Return a sampler that is the complement of the range specified of the current sampler.
close() - Method in class org.apache.spark.api.java.JavaSparkContext
 
close() - Method in class org.apache.spark.io.NioBufferedFileInputStream
 
close() - Method in class org.apache.spark.io.ReadAheadInputStream
 
close() - Method in class org.apache.spark.io.SnappyOutputStreamWrapper
 
close() - Method in class org.apache.spark.serializer.DeserializationStream
 
close() - Method in class org.apache.spark.serializer.SerializationStream
 
close(Throwable) - Method in class org.apache.spark.sql.ForeachWriter
Called when stopping to process one partition of new data in the executor side.
close() - Method in class org.apache.spark.sql.hive.execution.HiveOutputWriter
 
close() - Method in class org.apache.spark.sql.SparkSession
Synonym for stop().
close() - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
close() - Method in class org.apache.spark.sql.vectorized.ColumnarBatch
Called to close all the columns in this batch.
close() - Method in class org.apache.spark.sql.vectorized.ColumnVector
Cleans up memory for this column vector.
close() - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
close() - Method in class org.apache.spark.storage.CountingWritableChannel
 
close() - Method in class org.apache.spark.storage.TimeTrackingOutputStream
 
close() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
 
close() - Method in class org.apache.spark.streaming.util.WriteAheadLog
Close this log and release any resources.
closeWriter(TaskAttemptContext) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
ClosureCleaner - Class in org.apache.spark.util
A cleaner that renders closures serializable if they can be done so safely.
ClosureCleaner() - Constructor for class org.apache.spark.util.ClosureCleaner
 
closureSerializer() - Method in class org.apache.spark.SparkEnv
 
cls() - Method in class org.apache.spark.sql.types.ObjectType
 
cls() - Method in class org.apache.spark.util.MethodIdentifier
 
clsTag() - Method in interface org.apache.spark.sql.Encoder
A ClassTag that can be used to construct an Array to contain a collection of T.
cluster() - Method in class org.apache.spark.ml.clustering.ClusteringSummary
Cluster centers of the transformed data.
cluster() - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering.Assignment
 
clusterCenters() - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
clusterCenters() - Method in class org.apache.spark.ml.clustering.KMeansModel
 
clusterCenters() - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Leaf cluster centers.
clusterCenters() - Method in class org.apache.spark.mllib.clustering.KMeansModel
 
clusterCenters() - Method in class org.apache.spark.mllib.clustering.StreamingKMeansModel
 
clusteredColumns - Variable in class org.apache.spark.sql.sources.v2.reader.partitioning.ClusteredDistribution
The names of the clustered columns.
ClusteredDistribution - Class in org.apache.spark.sql.sources.v2.reader.partitioning
A concrete implementation of Distribution.
ClusteredDistribution(String[]) - Constructor for class org.apache.spark.sql.sources.v2.reader.partitioning.ClusteredDistribution
 
ClusteringEvaluator - Class in org.apache.spark.ml.evaluation
Experimental
ClusteringEvaluator(String) - Constructor for class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
ClusteringEvaluator() - Constructor for class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
ClusteringSummary - Class in org.apache.spark.ml.clustering
Experimental Summary of clustering algorithms.
clusterSizes() - Method in class org.apache.spark.ml.clustering.ClusteringSummary
Size of (number of data points in) each cluster.
ClusterStats(Vector, double, long) - Constructor for class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette.ClusterStats
 
ClusterStats$() - Constructor for class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette.ClusterStats$
 
clusterWeights() - Method in class org.apache.spark.mllib.clustering.StreamingKMeansModel
 
cn() - Method in class org.apache.spark.mllib.feature.VocabWord
 
coalesce(int) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
coalesce(int, RDD<?>) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
Runs the packing algorithm and returns an array of PartitionGroups that if possible are load balanced and grouped by locality
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
coalesce(int, RDD<?>) - Method in interface org.apache.spark.rdd.PartitionCoalescer
Coalesce the partitions of the given RDD.
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD that is reduced into numPartitions partitions.
coalesce(int, boolean, Option<PartitionCoalescer>, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
coalesce(int) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset that has exactly numPartitions partitions, when the fewer partitions are requested.
coalesce(Column...) - Static method in class org.apache.spark.sql.functions
Returns the first column that is not null, or null if all inputs are null.
coalesce(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Returns the first column that is not null, or null if all inputs are null.
coalesce$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
coalesce$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
coalesce$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
coalesce$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
coalesce$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
coalesce$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
coalesce$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
coalesce$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
coalesce$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
coalesce$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
coalesce$default$3() - Static method in class org.apache.spark.api.r.RRDD
 
coalesce$default$3() - Static method in class org.apache.spark.graphx.EdgeRDD
 
coalesce$default$3() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
coalesce$default$3() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
coalesce$default$3() - Static method in class org.apache.spark.graphx.VertexRDD
 
coalesce$default$3() - Static method in class org.apache.spark.rdd.HadoopRDD
 
coalesce$default$3() - Static method in class org.apache.spark.rdd.JdbcRDD
 
coalesce$default$3() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
coalesce$default$3() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
coalesce$default$3() - Static method in class org.apache.spark.rdd.UnionRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.api.r.RRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.graphx.VertexRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
coalesce$default$4(int, boolean, Option<PartitionCoalescer>) - Static method in class org.apache.spark.rdd.UnionRDD
 
CoarseGrainedClusterMessages - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages
 
CoarseGrainedClusterMessages.AddWebUIFilter - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.AddWebUIFilter$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.GetExecutorLossReason - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.GetExecutorLossReason$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillExecutors - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillExecutors$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillExecutorsOnHost - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillExecutorsOnHost$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillTask - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.KillTask$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.LaunchTask - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.LaunchTask$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterClusterManager - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterClusterManager$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisteredExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutor - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutorFailed - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutorFailed$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RegisterExecutorResponse - Interface in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RemoveExecutor - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RemoveExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RemoveWorker - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RemoveWorker$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RequestExecutors - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RequestExecutors$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RetrieveLastAllocatedExecutorId$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.RetrieveSparkAppConfig$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.ReviveOffers$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.SetupDriver - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.SetupDriver$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.Shutdown$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.SparkAppConfig - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.SparkAppConfig$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StatusUpdate - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StatusUpdate$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StopDriver$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StopExecutor$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.StopExecutors$ - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.UpdateDelegationTokens - Class in org.apache.spark.scheduler.cluster
 
CoarseGrainedClusterMessages.UpdateDelegationTokens$ - Class in org.apache.spark.scheduler.cluster
 
code() - Method in class org.apache.spark.mllib.feature.VocabWord
 
CodegenMetrics - Class in org.apache.spark.metrics.source
Experimental Metrics for code generation.
CodegenMetrics() - Constructor for class org.apache.spark.metrics.source.CodegenMetrics
 
codeLen() - Method in class org.apache.spark.mllib.feature.VocabWord
 
coefficientMatrix() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
coefficients() - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
coefficients() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
A vector of model coefficients for "binomial" logistic regression.
coefficients() - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
coefficients() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
coefficients() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
 
coefficientStandardErrors() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionTrainingSummary
Standard error of estimated coefficients and intercept.
coefficientStandardErrors() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Standard error of estimated coefficients and intercept.
cogroup(JavaPairRDD<K, W>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, JavaPairRDD<K, W3>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, JavaPairRDD<K, W3>) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(JavaPairRDD<K, W>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, JavaPairRDD<K, W3>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, RDD<Tuple2<K, W3>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(RDD<Tuple2<K, W>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, RDD<Tuple2<K, W3>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(RDD<Tuple2<K, W>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(RDD<Tuple2<K, W>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other, return a resulting RDD that contains a tuple with the list of values for that key in this as well as other.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2, return a resulting RDD that contains a tuple with the list of values for that key in this, other1 and other2.
cogroup(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, RDD<Tuple2<K, W3>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
For each key k in this or other1 or other2 or other3, return a resulting RDD that contains a tuple with the list of values for that key in this, other1, other2 and other3.
cogroup(KeyValueGroupedDataset<K, U>, Function3<K, Iterator<V>, Iterator<U>, TraversableOnce<R>>, Encoder<R>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Scala-specific) Applies the given function to each cogrouped data.
cogroup(KeyValueGroupedDataset<K, U>, CoGroupFunction<K, V, U, R>, Encoder<R>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Java-specific) Applies the given function to each cogrouped data.
cogroup(JavaPairDStream<K, W>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(JavaPairDStream<K, W>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(JavaPairDStream<K, W>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
cogroup(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
cogroup(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
cogroup(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
cogroup(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
cogroup(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
cogroup(DStream<Tuple2<K, W>>, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(DStream<Tuple2<K, W>>, int, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
cogroup(DStream<Tuple2<K, W>>, Partitioner, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'cogroup' between RDDs of this DStream and other DStream.
CoGroupedRDD<K> - Class in org.apache.spark.rdd
Developer API An RDD that cogroups its parents.
CoGroupedRDD(Seq<RDD<? extends Product2<K, ?>>>, Partitioner, ClassTag<K>) - Constructor for class org.apache.spark.rdd.CoGroupedRDD
 
CoGroupFunction<K,V1,V2,R> - Interface in org.apache.spark.api.java.function
A function that returns zero or more output records from each grouping key and its values from 2 Datasets.
col(String) - Method in class org.apache.spark.sql.Dataset
Selects column based on the column name and returns it as a Column.
col(String) - Static method in class org.apache.spark.sql.functions
Returns a Column based on the given column name.
coldStartStrategy() - Static method in class org.apache.spark.ml.recommendation.ALS
 
coldStartStrategy() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
colIter() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
colIter() - Method in interface org.apache.spark.ml.linalg.Matrix
Returns an iterator of column vectors.
colIter() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
colIter() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
colIter() - Method in interface org.apache.spark.mllib.linalg.Matrix
Returns an iterator of column vectors.
colIter() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
collect() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
collect() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
collect() - Static method in class org.apache.spark.api.java.JavaRDD
 
collect() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an array that contains all of the elements in this RDD.
collect() - Static method in class org.apache.spark.api.r.RRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
collect() - Static method in class org.apache.spark.graphx.EdgeRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
collect() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
collect() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
collect() - Static method in class org.apache.spark.graphx.VertexRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
collect() - Static method in class org.apache.spark.rdd.HadoopRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
collect() - Static method in class org.apache.spark.rdd.JdbcRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
collect() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
collect() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
collect() - Method in class org.apache.spark.rdd.RDD
Return an array that contains all of the elements in this RDD.
collect(PartialFunction<T, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return an RDD that contains all matching values by applying f.
collect() - Static method in class org.apache.spark.rdd.UnionRDD
 
collect(PartialFunction<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
collect() - Method in class org.apache.spark.sql.Dataset
Returns an array that contains all rows in this Dataset.
collect(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
collect(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
collect(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
collect(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
collect(PartialFunction<A, B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
collect_list(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns a list of objects with duplicates.
collect_list(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns a list of objects with duplicates.
collect_set(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns a set of objects with duplicate elements eliminated.
collect_set(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns a set of objects with duplicate elements eliminated.
collectAsList() - Method in class org.apache.spark.sql.Dataset
Returns a Java list that contains all rows in this Dataset.
collectAsMap() - Method in class org.apache.spark.api.java.JavaPairRDD
Return the key-value pairs in this RDD to the master as a Map.
collectAsMap() - Method in class org.apache.spark.rdd.PairRDDFunctions
Return the key-value pairs in this RDD to the master as a Map.
collectAsync() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
collectAsync() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
collectAsync() - Static method in class org.apache.spark.api.java.JavaRDD
 
collectAsync() - Method in interface org.apache.spark.api.java.JavaRDDLike
The asynchronous version of collect, which returns a future for retrieving an array containing all of the elements in this RDD.
collectAsync() - Method in class org.apache.spark.rdd.AsyncRDDActions
Returns a future for retrieving all elements of this RDD.
collectEdges(EdgeDirection) - Method in class org.apache.spark.graphx.GraphOps
Returns an RDD that contains for each vertex v its local edges, i.e., the edges that are incident on v, in the user-specified direction.
collectFirst(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
collectFirst(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
collectFirst(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
collectFirst(PartialFunction<BaseType, B>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
collectFirst(PartialFunction<A, B>) - Static method in class org.apache.spark.sql.types.StructType
 
collectionAccumulator() - Method in class org.apache.spark.SparkContext
Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.
collectionAccumulator(String) - Method in class org.apache.spark.SparkContext
Create and register a CollectionAccumulator, which starts with empty list and accumulates inputs by adding them into the list.
CollectionAccumulator<T> - Class in org.apache.spark.util
An accumulator for collecting a list of elements.
CollectionAccumulator() - Constructor for class org.apache.spark.util.CollectionAccumulator
 
CollectionsUtils - Class in org.apache.spark.util
 
CollectionsUtils() - Constructor for class org.apache.spark.util.CollectionsUtils
 
collectLeaves() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
collectLeaves() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
collectLeaves() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
collectLeaves() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
collectNeighborIds(EdgeDirection) - Method in class org.apache.spark.graphx.GraphOps
Collect the neighbor vertex ids for each vertex.
collectNeighbors(EdgeDirection) - Method in class org.apache.spark.graphx.GraphOps
Collect the neighbor vertex attributes for each vertex.
collectPartitions(int[]) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
collectPartitions(int[]) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
collectPartitions(int[]) - Static method in class org.apache.spark.api.java.JavaRDD
 
collectPartitions(int[]) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an array that contains all of the elements in a specific partition of this RDD.
collectSubModels() - Method in interface org.apache.spark.ml.param.shared.HasCollectSubModels
Param for whether to collect a list of sub-models trained during tuning.
collectSubModels() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
collectSubModels() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
colPtrs() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
colPtrs() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
colRegex(String) - Method in class org.apache.spark.sql.Dataset
Selects column based on the column name specified as a regex and returns it as Column.
colsPerBlock() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
colStats(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.Statistics
Computes column-wise summary statistics for the input RDD[Vector].
Column - Class in org.apache.spark.sql.catalog
A column in Spark, as returned by listColumns method in Catalog.
Column(String, String, String, boolean, boolean, boolean) - Constructor for class org.apache.spark.sql.catalog.Column
 
Column - Class in org.apache.spark.sql
A column that will be computed based on the data in a DataFrame.
Column(Expression) - Constructor for class org.apache.spark.sql.Column
 
Column(String) - Constructor for class org.apache.spark.sql.Column
 
column(String) - Static method in class org.apache.spark.sql.functions
Returns a Column based on the given column name.
column(int) - Method in class org.apache.spark.sql.vectorized.ColumnarBatch
Returns the column at `ordinal`.
ColumnarArray - Class in org.apache.spark.sql.vectorized
Array abstraction in ColumnVector.
ColumnarArray(ColumnVector, int, int) - Constructor for class org.apache.spark.sql.vectorized.ColumnarArray
 
ColumnarBatch - Class in org.apache.spark.sql.vectorized
This class wraps multiple ColumnVectors as a row-wise table.
ColumnarBatch(ColumnVector[]) - Constructor for class org.apache.spark.sql.vectorized.ColumnarBatch
 
ColumnarMap - Class in org.apache.spark.sql.vectorized
Map abstraction in ColumnVector.
ColumnarMap(ColumnVector, ColumnVector, int, int) - Constructor for class org.apache.spark.sql.vectorized.ColumnarMap
 
ColumnarRow - Class in org.apache.spark.sql.vectorized
Row abstraction in ColumnVector.
ColumnarRow(ColumnVector, int) - Constructor for class org.apache.spark.sql.vectorized.ColumnarRow
 
ColumnName - Class in org.apache.spark.sql
A convenient class used for constructing schema.
ColumnName(String) - Constructor for class org.apache.spark.sql.ColumnName
 
ColumnPruner - Class in org.apache.spark.ml.feature
Utility transformer for removing temporary columns from a DataFrame.
ColumnPruner(String, Set<String>) - Constructor for class org.apache.spark.ml.feature.ColumnPruner
 
ColumnPruner(Set<String>) - Constructor for class org.apache.spark.ml.feature.ColumnPruner
 
columns() - Method in class org.apache.spark.sql.Dataset
Returns all column names as an array.
columnSchema() - Static method in class org.apache.spark.ml.image.ImageSchema
Schema for the image column: Row(String, Int, Int, Int, Int, Array[Byte])
columnSimilarities() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Compute all cosine similarities between columns of this matrix using the brute-force approach of computing normalized dot products.
columnSimilarities() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Compute all cosine similarities between columns of this matrix using the brute-force approach of computing normalized dot products.
columnSimilarities(double) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Compute similarities between columns of this matrix using a sampling approach.
columnsToPrune() - Method in class org.apache.spark.ml.feature.ColumnPruner
 
ColumnVector - Class in org.apache.spark.sql.vectorized
An interface representing in-memory columnar data in Spark.
combinations(int) - Static method in class org.apache.spark.sql.types.StructType
 
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean, Serializer) - Method in class org.apache.spark.api.java.JavaPairRDD
Generic function to combine the elements for each key using a custom set of aggregation functions.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Generic function to combine the elements for each key using a custom set of aggregation functions.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Simplified version of combineByKey that hash-partitions the output RDD and uses map-side aggregation.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>) - Method in class org.apache.spark.api.java.JavaPairRDD
Simplified version of combineByKey that hash-partitions the resulting RDD using the existing partitioner/parallelism level and using map-side aggregation.
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean, Serializer) - Method in class org.apache.spark.rdd.PairRDDFunctions
Generic function to combine the elements for each key using a custom set of aggregation functions.
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Simplified version of combineByKeyWithClassTag that hash-partitions the output RDD.
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Simplified version of combineByKeyWithClassTag that hash-partitions the resulting RDD using the existing partitioner/parallelism level.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Combine elements of each key in DStream's RDDs using custom function.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Combine elements of each key in DStream's RDDs using custom function.
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
combineByKey(Function<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
combineByKey(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean, ClassTag<C>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Combine elements of each key in DStream's RDDs using custom functions.
combineByKeyWithClassTag(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, Partitioner, boolean, Serializer, ClassTag<C>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Experimental Generic function to combine the elements for each key using a custom set of aggregation functions.
combineByKeyWithClassTag(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, int, ClassTag<C>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Experimental Simplified version of combineByKeyWithClassTag that hash-partitions the output RDD.
combineByKeyWithClassTag(Function1<V, C>, Function2<C, V, C>, Function2<C, C, C>, ClassTag<C>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Experimental Simplified version of combineByKeyWithClassTag that hash-partitions the resulting RDD using the existing partitioner/parallelism level.
combineCombinersByKey(Iterator<? extends Product2<K, C>>, TaskContext) - Method in class org.apache.spark.Aggregator
 
combineValuesByKey(Iterator<? extends Product2<K, V>>, TaskContext) - Method in class org.apache.spark.Aggregator
 
commit(Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
commit(Offset) - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.ContinuousReader
Informs the source that Spark has completed processing all data for offsets less than or equal to `end` and will only request offsets greater than `end` in the future.
commit(Offset) - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.MicroBatchReader
Informs the source that Spark has completed processing all data for offsets less than or equal to `end` and will only request offsets greater than `end` in the future.
commit(WriterCommitMessage[]) - Method in interface org.apache.spark.sql.sources.v2.writer.DataSourceWriter
Commits this writing job with a list of commit messages.
commit() - Method in interface org.apache.spark.sql.sources.v2.writer.DataWriter
Commits this writer after all records are written successfully, returns a commit message which will be sent back to driver side and passed to DataSourceWriter.commit(WriterCommitMessage[]).
commit(long, WriterCommitMessage[]) - Method in interface org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
Commits this writing job for the specified epoch with a list of commit messages.
commit(WriterCommitMessage[]) - Method in interface org.apache.spark.sql.sources.v2.writer.streaming.StreamWriter
 
commitJob(JobContext, Seq<FileCommitProtocol.TaskCommitMessage>) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Commits a job after the writes succeed.
commitJob(JobContext, Seq<FileCommitProtocol.TaskCommitMessage>) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
commitTask(TaskAttemptContext) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Commits a task after the writes succeed.
commitTask(TaskAttemptContext) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
commitTask(OutputCommitter, TaskAttemptContext, int, int) - Static method in class org.apache.spark.mapred.SparkHadoopMapRedUtil
Commits a task output.
commonHeaderNodes() - Static method in class org.apache.spark.ui.UIUtils
 
companion() - Static method in class org.apache.spark.sql.types.StructType
 
compare(PartitionGroup, PartitionGroup) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
compare(Option<PartitionGroup>, Option<PartitionGroup>) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
compare(Decimal) - Method in class org.apache.spark.sql.types.Decimal
 
compare(RDDInfo) - Method in class org.apache.spark.storage.RDDInfo
 
compareTo(A) - Static method in class org.apache.spark.sql.types.Decimal
 
compareTo(A) - Static method in class org.apache.spark.storage.RDDInfo
 
compareTo(SparkShutdownHook) - Method in class org.apache.spark.util.SparkShutdownHook
 
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
compileValue(Object) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Converts value to SQL expression.
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
compileValue(Object) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
Complete() - Static method in class org.apache.spark.sql.streaming.OutputMode
OutputMode in which all the rows in the streaming DataFrame/Dataset will be written to the sink every time there are some updates.
completed() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
completedIndices() - Method in class org.apache.spark.status.LiveJob
 
completedIndices() - Method in class org.apache.spark.status.LiveStage
 
completedStages() - Method in class org.apache.spark.status.LiveJob
 
completedTasks() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
completedTasks() - Method in class org.apache.spark.status.LiveExecutor
 
completedTasks() - Method in class org.apache.spark.status.LiveJob
 
completedTasks() - Method in class org.apache.spark.status.LiveStage
 
COMPLETION_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
completionTime() - Method in class org.apache.spark.scheduler.StageInfo
Time when all tasks in the stage completed or when the stage was cancelled.
completionTime() - Method in class org.apache.spark.status.api.v1.JobData
 
completionTime() - Method in class org.apache.spark.status.api.v1.StageData
 
completionTime() - Method in class org.apache.spark.status.LiveJob
 
ComplexFutureAction<T> - Class in org.apache.spark
A FutureAction for actions that could trigger multiple Spark jobs.
ComplexFutureAction(Function1<JobSubmitter, Future<T>>) - Constructor for class org.apache.spark.ComplexFutureAction
 
compose(Function1<A, T1>) - Static method in class org.apache.spark.sql.types.StructType
 
compressed() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
compressed() - Static method in class org.apache.spark.ml.linalg.DenseVector
 
compressed() - Method in interface org.apache.spark.ml.linalg.Matrix
Returns a matrix in dense column major, dense row major, sparse row major, or sparse column major format, whichever uses less storage.
compressed() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
compressed() - Static method in class org.apache.spark.ml.linalg.SparseVector
 
compressed() - Method in interface org.apache.spark.ml.linalg.Vector
Returns a vector in either dense or sparse format, whichever uses less storage.
compressed() - Static method in class org.apache.spark.mllib.linalg.DenseVector
 
compressed() - Static method in class org.apache.spark.mllib.linalg.SparseVector
 
compressed() - Method in interface org.apache.spark.mllib.linalg.Vector
Returns a vector in either dense or sparse format, whichever uses less storage.
compressedColMajor() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
compressedColMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Returns a matrix in dense or sparse column major format, whichever uses less storage.
compressedColMajor() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
compressedInputStream(InputStream) - Method in interface org.apache.spark.io.CompressionCodec
 
compressedInputStream(InputStream) - Method in class org.apache.spark.io.LZ4CompressionCodec
 
compressedInputStream(InputStream) - Method in class org.apache.spark.io.LZFCompressionCodec
 
compressedInputStream(InputStream) - Method in class org.apache.spark.io.SnappyCompressionCodec
 
compressedInputStream(InputStream) - Method in class org.apache.spark.io.ZStdCompressionCodec
 
compressedOutputStream(OutputStream) - Method in interface org.apache.spark.io.CompressionCodec
 
compressedOutputStream(OutputStream) - Method in class org.apache.spark.io.LZ4CompressionCodec
 
compressedOutputStream(OutputStream) - Method in class org.apache.spark.io.LZFCompressionCodec
 
compressedOutputStream(OutputStream) - Method in class org.apache.spark.io.SnappyCompressionCodec
 
compressedOutputStream(OutputStream) - Method in class org.apache.spark.io.ZStdCompressionCodec
 
compressedRowMajor() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
compressedRowMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Returns a matrix in dense or sparse row major format, whichever uses less storage.
compressedRowMajor() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
CompressionCodec - Interface in org.apache.spark.io
Developer API CompressionCodec allows the customization of choosing different compression implementations to be used in block storage.
compute(Partition, TaskContext) - Method in class org.apache.spark.api.r.BaseRRDD
 
compute(Partition, TaskContext) - Static method in class org.apache.spark.api.r.RRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.graphx.EdgeRDD
 
compute(Partition, TaskContext) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
compute(Partition, TaskContext) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
compute(Partition, TaskContext) - Method in class org.apache.spark.graphx.VertexRDD
Provides the RDD[(VertexId, VD)] equivalent output.
compute(Vector, double, Vector) - Method in class org.apache.spark.mllib.optimization.Gradient
Compute the gradient and loss given the features of a single data point.
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.Gradient
Compute the gradient and loss given the features of a single data point, add the gradient to a provided vector to avoid creating new objects, and return loss.
compute(Vector, double, Vector) - Method in class org.apache.spark.mllib.optimization.HingeGradient
 
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.HingeGradient
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.L1Updater
 
compute(Vector, double, Vector) - Method in class org.apache.spark.mllib.optimization.LeastSquaresGradient
 
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.LeastSquaresGradient
 
compute(Vector, double, Vector, Vector) - Method in class org.apache.spark.mllib.optimization.LogisticGradient
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.SimpleUpdater
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.SquaredL2Updater
 
compute(Vector, Vector, double, int, double) - Method in class org.apache.spark.mllib.optimization.Updater
Compute an updated value for weights given the gradient, stepSize, iteration number and regularization parameter.
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.CoGroupedRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.HadoopRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.JdbcRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.NewHadoopRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.PartitionPruningRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.RDD
Developer API Implemented by subclasses to compute a given partition.
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.ShuffledRDD
 
compute(Partition, TaskContext) - Method in class org.apache.spark.rdd.UnionRDD
 
compute(Time) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Generate an RDD for the given duration
compute(Time) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
compute(Time) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Method that generates an RDD for the given Duration
compute(Time) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
compute(Time) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
compute(Time) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.ConstantInputDStream
 
compute(Time) - Method in class org.apache.spark.streaming.dstream.DStream
Method that generates an RDD for the given time
compute(Time) - Method in class org.apache.spark.streaming.dstream.ReceiverInputDStream
 
computeClusterStats(Dataset<Row>, String, String) - Static method in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette
The method takes the input dataset and computes the aggregated values about a cluster which are needed by the algorithm.
computeColumnSummaryStatistics() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes column-wise summary statistics.
computeCorrelation(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
Compute the Pearson correlation for two datasets.
computeCorrelation(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.correlation.SpearmanCorrelation
Compute Spearman's correlation for two datasets.
computeCorrelationMatrix(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
Compute the Pearson correlation matrix S, for the input matrix, where S(i, j) is the correlation between column i and j.
computeCorrelationMatrix(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.correlation.SpearmanCorrelation
Compute Spearman's correlation matrix S, for the input matrix, where S(i, j) is the correlation between column i and j.
computeCorrelationMatrixFromCovariance(Matrix) - Static method in class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
Compute the Pearson correlation matrix from the covariance matrix.
computeCorrelationWithMatrixImpl(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
 
computeCorrelationWithMatrixImpl(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.correlation.SpearmanCorrelation
 
computeCost(Dataset<?>) - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
Computes the sum of squared distances between the input points and their corresponding cluster centers.
computeCost(Dataset<?>) - Method in class org.apache.spark.ml.clustering.KMeansModel
Return the K-means cost (sum of squared distances of points to their nearest center) for this model on the given data.
computeCost(Vector) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Computes the squared distance between the input point and the cluster center it belongs to.
computeCost(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Computes the sum of squared distances between the input points and their corresponding cluster centers.
computeCost(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Java-friendly version of computeCost().
computeCost(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.KMeansModel
Return the K-means cost (sum of squared distances of points to their nearest center) for this model on the given data.
computeCovariance() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes the covariance matrix, treating each row as an observation.
computeError(RDD<LabeledPoint>, DecisionTreeRegressionModel[], double[], Loss) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Method to calculate error of the base learner for the gradient boosting calculation.
computeError(org.apache.spark.mllib.tree.model.TreeEnsembleModel, RDD<LabeledPoint>) - Method in interface org.apache.spark.mllib.tree.loss.Loss
Method to calculate error of the base learner for the gradient boosting calculation.
computeError(double, double) - Method in interface org.apache.spark.mllib.tree.loss.Loss
Method to calculate loss when the predictions are already known.
computeFractionForSampleSize(int, long, boolean) - Static method in class org.apache.spark.util.random.SamplingUtils
Returns a sampling rate that guarantees a sample of size greater than or equal to sampleSizeLowerBound 99.99% of the time.
computeGramianMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Computes the Gramian matrix A^T A.
computeGramianMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes the Gramian matrix A^T A.
computeInitialPredictionAndError(RDD<LabeledPoint>, double, DecisionTreeRegressionModel, Loss) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Compute the initial predictions and errors for a dataset for the first iteration of gradient boosting.
computeInitialPredictionAndError(RDD<LabeledPoint>, double, DecisionTreeModel, Loss) - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
Developer API Compute the initial predictions and errors for a dataset for the first iteration of gradient boosting.
computePreferredLocations(Seq<InputFormatInfo>) - Static method in class org.apache.spark.scheduler.InputFormatInfo
Computes the preferred locations based on input(s) and returned a location to block map.
computePrincipalComponents(int) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes the top k principal components only.
computePrincipalComponentsAndExplainedVariance(int) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes the top k principal components and a vector of proportions of variance explained by each principal component.
computeSilhouetteCoefficient(Broadcast<Map<Object, SquaredEuclideanSilhouette.ClusterStats>>, Vector, double, double) - Static method in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette
It computes the Silhouette coefficient for a point.
computeSilhouetteScore(Dataset<?>, String, String) - Static method in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette
Compute the mean Silhouette values of all samples.
computeSVD(int, boolean, double) - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Computes the singular value decomposition of this IndexedRowMatrix.
computeSVD(int, boolean, double) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Computes singular value decomposition of this matrix.
computeThresholdByKey(Map<K, AcceptanceResult>, Map<K, Object>) - Static method in class org.apache.spark.util.random.StratifiedSamplingUtils
Given the result returned by getCounts, determine the threshold for accepting items to generate exact sample size.
concat(Column...) - Static method in class org.apache.spark.sql.functions
Concatenates multiple input columns together into a single column.
concat(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Concatenates multiple input columns together into a single column.
concat_ws(String, Column...) - Static method in class org.apache.spark.sql.functions
Concatenates multiple input string columns together into a single string column, using the given separator.
concat_ws(String, Seq<Column>) - Static method in class org.apache.spark.sql.functions
Concatenates multiple input string columns together into a single string column, using the given separator.
Conf(int, int, double, double, double, double, double, double) - Constructor for class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
conf() - Method in class org.apache.spark.SparkEnv
 
conf() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
conf() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
conf() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
conf() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
conf() - Method in class org.apache.spark.sql.hive.RelationConversions
 
conf() - Method in class org.apache.spark.sql.SparkSession
Runtime configuration interface for Spark.
confidence() - Method in class org.apache.spark.mllib.fpm.AssociationRules.Rule
Returns the confidence of the rule.
confidence() - Method in class org.apache.spark.partial.BoundedDouble
 
confidence() - Method in class org.apache.spark.util.sketch.CountMinSketch
Returns the confidence (or delta) of this CountMinSketch.
config(String, String) - Method in class org.apache.spark.sql.SparkSession.Builder
Sets a config option.
config(String, long) - Method in class org.apache.spark.sql.SparkSession.Builder
Sets a config option.
config(String, double) - Method in class org.apache.spark.sql.SparkSession.Builder
Sets a config option.
config(String, boolean) - Method in class org.apache.spark.sql.SparkSession.Builder
Sets a config option.
config(SparkConf) - Method in class org.apache.spark.sql.SparkSession.Builder
Sets a list of config options based on the given SparkConf.
config - Class in org.apache.spark.status
 
config() - Constructor for class org.apache.spark.status.config
 
ConfigEntryWithDefault<T> - Class in org.apache.spark.internal.config
 
ConfigEntryWithDefault(String, List<String>, T, Function1<String, T>, Function1<T, String>, String, boolean) - Constructor for class org.apache.spark.internal.config.ConfigEntryWithDefault
 
ConfigEntryWithDefaultFunction<T> - Class in org.apache.spark.internal.config
 
ConfigEntryWithDefaultFunction(String, List<String>, Function0<T>, Function1<String, T>, Function1<T, String>, String, boolean) - Constructor for class org.apache.spark.internal.config.ConfigEntryWithDefaultFunction
 
ConfigEntryWithDefaultString<T> - Class in org.apache.spark.internal.config
 
ConfigEntryWithDefaultString(String, List<String>, String, Function1<String, T>, Function1<T, String>, String, boolean) - Constructor for class org.apache.spark.internal.config.ConfigEntryWithDefaultString
 
ConfigHelpers - Class in org.apache.spark.internal.config
 
ConfigHelpers() - Constructor for class org.apache.spark.internal.config.ConfigHelpers
 
configTestLog4j(String) - Static method in class org.apache.spark.util.Utils
config a log4j properties used for testsuite
configuration() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
CONFIGURATION_INSTANTIATION_LOCK() - Static method in class org.apache.spark.rdd.HadoopRDD
Configuration's constructor is not threadsafe (see SPARK-1097 and HADOOP-10456).
CONFIGURATION_INSTANTIATION_LOCK() - Static method in class org.apache.spark.rdd.NewHadoopRDD
Configuration's constructor is not threadsafe (see SPARK-1097 and HADOOP-10456).
configureJobPropertiesForStorageHandler(TableDesc, Configuration, boolean) - Static method in class org.apache.spark.sql.hive.HiveTableUtil
 
confusionMatrix() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns confusion matrix: predicted classes are in columns, they are ordered by class label ascending, as in "labels"
connectedComponents() - Method in class org.apache.spark.graphx.GraphOps
Compute the connected component membership of each vertex and return a graph with the vertex value containing the lowest vertex id in the connected component containing that vertex.
connectedComponents(int) - Method in class org.apache.spark.graphx.GraphOps
Compute the connected component membership of each vertex and return a graph with the vertex value containing the lowest vertex id in the connected component containing that vertex.
ConnectedComponents - Class in org.apache.spark.graphx.lib
Connected components algorithm.
ConnectedComponents() - Constructor for class org.apache.spark.graphx.lib.ConnectedComponents
 
consequent() - Method in class org.apache.spark.mllib.fpm.AssociationRules.Rule
 
ConstantInputDStream<T> - Class in org.apache.spark.streaming.dstream
An input stream that always returns the same RDD on each time step.
ConstantInputDStream(StreamingContext, RDD<T>, ClassTag<T>) - Constructor for class org.apache.spark.streaming.dstream.ConstantInputDStream
 
constraints() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
constraints() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
constraints() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
constructTree(org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0.NodeData[]) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
Given a list of nodes from a tree, construct the tree.
constructTrees(RDD<org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0.NodeData>) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
 
constructURIForAuthentication(URI, org.apache.spark.SecurityManager) - Static method in class org.apache.spark.util.Utils
Construct a URI container information used for authentication.
contains(Param<?>) - Method in class org.apache.spark.ml.param.ParamMap
Checks whether a parameter is explicitly specified.
contains(String) - Method in class org.apache.spark.SparkConf
Does the configuration contain a given parameter?
contains(Object) - Method in class org.apache.spark.sql.Column
Contains the other element.
contains(String) - Method in class org.apache.spark.sql.types.Metadata
Tests whether this Metadata contains a binding for a key.
contains(A1) - Static method in class org.apache.spark.sql.types.StructType
 
containsBlock(BlockId) - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return whether the given block is stored in this block manager in O(1) time.
containsChild() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
containsChild() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
containsChild() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
containsChild() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
containsDelimiters() - Method in class org.apache.spark.sql.hive.execution.HiveOptions
 
containsKey(Object) - Method in class org.apache.spark.api.java.JavaUtils.SerializableMapWrapper
 
containsNull() - Method in class org.apache.spark.sql.types.ArrayType
 
containsSlice(GenSeq<B>) - Static method in class org.apache.spark.sql.types.StructType
 
contentType() - Method in class org.apache.spark.ui.JettyUtils.ServletParams
 
context() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
context() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
context() - Static method in class org.apache.spark.api.java.JavaRDD
 
context() - Method in interface org.apache.spark.api.java.JavaRDDLike
The SparkContext that this RDD was created on.
context() - Static method in class org.apache.spark.api.r.RRDD
 
context() - Static method in class org.apache.spark.graphx.EdgeRDD
 
context() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
context() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
context() - Static method in class org.apache.spark.graphx.VertexRDD
 
context() - Method in class org.apache.spark.InterruptibleIterator
 
context(SQLContext) - Static method in class org.apache.spark.ml.r.RWrappers
 
context(SQLContext) - Method in class org.apache.spark.ml.util.MLReader
 
context(SQLContext) - Method in class org.apache.spark.ml.util.MLWriter
 
context() - Static method in class org.apache.spark.rdd.HadoopRDD
 
context() - Static method in class org.apache.spark.rdd.JdbcRDD
 
context() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
context() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
context() - Method in class org.apache.spark.rdd.RDD
The SparkContext that this RDD was created on.
context() - Static method in class org.apache.spark.rdd.UnionRDD
 
context() - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
context() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return the StreamingContext associated with this DStream
context() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
context() - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
context() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
context() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
context() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
context() - Method in class org.apache.spark.streaming.dstream.DStream
Return the StreamingContext associated with this DStream
Continuous() - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
Continuous(long) - Static method in class org.apache.spark.sql.streaming.Trigger
A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.
Continuous(long, TimeUnit) - Static method in class org.apache.spark.sql.streaming.Trigger
A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.
Continuous(Duration) - Static method in class org.apache.spark.sql.streaming.Trigger
(Scala-friendly) A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.
Continuous(String) - Static method in class org.apache.spark.sql.streaming.Trigger
A trigger that continuously processes streaming data, asynchronously checkpointing at the specified interval.
ContinuousDataReader<T> - Interface in org.apache.spark.sql.sources.v2.reader.streaming
A variation on DataReader for use with streaming in continuous processing mode.
ContinuousReader - Interface in org.apache.spark.sql.sources.v2.reader.streaming
A mix-in interface for DataSourceReader.
ContinuousReadSupport - Interface in org.apache.spark.sql.sources.v2
A mix-in interface for DataSourceV2.
ContinuousSplit - Class in org.apache.spark.ml.tree
Split which tests a continuous feature.
conv(Column, int, int) - Static method in class org.apache.spark.sql.functions
Convert a number in a string column from one base to another.
CONVERT_METASTORE_ORC() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
CONVERT_METASTORE_PARQUET() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
CONVERT_METASTORE_PARQUET_WITH_SCHEMA_MERGING() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
convertMatrixColumnsFromML(Dataset<?>, String...) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts matrix columns in an input Dataset to the Matrix type from the new Matrix type under the spark.ml package.
convertMatrixColumnsFromML(Dataset<?>, Seq<String>) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts matrix columns in an input Dataset to the Matrix type from the new Matrix type under the spark.ml package.
convertMatrixColumnsToML(Dataset<?>, String...) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts Matrix columns in an input Dataset from the Matrix type to the new Matrix type under the spark.ml package.
convertMatrixColumnsToML(Dataset<?>, Seq<String>) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts Matrix columns in an input Dataset from the Matrix type to the new Matrix type under the spark.ml package.
convertToCanonicalEdges(Function2<ED, ED, ED>) - Method in class org.apache.spark.graphx.GraphOps
Convert bi-directional edges into uni-directional ones.
convertToNetty() - Method in class org.apache.spark.storage.EncryptedManagedBuffer
 
convertToTimeUnit(long, TimeUnit) - Static method in class org.apache.spark.streaming.ui.UIUtils
Convert milliseconds to the specified unit.
convertVectorColumnsFromML(Dataset<?>, String...) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts vector columns in an input Dataset to the Vector type from the new Vector type under the spark.ml package.
convertVectorColumnsFromML(Dataset<?>, Seq<String>) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts vector columns in an input Dataset to the Vector type from the new Vector type under the spark.ml package.
convertVectorColumnsToML(Dataset<?>, String...) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts vector columns in an input Dataset from the Vector type to the new Vector type under the spark.ml package.
convertVectorColumnsToML(Dataset<?>, Seq<String>) - Static method in class org.apache.spark.mllib.util.MLUtils
Converts vector columns in an input Dataset from the Vector type to the new Vector type under the spark.ml package.
CoordinateMatrix - Class in org.apache.spark.mllib.linalg.distributed
Represents a matrix in coordinate format.
CoordinateMatrix(RDD<MatrixEntry>, long, long) - Constructor for class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
 
CoordinateMatrix(RDD<MatrixEntry>) - Constructor for class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Alternative constructor leaving matrix dimensions to be determined automatically.
copy(ParamMap) - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.LinearSVC
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.LogisticRegression
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.NaiveBayes
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.OneVsRest
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
copy(ParamMap) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.KMeans
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.KMeansModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.LDA
 
copy(ParamMap) - Method in class org.apache.spark.ml.clustering.LocalLDAModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.Estimator
 
copy(ParamMap) - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
copy(ParamMap) - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
copy(ParamMap) - Method in class org.apache.spark.ml.evaluation.Evaluator
 
copy(ParamMap) - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
copy(ParamMap) - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.Binarizer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.Bucketizer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.ColumnPruner
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
copy(ParamMap) - Static method in class org.apache.spark.ml.feature.DCT
 
copy(ParamMap) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.HashingTF
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.IDF
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.IDFModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.Imputer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.ImputerModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.IndexToString
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.Interaction
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.MaxAbsScaler
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.MinHashLSH
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.MinHashLSHModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
copy(ParamMap) - Static method in class org.apache.spark.ml.feature.NGram
 
copy(ParamMap) - Static method in class org.apache.spark.ml.feature.Normalizer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.PCA
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.PCAModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.PolynomialExpansion
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.RFormula
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.RFormulaModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.SQLTransformer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.StandardScaler
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.StringIndexer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.Tokenizer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.VectorAssembler
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.VectorIndexer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.VectorSizeHint
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.VectorSlicer
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.Word2Vec
 
copy(ParamMap) - Method in class org.apache.spark.ml.feature.Word2VecModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
copy(ParamMap) - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
copy(Vector, Vector) - Static method in class org.apache.spark.ml.linalg.BLAS
y = x
copy() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
copy() - Method in class org.apache.spark.ml.linalg.DenseVector
 
copy() - Method in interface org.apache.spark.ml.linalg.Matrix
Get a deep copy of the matrix.
copy() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
copy() - Method in class org.apache.spark.ml.linalg.SparseVector
 
copy() - Method in interface org.apache.spark.ml.linalg.Vector
Makes a deep copy of this vector.
copy(ParamMap) - Method in class org.apache.spark.ml.Model
 
copy() - Method in class org.apache.spark.ml.param.ParamMap
Creates a copy of this param map.
copy(ParamMap) - Method in interface org.apache.spark.ml.param.Params
Creates a copy of this instance with the same UID and some extra params.
copy(ParamMap) - Method in class org.apache.spark.ml.Pipeline
 
copy(ParamMap) - Method in class org.apache.spark.ml.PipelineModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.PipelineStage
 
copy(ParamMap) - Method in class org.apache.spark.ml.Predictor
 
copy(ParamMap) - Method in class org.apache.spark.ml.recommendation.ALS
 
copy(ParamMap) - Method in class org.apache.spark.ml.recommendation.ALSModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.GBTRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.LinearRegression
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.LinearRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
copy(ParamMap) - Method in class org.apache.spark.ml.Transformer
 
copy(ParamMap) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
copy(ParamMap) - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
copy(ParamMap) - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
copy(ParamMap) - Method in class org.apache.spark.ml.UnaryTransformer
 
copy(Vector, Vector) - Static method in class org.apache.spark.mllib.linalg.BLAS
y = x
copy() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
copy() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
copy() - Method in interface org.apache.spark.mllib.linalg.Matrix
Get a deep copy of the matrix.
copy() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
copy() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
copy() - Method in interface org.apache.spark.mllib.linalg.Vector
Makes a deep copy of this vector.
copy() - Method in class org.apache.spark.mllib.random.ExponentialGenerator
 
copy() - Method in class org.apache.spark.mllib.random.GammaGenerator
 
copy() - Method in class org.apache.spark.mllib.random.LogNormalGenerator
 
copy() - Method in class org.apache.spark.mllib.random.PoissonGenerator
 
copy() - Method in interface org.apache.spark.mllib.random.RandomDataGenerator
Returns a copy of the RandomDataGenerator with a new instance of the rng object used in the class when applicable for non-locking concurrent usage.
copy() - Method in class org.apache.spark.mllib.random.StandardNormalGenerator
 
copy() - Method in class org.apache.spark.mllib.random.UniformGenerator
 
copy() - Method in class org.apache.spark.mllib.random.WeibullGenerator
 
copy() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
Returns a shallow copy of this instance.
copy(Kryo, T) - Static method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
copy() - Method in interface org.apache.spark.sql.Row
Make a copy of the current Row object.
copy() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
copy() - Method in class org.apache.spark.sql.vectorized.ColumnarMap
 
copy() - Method in class org.apache.spark.sql.vectorized.ColumnarRow
Revisit this.
copy() - Method in class org.apache.spark.util.AccumulatorV2
Creates a new copy of this accumulator.
copy() - Method in class org.apache.spark.util.CollectionAccumulator
 
copy() - Method in class org.apache.spark.util.DoubleAccumulator
 
copy() - Method in class org.apache.spark.util.LegacyAccumulatorWrapper
 
copy() - Method in class org.apache.spark.util.LongAccumulator
 
copy() - Method in class org.apache.spark.util.StatCounter
Clone this StatCounter
copyAndReset() - Method in class org.apache.spark.util.AccumulatorV2
Creates a new copy of this accumulator, which is zero value.
copyAndReset() - Method in class org.apache.spark.util.CollectionAccumulator
 
copyFileStreamNIO(FileChannel, FileChannel, long, long) - Static method in class org.apache.spark.util.Utils
 
copyStream(InputStream, OutputStream, boolean, boolean) - Static method in class org.apache.spark.util.Utils
Copy all data from an InputStream to an OutputStream.
copyToArray(Object, int) - Static method in class org.apache.spark.sql.types.StructType
 
copyToArray(Object) - Static method in class org.apache.spark.sql.types.StructType
 
copyToArray(Object, int, int) - Static method in class org.apache.spark.sql.types.StructType
 
copyToBuffer(Buffer<B>) - Static method in class org.apache.spark.sql.types.StructType
 
copyValues(T, ParamMap) - Method in interface org.apache.spark.ml.param.Params
Copies param values from this instance to another instance for params shared by them.
cores() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
coresGranted() - Method in class org.apache.spark.status.api.v1.ApplicationInfo
 
coresPerExecutor() - Method in class org.apache.spark.status.api.v1.ApplicationInfo
 
corr(Dataset<?>, String, String) - Static method in class org.apache.spark.ml.stat.Correlation
Experimental Compute the correlation matrix for the input Dataset of Vectors using the specified method.
corr(Dataset<?>, String) - Static method in class org.apache.spark.ml.stat.Correlation
Compute the Pearson correlation matrix for the input Dataset of Vectors.
corr(RDD<Object>, RDD<Object>, String) - Static method in class org.apache.spark.mllib.stat.correlation.Correlations
 
corr(RDD<Vector>) - Static method in class org.apache.spark.mllib.stat.Statistics
Compute the Pearson correlation matrix for the input RDD of Vectors.
corr(RDD<Vector>, String) - Static method in class org.apache.spark.mllib.stat.Statistics
Compute the correlation matrix for the input RDD of Vectors using the specified method.
corr(RDD<Object>, RDD<Object>) - Static method in class org.apache.spark.mllib.stat.Statistics
Compute the Pearson correlation for the input RDDs.
corr(JavaRDD<Double>, JavaRDD<Double>) - Static method in class org.apache.spark.mllib.stat.Statistics
Java-friendly version of corr()
corr(RDD<Object>, RDD<Object>, String) - Static method in class org.apache.spark.mllib.stat.Statistics
Compute the correlation for the input RDDs using the specified method.
corr(JavaRDD<Double>, JavaRDD<Double>, String) - Static method in class org.apache.spark.mllib.stat.Statistics
Java-friendly version of corr()
corr(String, String, String) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Calculates the correlation of two columns of a DataFrame.
corr(String, String) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Calculates the Pearson Correlation Coefficient of two columns of a DataFrame.
corr(Column, Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the Pearson Correlation Coefficient for two columns.
corr(String, String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the Pearson Correlation Coefficient for two columns.
Correlation - Class in org.apache.spark.ml.stat
API for correlation functions in MLlib, compatible with DataFrames and Datasets.
Correlation() - Constructor for class org.apache.spark.ml.stat.Correlation
 
CorrelationNames - Class in org.apache.spark.mllib.stat.correlation
Maintains supported and default correlation names.
CorrelationNames() - Constructor for class org.apache.spark.mllib.stat.correlation.CorrelationNames
 
Correlations - Class in org.apache.spark.mllib.stat.correlation
Delegates computation to the specific correlation object based on the input method name.
Correlations() - Constructor for class org.apache.spark.mllib.stat.correlation.Correlations
 
corresponds(GenSeq<B>, Function2<A, B, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
corrMatrix(RDD<Vector>, String) - Static method in class org.apache.spark.mllib.stat.correlation.Correlations
 
cos(Column) - Static method in class org.apache.spark.sql.functions
 
cos(String) - Static method in class org.apache.spark.sql.functions
 
cosh(Column) - Static method in class org.apache.spark.sql.functions
 
cosh(String) - Static method in class org.apache.spark.sql.functions
 
count() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
count() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
count() - Static method in class org.apache.spark.api.java.JavaRDD
 
count() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the number of elements in the RDD.
count() - Static method in class org.apache.spark.api.r.RRDD
 
count() - Static method in class org.apache.spark.graphx.EdgeRDD
 
count() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
The number of edges in the RDD.
count() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
The number of vertices in the RDD.
count() - Static method in class org.apache.spark.graphx.VertexRDD
 
count() - Method in class org.apache.spark.ml.clustering.ExpectationAggregator
 
count() - Method in class org.apache.spark.ml.regression.AFTAggregator
 
count(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
count(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
count() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Sample size.
count() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Sample size.
count() - Static method in class org.apache.spark.rdd.HadoopRDD
 
count() - Static method in class org.apache.spark.rdd.JdbcRDD
 
count() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
count() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
count() - Method in class org.apache.spark.rdd.RDD
Return the number of elements in the RDD.
count() - Static method in class org.apache.spark.rdd.UnionRDD
 
count() - Method in class org.apache.spark.sql.Dataset
Returns the number of rows in the Dataset.
count(MapFunction<T, Object>) - Static method in class org.apache.spark.sql.expressions.javalang.typed
Count aggregate function.
count(Function1<IN, Object>) - Static method in class org.apache.spark.sql.expressions.scalalang.typed
Count aggregate function.
count(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the number of items in a group.
count(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the number of items in a group.
count() - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Returns a Dataset that contains a tuple with each key and the number of items present for that key.
count() - Method in class org.apache.spark.sql.RelationalGroupedDataset
Count the number of rows for each group.
count(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
count() - Method in class org.apache.spark.status.RDDPartitionSeq
 
count() - Method in class org.apache.spark.storage.ReadableChannelFileRegion
 
count() - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
count() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD has a single element generated by counting each RDD of this DStream.
count() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
count() - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
count() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
count() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
count() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
count() - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD has a single element generated by counting each RDD of this DStream.
count() - Method in class org.apache.spark.util.DoubleAccumulator
Returns the number of elements added to the accumulator.
count() - Method in class org.apache.spark.util.LongAccumulator
Returns the number of elements added to the accumulator.
count() - Method in class org.apache.spark.util.StatCounter
 
countApprox(long, double) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
countApprox(long) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
countApprox(long, double) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
countApprox(long) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
countApprox(long, double) - Static method in class org.apache.spark.api.java.JavaRDD
 
countApprox(long) - Static method in class org.apache.spark.api.java.JavaRDD
 
countApprox(long, double) - Method in interface org.apache.spark.api.java.JavaRDDLike
Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.
countApprox(long) - Method in interface org.apache.spark.api.java.JavaRDDLike
Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.
countApprox(long, double) - Static method in class org.apache.spark.api.r.RRDD
 
countApprox(long, double) - Static method in class org.apache.spark.graphx.EdgeRDD
 
countApprox(long, double) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countApprox(long, double) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countApprox(long, double) - Static method in class org.apache.spark.graphx.VertexRDD
 
countApprox(long, double) - Static method in class org.apache.spark.rdd.HadoopRDD
 
countApprox(long, double) - Static method in class org.apache.spark.rdd.JdbcRDD
 
countApprox(long, double) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countApprox(long, double) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countApprox(long, double) - Method in class org.apache.spark.rdd.RDD
Approximate version of count() that returns a potentially incomplete result within a timeout, even if not all tasks have finished.
countApprox(long, double) - Static method in class org.apache.spark.rdd.UnionRDD
 
countApprox$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
countApprox$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
countApprox$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countApprox$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countApprox$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
countApprox$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
countApprox$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
countApprox$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countApprox$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countApprox$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.api.java.JavaRDD
 
countApproxDistinct(double) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return approximate number of distinct elements in the RDD.
countApproxDistinct(int, int) - Static method in class org.apache.spark.api.r.RRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.api.r.RRDD
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.graphx.EdgeRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.graphx.EdgeRDD
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countApproxDistinct(double) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countApproxDistinct(double) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.graphx.VertexRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.graphx.VertexRDD
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.rdd.HadoopRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.rdd.HadoopRDD
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.rdd.JdbcRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.rdd.JdbcRDD
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countApproxDistinct(int, int) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countApproxDistinct(int, int) - Method in class org.apache.spark.rdd.RDD
Return approximate number of distinct elements in the RDD.
countApproxDistinct(double) - Method in class org.apache.spark.rdd.RDD
Return approximate number of distinct elements in the RDD.
countApproxDistinct(int, int) - Static method in class org.apache.spark.rdd.UnionRDD
 
countApproxDistinct(double) - Static method in class org.apache.spark.rdd.UnionRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.api.r.RRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.graphx.EdgeRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.graphx.VertexRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.rdd.HadoopRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.rdd.JdbcRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countApproxDistinct$default$1() - Static method in class org.apache.spark.rdd.UnionRDD
 
countApproxDistinctByKey(double, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double) - Method in class org.apache.spark.api.java.JavaPairRDD
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(int, int, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return approximate number of distinct values for each key in this RDD.
countApproxDistinctByKey(double) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return approximate number of distinct values for each key in this RDD.
countAsync() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
countAsync() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
countAsync() - Static method in class org.apache.spark.api.java.JavaRDD
 
countAsync() - Method in interface org.apache.spark.api.java.JavaRDDLike
The asynchronous version of count, which returns a future for counting the number of elements in this RDD.
countAsync() - Method in class org.apache.spark.rdd.AsyncRDDActions
Returns a future for counting the number of elements in the RDD.
countByKey() - Method in class org.apache.spark.api.java.JavaPairRDD
Count the number of elements for each key, and return the result to the master as a Map.
countByKey() - Method in class org.apache.spark.rdd.PairRDDFunctions
Count the number of elements for each key, collecting the results to a local Map.
countByKeyApprox(long) - Method in class org.apache.spark.api.java.JavaPairRDD
Approximate version of countByKey that can return a partial result if it does not finish within a timeout.
countByKeyApprox(long, double) - Method in class org.apache.spark.api.java.JavaPairRDD
Approximate version of countByKey that can return a partial result if it does not finish within a timeout.
countByKeyApprox(long, double) - Method in class org.apache.spark.rdd.PairRDDFunctions
Approximate version of countByKey that can return a partial result if it does not finish within a timeout.
countByValue() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
countByValue() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
countByValue() - Static method in class org.apache.spark.api.java.JavaRDD
 
countByValue() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the count of each unique value in this RDD as a map of (value, count) pairs.
countByValue(Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countByValue(Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countByValue(Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return the count of each unique value in this RDD as a local map of (value, count) pairs.
countByValue(Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
countByValue() - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
countByValue(int) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
countByValue() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the counts of each distinct value in each RDD of this DStream.
countByValue(int) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the counts of each distinct value in each RDD of this DStream.
countByValue() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
countByValue(int) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
countByValue() - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
countByValue(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
countByValue() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
countByValue(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
countByValue() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
countByValue(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
countByValue() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
countByValue(int) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
countByValue(int, Ordering<T>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD contains the counts of each distinct value in each RDD of this DStream.
countByValue$default$1() - Static method in class org.apache.spark.api.r.RRDD
 
countByValue$default$1() - Static method in class org.apache.spark.graphx.EdgeRDD
 
countByValue$default$1() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countByValue$default$1() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countByValue$default$1() - Static method in class org.apache.spark.graphx.VertexRDD
 
countByValue$default$1() - Static method in class org.apache.spark.rdd.HadoopRDD
 
countByValue$default$1() - Static method in class org.apache.spark.rdd.JdbcRDD
 
countByValue$default$1() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countByValue$default$1() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countByValue$default$1() - Static method in class org.apache.spark.rdd.UnionRDD
 
countByValueAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
countByValueAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
countByValueAndWindow(Duration, Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the count of distinct elements in RDDs in a sliding window over this DStream.
countByValueAndWindow(Duration, Duration, int) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD contains the count of distinct elements in RDDs in a sliding window over this DStream.
countByValueAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
countByValueAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
countByValueAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
countByValueAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
countByValueAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
countByValueAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
countByValueAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
countByValueAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
countByValueAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
countByValueAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
countByValueAndWindow(Duration, Duration, int, Ordering<T>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD contains the count of distinct elements in RDDs in a sliding window over this DStream.
countByValueApprox(long, double) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
countByValueApprox(long) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
countByValueApprox(long, double) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
countByValueApprox(long) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
countByValueApprox(long, double) - Static method in class org.apache.spark.api.java.JavaRDD
 
countByValueApprox(long) - Static method in class org.apache.spark.api.java.JavaRDD
 
countByValueApprox(long, double) - Method in interface org.apache.spark.api.java.JavaRDDLike
Approximate version of countByValue().
countByValueApprox(long) - Method in interface org.apache.spark.api.java.JavaRDDLike
Approximate version of countByValue().
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countByValueApprox(long, double, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Approximate version of countByValue().
countByValueApprox(long, double, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countByValueApprox$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countByValueApprox$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countByValueApprox$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.api.r.RRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.graphx.EdgeRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.graphx.VertexRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.rdd.HadoopRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.rdd.JdbcRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
countByValueApprox$default$3(long, double) - Static method in class org.apache.spark.rdd.UnionRDD
 
countByWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
countByWindow(Duration, Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD has a single element generated by counting the number of elements in a window over this DStream.
countByWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
countByWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
countByWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
countByWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
countByWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
countByWindow(Duration, Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD has a single element generated by counting the number of elements in a sliding window over this DStream.
countDistinct(Column, Column...) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the number of distinct items in a group.
countDistinct(String, String...) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the number of distinct items in a group.
countDistinct(Column, Seq<Column>) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the number of distinct items in a group.
countDistinct(String, Seq<String>) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the number of distinct items in a group.
COUNTER() - Static method in class org.apache.spark.metrics.sink.StatsdMetricType
 
CountingWritableChannel - Class in org.apache.spark.storage
 
CountingWritableChannel(WritableByteChannel) - Constructor for class org.apache.spark.storage.CountingWritableChannel
 
countMinSketch(String, int, int, int) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Count-min Sketch over a specified column.
countMinSketch(String, double, double, int) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Count-min Sketch over a specified column.
countMinSketch(Column, int, int, int) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Count-min Sketch over a specified column.
countMinSketch(Column, double, double, int) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Builds a Count-min Sketch over a specified column.
CountMinSketch - Class in org.apache.spark.util.sketch
A Count-min sketch is a probabilistic data structure used for cardinality estimation using sub-linear space.
CountMinSketch() - Constructor for class org.apache.spark.util.sketch.CountMinSketch
 
CountMinSketch.Version - Enum in org.apache.spark.util.sketch
 
countTowardsTaskFailures() - Static method in class org.apache.spark.ExceptionFailure
 
countTowardsTaskFailures() - Method in class org.apache.spark.ExecutorLostFailure
 
countTowardsTaskFailures() - Method in class org.apache.spark.FetchFailed
Fetch failures lead to a different failure handling path: (1) we don't abort the stage after 4 task failures, instead we immediately go back to the stage which generated the map output, and regenerate the missing data.
countTowardsTaskFailures() - Static method in class org.apache.spark.Resubmitted
 
countTowardsTaskFailures() - Method in class org.apache.spark.TaskCommitDenied
If a task failed because its attempt to commit was denied, do not count this failure towards failing the stage.
countTowardsTaskFailures() - Method in interface org.apache.spark.TaskFailedReason
Whether this task failure should be counted towards the maximum number of times the task is allowed to fail before the stage is aborted.
countTowardsTaskFailures() - Method in class org.apache.spark.TaskKilled
 
countTowardsTaskFailures() - Static method in class org.apache.spark.TaskResultLost
 
countTowardsTaskFailures() - Static method in class org.apache.spark.UnknownReason
 
CountVectorizer - Class in org.apache.spark.ml.feature
Extracts a vocabulary from document collections and generates a CountVectorizerModel.
CountVectorizer(String) - Constructor for class org.apache.spark.ml.feature.CountVectorizer
 
CountVectorizer() - Constructor for class org.apache.spark.ml.feature.CountVectorizer
 
CountVectorizerModel - Class in org.apache.spark.ml.feature
Converts a text document to a sparse vector of token counts.
CountVectorizerModel(String, String[]) - Constructor for class org.apache.spark.ml.feature.CountVectorizerModel
 
CountVectorizerModel(String[]) - Constructor for class org.apache.spark.ml.feature.CountVectorizerModel
 
cov() - Method in class org.apache.spark.ml.stat.distribution.MultivariateGaussian
 
cov(String, String) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Calculate the sample covariance of two numerical columns of a DataFrame.
covar_pop(Column, Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the population covariance for two columns.
covar_pop(String, String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the population covariance for two columns.
covar_samp(Column, Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sample covariance for two columns.
covar_samp(String, String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sample covariance for two columns.
covs() - Method in class org.apache.spark.ml.clustering.ExpectationAggregator
 
crc32(Column) - Static method in class org.apache.spark.sql.functions
Calculates the cyclic redundancy check value (CRC32) of a binary column and returns the value as a bigint.
CreatableRelationProvider - Interface in org.apache.spark.sql.sources
 
create(boolean, boolean, boolean, boolean, int) - Static method in class org.apache.spark.api.java.StorageLevels
Create a new StorageLevel object.
create(JavaSparkContext, JdbcRDD.ConnectionFactory, String, long, long, int, Function<ResultSet, T>) - Static method in class org.apache.spark.rdd.JdbcRDD
Create an RDD that executes a SQL query on a JDBC connection and reads results.
create(JavaSparkContext, JdbcRDD.ConnectionFactory, String, long, long, int) - Static method in class org.apache.spark.rdd.JdbcRDD
Create an RDD that executes a SQL query on a JDBC connection and reads results.
create(RDD<T>, Function1<Object, Object>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
Create a PartitionPruningRDD.
create(Object...) - Static method in class org.apache.spark.sql.RowFactory
Create a Row from the given arguments.
create(String) - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
use Trigger.ProcessingTime(interval)
create(long, TimeUnit) - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
use Trigger.ProcessingTime(interval, unit)
create(long) - Static method in class org.apache.spark.util.sketch.BloomFilter
Creates a BloomFilter with the expected number of insertions and a default expected false positive probability of 3%.
create(long, double) - Static method in class org.apache.spark.util.sketch.BloomFilter
Creates a BloomFilter with the expected number of insertions and expected false positive probability.
create(long, long) - Static method in class org.apache.spark.util.sketch.BloomFilter
Creates a BloomFilter with given expectedNumItems and numBits, it will pick an optimal numHashFunctions which can minimize fpp for the bloom filter.
create(int, int, int) - Static method in class org.apache.spark.util.sketch.CountMinSketch
Creates a CountMinSketch with given depth, width, and random seed.
create(double, double, int) - Static method in class org.apache.spark.util.sketch.CountMinSketch
Creates a CountMinSketch with given relative error (eps), confidence, and random seed.
createArrayType(DataType) - Static method in class org.apache.spark.sql.types.DataTypes
Creates an ArrayType by specifying the data type of elements (elementType).
createArrayType(DataType, boolean) - Static method in class org.apache.spark.sql.types.DataTypes
Creates an ArrayType by specifying the data type of elements (elementType) and whether the array contains null values (containsNull).
createAttrGroupForAttrNames(String, int, boolean, boolean) - Static method in class org.apache.spark.ml.feature.OneHotEncoderCommon
Creates an `AttributeGroup` with the required number of `BinaryAttribute`.
createBatchDataReaderFactories() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsScanColumnarBatch
Similar to DataSourceReader.createDataReaderFactories(), but returns columnar data in batches.
createCombiner() - Method in class org.apache.spark.Aggregator
 
createCommitter(int) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
createCompiledClass(String, File, TestUtils.JavaSourceFromString, Seq<URL>) - Static method in class org.apache.spark.TestUtils
Creates a compiled class with the source file.
createCompiledClass(String, File, String, String, Seq<URL>) - Static method in class org.apache.spark.TestUtils
Creates a compiled class with the given name.
createContinuousReader(Optional<StructType>, String, DataSourceOptions) - Method in interface org.apache.spark.sql.sources.v2.ContinuousReadSupport
Creates a ContinuousReader to scan the data from this data source.
createCryptoInputStream(InputStream, SparkConf, byte[]) - Static method in class org.apache.spark.security.CryptoStreamUtils
Helper method to wrap InputStream with CryptoInputStream for decryption.
createCryptoOutputStream(OutputStream, SparkConf, byte[]) - Static method in class org.apache.spark.security.CryptoStreamUtils
Helper method to wrap OutputStream with CryptoOutputStream for encryption.
createDataFrame(RDD<A>, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a DataFrame from an RDD of Product (e.g.
createDataFrame(Seq<A>, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a DataFrame from a local Seq of Product.
createDataFrame(RDD<Row>, StructType) - Method in class org.apache.spark.sql.SparkSession
Developer API Creates a DataFrame from an RDD containing Rows using the given schema.
createDataFrame(JavaRDD<Row>, StructType) - Method in class org.apache.spark.sql.SparkSession
Developer API Creates a DataFrame from a JavaRDD containing Rows using the given schema.
createDataFrame(List<Row>, StructType) - Method in class org.apache.spark.sql.SparkSession
Developer API Creates a DataFrame from a java.util.List containing Rows using the given schema.
createDataFrame(RDD<?>, Class<?>) - Method in class org.apache.spark.sql.SparkSession
Applies a schema to an RDD of Java Beans.
createDataFrame(JavaRDD<?>, Class<?>) - Method in class org.apache.spark.sql.SparkSession
Applies a schema to an RDD of Java Beans.
createDataFrame(List<?>, Class<?>) - Method in class org.apache.spark.sql.SparkSession
Applies a schema to a List of Java Beans.
createDataFrame(RDD<A>, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SQLContext
 
createDataFrame(Seq<A>, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SQLContext
 
createDataFrame(RDD<Row>, StructType) - Method in class org.apache.spark.sql.SQLContext
 
createDataFrame(JavaRDD<Row>, StructType) - Method in class org.apache.spark.sql.SQLContext
 
createDataFrame(List<Row>, StructType) - Method in class org.apache.spark.sql.SQLContext
 
createDataFrame(RDD<?>, Class<?>) - Method in class org.apache.spark.sql.SQLContext
 
createDataFrame(JavaRDD<?>, Class<?>) - Method in class org.apache.spark.sql.SQLContext
 
createDataFrame(List<?>, Class<?>) - Method in class org.apache.spark.sql.SQLContext
 
createDataReader() - Method in interface org.apache.spark.sql.sources.v2.reader.DataReaderFactory
Returns a data reader to do the actual reading work.
createDataReaderFactories() - Method in interface org.apache.spark.sql.sources.v2.reader.DataSourceReader
Returns a list of reader factories.
createDataReaderFactories() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsScanColumnarBatch
 
createDataReaderFactories() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsScanUnsafeRow
 
createDataset(Seq<T>, Encoder<T>) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a Dataset from a local Seq of data of a given type.
createDataset(RDD<T>, Encoder<T>) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a Dataset from an RDD of a given type.
createDataset(List<T>, Encoder<T>) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a Dataset from a java.util.List of a given type.
createDataset(Seq<T>, Encoder<T>) - Method in class org.apache.spark.sql.SQLContext
 
createDataset(RDD<T>, Encoder<T>) - Method in class org.apache.spark.sql.SQLContext
 
createDataset(List<T>, Encoder<T>) - Method in class org.apache.spark.sql.SQLContext
 
createDataWriter(int, int) - Method in interface org.apache.spark.sql.sources.v2.writer.DataWriterFactory
Returns a data writer to do the actual writing work.
createDecimalType(int, int) - Static method in class org.apache.spark.sql.types.DataTypes
Creates a DecimalType by specifying the precision and scale.
createDecimalType() - Static method in class org.apache.spark.sql.types.DataTypes
Creates a DecimalType with default precision and scale, which are 10 and 0.
createDF(RDD<byte[]>, StructType, SparkSession) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
createDirectory(String, String) - Static method in class org.apache.spark.util.Utils
Create a directory inside the given parent directory.
createdTempDir() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
createdTempDir() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
createdTempDir_$eq(Option<Path>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
createdTempDir_$eq(Option<Path>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
createExternalTable(String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Deprecated.
use createTable instead. Since 2.2.0.
createExternalTable(String, String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Deprecated.
use createTable instead. Since 2.2.0.
createExternalTable(String, String, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Deprecated.
use createTable instead. Since 2.2.0.
createExternalTable(String, String, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Deprecated.
use createTable instead. Since 2.2.0.
createExternalTable(String, String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Deprecated.
use createTable instead. Since 2.2.0.
createExternalTable(String, String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Deprecated.
use createTable instead. Since 2.2.0.
createExternalTable(String, String) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
use sparkSession.catalog.createTable instead. Since 2.2.0.
createExternalTable(String, String, String) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
use sparkSession.catalog.createTable instead. Since 2.2.0.
createExternalTable(String, String, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
use sparkSession.catalog.createTable instead. Since 2.2.0.
createExternalTable(String, String, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
use sparkSession.catalog.createTable instead. Since 2.2.0.
createExternalTable(String, String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
use sparkSession.catalog.createTable instead. Since 2.2.0.
createExternalTable(String, String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
use sparkSession.catalog.createTable instead. Since 2.2.0.
createFilter(StructType, Filter[]) - Static method in class org.apache.spark.sql.hive.orc.OrcFilters
 
createGlobalTempView(String) - Method in class org.apache.spark.sql.Dataset
Creates a global temporary view using the given name.
CreateHiveTableAsSelectCommand - Class in org.apache.spark.sql.hive.execution
Create table and insert the query result into it.
CreateHiveTableAsSelectCommand(CatalogTable, LogicalPlan, Seq<String>, SaveMode) - Constructor for class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
createInputStream() - Method in class org.apache.spark.storage.EncryptedManagedBuffer
 
createInternalRowWriterFactory() - Method in interface org.apache.spark.sql.sources.v2.writer.SupportsWriteInternalRow
 
createJar(Seq<File>, File, Option<String>) - Static method in class org.apache.spark.TestUtils
Create a jar file that contains this set of files.
createJarWithClasses(Seq<String>, String, Seq<Tuple2<String, String>>, Seq<URL>) - Static method in class org.apache.spark.TestUtils
Create a jar that defines classes with the given names.
createJarWithFiles(Map<String, String>, File) - Static method in class org.apache.spark.TestUtils
Create a jar file containing multiple files.
createJobContext(String, int) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
createJobID(Date, int) - Static method in class org.apache.spark.internal.io.SparkHadoopWriterUtils
 
createJobTrackerID(Date) - Static method in class org.apache.spark.internal.io.SparkHadoopWriterUtils
 
createKey(SparkConf) - Static method in class org.apache.spark.security.CryptoStreamUtils
Creates a new encryption key.
createLogForDriver(SparkConf, String, Configuration) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
Create a WriteAheadLog for the driver.
createLogForReceiver(SparkConf, String, Configuration) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
Create a WriteAheadLog for the receiver.
createMapType(DataType, DataType) - Static method in class org.apache.spark.sql.types.DataTypes
Creates a MapType by specifying the data type of keys (keyType) and values (keyType).
createMapType(DataType, DataType, boolean) - Static method in class org.apache.spark.sql.types.DataTypes
Creates a MapType by specifying the data type of keys (keyType), the data type of values (keyType), and whether values contain any null value (valueContainsNull).
createMetrics(long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long, long) - Static method in class org.apache.spark.status.LiveEntityHelpers
 
createMetrics(long) - Static method in class org.apache.spark.status.LiveEntityHelpers
 
createMicroBatchReader(Optional<StructType>, String, DataSourceOptions) - Method in interface org.apache.spark.sql.sources.v2.MicroBatchReadSupport
Creates a MicroBatchReader to read batches of data from this data source in a streaming query.
createOrReplaceGlobalTempView(String) - Method in class org.apache.spark.sql.Dataset
Creates or replaces a global temporary view using the given name.
createOrReplaceTempView(String) - Method in class org.apache.spark.sql.Dataset
Creates a local temporary view using the given name.
createOutputOperationFailureForUI(String) - Static method in class org.apache.spark.streaming.ui.UIUtils
 
createPathFromString(String, JobConf) - Static method in class org.apache.spark.internal.io.SparkHadoopWriterUtils
 
createPMMLModelExport(Object) - Static method in class org.apache.spark.mllib.pmml.export.PMMLModelExportFactory
Factory object to help creating the necessary PMMLModelExport implementation taking as input the machine learning model (for example KMeansModel).
createProxyHandler(Function1<String, Option<String>>) - Static method in class org.apache.spark.ui.JettyUtils
Create a handler for proxying request to Workers and Application Drivers
createProxyLocationHeader(String, HttpServletRequest, URI) - Static method in class org.apache.spark.ui.JettyUtils
 
createProxyURI(String, String, String, String) - Static method in class org.apache.spark.ui.JettyUtils
 
createRDDFromArray(JavaSparkContext, byte[][]) - Static method in class org.apache.spark.api.r.RRDD
Create an RRDD given a sequence of byte arrays.
createRDDFromFile(JavaSparkContext, String, int) - Static method in class org.apache.spark.api.r.RRDD
Create an RRDD given a temporary file name.
createReadableChannel(ReadableByteChannel, SparkConf, byte[]) - Static method in class org.apache.spark.security.CryptoStreamUtils
Wrap a ReadableByteChannel for decryption.
createReader(DataSourceOptions) - Method in interface org.apache.spark.sql.sources.v2.ReadSupport
Creates a DataSourceReader to scan the data from this data source.
createReader(StructType, DataSourceOptions) - Method in interface org.apache.spark.sql.sources.v2.ReadSupportWithSchema
Create a DataSourceReader to scan the data from this data source.
createRedirectHandler(String, String, Function1<HttpServletRequest, BoxedUnit>, String, Set<String>) - Static method in class org.apache.spark.ui.JettyUtils
Create a handler that always redirects the user to the given path
createRelation(SQLContext, SaveMode, Map<String, String>, Dataset<Row>) - Method in interface org.apache.spark.sql.sources.CreatableRelationProvider
Saves a DataFrame to a destination (using data source-specific parameters)
createRelation(SQLContext, Map<String, String>) - Method in interface org.apache.spark.sql.sources.RelationProvider
Returns a new base relation with the given parameters.
createRelation(SQLContext, Map<String, String>, StructType) - Method in interface org.apache.spark.sql.sources.SchemaRelationProvider
Returns a new base relation with the given parameters and user defined schema.
createSecret(SparkConf) - Static method in class org.apache.spark.util.Utils
 
createServlet(JettyUtils.ServletParams<T>, org.apache.spark.SecurityManager, SparkConf) - Static method in class org.apache.spark.ui.JettyUtils
 
createServletHandler(String, JettyUtils.ServletParams<T>, org.apache.spark.SecurityManager, SparkConf, String) - Static method in class org.apache.spark.ui.JettyUtils
Create a context handler that responds to a request with the given path prefix
createServletHandler(String, HttpServlet, String) - Static method in class org.apache.spark.ui.JettyUtils
Create a context handler that responds to a request with the given path prefix
createSink(SQLContext, Map<String, String>, Seq<String>, OutputMode) - Method in interface org.apache.spark.sql.sources.StreamSinkProvider
 
createSource(SQLContext, String, Option<StructType>, String, Map<String, String>) - Method in interface org.apache.spark.sql.sources.StreamSourceProvider
 
createSparkContext(String, String, String, String[], Map<Object, Object>, Map<Object, Object>) - Static method in class org.apache.spark.api.r.RRDD
 
createStaticHandler(String, String) - Static method in class org.apache.spark.ui.JettyUtils
Create a handler for serving files from a static directory
createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, ClassTag<T>) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, String, String, ClassTag<T>) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function1<Record, T>, String, String, String, String, String, ClassTag<T>) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(StreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, String, String) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>, String, String) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, Function<Record, T>, Class<T>, String, String, String, String, String) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(JavaStreamingContext, String, String, String, String, InitialPositionInStream, Duration, StorageLevel, String, String) - Static method in class org.apache.spark.streaming.kinesis.KinesisUtils
Deprecated.
Use KinesisInputDStream.builder instead. Since 2.2.0.
createStream(JavaStreamingContext, String, String, String, String, int, Duration, StorageLevel, String, String, String, String, String) - Method in class org.apache.spark.streaming.kinesis.KinesisUtilsPythonHelper
 
createStreamWriter(String, StructType, OutputMode, DataSourceOptions) - Method in interface org.apache.spark.sql.sources.v2.StreamWriteSupport
Creates an optional StreamWriter to save the data to this data source.
createStructField(String, String, boolean) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
createStructField(String, DataType, boolean, Metadata) - Static method in class org.apache.spark.sql.types.DataTypes
Creates a StructField by specifying the name (name), data type (dataType) and whether values of this field can be null values (nullable).
createStructField(String, DataType, boolean) - Static method in class org.apache.spark.sql.types.DataTypes
Creates a StructField with empty metadata.
createStructType(Seq<StructField>) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
createStructType(List<StructField>) - Static method in class org.apache.spark.sql.types.DataTypes
Creates a StructType with the given list of StructFields (fields).
createStructType(StructField[]) - Static method in class org.apache.spark.sql.types.DataTypes
Creates a StructType with the given StructField array (fields).
createTable(String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Experimental Creates a table from the given path and returns the corresponding DataFrame.
createTable(String, String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Experimental Creates a table from the given path based on a data source and returns the corresponding DataFrame.
createTable(String, String, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Experimental Creates a table based on the dataset in a data source and a set of options.
createTable(String, String, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Experimental (Scala-specific) Creates a table based on the dataset in a data source and a set of options.
createTable(String, String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Experimental Create a table based on the dataset in a data source, a schema and a set of options.
createTable(String, String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.catalog.Catalog
Experimental (Scala-specific) Create a table based on the dataset in a data source, a schema and a set of options.
createTaskAttemptContext(String, int, int, int) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
createTempDir(String, String) - Static method in class org.apache.spark.util.Utils
Create a temporary directory inside the given parent directory.
createTempView(String) - Method in class org.apache.spark.sql.Dataset
Creates a local temporary view using the given name.
createUnsafe(long, int, int) - Static method in class org.apache.spark.sql.types.Decimal
Creates a decimal from unscaled, precision and scale without checking the bounds.
createUnsafeRowReaderFactories() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsScanUnsafeRow
Similar to DataSourceReader.createDataReaderFactories(), but returns data in unsafe row format.
createWorkspace(int) - Static method in class org.apache.spark.mllib.optimization.NNLS
 
createWritableChannel(WritableByteChannel, SparkConf, byte[]) - Static method in class org.apache.spark.security.CryptoStreamUtils
Wrap a WritableByteChannel for encryption.
createWriter(String, StructType, SaveMode, DataSourceOptions) - Method in interface org.apache.spark.sql.sources.v2.WriteSupport
Creates an optional DataSourceWriter to save the data to this data source.
createWriterFactory() - Method in interface org.apache.spark.sql.sources.v2.writer.DataSourceWriter
Creates a writer factory which will be serialized and sent to executors.
createWriterFactory() - Method in interface org.apache.spark.sql.sources.v2.writer.SupportsWriteInternalRow
 
crossJoin(Dataset<?>) - Method in class org.apache.spark.sql.Dataset
Explicit cartesian join with another DataFrame.
crosstab(String, String) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Computes a pair-wise frequency table of the given columns.
CrossValidator - Class in org.apache.spark.ml.tuning
K-fold cross validation performs model selection by splitting the dataset into a set of non-overlapping randomly partitioned folds which are used as separate training and test datasets e.g., with k=3 folds, K-fold cross validation will generate 3 (training, test) dataset pairs, each of which uses 2/3 of the data for training and 1/3 for testing.
CrossValidator(String) - Constructor for class org.apache.spark.ml.tuning.CrossValidator
 
CrossValidator() - Constructor for class org.apache.spark.ml.tuning.CrossValidator
 
CrossValidatorModel - Class in org.apache.spark.ml.tuning
CrossValidatorModel contains the model with the highest average cross-validation metric across folds and uses this model to transform input data.
CrossValidatorModel.CrossValidatorModelWriter - Class in org.apache.spark.ml.tuning
Writer for CrossValidatorModel.
CryptoStreamUtils - Class in org.apache.spark.security
A util class for manipulating IO encryption and decryption streams.
CryptoStreamUtils() - Constructor for class org.apache.spark.security.CryptoStreamUtils
 
csv(String...) - Method in class org.apache.spark.sql.DataFrameReader
Loads CSV files and returns the result as a DataFrame.
csv(String) - Method in class org.apache.spark.sql.DataFrameReader
Loads a CSV file and returns the result as a DataFrame.
csv(Dataset<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads an Dataset[String] storing CSV rows and returns the result as a DataFrame.
csv(Seq<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads CSV files and returns the result as a DataFrame.
csv(String) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame in CSV format at the specified path.
csv(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads a CSV file stream and returns the result as a DataFrame.
cube(Column...) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them.
cube(String, String...) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them.
cube(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them.
cube(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional cube for the current Dataset using the specified columns, so we can run aggregation on them.
CubeType$() - Constructor for class org.apache.spark.sql.RelationalGroupedDataset.CubeType$
 
cume_dist() - Static method in class org.apache.spark.sql.functions
Window function: returns the cumulative distribution of values within a window partition, i.e.
current_date() - Static method in class org.apache.spark.sql.functions
Returns the current date as a date column.
current_timestamp() - Static method in class org.apache.spark.sql.functions
Returns the current timestamp as a timestamp column.
currentAttemptId() - Method in interface org.apache.spark.SparkStageInfo
 
currentAttemptId() - Method in class org.apache.spark.SparkStageInfoImpl
 
currentDatabase() - Method in class org.apache.spark.sql.catalog.Catalog
Returns the current default database in this session.
currentRow() - Static method in class org.apache.spark.sql.expressions.Window
Value representing the current row.
currentRow() - Static method in class org.apache.spark.sql.functions
Window function: returns the special frame boundary that represents the current row in the window partition.
currPrefLocs(Partition, RDD<?>) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 

D

dapply(Dataset<Row>, byte[], byte[], Object[], StructType) - Static method in class org.apache.spark.sql.api.r.SQLUtils
The helper function for dapply() on R side.
Data(Vector, double, Option<Object>) - Constructor for class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$.Data
 
Data(double[], double[], double[][]) - Constructor for class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$.Data
 
Data(double[], double[], double[][], String) - Constructor for class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$.Data
 
Data(int) - Constructor for class org.apache.spark.mllib.feature.ChiSqSelectorModel.SaveLoadV1_0$.Data
 
Data(Vector, double) - Constructor for class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$.Data
 
data() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.LaunchTask
 
data() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
Database - Class in org.apache.spark.sql.catalog
A database in Spark, as returned by the listDatabases method defined in Catalog.
Database(String, String, String) - Constructor for class org.apache.spark.sql.catalog.Database
 
database() - Method in class org.apache.spark.sql.catalog.Function
 
database() - Method in class org.apache.spark.sql.catalog.Table
 
databaseExists(String) - Method in class org.apache.spark.sql.catalog.Catalog
Check if the database with the specified name exists.
databaseTypeDefinition() - Method in class org.apache.spark.sql.jdbc.JdbcType
 
dataDistribution() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
DATAFRAME_DAPPLY() - Static method in class org.apache.spark.api.r.RRunnerModes
 
DATAFRAME_GAPPLY() - Static method in class org.apache.spark.api.r.RRunnerModes
 
DataFrameNaFunctions - Class in org.apache.spark.sql
Functionality for working with missing data in DataFrames.
DataFrameReader - Class in org.apache.spark.sql
Interface used to load a Dataset from external storage systems (e.g.
DataFrameStatFunctions - Class in org.apache.spark.sql
Statistic functions for DataFrames.
DataFrameWriter<T> - Class in org.apache.spark.sql
Interface used to write a Dataset to external storage systems (e.g.
DataReader<T> - Interface in org.apache.spark.sql.sources.v2.reader
A data reader returned by DataReaderFactory.createDataReader() and is responsible for outputting data for a RDD partition.
DataReaderFactory<T> - Interface in org.apache.spark.sql.sources.v2.reader
A reader factory returned by DataSourceReader.createDataReaderFactories() and is responsible for creating the actual data reader.
Dataset<T> - Class in org.apache.spark.sql
A Dataset is a strongly typed collection of domain-specific objects that can be transformed in parallel using functional or relational operations.
Dataset(SparkSession, LogicalPlan, Encoder<T>) - Constructor for class org.apache.spark.sql.Dataset
 
Dataset(SQLContext, LogicalPlan, Encoder<T>) - Constructor for class org.apache.spark.sql.Dataset
 
DatasetHolder<T> - Class in org.apache.spark.sql
A container for a Dataset, used for implicit conversions in Scala.
DataSourceOptions - Class in org.apache.spark.sql.sources.v2
An immutable string-to-string map in which keys are case-insensitive.
DataSourceOptions(Map<String, String>) - Constructor for class org.apache.spark.sql.sources.v2.DataSourceOptions
 
DataSourceReader - Interface in org.apache.spark.sql.sources.v2.reader
DataSourceRegister - Interface in org.apache.spark.sql.sources
Data sources should implement this trait so that they can register an alias to their data source.
DataSourceV2 - Interface in org.apache.spark.sql.sources.v2
The base interface for data source v2.
DataSourceWriter - Interface in org.apache.spark.sql.sources.v2.writer
DataStreamReader - Class in org.apache.spark.sql.streaming
Interface used to load a streaming Dataset from external storage systems (e.g.
DataStreamWriter<T> - Class in org.apache.spark.sql.streaming
Interface used to write a streaming Dataset to external storage systems (e.g.
dataTablesHeaderNodes() - Static method in class org.apache.spark.ui.UIUtils
 
dataType() - Method in class org.apache.spark.sql.catalog.Column
 
dataType() - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
The DataType of the returned value of this UserDefinedAggregateFunction.
dataType() - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
DataType - Class in org.apache.spark.sql.types
The base type of all Spark SQL data types.
DataType() - Constructor for class org.apache.spark.sql.types.DataType
 
dataType() - Method in class org.apache.spark.sql.types.StructField
 
dataType() - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the data type of this column vector.
DataTypes - Class in org.apache.spark.sql.types
To get/create specific data type, users should use singleton objects and factory methods provided by this class.
DataTypes() - Constructor for class org.apache.spark.sql.types.DataTypes
 
DataValidators - Class in org.apache.spark.mllib.util
Developer API A collection of methods used to validate data before applying ML algorithms.
DataValidators() - Constructor for class org.apache.spark.mllib.util.DataValidators
 
DataWriter<T> - Interface in org.apache.spark.sql.sources.v2.writer
A data writer returned by DataWriterFactory.createDataWriter(int, int) and is responsible for writing data for an input RDD partition.
DataWriterFactory<T> - Interface in org.apache.spark.sql.sources.v2.writer
A factory of DataWriter returned by DataSourceWriter.createWriterFactory(), which is responsible for creating and initializing the actual data writer at executor side.
date() - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type date.
DATE() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable date type.
date_add(Column, int) - Static method in class org.apache.spark.sql.functions
Returns the date that is days days after start
date_format(Column, String) - Static method in class org.apache.spark.sql.functions
Converts a date/timestamp/string to a value of string in the format specified by the date format given by the second argument.
date_sub(Column, int) - Static method in class org.apache.spark.sql.functions
Returns the date that is days days before start
date_trunc(String, Column) - Static method in class org.apache.spark.sql.functions
Returns timestamp truncated to the unit specified by the format.
datediff(Column, Column) - Static method in class org.apache.spark.sql.functions
Returns the number of days from start to end.
DateType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the DateType object.
DateType - Class in org.apache.spark.sql.types
A date type, supporting "0001-01-01" through "9999-12-31".
dayofmonth(Column) - Static method in class org.apache.spark.sql.functions
Extracts the day of the month as an integer from a given date/timestamp/string.
dayofweek(Column) - Static method in class org.apache.spark.sql.functions
Extracts the day of the week as an integer from a given date/timestamp/string.
dayofyear(Column) - Static method in class org.apache.spark.sql.functions
Extracts the day of the year as an integer from a given date/timestamp/string.
DB2Dialect - Class in org.apache.spark.sql.jdbc
 
DB2Dialect() - Constructor for class org.apache.spark.sql.jdbc.DB2Dialect
 
DCT - Class in org.apache.spark.ml.feature
A feature transformer that takes the 1D discrete cosine transform of a real vector.
DCT(String) - Constructor for class org.apache.spark.ml.feature.DCT
 
DCT() - Constructor for class org.apache.spark.ml.feature.DCT
 
deallocate() - Method in class org.apache.spark.storage.ReadableChannelFileRegion
 
decayFactor() - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
 
decimal() - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type decimal.
decimal(int, int) - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type decimal.
DECIMAL() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable decimal type.
Decimal - Class in org.apache.spark.sql.types
A mutable implementation of BigDecimal that can hold a Long if values are small enough.
Decimal() - Constructor for class org.apache.spark.sql.types.Decimal
 
Decimal.DecimalAsIfIntegral$ - Class in org.apache.spark.sql.types
A Integral evidence parameter for Decimals.
Decimal.DecimalIsFractional$ - Class in org.apache.spark.sql.types
A Fractional evidence parameter for Decimals.
DecimalAsIfIntegral$() - Constructor for class org.apache.spark.sql.types.Decimal.DecimalAsIfIntegral$
 
DecimalIsFractional$() - Constructor for class org.apache.spark.sql.types.Decimal.DecimalIsFractional$
 
DecimalType - Class in org.apache.spark.sql.types
The data type representing java.math.BigDecimal values.
DecimalType(int, int) - Constructor for class org.apache.spark.sql.types.DecimalType
 
DecimalType(int) - Constructor for class org.apache.spark.sql.types.DecimalType
 
DecimalType() - Constructor for class org.apache.spark.sql.types.DecimalType
 
DecimalType.Expression$ - Class in org.apache.spark.sql.types
 
DecimalType.Fixed$ - Class in org.apache.spark.sql.types
 
DecisionTree - Class in org.apache.spark.mllib.tree
A class which implements a decision tree learning algorithm for classification and regression.
DecisionTree(Strategy) - Constructor for class org.apache.spark.mllib.tree.DecisionTree
 
DecisionTreeClassificationModel - Class in org.apache.spark.ml.classification
Decision tree model (http://en.wikipedia.org/wiki/Decision_tree_learning) for classification.
DecisionTreeClassifier - Class in org.apache.spark.ml.classification
Decision tree learning algorithm (http://en.wikipedia.org/wiki/Decision_tree_learning) for classification.
DecisionTreeClassifier(String) - Constructor for class org.apache.spark.ml.classification.DecisionTreeClassifier
 
DecisionTreeClassifier() - Constructor for class org.apache.spark.ml.classification.DecisionTreeClassifier
 
DecisionTreeModel - Class in org.apache.spark.mllib.tree.model
Decision tree model for classification or regression.
DecisionTreeModel(Node, Enumeration.Value) - Constructor for class org.apache.spark.mllib.tree.model.DecisionTreeModel
 
DecisionTreeModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.tree.model
 
DecisionTreeModel.SaveLoadV1_0$.NodeData - Class in org.apache.spark.mllib.tree.model
Model data for model import/export
DecisionTreeModel.SaveLoadV1_0$.PredictData - Class in org.apache.spark.mllib.tree.model
 
DecisionTreeModel.SaveLoadV1_0$.SplitData - Class in org.apache.spark.mllib.tree.model
 
DecisionTreeModelReadWrite - Class in org.apache.spark.ml.tree
Helper classes for tree model persistence
DecisionTreeModelReadWrite() - Constructor for class org.apache.spark.ml.tree.DecisionTreeModelReadWrite
 
DecisionTreeModelReadWrite.NodeData - Class in org.apache.spark.ml.tree
Info for a Node
DecisionTreeModelReadWrite.NodeData$ - Class in org.apache.spark.ml.tree
 
DecisionTreeModelReadWrite.SplitData - Class in org.apache.spark.ml.tree
Info for a Split
DecisionTreeModelReadWrite.SplitData$ - Class in org.apache.spark.ml.tree
 
DecisionTreeRegressionModel - Class in org.apache.spark.ml.regression
Decision tree (Wikipedia) model for regression.
DecisionTreeRegressor - Class in org.apache.spark.ml.regression
Decision tree learning algorithm for regression.
DecisionTreeRegressor(String) - Constructor for class org.apache.spark.ml.regression.DecisionTreeRegressor
 
DecisionTreeRegressor() - Constructor for class org.apache.spark.ml.regression.DecisionTreeRegressor
 
decode(Column, String) - Static method in class org.apache.spark.sql.functions
Computes the first argument into a string from a binary using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
decodeFileNameInURI(URI) - Static method in class org.apache.spark.util.Utils
Get the file name from uri's raw path and decode it.
decodeLabel(Vector) - Static method in class org.apache.spark.ml.classification.LabelConverter
Converts a vector to a label.
decodeURLParameter(String) - Static method in class org.apache.spark.ui.UIUtils
Decode URLParameter if URL is encoded by YARN-WebAppProxyServlet.
DEFAULT_CONNECTION_TIMEOUT() - Static method in class org.apache.spark.api.r.SparkRDefaults
 
DEFAULT_DRIVER_MEM_MB() - Static method in class org.apache.spark.util.Utils
Define a default value for driver memory here since this value is referenced across the code base and nearly all files already use Utils.scala
DEFAULT_HEARTBEAT_INTERVAL() - Static method in class org.apache.spark.api.r.SparkRDefaults
 
DEFAULT_MAX_FAILURES() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
DEFAULT_MAX_TO_STRING_FIELDS() - Static method in class org.apache.spark.util.Utils
The performance overhead of creating and logging strings for wide schemas can be large.
DEFAULT_NUM_RBACKEND_THREADS() - Static method in class org.apache.spark.api.r.SparkRDefaults
 
DEFAULT_NUMBER_EXECUTORS() - Static method in class org.apache.spark.scheduler.cluster.SchedulerBackendUtils
 
DEFAULT_ROLLING_INTERVAL_SECS() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
DEFAULT_SHUTDOWN_PRIORITY() - Static method in class org.apache.spark.util.ShutdownHookManager
 
defaultAttr() - Static method in class org.apache.spark.ml.attribute.BinaryAttribute
The default binary attribute.
defaultAttr() - Static method in class org.apache.spark.ml.attribute.NominalAttribute
The default nominal attribute.
defaultAttr() - Static method in class org.apache.spark.ml.attribute.NumericAttribute
The default numeric attribute.
defaultCopy(ParamMap) - Method in interface org.apache.spark.ml.param.Params
Default implementation of copy with extra params.
defaultCorrName() - Static method in class org.apache.spark.mllib.stat.correlation.CorrelationNames
 
DefaultCredentials - Class in org.apache.spark.streaming.kinesis
Returns DefaultAWSCredentialsProviderChain for authentication.
DefaultCredentials() - Constructor for class org.apache.spark.streaming.kinesis.DefaultCredentials
 
defaultLink() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
 
defaultLink() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
 
defaultLink() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
defaultLink() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
 
defaultMinPartitions() - Method in class org.apache.spark.api.java.JavaSparkContext
Default min number of partitions for Hadoop RDDs when not given by user
defaultMinPartitions() - Method in class org.apache.spark.SparkContext
Default min number of partitions for Hadoop RDDs when not given by user Notice that we use math.min so the "defaultMinPartitions" cannot be higher than 2.
defaultParallelism() - Method in class org.apache.spark.api.java.JavaSparkContext
Default level of parallelism to use when not given by user (e.g.
defaultParallelism() - Method in class org.apache.spark.SparkContext
Default level of parallelism to use when not given by user (e.g.
defaultParamMap() - Method in interface org.apache.spark.ml.param.Params
Internal param map for default values.
defaultParams(String) - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
Returns default configuration for the boosting algorithm
defaultParams(Enumeration.Value) - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
Returns default configuration for the boosting algorithm
DefaultParamsReadable<T> - Interface in org.apache.spark.ml.util
Developer API
DefaultParamsWritable - Interface in org.apache.spark.ml.util
Developer API
DefaultPartitionCoalescer - Class in org.apache.spark.rdd
Coalesce the partitions of a parent RDD (prev) into fewer partitions, so that each partition of this RDD computes one or more of the parent ones.
DefaultPartitionCoalescer(double) - Constructor for class org.apache.spark.rdd.DefaultPartitionCoalescer
 
DefaultPartitionCoalescer.PartitionLocations - Class in org.apache.spark.rdd
 
defaultPartitioner(RDD<?>, Seq<RDD<?>>) - Static method in class org.apache.spark.Partitioner
Choose a partitioner to use for a cogroup-like operation between a number of RDDs.
defaultSize() - Method in class org.apache.spark.sql.types.ArrayType
The default size of a value of the ArrayType is the default size of the element type.
defaultSize() - Method in class org.apache.spark.sql.types.BinaryType
The default size of a value of the BinaryType is 100 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.BooleanType
The default size of a value of the BooleanType is 1 byte.
defaultSize() - Method in class org.apache.spark.sql.types.ByteType
The default size of a value of the ByteType is 1 byte.
defaultSize() - Method in class org.apache.spark.sql.types.CalendarIntervalType
 
defaultSize() - Static method in class org.apache.spark.sql.types.CharType
 
defaultSize() - Method in class org.apache.spark.sql.types.DataType
The default size of a value of this data type, used internally for size estimation.
defaultSize() - Method in class org.apache.spark.sql.types.DateType
The default size of a value of the DateType is 4 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.DecimalType
The default size of a value of the DecimalType is 8 bytes when precision is at most 18, and 16 bytes otherwise.
defaultSize() - Method in class org.apache.spark.sql.types.DoubleType
The default size of a value of the DoubleType is 8 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.FloatType
The default size of a value of the FloatType is 4 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.HiveStringType
 
defaultSize() - Method in class org.apache.spark.sql.types.IntegerType
The default size of a value of the IntegerType is 4 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.LongType
The default size of a value of the LongType is 8 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.MapType
The default size of a value of the MapType is (the default size of the key type + the default size of the value type).
defaultSize() - Method in class org.apache.spark.sql.types.NullType
 
defaultSize() - Static method in class org.apache.spark.sql.types.NumericType
 
defaultSize() - Method in class org.apache.spark.sql.types.ObjectType
 
defaultSize() - Method in class org.apache.spark.sql.types.ShortType
The default size of a value of the ShortType is 2 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.StringType
The default size of a value of the StringType is 20 bytes.
defaultSize() - Method in class org.apache.spark.sql.types.StructType
The default size of a value of the StructType is the total default sizes of all field types.
defaultSize() - Method in class org.apache.spark.sql.types.TimestampType
The default size of a value of the TimestampType is 8 bytes.
defaultSize() - Static method in class org.apache.spark.sql.types.VarcharType
 
defaultStrategy(String) - Static method in class org.apache.spark.mllib.tree.configuration.Strategy
Construct a default set of parameters for DecisionTree
defaultStrategy(Enumeration.Value) - Static method in class org.apache.spark.mllib.tree.configuration.Strategy
Construct a default set of parameters for DecisionTree
DefaultTopologyMapper - Class in org.apache.spark.storage
A TopologyMapper that assumes all nodes are in the same rack
DefaultTopologyMapper(SparkConf) - Constructor for class org.apache.spark.storage.DefaultTopologyMapper
 
defaultValue() - Method in class org.apache.spark.internal.config.ConfigEntryWithDefault
 
defaultValue() - Method in class org.apache.spark.internal.config.ConfigEntryWithDefaultFunction
 
defaultValue() - Method in class org.apache.spark.internal.config.ConfigEntryWithDefaultString
 
defaultValueString() - Method in class org.apache.spark.internal.config.ConfigEntryWithDefault
 
defaultValueString() - Method in class org.apache.spark.internal.config.ConfigEntryWithDefaultFunction
 
defaultValueString() - Method in class org.apache.spark.internal.config.ConfigEntryWithDefaultString
 
degree() - Method in class org.apache.spark.ml.feature.PolynomialExpansion
The polynomial degree to expand, which should be greater than equal to 1.
degrees() - Method in class org.apache.spark.graphx.GraphOps
The degree of each vertex in the graph.
degrees(Column) - Static method in class org.apache.spark.sql.functions
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.
degrees(String) - Static method in class org.apache.spark.sql.functions
Converts an angle measured in radians to an approximately equivalent angle measured in degrees.
degreesOfFreedom() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
Degrees of freedom.
degreesOfFreedom() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Degrees of freedom
degreesOfFreedom() - Method in class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
degreesOfFreedom() - Method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult
 
degreesOfFreedom() - Method in interface org.apache.spark.mllib.stat.test.TestResult
Returns the degree(s) of freedom of the hypothesis test.
delegate() - Method in class org.apache.spark.InterruptibleIterator
 
deleteCheckpointFiles() - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
Developer API
deleteRecursively(File) - Static method in class org.apache.spark.util.Utils
Delete a file or directory and its contents recursively.
deleteWithJob(FileSystem, Path, boolean) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Specifies that a file should be deleted with the commit of this job.
deleteWithJob(FileSystem, Path, boolean) - Static method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
delimiterOptions() - Static method in class org.apache.spark.sql.hive.execution.HiveOptions
 
delta() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Tweedie$
Constant used in initialization and deviance to avoid numerical issues.
dense(int, int, double[]) - Static method in class org.apache.spark.ml.linalg.Matrices
Creates a column-major dense matrix.
dense(double, double...) - Static method in class org.apache.spark.ml.linalg.Vectors
Creates a dense vector from its values.
dense(double, Seq<Object>) - Static method in class org.apache.spark.ml.linalg.Vectors
Creates a dense vector from its values.
dense(double[]) - Static method in class org.apache.spark.ml.linalg.Vectors
Creates a dense vector from a double array.
dense(int, int, double[]) - Static method in class org.apache.spark.mllib.linalg.Matrices
Creates a column-major dense matrix.
dense(double, double...) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a dense vector from its values.
dense(double, Seq<Object>) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a dense vector from its values.
dense(double[]) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a dense vector from a double array.
dense_rank() - Static method in class org.apache.spark.sql.functions
Window function: returns the rank of rows within a window partition, without any gaps.
DenseMatrix - Class in org.apache.spark.ml.linalg
Column-major dense matrix.
DenseMatrix(int, int, double[], boolean) - Constructor for class org.apache.spark.ml.linalg.DenseMatrix
 
DenseMatrix(int, int, double[]) - Constructor for class org.apache.spark.ml.linalg.DenseMatrix
Column-major dense matrix.
DenseMatrix - Class in org.apache.spark.mllib.linalg
Column-major dense matrix.
DenseMatrix(int, int, double[], boolean) - Constructor for class org.apache.spark.mllib.linalg.DenseMatrix
 
DenseMatrix(int, int, double[]) - Constructor for class org.apache.spark.mllib.linalg.DenseMatrix
Column-major dense matrix.
DenseVector - Class in org.apache.spark.ml.linalg
A dense vector represented by a value array.
DenseVector(double[]) - Constructor for class org.apache.spark.ml.linalg.DenseVector
 
DenseVector - Class in org.apache.spark.mllib.linalg
A dense vector represented by a value array.
DenseVector(double[]) - Constructor for class org.apache.spark.mllib.linalg.DenseVector
 
dependencies() - Static method in class org.apache.spark.api.r.RRDD
 
dependencies() - Static method in class org.apache.spark.graphx.EdgeRDD
 
dependencies() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
dependencies() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
dependencies() - Static method in class org.apache.spark.graphx.VertexRDD
 
dependencies() - Static method in class org.apache.spark.rdd.HadoopRDD
 
dependencies() - Static method in class org.apache.spark.rdd.JdbcRDD
 
dependencies() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
dependencies() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
dependencies() - Method in class org.apache.spark.rdd.RDD
Get the list of dependencies of this RDD, taking into account whether the RDD is checkpointed or not.
dependencies() - Static method in class org.apache.spark.rdd.UnionRDD
 
dependencies() - Method in class org.apache.spark.streaming.dstream.DStream
List of parent DStreams on which this DStream depends on
dependencies() - Method in class org.apache.spark.streaming.dstream.InputDStream
 
Dependency<T> - Class in org.apache.spark
Developer API Base class for dependencies.
Dependency() - Constructor for class org.apache.spark.Dependency
 
DEPLOY_MODE - Static variable in class org.apache.spark.launcher.SparkLauncher
The Spark deploy mode.
deployMode() - Method in class org.apache.spark.SparkContext
 
depth() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
depth() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
depth() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Get depth of tree.
depth() - Method in class org.apache.spark.util.sketch.CountMinSketch
Depth of this CountMinSketch.
DerbyDialect - Class in org.apache.spark.sql.jdbc
 
DerbyDialect() - Constructor for class org.apache.spark.sql.jdbc.DerbyDialect
 
deriv(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.CLogLog$
 
deriv(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Identity$
 
deriv(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Inverse$
 
deriv(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Log$
 
deriv(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Logit$
 
deriv(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Probit$
 
deriv(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Sqrt$
 
desc() - Method in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
desc() - Method in class org.apache.spark.sql.Column
Returns a sort expression based on the descending order of the column.
desc(String) - Static method in class org.apache.spark.sql.functions
Returns a sort expression based on the descending order of the column.
desc() - Method in class org.apache.spark.util.MethodIdentifier
 
desc_nulls_first() - Method in class org.apache.spark.sql.Column
Returns a sort expression based on the descending order of the column, and null values appear before non-null values.
desc_nulls_first(String) - Static method in class org.apache.spark.sql.functions
Returns a sort expression based on the descending order of the column, and null values appear before non-null values.
desc_nulls_last() - Method in class org.apache.spark.sql.Column
Returns a sort expression based on the descending order of the column, and null values appear after non-null values.
desc_nulls_last(String) - Static method in class org.apache.spark.sql.functions
Returns a sort expression based on the descending order of the column, and null values appear after non-null values.
describe(String...) - Method in class org.apache.spark.sql.Dataset
Computes basic statistics for numeric and string columns, including count, mean, stddev, min, and max.
describe(Seq<String>) - Method in class org.apache.spark.sql.Dataset
Computes basic statistics for numeric and string columns, including count, mean, stddev, min, and max.
describeTopics(int) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
describeTopics() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
describeTopics(int) - Method in class org.apache.spark.ml.clustering.LDAModel
Return the topics described by their top-weighted terms.
describeTopics() - Method in class org.apache.spark.ml.clustering.LDAModel
 
describeTopics(int) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
describeTopics() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
describeTopics(int) - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
 
describeTopics(int) - Method in class org.apache.spark.mllib.clustering.LDAModel
Return the topics described by weighted terms.
describeTopics() - Method in class org.apache.spark.mllib.clustering.LDAModel
Return the topics described by weighted terms.
describeTopics(int) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
description() - Method in class org.apache.spark.ExceptionFailure
 
description() - Method in class org.apache.spark.sql.catalog.Column
 
description() - Method in class org.apache.spark.sql.catalog.Database
 
description() - Method in class org.apache.spark.sql.catalog.Function
 
description() - Method in class org.apache.spark.sql.catalog.Table
 
description() - Method in class org.apache.spark.sql.streaming.SinkProgress
 
description() - Method in class org.apache.spark.sql.streaming.SourceProgress
 
description() - Method in class org.apache.spark.status.api.v1.JobData
 
description() - Method in class org.apache.spark.status.api.v1.StageData
 
description() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
description() - Method in class org.apache.spark.status.LiveStage
 
description() - Method in class org.apache.spark.storage.StorageLevel
 
description() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
DESER_CPU_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
DESER_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
DeserializationStream - Class in org.apache.spark.serializer
Developer API A stream for reading serialized objects.
DeserializationStream() - Constructor for class org.apache.spark.serializer.DeserializationStream
 
deserialize(Object) - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
deserialize(ByteBuffer, ClassLoader, ClassTag<T>) - Method in class org.apache.spark.serializer.DummySerializerInstance
 
deserialize(ByteBuffer, ClassTag<T>) - Method in class org.apache.spark.serializer.DummySerializerInstance
 
deserialize(ByteBuffer, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializerInstance
 
deserialize(ByteBuffer, ClassLoader, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializerInstance
 
deserialize(byte[]) - Static method in class org.apache.spark.util.Utils
Deserialize an object using Java serialization
deserialize(byte[], ClassLoader) - Static method in class org.apache.spark.util.Utils
Deserialize an object using Java serialization and the given ClassLoader
deserialized() - Method in class org.apache.spark.storage.StorageLevel
 
DeserializedMemoryEntry<T> - Class in org.apache.spark.storage.memory
 
DeserializedMemoryEntry(Object, long, ClassTag<T>) - Constructor for class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
deserializeLongValue(byte[]) - Static method in class org.apache.spark.util.Utils
Deserialize a Long value (used for PythonPartitioner)
deserializeOffset(String) - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.ContinuousReader
Deserialize a JSON string into an Offset of the implementation-defined offset type.
deserializeOffset(String) - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.MicroBatchReader
Deserialize a JSON string into an Offset of the implementation-defined offset type.
deserializeStream(InputStream) - Method in class org.apache.spark.serializer.DummySerializerInstance
 
deserializeStream(InputStream) - Method in class org.apache.spark.serializer.SerializerInstance
 
deserializeViaNestedStream(InputStream, SerializerInstance, Function1<DeserializationStream, BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Deserialize via nested stream using specific serializer
destroy() - Method in class org.apache.spark.broadcast.Broadcast
Destroy all data and metadata related to this broadcast variable.
details() - Method in class org.apache.spark.scheduler.StageInfo
 
details() - Method in class org.apache.spark.status.api.v1.StageData
 
DETERMINATE() - Static method in class org.apache.spark.rdd.DeterministicLevel
 
determineBounds(ArrayBuffer<Tuple2<K, Object>>, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.RangePartitioner
Determines the bounds for range partitioning from candidates with weights indicating how many items each represents.
DetermineTableStats - Class in org.apache.spark.sql.hive
 
DetermineTableStats(SparkSession) - Constructor for class org.apache.spark.sql.hive.DetermineTableStats
 
deterministic() - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Returns true iff this function is deterministic, i.e.
deterministic() - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
Returns true iff the UDF is deterministic, i.e.
DeterministicLevel - Class in org.apache.spark.rdd
The deterministic level of RDD's output (i.e.
DeterministicLevel() - Constructor for class org.apache.spark.rdd.DeterministicLevel
 
deviance(double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
 
deviance(double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
 
deviance(double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
deviance(double, double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
 
deviance() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
The deviance for the fitted model.
devianceResiduals() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
The weighted residuals, the usual residuals rescaled by the square root of the instance weights.
dfToCols(Dataset<Row>) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
dfToRowRDD(Dataset<Row>) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
dgemm(double, DenseMatrix<Object>, DenseMatrix<Object>, double, DenseMatrix<Object>) - Static method in class org.apache.spark.ml.ann.BreezeUtil
DGEMM: C := alpha * A * B + beta * C
dgemv(double, DenseMatrix<Object>, DenseVector<Object>, double, DenseVector<Object>) - Static method in class org.apache.spark.ml.ann.BreezeUtil
DGEMV: y := alpha * A * x + beta * y
diag(Vector) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
Generate a diagonal matrix in DenseMatrix format from the supplied values.
diag(Vector) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a diagonal matrix in Matrix format from the supplied values.
diag(Vector) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
Generate a diagonal matrix in DenseMatrix format from the supplied values.
diag(Vector) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a diagonal matrix in Matrix format from the supplied values.
diff(RDD<Tuple2<Object, VD>>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
diff(VertexRDD<VD>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
diff(RDD<Tuple2<Object, VD>>) - Method in class org.apache.spark.graphx.VertexRDD
For each vertex present in both this and other, diff returns only those vertices with differing values; for values that are different, keeps the values from other.
diff(VertexRDD<VD>) - Method in class org.apache.spark.graphx.VertexRDD
For each vertex present in both this and other, diff returns only those vertices with differing values; for values that are different, keeps the values from other.
diff(GenSeq<B>) - Static method in class org.apache.spark.sql.types.StructType
 
dir() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
directory(File) - Method in class org.apache.spark.launcher.SparkLauncher
Sets the working directory of spark-submit.
disableOutputSpecValidation() - Static method in class org.apache.spark.internal.io.SparkHadoopWriterUtils
Allows for the spark.hadoop.validateOutputSpecs checks to be disabled on a case-by-case basis; see SPARK-4835 for more details.
disconnect() - Method in interface org.apache.spark.launcher.SparkAppHandle
Disconnects the handle from the application, without stopping it.
DISK_BYTES_SPILLED() - Static method in class org.apache.spark.InternalAccumulator
 
DISK_ONLY - Static variable in class org.apache.spark.api.java.StorageLevels
 
DISK_ONLY() - Static method in class org.apache.spark.storage.StorageLevel
 
DISK_ONLY_2 - Static variable in class org.apache.spark.api.java.StorageLevels
 
DISK_ONLY_2() - Static method in class org.apache.spark.storage.StorageLevel
 
DISK_SPILL() - Static method in class org.apache.spark.status.TaskIndexNames
 
DiskBlockData - Class in org.apache.spark.storage
 
DiskBlockData(long, long, File, long) - Constructor for class org.apache.spark.storage.DiskBlockData
 
diskBytesSpilled() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
diskBytesSpilled() - Method in class org.apache.spark.status.api.v1.StageData
 
diskBytesSpilled() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
diskBytesSpilled() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
diskSize() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
diskSize() - Method in class org.apache.spark.storage.BlockStatus
 
diskSize() - Method in class org.apache.spark.storage.BlockUpdatedInfo
 
diskSize() - Method in class org.apache.spark.storage.RDDInfo
 
diskUsed() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
diskUsed() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
diskUsed() - Method in class org.apache.spark.status.api.v1.RDDPartitionInfo
 
diskUsed() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
diskUsed() - Method in class org.apache.spark.status.LiveExecutor
 
diskUsed() - Method in class org.apache.spark.status.LiveRDD
 
diskUsed() - Method in class org.apache.spark.status.LiveRDDDistribution
 
diskUsed() - Method in class org.apache.spark.status.LiveRDDPartition
 
diskUsed() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the disk space used by this block manager.
diskUsedByRdd(int) - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the disk space used by the given RDD in this block manager in O(1) time.
dispersion() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
The dispersion of the fitted model.
dispose() - Method in class org.apache.spark.storage.DiskBlockData
 
dispose() - Method in class org.apache.spark.storage.EncryptedBlockData
 
dispose(ByteBuffer) - Static method in class org.apache.spark.storage.StorageUtils
Attempt to clean up a ByteBuffer if it is direct or memory-mapped.
distinct() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD containing the distinct elements in this RDD.
distinct() - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD containing the distinct elements in this RDD.
distinct() - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
distinct() - Static method in class org.apache.spark.api.r.RRDD
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
distinct() - Static method in class org.apache.spark.graphx.EdgeRDD
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
distinct() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
distinct() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
distinct() - Static method in class org.apache.spark.graphx.VertexRDD
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
distinct() - Static method in class org.apache.spark.rdd.HadoopRDD
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
distinct() - Static method in class org.apache.spark.rdd.JdbcRDD
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
distinct() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
distinct(int, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
distinct() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
distinct(int, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD containing the distinct elements in this RDD.
distinct() - Method in class org.apache.spark.rdd.RDD
Return a new RDD containing the distinct elements in this RDD.
distinct(int, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
distinct() - Static method in class org.apache.spark.rdd.UnionRDD
 
distinct() - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset that contains only the unique rows from this Dataset.
distinct(Column...) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Creates a Column for this UDAF using the distinct values of the given Columns as input arguments.
distinct(Seq<Column>) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Creates a Column for this UDAF using the distinct values of the given Columns as input arguments.
distinct() - Static method in class org.apache.spark.sql.types.StructType
 
distinct$default$2(int) - Static method in class org.apache.spark.api.r.RRDD
 
distinct$default$2(int) - Static method in class org.apache.spark.graphx.EdgeRDD
 
distinct$default$2(int) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
distinct$default$2(int) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
distinct$default$2(int) - Static method in class org.apache.spark.graphx.VertexRDD
 
distinct$default$2(int) - Static method in class org.apache.spark.rdd.HadoopRDD
 
distinct$default$2(int) - Static method in class org.apache.spark.rdd.JdbcRDD
 
distinct$default$2(int) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
distinct$default$2(int) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
distinct$default$2(int) - Static method in class org.apache.spark.rdd.UnionRDD
 
DistributedLDAModel - Class in org.apache.spark.ml.clustering
Distributed model fitted by LDA.
DistributedLDAModel - Class in org.apache.spark.mllib.clustering
Distributed LDA model.
DistributedMatrix - Interface in org.apache.spark.mllib.linalg.distributed
Represents a distributively stored matrix backed by one or more RDDs.
Distribution - Interface in org.apache.spark.sql.sources.v2.reader.partitioning
An interface to represent data distribution requirement, which specifies how the records should be distributed among the data partitions(one DataReader outputs data for one partition).
distribution(LiveExecutor) - Method in class org.apache.spark.status.LiveRDD
 
distributionOpt(LiveExecutor) - Method in class org.apache.spark.status.LiveRDD
 
div(Decimal, Decimal) - Method in class org.apache.spark.sql.types.Decimal.DecimalIsFractional$
 
div(Duration) - Method in class org.apache.spark.streaming.Duration
 
divide(Object) - Method in class org.apache.spark.sql.Column
Division this expression by another expression.
doc() - Static method in class org.apache.spark.ml.param.DoubleParam
 
doc() - Static method in class org.apache.spark.ml.param.FloatParam
 
doc() - Method in class org.apache.spark.ml.param.Param
 
docConcentration() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
docConcentration() - Static method in class org.apache.spark.ml.clustering.LDA
 
docConcentration() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
docConcentration() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
 
docConcentration() - Method in class org.apache.spark.mllib.clustering.LDAModel
Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
docConcentration() - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
DocumentFrequencyAggregator(int) - Constructor for class org.apache.spark.mllib.feature.IDF.DocumentFrequencyAggregator
 
DocumentFrequencyAggregator() - Constructor for class org.apache.spark.mllib.feature.IDF.DocumentFrequencyAggregator
 
doesDirectoryContainAnyNewFiles(File, long) - Static method in class org.apache.spark.util.Utils
Determines if a directory contains any files newer than cutoff seconds.
doFetchFile(String, File, String, SparkConf, org.apache.spark.SecurityManager, Configuration) - Static method in class org.apache.spark.util.Utils
Download a file or directory to target directory.
Dot - Class in org.apache.spark.ml.feature
 
Dot() - Constructor for class org.apache.spark.ml.feature.Dot
 
dot(Vector, Vector) - Static method in class org.apache.spark.ml.linalg.BLAS
dot(x, y)
dot(Vector, Vector) - Static method in class org.apache.spark.mllib.linalg.BLAS
dot(x, y)
doTest(DStream<Tuple2<StatCounter, StatCounter>>) - Static method in class org.apache.spark.mllib.stat.test.StudentTTest
 
doTest(DStream<Tuple2<StatCounter, StatCounter>>) - Static method in class org.apache.spark.mllib.stat.test.WelchTTest
 
DOUBLE() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable double type.
doubleAccumulator(double) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().doubleAccumulator(). Since 2.0.0.
doubleAccumulator(double, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().doubleAccumulator(String). Since 2.0.0.
doubleAccumulator() - Method in class org.apache.spark.SparkContext
Create and register a double accumulator, which starts with 0 and accumulates inputs by add.
doubleAccumulator(String) - Method in class org.apache.spark.SparkContext
Create and register a double accumulator, which starts with 0 and accumulates inputs by add.
DoubleAccumulator - Class in org.apache.spark.util
An accumulator for computing sum, count, and averages for double precision floating numbers.
DoubleAccumulator() - Constructor for class org.apache.spark.util.DoubleAccumulator
 
DoubleAccumulatorParam$() - Constructor for class org.apache.spark.AccumulatorParam.DoubleAccumulatorParam$
Deprecated.
 
DoubleArrayArrayParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Array[Array[Double}] for Java.
DoubleArrayArrayParam(Params, String, String, Function1<double[][], Object>) - Constructor for class org.apache.spark.ml.param.DoubleArrayArrayParam
 
DoubleArrayArrayParam(Params, String, String) - Constructor for class org.apache.spark.ml.param.DoubleArrayArrayParam
 
DoubleArrayParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Array[Double} for Java.
DoubleArrayParam(Params, String, String, Function1<double[], Object>) - Constructor for class org.apache.spark.ml.param.DoubleArrayParam
 
DoubleArrayParam(Params, String, String) - Constructor for class org.apache.spark.ml.param.DoubleArrayParam
 
DoubleFlatMapFunction<T> - Interface in org.apache.spark.api.java.function
A function that returns zero or more records of type Double from each input record.
DoubleFunction<T> - Interface in org.apache.spark.api.java.function
A function that returns Doubles, and can be used to construct DoubleRDDs.
DoubleParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Double] for Java.
DoubleParam(String, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.DoubleParam
 
DoubleParam(String, String, String) - Constructor for class org.apache.spark.ml.param.DoubleParam
 
DoubleParam(Identifiable, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.DoubleParam
 
DoubleParam(Identifiable, String, String) - Constructor for class org.apache.spark.ml.param.DoubleParam
 
DoubleRDDFunctions - Class in org.apache.spark.rdd
Extra functions available on RDDs of Doubles through an implicit conversion.
DoubleRDDFunctions(RDD<Object>) - Constructor for class org.apache.spark.rdd.DoubleRDDFunctions
 
doubleRDDToDoubleRDDFunctions(RDD<Object>) - Static method in class org.apache.spark.rdd.RDD
 
DoubleType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the DoubleType object.
DoubleType - Class in org.apache.spark.sql.types
The data type representing Double values.
driver() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SetupDriver
 
DRIVER_EXTRA_CLASSPATH - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the driver class path.
DRIVER_EXTRA_JAVA_OPTIONS - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the driver VM options.
DRIVER_EXTRA_LIBRARY_PATH - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the driver native library path.
DRIVER_MEMORY - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the driver memory.
DRIVER_WAL_BATCHING_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
DRIVER_WAL_BATCHING_TIMEOUT_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
DRIVER_WAL_CLASS_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
DRIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
DRIVER_WAL_MAX_FAILURES_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
DRIVER_WAL_ROLLING_INTERVAL_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
driverLogs() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
drop() - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that drops rows containing any null or NaN values.
drop(String) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that drops rows containing null or NaN values.
drop(String[]) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that drops rows containing any null or NaN values in the specified columns.
drop(Seq<String>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that drops rows containing any null or NaN values in the specified columns.
drop(String, String[]) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that drops rows containing null or NaN values in the specified columns.
drop(String, Seq<String>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that drops rows containing null or NaN values in the specified columns.
drop(int) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values.
drop(int, String[]) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values in the specified columns.
drop(int, Seq<String>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that drops rows containing less than minNonNulls non-null and non-NaN values in the specified columns.
drop(String...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with columns dropped.
drop(String) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with a column dropped.
drop(Seq<String>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with columns dropped.
drop(Column) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with a column dropped.
drop(int) - Static method in class org.apache.spark.sql.types.StructType
 
dropDuplicates(String, String...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with duplicate rows removed, considering only the subset of columns.
dropDuplicates() - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset that contains only the unique rows from this Dataset.
dropDuplicates(Seq<String>) - Method in class org.apache.spark.sql.Dataset
(Scala-specific) Returns a new Dataset with duplicate rows removed, considering only the subset of columns.
dropDuplicates(String[]) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with duplicate rows removed, considering only the subset of columns.
dropDuplicates(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with duplicate rows removed, considering only the subset of columns.
dropGlobalTempView(String) - Method in class org.apache.spark.sql.catalog.Catalog
Drops the global temporary view with the given view name in the catalog.
dropLast() - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
Whether to drop the last category in the encoded vector (default: true)
dropLast() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
dropLast() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
dropRight(int) - Static method in class org.apache.spark.sql.types.StructType
 
dropTempTable(String) - Method in class org.apache.spark.sql.SQLContext
 
dropTempView(String) - Method in class org.apache.spark.sql.catalog.Catalog
Drops the local temporary view with the given view name in the catalog.
dropWhile(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
dspmv(int, double, DenseVector, DenseVector, double, DenseVector) - Static method in class org.apache.spark.ml.linalg.BLAS
y := alpha*A*x + beta*y
Dst - Static variable in class org.apache.spark.graphx.TripletFields
Expose the destination and edge fields but not the source field.
dstAttr() - Method in class org.apache.spark.graphx.EdgeContext
The vertex attribute of the edge's destination vertex.
dstAttr() - Method in class org.apache.spark.graphx.EdgeTriplet
The destination vertex attribute
dstAttr() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
dstId() - Method in class org.apache.spark.graphx.Edge
 
dstId() - Method in class org.apache.spark.graphx.EdgeContext
The vertex id of the edge's destination vertex.
dstId() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
dstream() - Method in class org.apache.spark.streaming.api.java.JavaDStream
 
dstream() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
 
dstream() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
dstream() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
dstream() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
dstream() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
dstream() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
DStream<T> - Class in org.apache.spark.streaming.dstream
A Discretized Stream (DStream), the basic abstraction in Spark Streaming, is a continuous sequence of RDDs (of the same type) representing a continuous stream of data (see org.apache.spark.rdd.RDD in the Spark core documentation for more details on RDDs).
DStream(StreamingContext, ClassTag<T>) - Constructor for class org.apache.spark.streaming.dstream.DStream
 
dtypes() - Method in class org.apache.spark.sql.Dataset
Returns all column names and their data types as an array.
DummySerializerInstance - Class in org.apache.spark.serializer
Unfortunately, we need a serializer instance in order to construct a DiskBlockObjectWriter.
duration() - Method in class org.apache.spark.scheduler.TaskInfo
 
duration() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
duration() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
duration() - Method in class org.apache.spark.status.api.v1.TaskData
 
DURATION() - Static method in class org.apache.spark.status.TaskIndexNames
 
Duration - Class in org.apache.spark.streaming
 
Duration(long) - Constructor for class org.apache.spark.streaming.Duration
 
duration() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
Return the duration of this output operation.
durationMs() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
Durations - Class in org.apache.spark.streaming
 
Durations() - Constructor for class org.apache.spark.streaming.Durations
 

E

Edge<ED> - Class in org.apache.spark.graphx
A single directed edge consisting of a source id, target id, and the data associated with the edge.
Edge(long, long, ED) - Constructor for class org.apache.spark.graphx.Edge
 
EdgeActiveness - Enum in org.apache.spark.graphx.impl
Criteria for filtering edges based on activeness.
EdgeContext<VD,ED,A> - Class in org.apache.spark.graphx
Represents an edge along with its neighboring vertices and allows sending messages along the edge.
EdgeContext() - Constructor for class org.apache.spark.graphx.EdgeContext
 
EdgeDirection - Class in org.apache.spark.graphx
The direction of a directed edge relative to a vertex.
edgeListFile(SparkContext, String, boolean, int, StorageLevel, StorageLevel) - Static method in class org.apache.spark.graphx.GraphLoader
Loads a graph from an edge list formatted file where each line contains two integers: a source id and a target id.
EdgeOnly - Static variable in class org.apache.spark.graphx.TripletFields
Expose only the edge field and not the source or destination field.
EdgePartition1D$() - Constructor for class org.apache.spark.graphx.PartitionStrategy.EdgePartition1D$
 
EdgePartition2D$() - Constructor for class org.apache.spark.graphx.PartitionStrategy.EdgePartition2D$
 
EdgeRDD<ED> - Class in org.apache.spark.graphx
EdgeRDD[ED, VD] extends RDD[Edge[ED} by storing the edges in columnar format on each partition for performance.
EdgeRDD(SparkContext, Seq<Dependency<?>>) - Constructor for class org.apache.spark.graphx.EdgeRDD
 
EdgeRDDImpl<ED,VD> - Class in org.apache.spark.graphx.impl
 
edges() - Method in class org.apache.spark.graphx.Graph
An RDD containing the edges and their associated attributes.
edges() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
EdgeTriplet<VD,ED> - Class in org.apache.spark.graphx
An edge triplet represents an edge along with the vertex attributes of its neighboring vertices.
EdgeTriplet() - Constructor for class org.apache.spark.graphx.EdgeTriplet
 
EigenValueDecomposition - Class in org.apache.spark.mllib.linalg
Compute eigen-decomposition.
EigenValueDecomposition() - Constructor for class org.apache.spark.mllib.linalg.EigenValueDecomposition
 
Either() - Static method in class org.apache.spark.graphx.EdgeDirection
Edges originating from *or* arriving at a vertex of interest.
elasticNetParam() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
elasticNetParam() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
elasticNetParam() - Method in interface org.apache.spark.ml.param.shared.HasElasticNetParam
Param for the ElasticNet mixing parameter, in range [0, 1].
elasticNetParam() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
elasticNetParam() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
elem(String, Function1<Object, Object>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
elem(Parsers) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
elementType() - Method in class org.apache.spark.sql.types.ArrayType
 
ElementwiseProduct - Class in org.apache.spark.ml.feature
Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.
ElementwiseProduct(String) - Constructor for class org.apache.spark.ml.feature.ElementwiseProduct
 
ElementwiseProduct() - Constructor for class org.apache.spark.ml.feature.ElementwiseProduct
 
ElementwiseProduct - Class in org.apache.spark.mllib.feature
Outputs the Hadamard product (i.e., the element-wise product) of each input vector with a provided "weight" vector.
ElementwiseProduct(Vector) - Constructor for class org.apache.spark.mllib.feature.ElementwiseProduct
 
EMLDAOptimizer - Class in org.apache.spark.mllib.clustering
Developer API
EMLDAOptimizer() - Constructor for class org.apache.spark.mllib.clustering.EMLDAOptimizer
 
empty() - Static method in class org.apache.spark.api.java.Optional
 
empty() - Static method in class org.apache.spark.ml.param.ParamMap
Returns an empty param map.
empty() - Method in class org.apache.spark.mllib.fpm.PrefixSpan.Prefix$
An empty Prefix instance.
empty() - Static method in class org.apache.spark.sql.sources.v2.DataSourceOptions
 
empty() - Static method in class org.apache.spark.sql.types.Metadata
Returns an empty Metadata.
empty() - Static method in class org.apache.spark.storage.BlockStatus
 
EMPTY_USER_GROUPS() - Static method in class org.apache.spark.util.Utils
 
emptyDataFrame() - Method in class org.apache.spark.sql.SparkSession
Returns a DataFrame with no rows or columns.
emptyDataFrame() - Method in class org.apache.spark.sql.SQLContext
Returns a DataFrame with no rows or columns.
emptyDataset(Encoder<T>) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a new Dataset of type T containing zero elements.
emptyJson() - Static method in class org.apache.spark.util.Utils
Return an empty JSON object
emptyNode(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Return a node with the given node id (but nothing else set).
emptyRDD() - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD that has no partitions or elements.
emptyRDD(ClassTag<T>) - Method in class org.apache.spark.SparkContext
Get an RDD that has no partitions or elements.
EmptyTaskCommitMessage$() - Constructor for class org.apache.spark.internal.io.FileCommitProtocol.EmptyTaskCommitMessage$
 
enableBatchRead() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsScanColumnarBatch
Returns true if the concrete data source reader can read data in batch according to the scan properties like required columns, pushes filters, etc.
enableHiveSupport() - Method in class org.apache.spark.sql.SparkSession.Builder
Enables Hive support, including connectivity to a persistent Hive metastore, support for Hive serdes, and Hive user-defined functions.
enableReceiverLog(SparkConf) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
encode(Column, String) - Static method in class org.apache.spark.sql.functions
Computes the first argument into a binary from a string using the provided character set (one of 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16').
encodeFileNameToURIRawPath(String) - Static method in class org.apache.spark.util.Utils
A file name may contain some invalid URI characters, such as " ".
encodeLabeledPoint(LabeledPoint, int) - Static method in class org.apache.spark.ml.classification.LabelConverter
Encodes a label as a vector.
Encoder<T> - Interface in org.apache.spark.sql
Experimental Used to convert a JVM object of type T to and from the internal Spark SQL representation.
Encoders - Class in org.apache.spark.sql
Experimental Methods for creating an Encoder.
Encoders() - Constructor for class org.apache.spark.sql.Encoders
 
EncryptedBlockData - Class in org.apache.spark.storage
 
EncryptedBlockData(File, long, SparkConf, byte[]) - Constructor for class org.apache.spark.storage.EncryptedBlockData
 
EncryptedManagedBuffer - Class in org.apache.spark.storage
 
EncryptedManagedBuffer(EncryptedBlockData) - Constructor for class org.apache.spark.storage.EncryptedManagedBuffer
 
endOffset() - Method in class org.apache.spark.sql.streaming.SourceProgress
 
endOffset() - Method in exception org.apache.spark.sql.streaming.StreamingQueryException
 
endsWith(Column) - Method in class org.apache.spark.sql.Column
String ends with.
endsWith(String) - Method in class org.apache.spark.sql.Column
String ends with another string literal.
endsWith(GenSeq<B>) - Static method in class org.apache.spark.sql.types.StructType
 
endTime() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
endTime() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
endTime() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
EnsembleCombiningStrategy - Class in org.apache.spark.mllib.tree.configuration
Enum to select ensemble combining strategy for base learners
EnsembleCombiningStrategy() - Constructor for class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
EnsembleModelReadWrite - Class in org.apache.spark.ml.tree
 
EnsembleModelReadWrite() - Constructor for class org.apache.spark.ml.tree.EnsembleModelReadWrite
 
EnsembleModelReadWrite.EnsembleNodeData - Class in org.apache.spark.ml.tree
Info for one Node in a tree ensemble
EnsembleModelReadWrite.EnsembleNodeData$ - Class in org.apache.spark.ml.tree
 
EnsembleNodeData(int, DecisionTreeModelReadWrite.NodeData) - Constructor for class org.apache.spark.ml.tree.EnsembleModelReadWrite.EnsembleNodeData
 
EnsembleNodeData$() - Constructor for class org.apache.spark.ml.tree.EnsembleModelReadWrite.EnsembleNodeData$
 
entries() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
 
Entropy - Class in org.apache.spark.mllib.tree.impurity
Class for calculating entropy during multiclass classification.
Entropy() - Constructor for class org.apache.spark.mllib.tree.impurity.Entropy
 
entrySet() - Method in class org.apache.spark.api.java.JavaUtils.SerializableMapWrapper
 
EnumUtil - Class in org.apache.spark.util
 
EnumUtil() - Constructor for class org.apache.spark.util.EnumUtil
 
environmentDetails() - Method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
environmentUpdateFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
environmentUpdateToJson(SparkListenerEnvironmentUpdate) - Static method in class org.apache.spark.util.JsonProtocol
 
EPSILON() - Static method in class org.apache.spark.ml.impl.Utils
 
epsilon() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
epsilon() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
eqNullSafe(Object) - Method in class org.apache.spark.sql.Column
Equality test that is safe for null values.
EqualNullSafe - Class in org.apache.spark.sql.sources
Performs equality comparison, similar to EqualTo.
EqualNullSafe(String, Object) - Constructor for class org.apache.spark.sql.sources.EqualNullSafe
 
equals(Object) - Static method in class org.apache.spark.Aggregator
 
equals(Object) - Method in class org.apache.spark.api.java.Optional
 
equals(Object) - Static method in class org.apache.spark.CleanAccum
 
equals(Object) - Static method in class org.apache.spark.CleanBroadcast
 
equals(Object) - Static method in class org.apache.spark.CleanCheckpoint
 
equals(Object) - Static method in class org.apache.spark.CleanRDD
 
equals(Object) - Static method in class org.apache.spark.CleanShuffle
 
equals(Object) - Static method in class org.apache.spark.ExceptionFailure
 
equals(Object) - Static method in class org.apache.spark.ExecutorLostFailure
 
equals(Object) - Static method in class org.apache.spark.ExecutorRegistered
 
equals(Object) - Static method in class org.apache.spark.ExecutorRemoved
 
equals(Object) - Static method in class org.apache.spark.ExpireDeadHosts
 
equals(Object) - Static method in class org.apache.spark.FetchFailed
 
equals(Object) - Static method in class org.apache.spark.graphx.Edge
 
equals(Object) - Method in class org.apache.spark.graphx.EdgeDirection
 
equals(Object) - Method in class org.apache.spark.HashPartitioner
 
equals(Object) - Method in class org.apache.spark.ml.attribute.AttributeGroup
 
equals(Object) - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
equals(Object) - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
equals(Object) - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
equals(Object) - Static method in class org.apache.spark.ml.feature.Dot
 
equals(Object) - Static method in class org.apache.spark.ml.feature.LabeledPoint
 
equals(Object) - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
equals(Object) - Method in class org.apache.spark.ml.linalg.DenseVector
 
equals(Object) - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
equals(Object) - Method in class org.apache.spark.ml.linalg.SparseVector
 
equals(Object) - Method in interface org.apache.spark.ml.linalg.Vector
 
equals(Object) - Static method in class org.apache.spark.ml.param.DoubleParam
 
equals(Object) - Static method in class org.apache.spark.ml.param.FloatParam
 
equals(Object) - Method in class org.apache.spark.ml.param.Param
 
equals(Object) - Static method in class org.apache.spark.ml.param.ParamPair
 
equals(Object) - Method in class org.apache.spark.ml.tree.CategoricalSplit
 
equals(Object) - Method in class org.apache.spark.ml.tree.ContinuousSplit
 
equals(Object) - Static method in class org.apache.spark.mllib.feature.VocabWord
 
equals(Object) - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
equals(Object) - Method in class org.apache.spark.mllib.linalg.DenseVector
 
equals(Object) - Static method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
equals(Object) - Static method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
equals(Object) - Static method in class org.apache.spark.mllib.linalg.QRDecomposition
 
equals(Object) - Static method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
equals(Object) - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
equals(Object) - Method in class org.apache.spark.mllib.linalg.SparseVector
 
equals(Object) - Method in interface org.apache.spark.mllib.linalg.Vector
 
equals(Object) - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
equals(Object) - Static method in class org.apache.spark.mllib.recommendation.Rating
 
equals(Object) - Static method in class org.apache.spark.mllib.regression.LabeledPoint
 
equals(Object) - Static method in class org.apache.spark.mllib.stat.test.BinarySample
 
equals(Object) - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
equals(Object) - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
equals(Object) - Method in class org.apache.spark.mllib.tree.model.Predict
 
equals(Object) - Static method in class org.apache.spark.mllib.tree.model.Split
 
equals(Object) - Method in class org.apache.spark.partial.BoundedDouble
 
equals(Object) - Method in interface org.apache.spark.Partition
 
equals(Object) - Method in class org.apache.spark.RangePartitioner
 
equals(Object) - Static method in class org.apache.spark.Resubmitted
 
equals(Object) - Static method in class org.apache.spark.rpc.netty.OnStart
 
equals(Object) - Static method in class org.apache.spark.rpc.netty.OnStop
 
equals(Object) - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
equals(Object) - Static method in class org.apache.spark.scheduler.AllJobsCancelled
 
equals(Object) - Static method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
equals(Object) - Static method in class org.apache.spark.scheduler.BlacklistedExecutor
 
equals(Object) - Method in class org.apache.spark.scheduler.cluster.ExecutorInfo
 
equals(Object) - Method in class org.apache.spark.scheduler.InputFormatInfo
 
equals(Object) - Static method in class org.apache.spark.scheduler.JobSucceeded
 
equals(Object) - Static method in class org.apache.spark.scheduler.local.KillTask
 
equals(Object) - Static method in class org.apache.spark.scheduler.local.ReviveOffers
 
equals(Object) - Static method in class org.apache.spark.scheduler.local.StatusUpdate
 
equals(Object) - Static method in class org.apache.spark.scheduler.local.StopExecutor
 
equals(Object) - Static method in class org.apache.spark.scheduler.ResubmitFailedStages
 
equals(Object) - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerJobStart
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerLogStart
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerStageCompleted
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
equals(Object) - Static method in class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
equals(Object) - Method in class org.apache.spark.scheduler.SplitInfo
 
equals(Object) - Static method in class org.apache.spark.scheduler.StopCoordinator
 
equals(Object) - Method in class org.apache.spark.sql.Column
 
equals(Object) - Static method in class org.apache.spark.sql.DatasetHolder
 
equals(Object) - Static method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
equals(Object) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
equals(Object) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
equals(Object) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
equals(Object) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
equals(Object) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
equals(Object) - Static method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
equals(Object) - Static method in class org.apache.spark.sql.hive.RelationConversions
 
equals(Object) - Static method in class org.apache.spark.sql.jdbc.JdbcType
 
equals(Object) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
equals(Object) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
equals(Object) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
equals(Object) - Method in interface org.apache.spark.sql.Row
 
equals(Object) - Static method in class org.apache.spark.sql.sources.And
 
equals(Object) - Static method in class org.apache.spark.sql.sources.EqualNullSafe
 
equals(Object) - Static method in class org.apache.spark.sql.sources.EqualTo
 
equals(Object) - Static method in class org.apache.spark.sql.sources.GreaterThan
 
equals(Object) - Static method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
equals(Object) - Method in class org.apache.spark.sql.sources.In
 
equals(Object) - Static method in class org.apache.spark.sql.sources.IsNotNull
 
equals(Object) - Static method in class org.apache.spark.sql.sources.IsNull
 
equals(Object) - Static method in class org.apache.spark.sql.sources.LessThan
 
equals(Object) - Static method in class org.apache.spark.sql.sources.LessThanOrEqual
 
equals(Object) - Static method in class org.apache.spark.sql.sources.Not
 
equals(Object) - Static method in class org.apache.spark.sql.sources.Or
 
equals(Object) - Static method in class org.apache.spark.sql.sources.StringContains
 
equals(Object) - Static method in class org.apache.spark.sql.sources.StringEndsWith
 
equals(Object) - Static method in class org.apache.spark.sql.sources.StringStartsWith
 
equals(Object) - Method in class org.apache.spark.sql.sources.v2.reader.streaming.Offset
Equality based on JSON string representation.
equals(Object) - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
equals(Object) - Static method in class org.apache.spark.sql.types.ArrayType
 
equals(Object) - Static method in class org.apache.spark.sql.types.CharType
 
equals(Object) - Method in class org.apache.spark.sql.types.Decimal
 
equals(Object) - Static method in class org.apache.spark.sql.types.DecimalType
 
equals(Object) - Static method in class org.apache.spark.sql.types.MapType
 
equals(Object) - Method in class org.apache.spark.sql.types.Metadata
 
equals(Object) - Static method in class org.apache.spark.sql.types.ObjectType
 
equals(Object) - Static method in class org.apache.spark.sql.types.StructField
 
equals(Object) - Method in class org.apache.spark.sql.types.StructType
 
equals(Object) - Static method in class org.apache.spark.sql.types.VarcharType
 
equals(Object) - Static method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
equals(Object) - Static method in class org.apache.spark.status.api.v1.ApplicationInfo
 
equals(Object) - Static method in class org.apache.spark.StopMapOutputTracker
 
equals(Object) - Method in class org.apache.spark.storage.BlockManagerId
 
equals(Object) - Static method in class org.apache.spark.storage.BlockStatus
 
equals(Object) - Static method in class org.apache.spark.storage.BlockUpdatedInfo
 
equals(Object) - Static method in class org.apache.spark.storage.BroadcastBlockId
 
equals(Object) - Static method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
equals(Object) - Static method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
equals(Object) - Static method in class org.apache.spark.storage.RDDBlockId
 
equals(Object) - Static method in class org.apache.spark.storage.ShuffleBlockId
 
equals(Object) - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
equals(Object) - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
equals(Object) - Method in class org.apache.spark.storage.StorageLevel
 
equals(Object) - Static method in class org.apache.spark.storage.StreamBlockId
 
equals(Object) - Static method in class org.apache.spark.storage.TaskResultBlockId
 
equals(Object) - Static method in class org.apache.spark.streaming.Duration
 
equals(Object) - Static method in class org.apache.spark.streaming.kinesis.DefaultCredentials
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.AllReceiverIds
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.BatchInfo
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.GetAllReceiverInfo
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StopAllReceivers
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
equals(Object) - Static method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
equals(Object) - Static method in class org.apache.spark.streaming.Time
 
equals(Object) - Static method in class org.apache.spark.Success
 
equals(Object) - Static method in class org.apache.spark.TaskCommitDenied
 
equals(Object) - Static method in class org.apache.spark.TaskKilled
 
equals(Object) - Static method in class org.apache.spark.TaskResultLost
 
equals(Object) - Static method in class org.apache.spark.TaskSchedulerIsSet
 
equals(Object) - Static method in class org.apache.spark.UnknownReason
 
equals(Object) - Static method in class org.apache.spark.util.MethodIdentifier
 
equals(Object) - Static method in class org.apache.spark.util.MutablePair
 
equalsStructurally(DataType, DataType, boolean) - Static method in class org.apache.spark.sql.types.DataType
Returns true if the two data types share the same "shape", i.e.
equalTo(Object) - Method in class org.apache.spark.sql.Column
Equality test.
EqualTo - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a value equal to value.
EqualTo(String, Object) - Constructor for class org.apache.spark.sql.sources.EqualTo
 
err(String) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
ERROR() - Static method in class org.apache.spark.status.TaskIndexNames
 
errorMessage() - Method in class org.apache.spark.status.api.v1.TaskData
 
errorMessage() - Method in class org.apache.spark.status.LiveTask
 
estimate(double[]) - Method in class org.apache.spark.mllib.stat.KernelDensity
Estimates probability density function at the given array of points.
estimate(Object) - Static method in class org.apache.spark.util.SizeEstimator
Estimate the number of bytes that the given object takes up on the JVM heap.
estimateCount(Object) - Method in class org.apache.spark.util.sketch.CountMinSketch
Returns the estimated frequency of item.
estimatedDocConcentration() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
estimatedDocConcentration() - Method in class org.apache.spark.ml.clustering.LDAModel
Value for docConcentration estimated from data.
estimatedDocConcentration() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
Estimator<M extends Model<M>> - Class in org.apache.spark.ml
Developer API Abstract class for estimators that fit models to data.
Estimator() - Constructor for class org.apache.spark.ml.Estimator
 
estimator() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
estimator() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
estimator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
estimator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
estimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
estimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
estimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
estimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
Evaluates the model on a test dataset.
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
evaluate(Dataset<?>, ParamMap) - Method in class org.apache.spark.ml.evaluation.Evaluator
Evaluates model output and returns a scalar metric.
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.evaluation.Evaluator
Evaluates model output and returns a scalar metric.
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
Evaluate the model on the given dataset, returning a summary of the results.
evaluate(Dataset<?>) - Method in class org.apache.spark.ml.regression.LinearRegressionModel
Evaluates the model on a test dataset.
evaluate(Row) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Calculates the final result of this UserDefinedAggregateFunction based on the given aggregation buffer.
evaluateEachIteration(RDD<LabeledPoint>, DecisionTreeRegressionModel[], double[], Loss, Enumeration.Value) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Method to compute error or loss for every iteration of gradient boosting.
evaluateEachIteration(RDD<LabeledPoint>, Loss) - Method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
Method to compute error or loss for every iteration of gradient boosting.
Evaluator - Class in org.apache.spark.ml.evaluation
Developer API Abstract class for evaluators that compute metrics from predictions.
Evaluator() - Constructor for class org.apache.spark.ml.evaluation.Evaluator
 
evaluator() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
evaluator() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
evaluator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
evaluator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
eventRates() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
eventTime() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
EventTimeTimeout() - Static method in class org.apache.spark.sql.streaming.GroupStateTimeout
Timeout based on event-time.
except(Dataset<T>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset containing rows in this Dataset but not in another Dataset.
exception() - Method in class org.apache.spark.ExceptionFailure
 
exception() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread
Contains the exception thrown while writing the parent iterator to the external process.
exception() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns the StreamingQueryException if the query was terminated by an exception.
exception() - Method in class org.apache.spark.sql.streaming.StreamingQueryListener.QueryTerminatedEvent
 
ExceptionFailure - Class in org.apache.spark
Developer API Task failed due to a runtime exception.
ExceptionFailure(String, String, StackTraceElement[], String, Option<ThrowableSerializationWrapper>, Seq<AccumulableInfo>, Seq<AccumulatorV2<?, ?>>) - Constructor for class org.apache.spark.ExceptionFailure
 
exceptionFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
exceptionString(Throwable) - Static method in class org.apache.spark.util.Utils
Return a nice string representation of the exception.
exceptionToJson(Exception) - Static method in class org.apache.spark.util.JsonProtocol
 
EXEC_CPU_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
EXEC_RUN_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
execId() - Method in class org.apache.spark.ExecutorLostFailure
 
execId() - Method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
execId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveExecutor
 
execute() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
executeAndGetOutput(Seq<String>, File, Map<String, String>, boolean) - Static method in class org.apache.spark.util.Utils
Execute a command and get its output, throwing an exception if it yields a code other than 0.
executeBroadcast() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
executeCollect() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
executeCollectPublic() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
executeCommand(Seq<String>, File, Map<String, String>, boolean) - Static method in class org.apache.spark.util.Utils
Execute a command and return the process running the command.
executeTake(int) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
executeToIterator() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
ExecutionListenerManager - Class in org.apache.spark.sql.util
Experimental
executor() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask
 
EXECUTOR() - Static method in class org.apache.spark.status.TaskIndexNames
 
EXECUTOR_CORES - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the number of executor CPU cores.
EXECUTOR_CPU_TIME() - Static method in class org.apache.spark.InternalAccumulator
 
EXECUTOR_DESERIALIZE_CPU_TIME() - Static method in class org.apache.spark.InternalAccumulator
 
EXECUTOR_DESERIALIZE_TIME() - Static method in class org.apache.spark.InternalAccumulator
 
EXECUTOR_EXTRA_CLASSPATH - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the executor class path.
EXECUTOR_EXTRA_JAVA_OPTIONS - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the executor VM options.
EXECUTOR_EXTRA_LIBRARY_PATH - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the executor native library path.
EXECUTOR_MEMORY - Static variable in class org.apache.spark.launcher.SparkLauncher
Configuration key for the executor memory.
EXECUTOR_RUN_TIME() - Static method in class org.apache.spark.InternalAccumulator
 
executorAddedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
executorAddedToJson(SparkListenerExecutorAdded) - Static method in class org.apache.spark.util.JsonProtocol
 
executorCpuTime() - Method in class org.apache.spark.status.api.v1.StageData
 
executorCpuTime() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
executorCpuTime() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
executorDeserializeCpuTime() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
executorDeserializeCpuTime() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
executorDeserializeTime() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
executorDeserializeTime() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
executorFailures() - Method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
executorHost() - Method in class org.apache.spark.scheduler.cluster.ExecutorInfo
 
executorHost() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
executorId() - Method in class org.apache.spark.ExecutorRegistered
 
executorId() - Method in class org.apache.spark.ExecutorRemoved
 
executorId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.GetExecutorLossReason
 
executorId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
executorId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
 
executorId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
executorId() - Method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
executorId() - Method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
executorId() - Method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
executorId() - Method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
executorId() - Method in class org.apache.spark.scheduler.TaskInfo
 
executorId() - Method in class org.apache.spark.SparkEnv
 
executorId() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
executorId() - Method in class org.apache.spark.status.api.v1.TaskData
 
executorId() - Method in class org.apache.spark.status.LiveExecutor
 
executorId() - Method in class org.apache.spark.status.LiveRDDDistribution
 
executorId() - Method in class org.apache.spark.storage.BlockManagerId
 
executorId() - Method in class org.apache.spark.storage.BlockManagerMessages.GetExecutorEndpointRef
 
executorId() - Method in class org.apache.spark.storage.BlockManagerMessages.HasCachedBlocks
 
executorId() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
executorId() - Method in class org.apache.spark.ui.storage.ExecutorStreamSummary
 
executorIds() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutors
 
ExecutorInfo - Class in org.apache.spark.scheduler.cluster
Developer API Stores information about an executor to pass from the scheduler to SparkListeners.
ExecutorInfo(String, int, Map<String, String>) - Constructor for class org.apache.spark.scheduler.cluster.ExecutorInfo
 
executorInfo() - Method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
executorInfoFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
executorInfoToJson(ExecutorInfo) - Static method in class org.apache.spark.util.JsonProtocol
 
ExecutorKilled - Class in org.apache.spark.scheduler
 
ExecutorKilled() - Constructor for class org.apache.spark.scheduler.ExecutorKilled
 
executorLogs() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
executorLogs() - Method in class org.apache.spark.status.LiveExecutor
 
ExecutorLostFailure - Class in org.apache.spark
Developer API The task failed because the executor that it was running on was lost.
ExecutorLostFailure(String, boolean, Option<String>) - Constructor for class org.apache.spark.ExecutorLostFailure
 
executorMetricsUpdateFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
executorMetricsUpdateToJson(SparkListenerExecutorMetricsUpdate) - Static method in class org.apache.spark.util.JsonProtocol
 
executorPct() - Method in class org.apache.spark.scheduler.RuntimePercentage
 
executorRef() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
ExecutorRegistered - Class in org.apache.spark
 
ExecutorRegistered(String) - Constructor for class org.apache.spark.ExecutorRegistered
 
ExecutorRemoved - Class in org.apache.spark
 
ExecutorRemoved(String) - Constructor for class org.apache.spark.ExecutorRemoved
 
executorRemovedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
executorRemovedToJson(SparkListenerExecutorRemoved) - Static method in class org.apache.spark.util.JsonProtocol
 
executorRunTime() - Method in class org.apache.spark.status.api.v1.StageData
 
executorRunTime() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
executorRunTime() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
executors() - Method in class org.apache.spark.status.api.v1.RDDPartitionInfo
 
executors() - Method in class org.apache.spark.status.LiveRDDPartition
 
ExecutorStageSummary - Class in org.apache.spark.status.api.v1
 
ExecutorStreamSummary - Class in org.apache.spark.ui.storage
 
ExecutorStreamSummary(Seq<org.apache.spark.status.StreamBlockData>) - Constructor for class org.apache.spark.ui.storage.ExecutorStreamSummary
 
executorSummaries() - Method in class org.apache.spark.status.LiveStage
 
ExecutorSummary - Class in org.apache.spark.status.api.v1
 
executorSummary() - Method in class org.apache.spark.status.api.v1.StageData
 
executorSummary(String) - Method in class org.apache.spark.status.LiveStage
 
exists() - Method in interface org.apache.spark.sql.streaming.GroupState
Whether state exists or not.
exists(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
exists(String) - Static method in class org.apache.spark.sql.types.UDTRegistration
Queries if a given user class is already registered or not.
exists() - Method in class org.apache.spark.streaming.State
Whether the state already exists
exitCausedByApp() - Method in class org.apache.spark.ExecutorLostFailure
 
exp(Column) - Static method in class org.apache.spark.sql.functions
Computes the exponential of the given value.
exp(String) - Static method in class org.apache.spark.sql.functions
Computes the exponential of the given column.
ExpectationAggregator - Class in org.apache.spark.ml.clustering
ExpectationAggregator computes the partial expectation results.
ExpectationAggregator(int, Broadcast<double[]>, Broadcast<Tuple2<DenseVector, DenseVector>[]>) - Constructor for class org.apache.spark.ml.clustering.ExpectationAggregator
 
ExpectationSum - Class in org.apache.spark.mllib.clustering
 
ExpectationSum(double, double[], DenseVector<Object>[], DenseMatrix<Object>[]) - Constructor for class org.apache.spark.mllib.clustering.ExpectationSum
 
expectedFpp() - Method in class org.apache.spark.util.sketch.BloomFilter
Returns the probability that BloomFilter.mightContain(Object) erroneously return true for an object that has not actually been put in the BloomFilter.
experimental() - Method in class org.apache.spark.sql.SparkSession
Experimental A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
experimental() - Method in class org.apache.spark.sql.SQLContext
Experimental A collection of methods that are considered experimental, but can be used to hook into the query planner for advanced functionality.
ExperimentalMethods - Class in org.apache.spark.sql
Experimental Holder for experimental methods for the bravest.
ExpireDeadHosts - Class in org.apache.spark
 
ExpireDeadHosts() - Constructor for class org.apache.spark.ExpireDeadHosts
 
expiryTime() - Method in class org.apache.spark.scheduler.BlacklistedExecutor
 
explain(boolean) - Method in class org.apache.spark.sql.Column
Prints the expression to the console for debugging purposes.
explain(boolean) - Method in class org.apache.spark.sql.Dataset
Prints the plans (logical and physical) to the console for debugging purposes.
explain() - Method in class org.apache.spark.sql.Dataset
Prints the physical plan to the console for debugging purposes.
explain() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Prints the physical plan to the console for debugging purposes.
explain(boolean) - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Prints the physical plan to the console for debugging purposes.
explainedVariance() - Method in class org.apache.spark.ml.feature.PCAModel
 
explainedVariance() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Returns the explained variance regression score.
explainedVariance() - Method in class org.apache.spark.mllib.evaluation.RegressionMetrics
Returns the variance explained by regression.
explainedVariance() - Method in class org.apache.spark.mllib.feature.PCAModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.LDA
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.DCT
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.IDF
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Imputer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Interaction
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.NGram
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.PCA
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormula
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
explainParam(Param<?>) - Method in interface org.apache.spark.ml.param.Params
Explains a param.
explainParam(Param<?>) - Static method in class org.apache.spark.ml.Pipeline
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.PipelineModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
explainParam(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
explainParams() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
explainParams() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
explainParams() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
explainParams() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
explainParams() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
explainParams() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
explainParams() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
explainParams() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
explainParams() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
explainParams() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
explainParams() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
explainParams() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
explainParams() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
explainParams() - Static method in class org.apache.spark.ml.clustering.KMeans
 
explainParams() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
explainParams() - Static method in class org.apache.spark.ml.clustering.LDA
 
explainParams() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
explainParams() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
explainParams() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
explainParams() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
explainParams() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
explainParams() - Static method in class org.apache.spark.ml.feature.Binarizer
 
explainParams() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
explainParams() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
explainParams() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
explainParams() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
explainParams() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
explainParams() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.DCT
 
explainParams() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
explainParams() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
explainParams() - Static method in class org.apache.spark.ml.feature.HashingTF
 
explainParams() - Static method in class org.apache.spark.ml.feature.IDF
 
explainParams() - Static method in class org.apache.spark.ml.feature.IDFModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.Imputer
 
explainParams() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.IndexToString
 
explainParams() - Static method in class org.apache.spark.ml.feature.Interaction
 
explainParams() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
explainParams() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
explainParams() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
explainParams() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.NGram
 
explainParams() - Static method in class org.apache.spark.ml.feature.Normalizer
 
explainParams() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
explainParams() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
explainParams() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.PCA
 
explainParams() - Static method in class org.apache.spark.ml.feature.PCAModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
explainParams() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
explainParams() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
explainParams() - Static method in class org.apache.spark.ml.feature.RFormula
 
explainParams() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
explainParams() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
explainParams() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
explainParams() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
explainParams() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
explainParams() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
explainParams() - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
explainParams() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
explainParams() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
explainParams() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
explainParams() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
explainParams() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
explainParams() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
explainParams() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
explainParams() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
explainParams() - Method in interface org.apache.spark.ml.param.Params
Explains all params of this instance.
explainParams() - Static method in class org.apache.spark.ml.Pipeline
 
explainParams() - Static method in class org.apache.spark.ml.PipelineModel
 
explainParams() - Static method in class org.apache.spark.ml.recommendation.ALS
 
explainParams() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
explainParams() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
explainParams() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
explainParams() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
explainParams() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
explainParams() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
explainParams() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
explainParams() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
explainParams() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
explainParams() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
explainParams() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
explainParams() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
explode(Seq<Column>, Function1<Row, TraversableOnce<A>>, TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.Dataset
Deprecated.
use flatMap() or select() with functions.explode() instead. Since 2.0.0.
explode(String, String, Function1<A, TraversableOnce<B>>, TypeTags.TypeTag<B>) - Method in class org.apache.spark.sql.Dataset
Deprecated.
use flatMap() or select() with functions.explode() instead. Since 2.0.0.
explode(Column) - Static method in class org.apache.spark.sql.functions
Creates a new row for each element in the given array or map column.
explode_outer(Column) - Static method in class org.apache.spark.sql.functions
Creates a new row for each element in the given array or map column.
expm1(Column) - Static method in class org.apache.spark.sql.functions
Computes the exponential of the given value minus one.
expm1(String) - Static method in class org.apache.spark.sql.functions
Computes the exponential of the given column.
ExponentialGenerator - Class in org.apache.spark.mllib.random
Developer API Generates i.i.d.
ExponentialGenerator(double) - Constructor for class org.apache.spark.mllib.random.ExponentialGenerator
 
exponentialJavaRDD(JavaSparkContext, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.exponentialRDD.
exponentialJavaRDD(JavaSparkContext, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.exponentialJavaRDD with the default seed.
exponentialJavaRDD(JavaSparkContext, double, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.exponentialJavaRDD with the default number of partitions and the default seed.
exponentialJavaVectorRDD(JavaSparkContext, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.exponentialVectorRDD.
exponentialJavaVectorRDD(JavaSparkContext, double, long, int, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.exponentialJavaVectorRDD with the default seed.
exponentialJavaVectorRDD(JavaSparkContext, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.exponentialJavaVectorRDD with the default number of partitions and the default seed.
exponentialRDD(SparkContext, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD comprised of i.i.d. samples from the exponential distribution with the input mean.
exponentialVectorRDD(SparkContext, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD[Vector] with vectors containing i.i.d. samples drawn from the exponential distribution with the input mean.
expr() - Method in class org.apache.spark.sql.Column
 
expr(String) - Static method in class org.apache.spark.sql.functions
Parses the expression string into the column that it represents, similar to Dataset.selectExpr(java.lang.String...).
Expression$() - Constructor for class org.apache.spark.sql.types.DecimalType.Expression$
 
expressions() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
expressions() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
expressions() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
expressions() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
extensionsForCompressionCodecNames() - Static method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
externalBlockStoreSize() - Method in class org.apache.spark.storage.RDDInfo
 
extractDistribution(Function1<BatchInfo, Option<Object>>) - Method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
extractDoubleDistribution(Seq<Tuple2<TaskInfo, TaskMetrics>>, Function2<TaskInfo, TaskMetrics, Object>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
extractFn() - Method in class org.apache.spark.ui.JettyUtils.ServletParams
 
extractHostPortFromSparkUrl(String) - Static method in class org.apache.spark.util.Utils
Return a pair of host and port extracted from the sparkUrl.
extractLongDistribution(Seq<Tuple2<TaskInfo, TaskMetrics>>, Function2<TaskInfo, TaskMetrics, Object>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
extractParamMap() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.KMeans
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.KMeans
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.LDA
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.LDA
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
extractParamMap() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
extractParamMap() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
extractParamMap() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
extractParamMap() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
extractParamMap() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Binarizer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Binarizer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.DCT
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.DCT
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.HashingTF
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.HashingTF
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.IDF
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.IDF
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.IDFModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.IDFModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Imputer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Imputer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.IndexToString
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.IndexToString
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Interaction
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Interaction
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.NGram
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.NGram
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Normalizer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Normalizer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.PCA
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.PCA
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.PCAModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.PCAModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.RFormula
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.RFormula
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
extractParamMap() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
extractParamMap() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
extractParamMap() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
extractParamMap(ParamMap) - Method in interface org.apache.spark.ml.param.Params
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values less than user-supplied values less than extra.
extractParamMap() - Method in interface org.apache.spark.ml.param.Params
extractParamMap with no extra values.
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.Pipeline
 
extractParamMap() - Static method in class org.apache.spark.ml.Pipeline
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.PipelineModel
 
extractParamMap() - Static method in class org.apache.spark.ml.PipelineModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.recommendation.ALS
 
extractParamMap() - Static method in class org.apache.spark.ml.recommendation.ALS
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
extractParamMap() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
extractParamMap() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
extractParamMap() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
extractParamMap() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
extractParamMap() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
extractParamMap(ParamMap) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
extractParamMap() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
extraOptimizations() - Method in class org.apache.spark.sql.ExperimentalMethods
 
extraStrategies() - Method in class org.apache.spark.sql.ExperimentalMethods
Allows extra strategies to be injected into the query planner at runtime.
eye(int) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
Generate an Identity Matrix in DenseMatrix format.
eye(int) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a dense Identity Matrix in Matrix format.
eye(int) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
Generate an Identity Matrix in DenseMatrix format.
eye(int) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a dense Identity Matrix in Matrix format.

F

f() - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
f1Measure() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns document-based f1-measure averaged by the number of documents
f1Measure(double) - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns f1-measure for a given label (category)
factorial(Column) - Static method in class org.apache.spark.sql.functions
Computes the factorial of the given value.
failed() - Method in class org.apache.spark.scheduler.TaskInfo
 
FAILED() - Static method in class org.apache.spark.TaskState
 
failedStages() - Method in class org.apache.spark.status.LiveJob
 
failedTasks() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
failedTasks() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
failedTasks() - Method in class org.apache.spark.status.LiveExecutor
 
failedTasks() - Method in class org.apache.spark.status.LiveExecutorStageSummary
 
failedTasks() - Method in class org.apache.spark.status.LiveJob
 
failedTasks() - Method in class org.apache.spark.status.LiveStage
 
failure(String) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
failureReason() - Method in class org.apache.spark.scheduler.StageInfo
If the stage failed, the reason why.
failureReason() - Method in class org.apache.spark.status.api.v1.StageData
 
failureReason() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
failureReason() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
failureReasonCell(String, int, boolean) - Static method in class org.apache.spark.streaming.ui.UIUtils
 
FAIR() - Static method in class org.apache.spark.scheduler.SchedulingMode
 
FAKE_HIVE_VERSION() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
FalsePositiveRate - Class in org.apache.spark.mllib.evaluation.binary
False positive rate.
FalsePositiveRate() - Constructor for class org.apache.spark.mllib.evaluation.binary.FalsePositiveRate
 
falsePositiveRate(double) - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns false positive rate for a given label (category)
falsePositiveRateByLabel() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns false positive rate for each label (category).
family() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
family() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
family() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
family() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
Family$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Family$
 
FamilyAndLink$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.FamilyAndLink$
 
fastEquals(TreeNode<?>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
fastEquals(TreeNode<?>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
fastEquals(TreeNode<?>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
fastEquals(TreeNode<?>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
fdr() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
fdr() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
fdr() - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
feature() - Method in class org.apache.spark.mllib.feature.ChiSqSelectorModel.SaveLoadV1_0$.Data
 
feature() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.SplitData
 
feature() - Method in class org.apache.spark.mllib.tree.model.Split
 
FeatureHasher - Class in org.apache.spark.ml.feature
Feature hashing projects a set of categorical or numerical features into a feature vector of specified dimension (typically substantially smaller than that of the original feature space).
FeatureHasher(String) - Constructor for class org.apache.spark.ml.feature.FeatureHasher
 
FeatureHasher() - Constructor for class org.apache.spark.ml.feature.FeatureHasher
 
featureImportances() - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
Estimate of the importance of each feature.
featureImportances() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
Estimate of the importance of each feature.
featureImportances() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
Estimate of the importance of each feature.
featureImportances() - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
Estimate of the importance of each feature.
featureImportances() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
Estimate of the importance of each feature.
featureImportances() - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
Estimate of the importance of each feature.
featureIndex() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
featureIndex() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
featureIndex() - Method in class org.apache.spark.ml.tree.CategoricalSplit
 
featureIndex() - Method in class org.apache.spark.ml.tree.ContinuousSplit
 
featureIndex() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData
 
featureIndex() - Method in interface org.apache.spark.ml.tree.Split
Index of feature which this split tests
features() - Method in class org.apache.spark.ml.feature.LabeledPoint
 
features() - Method in class org.apache.spark.mllib.regression.LabeledPoint
 
featuresCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
featuresCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
featuresCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
featuresCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
featuresCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
featuresCol() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Field in "predictions" which gives the features of each instance as a vector.
featuresCol() - Method in class org.apache.spark.ml.classification.LogisticRegressionSummaryImpl
 
featuresCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
featuresCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
featuresCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
featuresCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
featuresCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
featuresCol() - Method in class org.apache.spark.ml.clustering.ClusteringSummary
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.KMeans
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.LDA
 
featuresCol() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
featuresCol() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
featuresCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
featuresCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
featuresCol() - Static method in class org.apache.spark.ml.feature.RFormula
 
featuresCol() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
featuresCol() - Method in interface org.apache.spark.ml.param.shared.HasFeaturesCol
Param for features column name.
featuresCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
featuresCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
featuresCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
featuresCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
featuresCol() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
featuresCol() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
featuresCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
featuresCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
featuresCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
featuresCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
featuresCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
featuresCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
featuresCol() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
 
featuresCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
featuresCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
featureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
featureSum() - Method in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette.ClusterStats
 
FeatureType - Class in org.apache.spark.mllib.tree.configuration
Enum to describe whether a feature is "continuous" or "categorical"
FeatureType() - Constructor for class org.apache.spark.mllib.tree.configuration.FeatureType
 
featureType() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.SplitData
 
featureType() - Method in class org.apache.spark.mllib.tree.model.Split
 
FETCH_WAIT_TIME() - Method in class org.apache.spark.InternalAccumulator.shuffleRead$
 
FetchFailed - Class in org.apache.spark
Developer API Task failed to fetch shuffle data from a remote node.
FetchFailed(BlockManagerId, int, int, int, String) - Constructor for class org.apache.spark.FetchFailed
 
fetchFile(String, File, SparkConf, org.apache.spark.SecurityManager, Configuration, long, boolean) - Static method in class org.apache.spark.util.Utils
Download a file or directory to target directory.
fetchPct() - Method in class org.apache.spark.scheduler.RuntimePercentage
 
fetchWaitTime() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
fetchWaitTime() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetrics
 
field() - Method in class org.apache.spark.storage.BroadcastBlockId
 
fieldIndex(String) - Method in interface org.apache.spark.sql.Row
Returns the index of a given field name.
fieldIndex(String) - Method in class org.apache.spark.sql.types.StructType
Returns the index of a given field.
fieldNames() - Method in class org.apache.spark.sql.types.StructType
Returns all field names in an array.
fields() - Method in class org.apache.spark.sql.types.StructType
 
FIFO() - Static method in class org.apache.spark.scheduler.SchedulingMode
 
FILE_FORMAT() - Static method in class org.apache.spark.sql.hive.execution.HiveOptions
 
FileBasedTopologyMapper - Class in org.apache.spark.storage
A simple file based topology mapper.
FileBasedTopologyMapper(SparkConf) - Constructor for class org.apache.spark.storage.FileBasedTopologyMapper
 
FileCommitProtocol - Class in org.apache.spark.internal.io
An interface to define how a single Spark job commits its outputs.
FileCommitProtocol() - Constructor for class org.apache.spark.internal.io.FileCommitProtocol
 
FileCommitProtocol.EmptyTaskCommitMessage$ - Class in org.apache.spark.internal.io
 
FileCommitProtocol.TaskCommitMessage - Class in org.apache.spark.internal.io
 
fileFormat() - Method in class org.apache.spark.sql.hive.execution.HiveOptions
 
files() - Method in class org.apache.spark.SparkContext
 
fileStream(String, Class<K>, Class<V>, Class<F>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
fileStream(String, Class<K>, Class<V>, Class<F>, Function<Path, Boolean>, boolean) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
fileStream(String, Class<K>, Class<V>, Class<F>, Function<Path, Boolean>, boolean, Configuration) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
fileStream(String, ClassTag<K>, ClassTag<V>, ClassTag<F>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
fileStream(String, Function1<Path, Object>, boolean, ClassTag<K>, ClassTag<V>, ClassTag<F>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
fileStream(String, Function1<Path, Object>, boolean, Configuration, ClassTag<K>, ClassTag<V>, ClassTag<F>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them using the given key-value types and input format.
fill(long) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null or NaN values in numeric columns with value.
fill(double) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null or NaN values in numeric columns with value.
fill(String) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null values in string columns with value.
fill(long, String[]) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null or NaN values in specified numeric columns.
fill(double, String[]) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null or NaN values in specified numeric columns.
fill(long, Seq<String>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that replaces null or NaN values in specified numeric columns.
fill(double, Seq<String>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that replaces null or NaN values in specified numeric columns.
fill(String, String[]) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null values in specified string columns.
fill(String, Seq<String>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that replaces null values in specified string columns.
fill(boolean) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null values in boolean columns with value.
fill(boolean, Seq<String>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that replaces null values in specified boolean columns.
fill(boolean, String[]) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null values in specified boolean columns.
fill(Map<String, Object>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Returns a new DataFrame that replaces null values.
fill(Map<String, Object>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Returns a new DataFrame that replaces null values.
fillInStackTrace() - Static method in exception org.apache.spark.sql.AnalysisException
 
filter(Function<Double, Boolean>) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD containing only the elements that satisfy a predicate.
filter(Function<Tuple2<K, V>, Boolean>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD containing only the elements that satisfy a predicate.
filter(Function<T, Boolean>) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD containing only the elements that satisfy a predicate.
filter(Function1<T, Object>) - Static method in class org.apache.spark.api.r.RRDD
 
filter(Function1<T, Object>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
filter(Function1<Graph<VD, ED>, Graph<VD2, ED2>>, Function1<EdgeTriplet<VD2, ED2>, Object>, Function2<Object, VD2, Object>, ClassTag<VD2>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.GraphOps
Filter the graph by computing some values to filter on, and applying the predicates.
filter(Function1<EdgeTriplet<VD, ED>, Object>, Function2<Object, VD, Object>) - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
filter(Function1<Tuple2<Object, VD>, Object>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
filter(Function1<Tuple2<Object, VD>, Object>) - Method in class org.apache.spark.graphx.VertexRDD
Restricts the vertex set to the set of vertices satisfying the given predicate.
filter(Params) - Method in class org.apache.spark.ml.param.ParamMap
Filters this param map for the given parent.
filter(Function1<T, Object>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
filter(Function1<T, Object>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
filter(Function1<T, Object>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
filter(Function1<T, Object>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
filter(Function1<T, Object>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD containing only the elements that satisfy a predicate.
filter(Function1<T, Object>) - Static method in class org.apache.spark.rdd.UnionRDD
 
filter(Column) - Method in class org.apache.spark.sql.Dataset
Filters rows using the given condition.
filter(String) - Method in class org.apache.spark.sql.Dataset
Filters rows using the given SQL expression.
filter(Function1<T, Object>) - Method in class org.apache.spark.sql.Dataset
Experimental (Scala-specific) Returns a new Dataset that only contains elements where func returns true.
filter(FilterFunction<T>) - Method in class org.apache.spark.sql.Dataset
Experimental (Java-specific) Returns a new Dataset that only contains elements where func returns true.
Filter - Class in org.apache.spark.sql.sources
A filter predicate for data sources.
Filter() - Constructor for class org.apache.spark.sql.sources.Filter
 
filter(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
filter() - Method in class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds
 
filter(Function<T, Boolean>) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Return a new DStream containing only the elements that satisfy a predicate.
filter(Function<T, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
filter(Function<Tuple2<K, V>, Boolean>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream containing only the elements that satisfy a predicate.
filter(Function<Tuple2<K, V>, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
filter(Function<Tuple2<K, V>, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
filter(Function<T, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
filter(Function1<T, Object>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream containing only the elements that satisfy a predicate.
filterByRange(K, K) - Method in class org.apache.spark.rdd.OrderedRDDFunctions
Returns an RDD containing only the elements in the inclusive range lower to upper.
FilterFunction<T> - Interface in org.apache.spark.api.java.function
Base interface for a function used in Dataset's filter function.
filterName() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter
 
filterNot(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
filterParams() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter
 
finalStorageLevel() - Static method in class org.apache.spark.ml.recommendation.ALS
 
find(Function1<BaseType, Object>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
find(Function1<BaseType, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
find(Function1<BaseType, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
find(Function1<BaseType, Object>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
find(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
findClass(String) - Method in class org.apache.spark.util.ParentClassLoader
 
findMissingPartitions() - Method in class org.apache.spark.ShuffleStatus
Returns the sequence of partition ids that are missing (i.e.
findSynonyms(String, int) - Method in class org.apache.spark.ml.feature.Word2VecModel
Find "num" number of words closest in similarity to the given word, not including the word itself.
findSynonyms(Vector, int) - Method in class org.apache.spark.ml.feature.Word2VecModel
Find "num" number of words whose vector representation is most similar to the supplied vector.
findSynonyms(String, int) - Method in class org.apache.spark.mllib.feature.Word2VecModel
Find synonyms of a word; do not include the word itself in results.
findSynonyms(Vector, int) - Method in class org.apache.spark.mllib.feature.Word2VecModel
Find synonyms of the vector representation of a word, possibly including any words in the model vocabulary whose vector respresentation is the supplied vector.
findSynonymsArray(Vector, int) - Method in class org.apache.spark.ml.feature.Word2VecModel
Find "num" number of words whose vector representation is most similar to the supplied vector.
findSynonymsArray(String, int) - Method in class org.apache.spark.ml.feature.Word2VecModel
Find "num" number of words closest in similarity to the given word, not including the word itself.
finish(BUF) - Method in class org.apache.spark.sql.expressions.Aggregator
Transform the output of the reduction.
finished() - Method in class org.apache.spark.scheduler.TaskInfo
 
FINISHED() - Static method in class org.apache.spark.TaskState
 
finishTime() - Method in class org.apache.spark.scheduler.TaskInfo
The time when the task has completed successfully (including the time to remotely fetch results, if necessary).
first() - Method in class org.apache.spark.api.java.JavaDoubleRDD
 
first() - Method in class org.apache.spark.api.java.JavaPairRDD
 
first() - Static method in class org.apache.spark.api.java.JavaRDD
 
first() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the first element in this RDD.
first() - Static method in class org.apache.spark.api.r.RRDD
 
first() - Static method in class org.apache.spark.graphx.EdgeRDD
 
first() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
first() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
first() - Static method in class org.apache.spark.graphx.VertexRDD
 
first() - Static method in class org.apache.spark.rdd.HadoopRDD
 
first() - Static method in class org.apache.spark.rdd.JdbcRDD
 
first() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
first() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
first() - Method in class org.apache.spark.rdd.RDD
Return the first element in this RDD.
first() - Static method in class org.apache.spark.rdd.UnionRDD
 
first() - Method in class org.apache.spark.sql.Dataset
Returns the first row.
first(Column, boolean) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the first value in a group.
first(String, boolean) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the first value of a column in a group.
first(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the first value in a group.
first(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the first value of a column in a group.
firstFailureReason() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
firstLaunchTime() - Method in class org.apache.spark.status.LiveStage
 
firstTaskLaunchedTime() - Method in class org.apache.spark.status.api.v1.StageData
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.classification.OneVsRest
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.clustering.KMeans
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.clustering.LDA
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Method in class org.apache.spark.ml.Estimator
Fits a single model to the input data with optional parameters.
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Method in class org.apache.spark.ml.Estimator
Fits a single model to the input data with optional parameters.
fit(Dataset<?>, ParamMap) - Method in class org.apache.spark.ml.Estimator
Fits a single model to the input data with provided parameter map.
fit(Dataset<?>) - Method in class org.apache.spark.ml.Estimator
Fits a model to the input data.
fit(Dataset<?>, ParamMap[]) - Method in class org.apache.spark.ml.Estimator
Fits multiple models to the input data with multiple sets of parameters.
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.IDF
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.Imputer
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.MaxAbsScaler
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.PCA
Computes a PCAModel that contains the principal components of the input vectors.
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.RFormula
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.StandardScaler
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.StringIndexer
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.VectorIndexer
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.feature.Word2Vec
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.Pipeline
Fits the pipeline to the input dataset with additional parameters.
fit(Dataset<?>) - Method in class org.apache.spark.ml.Predictor
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.recommendation.ALS
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
fit(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
fit(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
fit(Dataset<?>, ParamMap[]) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
fit(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
fit(Dataset<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
fit(Dataset<?>) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
fit(RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.feature.ChiSqSelector
Returns a ChiSquared feature selector.
fit(RDD<Vector>) - Method in class org.apache.spark.mllib.feature.IDF
Computes the inverse document frequency.
fit(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.feature.IDF
Computes the inverse document frequency.
fit(RDD<Vector>) - Method in class org.apache.spark.mllib.feature.PCA
Computes a PCAModel that contains the principal components of the input vectors.
fit(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.feature.PCA
Java-friendly version of fit().
fit(RDD<Vector>) - Method in class org.apache.spark.mllib.feature.StandardScaler
Computes the mean and variance and stores as a model to be used for later scaling.
fit(RDD<S>) - Method in class org.apache.spark.mllib.feature.Word2Vec
Computes the vector representation of each word in vocabulary.
fit(JavaRDD<S>) - Method in class org.apache.spark.mllib.feature.Word2Vec
Computes the vector representation of each word in vocabulary (Java version).
fitIntercept() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
fitIntercept() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
fitIntercept() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
fitIntercept() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
fitIntercept() - Method in interface org.apache.spark.ml.param.shared.HasFitIntercept
Param for whether to fit an intercept term.
fitIntercept() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
fitIntercept() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
fitIntercept() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
fitIntercept() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
fitIntercept() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
fitIntercept() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
Fixed$() - Constructor for class org.apache.spark.sql.types.DecimalType.Fixed$
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.api.java.JavaRDD
 
flatMap(FlatMapFunction<T, U>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
flatMap(Function1<T, TraversableOnce<U>>, Encoder<U>) - Method in class org.apache.spark.sql.Dataset
Experimental (Scala-specific) Returns a new Dataset by first applying a function to all elements of this Dataset, and then flattening the results.
flatMap(FlatMapFunction<T, U>, Encoder<U>) - Method in class org.apache.spark.sql.Dataset
Experimental (Java-specific) Returns a new Dataset by first applying a function to all elements of this Dataset, and then flattening the results.
flatMap(Function1<BaseType, TraversableOnce<A>>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
flatMap(Function1<BaseType, TraversableOnce<A>>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
flatMap(Function1<BaseType, TraversableOnce<A>>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
flatMap(Function1<BaseType, TraversableOnce<A>>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
flatMap(Function1<A, GenTraversableOnce<B>>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
flatMap(FlatMapFunction<T, U>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream by applying a function to all elements of this DStream, and then flattening the results
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
flatMap(FlatMapFunction<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
flatMap(Function1<T, TraversableOnce<U>>, ClassTag<U>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream by applying a function to all elements of this DStream, and then flattening the results
FlatMapFunction<T,R> - Interface in org.apache.spark.api.java.function
A function that returns zero or more output records from each input record.
FlatMapFunction2<T1,T2,R> - Interface in org.apache.spark.api.java.function
A function that takes two inputs and returns zero or more output records.
flatMapGroups(Function2<K, Iterator<V>, TraversableOnce<U>>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Scala-specific) Applies the given function to each group of data.
flatMapGroups(FlatMapGroupsFunction<K, V, U>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Java-specific) Applies the given function to each group of data.
FlatMapGroupsFunction<K,V,R> - Interface in org.apache.spark.api.java.function
A function that returns zero or more output records from each grouping key and its values.
flatMapGroupsWithState(OutputMode, GroupStateTimeout, Function3<K, Iterator<V>, GroupState<S>, Iterator<U>>, Encoder<S>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
::Experimental:: (Scala-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state.
flatMapGroupsWithState(FlatMapGroupsWithStateFunction<K, V, S, U>, OutputMode, Encoder<S>, Encoder<U>, GroupStateTimeout) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
::Experimental:: (Java-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state.
FlatMapGroupsWithStateFunction<K,V,S,R> - Interface in org.apache.spark.api.java.function
::Experimental:: Base interface for a map function used in org.apache.spark.sql.KeyValueGroupedDataset.flatMapGroupsWithState( FlatMapGroupsWithStateFunction, org.apache.spark.sql.streaming.OutputMode, org.apache.spark.sql.Encoder, org.apache.spark.sql.Encoder)
flatMapToDouble(DoubleFlatMapFunction<T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
flatMapToDouble(DoubleFlatMapFunction<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
flatMapToDouble(DoubleFlatMapFunction<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
flatMapToDouble(DoubleFlatMapFunction<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.api.java.JavaRDD
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by first applying a function to all elements of this RDD, and then flattening the results.
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream by applying a function to all elements of this DStream, and then flattening the results
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
flatMapToPair(PairFlatMapFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
flatMapValues(Function<V, Iterable<U>>) - Method in class org.apache.spark.api.java.JavaPairRDD
Pass each value in the key-value pair RDD through a flatMap function without changing the keys; this also retains the original RDD's partitioning.
flatMapValues(Function1<V, TraversableOnce<U>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Pass each value in the key-value pair RDD through a flatMap function without changing the keys; this also retains the original RDD's partitioning.
flatMapValues(Function<V, Iterable<U>>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying a flatmap function to the value of each key-value pairs in 'this' DStream without changing the key.
flatMapValues(Function<V, Iterable<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
flatMapValues(Function<V, Iterable<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
flatMapValues(Function1<V, TraversableOnce<U>>, ClassTag<U>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying a flatmap function to the value of each key-value pairs in 'this' DStream without changing the key.
flatten(Function1<A, GenTraversableOnce<B>>) - Static method in class org.apache.spark.sql.types.StructType
 
FLOAT() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable float type.
FloatAccumulatorParam$() - Constructor for class org.apache.spark.AccumulatorParam.FloatAccumulatorParam$
Deprecated.
 
FloatParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Float] for Java.
FloatParam(String, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.FloatParam
 
FloatParam(String, String, String) - Constructor for class org.apache.spark.ml.param.FloatParam
 
FloatParam(Identifiable, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.FloatParam
 
FloatParam(Identifiable, String, String) - Constructor for class org.apache.spark.ml.param.FloatParam
 
FloatType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the FloatType object.
FloatType - Class in org.apache.spark.sql.types
The data type representing Float values.
floor(Column) - Static method in class org.apache.spark.sql.functions
Computes the floor of the given value.
floor(String) - Static method in class org.apache.spark.sql.functions
Computes the floor of the given column.
floor() - Method in class org.apache.spark.sql.types.Decimal
 
floor(Duration) - Method in class org.apache.spark.streaming.Time
 
floor(Duration, Time) - Method in class org.apache.spark.streaming.Time
 
flush() - Method in class org.apache.spark.io.SnappyOutputStreamWrapper
 
flush() - Method in class org.apache.spark.serializer.SerializationStream
 
flush() - Method in class org.apache.spark.storage.TimeTrackingOutputStream
 
fMeasure(double, double) - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns f-measure for a given label (category)
fMeasure(double) - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns f1-measure for a given label (category)
fMeasure() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Deprecated.
Use accuracy. Since 2.0.0.
fMeasureByLabel(double) - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns f-measure for each label (category).
fMeasureByLabel() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns f1-measure for each label (category).
fMeasureByThreshold() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
Returns a dataframe with two fields (threshold, F-Measure) curve with beta = 1.0.
fMeasureByThreshold(double) - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Returns the (threshold, F-Measure) curve.
fMeasureByThreshold() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Returns the (threshold, F-Measure) curve with beta = 1.0.
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
fold(T, Function2<T, T, T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value".
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.api.r.RRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
fold(T, Function2<T, T, T>) - Method in class org.apache.spark.rdd.RDD
Aggregate the elements of each partition, and then the results for all the partitions, using a given associative function and a neutral "zero value".
fold(T, Function2<T, T, T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
fold(A1, Function2<A1, A1, A1>) - Static method in class org.apache.spark.sql.types.StructType
 
foldByKey(V, Partitioner, Function2<V, V, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g ., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
foldByKey(V, int, Function2<V, V, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g ., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
foldByKey(V, Function2<V, V, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
foldByKey(V, Partitioner, Function2<V, V, V>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
foldByKey(V, int, Function2<V, V, V>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
foldByKey(V, Function2<V, V, V>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).
foldLeft(B, Function2<B, A, B>) - Static method in class org.apache.spark.sql.types.StructType
 
foldRight(B, Function2<A, B, B>) - Static method in class org.apache.spark.sql.types.StructType
 
forall(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
forceIndexLabel() - Static method in class org.apache.spark.ml.feature.RFormula
 
forceIndexLabel() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
foreach(VoidFunction<T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
foreach(VoidFunction<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
foreach(VoidFunction<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
foreach(VoidFunction<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Applies a function f to all elements of this RDD.
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.api.r.RRDD
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.graphx.VertexRDD
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
foreach(Function1<T, BoxedUnit>) - Method in class org.apache.spark.rdd.RDD
Applies a function f to all elements of this RDD.
foreach(Function1<T, BoxedUnit>) - Static method in class org.apache.spark.rdd.UnionRDD
 
foreach(Function1<T, BoxedUnit>) - Method in class org.apache.spark.sql.Dataset
Applies a function f to all rows.
foreach(ForeachFunction<T>) - Method in class org.apache.spark.sql.Dataset
(Java-specific) Runs func on each element of this Dataset.
foreach(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
foreach(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
foreach(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
foreach(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
foreach(ForeachWriter<T>) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Starts the execution of the streaming query, which will continually send results to the given ForeachWriter as new data arrives.
foreach(Function1<A, U>) - Static method in class org.apache.spark.sql.types.StructType
 
foreach(Function1<A, U>) - Static method in class org.apache.spark.sql.types.StructType
 
foreachActive(Function3<Object, Object, Object, BoxedUnit>) - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
foreachActive(Function2<Object, Object, BoxedUnit>) - Method in class org.apache.spark.ml.linalg.DenseVector
 
foreachActive(Function3<Object, Object, Object, BoxedUnit>) - Method in interface org.apache.spark.ml.linalg.Matrix
Applies a function f to all the active elements of dense and sparse matrix.
foreachActive(Function3<Object, Object, Object, BoxedUnit>) - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
foreachActive(Function2<Object, Object, BoxedUnit>) - Method in class org.apache.spark.ml.linalg.SparseVector
 
foreachActive(Function2<Object, Object, BoxedUnit>) - Method in interface org.apache.spark.ml.linalg.Vector
Applies a function f to all the active elements of dense and sparse vector.
foreachActive(Function2<Object, Object, BoxedUnit>) - Method in class org.apache.spark.mllib.linalg.DenseVector
 
foreachActive(Function3<Object, Object, Object, BoxedUnit>) - Method in interface org.apache.spark.mllib.linalg.Matrix
Applies a function f to all the active elements of dense and sparse matrix.
foreachActive(Function2<Object, Object, BoxedUnit>) - Method in class org.apache.spark.mllib.linalg.SparseVector
 
foreachActive(Function2<Object, Object, BoxedUnit>) - Method in interface org.apache.spark.mllib.linalg.Vector
Applies a function f to all the active elements of dense and sparse vector.
foreachAsync(VoidFunction<T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
foreachAsync(VoidFunction<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
foreachAsync(VoidFunction<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
foreachAsync(VoidFunction<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
The asynchronous version of the foreach action, which applies a function f to all the elements of this RDD.
foreachAsync(Function1<T, BoxedUnit>) - Method in class org.apache.spark.rdd.AsyncRDDActions
Applies a function f to all elements of this RDD.
ForeachFunction<T> - Interface in org.apache.spark.api.java.function
Base interface for a function used in Dataset's foreach function.
foreachPartition(VoidFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
foreachPartition(VoidFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
foreachPartition(VoidFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaRDD
 
foreachPartition(VoidFunction<Iterator<T>>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Applies a function f to each partition of this RDD.
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.api.r.RRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.graphx.VertexRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Method in class org.apache.spark.rdd.RDD
Applies a function f to each partition of this RDD.
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Static method in class org.apache.spark.rdd.UnionRDD
 
foreachPartition(Function1<Iterator<T>, BoxedUnit>) - Method in class org.apache.spark.sql.Dataset
Applies a function f to each partition of this Dataset.
foreachPartition(ForeachPartitionFunction<T>) - Method in class org.apache.spark.sql.Dataset
(Java-specific) Runs func on each partition of this Dataset.
foreachPartitionAsync(VoidFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
foreachPartitionAsync(VoidFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
foreachPartitionAsync(VoidFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaRDD
 
foreachPartitionAsync(VoidFunction<Iterator<T>>) - Method in interface org.apache.spark.api.java.JavaRDDLike
The asynchronous version of the foreachPartition action, which applies a function f to each partition of this RDD.
foreachPartitionAsync(Function1<Iterator<T>, BoxedUnit>) - Method in class org.apache.spark.rdd.AsyncRDDActions
Applies a function f to each partition of this RDD.
ForeachPartitionFunction<T> - Interface in org.apache.spark.api.java.function
Base interface for a function used in Dataset's foreachPartition function.
foreachRDD(VoidFunction<R>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
foreachRDD(VoidFunction2<R, Time>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
foreachRDD(VoidFunction<R>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Apply a function to each RDD in this DStream.
foreachRDD(VoidFunction2<R, Time>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Apply a function to each RDD in this DStream.
foreachRDD(VoidFunction<R>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
foreachRDD(VoidFunction2<R, Time>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
foreachRDD(VoidFunction<R>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
foreachRDD(VoidFunction2<R, Time>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
foreachRDD(VoidFunction<R>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
foreachRDD(VoidFunction2<R, Time>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
foreachRDD(VoidFunction<R>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
foreachRDD(VoidFunction2<R, Time>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
foreachRDD(VoidFunction<R>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
foreachRDD(VoidFunction2<R, Time>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
foreachRDD(Function1<RDD<T>, BoxedUnit>) - Method in class org.apache.spark.streaming.dstream.DStream
Apply a function to each RDD in this DStream.
foreachRDD(Function2<RDD<T>, Time, BoxedUnit>) - Method in class org.apache.spark.streaming.dstream.DStream
Apply a function to each RDD in this DStream.
foreachUp(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
foreachUp(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
foreachUp(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
foreachUp(Function1<BaseType, BoxedUnit>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
ForeachWriter<T> - Class in org.apache.spark.sql
A class to consume data generated by a StreamingQuery.
ForeachWriter() - Constructor for class org.apache.spark.sql.ForeachWriter
 
format(String) - Method in class org.apache.spark.sql.DataFrameReader
Specifies the input data source format.
format(String) - Method in class org.apache.spark.sql.DataFrameWriter
Specifies the underlying output data source.
format(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Specifies the input data source format.
format(String) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Specifies the underlying output data source.
format_number(Column, int) - Static method in class org.apache.spark.sql.functions
Formats numeric column x to a format like '#,###,###.##', rounded to d decimal places with HALF_EVEN round mode, and returns the result as a string column.
format_string(String, Column...) - Static method in class org.apache.spark.sql.functions
Formats the arguments in printf-style and returns the result as a string column.
format_string(String, Seq<Column>) - Static method in class org.apache.spark.sql.functions
Formats the arguments in printf-style and returns the result as a string column.
formatBatchTime(long, long, boolean, TimeZone) - Static method in class org.apache.spark.streaming.ui.UIUtils
If batchInterval is less than 1 second, format batchTime with milliseconds.
formatDate(Date) - Static method in class org.apache.spark.ui.UIUtils
 
formatDate(long) - Static method in class org.apache.spark.ui.UIUtils
 
formatDuration(long) - Static method in class org.apache.spark.ui.UIUtils
 
formatDurationVerbose(long) - Static method in class org.apache.spark.ui.UIUtils
Generate a verbose human-readable string representing a duration such as "5 second 35 ms"
formatNumber(double) - Static method in class org.apache.spark.ui.UIUtils
Generate a human-readable string representing a number (e.g.
formatVersion() - Method in interface org.apache.spark.mllib.util.Saveable
Current version of model save/load format.
formula() - Static method in class org.apache.spark.ml.feature.RFormula
 
formula() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
FPGrowth - Class in org.apache.spark.ml.fpm
Experimental A parallel FP-growth algorithm to mine frequent itemsets.
FPGrowth(String) - Constructor for class org.apache.spark.ml.fpm.FPGrowth
 
FPGrowth() - Constructor for class org.apache.spark.ml.fpm.FPGrowth
 
FPGrowth - Class in org.apache.spark.mllib.fpm
A parallel FP-growth algorithm to mine frequent itemsets.
FPGrowth() - Constructor for class org.apache.spark.mllib.fpm.FPGrowth
Constructs a default instance with default parameters {minSupport: 0.3, numPartitions: same as the input data}.
FPGrowth.FreqItemset<Item> - Class in org.apache.spark.mllib.fpm
Frequent itemset.
FPGrowthModel - Class in org.apache.spark.ml.fpm
Experimental Model fitted by FPGrowth.
FPGrowthModel<Item> - Class in org.apache.spark.mllib.fpm
Model trained by FPGrowth, which holds frequent itemsets.
FPGrowthModel(RDD<FPGrowth.FreqItemset<Item>>, ClassTag<Item>) - Constructor for class org.apache.spark.mllib.fpm.FPGrowthModel
 
FPGrowthModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.fpm
 
fpr() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
fpr() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
fpr() - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
freq() - Method in class org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
 
freq() - Method in class org.apache.spark.mllib.fpm.PrefixSpan.FreqSequence
 
freqItems(String[], double) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Finding frequent items for columns, possibly with false positives.
freqItems(String[]) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Finding frequent items for columns, possibly with false positives.
freqItems(Seq<String>, double) - Method in class org.apache.spark.sql.DataFrameStatFunctions
(Scala-specific) Finding frequent items for columns, possibly with false positives.
freqItems(Seq<String>) - Method in class org.apache.spark.sql.DataFrameStatFunctions
(Scala-specific) Finding frequent items for columns, possibly with false positives.
FreqItemset(Object, long) - Constructor for class org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
 
freqItemsets() - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
freqItemsets() - Method in class org.apache.spark.mllib.fpm.FPGrowthModel
 
FreqSequence(Object[], long) - Constructor for class org.apache.spark.mllib.fpm.PrefixSpan.FreqSequence
 
freqSequences() - Method in class org.apache.spark.mllib.fpm.PrefixSpanModel
 
from_json(Column, StructType, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Scala-specific) Parses a column containing a JSON string into a StructType with the specified schema.
from_json(Column, DataType, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Scala-specific) Parses a column containing a JSON string into a StructType or ArrayType of StructTypes with the specified schema.
from_json(Column, StructType, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Java-specific) Parses a column containing a JSON string into a StructType with the specified schema.
from_json(Column, DataType, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Java-specific) Parses a column containing a JSON string into a StructType or ArrayType of StructTypes with the specified schema.
from_json(Column, StructType) - Static method in class org.apache.spark.sql.functions
Parses a column containing a JSON string into a StructType with the specified schema.
from_json(Column, DataType) - Static method in class org.apache.spark.sql.functions
Parses a column containing a JSON string into a StructType or ArrayType of StructTypes with the specified schema.
from_json(Column, String, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Java-specific) Parses a column containing a JSON string into a StructType or ArrayType of StructTypes with the specified schema.
from_json(Column, String, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Scala-specific) Parses a column containing a JSON string into a StructType or ArrayType of StructTypes with the specified schema.
from_unixtime(Column) - Static method in class org.apache.spark.sql.functions
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
from_unixtime(Column, String) - Static method in class org.apache.spark.sql.functions
Converts the number of seconds from unix epoch (1970-01-01 00:00:00 UTC) to a string representing the timestamp of that moment in the current system time zone in the given format.
from_utc_timestamp(Column, String) - Static method in class org.apache.spark.sql.functions
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in UTC, and renders that time as a timestamp in the given time zone.
fromCOO(int, int, Iterable<Tuple3<Object, Object, Object>>) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
Generate a SparseMatrix from Coordinate List (COO) format.
fromCOO(int, int, Iterable<Tuple3<Object, Object, Object>>) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
Generate a SparseMatrix from Coordinate List (COO) format.
fromDDL(String) - Static method in class org.apache.spark.sql.types.StructType
Creates StructType for a given DDL-formatted string, which is a comma separated list of field definitions, e.g., a INT, b STRING.
fromDecimal(Object) - Static method in class org.apache.spark.sql.types.Decimal
 
fromDStream(DStream<T>, ClassTag<T>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
Convert a scala DStream to a Java-friendly JavaDStream.
fromEdgePartitions(RDD<Tuple2<Object, EdgePartition<ED, VD>>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from EdgePartitions, setting referenced vertices to defaultVertexAttr.
fromEdges(RDD<Edge<ED>>, ClassTag<ED>, ClassTag<VD>) - Static method in class org.apache.spark.graphx.EdgeRDD
Creates an EdgeRDD from a set of edges.
fromEdges(RDD<Edge<ED>>, VD, StorageLevel, StorageLevel, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.Graph
Construct a graph from a collection of edges.
fromEdges(EdgeRDD<?>, int, VD, ClassTag<VD>) - Static method in class org.apache.spark.graphx.VertexRDD
Constructs a VertexRDD containing all vertices referred to in edges.
fromEdgeTuples(RDD<Tuple2<Object, Object>>, VD, Option<PartitionStrategy>, StorageLevel, StorageLevel, ClassTag<VD>) - Static method in class org.apache.spark.graphx.Graph
Construct a graph from a collection of edges encoded as vertex id pairs.
fromExistingRDDs(VertexRDD<VD>, EdgeRDD<ED>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
Create a graph from a VertexRDD and an EdgeRDD with the same replicated vertex type as the vertices.
fromInputDStream(InputDStream<T>, ClassTag<T>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
Convert a scala InputDStream to a Java-friendly JavaInputDStream.
fromInputDStream(InputDStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
Convert a scala InputDStream of pairs to a Java-friendly JavaPairInputDStream.
fromJavaDStream(JavaDStream<Tuple2<K, V>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
fromJavaRDD(JavaRDD<Tuple2<K, V>>) - Static method in class org.apache.spark.api.java.JavaPairRDD
Convert a JavaRDD of key-value pairs to JavaPairRDD.
fromJson(String) - Static method in class org.apache.spark.ml.linalg.JsonMatrixConverter
Parses the JSON representation of a Matrix into a Matrix.
fromJson(String) - Static method in class org.apache.spark.ml.linalg.JsonVectorConverter
Parses the JSON representation of a vector into a Vector.
fromJson(String) - Static method in class org.apache.spark.mllib.linalg.Vectors
Parses the JSON representation of a vector into a Vector.
fromJson(String) - Static method in class org.apache.spark.sql.types.DataType
 
fromJson(String) - Static method in class org.apache.spark.sql.types.Metadata
Creates a Metadata instance from JSON.
fromKinesisInitialPosition(InitialPositionInStream) - Static method in class org.apache.spark.streaming.kinesis.KinesisInitialPositions
Returns instance of [[KinesisInitialPosition]] based on the passed [[InitialPositionInStream]].
fromML(DenseMatrix) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
Convert new linalg type to spark.mllib type.
fromML(DenseVector) - Static method in class org.apache.spark.mllib.linalg.DenseVector
Convert new linalg type to spark.mllib type.
fromML(Matrix) - Static method in class org.apache.spark.mllib.linalg.Matrices
Convert new linalg type to spark.mllib type.
fromML(SparseMatrix) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
Convert new linalg type to spark.mllib type.
fromML(SparseVector) - Static method in class org.apache.spark.mllib.linalg.SparseVector
Convert new linalg type to spark.mllib type.
fromML(Vector) - Static method in class org.apache.spark.mllib.linalg.Vectors
Convert new linalg type to spark.mllib type.
fromName(String) - Static method in class org.apache.spark.ml.attribute.AttributeType
Gets the AttributeType object from its name.
fromNullable(T) - Static method in class org.apache.spark.api.java.Optional
 
fromOld(Node, Map<Object, Object>) - Static method in class org.apache.spark.ml.tree.Node
Create a new Node from the old Node format, recursively creating child nodes as needed.
fromPairDStream(DStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
fromPairRDD(RDD<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Static method in class org.apache.spark.mllib.rdd.MLPairRDDFunctions
Implicit conversion from a pair RDD to MLPairRDDFunctions.
fromParams(GeneralizedLinearRegressionBase) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Family$
Gets the Family object based on param family and variancePower.
fromParams(GeneralizedLinearRegressionBase) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Link$
Gets the Link object based on param family, link and linkPower.
fromRDD(RDD<Object>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
fromRDD(RDD<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
fromRDD(RDD<T>, ClassTag<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
fromRDD(RDD<T>, ClassTag<T>) - Static method in class org.apache.spark.mllib.rdd.RDDFunctions
Implicit conversion from an RDD to RDDFunctions.
fromRdd(RDD<?>) - Static method in class org.apache.spark.storage.RDDInfo
 
fromReceiverInputDStream(ReceiverInputDStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
Convert a scala ReceiverInputDStream to a Java-friendly JavaReceiverInputDStream.
fromReceiverInputDStream(ReceiverInputDStream<T>, ClassTag<T>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
Convert a scala ReceiverInputDStream to a Java-friendly JavaReceiverInputDStream.
fromSparkContext(SparkContext) - Static method in class org.apache.spark.api.java.JavaSparkContext
 
fromStage(Stage, int, Option<Object>, TaskMetrics, Seq<Seq<TaskLocation>>) - Static method in class org.apache.spark.scheduler.StageInfo
Construct a StageInfo from a Stage.
fromString(String) - Static method in enum org.apache.spark.JobExecutionStatus
 
fromString(String) - Static method in class org.apache.spark.mllib.tree.impurity.Impurities
 
fromString(String) - Static method in class org.apache.spark.mllib.tree.loss.Losses
 
fromString(String) - Static method in enum org.apache.spark.status.api.v1.ApplicationStatus
 
fromString(String) - Static method in enum org.apache.spark.status.api.v1.StageStatus
 
fromString(String) - Static method in enum org.apache.spark.status.api.v1.streaming.BatchStatus
 
fromString(String) - Static method in enum org.apache.spark.status.api.v1.TaskSorting
 
fromString(String) - Static method in class org.apache.spark.storage.StorageLevel
Developer API Return the StorageLevel object with the specified name.
fromStructField(StructField) - Static method in class org.apache.spark.ml.attribute.AttributeGroup
Creates an attribute group from a StructField instance.
fullOuterJoin(JavaPairRDD<K, W>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a full outer join of this and other.
fullOuterJoin(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a full outer join of this and other.
fullOuterJoin(JavaPairRDD<K, W>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a full outer join of this and other.
fullOuterJoin(RDD<Tuple2<K, W>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a full outer join of this and other.
fullOuterJoin(RDD<Tuple2<K, W>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a full outer join of this and other.
fullOuterJoin(RDD<Tuple2<K, W>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a full outer join of this and other.
fullOuterJoin(JavaPairDStream<K, W>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'full outer join' between RDDs of this DStream and other DStream.
fullOuterJoin(JavaPairDStream<K, W>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'full outer join' between RDDs of this DStream and other DStream.
fullOuterJoin(JavaPairDStream<K, W>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'full outer join' between RDDs of this DStream and other DStream.
fullOuterJoin(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
fullOuterJoin(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
fullOuterJoin(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
fullOuterJoin(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
fullOuterJoin(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
fullOuterJoin(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
fullOuterJoin(DStream<Tuple2<K, W>>, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'full outer join' between RDDs of this DStream and other DStream.
fullOuterJoin(DStream<Tuple2<K, W>>, int, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'full outer join' between RDDs of this DStream and other DStream.
fullOuterJoin(DStream<Tuple2<K, W>>, Partitioner, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'full outer join' between RDDs of this DStream and other DStream.
fullStackTrace() - Method in class org.apache.spark.ExceptionFailure
 
Function<T1,R> - Interface in org.apache.spark.api.java.function
Base interface for functions whose return types do not create special RDDs.
Function - Class in org.apache.spark.sql.catalog
A user-defined function in Spark, as returned by listFunctions method in Catalog.
Function(String, String, String, String, boolean) - Constructor for class org.apache.spark.sql.catalog.Function
 
function(Function4<Time, KeyType, Option<ValueType>, State<StateType>, Option<MappedType>>) - Static method in class org.apache.spark.streaming.StateSpec
Create a StateSpec for setting all the specifications of the mapWithState operation on a pair DStream.
function(Function3<KeyType, Option<ValueType>, State<StateType>, MappedType>) - Static method in class org.apache.spark.streaming.StateSpec
Create a StateSpec for setting all the specifications of the mapWithState operation on a pair DStream.
function(Function4<Time, KeyType, Optional<ValueType>, State<StateType>, Optional<MappedType>>) - Static method in class org.apache.spark.streaming.StateSpec
Create a StateSpec for setting all the specifications of the mapWithState operation on a JavaPairDStream.
function(Function3<KeyType, Optional<ValueType>, State<StateType>, MappedType>) - Static method in class org.apache.spark.streaming.StateSpec
Create a StateSpec for setting all the specifications of the mapWithState operation on a JavaPairDStream.
Function0<R> - Interface in org.apache.spark.api.java.function
A zero-argument function that returns an R.
Function2<T1,T2,R> - Interface in org.apache.spark.api.java.function
A two-argument function that takes arguments of type T1 and T2 and returns an R.
Function3<T1,T2,T3,R> - Interface in org.apache.spark.api.java.function
A three-argument function that takes arguments of type T1, T2 and T3 and returns an R.
Function4<T1,T2,T3,T4,R> - Interface in org.apache.spark.api.java.function
A four-argument function that takes arguments of type T1, T2, T3 and T4 and returns an R.
functionExists(String) - Method in class org.apache.spark.sql.catalog.Catalog
Check if the function with the specified name exists.
functionExists(String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Check if the function with the specified name exists in the specified database.
functions - Class in org.apache.spark.sql
Functions available for DataFrame operations.
functions() - Constructor for class org.apache.spark.sql.functions
 
FutureAction<T> - Interface in org.apache.spark
A future for the result of an action to support cancellation.
futureExecutionContext() - Static method in class org.apache.spark.rdd.AsyncRDDActions
 
fwe() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
fwe() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
fwe() - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 

G

gain() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
gain() - Method in class org.apache.spark.ml.tree.InternalNode
 
gain() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
Gamma$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
 
gamma1() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
gamma2() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
gamma6() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
gamma7() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
GammaGenerator - Class in org.apache.spark.mllib.random
Developer API Generates i.i.d.
GammaGenerator(double, double) - Constructor for class org.apache.spark.mllib.random.GammaGenerator
 
gammaJavaRDD(JavaSparkContext, double, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.gammaRDD.
gammaJavaRDD(JavaSparkContext, double, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.gammaJavaRDD with the default seed.
gammaJavaRDD(JavaSparkContext, double, double, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.gammaJavaRDD with the default number of partitions and the default seed.
gammaJavaVectorRDD(JavaSparkContext, double, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.gammaVectorRDD.
gammaJavaVectorRDD(JavaSparkContext, double, double, long, int, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.gammaJavaVectorRDD with the default seed.
gammaJavaVectorRDD(JavaSparkContext, double, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.gammaJavaVectorRDD with the default number of partitions and the default seed.
gammaRDD(SparkContext, double, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD comprised of i.i.d. samples from the gamma distribution with the input shape and scale.
gammaVectorRDD(SparkContext, double, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD[Vector] with vectors containing i.i.d. samples drawn from the gamma distribution with the input shape and scale.
gapply(RelationalGroupedDataset, byte[], byte[], Object[], StructType) - Static method in class org.apache.spark.sql.api.r.SQLUtils
The helper function for gapply() on R side.
gaps() - Method in class org.apache.spark.ml.feature.RegexTokenizer
Indicates whether regex splits on gaps (true) or matches tokens (false).
GAUGE() - Static method in class org.apache.spark.metrics.sink.StatsdMetricType
 
Gaussian$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
GaussianMixture - Class in org.apache.spark.ml.clustering
Gaussian Mixture clustering.
GaussianMixture(String) - Constructor for class org.apache.spark.ml.clustering.GaussianMixture
 
GaussianMixture() - Constructor for class org.apache.spark.ml.clustering.GaussianMixture
 
GaussianMixture - Class in org.apache.spark.mllib.clustering
This class performs expectation maximization for multivariate Gaussian Mixture Models (GMMs).
GaussianMixture() - Constructor for class org.apache.spark.mllib.clustering.GaussianMixture
Constructs a default instance.
GaussianMixtureModel - Class in org.apache.spark.ml.clustering
Multivariate Gaussian Mixture Model (GMM) consisting of k Gaussians, where points are drawn from each Gaussian i with probability weights(i).
GaussianMixtureModel - Class in org.apache.spark.mllib.clustering
Multivariate Gaussian Mixture Model (GMM) consisting of k Gaussians, where points are drawn from each Gaussian i=1..k with probability w(i); mu(i) and sigma(i) are the respective mean and covariance for each Gaussian distribution i=1..k.
GaussianMixtureModel(double[], MultivariateGaussian[]) - Constructor for class org.apache.spark.mllib.clustering.GaussianMixtureModel
 
GaussianMixtureSummary - Class in org.apache.spark.ml.clustering
Experimental Summary of GaussianMixture.
gaussians() - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
gaussians() - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
 
gaussiansDF() - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
Retrieve Gaussian distributions as a DataFrame.
GBTClassificationModel - Class in org.apache.spark.ml.classification
Gradient-Boosted Trees (GBTs) (http://en.wikipedia.org/wiki/Gradient_boosting) model for classification.
GBTClassificationModel(String, DecisionTreeRegressionModel[], double[]) - Constructor for class org.apache.spark.ml.classification.GBTClassificationModel
Construct a GBTClassificationModel
GBTClassifier - Class in org.apache.spark.ml.classification
Gradient-Boosted Trees (GBTs) (http://en.wikipedia.org/wiki/Gradient_boosting) learning algorithm for classification.
GBTClassifier(String) - Constructor for class org.apache.spark.ml.classification.GBTClassifier
 
GBTClassifier() - Constructor for class org.apache.spark.ml.classification.GBTClassifier
 
GBTRegressionModel - Class in org.apache.spark.ml.regression
Gradient-Boosted Trees (GBTs) model for regression.
GBTRegressionModel(String, DecisionTreeRegressionModel[], double[]) - Constructor for class org.apache.spark.ml.regression.GBTRegressionModel
Construct a GBTRegressionModel
GBTRegressor - Class in org.apache.spark.ml.regression
Gradient-Boosted Trees (GBTs) learning algorithm for regression.
GBTRegressor(String) - Constructor for class org.apache.spark.ml.regression.GBTRegressor
 
GBTRegressor() - Constructor for class org.apache.spark.ml.regression.GBTRegressor
 
GC_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
GC_TIME() - Static method in class org.apache.spark.ui.ToolTips
 
gemm(double, Matrix, DenseMatrix, double, DenseMatrix) - Static method in class org.apache.spark.ml.linalg.BLAS
C := alpha * A * B + beta * C
gemm(double, Matrix, DenseMatrix, double, DenseMatrix) - Static method in class org.apache.spark.mllib.linalg.BLAS
C := alpha * A * B + beta * C
gemv(double, Matrix, Vector, double, DenseVector) - Static method in class org.apache.spark.ml.linalg.BLAS
y := alpha * A * x + beta * y
gemv(double, Matrix, Vector, double, DenseVector) - Static method in class org.apache.spark.mllib.linalg.BLAS
y := alpha * A * x + beta * y
GeneralizedLinearAlgorithm<M extends GeneralizedLinearModel> - Class in org.apache.spark.mllib.regression
Developer API GeneralizedLinearAlgorithm implements methods to train a Generalized Linear Model (GLM).
GeneralizedLinearAlgorithm() - Constructor for class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
 
GeneralizedLinearModel - Class in org.apache.spark.mllib.regression
Developer API GeneralizedLinearModel (GLM) represents a model trained using GeneralizedLinearAlgorithm.
GeneralizedLinearModel(Vector, double) - Constructor for class org.apache.spark.mllib.regression.GeneralizedLinearModel
 
GeneralizedLinearRegression - Class in org.apache.spark.ml.regression
Experimental
GeneralizedLinearRegression(String) - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
GeneralizedLinearRegression() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
GeneralizedLinearRegression.Binomial$ - Class in org.apache.spark.ml.regression
Binomial exponential family distribution.
GeneralizedLinearRegression.CLogLog$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Family$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.FamilyAndLink$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Gamma$ - Class in org.apache.spark.ml.regression
Gamma exponential family distribution.
GeneralizedLinearRegression.Gaussian$ - Class in org.apache.spark.ml.regression
Gaussian exponential family distribution.
GeneralizedLinearRegression.Identity$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Inverse$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Link$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Log$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Logit$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Poisson$ - Class in org.apache.spark.ml.regression
Poisson exponential family distribution.
GeneralizedLinearRegression.Probit$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Sqrt$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegression.Tweedie$ - Class in org.apache.spark.ml.regression
 
GeneralizedLinearRegressionModel - Class in org.apache.spark.ml.regression
Experimental Model produced by GeneralizedLinearRegression.
GeneralizedLinearRegressionSummary - Class in org.apache.spark.ml.regression
Experimental Summary of GeneralizedLinearRegression model and predictions.
GeneralizedLinearRegressionTrainingSummary - Class in org.apache.spark.ml.regression
Experimental Summary of GeneralizedLinearRegression fitting and model.
generateAssociationRules(double) - Method in class org.apache.spark.mllib.fpm.FPGrowthModel
Generates association rules for the Items in freqItemsets.
generateKMeansRDD(SparkContext, int, int, int, double, int) - Static method in class org.apache.spark.mllib.util.KMeansDataGenerator
Generate an RDD containing test data for KMeans.
generateLinearInput(double, double[], int, int, double) - Static method in class org.apache.spark.mllib.util.LinearDataGenerator
For compatibility, the generated data without specifying the mean and variance will have zero mean and variance of (1.0/3.0) since the original output range is [-1, 1] with uniform distribution, and the variance of uniform distribution is (b - a)^2^ / 12 which will be (1.0/3.0)
generateLinearInput(double, double[], double[], double[], int, int, double) - Static method in class org.apache.spark.mllib.util.LinearDataGenerator
 
generateLinearInput(double, double[], double[], double[], int, int, double, double) - Static method in class org.apache.spark.mllib.util.LinearDataGenerator
 
generateLinearInputAsList(double, double[], int, int, double) - Static method in class org.apache.spark.mllib.util.LinearDataGenerator
Return a Java List of synthetic data randomly generated according to a multi collinear model.
generateLinearRDD(SparkContext, int, int, double, int, double) - Static method in class org.apache.spark.mllib.util.LinearDataGenerator
Generate an RDD containing sample data for Linear Regression models - including Ridge, Lasso, and unregularized variants.
generateLogisticRDD(SparkContext, int, int, double, int, double) - Static method in class org.apache.spark.mllib.util.LogisticRegressionDataGenerator
Generate an RDD containing test data for LogisticRegression.
generateRandomEdges(int, int, int, long) - Static method in class org.apache.spark.graphx.util.GraphGenerators
 
generateTreeString(int, Seq<Object>, StringBuilder, boolean, String, boolean) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
generateTreeString(int, Seq<Object>, StringBuilder, boolean, String, boolean) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
generateTreeString(int, Seq<Object>, StringBuilder, boolean, String, boolean) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
generateTreeString(int, Seq<Object>, StringBuilder, boolean, String, boolean) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
generateTreeString$default$5() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
generateTreeString$default$5() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
generateTreeString$default$5() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
generateTreeString$default$5() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
generateTreeString$default$6() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
generateTreeString$default$6() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
generateTreeString$default$6() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
generateTreeString$default$6() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
genericBuilder() - Static method in class org.apache.spark.sql.types.StructType
 
geq(Object) - Method in class org.apache.spark.sql.Column
Greater than or equal to an expression.
get(Object) - Method in class org.apache.spark.api.java.JavaUtils.SerializableMapWrapper
 
get() - Method in class org.apache.spark.api.java.Optional
 
get() - Method in interface org.apache.spark.FutureAction
Blocks and returns the result of this job.
get(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.LDA
 
get(Param<T>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
get(Param<T>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
get(Param<T>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
get(Param<T>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.DCT
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.IDF
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Imputer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Interaction
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.NGram
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.PCA
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormula
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
get(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
get(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
get(Param<T>) - Method in class org.apache.spark.ml.param.ParamMap
Optionally returns the value associated with a param.
get(Param<T>) - Method in interface org.apache.spark.ml.param.Params
Optionally returns the user-supplied value of a param.
get(Param<T>) - Static method in class org.apache.spark.ml.Pipeline
 
get(Param<T>) - Static method in class org.apache.spark.ml.PipelineModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
get(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
get(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
get(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
get(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
get(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
get(String) - Method in class org.apache.spark.SparkConf
Get a parameter; throws a NoSuchElementException if it's not set
get(String, String) - Method in class org.apache.spark.SparkConf
Get a parameter, falling back to a default if not set
get() - Static method in class org.apache.spark.SparkEnv
Returns the SparkEnv.
get(String) - Static method in class org.apache.spark.SparkFiles
Get the absolute path of a file added through SparkContext.addFile().
get(String) - Static method in class org.apache.spark.sql.jdbc.JdbcDialects
Fetch the JdbcDialect class corresponding to a given database url.
get(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i.
get(String) - Method in class org.apache.spark.sql.RuntimeConfig
Returns the value of Spark runtime configuration property for the given key.
get(String, String) - Method in class org.apache.spark.sql.RuntimeConfig
Returns the value of Spark runtime configuration property for the given key.
get(String) - Method in class org.apache.spark.sql.sources.v2.DataSourceOptions
Returns the option value to which the specified key is mapped, case-insensitively.
get() - Method in interface org.apache.spark.sql.sources.v2.reader.DataReader
Return the current record.
get() - Method in interface org.apache.spark.sql.streaming.GroupState
Get the state value if it exists, or throw NoSuchElementException.
get(UUID) - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
Returns the query if there is an active query with the given id, or null.
get(String) - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
Returns the query if there is an active query with the given id, or null.
get(int, DataType) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
get(int, DataType) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
get() - Method in class org.apache.spark.streaming.State
Get the state if it exists, otherwise it will throw java.util.NoSuchElementException.
get() - Static method in class org.apache.spark.TaskContext
Return the currently active TaskContext.
get(long) - Static method in class org.apache.spark.util.AccumulatorContext
Returns the AccumulatorV2 registered with the given ID, if any.
get_json_object(Column, String) - Static method in class org.apache.spark.sql.functions
Extracts json object from a json string based on json path specified, and returns json string of the extracted json object.
getAcceptanceResults(RDD<Tuple2<K, V>>, boolean, Map<K, Object>, Option<Map<K, Object>>, long) - Static method in class org.apache.spark.util.random.StratifiedSamplingUtils
Count the number of items instantly accepted and generate the waitlist for each stratum.
getAcceptsNull() - Static method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
getActive() - Static method in class org.apache.spark.streaming.StreamingContext
Experimental
getActiveJobIds() - Method in class org.apache.spark.api.java.JavaSparkStatusTracker
Returns an array containing the ids of all active jobs.
getActiveJobIds() - Method in class org.apache.spark.SparkStatusTracker
Returns an array containing the ids of all active jobs.
getActiveOrCreate(Function0<StreamingContext>) - Static method in class org.apache.spark.streaming.StreamingContext
Experimental
getActiveOrCreate(String, Function0<StreamingContext>, Configuration, boolean) - Static method in class org.apache.spark.streaming.StreamingContext
Experimental
getActiveSession() - Static method in class org.apache.spark.sql.SparkSession
Returns the active SparkSession for the current thread, returned by the builder.
getActiveStageIds() - Method in class org.apache.spark.api.java.JavaSparkStatusTracker
Returns an array containing the ids of all active stages.
getActiveStageIds() - Method in class org.apache.spark.SparkStatusTracker
Returns an array containing the ids of all active stages.
getAggregationDepth() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getAggregationDepth() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getAggregationDepth() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getAggregationDepth() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getAggregationDepth() - Method in interface org.apache.spark.ml.param.shared.HasAggregationDepth
 
getAggregationDepth() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getAggregationDepth() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getAggregationDepth() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getAggregationDepth() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getAlgo() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getAll() - Method in class org.apache.spark.SparkConf
Get all parameters as a list of pairs
getAll() - Method in class org.apache.spark.sql.RuntimeConfig
Returns all properties set in this conf.
getAllConfs() - Method in class org.apache.spark.sql.SQLContext
Return all the configuration properties that have been set (i.e.
getAllPools() - Method in class org.apache.spark.SparkContext
Developer API Return pools for fair scheduler
getAllPrefLocs(RDD<?>) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer.PartitionLocations
 
GetAllReceiverInfo - Class in org.apache.spark.streaming.scheduler
 
GetAllReceiverInfo() - Constructor for class org.apache.spark.streaming.scheduler.GetAllReceiverInfo
 
getAllWithPrefix(String) - Method in class org.apache.spark.SparkConf
Get all parameters that start with prefix
getAlpha() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getAlpha() - Method in class org.apache.spark.mllib.clustering.LDA
Alias for getDocConcentration
getAnyValAs(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i.
getAppId() - Method in interface org.apache.spark.launcher.SparkAppHandle
Returns the application ID, or null if not yet known.
getAppId() - Method in class org.apache.spark.SparkConf
Returns the Spark application id, valid in the Driver after TaskScheduler registration and from the start in the Executor.
getArray(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getArray(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getArray(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getArray(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the array type value for rowId.
getAs(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i.
getAs(String) - Method in interface org.apache.spark.sql.Row
Returns the value of a given fieldName.
getAssociationRulesFromFP(Dataset<?>, String, String, double, ClassTag<T>) - Static method in class org.apache.spark.ml.fpm.AssociationRules
Computes the association rules with confidence above minConfidence.
getAsymmetricAlpha() - Method in class org.apache.spark.mllib.clustering.LDA
Alias for getAsymmetricDocConcentration
getAsymmetricDocConcentration() - Method in class org.apache.spark.mllib.clustering.LDA
Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
getAttr(String) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Gets an attribute by its name.
getAttr(int) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Gets an attribute by its index.
getAvroSchema() - Method in class org.apache.spark.SparkConf
Gets all the avro schemas in the configuration used in the generic Avro record serializer
getBatchingTimeout(SparkConf) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
How long we will wait for the wrappedLog in the BatchedWriteAheadLog to write the records before we fail the write attempt to unblock receivers.
getBernoulliSamplingFunction(RDD<Tuple2<K, V>>, Map<K, Object>, boolean, long) - Static method in class org.apache.spark.util.random.StratifiedSamplingUtils
Return the per partition sampling function used for sampling without replacement.
getBeta() - Method in class org.apache.spark.mllib.clustering.LDA
Alias for getTopicConcentration
getBinary() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getBinary() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getBinary() - Method in class org.apache.spark.ml.feature.HashingTF
 
getBinary(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getBinary(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getBinary(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getBinary(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the binary type value for rowId.
getBlock(BlockId) - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the given block stored in this block manager in O(1) time.
getBlockSize() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
GetBlockStatus(BlockId, boolean) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus
 
GetBlockStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus$
 
getBoolean(String, boolean) - Method in class org.apache.spark.SparkConf
Get a parameter as a boolean, falling back to a default if not set
getBoolean(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a primitive boolean.
getBoolean(String, boolean) - Method in class org.apache.spark.sql.sources.v2.DataSourceOptions
Returns the boolean value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
getBoolean(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Boolean.
getBoolean(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getBoolean(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getBoolean(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getBoolean(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the boolean type value for rowId.
getBooleanArray(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Boolean array.
getBooleans(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Gets boolean type values from [rowId, rowId + count).
getBucketLength() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getBucketLength() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
getByte(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a primitive byte.
getByte(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getByte(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getByte(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getByte(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the byte type value for rowId.
getBytes(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Gets byte type values from [rowId, rowId + count).
getCachedBlockManagerId(BlockManagerId) - Static method in class org.apache.spark.storage.BlockManagerId
 
getCachedMetadata(String) - Static method in class org.apache.spark.rdd.HadoopRDD
The three methods below are helpers for accessing the local map, a property of the SparkEnv of the local process.
getCacheNodeIds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getCacheNodeIds() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getCallSite(Function1<String, Object>) - Static method in class org.apache.spark.util.Utils
When called inside a class in the spark package, returns the name of the user code class (outside the spark package) that called into Spark, as well as which Spark method they called.
getCaseSensitive() - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
getCatalystType(int, String, int, MetadataBuilder) - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
getCatalystType(int, String, int, MetadataBuilder) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Get the custom datatype mapping for the given jdbc meta information.
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
getCatalystType(int, String, int, MetadataBuilder) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
getCategoricalCols() - Method in class org.apache.spark.ml.feature.FeatureHasher
 
getCategoricalFeatures(StructField) - Static method in class org.apache.spark.ml.util.MetadataUtils
Examine a schema to identify categorical (Binary and Nominal) features.
getCategoricalFeaturesInfo() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getCause() - Static method in exception org.apache.spark.sql.AnalysisException
 
getCensorCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getCensorCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getCheckpointDir() - Method in class org.apache.spark.api.java.JavaSparkContext
 
getCheckpointDir() - Method in class org.apache.spark.SparkContext
 
getCheckpointFile() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
getCheckpointFile() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
getCheckpointFile() - Static method in class org.apache.spark.api.java.JavaRDD
 
getCheckpointFile() - Method in interface org.apache.spark.api.java.JavaRDDLike
Gets the name of the file to which this RDD was checkpointed
getCheckpointFile() - Static method in class org.apache.spark.api.r.RRDD
 
getCheckpointFile() - Static method in class org.apache.spark.graphx.EdgeRDD
 
getCheckpointFile() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
getCheckpointFile() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
getCheckpointFile() - Static method in class org.apache.spark.graphx.VertexRDD
 
getCheckpointFile() - Static method in class org.apache.spark.rdd.HadoopRDD
 
getCheckpointFile() - Static method in class org.apache.spark.rdd.JdbcRDD
 
getCheckpointFile() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
getCheckpointFile() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
getCheckpointFile() - Method in class org.apache.spark.rdd.RDD
Gets the name of the directory to which this RDD was checkpointed.
getCheckpointFile() - Static method in class org.apache.spark.rdd.UnionRDD
 
getCheckpointFiles() - Method in class org.apache.spark.graphx.Graph
Gets the name of the files to which this Graph was checkpointed.
getCheckpointFiles() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
getCheckpointFiles() - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
Developer API
getCheckpointInterval() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.clustering.LDA
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getCheckpointInterval() - Method in interface org.apache.spark.ml.param.shared.HasCheckpointInterval
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getCheckpointInterval() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getCheckpointInterval() - Method in class org.apache.spark.mllib.clustering.LDA
Period (in iterations) between checkpoints.
getCheckpointInterval() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getChild(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getClassifier() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getClassifier() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getColdStartStrategy() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getColdStartStrategy() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
getCollectSubModels() - Method in interface org.apache.spark.ml.param.shared.HasCollectSubModels
 
getCollectSubModels() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getCollectSubModels() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getCombOp() - Static method in class org.apache.spark.util.random.StratifiedSamplingUtils
Returns the function used combine results returned by seqOp from different partitions.
getComment() - Method in class org.apache.spark.sql.types.StructField
Return the comment of this StructField.
getConf() - Method in class org.apache.spark.api.java.JavaSparkContext
Return a copy of this JavaSparkContext's configuration.
getConf() - Static method in class org.apache.spark.ml.image.SamplePathFilter
 
getConf() - Method in class org.apache.spark.rdd.HadoopRDD
 
getConf() - Method in class org.apache.spark.rdd.NewHadoopRDD
 
getConf() - Method in class org.apache.spark.SparkContext
Return a copy of this SparkContext's configuration.
getConf(String) - Method in class org.apache.spark.sql.SQLContext
Return the value of Spark SQL configuration property for the given key.
getConf(String, String) - Method in class org.apache.spark.sql.SQLContext
Return the value of Spark SQL configuration property for the given key.
getConfiguration() - Method in class org.apache.spark.input.PortableDataStream
 
getConfiguredLocalDirs(SparkConf) - Static method in class org.apache.spark.util.Utils
Return the configured local directories where Spark can write files.
getConnection() - Method in interface org.apache.spark.rdd.JdbcRDD.ConnectionFactory
 
getContextOrSparkClassLoader() - Static method in class org.apache.spark.util.Utils
Get the Context ClassLoader on this thread or, if not present, the ClassLoader that loaded Spark.
getConvergenceTol() - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Return the largest change in log-likelihood at which convergence is considered to have occurred.
getCorrelationFromName(String) - Static method in class org.apache.spark.mllib.stat.correlation.Correlations
 
getCount() - Method in class org.apache.spark.storage.CountingWritableChannel
 
getCurrentProcessingTimeMs() - Method in interface org.apache.spark.sql.streaming.GroupState
Get the current processing time as milliseconds in epoch time.
getCurrentUserGroups(SparkConf, String) - Static method in class org.apache.spark.util.Utils
 
getCurrentUserName() - Static method in class org.apache.spark.util.Utils
Returns the current user name.
getCurrentWatermarkMs() - Method in interface org.apache.spark.sql.streaming.GroupState
Get the current event time watermark as milliseconds in epoch time.
getData(Row) - Static method in class org.apache.spark.ml.image.ImageSchema
Gets the image data
getDatabase(String) - Method in class org.apache.spark.sql.catalog.Catalog
Get the database with the specified name.
getDate(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of date type as java.sql.Date.
getDecimal(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of decimal type as java.math.BigDecimal.
getDecimal(int, int, int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getDecimal(int, int, int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getDecimal(int, int, int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getDecimal(int, int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the decimal type value for rowId.
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.LDA
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.DCT
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IDF
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Imputer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Interaction
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.NGram
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PCA
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormula
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getDefault(Param<T>) - Method in interface org.apache.spark.ml.param.Params
Gets the default value of a parameter.
getDefault(Param<T>) - Static method in class org.apache.spark.ml.Pipeline
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.PipelineModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
getDefaultPropertiesFile(Map<String, String>) - Static method in class org.apache.spark.util.Utils
Return the path of the default Spark properties file.
getDefaultSession() - Static method in class org.apache.spark.sql.SparkSession
Returns the default SparkSession that is returned by the builder.
getDegree() - Method in class org.apache.spark.ml.feature.PolynomialExpansion
 
getDenseSizeInBytes() - Method in interface org.apache.spark.ml.linalg.Matrix
Gets the size of the dense representation of this `Matrix`.
getDependencies() - Method in class org.apache.spark.rdd.CoGroupedRDD
 
getDependencies() - Method in class org.apache.spark.rdd.ShuffledRDD
 
getDependencies() - Method in class org.apache.spark.rdd.UnionRDD
 
getDeprecatedConfig(String, Map<String, String>) - Static method in class org.apache.spark.SparkConf
Looks for available deprecated keys for the given config option, and return the first value available.
getDistributions() - Method in class org.apache.spark.status.LiveRDD
 
getDocConcentration() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getDocConcentration() - Static method in class org.apache.spark.ml.clustering.LDA
 
getDocConcentration() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getDocConcentration() - Method in class org.apache.spark.mllib.clustering.LDA
Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
getDouble(String, double) - Method in class org.apache.spark.SparkConf
Get a parameter as a double, falling back to a default if not set
getDouble(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a primitive double.
getDouble(String, double) - Method in class org.apache.spark.sql.sources.v2.DataSourceOptions
Returns the double value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
getDouble(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Double.
getDouble(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getDouble(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getDouble(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getDouble(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the double type value for rowId.
getDoubleArray(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Double array.
getDoubles(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Gets double type values from [rowId, rowId + count).
getDropLast() - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
getDropLast() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
getDropLast() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
getDynamicAllocationInitialExecutors(SparkConf) - Static method in class org.apache.spark.util.Utils
Return the initial number of executors for dynamic allocation.
getElasticNetParam() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getElasticNetParam() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getElasticNetParam() - Method in interface org.apache.spark.ml.param.shared.HasElasticNetParam
 
getElasticNetParam() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getElasticNetParam() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getEncryptionEnabled(JavaSparkContext) - Static method in class org.apache.spark.api.r.RUtils
 
getEndOffset() - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.MicroBatchReader
Return the specified (if explicitly set through setOffsetRange) or inferred end offset for this reader.
getEndTimeEpoch() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
getEpsilon() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getEpsilon() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getEpsilon() - Method in class org.apache.spark.mllib.clustering.KMeans
The distance threshold within which we've consider centers to have converged.
getEstimator() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getEstimator() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getEstimator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getEstimator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
getEstimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getEstimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getEstimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getEstimatorParamMaps() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
getEvaluator() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getEvaluator() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getEvaluator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getEvaluator() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
GetExecutorEndpointRef(String) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetExecutorEndpointRef
 
GetExecutorEndpointRef$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetExecutorEndpointRef$
 
getExecutorEnv() - Method in class org.apache.spark.SparkConf
Get all executor environment variables set on this SparkConf
getExecutorInfos() - Method in class org.apache.spark.SparkStatusTracker
Returns information of all known executors, including host, port, cacheSize, numRunningTasks.
GetExecutorLossReason(String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.GetExecutorLossReason
 
GetExecutorLossReason$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.GetExecutorLossReason$
 
getExecutorMemoryStatus() - Method in class org.apache.spark.SparkContext
Return a map from the slave to the max memory available for caching and the remaining memory available for caching.
getExecutorStorageStatus() - Method in class org.apache.spark.SparkContext
Deprecated.
This method may change or be removed in a future release. Since 2.2.0.
getFamily() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getFamily() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getFamily() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getFamily() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getFdr() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getFdr() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getFeatureIndex() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getFeatureIndex() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getFeatureIndicesFromNames(StructField, String[]) - Static method in class org.apache.spark.ml.util.MetadataUtils
Takes a Vector column and a list of feature names, and returns the corresponding list of feature indices in the column, in order.
getFeaturesAndLabels(RFormulaModel, Dataset<?>) - Static method in class org.apache.spark.ml.r.RWrapperUtils
Get the feature names and original labels from the schema of DataFrame transformed by RFormulaModel.
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.LDA
 
getFeaturesCol() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
getFeaturesCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getFeaturesCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.feature.RFormula
 
getFeaturesCol() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getFeaturesCol() - Method in interface org.apache.spark.ml.param.shared.HasFeaturesCol
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getFeaturesCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getFeatureSubsetStrategy() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getField(String) - Method in class org.apache.spark.sql.Column
An expression that gets a field by name in a StructType.
getFileLength(File, SparkConf) - Static method in class org.apache.spark.util.Utils
Return the file length, if the file is compressed it returns the uncompressed file length.
getFilePath(File, String) - Static method in class org.apache.spark.util.Utils
Return the absolute path of a file in the given directory.
getFileReader(String, Option<Configuration>, boolean) - Static method in class org.apache.spark.sql.hive.orc.OrcFileOperator
Retrieves an ORC file reader from a given path.
getFileSegmentLocations(String, long, long, Configuration) - Static method in class org.apache.spark.streaming.util.HdfsUtils
Get the locations of the HDFS blocks containing the given file segment.
getFileSystemForPath(Path, Configuration) - Static method in class org.apache.spark.streaming.util.HdfsUtils
 
getFinalStorageLevel() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getFinalValue() - Method in class org.apache.spark.partial.PartialResult
Blocking method to wait for and return the final value.
getFitIntercept() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getFitIntercept() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getFitIntercept() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getFitIntercept() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getFitIntercept() - Method in interface org.apache.spark.ml.param.shared.HasFitIntercept
 
getFitIntercept() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getFitIntercept() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getFitIntercept() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getFitIntercept() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getFitIntercept() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getFitIntercept() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getFloat(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a primitive float.
getFloat(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getFloat(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getFloat(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getFloat(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the float type value for rowId.
getFloats(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Gets float type values from [rowId, rowId + count).
getForceIndexLabel() - Static method in class org.apache.spark.ml.feature.RFormula
 
getForceIndexLabel() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getFormattedClassName(Object) - Static method in class org.apache.spark.util.Utils
Return the class name of the given object, removing all dollar signs
getFormula() - Static method in class org.apache.spark.ml.feature.RFormula
 
getFormula() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getFpr() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getFpr() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getFunction(String) - Method in class org.apache.spark.sql.catalog.Catalog
Get the function with the specified name.
getFunction(String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Get the function with the specified name.
getFwe() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getFwe() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getGaps() - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
getGroups(String) - Method in interface org.apache.spark.security.GroupMappingServiceProvider
Get the groups the user belongs to.
getHadoopFileSystem(URI, Configuration) - Static method in class org.apache.spark.util.Utils
Return a Hadoop FileSystem with the scheme encoded in the given path.
getHadoopFileSystem(String, Configuration) - Static method in class org.apache.spark.util.Utils
Return a Hadoop FileSystem with the scheme encoded in the given path.
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.RFormula
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
getHandleInvalid() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
getHandleInvalid() - Method in interface org.apache.spark.ml.param.shared.HasHandleInvalid
 
getHeight(Row) - Static method in class org.apache.spark.ml.image.ImageSchema
Gets the height of the image
getHiveWriteCompression(TableDesc, SQLConf) - Static method in class org.apache.spark.sql.hive.execution.HiveOptions
 
getImplicitPrefs() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getImpurity() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getImpurity() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getImpurity() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getImpurity() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getImpurity() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getImpurity() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getImpurity() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getImpurity() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getImpurity() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getImpurity() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getImpurity() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getImpurity() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getImpurity() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getIndices() - Method in class org.apache.spark.ml.feature.VectorSlicer
 
getInitializationMode() - Method in class org.apache.spark.mllib.clustering.KMeans
The initialization algorithm.
getInitializationSteps() - Method in class org.apache.spark.mllib.clustering.KMeans
Number of steps for the k-means|| initialization mode
getInitialModel() - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Return the user supplied initial GMM, if supplied
getInitialPositionInStream(int) - Method in class org.apache.spark.streaming.kinesis.KinesisUtilsPythonHelper
 
getInitialTargetExecutorNumber(SparkConf, int) - Static method in class org.apache.spark.scheduler.cluster.SchedulerBackendUtils
Getting the initial target number of executors depends on whether dynamic allocation is enabled.
getInitialWeights() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getInitMode() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getInitMode() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getInitSteps() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getInitSteps() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.Binarizer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getInputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.DCT
 
getInputCol() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
getInputCol() - Static method in class org.apache.spark.ml.feature.HashingTF
 
getInputCol() - Static method in class org.apache.spark.ml.feature.IDF
 
getInputCol() - Static method in class org.apache.spark.ml.feature.IDFModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.IndexToString
 
getInputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
getInputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
getInputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
getInputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.NGram
 
getInputCol() - Static method in class org.apache.spark.ml.feature.Normalizer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
getInputCol() - Static method in class org.apache.spark.ml.feature.PCA
 
getInputCol() - Static method in class org.apache.spark.ml.feature.PCAModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
getInputCol() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
getInputCol() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
getInputCol() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
getInputCol() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
getInputCol() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
getInputCol() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getInputCol() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getInputCol() - Method in interface org.apache.spark.ml.param.shared.HasInputCol
 
getInputCols() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getInputCols() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
getInputCols() - Static method in class org.apache.spark.ml.feature.Imputer
 
getInputCols() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
getInputCols() - Static method in class org.apache.spark.ml.feature.Interaction
 
getInputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
getInputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
getInputCols() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getInputCols() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
getInputCols() - Method in interface org.apache.spark.ml.param.shared.HasInputCols
 
getInputFilePath() - Static method in class org.apache.spark.rdd.InputFileBlockHolder
Returns the holding file name or empty string if it is unknown.
getInputStream(String, Configuration) - Static method in class org.apache.spark.streaming.util.HdfsUtils
 
getInt(String, int) - Method in class org.apache.spark.SparkConf
Get a parameter as an integer, falling back to a default if not set
getInt(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a primitive int.
getInt(String, int) - Method in class org.apache.spark.sql.sources.v2.DataSourceOptions
Returns the integer value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
getInt(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getInt(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getInt(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getInt(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the int type value for rowId.
getIntermediateStorageLevel() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getInterval(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getInterval(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getInterval(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the calendar interval type value for rowId.
getInts(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Gets int type values from [rowId, rowId + count).
getInverse() - Method in class org.apache.spark.ml.feature.DCT
 
getIsotonic() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getIsotonic() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getItem(Object) - Method in class org.apache.spark.sql.Column
An expression that gets an item at position ordinal out of an array, or gets a value by key key in a MapType.
getItemCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getItemCol() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
getItemsCol() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getItemsCol() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getIteratorSize(Iterator<T>) - Static method in class org.apache.spark.util.Utils
Counts the number of elements of an iterator using a while loop rather than calling TraversableOnce.size() because it uses a for loop, which is slightly slower in the current version of Scala.
getIteratorZipWithIndex(Iterator<T>, long) - Static method in class org.apache.spark.util.Utils
Generate a zipWithIndex iterator, avoid index value overflowing problem in scala's zipWithIndex
getJavaMap(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of array type as a java.util.Map.
getJavaSparkContext(SparkSession) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
getJDBCType(DataType) - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
getJDBCType(DataType) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Retrieve the jdbc / sql type for a given datatype.
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
getJDBCType(DataType) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
getJobIdsForGroup(String) - Method in class org.apache.spark.api.java.JavaSparkStatusTracker
Return a list of all known jobs in a particular job group.
getJobIdsForGroup(String) - Method in class org.apache.spark.SparkStatusTracker
Return a list of all known jobs in a particular job group.
getJobInfo(int) - Method in class org.apache.spark.api.java.JavaSparkStatusTracker
Returns job information, or null if the job info could not be found or was garbage collected.
getJobInfo(int) - Method in class org.apache.spark.SparkStatusTracker
Returns job information, or None if the job info could not be found or was garbage collected.
getK() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getK() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getK() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getK() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getK() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getK() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getK() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getK() - Static method in class org.apache.spark.ml.clustering.LDA
 
getK() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getK() - Static method in class org.apache.spark.ml.feature.PCA
 
getK() - Static method in class org.apache.spark.ml.feature.PCAModel
 
getK() - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Gets the desired number of leaf clusters.
getK() - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Return the number of Gaussians in the mixture model
getK() - Method in class org.apache.spark.mllib.clustering.KMeans
Number of clusters to create (k).
getK() - Method in class org.apache.spark.mllib.clustering.LDA
Number of topics to infer, i.e., the number of soft cluster centers.
getKappa() - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
Learning rate: exponential decay rate
getKeepLastCheckpoint() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getKeepLastCheckpoint() - Static method in class org.apache.spark.ml.clustering.LDA
 
getKeepLastCheckpoint() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getKeepLastCheckpoint() - Method in class org.apache.spark.mllib.clustering.EMLDAOptimizer
If using checkpointing, this indicates whether to keep the last checkpoint (vs clean up).
getLabelCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getLabelCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getLabelCol() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
getLabelCol() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
getLabelCol() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
getLabelCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getLabelCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getLabelCol() - Static method in class org.apache.spark.ml.feature.RFormula
 
getLabelCol() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getLabelCol() - Method in interface org.apache.spark.ml.param.shared.HasLabelCol
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getLabelCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getLabels() - Method in class org.apache.spark.ml.feature.IndexToString
 
getLambda() - Method in class org.apache.spark.mllib.classification.NaiveBayes
Get the smoothing parameter.
getLastUpdatedEpoch() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
getLayers() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getLDAModel(double[]) - Method in interface org.apache.spark.mllib.clustering.LDAOptimizer
 
getLearningDecay() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getLearningDecay() - Static method in class org.apache.spark.ml.clustering.LDA
 
getLearningDecay() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getLearningOffset() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getLearningOffset() - Static method in class org.apache.spark.ml.clustering.LDA
 
getLearningOffset() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getLearningRate() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
getLeastGroupHash(String) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
Sorts and gets the least element of the list associated with key in groupHash The returned PartitionGroup is the least loaded of all groups that represent the machine "key"
getLength() - Static method in class org.apache.spark.rdd.InputFileBlockHolder
Returns the length of the block being read, or -1 if it is unknown.
getLink() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getLink() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getLinkPower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getLinkPower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getLinkPredictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getLinkPredictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getList(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of array type as java.util.List.
getLocalDir(SparkConf) - Static method in class org.apache.spark.util.Utils
Get the path of a temporary directory.
getLocalizedMessage() - Static method in exception org.apache.spark.sql.AnalysisException
 
getLocalProperty(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Get a local property set in this thread, or null if it is missing.
getLocalProperty(String) - Method in class org.apache.spark.SparkContext
Get a local property set in this thread, or null if it is missing.
getLocalProperty(String) - Method in class org.apache.spark.TaskContext
Get a local property set upstream in the driver, or null if it is missing.
getLocalUserJarsForShell(SparkConf) - Static method in class org.apache.spark.util.Utils
Return the local jar files which will be added to REPL's classpath.
GetLocations(BlockId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocations
 
GetLocations$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocations$
 
GetLocationsAndStatus(BlockId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocationsAndStatus
 
GetLocationsAndStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocationsAndStatus$
 
GetLocationsMultipleBlockIds(BlockId[]) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocationsMultipleBlockIds
 
GetLocationsMultipleBlockIds$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetLocationsMultipleBlockIds$
 
getLong(String, long) - Method in class org.apache.spark.SparkConf
Get a parameter as a long, falling back to a default if not set
getLong(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a primitive long.
getLong(String, long) - Method in class org.apache.spark.sql.sources.v2.DataSourceOptions
Returns the long value to which the specified key is mapped, or defaultValue if there is no mapping for the key.
getLong(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Long.
getLong(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getLong(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getLong(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getLong(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the long type value for rowId.
getLongArray(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Long array.
getLongs(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Gets long type values from [rowId, rowId + count).
getLoss() - Method in interface org.apache.spark.ml.param.shared.HasLoss
 
getLoss() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getLoss() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getLoss() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
getLossType() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getLossType() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getLossType() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getLossType() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getLowerBound(double, long, double) - Static method in class org.apache.spark.util.random.BinomialBounds
Returns a threshold p such that if we conduct n Bernoulli trials with success rate = p, it is very unlikely to have more than fraction * n successes.
getLowerBound(double) - Static method in class org.apache.spark.util.random.PoissonBounds
Returns a lambda such that Pr[X > s] is very small, where X ~ Pois(lambda).
getLowerBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getLowerBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getLowerBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getLowerBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getMap(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of map type as a Scala Map.
getMap(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getMap(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getMap(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getMap(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the map type value for rowId.
GetMatchingBlockIds(Function1<BlockId, Object>, boolean) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds
 
GetMatchingBlockIds$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds$
 
getMax() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
getMax() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
getMaxBins() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getMaxBins() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getMaxBins() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getMaxBins() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getMaxBins() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getMaxBins() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getMaxBins() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getMaxBins() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getMaxBins() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getMaxBins() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getMaxBins() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getMaxBins() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getMaxBins() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getMaxCategories() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
getMaxCategories() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
getMaxDepth() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getMaxDepth() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getMaxDepth() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getMaxDepth() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getMaxDepth() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getMaxDepth() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getMaxDepth() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getMaxDepth() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getMaxDepth() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getMaxDepth() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getMaxDepth() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getMaxDepth() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getMaxDepth() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getMaxFailures(SparkConf, boolean) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
getMaxIter() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getMaxIter() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getMaxIter() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getMaxIter() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getMaxIter() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getMaxIter() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getMaxIter() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.LDA
 
getMaxIter() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getMaxIter() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getMaxIter() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getMaxIter() - Method in interface org.apache.spark.ml.param.shared.HasMaxIter
 
getMaxIter() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getMaxIter() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getMaxIterations() - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Gets the max number of k-means iterations to split clusters.
getMaxIterations() - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Return the maximum number of iterations allowed
getMaxIterations() - Method in class org.apache.spark.mllib.clustering.KMeans
Maximum number of iterations allowed.
getMaxIterations() - Method in class org.apache.spark.mllib.clustering.LDA
Maximum number of iterations allowed.
getMaxLocalProjDBSize() - Method in class org.apache.spark.mllib.fpm.PrefixSpan
Gets the maximum number of items allowed in a projected database before local processing.
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getMaxMemoryInMB() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getMaxMemoryInMB() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getMaxPatternLength() - Method in class org.apache.spark.mllib.fpm.PrefixSpan
Gets the maximal pattern length (i.e.
getMaxResultSize(SparkConf) - Static method in class org.apache.spark.util.Utils
 
getMaxSentenceLength() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getMaxSentenceLength() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
GetMemoryStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetMemoryStatus$
 
getMessage() - Method in exception org.apache.spark.sql.AnalysisException
 
getMetadata(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Metadata.
getMetadataArray(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a Metadata array.
getMetricName() - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
getMetricName() - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
getMetricName() - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
getMetricName() - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
getMetricsSources(String) - Method in class org.apache.spark.TaskContext
::DeveloperApi:: Returns all metrics sources with the given name which are associated with the instance which runs the task.
getMin() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
getMin() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
getMinConfidence() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getMinConfidence() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getMinCount() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getMinCount() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getMinDF() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getMinDF() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getMinDivisibleClusterSize() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getMinDivisibleClusterSize() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getMinDivisibleClusterSize() - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Gets the minimum number of points (if greater than or equal to 1.0) or the minimum proportion of points (if less than 1.0) of a divisible cluster.
getMinDocFreq() - Static method in class org.apache.spark.ml.feature.IDF
 
getMinDocFreq() - Static method in class org.apache.spark.ml.feature.IDFModel
 
getMiniBatchFraction() - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
Mini-batch fraction, which sets the fraction of document sampled and used in each iteration
getMinInfoGain() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getMinInfoGain() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getMinInfoGain() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getMinInfoGain() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getMinInfoGain() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getMinInfoGain() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getMinInfoGain() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getMinInfoGain() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getMinInfoGain() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getMinInfoGain() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getMinInfoGain() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getMinInfoGain() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getMinInfoGain() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getMinInstancesPerNode() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getMinInstancesPerNode() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getMinSupport() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getMinSupport() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getMinSupport() - Method in class org.apache.spark.mllib.fpm.PrefixSpan
Get the minimal support (i.e.
getMinTF() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getMinTF() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getMinTokenLength() - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
getMissingValue() - Static method in class org.apache.spark.ml.feature.Imputer
 
getMissingValue() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
getMode(Row) - Static method in class org.apache.spark.ml.image.ImageSchema
Gets the OpenCV representation as an int
getModelType() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getModelType() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getModelType() - Method in class org.apache.spark.mllib.classification.NaiveBayes
Get the model type.
getN() - Method in class org.apache.spark.ml.feature.NGram
 
getNames() - Method in class org.apache.spark.ml.feature.VectorSlicer
 
getNChannels(Row) - Static method in class org.apache.spark.ml.image.ImageSchema
Gets the number of channels in the image
getNode(int, Node) - Static method in class org.apache.spark.mllib.tree.model.Node
Traces down from a root node to get the node with the given node index.
getNonnegative() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getNumBuckets() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getNumBucketsArray() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getNumClasses(StructField) - Static method in class org.apache.spark.ml.util.MetadataUtils
Examine a schema to identify the number of classes in a label column.
getNumClasses() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getNumFeatures() - Method in class org.apache.spark.ml.feature.FeatureHasher
 
getNumFeatures() - Method in class org.apache.spark.ml.feature.HashingTF
 
getNumFeatures() - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
 
getNumFeatures() - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
 
getNumFeatures() - Method in class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
The dimension of training features.
getNumFeatures() - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
 
getNumFeatures() - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
 
getNumFeatures() - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
 
getNumFolds() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getNumFolds() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getNumHashTables() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getNumHashTables() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
getNumHashTables() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
getNumHashTables() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
getNumItemBlocks() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getNumIterations() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
getNumObjFields() - Method in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
getNumPartitions() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
getNumPartitions() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
getNumPartitions() - Static method in class org.apache.spark.api.java.JavaRDD
 
getNumPartitions() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return the number of partitions in this RDD.
getNumPartitions() - Static method in class org.apache.spark.api.r.RRDD
 
getNumPartitions() - Static method in class org.apache.spark.graphx.EdgeRDD
 
getNumPartitions() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
getNumPartitions() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
getNumPartitions() - Static method in class org.apache.spark.graphx.VertexRDD
 
getNumPartitions() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getNumPartitions() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getNumPartitions() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getNumPartitions() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getNumPartitions() - Static method in class org.apache.spark.rdd.HadoopRDD
 
getNumPartitions() - Static method in class org.apache.spark.rdd.JdbcRDD
 
getNumPartitions() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
getNumPartitions() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
getNumPartitions() - Method in class org.apache.spark.rdd.RDD
Returns the number of partitions of this RDD.
getNumPartitions() - Static method in class org.apache.spark.rdd.UnionRDD
 
getNumTopFeatures() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getNumTopFeatures() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getNumTrees() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
Number of trees in ensemble
getNumTrees() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getNumTrees() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getNumTrees() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
Number of trees in ensemble
getNumTrees() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getNumTrees() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getNumUserBlocks() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getNumValues() - Method in class org.apache.spark.ml.attribute.NominalAttribute
Get the number of values, either from numValues or from values.
getObjectInspector(String, Option<Configuration>) - Static method in class org.apache.spark.sql.hive.orc.OrcFileOperator
 
getObjFieldValues(Object, Object[]) - Method in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
getOffset() - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.ContinuousDataReader
Get the offset of the current record, or the start offset if no records have been read.
getOffsetCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getOffsetCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getOptimizeDocConcentration() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getOptimizeDocConcentration() - Static method in class org.apache.spark.ml.clustering.LDA
 
getOptimizeDocConcentration() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getOptimizeDocConcentration() - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
Optimize docConcentration, indicates whether docConcentration (Dirichlet parameter for document-topic distribution) will be optimized during training.
getOptimizer() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getOptimizer() - Static method in class org.apache.spark.ml.clustering.LDA
 
getOptimizer() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getOptimizer() - Method in class org.apache.spark.mllib.clustering.LDA
Developer API
getOption(String) - Method in class org.apache.spark.SparkConf
Get a parameter as an Option
getOption(String) - Method in class org.apache.spark.sql.RuntimeConfig
Returns the value of Spark runtime configuration property for the given key.
getOption() - Method in interface org.apache.spark.sql.streaming.GroupState
Get the state value as a scala Option.
getOption() - Method in class org.apache.spark.streaming.State
Get the state as a scala.Option.
getOrCreate(SparkConf) - Static method in class org.apache.spark.SparkContext
This function may be used to get or instantiate a SparkContext and register it as a singleton object.
getOrCreate() - Static method in class org.apache.spark.SparkContext
This function may be used to get or instantiate a SparkContext and register it as a singleton object.
getOrCreate() - Method in class org.apache.spark.sql.SparkSession.Builder
Gets an existing SparkSession or, if there is no existing one, creates a new one based on the options set in this builder.
getOrCreate(SparkContext) - Static method in class org.apache.spark.sql.SQLContext
Deprecated.
Use SparkSession.builder instead. Since 2.0.0.
getOrCreate(String, Function0<JavaStreamingContext>) - Static method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Either recreate a StreamingContext from checkpoint data or create a new StreamingContext.
getOrCreate(String, Function0<JavaStreamingContext>, Configuration) - Static method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Either recreate a StreamingContext from checkpoint data or create a new StreamingContext.
getOrCreate(String, Function0<JavaStreamingContext>, Configuration, boolean) - Static method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Either recreate a StreamingContext from checkpoint data or create a new StreamingContext.
getOrCreate(String, Function0<StreamingContext>, Configuration, boolean) - Static method in class org.apache.spark.streaming.StreamingContext
Either recreate a StreamingContext from checkpoint data or create a new StreamingContext.
getOrCreateSparkSession(JavaSparkContext, Map<Object, Object>, boolean) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.LDA
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.DCT
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IDF
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Imputer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Interaction
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.NGram
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PCA
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormula
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getOrDefault(Param<T>) - Method in interface org.apache.spark.ml.param.Params
Gets the value of a param in the embedded param map or its default value.
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.Pipeline
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.PipelineModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getOrDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
getOrElse(Param<T>, T) - Method in class org.apache.spark.ml.param.ParamMap
Returns the value associated with a param or a default value.
getOrigin(Row) - Static method in class org.apache.spark.ml.image.ImageSchema
Gets the origin of the image
getOutputAttrGroupFromData(Dataset<?>, Seq<String>, Seq<String>, boolean) - Static method in class org.apache.spark.ml.feature.OneHotEncoderCommon
This method is called when we want to generate AttributeGroup from actual data for one-hot encoder.
getOutputCol() - Static method in class org.apache.spark.ml.feature.Binarizer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.DCT
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.HashingTF
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.IDF
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.IDFModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.IndexToString
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.Interaction
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.NGram
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.Normalizer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.PCA
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.PCAModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getOutputCol() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getOutputCol() - Method in interface org.apache.spark.ml.param.shared.HasOutputCol
 
getOutputCols() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getOutputCols() - Static method in class org.apache.spark.ml.feature.Imputer
 
getOutputCols() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
getOutputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
getOutputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
getOutputCols() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getOutputCols() - Method in interface org.apache.spark.ml.param.shared.HasOutputCols
 
getOutputStream(String, Configuration) - Static method in class org.apache.spark.streaming.util.HdfsUtils
 
getP() - Method in class org.apache.spark.ml.feature.Normalizer
 
getParallelism() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getParallelism() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getParallelism() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getParam(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getParam(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getParam(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getParam(String) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getParam(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getParam(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getParam(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getParam(String) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getParam(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.KMeans
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.LDA
 
getParam(String) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getParam(String) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
getParam(String) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
getParam(String) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
getParam(String) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Binarizer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getParam(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getParam(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
getParam(String) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.DCT
 
getParam(String) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
getParam(String) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
getParam(String) - Static method in class org.apache.spark.ml.feature.HashingTF
 
getParam(String) - Static method in class org.apache.spark.ml.feature.IDF
 
getParam(String) - Static method in class org.apache.spark.ml.feature.IDFModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Imputer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.IndexToString
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Interaction
 
getParam(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
getParam(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
getParam(String) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
getParam(String) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.NGram
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Normalizer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
getParam(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
getParam(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.PCA
 
getParam(String) - Static method in class org.apache.spark.ml.feature.PCAModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
getParam(String) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.RFormula
 
getParam(String) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
getParam(String) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
getParam(String) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
getParam(String) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
getParam(String) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
getParam(String) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
getParam(String) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getParam(String) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getParam(String) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getParam(String) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getParam(String) - Method in interface org.apache.spark.ml.param.Params
Gets a param by its name.
getParam(String) - Static method in class org.apache.spark.ml.Pipeline
 
getParam(String) - Static method in class org.apache.spark.ml.PipelineModel
 
getParam(String) - Static method in class org.apache.spark.ml.recommendation.ALS
 
getParam(String) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getParam(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getParam(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getParam(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getParam(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getParam(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getParam(String) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getParam(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getParam(String) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getParam(String) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getParam(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getParam(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
getParents(int) - Method in class org.apache.spark.NarrowDependency
Get the parent partitions for a child partition.
getParents(int) - Method in class org.apache.spark.OneToOneDependency
 
getParents(int) - Method in class org.apache.spark.RangeDependency
 
getPartition(long, long, int) - Method in class org.apache.spark.graphx.PartitionStrategy.CanonicalRandomVertexCut$
 
getPartition(long, long, int) - Method in class org.apache.spark.graphx.PartitionStrategy.EdgePartition1D$
 
getPartition(long, long, int) - Method in class org.apache.spark.graphx.PartitionStrategy.EdgePartition2D$
 
getPartition(long, long, int) - Method in interface org.apache.spark.graphx.PartitionStrategy
Returns the partition number for a given edge.
getPartition(long, long, int) - Method in class org.apache.spark.graphx.PartitionStrategy.RandomVertexCut$
 
getPartition(Object) - Method in class org.apache.spark.HashPartitioner
 
getPartition(Object) - Method in class org.apache.spark.Partitioner
 
getPartition(Object) - Method in class org.apache.spark.RangePartitioner
 
getPartitionId() - Static method in class org.apache.spark.TaskContext
Returns the partition id of currently active TaskContext.
getPartitions() - Method in class org.apache.spark.api.r.BaseRRDD
 
getPartitions() - Static method in class org.apache.spark.api.r.RRDD
 
getPartitions() - Method in class org.apache.spark.rdd.CoGroupedRDD
 
getPartitions() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
getPartitions() - Method in class org.apache.spark.rdd.HadoopRDD
 
getPartitions() - Method in class org.apache.spark.rdd.JdbcRDD
 
getPartitions() - Method in class org.apache.spark.rdd.NewHadoopRDD
 
getPartitions() - Method in class org.apache.spark.rdd.ShuffledRDD
 
getPartitions() - Method in class org.apache.spark.rdd.UnionRDD
 
getPartitions() - Method in class org.apache.spark.status.LiveRDD
 
getPath() - Method in class org.apache.spark.input.PortableDataStream
 
getPattern() - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
GetPeers(BlockManagerId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetPeers
 
GetPeers$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetPeers$
 
getPercentile() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getPercentile() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getPersistentRDDs() - Method in class org.apache.spark.api.java.JavaSparkContext
Returns a Java map of JavaRDDs that have marked themselves as persistent via cache() call.
getPersistentRDDs() - Method in class org.apache.spark.SparkContext
Returns an immutable map of RDDs that have marked themselves as persistent via cache() call.
getPoissonSamplingFunction(RDD<Tuple2<K, V>>, Map<K, Object>, boolean, long, ClassTag<K>, ClassTag<V>) - Static method in class org.apache.spark.util.random.StratifiedSamplingUtils
Return the per partition sampling function used for sampling with replacement.
getPoolForName(String) - Method in class org.apache.spark.SparkContext
Developer API Return the pool associated with the given name, if one exists
getPosition() - Method in class org.apache.spark.streaming.kinesis.KinesisInitialPositions.AtTimestamp
 
getPosition() - Method in class org.apache.spark.streaming.kinesis.KinesisInitialPositions.Latest
 
getPosition() - Method in class org.apache.spark.streaming.kinesis.KinesisInitialPositions.TrimHorizon
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getPredictionCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getPredictionCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getPredictionCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getPredictionCol() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
getPredictionCol() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
getPredictionCol() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
getPredictionCol() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
getPredictionCol() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
getPredictionCol() - Method in interface org.apache.spark.ml.param.shared.HasPredictionCol
 
getPredictionCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getPredictionCol() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getPredictionCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getPreferredLocations(Partition) - Method in class org.apache.spark.rdd.HadoopRDD
 
getPreferredLocations(Partition) - Method in class org.apache.spark.rdd.NewHadoopRDD
 
getPreferredLocations(Partition) - Method in class org.apache.spark.rdd.UnionRDD
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getProbabilityCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getProbabilityCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getProbabilityCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getProbabilityCol() - Method in interface org.apache.spark.ml.param.shared.HasProbabilityCol
 
getProcessName() - Static method in class org.apache.spark.util.Utils
Returns the name of this JVM process.
getPropertiesFromFile(String) - Static method in class org.apache.spark.util.Utils
Load properties present in the given file.
getQuantileCalculationStrategy() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getQuantileProbabilities() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getQuantileProbabilities() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getQuantilesCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getQuantilesCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getRandomSample(Seq<T>, int, Random) - Static method in class org.apache.spark.storage.BlockReplicationUtils
Get a random sample of size m from the elems
getRank() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getRatingCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getRawPredictionCol() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
getRawPredictionCol() - Method in interface org.apache.spark.ml.param.shared.HasRawPredictionCol
 
getRddBlockLocations(int, Seq<StorageStatus>) - Static method in class org.apache.spark.storage.StorageUtils
Return a mapping from block ID to its locations for each block that belongs to the given RDD.
getRDDStorageInfo() - Method in class org.apache.spark.SparkContext
Developer API Return information about what RDDs are cached, if they are in mem or on disk, how much space they take, etc.
getReceiver() - Method in class org.apache.spark.streaming.dstream.ReceiverInputDStream
Gets the receiver object that will be sent to the worker nodes to receive data.
getRegParam() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getRegParam() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getRegParam() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getRegParam() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getRegParam() - Method in interface org.apache.spark.ml.param.shared.HasRegParam
 
getRegParam() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getRegParam() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getRegParam() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getRegParam() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getRegParam() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getRelativeError() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
getResource(String) - Method in class org.apache.spark.util.ChildFirstURLClassLoader
 
getResources(String) - Method in class org.apache.spark.util.ChildFirstURLClassLoader
 
getRollingIntervalSecs(SparkConf, boolean) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
getRootDirectory() - Static method in class org.apache.spark.SparkFiles
Get the root directory that contains files added through SparkContext.addFile().
getRow(int) - Method in class org.apache.spark.sql.vectorized.ColumnarBatch
Returns the row in this batch at `rowId`.
getRuns() - Method in class org.apache.spark.mllib.clustering.KMeans
Deprecated.
This has no effect and always returns 1. Since 2.1.0.
getScalingVec() - Method in class org.apache.spark.ml.feature.ElementwiseProduct
 
getSchedulingMode() - Method in class org.apache.spark.SparkContext
Return current scheduling mode
getSchemaQuery(String) - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
getSchemaQuery(String) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
The SQL query that should be used to discover the schema of a table.
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
getSchemaQuery(String) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
getSeed() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getSeed() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getSeed() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getSeed() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getSeed() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getSeed() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getSeed() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getSeed() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
getSeed() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
getSeed() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getSeed() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getSeed() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getSeed() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getSeed() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getSeed() - Static method in class org.apache.spark.ml.clustering.LDA
 
getSeed() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getSeed() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
getSeed() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
getSeed() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getSeed() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getSeed() - Method in interface org.apache.spark.ml.param.shared.HasSeed
 
getSeed() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getSeed() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getSeed() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getSeed() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getSeed() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getSeed() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getSeed() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getSeed() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
getSeed() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
getSeed() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getSeed() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
getSeed() - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Gets the random seed.
getSeed() - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Return the random seed
getSeed() - Method in class org.apache.spark.mllib.clustering.KMeans
The random seed for cluster initialization.
getSeed() - Method in class org.apache.spark.mllib.clustering.LDA
Random seed for cluster initialization.
getSelectorType() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
getSelectorType() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
getSeq(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of array type as a Scala Seq.
getSeqOp(boolean, Map<K, Object>, StratifiedSamplingUtils.RandomDataGenerator, Option<Map<K, Object>>) - Static method in class org.apache.spark.util.random.StratifiedSamplingUtils
Returns the function used by aggregate to collect sampling statistics for each partition.
getSessionConf(SparkSession) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
getShort(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a primitive short.
getShort(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getShort(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getShort(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getShort(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the short type value for rowId.
getShorts(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Gets short type values from [rowId, rowId + count).
getSimpleMessage() - Method in exception org.apache.spark.sql.AnalysisException
 
getSimpleName(Class<?>) - Static method in class org.apache.spark.util.Utils
Safer than Class obj's getSimpleName which may throw Malformed class name error in scala.
getSize() - Method in class org.apache.spark.ml.feature.VectorSizeHint
group getParam
getSizeAsBytes(String) - Method in class org.apache.spark.SparkConf
Get a size parameter as bytes; throws a NoSuchElementException if it's not set.
getSizeAsBytes(String, String) - Method in class org.apache.spark.SparkConf
Get a size parameter as bytes, falling back to a default if not set.
getSizeAsBytes(String, long) - Method in class org.apache.spark.SparkConf
Get a size parameter as bytes, falling back to a default if not set.
getSizeAsGb(String) - Method in class org.apache.spark.SparkConf
Get a size parameter as Gibibytes; throws a NoSuchElementException if it's not set.
getSizeAsGb(String, String) - Method in class org.apache.spark.SparkConf
Get a size parameter as Gibibytes, falling back to a default if not set.
getSizeAsKb(String) - Method in class org.apache.spark.SparkConf
Get a size parameter as Kibibytes; throws a NoSuchElementException if it's not set.
getSizeAsKb(String, String) - Method in class org.apache.spark.SparkConf
Get a size parameter as Kibibytes, falling back to a default if not set.
getSizeAsMb(String) - Method in class org.apache.spark.SparkConf
Get a size parameter as Mebibytes; throws a NoSuchElementException if it's not set.
getSizeAsMb(String, String) - Method in class org.apache.spark.SparkConf
Get a size parameter as Mebibytes, falling back to a default if not set.
getSizeInBytes() - Method in interface org.apache.spark.ml.linalg.Matrix
Gets the current size in bytes of this `Matrix`.
getSlotDescs() - Method in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
getSmoothing() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getSmoothing() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getSolver() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getSolver() - Method in interface org.apache.spark.ml.param.shared.HasSolver
 
getSolver() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getSolver() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getSolver() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getSolver() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getSparkClassLoader() - Static method in class org.apache.spark.util.Utils
Get the ClassLoader which loaded Spark.
getSparkHome() - Method in class org.apache.spark.api.java.JavaSparkContext
Get Spark's home location from either a value set through the constructor, or the spark.home Java property, or the SPARK_HOME environment variable (in that order of preference).
getSparkOrYarnConfig(SparkConf, String, String) - Static method in class org.apache.spark.util.Utils
Return the value of a config either through the SparkConf or the Hadoop configuration if this is Yarn mode.
getSparseSizeInBytes(boolean) - Method in interface org.apache.spark.ml.linalg.Matrix
Gets the size of the minimal sparse representation of this `Matrix`.
getSplit() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData
 
getSplits() - Method in class org.apache.spark.ml.feature.Bucketizer
 
getSplitsArray() - Method in class org.apache.spark.ml.feature.Bucketizer
 
getStackTrace() - Static method in exception org.apache.spark.sql.AnalysisException
 
getStageInfo(int) - Method in class org.apache.spark.api.java.JavaSparkStatusTracker
Returns stage information, or null if the stage info could not be found or was garbage collected.
getStageInfo(int) - Method in class org.apache.spark.SparkStatusTracker
Returns stage information, or None if the stage info could not be found or was garbage collected.
getStagePath(String, int, int, String) - Method in class org.apache.spark.ml.Pipeline.SharedReadWrite$
Get path for saving the given stage.
getStages() - Method in class org.apache.spark.ml.Pipeline
 
getStandardization() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getStandardization() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getStandardization() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getStandardization() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getStandardization() - Method in interface org.apache.spark.ml.param.shared.HasStandardization
 
getStandardization() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getStandardization() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getStartOffset() - Static method in class org.apache.spark.rdd.InputFileBlockHolder
Returns the starting offset of the block currently being read, or -1 if it is unknown.
getStartOffset() - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.ContinuousReader
Return the specified or inferred start offset for this reader.
getStartOffset() - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.MicroBatchReader
Returns the specified (if explicitly set through setOffsetRange) or inferred start offset for this reader.
getStartTimeEpoch() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
getState() - Method in interface org.apache.spark.launcher.SparkAppHandle
Returns the current application state.
getState() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Developer API
getState() - Method in class org.apache.spark.streaming.StreamingContext
Developer API
getStatement() - Method in class org.apache.spark.ml.feature.SQLTransformer
 
getStatistics() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsReportStatistics
Returns the basic statistics of this data source.
getStderr(Process, long) - Static method in class org.apache.spark.util.Utils
Return the stderr of a process after waiting for the process to terminate.
getStepSize() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getStepSize() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getStepSize() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getStepSize() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getStepSize() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getStepSize() - Method in interface org.apache.spark.ml.param.shared.HasStepSize
 
getStepSize() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getStepSize() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getStopWords() - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
getStorageLevel() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
getStorageLevel() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
getStorageLevel() - Static method in class org.apache.spark.api.java.JavaRDD
 
getStorageLevel() - Method in interface org.apache.spark.api.java.JavaRDDLike
Get the RDD's current storage level, or StorageLevel.NONE if none is set.
getStorageLevel() - Static method in class org.apache.spark.api.r.RRDD
 
getStorageLevel() - Static method in class org.apache.spark.graphx.EdgeRDD
 
getStorageLevel() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
getStorageLevel() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
getStorageLevel() - Static method in class org.apache.spark.graphx.VertexRDD
 
getStorageLevel() - Static method in class org.apache.spark.rdd.HadoopRDD
 
getStorageLevel() - Static method in class org.apache.spark.rdd.JdbcRDD
 
getStorageLevel() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
getStorageLevel() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
getStorageLevel() - Method in class org.apache.spark.rdd.RDD
Get the RDD's current storage level, or StorageLevel.NONE if none is set.
getStorageLevel() - Static method in class org.apache.spark.rdd.UnionRDD
 
GetStorageStatus$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.GetStorageStatus$
 
getStrategy() - Static method in class org.apache.spark.ml.feature.Imputer
 
getStrategy() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
getString(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i as a String object.
getString(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a String.
getStringArray(String) - Method in class org.apache.spark.sql.types.Metadata
Gets a String array.
getStringIndexerOrderType() - Static method in class org.apache.spark.ml.feature.RFormula
 
getStringIndexerOrderType() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
getStringOrderType() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
getStringOrderType() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
getStruct(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of struct type as a Row object.
getStruct(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getStruct(int, int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getStruct(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the struct type value for rowId.
getSubsamplingRate() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.clustering.LDA
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
getSubsamplingRate() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
getSubsamplingRate() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getSuppressed() - Static method in exception org.apache.spark.sql.AnalysisException
 
getSystemProperties() - Static method in class org.apache.spark.util.Utils
Returns the system properties map that is thread-safe to iterator over.
getTable(String) - Method in class org.apache.spark.sql.catalog.Catalog
Get the table or view with the specified name.
getTable(String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Get the table or view with the specified name in the specified database.
getTableExistsQuery(String) - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
getTableExistsQuery(String) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Get the SQL query that should be used to find if the given table exists.
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
getTableExistsQuery(String) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
getTableNames(SparkSession, String) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
getTables(SparkSession, String) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
getTau0() - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
A (positive) learning parameter that downweights early iterations.
getThreadDump() - Static method in class org.apache.spark.util.Utils
Return a thread dump of all threads' stacktraces.
getThreadDumpForThread(long) - Static method in class org.apache.spark.util.Utils
 
getThreshold() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getThreshold() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getThreshold() - Method in class org.apache.spark.ml.classification.LogisticRegression
 
getThreshold() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getThreshold() - Method in class org.apache.spark.ml.feature.Binarizer
 
getThreshold() - Method in interface org.apache.spark.ml.param.shared.HasThreshold
 
getThreshold() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
Returns the threshold (if any) used for converting raw prediction scores into 0/1 predictions.
getThreshold() - Method in class org.apache.spark.mllib.classification.SVMModel
Returns the threshold (if any) used for converting raw prediction scores into 0/1 predictions.
getThresholds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
getThresholds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
getThresholds() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
getThresholds() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
getThresholds() - Method in class org.apache.spark.ml.classification.LogisticRegression
 
getThresholds() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getThresholds() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
getThresholds() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getThresholds() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getThresholds() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getThresholds() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
getThresholds() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
getThresholds() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
getThresholds() - Method in interface org.apache.spark.ml.param.shared.HasThresholds
 
getTimeAsMs(String) - Method in class org.apache.spark.SparkConf
Get a time parameter as milliseconds; throws a NoSuchElementException if it's not set.
getTimeAsMs(String, String) - Method in class org.apache.spark.SparkConf
Get a time parameter as milliseconds, falling back to a default if not set.
getTimeAsSeconds(String) - Method in class org.apache.spark.SparkConf
Get a time parameter as seconds; throws a NoSuchElementException if it's not set.
getTimeAsSeconds(String, String) - Method in class org.apache.spark.SparkConf
Get a time parameter as seconds, falling back to a default if not set.
getTimestamp(int) - Method in interface org.apache.spark.sql.Row
Returns the value at position i of date type as java.sql.Timestamp.
getTimestamp() - Method in class org.apache.spark.streaming.kinesis.KinesisInitialPositions.AtTimestamp
 
getTimeZoneOffset() - Static method in class org.apache.spark.ui.UIUtils
 
GETTING_RESULT_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
GETTING_RESULT_TIME() - Static method in class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
GETTING_RESULT_TIME() - Static method in class org.apache.spark.ui.ToolTips
 
gettingResult() - Method in class org.apache.spark.scheduler.TaskInfo
 
gettingResultTime() - Method in class org.apache.spark.scheduler.TaskInfo
The time when the task started remotely getting the result.
gettingResultTime() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
gettingResultTime(TaskData) - Static method in class org.apache.spark.status.AppStatusUtils
 
gettingResultTime(long, long, long) - Static method in class org.apache.spark.status.AppStatusUtils
 
getTol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getTol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getTol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getTol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getTol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
getTol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
getTol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
getTol() - Static method in class org.apache.spark.ml.clustering.KMeans
 
getTol() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
getTol() - Method in interface org.apache.spark.ml.param.shared.HasTol
 
getTol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
getTol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
getTol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getTol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getTol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getTol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getToLowercase() - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
getTopicConcentration() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getTopicConcentration() - Static method in class org.apache.spark.ml.clustering.LDA
 
getTopicConcentration() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getTopicConcentration() - Method in class org.apache.spark.mllib.clustering.LDA
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.
getTopicDistributionCol() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
getTopicDistributionCol() - Static method in class org.apache.spark.ml.clustering.LDA
 
getTopicDistributionCol() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
getTopologyForHost(String) - Method in class org.apache.spark.storage.DefaultTopologyMapper
 
getTopologyForHost(String) - Method in class org.apache.spark.storage.FileBasedTopologyMapper
 
getTopologyForHost(String) - Method in class org.apache.spark.storage.TopologyMapper
Gets the topology information given the host name
getTrainRatio() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
getTrainRatio() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
getTreeStrategy() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
getTruncateQuery(String) - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
getTruncateQuery(String) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
The SQL query that should be used to truncate a table.
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
The SQL query used to truncate a table.
getTruncateQuery(String) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
getUDTFor(String) - Static method in class org.apache.spark.sql.types.UDTRegistration
Returns the Class of UserDefinedType for the name of a given user class.
getUidMap(Params) - Static method in class org.apache.spark.ml.util.MetaAlgorithmReadWrite
Examine the given estimator (which may be a compound estimator) and extract a mapping from UIDs to corresponding Params instances.
getUiRoot(ServletContext) - Static method in class org.apache.spark.status.api.v1.UIRootFromServletContext
 
getUpperBound(double, long, double) - Static method in class org.apache.spark.util.random.BinomialBounds
Returns a threshold p such that if we conduct n Bernoulli trials with success rate = p, it is very unlikely to have less than fraction * n successes.
getUpperBound(double) - Static method in class org.apache.spark.util.random.PoissonBounds
Returns a lambda such that Pr[X < s] is very small, where X ~ Pois(lambda).
getUpperBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getUpperBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getUpperBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getUpperBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getUsedTimeMs(long) - Static method in class org.apache.spark.util.Utils
Return the string to tell how long has passed in milliseconds.
getUseNodeIdCache() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
getUserCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
getUserCol() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
getUserJars(SparkConf) - Static method in class org.apache.spark.util.Utils
Return the jar files pointed by the "spark.jars" property.
getUTF8String(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
getUTF8String(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
getUTF8String(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
getUTF8String(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the string type value for rowId.
getValidationTol() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
getValue(int) - Method in class org.apache.spark.ml.attribute.NominalAttribute
Gets a value given its index.
getValuesMap(Seq<String>) - Method in interface org.apache.spark.sql.Row
Returns a Map consisting of names and values for the requested fieldNames For primitive types if value is null it returns 'zero value' specific for primitive ie.
getVarianceCol() - Method in interface org.apache.spark.ml.param.shared.HasVarianceCol
 
getVarianceCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
getVarianceCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
getVariancePower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getVariancePower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getVectors() - Method in class org.apache.spark.ml.feature.Word2VecModel
Returns a dataframe with two fields, "word" and "vector", with "word" being a String and and the vector the DenseVector that it is mapped to.
getVectors() - Method in class org.apache.spark.mllib.feature.Word2VecModel
Returns a map of words to their vector representations.
getVectorSize() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getVectorSize() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getVocabSize() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
getVocabSize() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
getWeightCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
getWeightCol() - Method in interface org.apache.spark.ml.param.shared.HasWeightCol
 
getWeightCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
getWeightCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
getWeightCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
getWeightCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
getWeightCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
getWeightCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
getWidth(Row) - Static method in class org.apache.spark.ml.image.ImageSchema
Gets the width of the image
getWindowSize() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
getWindowSize() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
getWithMean() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
getWithMean() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
getWithStd() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
getWithStd() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
Gini - Class in org.apache.spark.mllib.tree.impurity
Class for calculating the Gini impurity (http://en.wikipedia.org/wiki/Decision_tree_learning#Gini_impurity) during multiclass classification.
Gini() - Constructor for class org.apache.spark.mllib.tree.impurity.Gini
 
GLMClassificationModel - Class in org.apache.spark.mllib.classification.impl
Helper class for import/export of GLM classification models.
GLMClassificationModel() - Constructor for class org.apache.spark.mllib.classification.impl.GLMClassificationModel
 
GLMClassificationModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.classification.impl
 
GLMClassificationModel.SaveLoadV1_0$.Data - Class in org.apache.spark.mllib.classification.impl
Model data for import/export
GLMRegressionModel - Class in org.apache.spark.mllib.regression.impl
Helper methods for import/export of GLM regression models.
GLMRegressionModel() - Constructor for class org.apache.spark.mllib.regression.impl.GLMRegressionModel
 
GLMRegressionModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.regression.impl
 
GLMRegressionModel.SaveLoadV1_0$.Data - Class in org.apache.spark.mllib.regression.impl
Model data for model import/export
glom() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
glom() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
glom() - Static method in class org.apache.spark.api.java.JavaRDD
 
glom() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD created by coalescing all elements within each partition into an array.
glom() - Static method in class org.apache.spark.api.r.RRDD
 
glom() - Static method in class org.apache.spark.graphx.EdgeRDD
 
glom() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
glom() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
glom() - Static method in class org.apache.spark.graphx.VertexRDD
 
glom() - Static method in class org.apache.spark.rdd.HadoopRDD
 
glom() - Static method in class org.apache.spark.rdd.JdbcRDD
 
glom() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
glom() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
glom() - Method in class org.apache.spark.rdd.RDD
Return an RDD created by coalescing all elements within each partition into an array.
glom() - Static method in class org.apache.spark.rdd.UnionRDD
 
glom() - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
glom() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying glom() to each RDD of this DStream.
glom() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
glom() - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
glom() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
glom() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
glom() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
glom() - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD is generated by applying glom() to each RDD of this DStream.
goodnessOfFit() - Method in class org.apache.spark.mllib.stat.test.ChiSqTest.NullHypothesis$
 
grad() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
gradient() - Method in class org.apache.spark.ml.regression.AFTAggregator
 
Gradient - Class in org.apache.spark.mllib.optimization
Developer API Class used to compute the gradient for a loss function, given a single data point.
Gradient() - Constructor for class org.apache.spark.mllib.optimization.Gradient
 
gradient(double, double) - Static method in class org.apache.spark.mllib.tree.loss.AbsoluteError
Method to calculate the gradients for the gradient boosting calculation for least absolute error calculation.
gradient(double, double) - Static method in class org.apache.spark.mllib.tree.loss.LogLoss
Method to calculate the loss gradients for the gradient boosting calculation for binary classification The gradient with respect to F(x) is: - 4 y / (1 + exp(2 y F(x)))
gradient(double, double) - Method in interface org.apache.spark.mllib.tree.loss.Loss
Method to calculate the gradients for the gradient boosting calculation.
gradient(double, double) - Static method in class org.apache.spark.mllib.tree.loss.SquaredError
Method to calculate the gradients for the gradient boosting calculation for least squares error calculation.
GradientBoostedTrees - Class in org.apache.spark.ml.tree.impl
 
GradientBoostedTrees() - Constructor for class org.apache.spark.ml.tree.impl.GradientBoostedTrees
 
GradientBoostedTrees - Class in org.apache.spark.mllib.tree
A class that implements Stochastic Gradient Boosting for regression and binary classification.
GradientBoostedTrees(BoostingStrategy) - Constructor for class org.apache.spark.mllib.tree.GradientBoostedTrees
 
GradientBoostedTreesModel - Class in org.apache.spark.mllib.tree.model
Represents a gradient boosted trees model.
GradientBoostedTreesModel(Enumeration.Value, DecisionTreeModel[], double[]) - Constructor for class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
GradientDescent - Class in org.apache.spark.mllib.optimization
Class used to solve an optimization problem using Gradient Descent.
Graph<VD,ED> - Class in org.apache.spark.graphx
The Graph abstractly represents a graph with arbitrary objects associated with vertices and edges.
GraphGenerators - Class in org.apache.spark.graphx.util
A collection of graph generating functions.
GraphGenerators() - Constructor for class org.apache.spark.graphx.util.GraphGenerators
 
GraphImpl<VD,ED> - Class in org.apache.spark.graphx.impl
An implementation of Graph to support computation on graphs.
GraphLoader - Class in org.apache.spark.graphx
Provides utilities for loading Graphs from files.
GraphLoader() - Constructor for class org.apache.spark.graphx.GraphLoader
 
GraphOps<VD,ED> - Class in org.apache.spark.graphx
Contains additional functionality for Graph.
GraphOps(Graph<VD, ED>, ClassTag<VD>, ClassTag<ED>) - Constructor for class org.apache.spark.graphx.GraphOps
 
graphToGraphOps(Graph<VD, ED>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.Graph
Implicitly extracts the GraphOps member from a graph.
GraphXUtils - Class in org.apache.spark.graphx
 
GraphXUtils() - Constructor for class org.apache.spark.graphx.GraphXUtils
 
greater(Duration) - Method in class org.apache.spark.streaming.Duration
 
greater(Time) - Method in class org.apache.spark.streaming.Time
 
greaterEq(Duration) - Method in class org.apache.spark.streaming.Duration
 
greaterEq(Time) - Method in class org.apache.spark.streaming.Time
 
GreaterThan - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a value greater than value.
GreaterThan(String, Object) - Constructor for class org.apache.spark.sql.sources.GreaterThan
 
GreaterThanOrEqual - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a value greater than or equal to value.
GreaterThanOrEqual(String, Object) - Constructor for class org.apache.spark.sql.sources.GreaterThanOrEqual
 
greatest(Column...) - Static method in class org.apache.spark.sql.functions
Returns the greatest value of the list of values, skipping null values.
greatest(String, String...) - Static method in class org.apache.spark.sql.functions
Returns the greatest value of the list of column names, skipping null values.
greatest(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Returns the greatest value of the list of values, skipping null values.
greatest(String, Seq<String>) - Static method in class org.apache.spark.sql.functions
Returns the greatest value of the list of column names, skipping null values.
gridGraph(SparkContext, int, int) - Static method in class org.apache.spark.graphx.util.GraphGenerators
Create rows by cols grid graph with each vertex connected to its row+1 and col+1 neighbors.
groupArr() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
groupBy(Function<T, U>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
groupBy(Function<T, U>, int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
groupBy(Function<T, U>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
groupBy(Function<T, U>, int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
groupBy(Function<T, U>) - Static method in class org.apache.spark.api.java.JavaRDD
 
groupBy(Function<T, U>, int) - Static method in class org.apache.spark.api.java.JavaRDD
 
groupBy(Function<T, U>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD of grouped elements.
groupBy(Function<T, U>, int) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD of grouped elements.
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.api.r.RRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.api.r.RRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.api.r.RRDD
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.VertexRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.graphx.VertexRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.graphx.VertexRDD
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
groupBy(Function1<T, K>, ClassTag<K>) - Method in class org.apache.spark.rdd.RDD
Return an RDD of grouped items.
groupBy(Function1<T, K>, int, ClassTag<K>) - Method in class org.apache.spark.rdd.RDD
Return an RDD of grouped elements.
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Method in class org.apache.spark.rdd.RDD
Return an RDD of grouped items.
groupBy(Function1<T, K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.UnionRDD
 
groupBy(Function1<T, K>, int, ClassTag<K>) - Static method in class org.apache.spark.rdd.UnionRDD
 
groupBy(Function1<T, K>, Partitioner, ClassTag<K>, Ordering<K>) - Static method in class org.apache.spark.rdd.UnionRDD
 
groupBy(Column...) - Method in class org.apache.spark.sql.Dataset
Groups the Dataset using the specified columns, so we can run aggregation on them.
groupBy(String, String...) - Method in class org.apache.spark.sql.Dataset
Groups the Dataset using the specified columns, so that we can run aggregation on them.
groupBy(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Groups the Dataset using the specified columns, so we can run aggregation on them.
groupBy(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Groups the Dataset using the specified columns, so that we can run aggregation on them.
groupBy(Function1<A, K>) - Static method in class org.apache.spark.sql.types.StructType
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.api.r.RRDD
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.graphx.EdgeRDD
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.graphx.VertexRDD
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.rdd.HadoopRDD
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.rdd.JdbcRDD
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
groupBy$default$4(Function1<T, K>, Partitioner) - Static method in class org.apache.spark.rdd.UnionRDD
 
groupByKey(Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Group the values for each key in the RDD into a single sequence.
groupByKey(int) - Method in class org.apache.spark.api.java.JavaPairRDD
Group the values for each key in the RDD into a single sequence.
groupByKey() - Method in class org.apache.spark.api.java.JavaPairRDD
Group the values for each key in the RDD into a single sequence.
groupByKey(Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Group the values for each key in the RDD into a single sequence.
groupByKey(int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Group the values for each key in the RDD into a single sequence.
groupByKey() - Method in class org.apache.spark.rdd.PairRDDFunctions
Group the values for each key in the RDD into a single sequence.
groupByKey(Function1<T, K>, Encoder<K>) - Method in class org.apache.spark.sql.Dataset
Experimental (Scala-specific) Returns a KeyValueGroupedDataset where the data is grouped by the given key func.
groupByKey(MapFunction<T, K>, Encoder<K>) - Method in class org.apache.spark.sql.Dataset
Experimental (Java-specific) Returns a KeyValueGroupedDataset where the data is grouped by the given key func.
groupByKey() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying groupByKey to each RDD.
groupByKey(int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying groupByKey to each RDD.
groupByKey(Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying groupByKey on each RDD of this DStream.
groupByKey() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
groupByKey(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
groupByKey(Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
groupByKey() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
groupByKey(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
groupByKey(Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
groupByKey() - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying groupByKey to each RDD.
groupByKey(int) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying groupByKey to each RDD.
groupByKey(Partitioner) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying groupByKey on each RDD.
groupByKeyAndWindow(Duration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying groupByKey over a sliding window.
groupByKeyAndWindow(Duration, Duration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying groupByKey over a sliding window.
groupByKeyAndWindow(Duration, Duration, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying groupByKey over a sliding window on this DStream.
groupByKeyAndWindow(Duration, Duration, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying groupByKey over a sliding window on this DStream.
groupByKeyAndWindow(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
groupByKeyAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
groupByKeyAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
groupByKeyAndWindow(Duration, Duration, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
groupByKeyAndWindow(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
groupByKeyAndWindow(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
groupByKeyAndWindow(Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
groupByKeyAndWindow(Duration, Duration, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
groupByKeyAndWindow(Duration) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying groupByKey over a sliding window.
groupByKeyAndWindow(Duration, Duration) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying groupByKey over a sliding window.
groupByKeyAndWindow(Duration, Duration, int) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying groupByKey over a sliding window on this DStream.
groupByKeyAndWindow(Duration, Duration, Partitioner) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Create a new DStream by applying groupByKey over a sliding window on this DStream.
GroupByType$() - Constructor for class org.apache.spark.sql.RelationalGroupedDataset.GroupByType$
 
grouped(int) - Static method in class org.apache.spark.sql.types.StructType
 
groupEdges(Function2<ED, ED, ED>) - Method in class org.apache.spark.graphx.Graph
Merges multiple edges between two vertices into a single edge.
groupEdges(Function2<ED, ED, ED>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
groupHash() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
grouping(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.
grouping(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: indicates whether a specified column in a GROUP BY list is aggregated or not, returns 1 for aggregated or 0 for not aggregated in the result set.
grouping_id(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the level of grouping, equals to
grouping_id(String, Seq<String>) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the level of grouping, equals to
GroupMappingServiceProvider - Interface in org.apache.spark.security
This Spark trait is used for mapping a given userName to a set of groups which it belongs to.
GroupState<S> - Interface in org.apache.spark.sql.streaming
Experimental
GroupStateTimeout - Class in org.apache.spark.sql.streaming
Represents the type of timeouts possible for the Dataset operations `mapGroupsWithState` and `flatMapGroupsWithState`.
GroupStateTimeout() - Constructor for class org.apache.spark.sql.streaming.GroupStateTimeout
 
groupWith(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
Alias for cogroup.
groupWith(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>) - Method in class org.apache.spark.api.java.JavaPairRDD
Alias for cogroup.
groupWith(JavaPairRDD<K, W1>, JavaPairRDD<K, W2>, JavaPairRDD<K, W3>) - Method in class org.apache.spark.api.java.JavaPairRDD
Alias for cogroup.
groupWith(RDD<Tuple2<K, W>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Alias for cogroup.
groupWith(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Alias for cogroup.
groupWith(RDD<Tuple2<K, W1>>, RDD<Tuple2<K, W2>>, RDD<Tuple2<K, W3>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Alias for cogroup.
gt(double) - Static method in class org.apache.spark.ml.param.ParamValidators
Check if value is greater than lowerBound
gt(Object) - Method in class org.apache.spark.sql.Column
Greater than.
gtEq(double) - Static method in class org.apache.spark.ml.param.ParamValidators
Check if value is greater than or equal to lowerBound
guard(Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 

H

hadoopConfiguration() - Method in class org.apache.spark.api.java.JavaSparkContext
Returns the Hadoop configuration used for the Hadoop code (e.g.
hadoopConfiguration() - Method in class org.apache.spark.SparkContext
A default Hadoop Configuration for the Hadoop code (e.g.
hadoopDelegationCreds() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SparkAppConfig
 
hadoopFile(String, Class<F>, Class<K>, Class<V>, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a Hadoop file with an arbitrary InputFormat.
hadoopFile(String, Class<F>, Class<K>, Class<V>) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a Hadoop file with an arbitrary InputFormat
hadoopFile(String, Class<? extends InputFormat<K, V>>, Class<K>, Class<V>, int) - Method in class org.apache.spark.SparkContext
Get an RDD for a Hadoop file with an arbitrary InputFormat
hadoopFile(String, int, ClassTag<K>, ClassTag<V>, ClassTag<F>) - Method in class org.apache.spark.SparkContext
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
hadoopFile(String, ClassTag<K>, ClassTag<V>, ClassTag<F>) - Method in class org.apache.spark.SparkContext
Smarter version of hadoopFile() that uses class tags to figure out the classes of keys, values and the InputFormat so that users don't need to pass them directly.
HadoopMapPartitionsWithSplitRDD$() - Constructor for class org.apache.spark.rdd.HadoopRDD.HadoopMapPartitionsWithSplitRDD$
 
HadoopMapRedCommitProtocol - Class in org.apache.spark.internal.io
An FileCommitProtocol implementation backed by an underlying Hadoop OutputCommitter (from the old mapred API).
HadoopMapRedCommitProtocol(String, String) - Constructor for class org.apache.spark.internal.io.HadoopMapRedCommitProtocol
 
HadoopMapReduceCommitProtocol - Class in org.apache.spark.internal.io
An FileCommitProtocol implementation backed by an underlying Hadoop OutputCommitter (from the newer mapreduce API, not the old mapred API).
HadoopMapReduceCommitProtocol(String, String, boolean) - Constructor for class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
hadoopRDD(JobConf, Class<F>, Class<K>, Class<V>, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any other necessary info (e.g.
hadoopRDD(JobConf, Class<F>, Class<K>, Class<V>) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf giving its InputFormat and any other necessary info (e.g.
HadoopRDD<K,V> - Class in org.apache.spark.rdd
Developer API An RDD that provides core functionality for reading data stored in Hadoop (e.g., files in HDFS, sources in HBase, or S3), using the older MapReduce API (org.apache.hadoop.mapred).
HadoopRDD(SparkContext, Broadcast<org.apache.spark.util.SerializableConfiguration>, Option<Function1<JobConf, BoxedUnit>>, Class<? extends InputFormat<K, V>>, Class<K>, Class<V>, int) - Constructor for class org.apache.spark.rdd.HadoopRDD
 
HadoopRDD(SparkContext, JobConf, Class<? extends InputFormat<K, V>>, Class<K>, Class<V>, int) - Constructor for class org.apache.spark.rdd.HadoopRDD
 
hadoopRDD(JobConf, Class<? extends InputFormat<K, V>>, Class<K>, Class<V>, int) - Method in class org.apache.spark.SparkContext
Get an RDD for a Hadoop-readable dataset from a Hadoop JobConf given its InputFormat and other necessary info (e.g.
HadoopRDD.HadoopMapPartitionsWithSplitRDD$ - Class in org.apache.spark.rdd
 
HadoopWriteConfigUtil<K,V> - Class in org.apache.spark.internal.io
Interface for create output format/committer/writer used during saving an RDD using a Hadoop OutputFormat (both from the old mapred API and the new mapreduce API)
HadoopWriteConfigUtil(ClassTag<V>) - Constructor for class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
hammingLoss() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns Hamming-loss
handleInvalid() - Method in class org.apache.spark.ml.feature.Bucketizer
Param for how to handle invalid entries.
handleInvalid() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.RFormula
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
handleInvalid() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
handleInvalid() - Method in class org.apache.spark.ml.feature.VectorSizeHint
Param for how to handle invalid entries.
handleInvalid() - Method in interface org.apache.spark.ml.param.shared.HasHandleInvalid
Param for how to handle invalid entries.
hasAccumulators(StageData) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HasAggregationDepth - Interface in org.apache.spark.ml.param.shared
Trait for shared param aggregationDepth (default: 2).
hasAttr(String) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Test whether this attribute group contains a specific attribute.
hasBytesSpilled(StageData) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HasCachedBlocks(String) - Constructor for class org.apache.spark.storage.BlockManagerMessages.HasCachedBlocks
 
HasCachedBlocks$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.HasCachedBlocks$
 
hasCachedSerializedBroadcast() - Method in class org.apache.spark.ShuffleStatus
 
HasCheckpointInterval - Interface in org.apache.spark.ml.param.shared
Trait for shared param checkpointInterval.
HasCollectSubModels - Interface in org.apache.spark.ml.param.shared
Trait for shared param collectSubModels (default: false).
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.LDA
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.DCT
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IDF
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Imputer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Interaction
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.NGram
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PCA
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormula
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
hasDefault(Param<T>) - Method in interface org.apache.spark.ml.param.Params
Tests whether the input param has a default value set.
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.Pipeline
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.PipelineModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
hasDefault(Param<T>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
hasDefiniteSize() - Static method in class org.apache.spark.sql.types.StructType
 
HasElasticNetParam - Interface in org.apache.spark.ml.param.shared
Trait for shared param elasticNetParam.
HasFeaturesCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param featuresCol (default: "features").
HasFitIntercept - Interface in org.apache.spark.ml.param.shared
Trait for shared param fitIntercept (default: true).
hash(Column...) - Static method in class org.apache.spark.sql.functions
Calculates the hash code of given columns, and returns the result as an int column.
hash(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Calculates the hash code of given columns, and returns the result as an int column.
HasHandleInvalid - Interface in org.apache.spark.ml.param.shared
Trait for shared param handleInvalid.
hashCode() - Method in class org.apache.spark.api.java.Optional
 
hashCode() - Method in class org.apache.spark.graphx.EdgeDirection
 
hashCode() - Method in class org.apache.spark.HashPartitioner
 
hashCode() - Method in class org.apache.spark.ml.attribute.AttributeGroup
 
hashCode() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
hashCode() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
hashCode() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
hashCode() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
hashCode() - Method in class org.apache.spark.ml.linalg.DenseVector
 
hashCode() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
hashCode() - Method in class org.apache.spark.ml.linalg.SparseVector
 
hashCode() - Method in interface org.apache.spark.ml.linalg.Vector
Returns a hash code value for the vector.
hashCode() - Static method in class org.apache.spark.ml.param.DoubleParam
 
hashCode() - Static method in class org.apache.spark.ml.param.FloatParam
 
hashCode() - Method in class org.apache.spark.ml.param.Param
 
hashCode() - Method in class org.apache.spark.ml.tree.CategoricalSplit
 
hashCode() - Method in class org.apache.spark.ml.tree.ContinuousSplit
 
hashCode() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
hashCode() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
hashCode() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
hashCode() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
hashCode() - Method in interface org.apache.spark.mllib.linalg.Vector
Returns a hash code value for the vector.
hashCode() - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
hashCode() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
hashCode() - Method in class org.apache.spark.mllib.tree.model.Predict
 
hashCode() - Method in class org.apache.spark.partial.BoundedDouble
 
hashCode() - Method in interface org.apache.spark.Partition
 
hashCode() - Method in class org.apache.spark.RangePartitioner
 
hashCode() - Method in class org.apache.spark.scheduler.cluster.ExecutorInfo
 
hashCode() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
hashCode() - Method in class org.apache.spark.scheduler.SplitInfo
 
hashCode() - Method in class org.apache.spark.sql.Column
 
hashCode() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
hashCode() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
hashCode() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
hashCode() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
hashCode() - Method in interface org.apache.spark.sql.Row
 
hashCode() - Method in class org.apache.spark.sql.sources.In
 
hashCode() - Method in class org.apache.spark.sql.sources.v2.reader.streaming.Offset
 
hashCode() - Method in class org.apache.spark.sql.types.Decimal
 
hashCode() - Method in class org.apache.spark.sql.types.Metadata
 
hashCode() - Method in class org.apache.spark.sql.types.StructType
 
hashCode() - Method in class org.apache.spark.storage.BlockManagerId
 
hashCode() - Method in class org.apache.spark.storage.StorageLevel
 
HashingTF - Class in org.apache.spark.ml.feature
Maps a sequence of terms to their term frequencies using the hashing trick.
HashingTF(String) - Constructor for class org.apache.spark.ml.feature.HashingTF
 
HashingTF() - Constructor for class org.apache.spark.ml.feature.HashingTF
 
HashingTF - Class in org.apache.spark.mllib.feature
Maps a sequence of terms to their term frequencies using the hashing trick.
HashingTF(int) - Constructor for class org.apache.spark.mllib.feature.HashingTF
 
HashingTF() - Constructor for class org.apache.spark.mllib.feature.HashingTF
 
HashPartitioner - Class in org.apache.spark
A Partitioner that implements hash-based partitioning using Java's Object.hashCode.
HashPartitioner(int) - Constructor for class org.apache.spark.HashPartitioner
 
hasInput(StageData) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HasInputCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param inputCol.
HasInputCols - Interface in org.apache.spark.ml.param.shared
Trait for shared param inputCols.
hasInputOutputFormat() - Method in class org.apache.spark.sql.hive.execution.HiveOptions
 
HasLabelCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param labelCol (default: "label").
HasLoss - Interface in org.apache.spark.ml.param.shared
Trait for shared param loss.
HasMaxIter - Interface in org.apache.spark.ml.param.shared
Trait for shared param maxIter.
hasMemoryInfo() - Method in class org.apache.spark.status.LiveExecutor
 
hasNext() - Method in class org.apache.spark.InterruptibleIterator
 
hasNull() - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
hasNull() - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns true if this column vector contains any null values.
hasOutput(StageData) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HasOutputCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param outputCol (default: uid + "__output").
HasOutputCols - Interface in org.apache.spark.ml.param.shared
Trait for shared param outputCols.
hasParam(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
hasParam(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.KMeans
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.LDA
 
hasParam(String) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
hasParam(String) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
hasParam(String) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
hasParam(String) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
hasParam(String) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Binarizer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.DCT
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.HashingTF
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.IDF
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.IDFModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Imputer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.IndexToString
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Interaction
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.NGram
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Normalizer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.PCA
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.PCAModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.RFormula
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
hasParam(String) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
hasParam(String) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
hasParam(String) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
hasParam(String) - Method in interface org.apache.spark.ml.param.Params
Tests whether this instance contains a param with a given name.
hasParam(String) - Static method in class org.apache.spark.ml.Pipeline
 
hasParam(String) - Static method in class org.apache.spark.ml.PipelineModel
 
hasParam(String) - Static method in class org.apache.spark.ml.recommendation.ALS
 
hasParam(String) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
hasParam(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
hasParam(String) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
hasParam(String) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
hasParam(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
hasParam(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
hasParent() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
hasParent() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
hasParent() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
hasParent() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
hasParent() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
hasParent() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
hasParent() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.IDFModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.PCAModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
hasParent() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
hasParent() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
hasParent() - Method in class org.apache.spark.ml.Model
Indicates whether this Model has a corresponding parent.
hasParent() - Static method in class org.apache.spark.ml.PipelineModel
 
hasParent() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
hasParent() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
hasParent() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
hasParent() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
HasPredictionCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param predictionCol (default: "prediction").
HasProbabilityCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param probabilityCol (default: "probability").
HasRawPredictionCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param rawPredictionCol (default: "rawPrediction").
HasRegParam - Interface in org.apache.spark.ml.param.shared
Trait for shared param regParam.
hasRootAsShutdownDeleteDir(File) - Static method in class org.apache.spark.util.ShutdownHookManager
 
HasSeed - Interface in org.apache.spark.ml.param.shared
Trait for shared param seed (default: this.getClass.getName.hashCode.toLong).
hasShuffleRead(StageData) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
hasShuffleWrite(StageData) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
hasShutdownDeleteDir(File) - Static method in class org.apache.spark.util.ShutdownHookManager
 
HasSolver - Interface in org.apache.spark.ml.param.shared
Trait for shared param solver.
HasStandardization - Interface in org.apache.spark.ml.param.shared
Trait for shared param standardization (default: true).
HasStepSize - Interface in org.apache.spark.ml.param.shared
Trait for shared param stepSize.
hasSubModels() - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
hasSubModels() - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
hasSummary() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
Indicates whether a training summary exists for this model instance.
hasSummary() - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
Return true if there exists summary of model.
hasSummary() - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
Return true if there exists summary of model.
hasSummary() - Method in class org.apache.spark.ml.clustering.KMeansModel
Return true if there exists summary of model.
hasSummary() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
Indicates if summary is available.
hasSummary() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
Indicates whether a training summary exists for this model instance.
HasThreshold - Interface in org.apache.spark.ml.param.shared
Trait for shared param threshold.
HasThresholds - Interface in org.apache.spark.ml.param.shared
Trait for shared param thresholds.
hasTimedOut() - Method in interface org.apache.spark.sql.streaming.GroupState
Whether the function has been called because the key has timed out.
HasTol - Interface in org.apache.spark.ml.param.shared
Trait for shared param tol.
hasValue(String) - Method in class org.apache.spark.ml.attribute.NominalAttribute
Tests whether this attribute contains a specific value.
HasVarianceCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param varianceCol.
HasWeightCol - Interface in org.apache.spark.ml.param.shared
Trait for shared param weightCol.
hasWriteObjectMethod() - Method in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
hasWriteReplaceMethod() - Method in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
HdfsUtils - Class in org.apache.spark.streaming.util
 
HdfsUtils() - Constructor for class org.apache.spark.streaming.util.HdfsUtils
 
head(int) - Method in class org.apache.spark.sql.Dataset
Returns the first n rows.
head() - Method in class org.apache.spark.sql.Dataset
Returns the first row.
head() - Static method in class org.apache.spark.sql.types.StructType
 
head() - Static method in class org.apache.spark.sql.types.StructType
 
HEADER_ACCUMULATORS() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_ATTEMPT() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_DESER_TIME() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_DISK_SPILL() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_DURATION() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_ERROR() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_EXECUTOR() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_GC_TIME() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_GETTING_RESULT_TIME() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_HOST() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_ID() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_INPUT_SIZE() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_LAUNCH_TIME() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_LOCALITY() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_MEM_SPILL() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_OUTPUT_SIZE() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_PEAK_MEM() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_SCHEDULER_DELAY() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_SER_TIME() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_SHUFFLE_READ_TIME() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_SHUFFLE_REMOTE_READS() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_SHUFFLE_TOTAL_READS() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_SHUFFLE_WRITE_SIZE() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_SHUFFLE_WRITE_TIME() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_STATUS() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
HEADER_TASK_INDEX() - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
headerSparkPage(String, Function0<Seq<Node>>, SparkUITab, Option<Object>, Option<String>, boolean, boolean) - Static method in class org.apache.spark.ui.UIUtils
Returns a spark page with correctly formatted headers
headOption() - Static method in class org.apache.spark.sql.types.StructType
 
hex(Column) - Static method in class org.apache.spark.sql.functions
Computes hex value of the given column.
high() - Method in class org.apache.spark.partial.BoundedDouble
 
HingeGradient - Class in org.apache.spark.mllib.optimization
Developer API Compute gradient and loss for a Hinge loss function, as used in SVM binary classification.
HingeGradient() - Constructor for class org.apache.spark.mllib.optimization.HingeGradient
 
hint(String, Object...) - Method in class org.apache.spark.sql.Dataset
Specifies some hint on the current Dataset.
hint(String, Seq<Object>) - Method in class org.apache.spark.sql.Dataset
Specifies some hint on the current Dataset.
histogram(int) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute a histogram of the data using bucketCount number of buckets evenly spaced between the minimum and maximum of the RDD.
histogram(double[]) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute a histogram using the provided buckets.
histogram(Double[], boolean) - Method in class org.apache.spark.api.java.JavaDoubleRDD
 
histogram(int) - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute a histogram of the data using bucketCount number of buckets evenly spaced between the minimum and maximum of the RDD.
histogram(double[], boolean) - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute a histogram using the provided buckets.
HIVE_GENERIC_UDF_MACRO_CLS() - Static method in class org.apache.spark.sql.hive.HiveShim
 
HIVE_METASTORE_BARRIER_PREFIXES() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
HIVE_METASTORE_JARS() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
HIVE_METASTORE_SHARED_PREFIXES() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
HIVE_METASTORE_VERSION() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
HIVE_THRIFT_SERVER_ASYNC() - Static method in class org.apache.spark.sql.hive.HiveUtils
 
HiveAnalysis - Class in org.apache.spark.sql.hive
Replaces generic operations with specific variants that are designed to work with Hive.
HiveAnalysis() - Constructor for class org.apache.spark.sql.hive.HiveAnalysis
 
HiveCatalogMetrics - Class in org.apache.spark.metrics.source
Experimental Metrics for access to the hive external catalog.
HiveCatalogMetrics() - Constructor for class org.apache.spark.metrics.source.HiveCatalogMetrics
 
HiveContext - Class in org.apache.spark.sql.hive
Deprecated.
Use SparkSession.builder.enableHiveSupport instead. Since 2.0.0.
HiveContext(SparkContext) - Constructor for class org.apache.spark.sql.hive.HiveContext
Deprecated.
 
HiveContext(JavaSparkContext) - Constructor for class org.apache.spark.sql.hive.HiveContext
Deprecated.
 
HiveFileFormat - Class in org.apache.spark.sql.hive.execution
FileFormat for writing Hive tables.
HiveFileFormat(org.apache.spark.sql.hive.HiveShim.ShimFileSinkDesc) - Constructor for class org.apache.spark.sql.hive.execution.HiveFileFormat
 
HiveFileFormat() - Constructor for class org.apache.spark.sql.hive.execution.HiveFileFormat
 
HiveFunctionWrapper$() - Constructor for class org.apache.spark.sql.hive.HiveShim.HiveFunctionWrapper$
 
HiveOptions - Class in org.apache.spark.sql.hive.execution
Options for the Hive data source.
HiveOptions(CaseInsensitiveMap<String>) - Constructor for class org.apache.spark.sql.hive.execution.HiveOptions
 
HiveOptions(Map<String, String>) - Constructor for class org.apache.spark.sql.hive.execution.HiveOptions
 
HiveOutputWriter - Class in org.apache.spark.sql.hive.execution
 
HiveOutputWriter(String, org.apache.spark.sql.hive.HiveShim.ShimFileSinkDesc, JobConf, StructType) - Constructor for class org.apache.spark.sql.hive.execution.HiveOutputWriter
 
HiveScriptIOSchema - Class in org.apache.spark.sql.hive.execution
 
HiveScriptIOSchema(Seq<Tuple2<String, String>>, Seq<Tuple2<String, String>>, Option<String>, Option<String>, Seq<Tuple2<String, String>>, Seq<Tuple2<String, String>>, Option<String>, Option<String>, boolean) - Constructor for class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
HiveSessionResourceLoader - Class in org.apache.spark.sql.hive
 
HiveSessionResourceLoader(SparkSession, org.apache.spark.sql.hive.client.HiveClient) - Constructor for class org.apache.spark.sql.hive.HiveSessionResourceLoader
 
HiveSessionStateBuilder - Class in org.apache.spark.sql.hive
Builder that produces a Hive-aware SessionState.
HiveSessionStateBuilder(SparkSession, Option<SessionState>) - Constructor for class org.apache.spark.sql.hive.HiveSessionStateBuilder
 
HiveShim - Class in org.apache.spark.sql.hive
 
HiveShim() - Constructor for class org.apache.spark.sql.hive.HiveShim
 
HiveShim.HiveFunctionWrapper$ - Class in org.apache.spark.sql.hive
 
HiveStringType - Class in org.apache.spark.sql.types
A hive string type for compatibility.
HiveStringType() - Constructor for class org.apache.spark.sql.types.HiveStringType
 
HiveTableUtil - Class in org.apache.spark.sql.hive
 
HiveTableUtil() - Constructor for class org.apache.spark.sql.hive.HiveTableUtil
 
HiveUDAFBuffer - Class in org.apache.spark.sql.hive
 
HiveUDAFBuffer(GenericUDAFEvaluator.AggregationBuffer, boolean) - Constructor for class org.apache.spark.sql.hive.HiveUDAFBuffer
 
HiveUtils - Class in org.apache.spark.sql.hive
 
HiveUtils() - Constructor for class org.apache.spark.sql.hive.HiveUtils
 
horzcat(Matrix[]) - Static method in class org.apache.spark.ml.linalg.Matrices
Horizontally concatenate a sequence of matrices.
horzcat(Matrix[]) - Static method in class org.apache.spark.mllib.linalg.Matrices
Horizontally concatenate a sequence of matrices.
host() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutorsOnHost
 
host() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveWorker
 
host() - Method in class org.apache.spark.scheduler.TaskInfo
 
host() - Method in interface org.apache.spark.SparkExecutorInfo
 
host() - Method in class org.apache.spark.SparkExecutorInfoImpl
 
host() - Method in class org.apache.spark.status.api.v1.TaskData
 
host() - Method in class org.apache.spark.status.LiveExecutor
 
HOST() - Static method in class org.apache.spark.status.TaskIndexNames
 
host() - Method in class org.apache.spark.storage.BlockManagerId
 
hostId() - Method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
hostId() - Method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
hostLocation() - Method in class org.apache.spark.scheduler.SplitInfo
 
hostname() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
hostname() - Method in class org.apache.spark.status.LiveExecutor
 
hostPort() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
hostPort() - Method in class org.apache.spark.status.LiveExecutor
 
hostPort() - Method in class org.apache.spark.storage.BlockManagerId
 
hostToLocalTaskCount() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors
 
hour(Column) - Static method in class org.apache.spark.sql.functions
Extracts the hours as an integer from a given date/timestamp/string.
hours() - Static method in class org.apache.spark.scheduler.StatsReportListener
 
htmlResponderToServlet(Function1<HttpServletRequest, Seq<Node>>) - Static method in class org.apache.spark.ui.JettyUtils
 
httpResponseCode(URL, String, Seq<Tuple2<String, String>>) - Static method in class org.apache.spark.TestUtils
Returns the response code from an HTTP(S) URL.
hypot(Column, Column) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
hypot(Column, String) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
hypot(String, Column) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
hypot(String, String) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
hypot(Column, double) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
hypot(String, double) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
hypot(double, Column) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.
hypot(double, String) - Static method in class org.apache.spark.sql.functions
Computes sqrt(a^2^ + b^2^) without intermediate overflow or underflow.

I

i() - Method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
id() - Method in class org.apache.spark.Accumulable
Deprecated.
 
id() - Static method in class org.apache.spark.Accumulator
Deprecated.
 
id() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
id() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
id() - Static method in class org.apache.spark.api.java.JavaRDD
 
id() - Method in interface org.apache.spark.api.java.JavaRDDLike
A unique ID for this RDD (within its SparkContext).
id() - Static method in class org.apache.spark.api.r.RRDD
 
id() - Method in class org.apache.spark.broadcast.Broadcast
 
id() - Static method in class org.apache.spark.graphx.EdgeRDD
 
id() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
id() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
id() - Static method in class org.apache.spark.graphx.VertexRDD
 
id() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
id() - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering.Assignment
 
id() - Method in class org.apache.spark.mllib.tree.model.Node
 
id() - Static method in class org.apache.spark.rdd.HadoopRDD
 
id() - Static method in class org.apache.spark.rdd.JdbcRDD
 
id() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
id() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
id() - Method in class org.apache.spark.rdd.RDD
A unique ID for this RDD (within its SparkContext).
id() - Static method in class org.apache.spark.rdd.UnionRDD
 
id() - Method in class org.apache.spark.scheduler.AccumulableInfo
 
id() - Method in class org.apache.spark.scheduler.TaskInfo
 
id() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns the unique id of this query that persists across restarts from checkpoint data.
id() - Method in class org.apache.spark.sql.streaming.StreamingQueryListener.QueryStartedEvent
 
id() - Method in class org.apache.spark.sql.streaming.StreamingQueryListener.QueryTerminatedEvent
 
id() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
id() - Method in class org.apache.spark.status.api.v1.AccumulableInfo
 
id() - Method in class org.apache.spark.status.api.v1.ApplicationInfo
 
id() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
id() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
id() - Method in class org.apache.spark.storage.RDDInfo
 
id() - Method in class org.apache.spark.streaming.dstream.InputDStream
This is an unique identifier for the input stream.
id() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
id() - Method in class org.apache.spark.util.AccumulatorV2
Returns the id of this accumulator, can only be called after registration.
Identifiable - Interface in org.apache.spark.ml.util
Developer API
Identity$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Identity$
 
IDF - Class in org.apache.spark.ml.feature
Compute the Inverse Document Frequency (IDF) given a collection of documents.
IDF(String) - Constructor for class org.apache.spark.ml.feature.IDF
 
IDF() - Constructor for class org.apache.spark.ml.feature.IDF
 
idf() - Method in class org.apache.spark.ml.feature.IDFModel
Returns the IDF vector.
IDF - Class in org.apache.spark.mllib.feature
Inverse document frequency (IDF).
IDF(int) - Constructor for class org.apache.spark.mllib.feature.IDF
 
IDF() - Constructor for class org.apache.spark.mllib.feature.IDF
 
idf() - Method in class org.apache.spark.mllib.feature.IDF.DocumentFrequencyAggregator
Returns the current IDF vector.
idf() - Method in class org.apache.spark.mllib.feature.IDFModel
 
IDF.DocumentFrequencyAggregator - Class in org.apache.spark.mllib.feature
Document frequency aggregator.
IDFModel - Class in org.apache.spark.ml.feature
Model fitted by IDF.
IDFModel - Class in org.apache.spark.mllib.feature
Represents an IDF model that can transform term frequency vectors.
ifPartitionNotExists() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
imageFields() - Static method in class org.apache.spark.ml.image.ImageSchema
 
ImageSchema - Class in org.apache.spark.ml.image
Experimental Defines the image schema and methods to read and manipulate images.
ImageSchema() - Constructor for class org.apache.spark.ml.image.ImageSchema
 
imageSchema() - Static method in class org.apache.spark.ml.image.ImageSchema
DataFrame with a single column of images named "image" (nullable)
implicitPrefs() - Static method in class org.apache.spark.ml.recommendation.ALS
 
implicits() - Method in class org.apache.spark.sql.SparkSession
Accessor for nested Scala object
implicits() - Method in class org.apache.spark.sql.SQLContext
Accessor for nested Scala object
implicits$() - Constructor for class org.apache.spark.sql.SparkSession.implicits$
 
implicits$() - Constructor for class org.apache.spark.sql.SQLContext.implicits$
 
improveException(Object, NotSerializableException) - Static method in class org.apache.spark.serializer.SerializationDebugger
Improve the given NotSerializableException with the serialization path leading from the given object to the problematic object.
Impurities - Class in org.apache.spark.mllib.tree.impurity
Factory for Impurity instances.
Impurities() - Constructor for class org.apache.spark.mllib.tree.impurity.Impurities
 
impurity() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
impurity() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
impurity() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
impurity() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
impurity() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
impurity() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
impurity() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
impurity() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
impurity() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
impurity() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
impurity() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
impurity() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
impurity() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
impurity() - Method in class org.apache.spark.ml.tree.InternalNode
 
impurity() - Method in class org.apache.spark.ml.tree.LeafNode
 
impurity() - Method in class org.apache.spark.ml.tree.Node
Impurity measure at this node (for training data)
impurity() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
Impurity - Interface in org.apache.spark.mllib.tree.impurity
Trait for calculating information gain.
impurity() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
impurity() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
impurity() - Method in class org.apache.spark.mllib.tree.model.Node
 
impurityStats() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
Imputer - Class in org.apache.spark.ml.feature
Experimental Imputation estimator for completing missing values, either using the mean or the median of the columns in which the missing values are located.
Imputer(String) - Constructor for class org.apache.spark.ml.feature.Imputer
 
Imputer() - Constructor for class org.apache.spark.ml.feature.Imputer
 
ImputerModel - Class in org.apache.spark.ml.feature
Experimental Model fitted by Imputer.
In() - Static method in class org.apache.spark.graphx.EdgeDirection
Edges arriving at a vertex.
In - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to one of the values in the array.
In(String, Object[]) - Constructor for class org.apache.spark.sql.sources.In
 
INACTIVE() - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
inArray(Object) - Static method in class org.apache.spark.ml.param.ParamValidators
Check for value in an allowed set of values.
inArray(List<T>) - Static method in class org.apache.spark.ml.param.ParamValidators
Check for value in an allowed set of values.
InBlock$() - Constructor for class org.apache.spark.ml.recommendation.ALS.InBlock$
 
IncompatibleMergeException - Exception in org.apache.spark.util.sketch
 
IncompatibleMergeException(String) - Constructor for exception org.apache.spark.util.sketch.IncompatibleMergeException
 
incrementFetchedPartitions(int) - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
 
incrementFileCacheHits(int) - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
 
incrementFilesDiscovered(int) - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
 
incrementHiveClientCalls(int) - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
 
incrementParallelListingJobCount(int) - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
 
inDegrees() - Method in class org.apache.spark.graphx.GraphOps
The in-degree of each vertex in the graph.
independence() - Method in class org.apache.spark.mllib.stat.test.ChiSqTest.NullHypothesis$
 
INDETERMINATE() - Static method in class org.apache.spark.rdd.DeterministicLevel
 
index() - Method in class org.apache.spark.ml.attribute.Attribute
Index of the attribute.
INDEX() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
index() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
index() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
index() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
index() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
index(int, int) - Method in interface org.apache.spark.ml.linalg.Matrix
Return the index for the (i, j)-th element in the backing array.
index() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
index(int, int) - Method in interface org.apache.spark.mllib.linalg.Matrix
Return the index for the (i, j)-th element in the backing array.
index() - Method in interface org.apache.spark.Partition
Get the partition's index within its parent RDD
index() - Method in class org.apache.spark.scheduler.TaskInfo
The index of this task within its task set.
index() - Method in class org.apache.spark.status.api.v1.TaskData
 
IndexedRow - Class in org.apache.spark.mllib.linalg.distributed
Represents a row of IndexedRowMatrix.
IndexedRow(long, Vector) - Constructor for class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
IndexedRowMatrix - Class in org.apache.spark.mllib.linalg.distributed
Represents a row-oriented DistributedMatrix with indexed rows.
IndexedRowMatrix(RDD<IndexedRow>, long, int) - Constructor for class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
 
IndexedRowMatrix(RDD<IndexedRow>) - Constructor for class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Alternative constructor leaving matrix dimensions to be determined automatically.
indexName(String) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
indexOf(String) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Index of an attribute specified by name.
indexOf(String) - Method in class org.apache.spark.ml.attribute.NominalAttribute
Index of a specific value.
indexOf(Object) - Method in class org.apache.spark.mllib.feature.HashingTF
Returns the index of the input term.
indexOf(B) - Static method in class org.apache.spark.sql.types.StructType
 
indexOf(B, int) - Static method in class org.apache.spark.sql.types.StructType
 
indexOfSlice(GenSeq<B>) - Static method in class org.apache.spark.sql.types.StructType
 
indexOfSlice(GenSeq<B>, int) - Static method in class org.apache.spark.sql.types.StructType
 
indexToLevel(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Return the level of a tree which the given node is in.
IndexToString - Class in org.apache.spark.ml.feature
A Transformer that maps a column of indices back to a new column of corresponding string values.
IndexToString(String) - Constructor for class org.apache.spark.ml.feature.IndexToString
 
IndexToString() - Constructor for class org.apache.spark.ml.feature.IndexToString
 
indexWhere(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
indexWhere(Function1<A, Object>, int) - Static method in class org.apache.spark.sql.types.StructType
 
indices() - Method in class org.apache.spark.ml.feature.VectorSlicer
An array of indices to select features from a vector column.
indices() - Method in class org.apache.spark.ml.linalg.SparseVector
 
indices() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
indices() - Static method in class org.apache.spark.sql.types.StructType
 
inferSchema(SparkSession, Map<String, String>, Seq<FileStatus>) - Method in class org.apache.spark.sql.hive.execution.HiveFileFormat
 
inferSchema(CatalogTable) - Static method in class org.apache.spark.sql.hive.HiveUtils
Infers the schema for Hive serde tables and returns the CatalogTable with the inferred schema.
inferSchema(SparkSession, Map<String, String>, Seq<FileStatus>) - Method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
info() - Method in class org.apache.spark.status.LiveRDD
 
info() - Method in class org.apache.spark.status.LiveStage
 
info() - Method in class org.apache.spark.status.LiveTask
 
infoChanged(SparkAppHandle) - Method in interface org.apache.spark.launcher.SparkAppHandle.Listener
Callback for changes in any information that is not the handle's state.
infoGain() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
InformationGainStats - Class in org.apache.spark.mllib.tree.model
Developer API Information gain statistics for each split param: gain information gain value param: impurity current node impurity param: leftImpurity left node impurity param: rightImpurity right node impurity param: leftPredict left node predict param: rightPredict right node predict
InformationGainStats(double, double, double, double, Predict, Predict) - Constructor for class org.apache.spark.mllib.tree.model.InformationGainStats
 
init() - Static method in class org.apache.spark.sql.types.StructType
 
initcap(Column) - Static method in class org.apache.spark.sql.functions
Returns a new string column by converting the first letter of each word to uppercase.
initCause(Throwable) - Static method in exception org.apache.spark.sql.AnalysisException
 
initDaemon(Logger) - Static method in class org.apache.spark.util.Utils
Utility function that should be called early in main() for daemons to set up some common diagnostic state.
initHadoopOutputMetrics(TaskContext) - Static method in class org.apache.spark.internal.io.SparkHadoopWriterUtils
 
initialHash() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
initialize(double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
 
initialize(double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
 
initialize(double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
initialize(double, double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
 
initialize(RDD<Tuple2<Object, Vector>>, LDA) - Method in interface org.apache.spark.mllib.clustering.LDAOptimizer
Initializer for the optimizer.
initialize(MutableAggregationBuffer) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Initializes the given aggregation buffer, i.e.
Initialized() - Static method in class org.apache.spark.rdd.CheckpointState
 
initializeLogging(boolean, boolean) - Method in interface org.apache.spark.internal.Logging
 
initializeLogIfNecessary(boolean) - Method in interface org.apache.spark.internal.Logging
 
initializeLogIfNecessary(boolean, boolean) - Method in interface org.apache.spark.internal.Logging
 
initialState(RDD<Tuple2<KeyType, StateType>>) - Method in class org.apache.spark.streaming.StateSpec
Set the RDD containing the initial states that will be used by mapWithState
initialState(JavaPairRDD<KeyType, StateType>) - Method in class org.apache.spark.streaming.StateSpec
Set the RDD containing the initial states that will be used by mapWithState
initialValue() - Method in class org.apache.spark.partial.PartialResult
 
initialWeights() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
initInputSerDe(Seq<Expression>) - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
initMode() - Static method in class org.apache.spark.ml.clustering.KMeans
 
initMode() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
initOutputFormat(JobContext) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
initOutputSerDe(Seq<Attribute>) - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
inits() - Static method in class org.apache.spark.sql.types.StructType
 
initSteps() - Static method in class org.apache.spark.ml.clustering.KMeans
 
initSteps() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
initWriter(TaskAttemptContext, int) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
injectCheckRule(Function1<SparkSession, Function1<LogicalPlan, BoxedUnit>>) - Method in class org.apache.spark.sql.SparkSessionExtensions
Inject an check analysis Rule builder into the SparkSession.
injectOptimizerRule(Function1<SparkSession, Rule<LogicalPlan>>) - Method in class org.apache.spark.sql.SparkSessionExtensions
Inject an optimizer Rule builder into the SparkSession.
injectParser(Function2<SparkSession, ParserInterface, ParserInterface>) - Method in class org.apache.spark.sql.SparkSessionExtensions
Inject a custom parser into the SparkSession.
injectPlannerStrategy(Function1<SparkSession, SparkStrategy>) - Method in class org.apache.spark.sql.SparkSessionExtensions
Inject a planner Strategy builder into the SparkSession.
injectPostHocResolutionRule(Function1<SparkSession, Rule<LogicalPlan>>) - Method in class org.apache.spark.sql.SparkSessionExtensions
Inject an analyzer Rule builder into the SparkSession.
injectResolutionRule(Function1<SparkSession, Rule<LogicalPlan>>) - Method in class org.apache.spark.sql.SparkSessionExtensions
Inject an analyzer resolution Rule builder into the SparkSession.
InnerClosureFinder - Class in org.apache.spark.util
 
InnerClosureFinder(Set<Class<?>>) - Constructor for class org.apache.spark.util.InnerClosureFinder
 
innerJoin(EdgeRDD<ED2>, Function4<Object, Object, ED, ED2, ED3>, ClassTag<ED2>, ClassTag<ED3>) - Method in class org.apache.spark.graphx.EdgeRDD
Inner joins this EdgeRDD with another EdgeRDD, assuming both are partitioned using the same PartitionStrategy.
innerJoin(EdgeRDD<ED2>, Function4<Object, Object, ED, ED2, ED3>, ClassTag<ED2>, ClassTag<ED3>) - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
innerJoin(RDD<Tuple2<Object, U>>, Function3<Object, VD, U, VD2>, ClassTag<U>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
innerJoin(RDD<Tuple2<Object, U>>, Function3<Object, VD, U, VD2>, ClassTag<U>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.VertexRDD
Inner joins this VertexRDD with an RDD containing vertex attribute pairs.
innerZipJoin(VertexRDD<U>, Function3<Object, VD, U, VD2>, ClassTag<U>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
innerZipJoin(VertexRDD<U>, Function3<Object, VD, U, VD2>, ClassTag<U>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.VertexRDD
Efficiently inner joins this VertexRDD with another VertexRDD sharing the same index.
InProcessLauncher - Class in org.apache.spark.launcher
In-process launcher for Spark applications.
InProcessLauncher() - Constructor for class org.apache.spark.launcher.InProcessLauncher
 
input() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
INPUT() - Static method in class org.apache.spark.ui.ToolTips
 
input$() - Constructor for class org.apache.spark.InternalAccumulator.input$
 
input_file_name() - Static method in class org.apache.spark.sql.functions
Creates a string column for the file name of the current Spark task.
INPUT_FORMAT() - Static method in class org.apache.spark.sql.hive.execution.HiveOptions
 
INPUT_METRICS_PREFIX() - Static method in class org.apache.spark.InternalAccumulator
 
INPUT_RECORDS() - Static method in class org.apache.spark.status.TaskIndexNames
 
INPUT_SIZE() - Static method in class org.apache.spark.status.TaskIndexNames
 
inputBytes() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
inputBytes() - Method in class org.apache.spark.status.api.v1.StageData
 
inputCol() - Static method in class org.apache.spark.ml.feature.Binarizer
 
inputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
inputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
inputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
inputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.DCT
 
inputCol() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
inputCol() - Static method in class org.apache.spark.ml.feature.HashingTF
 
inputCol() - Static method in class org.apache.spark.ml.feature.IDF
 
inputCol() - Static method in class org.apache.spark.ml.feature.IDFModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.IndexToString
 
inputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
inputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
inputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
inputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.NGram
 
inputCol() - Static method in class org.apache.spark.ml.feature.Normalizer
 
inputCol() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
inputCol() - Static method in class org.apache.spark.ml.feature.PCA
 
inputCol() - Static method in class org.apache.spark.ml.feature.PCAModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
inputCol() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
inputCol() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
inputCol() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
inputCol() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
inputCol() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
inputCol() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
inputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
inputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
inputCol() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
inputCol() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
inputCol() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
inputCol() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
inputCol() - Method in interface org.apache.spark.ml.param.shared.HasInputCol
Param for input column name.
inputCols() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
inputCols() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
inputCols() - Static method in class org.apache.spark.ml.feature.Imputer
 
inputCols() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
inputCols() - Static method in class org.apache.spark.ml.feature.Interaction
 
inputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
inputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
inputCols() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
inputCols() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
inputCols() - Method in interface org.apache.spark.ml.param.shared.HasInputCols
Param for input column names.
inputDStream() - Method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
inputDStream() - Method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
inputDStream() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
inputDStream() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
InputDStream<T> - Class in org.apache.spark.streaming.dstream
This is the abstract base class for all input streams.
InputDStream(StreamingContext, ClassTag<T>) - Constructor for class org.apache.spark.streaming.dstream.InputDStream
 
InputFileBlockHolder - Class in org.apache.spark.rdd
This holds file names of the current Spark task.
InputFileBlockHolder() - Constructor for class org.apache.spark.rdd.InputFileBlockHolder
 
inputFiles() - Method in class org.apache.spark.sql.Dataset
Returns a best-effort snapshot of the files that compose this Dataset.
inputFormat() - Method in class org.apache.spark.sql.hive.execution.HiveOptions
 
inputFormatClazz() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
inputFormatClazz() - Method in class org.apache.spark.scheduler.SplitInfo
 
InputFormatInfo - Class in org.apache.spark.scheduler
Developer API Parses and holds information about inputFormat (and files) specified as a parameter.
InputFormatInfo(Configuration, Class<?>, String) - Constructor for class org.apache.spark.scheduler.InputFormatInfo
 
InputMetricDistributions - Class in org.apache.spark.status.api.v1
 
InputMetrics - Class in org.apache.spark.status.api.v1
 
inputMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
inputMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
inputRecords() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
inputRecords() - Method in class org.apache.spark.status.api.v1.StageData
 
inputRowFormat() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
inputRowFormatMap() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
inputRowsPerSecond() - Method in class org.apache.spark.sql.streaming.SourceProgress
 
inputRowsPerSecond() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
The aggregate (across all sources) rate of data arriving.
inputSchema() - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
A StructType represents data types of input arguments of this aggregate function.
inputSerdeClass() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
inputSerdeProps() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
inputSet() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
inputSet() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
inputSet() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
inputSet() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
inputSize() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
inputStreamId() - Method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
inputTypes() - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
inRange(double, double, boolean, boolean) - Static method in class org.apache.spark.ml.param.ParamValidators
Check for value in range lowerBound to upperBound.
inRange(double, double) - Static method in class org.apache.spark.ml.param.ParamValidators
Version of `inRange()` which uses inclusive be default: [lowerBound, upperBound]
insert(Dataset<Row>, boolean) - Method in interface org.apache.spark.sql.sources.InsertableRelation
 
InsertableRelation - Interface in org.apache.spark.sql.sources
A BaseRelation that can be used to insert data into it through the insert method.
insertInto(String) - Method in class org.apache.spark.sql.DataFrameWriter
Inserts the content of the DataFrame to the specified table.
InsertIntoHiveDirCommand - Class in org.apache.spark.sql.hive.execution
Command for writing the results of query to file system.
InsertIntoHiveDirCommand(boolean, CatalogStorageFormat, LogicalPlan, boolean, Seq<String>) - Constructor for class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
InsertIntoHiveTable - Class in org.apache.spark.sql.hive.execution
Command for writing data out to a Hive table.
InsertIntoHiveTable(CatalogTable, Map<String, Option<String>>, LogicalPlan, boolean, boolean, Seq<String>) - Constructor for class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
inShutdown() - Static method in class org.apache.spark.util.ShutdownHookManager
Detect whether this thread might be executing a shutdown hook.
inspectorToDataType(ObjectInspector) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
instance() - Static method in class org.apache.spark.mllib.tree.impurity.Entropy
Get this impurity instance.
instance() - Static method in class org.apache.spark.mllib.tree.impurity.Gini
Get this impurity instance.
instance() - Static method in class org.apache.spark.mllib.tree.impurity.Variance
Get this impurity instance.
INSTANCE - Static variable in class org.apache.spark.serializer.DummySerializerInstance
 
instantiate(String, String, String, boolean) - Static method in class org.apache.spark.internal.io.FileCommitProtocol
Instantiates a FileCommitProtocol using the given className.
instr(Column, String) - Static method in class org.apache.spark.sql.functions
Locate the position of the first occurrence of substr column in the given string.
INT() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable int type.
intAccumulator(int) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().longAccumulator(). Since 2.0.0.
intAccumulator(int, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Deprecated.
use sc().longAccumulator(String). Since 2.0.0.
IntAccumulatorParam$() - Constructor for class org.apache.spark.AccumulatorParam.IntAccumulatorParam$
Deprecated.
 
IntArrayParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Array[Int} for Java.
IntArrayParam(Params, String, String, Function1<int[], Object>) - Constructor for class org.apache.spark.ml.param.IntArrayParam
 
IntArrayParam(Params, String, String) - Constructor for class org.apache.spark.ml.param.IntArrayParam
 
IntegerType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the IntegerType object.
IntegerType - Class in org.apache.spark.sql.types
The data type representing Int values.
INTER_JOB_WAIT_MS() - Static method in class org.apache.spark.ui.UIWorkloadGenerator
 
Interaction - Class in org.apache.spark.ml.feature
Implements the feature interaction transform.
Interaction(String) - Constructor for class org.apache.spark.ml.feature.Interaction
 
Interaction() - Constructor for class org.apache.spark.ml.feature.Interaction
 
intercept() - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
intercept() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
The model intercept for "binomial" logistic regression.
intercept() - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
intercept() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
intercept() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
 
intercept() - Method in class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$.Data
 
intercept() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
intercept() - Method in class org.apache.spark.mllib.classification.SVMModel
 
intercept() - Method in class org.apache.spark.mllib.regression.GeneralizedLinearModel
 
intercept() - Method in class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$.Data
 
intercept() - Method in class org.apache.spark.mllib.regression.LassoModel
 
intercept() - Method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
intercept() - Method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
interceptVector() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
intermediateStorageLevel() - Static method in class org.apache.spark.ml.recommendation.ALS
 
InternalAccumulator - Class in org.apache.spark
A collection of fields and methods concerned with internal accumulators that represent task level metrics.
InternalAccumulator() - Constructor for class org.apache.spark.InternalAccumulator
 
InternalAccumulator.input$ - Class in org.apache.spark
 
InternalAccumulator.output$ - Class in org.apache.spark
 
InternalAccumulator.shuffleRead$ - Class in org.apache.spark
 
InternalAccumulator.shuffleWrite$ - Class in org.apache.spark
 
InternalNode - Class in org.apache.spark.ml.tree
Internal Decision Tree node.
InterruptibleIterator<T> - Class in org.apache.spark
Developer API An iterator that wraps around an existing iterator to provide task killing functionality.
InterruptibleIterator(TaskContext, Iterator<T>) - Constructor for class org.apache.spark.InterruptibleIterator
 
interruptThread() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask
 
interruptThread() - Method in class org.apache.spark.scheduler.local.KillTask
 
intersect(Dataset<T>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset containing rows only in both this Dataset and another Dataset.
intersect(GenSeq<B>) - Static method in class org.apache.spark.sql.types.StructType
 
intersection(JavaDoubleRDD) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return the intersection of this RDD and another one.
intersection(JavaPairRDD<K, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return the intersection of this RDD and another one.
intersection(JavaRDD<T>) - Method in class org.apache.spark.api.java.JavaRDD
Return the intersection of this RDD and another one.
intersection(RDD<T>) - Static method in class org.apache.spark.api.r.RRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.api.r.RRDD
 
intersection(RDD<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.graphx.EdgeRDD
 
intersection(RDD<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
intersection(RDD<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
intersection(RDD<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.graphx.VertexRDD
 
intersection(RDD<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.rdd.HadoopRDD
 
intersection(RDD<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.rdd.JdbcRDD
 
intersection(RDD<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
intersection(RDD<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
intersection(RDD<T>) - Method in class org.apache.spark.rdd.RDD
Return the intersection of this RDD and another one.
intersection(RDD<T>, Partitioner, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return the intersection of this RDD and another one.
intersection(RDD<T>, int) - Method in class org.apache.spark.rdd.RDD
Return the intersection of this RDD and another one.
intersection(RDD<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
intersection(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
intersection(RDD<T>, int) - Static method in class org.apache.spark.rdd.UnionRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.api.r.RRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.EdgeRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.VertexRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.HadoopRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.JdbcRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
intersection$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.UnionRDD
 
intervalMs() - Method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
IntParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Int] for Java.
IntParam(String, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.IntParam
 
IntParam(String, String, String) - Constructor for class org.apache.spark.ml.param.IntParam
 
IntParam(Identifiable, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.IntParam
 
IntParam(Identifiable, String, String) - Constructor for class org.apache.spark.ml.param.IntParam
 
IntParam - Class in org.apache.spark.util
An extractor object for parsing strings into integers.
IntParam() - Constructor for class org.apache.spark.util.IntParam
 
invalidateSerializedMapOutputStatusCache() - Method in class org.apache.spark.ShuffleStatus
Clears the cached serialized map output statuses.
invalidateStatsCache() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
invalidateStatsCache() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
invalidateStatsCache() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
inverse() - Method in class org.apache.spark.ml.feature.DCT
Indicates whether to perform the inverse DCT (true) or forward DCT (false).
inverse(double[], int) - Static method in class org.apache.spark.mllib.linalg.CholeskyDecomposition
Computes the inverse of a real symmetric positive definite matrix A using the Cholesky factorization A = U**T*U.
Inverse$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Inverse$
 
invoke(Class<?>, Object, String, Seq<Tuple2<Class<?>, Object>>) - Static method in class org.apache.spark.util.Utils
 
invokedMethod(Object, Class<?>, String) - Static method in class org.apache.spark.graphx.util.BytecodeUtils
Test whether the given closure invokes the specified method in the specified class.
invokeWriteReplace(Object) - Method in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
ioEncryptionKey() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SparkAppConfig
 
ioschema() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
is32BitDecimalType(DataType) - Static method in class org.apache.spark.sql.types.DecimalType
Returns if dt is a DecimalType that fits inside an int
is64BitDecimalType(DataType) - Static method in class org.apache.spark.sql.types.DecimalType
Returns if dt is a DecimalType that fits inside a long
isActive() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns true if this query is actively running.
isActive() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
isActive() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
isActive() - Method in class org.apache.spark.status.LiveExecutor
 
isAddIntercept() - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
 
isAddIntercept() - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
 
isAddIntercept() - Method in class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
Get if the algorithm uses addIntercept
isAddIntercept() - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
 
isAddIntercept() - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
 
isAddIntercept() - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
 
isAllowed(Enumeration.Value, Enumeration.Value) - Static method in class org.apache.spark.scheduler.TaskLocality
 
isBatchingEnabled(SparkConf, boolean) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
isBindCollision(Throwable) - Static method in class org.apache.spark.util.Utils
Return whether the exception is caused by an address-port collision when binding.
isBlacklisted() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
isBlacklisted() - Method in class org.apache.spark.status.LiveExecutor
 
isBroadcast() - Method in class org.apache.spark.storage.BlockId
 
isBroadcast() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
isBroadcast() - Static method in class org.apache.spark.storage.RDDBlockId
 
isBroadcast() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
isBroadcast() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
isBroadcast() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
isBroadcast() - Static method in class org.apache.spark.storage.StreamBlockId
 
isBroadcast() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
isBucket() - Method in class org.apache.spark.sql.catalog.Column
 
isByteArrayDecimalType(DataType) - Static method in class org.apache.spark.sql.types.DecimalType
Returns if dt is a DecimalType that doesn't fit inside a long
isCached(String) - Method in class org.apache.spark.sql.catalog.Catalog
Returns true if the table is currently cached in-memory.
isCached(String) - Method in class org.apache.spark.sql.SQLContext
Returns true if the table is currently cached in-memory.
isCached() - Method in class org.apache.spark.storage.BlockStatus
 
isCached() - Method in class org.apache.spark.storage.RDDInfo
 
isCancelled() - Method in class org.apache.spark.ComplexFutureAction
 
isCancelled() - Method in interface org.apache.spark.FutureAction
Returns whether the action has been cancelled.
isCancelled() - Method in class org.apache.spark.SimpleFutureAction
 
isCascadingTruncateTable() - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
isCascadingTruncateTable() - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Return Some[true] iff TRUNCATE TABLE causes cascading default.
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
isCascadingTruncateTable() - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
isCheckpointed() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
isCheckpointed() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
isCheckpointed() - Static method in class org.apache.spark.api.java.JavaRDD
 
isCheckpointed() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return whether this RDD has been checkpointed or not
isCheckpointed() - Static method in class org.apache.spark.api.r.RRDD
 
isCheckpointed() - Static method in class org.apache.spark.graphx.EdgeRDD
 
isCheckpointed() - Method in class org.apache.spark.graphx.Graph
Return whether this Graph has been checkpointed or not.
isCheckpointed() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
isCheckpointed() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
isCheckpointed() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
isCheckpointed() - Static method in class org.apache.spark.graphx.VertexRDD
 
isCheckpointed() - Static method in class org.apache.spark.rdd.HadoopRDD
 
isCheckpointed() - Static method in class org.apache.spark.rdd.JdbcRDD
 
isCheckpointed() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
isCheckpointed() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
isCheckpointed() - Method in class org.apache.spark.rdd.RDD
Return whether this RDD is checkpointed and materialized, either reliably or locally.
isCheckpointed() - Static method in class org.apache.spark.rdd.UnionRDD
 
isCliSessionState() - Static method in class org.apache.spark.sql.hive.HiveUtils
Check current Thread's SessionState type
isColMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Indicates whether the values backing this matrix are arranged in column major order.
isCompatible(BloomFilter) - Method in class org.apache.spark.util.sketch.BloomFilter
Determines whether a given bloom filter is compatible with this bloom filter.
isCompleted() - Method in class org.apache.spark.ComplexFutureAction
 
isCompleted() - Method in interface org.apache.spark.FutureAction
Returns whether the action has already been completed with a value or an exception.
isCompleted() - Method in class org.apache.spark.SimpleFutureAction
 
isCompleted() - Method in class org.apache.spark.TaskContext
Returns true if the task has completed.
isDataAvailable() - Method in class org.apache.spark.sql.streaming.StreamingQueryStatus
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.LDA
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.DCT
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.IDF
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Imputer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Interaction
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.NGram
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.PCA
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormula
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
isDefined(Param<?>) - Method in interface org.apache.spark.ml.param.Params
Checks whether a param is explicitly set or has a default value.
isDefined(Param<?>) - Static method in class org.apache.spark.ml.Pipeline
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.PipelineModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
isDefined(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
isDefinedAt(A) - Static method in class org.apache.spark.sql.types.StructType
 
isDistributed() - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
isDistributed() - Method in class org.apache.spark.ml.clustering.LDAModel
Indicates whether this instance is of type DistributedLDAModel
isDistributed() - Method in class org.apache.spark.ml.clustering.LocalLDAModel
 
isDriver() - Method in class org.apache.spark.storage.BlockManagerId
 
isDynamicAllocationEnabled(SparkConf) - Static method in class org.apache.spark.util.Utils
Return whether dynamic allocation is enabled in the given conf.
isEmpty() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
isEmpty() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
isEmpty() - Static method in class org.apache.spark.api.java.JavaRDD
 
isEmpty() - Method in interface org.apache.spark.api.java.JavaRDDLike
 
isEmpty() - Static method in class org.apache.spark.api.r.RRDD
 
isEmpty() - Static method in class org.apache.spark.graphx.EdgeRDD
 
isEmpty() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
isEmpty() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
isEmpty() - Static method in class org.apache.spark.graphx.VertexRDD
 
isEmpty() - Static method in class org.apache.spark.rdd.HadoopRDD
 
isEmpty() - Static method in class org.apache.spark.rdd.JdbcRDD
 
isEmpty() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
isEmpty() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
isEmpty() - Method in class org.apache.spark.rdd.RDD
 
isEmpty() - Static method in class org.apache.spark.rdd.UnionRDD
 
isEmpty() - Static method in class org.apache.spark.sql.types.StructType
 
isExecutorStartupConf(String) - Static method in class org.apache.spark.SparkConf
Return whether the given config should be passed to an executor on start-up.
isExperiment() - Method in class org.apache.spark.mllib.stat.test.BinarySample
 
isFailed(Enumeration.Value) - Static method in class org.apache.spark.TaskState
 
isFatalError(Throwable) - Static method in class org.apache.spark.util.Utils
Returns true if the given exception was fatal.
isFile(Path) - Static method in class org.apache.spark.ml.image.SamplePathFilter
 
isFinal() - Method in enum org.apache.spark.launcher.SparkAppHandle.State
Whether this state is a final state, meaning the application is not running anymore once it's reached.
isFinished(Enumeration.Value) - Static method in class org.apache.spark.TaskState
 
isImmutable() - Static method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
isin(Object...) - Method in class org.apache.spark.sql.Column
A boolean expression that is evaluated to true if the value of this expression is contained by the evaluated values of the arguments.
isin(Seq<Object>) - Method in class org.apache.spark.sql.Column
A boolean expression that is evaluated to true if the value of this expression is contained by the evaluated values of the arguments.
isInDirectory(File, File) - Static method in class org.apache.spark.util.Utils
Return whether the specified file is a parent directory of the child file.
isInitialValueFinal() - Method in class org.apache.spark.partial.PartialResult
 
isInterrupted() - Method in class org.apache.spark.TaskContext
Returns true if the task has been killed.
isLargerBetter() - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
isLargerBetter() - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
isLargerBetter() - Method in class org.apache.spark.ml.evaluation.Evaluator
Indicates whether the metric returned by evaluate should be maximized (true, default) or minimized (false).
isLargerBetter() - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
isLargerBetter() - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
isLeaf() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
isLeaf() - Method in class org.apache.spark.mllib.tree.model.Node
 
isLeftChild(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Returns true if this is a left child.
isLocal() - Method in class org.apache.spark.api.java.JavaSparkContext
 
isLocal() - Method in class org.apache.spark.SparkContext
 
isLocal() - Method in class org.apache.spark.sql.Dataset
Returns true if the collect and take methods can be run locally (without any Spark executors).
isLocal() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
isLocalMaster(SparkConf) - Static method in class org.apache.spark.util.Utils
 
isMac() - Static method in class org.apache.spark.util.Utils
Whether the underlying operating system is Mac OS X.
isMulticlassClassification() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
isMulticlassWithCategoricalFeatures() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
isMultipleOf(Duration) - Method in class org.apache.spark.streaming.Duration
 
isMultipleOf(Duration) - Method in class org.apache.spark.streaming.Time
 
isNaN() - Method in class org.apache.spark.sql.Column
True if the current expression is NaN.
isnan(Column) - Static method in class org.apache.spark.sql.functions
Return true iff the column is NaN.
isNominal() - Method in class org.apache.spark.ml.attribute.Attribute
Tests whether this attribute is nominal, true for NominalAttribute and BinaryAttribute.
isNominal() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
isNominal() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
isNominal() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
isNominal() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
isNotNull() - Method in class org.apache.spark.sql.Column
True if the current expression is NOT null.
IsNotNull - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a non-null value.
IsNotNull(String) - Constructor for class org.apache.spark.sql.sources.IsNotNull
 
isNull() - Method in class org.apache.spark.sql.Column
True if the current expression is null.
isnull(Column) - Static method in class org.apache.spark.sql.functions
Return true iff the column is null.
IsNull - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to null.
IsNull(String) - Constructor for class org.apache.spark.sql.sources.IsNull
 
isNullAt(int) - Method in interface org.apache.spark.sql.Row
Checks whether the value at position i is null.
isNullAt(int) - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
isNullAt(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
isNullAt(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
isNullAt(int) - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns whether the value at rowId is NULL.
isNumeric() - Method in class org.apache.spark.ml.attribute.Attribute
Tests whether this attribute is numeric, true for NumericAttribute and BinaryAttribute.
isNumeric() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
isNumeric() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
isNumeric() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
isNumeric() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
isOpen() - Method in class org.apache.spark.storage.CountingWritableChannel
 
isOrdinal() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
isotonic() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
isotonic() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
isotonic() - Method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
 
IsotonicRegression - Class in org.apache.spark.ml.regression
Isotonic regression.
IsotonicRegression(String) - Constructor for class org.apache.spark.ml.regression.IsotonicRegression
 
IsotonicRegression() - Constructor for class org.apache.spark.ml.regression.IsotonicRegression
 
IsotonicRegression - Class in org.apache.spark.mllib.regression
Isotonic regression.
IsotonicRegression() - Constructor for class org.apache.spark.mllib.regression.IsotonicRegression
Constructs IsotonicRegression instance with default parameter isotonic = true.
IsotonicRegressionModel - Class in org.apache.spark.ml.regression
Model fitted by IsotonicRegression.
IsotonicRegressionModel - Class in org.apache.spark.mllib.regression
Regression model for isotonic regression.
IsotonicRegressionModel(double[], double[], boolean) - Constructor for class org.apache.spark.mllib.regression.IsotonicRegressionModel
 
IsotonicRegressionModel(Iterable<Object>, Iterable<Object>, Boolean) - Constructor for class org.apache.spark.mllib.regression.IsotonicRegressionModel
A Java-friendly constructor that takes two Iterable parameters and one Boolean parameter.
isOutputSpecValidationEnabled(SparkConf) - Static method in class org.apache.spark.internal.io.SparkHadoopWriterUtils
 
isPartition() - Method in class org.apache.spark.sql.catalog.Column
 
isPresent() - Method in class org.apache.spark.api.java.Optional
 
isRDD() - Method in class org.apache.spark.storage.BlockId
 
isRDD() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
isRDD() - Static method in class org.apache.spark.storage.RDDBlockId
 
isRDD() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
isRDD() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
isRDD() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
isRDD() - Static method in class org.apache.spark.storage.StreamBlockId
 
isRDD() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
isRegistered() - Method in class org.apache.spark.util.AccumulatorV2
Returns true if this accumulator has been registered.
isRInstalled() - Static method in class org.apache.spark.api.r.RUtils
Check if R is installed before running tests that use R commands.
isRowMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Indicates whether the values backing this matrix are arranged in row major order.
isRunningLocally() - Method in class org.apache.spark.TaskContext
Deprecated.
Local execution was removed, so this always returns false. Since 2.0.0.
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeans
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.LDA
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Binarizer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.DCT
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.HashingTF
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.IDF
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Imputer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.IndexToString
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Interaction
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.NGram
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.PCA
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormula
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
isSet(Param<?>) - Method in interface org.apache.spark.ml.param.Params
Checks whether a param is explicitly set.
isSet(Param<?>) - Static method in class org.apache.spark.ml.Pipeline
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.PipelineModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALS
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
isSet(Param<?>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
isShuffle() - Method in class org.apache.spark.storage.BlockId
 
isShuffle() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
isShuffle() - Static method in class org.apache.spark.storage.RDDBlockId
 
isShuffle() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
isShuffle() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
isShuffle() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
isShuffle() - Static method in class org.apache.spark.storage.StreamBlockId
 
isShuffle() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
isSparkPortConf(String) - Static method in class org.apache.spark.SparkConf
Return true if the given config matches either spark.*.port or spark.port.*.
isSparkRInstalled() - Static method in class org.apache.spark.api.r.RUtils
Check if SparkR is installed before running tests that use SparkR.
isSplitable(SparkSession, Map<String, String>, Path) - Method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
isStarted() - Method in class org.apache.spark.streaming.receiver.Receiver
Check if the receiver has started or not.
isStopped() - Method in class org.apache.spark.SparkContext
 
isStopped() - Method in class org.apache.spark.streaming.receiver.Receiver
Check if receiver has been marked for stopping.
isStreaming() - Method in class org.apache.spark.sql.Dataset
Returns true if this Dataset contains one or more sources that continuously return data as it arrives.
isStreaming() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
isStreaming() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
isStreaming() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
isSymlink(File) - Static method in class org.apache.spark.util.Utils
Check to see if file is a symbolic link.
isTemporary() - Method in class org.apache.spark.sql.catalog.Function
 
isTemporary() - Method in class org.apache.spark.sql.catalog.Table
 
isTesting() - Static method in class org.apache.spark.util.Utils
Indicates whether Spark is currently running unit tests.
isTimingOut() - Method in class org.apache.spark.streaming.State
Whether the state is timing out and going to be removed by the system after the current batch.
isTraceEnabled() - Method in interface org.apache.spark.internal.Logging
 
isTransposed() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
isTransposed() - Method in interface org.apache.spark.ml.linalg.Matrix
Flag that keeps track whether the matrix is transposed or not.
isTransposed() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
isTransposed() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
isTransposed() - Method in interface org.apache.spark.mllib.linalg.Matrix
Flag that keeps track whether the matrix is transposed or not.
isTransposed() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
isTraversableAgain() - Static method in class org.apache.spark.sql.types.StructType
 
isTriggerActive() - Method in class org.apache.spark.sql.streaming.StreamingQueryStatus
 
isValid() - Static method in class org.apache.spark.ml.param.DoubleParam
 
isValid() - Static method in class org.apache.spark.ml.param.FloatParam
 
isValid() - Method in class org.apache.spark.ml.param.Param
 
isValid() - Method in class org.apache.spark.storage.StorageLevel
 
isWindows() - Static method in class org.apache.spark.util.Utils
Whether the underlying operating system is Windows.
isZero() - Method in class org.apache.spark.sql.types.Decimal
 
isZero() - Method in class org.apache.spark.streaming.Duration
 
isZero() - Method in class org.apache.spark.util.AccumulatorV2
Returns if this accumulator is zero value or not.
isZero() - Method in class org.apache.spark.util.CollectionAccumulator
Returns false if this accumulator instance has any values in it.
isZero() - Method in class org.apache.spark.util.DoubleAccumulator
Returns false if this accumulator has had any values added to it or the sum is non-zero.
isZero() - Method in class org.apache.spark.util.LegacyAccumulatorWrapper
 
isZero() - Method in class org.apache.spark.util.LongAccumulator
Returns false if this accumulator has had any values added to it or the sum is non-zero.
item() - Method in class org.apache.spark.ml.recommendation.ALS.Rating
 
itemCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
itemCol() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
itemFactors() - Method in class org.apache.spark.ml.recommendation.ALSModel
 
items() - Method in class org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
 
itemsCol() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
itemsCol() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.api.java.JavaRDD
 
iterator(Partition, TaskContext) - Method in interface org.apache.spark.api.java.JavaRDDLike
Internal method to this RDD; will read from cache if applicable, or otherwise compute it.
iterator(Partition, TaskContext) - Static method in class org.apache.spark.api.r.RRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.graphx.EdgeRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.graphx.VertexRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.rdd.HadoopRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.rdd.JdbcRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
iterator(Partition, TaskContext) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
iterator(Partition, TaskContext) - Method in class org.apache.spark.rdd.RDD
Internal method to this RDD; will read from cache if applicable, or otherwise compute it.
iterator(Partition, TaskContext) - Static method in class org.apache.spark.rdd.UnionRDD
 
iterator() - Method in class org.apache.spark.sql.types.StructType
 
iterator() - Method in class org.apache.spark.status.RDDPartitionSeq
 
IV_LENGTH_IN_BYTES() - Static method in class org.apache.spark.security.CryptoStreamUtils
 

J

j() - Method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
jarOfClass(Class<?>) - Static method in class org.apache.spark.api.java.JavaSparkContext
Find the JAR from which a given class was loaded, to make it easy for users to pass their JARs to SparkContext.
jarOfClass(Class<?>) - Static method in class org.apache.spark.SparkContext
Find the JAR from which a given class was loaded, to make it easy for users to pass their JARs to SparkContext.
jarOfClass(Class<?>) - Static method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Find the JAR from which a given class was loaded, to make it easy for users to pass their JARs to StreamingContext.
jarOfClass(Class<?>) - Static method in class org.apache.spark.streaming.StreamingContext
Find the JAR from which a given class was loaded, to make it easy for users to pass their JARs to StreamingContext.
jarOfObject(Object) - Static method in class org.apache.spark.api.java.JavaSparkContext
Find the JAR that contains the class of a particular object, to make it easy for users to pass their JARs to SparkContext.
jarOfObject(Object) - Static method in class org.apache.spark.SparkContext
Find the JAR that contains the class of a particular object, to make it easy for users to pass their JARs to SparkContext.
jars() - Method in class org.apache.spark.api.java.JavaSparkContext
 
jars() - Method in class org.apache.spark.SparkContext
 
javaAntecedent() - Method in class org.apache.spark.mllib.fpm.AssociationRules.Rule
Returns antecedent in a Java List.
javaCategoryMaps() - Method in class org.apache.spark.ml.feature.VectorIndexerModel
Java-friendly version of categoryMaps
javaConsequent() - Method in class org.apache.spark.mllib.fpm.AssociationRules.Rule
Returns consequent in a Java List.
JavaDoubleRDD - Class in org.apache.spark.api.java
 
JavaDoubleRDD(RDD<Object>) - Constructor for class org.apache.spark.api.java.JavaDoubleRDD
 
JavaDStream<T> - Class in org.apache.spark.streaming.api.java
A Java-friendly interface to DStream, the basic abstraction in Spark Streaming that represents a continuous stream of data.
JavaDStream(DStream<T>, ClassTag<T>) - Constructor for class org.apache.spark.streaming.api.java.JavaDStream
 
JavaDStreamLike<T,This extends JavaDStreamLike<T,This,R>,R extends JavaRDDLike<T,R>> - Interface in org.apache.spark.streaming.api.java
 
JavaFutureAction<T> - Interface in org.apache.spark.api.java
 
JavaHadoopRDD<K,V> - Class in org.apache.spark.api.java
 
JavaHadoopRDD(HadoopRDD<K, V>, ClassTag<K>, ClassTag<V>) - Constructor for class org.apache.spark.api.java.JavaHadoopRDD
 
javaHome() - Method in class org.apache.spark.status.api.v1.RuntimeInfo
 
JavaInputDStream<T> - Class in org.apache.spark.streaming.api.java
A Java-friendly interface to InputDStream.
JavaInputDStream(InputDStream<T>, ClassTag<T>) - Constructor for class org.apache.spark.streaming.api.java.JavaInputDStream
 
javaItems() - Method in class org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
Returns items in a Java List.
JavaIterableWrapperSerializer - Class in org.apache.spark.serializer
A Kryo serializer for serializing results returned by asJavaIterable.
JavaIterableWrapperSerializer() - Constructor for class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
JavaMapWithStateDStream<KeyType,ValueType,StateType,MappedType> - Class in org.apache.spark.streaming.api.java
Experimental DStream representing the stream of data generated by mapWithState operation on a JavaPairDStream.
JavaNewHadoopRDD<K,V> - Class in org.apache.spark.api.java
 
JavaNewHadoopRDD(NewHadoopRDD<K, V>, ClassTag<K>, ClassTag<V>) - Constructor for class org.apache.spark.api.java.JavaNewHadoopRDD
 
javaOcvTypes() - Static method in class org.apache.spark.ml.image.ImageSchema
(Java-specific) OpenCV type mapping supported
JavaPackage - Class in org.apache.spark.mllib
A dummy class as a workaround to show the package doc of spark.mllib in generated Java API docs.
JavaPairDStream<K,V> - Class in org.apache.spark.streaming.api.java
A Java-friendly interface to a DStream of key-value pairs, which provides extra methods like reduceByKey and join.
JavaPairDStream(DStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Constructor for class org.apache.spark.streaming.api.java.JavaPairDStream
 
JavaPairInputDStream<K,V> - Class in org.apache.spark.streaming.api.java
A Java-friendly interface to InputDStream of key-value pairs.
JavaPairInputDStream(InputDStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Constructor for class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
JavaPairRDD<K,V> - Class in org.apache.spark.api.java
 
JavaPairRDD(RDD<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Constructor for class org.apache.spark.api.java.JavaPairRDD
 
JavaPairReceiverInputDStream<K,V> - Class in org.apache.spark.streaming.api.java
A Java-friendly interface to ReceiverInputDStream, the abstract class for defining any input stream that receives data over the network.
JavaPairReceiverInputDStream(ReceiverInputDStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Constructor for class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
JavaParams - Class in org.apache.spark.ml.param
Developer API Java-friendly wrapper for Params.
JavaParams() - Constructor for class org.apache.spark.ml.param.JavaParams
 
JavaRDD<T> - Class in org.apache.spark.api.java
 
JavaRDD(RDD<T>, ClassTag<T>) - Constructor for class org.apache.spark.api.java.JavaRDD
 
javaRDD() - Method in class org.apache.spark.sql.Dataset
Returns the content of the Dataset as a JavaRDD of Ts.
JavaRDDLike<T,This extends JavaRDDLike<T,This>> - Interface in org.apache.spark.api.java
Defines operations common to several Java RDD implementations.
JavaReceiverInputDStream<T> - Class in org.apache.spark.streaming.api.java
A Java-friendly interface to ReceiverInputDStream, the abstract class for defining any input stream that receives data over the network.
JavaReceiverInputDStream(ReceiverInputDStream<T>, ClassTag<T>) - Constructor for class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
javaSequence() - Method in class org.apache.spark.mllib.fpm.PrefixSpan.FreqSequence
Returns sequence as a Java List of lists for Java users.
javaSerialization(ClassTag<T>) - Static method in class org.apache.spark.sql.Encoders
(Scala-specific) Creates an encoder that serializes objects of type T using generic Java serialization.
javaSerialization(Class<T>) - Static method in class org.apache.spark.sql.Encoders
Creates an encoder that serializes objects of type T using generic Java serialization.
JavaSerializer - Class in org.apache.spark.serializer
Developer API A Spark serializer that uses Java's built-in serialization.
JavaSerializer(SparkConf) - Constructor for class org.apache.spark.serializer.JavaSerializer
 
JavaSparkContext - Class in org.apache.spark.api.java
A Java-friendly version of SparkContext that returns JavaRDDs and works with Java collections instead of Scala ones.
JavaSparkContext(SparkContext) - Constructor for class org.apache.spark.api.java.JavaSparkContext
 
JavaSparkContext() - Constructor for class org.apache.spark.api.java.JavaSparkContext
Create a JavaSparkContext that loads settings from system properties (for instance, when launching with ./bin/spark-submit).
JavaSparkContext(SparkConf) - Constructor for class org.apache.spark.api.java.JavaSparkContext
 
JavaSparkContext(String, String) - Constructor for class org.apache.spark.api.java.JavaSparkContext
 
JavaSparkContext(String, String, SparkConf) - Constructor for class org.apache.spark.api.java.JavaSparkContext
 
JavaSparkContext(String, String, String, String) - Constructor for class org.apache.spark.api.java.JavaSparkContext
 
JavaSparkContext(String, String, String, String[]) - Constructor for class org.apache.spark.api.java.JavaSparkContext
 
JavaSparkContext(String, String, String, String[], Map<String, String>) - Constructor for class org.apache.spark.api.java.JavaSparkContext
 
JavaSparkStatusTracker - Class in org.apache.spark.api.java
Low-level status reporting APIs for monitoring job and stage progress.
JavaStreamingContext - Class in org.apache.spark.streaming.api.java
A Java-friendly version of StreamingContext which is the main entry point for Spark Streaming functionality.
JavaStreamingContext(StreamingContext) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
 
JavaStreamingContext(String, String, Duration) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a StreamingContext.
JavaStreamingContext(String, String, Duration, String, String) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a StreamingContext.
JavaStreamingContext(String, String, Duration, String, String[]) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a StreamingContext.
JavaStreamingContext(String, String, Duration, String, String[], Map<String, String>) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a StreamingContext.
JavaStreamingContext(JavaSparkContext, Duration) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a JavaStreamingContext using an existing JavaSparkContext.
JavaStreamingContext(SparkConf, Duration) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a JavaStreamingContext using a SparkConf configuration.
JavaStreamingContext(String) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Recreate a JavaStreamingContext from a checkpoint file.
JavaStreamingContext(String, Configuration) - Constructor for class org.apache.spark.streaming.api.java.JavaStreamingContext
Re-creates a JavaStreamingContext from a checkpoint file.
javaTopicAssignments() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Java-friendly version of topicAssignments
javaTopicDistributions() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Java-friendly version of topicDistributions
javaTopTopicsPerDocument(int) - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Java-friendly version of topTopicsPerDocument
javaTypeToDataType(Type) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
JavaUtils - Class in org.apache.spark.api.java
 
JavaUtils() - Constructor for class org.apache.spark.api.java.JavaUtils
 
JavaUtils.SerializableMapWrapper<A,B> - Class in org.apache.spark.api.java
 
javaVersion() - Method in class org.apache.spark.status.api.v1.RuntimeInfo
 
jdbc(String, String, Properties) - Method in class org.apache.spark.sql.DataFrameReader
Construct a DataFrame representing the database table accessible via JDBC URL url named table and connection properties.
jdbc(String, String, String, long, long, int, Properties) - Method in class org.apache.spark.sql.DataFrameReader
Construct a DataFrame representing the database table accessible via JDBC URL url named table.
jdbc(String, String, String[], Properties) - Method in class org.apache.spark.sql.DataFrameReader
Construct a DataFrame representing the database table accessible via JDBC URL url named table using connection properties.
jdbc(String, String, Properties) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame to an external database table via JDBC.
jdbc(String, String) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().jdbc().
jdbc(String, String, String, long, long, int) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().jdbc().
jdbc(String, String, String[]) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().jdbc().
JdbcDialect - Class in org.apache.spark.sql.jdbc
Developer API Encapsulates everything (extensions, workarounds, quirks) to handle the SQL dialect of a certain database or jdbc driver.
JdbcDialect() - Constructor for class org.apache.spark.sql.jdbc.JdbcDialect
 
JdbcDialects - Class in org.apache.spark.sql.jdbc
Developer API Registry of dialects that apply to every new jdbc org.apache.spark.sql.DataFrame.
JdbcDialects() - Constructor for class org.apache.spark.sql.jdbc.JdbcDialects
 
jdbcNullType() - Method in class org.apache.spark.sql.jdbc.JdbcType
 
JdbcRDD<T> - Class in org.apache.spark.rdd
An RDD that executes a SQL query on a JDBC connection and reads results.
JdbcRDD(SparkContext, Function0<Connection>, String, long, long, int, Function1<ResultSet, T>, ClassTag<T>) - Constructor for class org.apache.spark.rdd.JdbcRDD
 
JdbcRDD.ConnectionFactory - Interface in org.apache.spark.rdd
 
JdbcType - Class in org.apache.spark.sql.jdbc
Developer API A database type definition coupled with the jdbc type needed to send null values to the database.
JdbcType(String, int) - Constructor for class org.apache.spark.sql.jdbc.JdbcType
 
JettyUtils - Class in org.apache.spark.ui
Utilities for launching a web server using Jetty's HTTP Server class
JettyUtils() - Constructor for class org.apache.spark.ui.JettyUtils
 
JettyUtils.ServletParams<T> - Class in org.apache.spark.ui
 
JettyUtils.ServletParams$ - Class in org.apache.spark.ui
 
JOB_DAG() - Static method in class org.apache.spark.ui.ToolTips
 
JOB_TIMELINE() - Static method in class org.apache.spark.ui.ToolTips
 
JobData - Class in org.apache.spark.status.api.v1
 
jobEndFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
jobEndToJson(SparkListenerJobEnd) - Static method in class org.apache.spark.util.JsonProtocol
 
JobExecutionStatus - Enum in org.apache.spark
 
jobGroup() - Method in class org.apache.spark.status.api.v1.JobData
 
jobId() - Method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
jobId() - Method in class org.apache.spark.scheduler.SparkListenerJobStart
 
jobId() - Method in interface org.apache.spark.SparkJobInfo
 
jobId() - Method in class org.apache.spark.SparkJobInfoImpl
 
jobId() - Method in class org.apache.spark.status.api.v1.JobData
 
jobId() - Method in class org.apache.spark.status.LiveJob
 
jobID() - Method in class org.apache.spark.TaskCommitDenied
 
jobIds() - Method in interface org.apache.spark.api.java.JavaFutureAction
Returns the job IDs run by the underlying async operation.
jobIds() - Method in class org.apache.spark.ComplexFutureAction
 
jobIds() - Method in interface org.apache.spark.FutureAction
Returns the job IDs run by the underlying async operation.
jobIds() - Method in class org.apache.spark.SimpleFutureAction
 
jobIds() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
jobIds() - Method in class org.apache.spark.status.LiveStage
 
JobResult - Interface in org.apache.spark.scheduler
Developer API A result of a job in the DAGScheduler.
jobResult() - Method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
jobResultFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
jobResultToJson(JobResult) - Static method in class org.apache.spark.util.JsonProtocol
 
jobs() - Method in class org.apache.spark.status.LiveStage
 
jobStartFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
jobStartToJson(SparkListenerJobStart) - Static method in class org.apache.spark.util.JsonProtocol
 
JobSubmitter - Interface in org.apache.spark
Handle via which a "run" function passed to a ComplexFutureAction can submit jobs for execution.
JobSucceeded - Class in org.apache.spark.scheduler
 
JobSucceeded() - Constructor for class org.apache.spark.scheduler.JobSucceeded
 
join(JavaPairRDD<K, W>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD containing all pairs of elements with matching keys in this and other.
join(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD containing all pairs of elements with matching keys in this and other.
join(JavaPairRDD<K, W>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD containing all pairs of elements with matching keys in this and other.
join(RDD<Tuple2<K, W>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD containing all pairs of elements with matching keys in this and other.
join(RDD<Tuple2<K, W>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD containing all pairs of elements with matching keys in this and other.
join(RDD<Tuple2<K, W>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD containing all pairs of elements with matching keys in this and other.
join(Dataset<?>) - Method in class org.apache.spark.sql.Dataset
Join with another DataFrame.
join(Dataset<?>, String) - Method in class org.apache.spark.sql.Dataset
Inner equi-join with another DataFrame using the given column.
join(Dataset<?>, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Inner equi-join with another DataFrame using the given columns.
join(Dataset<?>, Seq<String>, String) - Method in class org.apache.spark.sql.Dataset
Equi-join with another DataFrame using the given columns.
join(Dataset<?>, Column) - Method in class org.apache.spark.sql.Dataset
Inner join with another DataFrame, using the given join expression.
join(Dataset<?>, Column, String) - Method in class org.apache.spark.sql.Dataset
Join with another DataFrame, using the given join expression.
join(JavaPairDStream<K, W>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'join' between RDDs of this DStream and other DStream.
join(JavaPairDStream<K, W>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'join' between RDDs of this DStream and other DStream.
join(JavaPairDStream<K, W>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'join' between RDDs of this DStream and other DStream.
join(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
join(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
join(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
join(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
join(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
join(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
join(DStream<Tuple2<K, W>>, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'join' between RDDs of this DStream and other DStream.
join(DStream<Tuple2<K, W>>, int, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'join' between RDDs of this DStream and other DStream.
join(DStream<Tuple2<K, W>>, Partitioner, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'join' between RDDs of this DStream and other DStream.
joinVertices(RDD<Tuple2<Object, U>>, Function3<Object, VD, U, VD>, ClassTag<U>) - Method in class org.apache.spark.graphx.GraphOps
Join the vertices with an RDD and then apply a function from the vertex and RDD entry to a new vertex value.
joinWith(Dataset<U>, Column, String) - Method in class org.apache.spark.sql.Dataset
Experimental Joins this Dataset returning a Tuple2 for each pair where condition evaluates to true.
joinWith(Dataset<U>, Column) - Method in class org.apache.spark.sql.Dataset
Experimental Using inner equi-join to join this Dataset returning a Tuple2 for each pair where condition evaluates to true.
json(String...) - Method in class org.apache.spark.sql.DataFrameReader
Loads JSON files and returns the results as a DataFrame.
json(String) - Method in class org.apache.spark.sql.DataFrameReader
Loads a JSON file and returns the results as a DataFrame.
json(Seq<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads JSON files and returns the results as a DataFrame.
json(JavaRDD<String>) - Method in class org.apache.spark.sql.DataFrameReader
Deprecated.
Use json(Dataset[String]) instead. Since 2.2.0.
json(RDD<String>) - Method in class org.apache.spark.sql.DataFrameReader
Deprecated.
Use json(Dataset[String]) instead. Since 2.2.0.
json(Dataset<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads a Dataset[String] storing JSON objects (JSON Lines text format or newline-delimited JSON) and returns the result as a DataFrame.
json(String) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame in JSON format ( JSON Lines text format or newline-delimited JSON) at the specified path.
json() - Method in class org.apache.spark.sql.sources.v2.reader.streaming.Offset
A JSON-serialized representation of an Offset that is used for saving offsets to the offset log.
json(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads a JSON file stream and returns the results as a DataFrame.
json() - Method in class org.apache.spark.sql.streaming.SinkProgress
The compact JSON representation of this progress.
json() - Method in class org.apache.spark.sql.streaming.SourceProgress
The compact JSON representation of this progress.
json() - Method in class org.apache.spark.sql.streaming.StateOperatorProgress
The compact JSON representation of this progress.
json() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
The compact JSON representation of this progress.
json() - Method in class org.apache.spark.sql.streaming.StreamingQueryStatus
The compact JSON representation of this status.
json() - Static method in class org.apache.spark.sql.types.ArrayType
 
json() - Static method in class org.apache.spark.sql.types.BinaryType
 
json() - Static method in class org.apache.spark.sql.types.BooleanType
 
json() - Static method in class org.apache.spark.sql.types.ByteType
 
json() - Static method in class org.apache.spark.sql.types.CalendarIntervalType
 
json() - Static method in class org.apache.spark.sql.types.CharType
 
json() - Method in class org.apache.spark.sql.types.DataType
The compact JSON representation of this data type.
json() - Static method in class org.apache.spark.sql.types.DateType
 
json() - Static method in class org.apache.spark.sql.types.DecimalType
 
json() - Static method in class org.apache.spark.sql.types.DoubleType
 
json() - Static method in class org.apache.spark.sql.types.FloatType
 
json() - Static method in class org.apache.spark.sql.types.HiveStringType
 
json() - Static method in class org.apache.spark.sql.types.IntegerType
 
json() - Static method in class org.apache.spark.sql.types.LongType
 
json() - Static method in class org.apache.spark.sql.types.MapType
 
json() - Method in class org.apache.spark.sql.types.Metadata
Converts to its JSON representation.
json() - Static method in class org.apache.spark.sql.types.NullType
 
json() - Static method in class org.apache.spark.sql.types.NumericType
 
json() - Static method in class org.apache.spark.sql.types.ObjectType
 
json() - Static method in class org.apache.spark.sql.types.ShortType
 
json() - Static method in class org.apache.spark.sql.types.StringType
 
json() - Static method in class org.apache.spark.sql.types.StructType
 
json() - Static method in class org.apache.spark.sql.types.TimestampType
 
json() - Static method in class org.apache.spark.sql.types.VarcharType
 
json_tuple(Column, String...) - Static method in class org.apache.spark.sql.functions
Creates a new row for a json column according to the given field names.
json_tuple(Column, Seq<String>) - Static method in class org.apache.spark.sql.functions
Creates a new row for a json column according to the given field names.
jsonDecode(String) - Method in class org.apache.spark.ml.param.BooleanParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.DoubleArrayArrayParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.DoubleArrayParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.DoubleParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.FloatParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.IntArrayParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.IntParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.LongParam
 
jsonDecode(String) - Method in class org.apache.spark.ml.param.Param
Decodes a param value from JSON.
jsonDecode(String) - Method in class org.apache.spark.ml.param.StringArrayParam
 
jsonEncode(boolean) - Method in class org.apache.spark.ml.param.BooleanParam
 
jsonEncode(double[][]) - Method in class org.apache.spark.ml.param.DoubleArrayArrayParam
 
jsonEncode(double[]) - Method in class org.apache.spark.ml.param.DoubleArrayParam
 
jsonEncode(double) - Method in class org.apache.spark.ml.param.DoubleParam
 
jsonEncode(float) - Method in class org.apache.spark.ml.param.FloatParam
 
jsonEncode(int[]) - Method in class org.apache.spark.ml.param.IntArrayParam
 
jsonEncode(int) - Method in class org.apache.spark.ml.param.IntParam
 
jsonEncode(long) - Method in class org.apache.spark.ml.param.LongParam
 
jsonEncode(T) - Method in class org.apache.spark.ml.param.Param
Encodes a param value into JSON, which can be decoded by `jsonDecode()`.
jsonEncode(String[]) - Method in class org.apache.spark.ml.param.StringArrayParam
 
jsonFile(String) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonFile(String, StructType) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonFile(String, double) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
JsonMatrixConverter - Class in org.apache.spark.ml.linalg
 
JsonMatrixConverter() - Constructor for class org.apache.spark.ml.linalg.JsonMatrixConverter
 
jsonOption(JsonAST.JValue) - Static method in class org.apache.spark.util.Utils
Return an option that translates JNothing to None
JsonProtocol - Class in org.apache.spark.util
Serializes SparkListener events to/from JSON.
JsonProtocol() - Constructor for class org.apache.spark.util.JsonProtocol
 
jsonRDD(RDD<String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonRDD(JavaRDD<String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonRDD(RDD<String>, StructType) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonRDD(JavaRDD<String>, StructType) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonRDD(RDD<String>, double) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonRDD(JavaRDD<String>, double) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().json().
jsonResponderToServlet(Function1<HttpServletRequest, JsonAST.JValue>) - Static method in class org.apache.spark.ui.JettyUtils
 
JsonVectorConverter - Class in org.apache.spark.ml.linalg
 
JsonVectorConverter() - Constructor for class org.apache.spark.ml.linalg.JsonVectorConverter
 
jValueDecode(JsonAST.JValue) - Static method in class org.apache.spark.ml.param.DoubleParam
Decodes a param value from JValue.
jValueDecode(JsonAST.JValue) - Static method in class org.apache.spark.ml.param.FloatParam
Decodes a param value from JValue.
jValueEncode(double) - Static method in class org.apache.spark.ml.param.DoubleParam
Encodes a param value into JValue.
jValueEncode(float) - Static method in class org.apache.spark.ml.param.FloatParam
Encodes a param value into JValue.
JVM_GC_TIME() - Static method in class org.apache.spark.InternalAccumulator
 
jvmGcTime() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
jvmGcTime() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 

K

k() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
k() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
k() - Method in class org.apache.spark.ml.clustering.ClusteringSummary
 
k() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
k() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
k() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
k() - Static method in class org.apache.spark.ml.clustering.KMeans
 
k() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
k() - Static method in class org.apache.spark.ml.clustering.LDA
 
k() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
k() - Static method in class org.apache.spark.ml.feature.PCA
 
k() - Static method in class org.apache.spark.ml.feature.PCAModel
 
k() - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Number of leaf clusters.
k() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
 
k() - Method in class org.apache.spark.mllib.clustering.ExpectationSum
 
k() - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
Number of gaussians in mixture
k() - Method in class org.apache.spark.mllib.clustering.KMeansModel
Total number of clusters.
k() - Method in class org.apache.spark.mllib.clustering.LDAModel
Number of topics
k() - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
k() - Method in class org.apache.spark.mllib.clustering.PowerIterationClusteringModel
 
k() - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
 
k() - Method in class org.apache.spark.mllib.feature.PCA
 
k() - Method in class org.apache.spark.mllib.feature.PCAModel
 
K_MEANS_PARALLEL() - Static method in class org.apache.spark.mllib.clustering.KMeans
 
kClassTag() - Method in class org.apache.spark.api.java.JavaHadoopRDD
 
kClassTag() - Method in class org.apache.spark.api.java.JavaNewHadoopRDD
 
kClassTag() - Method in class org.apache.spark.api.java.JavaPairRDD
 
kClassTag() - Method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
kClassTag() - Method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
keepLastCheckpoint() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
keepLastCheckpoint() - Static method in class org.apache.spark.ml.clustering.LDA
 
keepLastCheckpoint() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
KernelDensity - Class in org.apache.spark.mllib.stat
Kernel density estimation.
KernelDensity() - Constructor for class org.apache.spark.mllib.stat.KernelDensity
 
keyArray() - Method in class org.apache.spark.sql.vectorized.ColumnarMap
 
keyAs(Encoder<L>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Returns a new KeyValueGroupedDataset where the type of the key has been mapped to the specified type.
keyBy(Function<T, U>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
keyBy(Function<T, U>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
keyBy(Function<T, U>) - Static method in class org.apache.spark.api.java.JavaRDD
 
keyBy(Function<T, U>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Creates tuples of the elements in this RDD by applying f.
keyBy(Function1<T, K>) - Static method in class org.apache.spark.api.r.RRDD
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.graphx.VertexRDD
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
keyBy(Function1<T, K>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
keyBy(Function1<T, K>) - Method in class org.apache.spark.rdd.RDD
Creates tuples of the elements in this RDD by applying f.
keyBy(Function1<T, K>) - Static method in class org.apache.spark.rdd.UnionRDD
 
keyOrdering() - Method in class org.apache.spark.ShuffleDependency
 
keyPrefix() - Method in interface org.apache.spark.sql.sources.v2.SessionConfigSupport
Key prefix of the session configs to propagate.
keys() - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the keys of each tuple.
keys() - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD with the keys of each tuple.
keys() - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Returns a Dataset that contains each unique key.
keyType() - Method in class org.apache.spark.sql.types.MapType
 
KeyValueGroupedDataset<K,V> - Class in org.apache.spark.sql
Experimental A Dataset has been logically grouped by a user specified grouping key.
kFold(RDD<T>, int, int, ClassTag<T>) - Static method in class org.apache.spark.mllib.util.MLUtils
Return a k element array of pairs of RDDs with the first element of each pair containing the training data, a complement of the validation data and the second element, the validation data, containing a unique 1/kth of the data.
kFold(RDD<T>, int, long, ClassTag<T>) - Static method in class org.apache.spark.mllib.util.MLUtils
Version of kFold() taking a Long seed.
kill() - Method in interface org.apache.spark.launcher.SparkAppHandle
Tries to kill the underlying application.
killed() - Method in class org.apache.spark.scheduler.TaskInfo
 
KILLED() - Static method in class org.apache.spark.TaskState
 
killedSummary() - Method in class org.apache.spark.status.LiveJob
 
killedSummary() - Method in class org.apache.spark.status.LiveStage
 
killedTasks() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
killedTasks() - Method in class org.apache.spark.status.LiveExecutorStageSummary
 
killedTasks() - Method in class org.apache.spark.status.LiveJob
 
killedTasks() - Method in class org.apache.spark.status.LiveStage
 
killedTasksSummary() - Method in class org.apache.spark.status.api.v1.JobData
 
killedTasksSummary() - Method in class org.apache.spark.status.api.v1.StageData
 
killExecutor(String) - Method in class org.apache.spark.SparkContext
Developer API Request that the cluster manager kill the specified executor.
KillExecutors(Seq<String>) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutors
 
killExecutors(Seq<String>) - Method in class org.apache.spark.SparkContext
Developer API Request that the cluster manager kill the specified executors.
KillExecutors$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutors$
 
KillExecutorsOnHost(String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutorsOnHost
 
KillExecutorsOnHost$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutorsOnHost$
 
KillTask(long, String, boolean, String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask
 
KillTask - Class in org.apache.spark.scheduler.local
 
KillTask(long, boolean, String) - Constructor for class org.apache.spark.scheduler.local.KillTask
 
KillTask$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask$
 
killTaskAttempt(long, boolean, String) - Method in class org.apache.spark.SparkContext
Kill and reschedule the given task attempt.
KinesisInitialPositions - Class in org.apache.spark.streaming.kinesis
 
KinesisInitialPositions() - Constructor for class org.apache.spark.streaming.kinesis.KinesisInitialPositions
 
KinesisInitialPositions.AtTimestamp - Class in org.apache.spark.streaming.kinesis
 
KinesisInitialPositions.Latest - Class in org.apache.spark.streaming.kinesis
 
KinesisInitialPositions.TrimHorizon - Class in org.apache.spark.streaming.kinesis
 
KinesisUtils - Class in org.apache.spark.streaming.kinesis
 
KinesisUtils() - Constructor for class org.apache.spark.streaming.kinesis.KinesisUtils
 
KinesisUtilsPythonHelper - Class in org.apache.spark.streaming.kinesis
This is a helper class that wraps the methods in KinesisUtils into more Python-friendly class and function so that it can be easily instantiated and called from Python's KinesisUtils.
KinesisUtilsPythonHelper() - Constructor for class org.apache.spark.streaming.kinesis.KinesisUtilsPythonHelper
 
kManifest() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
kManifest() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
kManifest() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
KMeans - Class in org.apache.spark.ml.clustering
K-means clustering with support for k-means|| initialization proposed by Bahmani et al.
KMeans(String) - Constructor for class org.apache.spark.ml.clustering.KMeans
 
KMeans() - Constructor for class org.apache.spark.ml.clustering.KMeans
 
KMeans - Class in org.apache.spark.mllib.clustering
K-means clustering with a k-means++ like initialization mode (the k-means|| algorithm by Bahmani et al).
KMeans() - Constructor for class org.apache.spark.mllib.clustering.KMeans
Constructs a KMeans instance with default parameters: {k: 2, maxIterations: 20, initializationMode: "k-means||", initializationSteps: 2, epsilon: 1e-4, seed: random}.
KMeansDataGenerator - Class in org.apache.spark.mllib.util
Developer API Generate test data for KMeans.
KMeansDataGenerator() - Constructor for class org.apache.spark.mllib.util.KMeansDataGenerator
 
KMeansModel - Class in org.apache.spark.ml.clustering
Model fitted by KMeans.
KMeansModel - Class in org.apache.spark.mllib.clustering
A clustering model for K-means.
KMeansModel(Vector[]) - Constructor for class org.apache.spark.mllib.clustering.KMeansModel
 
KMeansModel(Iterable<Vector>) - Constructor for class org.apache.spark.mllib.clustering.KMeansModel
A Java-friendly constructor that takes an Iterable of Vectors.
KMeansModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.clustering
 
kMeansPlusPlus(int, VectorWithNorm[], double[], int, int) - Static method in class org.apache.spark.mllib.clustering.LocalKMeans
Run K-means++ on the weighted point set points.
KMeansSummary - Class in org.apache.spark.ml.clustering
Experimental Summary of KMeans.
kolmogorovSmirnovTest(RDD<Object>, String, double...) - Static method in class org.apache.spark.mllib.stat.Statistics
Convenience function to conduct a one-sample, two-sided Kolmogorov-Smirnov test for probability distribution equality.
kolmogorovSmirnovTest(JavaDoubleRDD, String, double...) - Static method in class org.apache.spark.mllib.stat.Statistics
Java-friendly version of kolmogorovSmirnovTest()
kolmogorovSmirnovTest(RDD<Object>, Function1<Object, Object>) - Static method in class org.apache.spark.mllib.stat.Statistics
Conduct the two-sided Kolmogorov-Smirnov (KS) test for data sampled from a continuous distribution.
kolmogorovSmirnovTest(RDD<Object>, String, Seq<Object>) - Static method in class org.apache.spark.mllib.stat.Statistics
Convenience function to conduct a one-sample, two-sided Kolmogorov-Smirnov test for probability distribution equality.
kolmogorovSmirnovTest(JavaDoubleRDD, String, Seq<Object>) - Static method in class org.apache.spark.mllib.stat.Statistics
Java-friendly version of kolmogorovSmirnovTest()
KolmogorovSmirnovTest - Class in org.apache.spark.mllib.stat.test
Conduct the two-sided Kolmogorov Smirnov (KS) test for data sampled from a continuous distribution.
KolmogorovSmirnovTest() - Constructor for class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest
 
KolmogorovSmirnovTest.NullHypothesis$ - Class in org.apache.spark.mllib.stat.test
 
KolmogorovSmirnovTestResult - Class in org.apache.spark.mllib.stat.test
Object containing the test results for the Kolmogorov-Smirnov test.
kryo(ClassTag<T>) - Static method in class org.apache.spark.sql.Encoders
(Scala-specific) Creates an encoder that serializes objects of type T using Kryo.
kryo(Class<T>) - Static method in class org.apache.spark.sql.Encoders
Creates an encoder that serializes objects of type T using Kryo.
KryoRegistrator - Interface in org.apache.spark.serializer
Interface implemented by clients to register their classes with Kryo when using Kryo serialization.
KryoSerializer - Class in org.apache.spark.serializer
A Spark serializer that uses the Kryo serialization library.
KryoSerializer(SparkConf) - Constructor for class org.apache.spark.serializer.KryoSerializer
 
kurtosis(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the kurtosis of the values in a group.
kurtosis(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the kurtosis of the values in a group.
KVUtils - Class in org.apache.spark.status
 
KVUtils() - Constructor for class org.apache.spark.status.KVUtils
 

L

L1Updater - Class in org.apache.spark.mllib.optimization
Developer API Updater for L1 regularized problems.
L1Updater() - Constructor for class org.apache.spark.mllib.optimization.L1Updater
 
label() - Method in class org.apache.spark.ml.feature.LabeledPoint
 
label() - Method in class org.apache.spark.mllib.regression.LabeledPoint
 
labelCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
labelCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
labelCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
labelCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
labelCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
labelCol() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Field in "predictions" which gives the true label of each instance (if available).
labelCol() - Method in class org.apache.spark.ml.classification.LogisticRegressionSummaryImpl
 
labelCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
labelCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
labelCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
labelCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
labelCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
labelCol() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
labelCol() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
labelCol() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
labelCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
labelCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
labelCol() - Static method in class org.apache.spark.ml.feature.RFormula
 
labelCol() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
labelCol() - Method in interface org.apache.spark.ml.param.shared.HasLabelCol
Param for label column name.
labelCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
labelCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
labelCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
labelCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
labelCol() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
labelCol() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
labelCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
labelCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
labelCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
labelCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
labelCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
labelCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
labelCol() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
 
labelCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
labelCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
LabelConverter - Class in org.apache.spark.ml.classification
Label to vector converter.
LabelConverter() - Constructor for class org.apache.spark.ml.classification.LabelConverter
 
LabeledPoint - Class in org.apache.spark.ml.feature
Class that represents the features and label of a data point.
LabeledPoint(double, Vector) - Constructor for class org.apache.spark.ml.feature.LabeledPoint
 
LabeledPoint - Class in org.apache.spark.mllib.regression
Class that represents the features and labels of a data point.
LabeledPoint(double, Vector) - Constructor for class org.apache.spark.mllib.regression.LabeledPoint
 
LabelPropagation - Class in org.apache.spark.graphx.lib
Label Propagation algorithm.
LabelPropagation() - Constructor for class org.apache.spark.graphx.lib.LabelPropagation
 
labels() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns the sequence of labels in ascending order.
labels() - Method in class org.apache.spark.ml.feature.IndexToString
Optional param for array of labels specifying index-string mapping.
labels() - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
labels() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
labels() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$.Data
 
labels() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$.Data
 
labels() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns the sequence of labels in ascending order
labels() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns the sequence of labels in ascending order
lag(Column, int) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row.
lag(String, int) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows before the current row, and null if there is less than offset rows before the current row.
lag(String, int, Object) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.
lag(Column, int, Object) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows before the current row, and defaultValue if there is less than offset rows before the current row.
LassoModel - Class in org.apache.spark.mllib.regression
Regression model trained using Lasso.
LassoModel(Vector, double) - Constructor for class org.apache.spark.mllib.regression.LassoModel
 
LassoWithSGD - Class in org.apache.spark.mllib.regression
Train a regression model with L1-regularization using Stochastic Gradient Descent.
LassoWithSGD() - Constructor for class org.apache.spark.mllib.regression.LassoWithSGD
Deprecated.
Use ml.regression.LinearRegression with elasticNetParam = 1.0. Note the default regParam is 0.01 for LassoWithSGD, but is 0.0 for LinearRegression. Since 2.0.0.
last(Column, boolean) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the last value in a group.
last(String, boolean) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the last value of the column in a group.
last(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the last value in a group.
last(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the last value of the column in a group.
last() - Static method in class org.apache.spark.sql.types.StructType
 
last_day(Column) - Static method in class org.apache.spark.sql.functions
Given a date column, returns the last day of the month which the given date belongs to.
lastDir() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
lastError() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
lastError() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
lastErrorMessage() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
lastErrorMessage() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
lastErrorTime() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
lastErrorTime() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
lastIndexOf(B) - Static method in class org.apache.spark.sql.types.StructType
 
lastIndexOf(B, int) - Static method in class org.apache.spark.sql.types.StructType
 
lastIndexOfSlice(GenSeq<B>) - Static method in class org.apache.spark.sql.types.StructType
 
lastIndexOfSlice(GenSeq<B>, int) - Static method in class org.apache.spark.sql.types.StructType
 
lastIndexWhere(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
lastIndexWhere(Function1<A, Object>, int) - Static method in class org.apache.spark.sql.types.StructType
 
lastOption() - Static method in class org.apache.spark.sql.types.StructType
 
lastProgress() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns the most recent StreamingQueryProgress update of this streaming query.
lastStageNameAndDescription(org.apache.spark.status.AppStatusStore, JobData) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
lastUpdate() - Method in class org.apache.spark.status.LiveRDDDistribution
 
lastUpdated() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
Latest() - Constructor for class org.apache.spark.streaming.kinesis.KinesisInitialPositions.Latest
 
latestModel() - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Return the latest model.
latestModel() - Method in class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
Return the latest model.
launch() - Method in class org.apache.spark.launcher.SparkLauncher
Launches a sub-process that will start the configured Spark application.
LAUNCH_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
LAUNCHING() - Static method in class org.apache.spark.TaskState
 
LaunchTask(org.apache.spark.util.SerializableBuffer) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.LaunchTask
 
LaunchTask$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.LaunchTask$
 
launchTime() - Method in class org.apache.spark.scheduler.TaskInfo
 
launchTime() - Method in class org.apache.spark.status.api.v1.TaskData
 
layers() - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
layers() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
LBFGS - Class in org.apache.spark.mllib.optimization
Developer API Class used to solve an optimization problem using Limited-memory BFGS.
LBFGS(Gradient, Updater) - Constructor for class org.apache.spark.mllib.optimization.LBFGS
 
LDA - Class in org.apache.spark.ml.clustering
Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
LDA(String) - Constructor for class org.apache.spark.ml.clustering.LDA
 
LDA() - Constructor for class org.apache.spark.ml.clustering.LDA
 
LDA - Class in org.apache.spark.mllib.clustering
Latent Dirichlet Allocation (LDA), a topic model designed for text documents.
LDA() - Constructor for class org.apache.spark.mllib.clustering.LDA
Constructs a LDA instance with default parameters.
LDAModel - Class in org.apache.spark.ml.clustering
Model fitted by LDA.
LDAModel - Class in org.apache.spark.mllib.clustering
Latent Dirichlet Allocation (LDA) model.
LDAOptimizer - Interface in org.apache.spark.mllib.clustering
Developer API
LDAUtils - Class in org.apache.spark.mllib.clustering
Utility methods for LDA.
LDAUtils() - Constructor for class org.apache.spark.mllib.clustering.LDAUtils
 
lead(String, int) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row.
lead(Column, int) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows after the current row, and null if there is less than offset rows after the current row.
lead(String, int, Object) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.
lead(Column, int, Object) - Static method in class org.apache.spark.sql.functions
Window function: returns the value that is offset rows after the current row, and defaultValue if there is less than offset rows after the current row.
LeafNode - Class in org.apache.spark.ml.tree
Decision tree leaf node.
learningDecay() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
learningDecay() - Static method in class org.apache.spark.ml.clustering.LDA
 
learningDecay() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
learningOffset() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
learningOffset() - Static method in class org.apache.spark.ml.clustering.LDA
 
learningOffset() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
learningRate() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
least(Column...) - Static method in class org.apache.spark.sql.functions
Returns the least value of the list of values, skipping null values.
least(String, String...) - Static method in class org.apache.spark.sql.functions
Returns the least value of the list of column names, skipping null values.
least(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Returns the least value of the list of values, skipping null values.
least(String, Seq<String>) - Static method in class org.apache.spark.sql.functions
Returns the least value of the list of column names, skipping null values.
LeastSquaresGradient - Class in org.apache.spark.mllib.optimization
Developer API Compute gradient and loss for a Least-squared loss function, as used in linear regression.
LeastSquaresGradient() - Constructor for class org.apache.spark.mllib.optimization.LeastSquaresGradient
 
left() - Method in class org.apache.spark.sql.sources.And
 
left() - Method in class org.apache.spark.sql.sources.Or
 
leftCategories() - Method in class org.apache.spark.ml.tree.CategoricalSplit
Get sorted categories which split to the left
leftCategoriesOrThreshold() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData
 
leftChild() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
leftChild() - Method in class org.apache.spark.ml.tree.InternalNode
 
leftChildIndex(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Return the index of the left child of this node.
leftImpurity() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
leftJoin(RDD<Tuple2<Object, VD2>>, Function3<Object, VD, Option<VD2>, VD3>, ClassTag<VD2>, ClassTag<VD3>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
leftJoin(RDD<Tuple2<Object, VD2>>, Function3<Object, VD, Option<VD2>, VD3>, ClassTag<VD2>, ClassTag<VD3>) - Method in class org.apache.spark.graphx.VertexRDD
Left joins this VertexRDD with an RDD containing vertex attribute pairs.
leftNode() - Method in class org.apache.spark.mllib.tree.model.Node
 
leftNodeId() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
leftOuterJoin(JavaPairRDD<K, W>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a left outer join of this and other.
leftOuterJoin(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a left outer join of this and other.
leftOuterJoin(JavaPairRDD<K, W>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a left outer join of this and other.
leftOuterJoin(RDD<Tuple2<K, W>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a left outer join of this and other.
leftOuterJoin(RDD<Tuple2<K, W>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a left outer join of this and other.
leftOuterJoin(RDD<Tuple2<K, W>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a left outer join of this and other.
leftOuterJoin(JavaPairDStream<K, W>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'left outer join' between RDDs of this DStream and other DStream.
leftOuterJoin(JavaPairDStream<K, W>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'left outer join' between RDDs of this DStream and other DStream.
leftOuterJoin(JavaPairDStream<K, W>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'left outer join' between RDDs of this DStream and other DStream.
leftOuterJoin(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
leftOuterJoin(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
leftOuterJoin(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
leftOuterJoin(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
leftOuterJoin(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
leftOuterJoin(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
leftOuterJoin(DStream<Tuple2<K, W>>, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'left outer join' between RDDs of this DStream and other DStream.
leftOuterJoin(DStream<Tuple2<K, W>>, int, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'left outer join' between RDDs of this DStream and other DStream.
leftOuterJoin(DStream<Tuple2<K, W>>, Partitioner, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'left outer join' between RDDs of this DStream and other DStream.
leftPredict() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
leftZipJoin(VertexRDD<VD2>, Function3<Object, VD, Option<VD2>, VD3>, ClassTag<VD2>, ClassTag<VD3>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
leftZipJoin(VertexRDD<VD2>, Function3<Object, VD, Option<VD2>, VD3>, ClassTag<VD2>, ClassTag<VD3>) - Method in class org.apache.spark.graphx.VertexRDD
Left joins this RDD with another VertexRDD with the same index.
LegacyAccumulatorWrapper<R,T> - Class in org.apache.spark.util
 
LegacyAccumulatorWrapper(R, AccumulableParam<R, T>) - Constructor for class org.apache.spark.util.LegacyAccumulatorWrapper
 
length() - Method in class org.apache.spark.scheduler.SplitInfo
 
length(Column) - Static method in class org.apache.spark.sql.functions
Computes the character length of a given string or number of bytes of a binary string.
length() - Method in interface org.apache.spark.sql.Row
Number of elements in the Row.
length() - Method in class org.apache.spark.sql.types.CharType
 
length() - Method in class org.apache.spark.sql.types.HiveStringType
 
length() - Method in class org.apache.spark.sql.types.StructType
 
length() - Method in class org.apache.spark.sql.types.VarcharType
 
length() - Method in class org.apache.spark.status.RDDPartitionSeq
 
lengthCompare(int) - Static method in class org.apache.spark.sql.types.StructType
 
leq(Object) - Method in class org.apache.spark.sql.Column
Less than or equal to.
less(Duration) - Method in class org.apache.spark.streaming.Duration
 
less(Time) - Method in class org.apache.spark.streaming.Time
 
lessEq(Duration) - Method in class org.apache.spark.streaming.Duration
 
lessEq(Time) - Method in class org.apache.spark.streaming.Time
 
LessThan - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a value less than value.
LessThan(String, Object) - Constructor for class org.apache.spark.sql.sources.LessThan
 
LessThanOrEqual - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a value less than or equal to value.
LessThanOrEqual(String, Object) - Constructor for class org.apache.spark.sql.sources.LessThanOrEqual
 
levenshtein(Column, Column) - Static method in class org.apache.spark.sql.functions
Computes the Levenshtein distance of the two given string columns.
libraryPathEnvName() - Static method in class org.apache.spark.util.Utils
Return the current system LD_LIBRARY_PATH name
libraryPathEnvPrefix(Seq<String>) - Static method in class org.apache.spark.util.Utils
Return the prefix of a command that appends the given library paths to the system-specific library path environment variable.
LibSVMDataSource - Class in org.apache.spark.ml.source.libsvm
libsvm package implements Spark SQL data source API for loading LIBSVM data as DataFrame.
lift() - Static method in class org.apache.spark.sql.types.StructType
 
like(String) - Method in class org.apache.spark.sql.Column
SQL like expression.
limit(int) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset by taking the first n rows.
line() - Method in exception org.apache.spark.sql.AnalysisException
 
LinearDataGenerator - Class in org.apache.spark.mllib.util
Developer API Generate sample data used for Linear Data.
LinearDataGenerator() - Constructor for class org.apache.spark.mllib.util.LinearDataGenerator
 
LinearRegression - Class in org.apache.spark.ml.regression
Linear regression.
LinearRegression(String) - Constructor for class org.apache.spark.ml.regression.LinearRegression
 
LinearRegression() - Constructor for class org.apache.spark.ml.regression.LinearRegression
 
LinearRegressionModel - Class in org.apache.spark.ml.regression
Model produced by LinearRegression.
LinearRegressionModel - Class in org.apache.spark.mllib.regression
Regression model trained using LinearRegression.
LinearRegressionModel(Vector, double) - Constructor for class org.apache.spark.mllib.regression.LinearRegressionModel
 
LinearRegressionSummary - Class in org.apache.spark.ml.regression
Experimental Linear regression results evaluated on a dataset.
LinearRegressionTrainingSummary - Class in org.apache.spark.ml.regression
Experimental Linear regression training results.
LinearRegressionWithSGD - Class in org.apache.spark.mllib.regression
Train a linear regression model with no regularization using Stochastic Gradient Descent.
LinearRegressionWithSGD() - Constructor for class org.apache.spark.mllib.regression.LinearRegressionWithSGD
Deprecated.
Use ml.regression.LinearRegression or LBFGS. Since 2.0.0.
LinearSVC - Class in org.apache.spark.ml.classification
Experimental
LinearSVC(String) - Constructor for class org.apache.spark.ml.classification.LinearSVC
 
LinearSVC() - Constructor for class org.apache.spark.ml.classification.LinearSVC
 
LinearSVCModel - Class in org.apache.spark.ml.classification
Experimental Linear SVM Model trained by LinearSVC
link(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.CLogLog$
 
link(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Identity$
 
link(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Inverse$
 
link() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
link(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Log$
 
link(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Logit$
 
link(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Probit$
 
link(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Sqrt$
 
link() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
Link$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Link$
 
linkPower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
linkPower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
linkPredictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
linkPredictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
listColumns(String) - Method in class org.apache.spark.sql.catalog.Catalog
Returns a list of columns for the given table/view or temporary view.
listColumns(String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Returns a list of columns for the given table/view in the specified database.
listDatabases() - Method in class org.apache.spark.sql.catalog.Catalog
Returns a list of databases available across all sessions.
listenerManager() - Method in class org.apache.spark.sql.SparkSession
Experimental An interface to register custom QueryExecutionListeners that listen for execution metrics.
listenerManager() - Method in class org.apache.spark.sql.SQLContext
An interface to register custom QueryExecutionListeners that listen for execution metrics.
listFiles() - Method in class org.apache.spark.SparkContext
Returns a list of file paths that are added to resources.
listFunctions() - Method in class org.apache.spark.sql.catalog.Catalog
Returns a list of functions registered in the current database.
listFunctions(String) - Method in class org.apache.spark.sql.catalog.Catalog
Returns a list of functions registered in the specified database.
listingTable(Seq<String>, Function1<T, Seq<Node>>, Iterable<T>, boolean, Option<String>, Seq<String>, boolean, boolean) - Static method in class org.apache.spark.ui.UIUtils
Returns an HTML table constructed by generating a row for each object in a sequence.
listJars() - Method in class org.apache.spark.SparkContext
Returns a list of jar files that are added to resources.
listOrcFiles(String, Configuration) - Static method in class org.apache.spark.sql.hive.orc.OrcFileOperator
 
listTables() - Method in class org.apache.spark.sql.catalog.Catalog
Returns a list of tables/views in the current database.
listTables(String) - Method in class org.apache.spark.sql.catalog.Catalog
Returns a list of tables/views in the specified database.
lit(Object) - Static method in class org.apache.spark.sql.functions
Creates a Column of literal value.
literal(String) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
LIVE_ENTITY_UPDATE_PERIOD() - Static method in class org.apache.spark.status.config
 
LiveEntityHelpers - Class in org.apache.spark.status
 
LiveEntityHelpers() - Constructor for class org.apache.spark.status.LiveEntityHelpers
 
LiveExecutor - Class in org.apache.spark.status
 
LiveExecutor(String, long) - Constructor for class org.apache.spark.status.LiveExecutor
 
LiveExecutorStageSummary - Class in org.apache.spark.status
 
LiveExecutorStageSummary(int, int, String) - Constructor for class org.apache.spark.status.LiveExecutorStageSummary
 
LiveJob - Class in org.apache.spark.status
 
LiveJob(int, String, Option<Date>, Seq<Object>, Option<String>, int) - Constructor for class org.apache.spark.status.LiveJob
 
LiveRDD - Class in org.apache.spark.status
 
LiveRDD(RDDInfo) - Constructor for class org.apache.spark.status.LiveRDD
 
LiveRDDDistribution - Class in org.apache.spark.status
 
LiveRDDDistribution(LiveExecutor) - Constructor for class org.apache.spark.status.LiveRDDDistribution
 
LiveRDDPartition - Class in org.apache.spark.status
 
LiveRDDPartition(String) - Constructor for class org.apache.spark.status.LiveRDDPartition
 
LiveStage - Class in org.apache.spark.status
 
LiveStage() - Constructor for class org.apache.spark.status.LiveStage
 
LiveTask - Class in org.apache.spark.status
 
LiveTask(TaskInfo, int, int, Option<Object>) - Constructor for class org.apache.spark.status.LiveTask
 
load(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
load(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
load(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
load(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
load(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
load(String) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
load(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
load(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
load(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
load(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
load(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
load(String) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
load(String) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
load(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
load(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
load(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
load(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
load(String) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
load(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
load(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
load(String) - Static method in class org.apache.spark.ml.clustering.KMeans
 
load(String) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
load(String) - Static method in class org.apache.spark.ml.clustering.LDA
 
load(String) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
load(String) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
load(String) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
load(String) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
load(String) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
load(String) - Static method in class org.apache.spark.ml.feature.Binarizer
 
load(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
load(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
load(String) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
load(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
load(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
load(String) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
load(String) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
load(String) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
load(String) - Static method in class org.apache.spark.ml.feature.DCT
 
load(String) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
load(String) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
load(String) - Static method in class org.apache.spark.ml.feature.HashingTF
 
load(String) - Static method in class org.apache.spark.ml.feature.IDF
 
load(String) - Static method in class org.apache.spark.ml.feature.IDFModel
 
load(String) - Static method in class org.apache.spark.ml.feature.Imputer
 
load(String) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
load(String) - Static method in class org.apache.spark.ml.feature.IndexToString
 
load(String) - Static method in class org.apache.spark.ml.feature.Interaction
 
load(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
load(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
load(String) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
load(String) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
load(String) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
load(String) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
load(String) - Static method in class org.apache.spark.ml.feature.NGram
 
load(String) - Static method in class org.apache.spark.ml.feature.Normalizer
 
load(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
load(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
load(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
load(String) - Static method in class org.apache.spark.ml.feature.PCA
 
load(String) - Static method in class org.apache.spark.ml.feature.PCAModel
 
load(String) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
load(String) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
load(String) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
load(String) - Static method in class org.apache.spark.ml.feature.RFormula
 
load(String) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
load(String) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
load(String) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
load(String) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
load(String) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
load(String) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
load(String) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
load(String) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
load(String) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
load(String) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
load(String) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
load(String) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
load(String) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
load(String) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
load(String) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
load(String) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
load(String) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
load(String) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
load(String) - Static method in class org.apache.spark.ml.Pipeline
 
load(String, SparkContext, String) - Method in class org.apache.spark.ml.Pipeline.SharedReadWrite$
Load metadata and stages for a Pipeline or PipelineModel
load(String) - Static method in class org.apache.spark.ml.PipelineModel
 
load(String) - Static method in class org.apache.spark.ml.r.RWrappers
 
load(String) - Static method in class org.apache.spark.ml.recommendation.ALS
 
load(String) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
load(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
load(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
load(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
load(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
load(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
load(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
load(String) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
load(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
load(String) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
load(String) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
load(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
load(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
load(String) - Method in interface org.apache.spark.ml.util.MLReadable
Reads an ML instance from the input path, a shortcut of read.load(path).
load(String) - Method in class org.apache.spark.ml.util.MLReader
Loads the ML component from the input path.
load(SparkContext, String) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
load(SparkContext, String) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$
 
load(SparkContext, String) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.classification.SVMModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
 
load(SparkContext, String, int) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.clustering.DistributedLDAModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.clustering.KMeansModel
 
load(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.KMeansModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.clustering.PowerIterationClusteringModel
 
load(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.PowerIterationClusteringModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.feature.ChiSqSelectorModel
 
load(SparkContext, String) - Method in class org.apache.spark.mllib.feature.ChiSqSelectorModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.feature.Word2VecModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.fpm.FPGrowthModel
 
load(SparkContext, String) - Method in class org.apache.spark.mllib.fpm.FPGrowthModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.fpm.PrefixSpanModel
 
load(SparkContext, String) - Method in class org.apache.spark.mllib.fpm.PrefixSpanModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Load a model from the given path.
load(SparkContext, String) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.regression.LassoModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
 
load(SparkContext, String, String, int) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
load(SparkContext, String) - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
load(SparkContext, String) - Method in interface org.apache.spark.mllib.util.Loader
Load a model from the given path.
load(String...) - Method in class org.apache.spark.sql.DataFrameReader
Loads input in as a DataFrame, for data sources that support multiple paths.
load() - Method in class org.apache.spark.sql.DataFrameReader
Loads input in as a DataFrame, for data sources that don't require a path (e.g.
load(String) - Method in class org.apache.spark.sql.DataFrameReader
Loads input in as a DataFrame, for data sources that require a path (e.g.
load(Seq<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads input in as a DataFrame, for data sources that support multiple paths.
load(String) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().load(path).
load(String, String) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().format(source).load(path).
load(String, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().format(source).options(options).load().
load(String, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().format(source).options(options).load().
load(String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
load(String, StructType, Map<String, String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().format(source).schema(schema).options(options).load().
load() - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads input data stream in as a DataFrame, for data streams that don't require a path (e.g.
load(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads input in as a DataFrame, for data streams that read from some path.
loadClass(String, boolean) - Method in class org.apache.spark.util.ChildFirstURLClassLoader
 
loadClass(String, boolean) - Method in class org.apache.spark.util.ParentClassLoader
 
loadData(SparkContext, String, String) - Method in class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$
Helper method for loading GLM classification model data.
loadData(SparkContext, String, String, int) - Method in class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$
Helper method for loading GLM regression model data.
loadDefaultSparkProperties(SparkConf, String) - Static method in class org.apache.spark.util.Utils
Load default Spark properties from the given file.
loadDefaultStopWords(String) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
Loads the default stop words for the given language.
Loader<M extends Saveable> - Interface in org.apache.spark.mllib.util
Developer API
loadExtensions(Class<T>, Seq<String>, SparkConf) - Static method in class org.apache.spark.util.Utils
Create instances of extension classes.
loadImpl(String, SparkSession, String, String) - Static method in class org.apache.spark.ml.tree.EnsembleModelReadWrite
Helper method for loading a tree ensemble from disk.
loadImpl(Dataset<Row>, Item, ClassTag<Item>) - Method in class org.apache.spark.mllib.fpm.FPGrowthModel.SaveLoadV1_0$
 
loadImpl(Dataset<Row>, Item, ClassTag<Item>) - Method in class org.apache.spark.mllib.fpm.PrefixSpanModel.SaveLoadV1_0$
 
loadLabeledPoints(SparkContext, String, int) - Static method in class org.apache.spark.mllib.util.MLUtils
Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile.
loadLabeledPoints(SparkContext, String) - Static method in class org.apache.spark.mllib.util.MLUtils
Loads labeled points saved using RDD[LabeledPoint].saveAsTextFile with the default number of partitions.
loadLibSVMFile(SparkContext, String, int, int) - Static method in class org.apache.spark.mllib.util.MLUtils
Loads labeled data in the LIBSVM format into an RDD[LabeledPoint].
loadLibSVMFile(SparkContext, String, int) - Static method in class org.apache.spark.mllib.util.MLUtils
Loads labeled data in the LIBSVM format into an RDD[LabeledPoint], with the default number of partitions.
loadLibSVMFile(SparkContext, String) - Static method in class org.apache.spark.mllib.util.MLUtils
Loads binary labeled data in the LIBSVM format into an RDD[LabeledPoint], with number of features determined automatically and the default number of partitions.
loadTreeNodes(String, org.apache.spark.ml.util.DefaultParamsReader.Metadata, SparkSession) - Static method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite
Load a decision tree from a file.
loadVectors(SparkContext, String, int) - Static method in class org.apache.spark.mllib.util.MLUtils
Loads vectors saved using RDD[Vector].saveAsTextFile.
loadVectors(SparkContext, String) - Static method in class org.apache.spark.mllib.util.MLUtils
Loads vectors saved using RDD[Vector].saveAsTextFile with the default number of partitions.
LOCAL_BLOCKS_FETCHED() - Method in class org.apache.spark.InternalAccumulator.shuffleRead$
 
LOCAL_BYTES_READ() - Method in class org.apache.spark.InternalAccumulator.shuffleRead$
 
LOCAL_CLUSTER_REGEX() - Static method in class org.apache.spark.SparkMasterRegex
 
LOCAL_N_FAILURES_REGEX() - Static method in class org.apache.spark.SparkMasterRegex
 
LOCAL_N_REGEX() - Static method in class org.apache.spark.SparkMasterRegex
 
localBlocksFetched() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
localBlocksFetched() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetrics
 
localBytesRead() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetrics
 
localCanonicalHostName() - Static method in class org.apache.spark.util.Utils
Get the local machine's FQDN.
localCheckpoint() - Static method in class org.apache.spark.api.r.RRDD
 
localCheckpoint() - Static method in class org.apache.spark.graphx.EdgeRDD
 
localCheckpoint() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
localCheckpoint() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
localCheckpoint() - Static method in class org.apache.spark.graphx.VertexRDD
 
localCheckpoint() - Static method in class org.apache.spark.rdd.HadoopRDD
 
localCheckpoint() - Static method in class org.apache.spark.rdd.JdbcRDD
 
localCheckpoint() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
localCheckpoint() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
localCheckpoint() - Method in class org.apache.spark.rdd.RDD
Mark this RDD for local checkpointing using Spark's existing caching layer.
localCheckpoint() - Static method in class org.apache.spark.rdd.UnionRDD
 
localCheckpoint() - Method in class org.apache.spark.sql.Dataset
Eagerly locally checkpoints a Dataset and return the new Dataset.
localCheckpoint(boolean) - Method in class org.apache.spark.sql.Dataset
Locally checkpoints a Dataset and return the new Dataset.
localHostName() - Static method in class org.apache.spark.util.Utils
Get the local machine's hostname.
localHostNameForURI() - Static method in class org.apache.spark.util.Utils
Get the local machine's URI.
LOCALITY() - Static method in class org.apache.spark.status.TaskIndexNames
 
localityAwareTasks() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors
 
localitySummary() - Method in class org.apache.spark.status.LiveStage
 
LocalKMeans - Class in org.apache.spark.mllib.clustering
An utility object to run K-means locally.
LocalKMeans() - Constructor for class org.apache.spark.mllib.clustering.LocalKMeans
 
LocalLDAModel - Class in org.apache.spark.ml.clustering
Local (non-distributed) model fitted by LDA.
LocalLDAModel - Class in org.apache.spark.mllib.clustering
Local LDA model.
localSeqToDatasetHolder(Seq<T>, Encoder<T>) - Method in class org.apache.spark.sql.SQLImplicits
Creates a Dataset from a local Seq.
localSparkRPackagePath() - Static method in class org.apache.spark.api.r.RUtils
Get the SparkR package path in the local spark distribution.
localValue() - Method in class org.apache.spark.Accumulable
Deprecated.
Get the current value of this accumulator from within a task.
localValue() - Static method in class org.apache.spark.Accumulator
Deprecated.
 
locate(String, Column) - Static method in class org.apache.spark.sql.functions
Locate the position of the first occurrence of substr.
locate(String, Column, int) - Static method in class org.apache.spark.sql.functions
Locate the position of the first occurrence of substr in a string column, after position pos.
location() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
location() - Method in class org.apache.spark.ui.storage.ExecutorStreamSummary
 
locations() - Method in class org.apache.spark.storage.BlockManagerMessages.BlockLocationsAndStatus
 
locationUri() - Method in class org.apache.spark.sql.catalog.Database
 
log() - Method in interface org.apache.spark.internal.Logging
 
log(Function0<Parsers.Parser<T>>, String) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
log(Column) - Static method in class org.apache.spark.sql.functions
Computes the natural logarithm of the given value.
log(String) - Static method in class org.apache.spark.sql.functions
Computes the natural logarithm of the given column.
log(double, Column) - Static method in class org.apache.spark.sql.functions
Returns the first argument-base logarithm of the second argument.
log(double, String) - Static method in class org.apache.spark.sql.functions
Returns the first argument-base logarithm of the second argument.
Log$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Log$
 
log10(Column) - Static method in class org.apache.spark.sql.functions
Computes the logarithm of the given value in base 10.
log10(String) - Static method in class org.apache.spark.sql.functions
Computes the logarithm of the given value in base 10.
log1p(Column) - Static method in class org.apache.spark.sql.functions
Computes the natural logarithm of the given value plus one.
log1p(String) - Static method in class org.apache.spark.sql.functions
Computes the natural logarithm of the given column plus one.
log2(Column) - Static method in class org.apache.spark.sql.functions
Computes the logarithm of the given column in base 2.
log2(String) - Static method in class org.apache.spark.sql.functions
Computes the logarithm of the given value in base 2.
log_() - Method in interface org.apache.spark.internal.Logging
 
logDebug(Function0<String>) - Method in interface org.apache.spark.internal.Logging
 
logDebug(Function0<String>, Throwable) - Method in interface org.apache.spark.internal.Logging
 
logDeprecationWarning(String) - Static method in class org.apache.spark.SparkConf
Logs a warning message if the given config key is deprecated.
logError(Function0<String>) - Method in interface org.apache.spark.internal.Logging
 
logError(Function0<String>, Throwable) - Method in interface org.apache.spark.internal.Logging
 
logEvent() - Method in interface org.apache.spark.scheduler.SparkListenerEvent
 
Logging - Interface in org.apache.spark.internal
Utility trait for classes that want to log data.
logInfo(Function0<String>) - Method in interface org.apache.spark.internal.Logging
 
logInfo(Function0<String>, Throwable) - Method in interface org.apache.spark.internal.Logging
 
LogisticGradient - Class in org.apache.spark.mllib.optimization
Developer API Compute gradient and loss for a multinomial logistic loss function, as used in multi-class classification (it is also used in binary logistic regression).
LogisticGradient(int) - Constructor for class org.apache.spark.mllib.optimization.LogisticGradient
 
LogisticGradient() - Constructor for class org.apache.spark.mllib.optimization.LogisticGradient
 
LogisticRegression - Class in org.apache.spark.ml.classification
Logistic regression.
LogisticRegression(String) - Constructor for class org.apache.spark.ml.classification.LogisticRegression
 
LogisticRegression() - Constructor for class org.apache.spark.ml.classification.LogisticRegression
 
LogisticRegressionDataGenerator - Class in org.apache.spark.mllib.util
Developer API Generate test data for LogisticRegression.
LogisticRegressionDataGenerator() - Constructor for class org.apache.spark.mllib.util.LogisticRegressionDataGenerator
 
LogisticRegressionModel - Class in org.apache.spark.ml.classification
Model produced by LogisticRegression.
LogisticRegressionModel - Class in org.apache.spark.mllib.classification
Classification model trained using Multinomial/Binary Logistic Regression.
LogisticRegressionModel(Vector, double, int, int) - Constructor for class org.apache.spark.mllib.classification.LogisticRegressionModel
 
LogisticRegressionModel(Vector, double) - Constructor for class org.apache.spark.mllib.classification.LogisticRegressionModel
Constructs a LogisticRegressionModel with weights and intercept for binary classification.
LogisticRegressionSummary - Interface in org.apache.spark.ml.classification
Experimental Abstraction for logistic regression results for a given model.
LogisticRegressionSummaryImpl - Class in org.apache.spark.ml.classification
Multiclass logistic regression results for a given model.
LogisticRegressionSummaryImpl(Dataset<Row>, String, String, String, String) - Constructor for class org.apache.spark.ml.classification.LogisticRegressionSummaryImpl
 
LogisticRegressionTrainingSummary - Interface in org.apache.spark.ml.classification
Experimental Abstraction for multiclass logistic regression training results.
LogisticRegressionTrainingSummaryImpl - Class in org.apache.spark.ml.classification
Multiclass logistic regression training results.
LogisticRegressionTrainingSummaryImpl(Dataset<Row>, String, String, String, String, double[]) - Constructor for class org.apache.spark.ml.classification.LogisticRegressionTrainingSummaryImpl
 
LogisticRegressionWithLBFGS - Class in org.apache.spark.mllib.classification
Train a classification model for Multinomial/Binary Logistic Regression using Limited-memory BFGS.
LogisticRegressionWithLBFGS() - Constructor for class org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
 
LogisticRegressionWithSGD - Class in org.apache.spark.mllib.classification
Train a classification model for Binary Logistic Regression using Stochastic Gradient Descent.
LogisticRegressionWithSGD() - Constructor for class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
Deprecated.
Use ml.classification.LogisticRegression or LogisticRegressionWithLBFGS. Since 2.0.0.
Logit$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Logit$
 
logLikelihood(Dataset<?>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
logLikelihood() - Method in class org.apache.spark.ml.clustering.ExpectationAggregator
 
logLikelihood() - Method in class org.apache.spark.ml.clustering.GaussianMixtureSummary
 
logLikelihood(Dataset<?>) - Method in class org.apache.spark.ml.clustering.LDAModel
Calculates a lower bound on the log likelihood of the entire corpus.
logLikelihood(Dataset<?>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
logLikelihood() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, alpha, eta)
logLikelihood() - Method in class org.apache.spark.mllib.clustering.ExpectationSum
 
logLikelihood(RDD<Tuple2<Object, Vector>>) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
Calculates a lower bound on the log likelihood of the entire corpus.
logLikelihood(JavaPairRDD<Long, Vector>) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
Java-friendly version of logLikelihood
LogLoss - Class in org.apache.spark.mllib.tree.loss
Developer API Class for log loss calculation (for classification).
LogLoss() - Constructor for class org.apache.spark.mllib.tree.loss.LogLoss
 
logName() - Method in interface org.apache.spark.internal.Logging
 
LogNormalGenerator - Class in org.apache.spark.mllib.random
Developer API Generates i.i.d.
LogNormalGenerator(double, double) - Constructor for class org.apache.spark.mllib.random.LogNormalGenerator
 
logNormalGraph(SparkContext, int, int, double, double, long) - Static method in class org.apache.spark.graphx.util.GraphGenerators
Generate a graph whose vertex out degree distribution is log normal.
logNormalJavaRDD(JavaSparkContext, double, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.logNormalRDD.
logNormalJavaRDD(JavaSparkContext, double, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.logNormalJavaRDD with the default seed.
logNormalJavaRDD(JavaSparkContext, double, double, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.logNormalJavaRDD with the default number of partitions and the default seed.
logNormalJavaVectorRDD(JavaSparkContext, double, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.logNormalVectorRDD.
logNormalJavaVectorRDD(JavaSparkContext, double, double, long, int, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.logNormalJavaVectorRDD with the default seed.
logNormalJavaVectorRDD(JavaSparkContext, double, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.logNormalJavaVectorRDD with the default number of partitions and the default seed.
logNormalRDD(SparkContext, double, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD comprised of i.i.d. samples from the log normal distribution with the input mean and standard deviation
logNormalVectorRDD(SparkContext, double, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD[Vector] with vectors containing i.i.d. samples drawn from a log normal distribution.
logpdf(Vector) - Method in class org.apache.spark.ml.stat.distribution.MultivariateGaussian
Returns the log-density of this multivariate Gaussian at given point, x
logpdf(Vector) - Method in class org.apache.spark.mllib.stat.distribution.MultivariateGaussian
Returns the log-density of this multivariate Gaussian at given point, x
logPerplexity(Dataset<?>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
logPerplexity(Dataset<?>) - Method in class org.apache.spark.ml.clustering.LDAModel
Calculate an upper bound on perplexity.
logPerplexity(Dataset<?>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
logPerplexity(RDD<Tuple2<Object, Vector>>) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
Calculate an upper bound on perplexity.
logPerplexity(JavaPairRDD<Long, Vector>) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
Java-friendly version of logPerplexity
logPrior() - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
Log probability of the current parameter estimate: log P(topics, topic distributions for docs | Dirichlet hyperparameters)
logPrior() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Log probability of the current parameter estimate: log P(topics, topic distributions for docs | alpha, eta)
logStartFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
logStartToJson(SparkListenerLogStart) - Static method in class org.apache.spark.util.JsonProtocol
 
logTrace(Function0<String>) - Method in interface org.apache.spark.internal.Logging
 
logTrace(Function0<String>, Throwable) - Method in interface org.apache.spark.internal.Logging
 
logUncaughtExceptions(Function0<T>) - Static method in class org.apache.spark.util.Utils
Execute the given block, logging and re-throwing any uncaught exception.
logUrlMap() - Method in class org.apache.spark.scheduler.cluster.ExecutorInfo
 
logUrls() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
logWarning(Function0<String>) - Method in interface org.apache.spark.internal.Logging
 
logWarning(Function0<String>, Throwable) - Method in interface org.apache.spark.internal.Logging
 
LONG() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable long type.
longAccumulator() - Method in class org.apache.spark.SparkContext
Create and register a long accumulator, which starts with 0 and accumulates inputs by add.
longAccumulator(String) - Method in class org.apache.spark.SparkContext
Create and register a long accumulator, which starts with 0 and accumulates inputs by add.
LongAccumulator - Class in org.apache.spark.util
An accumulator for computing sum, count, and average of 64-bit integers.
LongAccumulator() - Constructor for class org.apache.spark.util.LongAccumulator
 
LongAccumulatorParam$() - Constructor for class org.apache.spark.AccumulatorParam.LongAccumulatorParam$
Deprecated.
 
longMetric(String) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
LongParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Long] for Java.
LongParam(String, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.LongParam
 
LongParam(String, String, String) - Constructor for class org.apache.spark.ml.param.LongParam
 
LongParam(Identifiable, String, String, Function1<Object, Object>) - Constructor for class org.apache.spark.ml.param.LongParam
 
LongParam(Identifiable, String, String) - Constructor for class org.apache.spark.ml.param.LongParam
 
LongType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the LongType object.
LongType - Class in org.apache.spark.sql.types
The data type representing Long values.
lookup(K) - Method in class org.apache.spark.api.java.JavaPairRDD
Return the list of values in the RDD for key key.
lookup(K) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return the list of values in the RDD for key key.
lookupRpcTimeout(SparkConf) - Static method in class org.apache.spark.util.RpcUtils
Returns the default Spark timeout to use for RPC remote endpoint lookup.
loss() - Method in interface org.apache.spark.ml.param.shared.HasLoss
Param for the loss function to be optimized.
loss() - Method in class org.apache.spark.ml.regression.AFTAggregator
 
loss() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
loss() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
loss() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
Loss - Interface in org.apache.spark.mllib.tree.loss
Developer API Trait for adding "pluggable" loss functions for the gradient boosting algorithm.
Losses - Class in org.apache.spark.mllib.tree.loss
 
Losses() - Constructor for class org.apache.spark.mllib.tree.loss.Losses
 
LossReasonPending - Class in org.apache.spark.scheduler
A loss reason that means we don't yet know why the executor exited.
LossReasonPending() - Constructor for class org.apache.spark.scheduler.LossReasonPending
 
lossType() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
lossType() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
lossType() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
lossType() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
LOST() - Static method in class org.apache.spark.TaskState
 
low() - Method in class org.apache.spark.partial.BoundedDouble
 
lower(Column) - Static method in class org.apache.spark.sql.functions
Converts a string column to lower case.
lowerBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
lowerBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
lowerBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
lowerBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
LowPrioritySQLImplicits - Interface in org.apache.spark.sql
Lower priority implicit methods for converting Scala objects into Datasets.
lpad(Column, int, String) - Static method in class org.apache.spark.sql.functions
Left-pad the string column with pad to a length of len.
lt(double) - Static method in class org.apache.spark.ml.param.ParamValidators
Check if value is less than upperBound
lt(Object) - Method in class org.apache.spark.sql.Column
Less than.
ltEq(double) - Static method in class org.apache.spark.ml.param.ParamValidators
Check if value is less than or equal to upperBound
ltrim(Column) - Static method in class org.apache.spark.sql.functions
Trim the spaces from left end for the specified string value.
ltrim(Column, String) - Static method in class org.apache.spark.sql.functions
Trim the specified character string from left end for the specified string column.
LZ4CompressionCodec - Class in org.apache.spark.io
Developer API LZ4 implementation of CompressionCodec.
LZ4CompressionCodec(SparkConf) - Constructor for class org.apache.spark.io.LZ4CompressionCodec
 
LZFCompressionCodec - Class in org.apache.spark.io
Developer API LZF implementation of CompressionCodec.
LZFCompressionCodec(SparkConf) - Constructor for class org.apache.spark.io.LZFCompressionCodec
 

M

main(String[]) - Static method in class org.apache.spark.ml.param.shared.SharedParamsCodeGen
 
main(String[]) - Static method in class org.apache.spark.mllib.util.KMeansDataGenerator
 
main(String[]) - Static method in class org.apache.spark.mllib.util.LinearDataGenerator
 
main(String[]) - Static method in class org.apache.spark.mllib.util.LogisticRegressionDataGenerator
 
main(String[]) - Static method in class org.apache.spark.mllib.util.MFDataGenerator
 
main(String[]) - Static method in class org.apache.spark.mllib.util.SVMDataGenerator
 
main(String[]) - Static method in class org.apache.spark.streaming.util.RawTextSender
 
main(String[]) - Static method in class org.apache.spark.ui.UIWorkloadGenerator
 
majorMinorVersion(String) - Static method in class org.apache.spark.util.VersionUtils
Given a Spark version string, return the (major version number, minor version number).
majorVersion(String) - Static method in class org.apache.spark.util.VersionUtils
Given a Spark version string, return the major version number.
makeBinarySearch(Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.util.CollectionsUtils
 
makeCopy(Object[]) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
makeCopy(Object[]) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
makeCopy(Object[]) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
makeCopy(Object[]) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
makeDescription(String, String, boolean) - Static method in class org.apache.spark.ui.UIUtils
Returns HTML rendering of a job or stage description.
makeDriverRef(String, SparkConf, org.apache.spark.rpc.RpcEnv) - Static method in class org.apache.spark.util.RpcUtils
Retrieve a RpcEndpointRef which is located in the driver via its name.
makeHref(boolean, String, String) - Static method in class org.apache.spark.ui.UIUtils
Return the correct Href after checking if master is running in the reverse proxy mode or not.
makeProgressBar(int, int, int, int, Map<String, Object>, int) - Static method in class org.apache.spark.ui.UIUtils
 
makeRDD(Seq<T>, int, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Distribute a local Scala collection to form an RDD.
makeRDD(Seq<Tuple2<T, Seq<String>>>, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Distribute a local Scala collection to form an RDD, with one or more location preferences (hostnames of Spark nodes) for each object.
map(Function<T, R>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
map(Function<T, R>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
map(Function<T, R>) - Static method in class org.apache.spark.api.java.JavaRDD
 
map(Function<T, R>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to all elements of this RDD.
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
map(Function1<Object, Object>) - Method in interface org.apache.spark.ml.linalg.Matrix
Map the values of this matrix using a function.
map(Function1<Object, Object>) - Method in interface org.apache.spark.mllib.linalg.Matrix
Map the values of this matrix using a function.
map(Function1<R, T>) - Method in class org.apache.spark.partial.PartialResult
Transform this PartialResult into a PartialResult of type T.
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
map(Function1<T, U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD by applying a function to all elements of this RDD.
map(Function1<T, U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
map(DataType, DataType) - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type map.
map(MapType) - Method in class org.apache.spark.sql.ColumnName
 
map(Function1<T, U>, Encoder<U>) - Method in class org.apache.spark.sql.Dataset
Experimental (Scala-specific) Returns a new Dataset that contains the result of applying func to each element.
map(MapFunction<T, U>, Encoder<U>) - Method in class org.apache.spark.sql.Dataset
Experimental (Java-specific) Returns a new Dataset that contains the result of applying func to each element.
map(Column...) - Static method in class org.apache.spark.sql.functions
Creates a new map column.
map(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Creates a new map column.
map(Function1<BaseType, A>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
map(Function1<BaseType, A>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
map(Function1<BaseType, A>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
map(Function1<BaseType, A>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
map(Function1<A, B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
map(Function<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
map(Function<T, U>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream by applying a function to all elements of this DStream.
map(Function<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
map(Function<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
map(Function<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
map(Function<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
map(Function<T, U>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
map(Function1<T, U>, ClassTag<U>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream by applying a function to all elements of this DStream.
map_keys(Column) - Static method in class org.apache.spark.sql.functions
Returns an unordered array containing the keys of the map.
map_values(Column) - Static method in class org.apache.spark.sql.functions
Returns an unordered array containing the values of the map.
mapAsSerializableJavaMap(Map<A, B>) - Static method in class org.apache.spark.api.java.JavaUtils
 
mapChildren(Function1<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
mapChildren(Function1<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
mapChildren(Function1<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
mapChildren(Function1<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
mapEdgePartitions(Function2<Object, EdgePartition<ED, VD>, EdgePartition<ED2, VD2>>, ClassTag<ED2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapEdges(Function1<Edge<ED>, ED2>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.Graph
Transforms each edge attribute in the graph using the map function.
mapEdges(Function2<Object, Iterator<Edge<ED>>, Iterator<ED2>>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.Graph
Transforms each edge attribute using the map function, passing it a whole partition at a time.
mapEdges(Function2<Object, Iterator<Edge<ED>>, Iterator<ED2>>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
mapExpressions(Function1<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
mapExpressions(Function1<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
mapExpressions(Function1<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
mapExpressions(Function1<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
mapFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
-------------------------------- * Util JSON deserialization methods |
MapFunction<T,U> - Interface in org.apache.spark.api.java.function
Base interface for a map function used in Dataset's map function.
mapGroups(Function2<K, Iterator<V>, U>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Scala-specific) Applies the given function to each group of data.
mapGroups(MapGroupsFunction<K, V, U>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Java-specific) Applies the given function to each group of data.
MapGroupsFunction<K,V,R> - Interface in org.apache.spark.api.java.function
Base interface for a map function used in GroupedDataset's mapGroup function.
mapGroupsWithState(Function3<K, Iterator<V>, GroupState<S>, U>, Encoder<S>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
::Experimental:: (Scala-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state.
mapGroupsWithState(GroupStateTimeout, Function3<K, Iterator<V>, GroupState<S>, U>, Encoder<S>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
::Experimental:: (Scala-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state.
mapGroupsWithState(MapGroupsWithStateFunction<K, V, S, U>, Encoder<S>, Encoder<U>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
::Experimental:: (Java-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state.
mapGroupsWithState(MapGroupsWithStateFunction<K, V, S, U>, Encoder<S>, Encoder<U>, GroupStateTimeout) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
::Experimental:: (Java-specific) Applies the given function to each group of data, while maintaining a user-defined per-group state.
MapGroupsWithStateFunction<K,V,S,R> - Interface in org.apache.spark.api.java.function
mapId() - Method in class org.apache.spark.FetchFailed
 
mapId() - Method in class org.apache.spark.storage.ShuffleBlockId
 
mapId() - Method in class org.apache.spark.storage.ShuffleDataBlockId
 
mapId() - Method in class org.apache.spark.storage.ShuffleIndexBlockId
 
mapOutputTracker() - Method in class org.apache.spark.SparkEnv
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitions(FlatMapFunction<Iterator<T>, U>, boolean) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitions(FlatMapFunction<Iterator<T>, U>, boolean) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitions(FlatMapFunction<Iterator<T>, U>, boolean) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to each partition of this RDD.
mapPartitions(FlatMapFunction<Iterator<T>, U>, boolean) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to each partition of this RDD.
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD by applying a function to each partition of this RDD.
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, Encoder<U>) - Method in class org.apache.spark.sql.Dataset
Experimental (Scala-specific) Returns a new Dataset that contains the result of applying func to each partition.
mapPartitions(MapPartitionsFunction<T, U>, Encoder<U>) - Method in class org.apache.spark.sql.Dataset
Experimental (Java-specific) Returns a new Dataset that contains the result of applying f to each partition.
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying mapPartitions() to each RDDs of this DStream.
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
mapPartitions(FlatMapFunction<Iterator<T>, U>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
mapPartitions(Function1<Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD is generated by applying mapPartitions() to each RDDs of this DStream.
mapPartitions$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
mapPartitions$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
mapPartitions$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapPartitions$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapPartitions$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
mapPartitions$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
mapPartitions$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
mapPartitions$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
mapPartitions$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
mapPartitions$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
MapPartitionsFunction<T,U> - Interface in org.apache.spark.api.java.function
Base interface for function used in Dataset's mapPartitions.
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
mapPartitionsInternal$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>, boolean) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>, boolean) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>, boolean) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to each partition of this RDD.
mapPartitionsToDouble(DoubleFlatMapFunction<Iterator<T>>, boolean) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to each partition of this RDD.
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>, boolean) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>, boolean) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>, boolean) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to each partition of this RDD.
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>, boolean) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to each partition of this RDD.
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying mapPartitions() to each RDDs of this DStream.
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
mapPartitionsToPair(PairFlatMapFunction<Iterator<T>, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
mapPartitionsWithIndex(Function2<Integer, Iterator<T>, Iterator<R>>, boolean) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitionsWithIndex(Function2<Integer, Iterator<T>, Iterator<R>>, boolean) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitionsWithIndex(Function2<Integer, Iterator<T>, Iterator<R>>, boolean) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitionsWithIndex(Function2<Integer, Iterator<T>, Iterator<R>>, boolean) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD by applying a function to each partition of this RDD, while tracking the index of the original partition.
mapPartitionsWithIndex(Function2<Object, Iterator<T>, Iterator<U>>, boolean, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.api.java.JavaRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
mapPartitionsWithIndex$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
mapPartitionsWithIndexInternal$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.api.r.RRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.graphx.EdgeRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.graphx.VertexRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.rdd.HadoopRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.rdd.JdbcRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
mapPartitionsWithIndexInternal$default$3() - Static method in class org.apache.spark.rdd.UnionRDD
 
mapPartitionsWithInputSplit(Function2<InputSplit, Iterator<Tuple2<K, V>>, Iterator<R>>, boolean) - Method in class org.apache.spark.api.java.JavaHadoopRDD
Maps over a partition, providing the InputSplit that was used as the base of the partition.
mapPartitionsWithInputSplit(Function2<InputSplit, Iterator<Tuple2<K, V>>, Iterator<R>>, boolean) - Method in class org.apache.spark.api.java.JavaNewHadoopRDD
Maps over a partition, providing the InputSplit that was used as the base of the partition.
mapPartitionsWithInputSplit(Function2<InputSplit, Iterator<Tuple2<K, V>>, Iterator<U>>, boolean, ClassTag<U>) - Method in class org.apache.spark.rdd.HadoopRDD
Maps over a partition, providing the InputSplit that was used as the base of the partition.
mapPartitionsWithInputSplit(Function2<InputSplit, Iterator<Tuple2<K, V>>, Iterator<U>>, boolean, ClassTag<U>) - Method in class org.apache.spark.rdd.NewHadoopRDD
Maps over a partition, providing the InputSplit that was used as the base of the partition.
mapredInputFormat() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
mapreduceInputFormat() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
mapSideCombine() - Method in class org.apache.spark.ShuffleDependency
 
mapStatuses() - Method in class org.apache.spark.ShuffleStatus
MapStatus for each partition.
mapToDouble(DoubleFunction<T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapToDouble(DoubleFunction<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapToDouble(DoubleFunction<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapToDouble(DoubleFunction<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to all elements of this RDD.
mapToJson(Map<String, String>) - Static method in class org.apache.spark.util.JsonProtocol
------------------------------ * Util JSON serialization methods |
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.api.java.JavaRDD
 
mapToPair(PairFunction<T, K2, V2>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return a new RDD by applying a function to all elements of this RDD.
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
mapToPair(PairFunction<T, K2, V2>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream by applying a function to all elements of this DStream.
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
mapToPair(PairFunction<T, K2, V2>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
mapTriplets(Function1<EdgeTriplet<VD, ED>, ED2>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.Graph
Transforms each edge attribute using the map function, passing it the adjacent vertex attributes as well.
mapTriplets(Function1<EdgeTriplet<VD, ED>, ED2>, TripletFields, ClassTag<ED2>) - Method in class org.apache.spark.graphx.Graph
Transforms each edge attribute using the map function, passing it the adjacent vertex attributes as well.
mapTriplets(Function2<Object, Iterator<EdgeTriplet<VD, ED>>, Iterator<ED2>>, TripletFields, ClassTag<ED2>) - Method in class org.apache.spark.graphx.Graph
Transforms each edge attribute a partition at a time using the map function, passing it the adjacent vertex attributes as well.
mapTriplets(Function2<Object, Iterator<EdgeTriplet<VD, ED>>, Iterator<ED2>>, TripletFields, ClassTag<ED2>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
MapType - Class in org.apache.spark.sql.types
The data type for Maps.
MapType(DataType, DataType, boolean) - Constructor for class org.apache.spark.sql.types.MapType
 
MapType() - Constructor for class org.apache.spark.sql.types.MapType
No-arg constructor for kryo.
mapValues(Function<V, U>) - Method in class org.apache.spark.api.java.JavaPairRDD
Pass each value in the key-value pair RDD through a map function without changing the keys; this also retains the original RDD's partitioning.
mapValues(Function1<Edge<ED>, ED2>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.EdgeRDD
Map the values in an edge partitioning preserving the structure but changing the values.
mapValues(Function1<Edge<ED>, ED2>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
mapValues(Function1<VD, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapValues(Function2<Object, VD, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
mapValues(Function1<VD, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.VertexRDD
Maps each vertex attribute, preserving the index.
mapValues(Function2<Object, VD, VD2>, ClassTag<VD2>) - Method in class org.apache.spark.graphx.VertexRDD
Maps each vertex attribute, additionally supplying the vertex ID.
mapValues(Function1<V, U>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Pass each value in the key-value pair RDD through a map function without changing the keys; this also retains the original RDD's partitioning.
mapValues(Function1<V, W>, Encoder<W>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Returns a new KeyValueGroupedDataset where the given function func has been applied to the data.
mapValues(MapFunction<V, W>, Encoder<W>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
Returns a new KeyValueGroupedDataset where the given function func has been applied to the data.
mapValues(Function<V, U>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying a map function to the value of each key-value pairs in 'this' DStream without changing the key.
mapValues(Function<V, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
mapValues(Function<V, U>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
mapValues(Function1<V, U>, ClassTag<U>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying a map function to the value of each key-value pairs in 'this' DStream without changing the key.
mapVertices(Function2<Object, VD, VD2>, ClassTag<VD2>, Predef.$eq$colon$eq<VD, VD2>) - Method in class org.apache.spark.graphx.Graph
Transforms each vertex attribute in the graph using the map function.
mapVertices(Function2<Object, VD, VD2>, ClassTag<VD2>, Predef.$eq$colon$eq<VD, VD2>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
mapVertices$default$3(Function2<Object, VD, VD2>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
mapWithState(StateSpec<K, V, StateType, MappedType>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Experimental Return a JavaMapWithStateDStream by applying a function to every key-value element of this stream, while maintaining some state data for each unique key.
mapWithState(StateSpec<K, V, StateType, MappedType>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
mapWithState(StateSpec<K, V, StateType, MappedType>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
mapWithState(StateSpec<K, V, StateType, MappedType>, ClassTag<StateType>, ClassTag<MappedType>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Experimental Return a MapWithStateDStream by applying a function to every key-value element of this stream, while maintaining some state data for each unique key.
MapWithStateDStream<KeyType,ValueType,StateType,MappedType> - Class in org.apache.spark.streaming.dstream
Experimental DStream representing the stream of data generated by mapWithState operation on a pair DStream.
MapWithStateDStream(StreamingContext, ClassTag<MappedType>) - Constructor for class org.apache.spark.streaming.dstream.MapWithStateDStream
 
mark(int) - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
markSupported() - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
mask(Graph<VD2, ED2>, ClassTag<VD2>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.Graph
Restricts the graph to only the vertices and edges that are also in other, but keeps the attributes from this graph.
mask(Graph<VD2, ED2>, ClassTag<VD2>, ClassTag<ED2>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
master() - Method in class org.apache.spark.api.java.JavaSparkContext
 
master() - Method in class org.apache.spark.SparkContext
 
master(String) - Method in class org.apache.spark.sql.SparkSession.Builder
Sets the Spark master URL to connect to, such as "local" to run locally, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
Matrices - Class in org.apache.spark.ml.linalg
Factory methods for Matrix.
Matrices() - Constructor for class org.apache.spark.ml.linalg.Matrices
 
Matrices - Class in org.apache.spark.mllib.linalg
Factory methods for Matrix.
Matrices() - Constructor for class org.apache.spark.mllib.linalg.Matrices
 
Matrix - Interface in org.apache.spark.ml.linalg
Trait for a local matrix.
Matrix - Interface in org.apache.spark.mllib.linalg
Trait for a local matrix.
MatrixEntry - Class in org.apache.spark.mllib.linalg.distributed
Represents an entry in a distributed matrix.
MatrixEntry(long, long, double) - Constructor for class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
MatrixFactorizationModel - Class in org.apache.spark.mllib.recommendation
Model representing the result of matrix factorization.
MatrixFactorizationModel(int, RDD<Tuple2<Object, double[]>>, RDD<Tuple2<Object, double[]>>) - Constructor for class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
 
MatrixFactorizationModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.recommendation
 
MatrixImplicits - Class in org.apache.spark.mllib.linalg
Implicit methods available in Scala for converting Matrix to Matrix and vice versa.
MatrixImplicits() - Constructor for class org.apache.spark.mllib.linalg.MatrixImplicits
 
MatrixType() - Static method in class org.apache.spark.ml.linalg.SQLDataTypes
Data type for Matrix.
max() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Returns the maximum element from this RDD as defined by the default comparator natural order.
max(Comparator<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
max(Comparator<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
max(Comparator<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Returns the maximum element from this RDD as defined by the specified Comparator[T].
max(Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
max(Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
max(Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
max(Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
max(Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
MAX() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
max() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
max() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
max() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
max(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
max(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
max() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Maximum value of each dimension.
max() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Maximum value of each column.
max(Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
max(Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
max(Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
max(Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
max(Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Returns the max of this RDD as defined by the implicit Ordering[T].
max(Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
max(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the maximum value of the expression in a group.
max(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the maximum value of the column in a group.
max(String...) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the max value for each numeric columns for each group.
max(Seq<String>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the max value for each numeric columns for each group.
max(Ordering<B>) - Static method in class org.apache.spark.sql.types.StructType
 
max(Duration) - Method in class org.apache.spark.streaming.Duration
 
max(Time) - Method in class org.apache.spark.streaming.Time
 
max(long, long) - Static method in class org.apache.spark.streaming.util.RawTextHelper
 
max() - Method in class org.apache.spark.util.StatCounter
 
MAX_FEATURES_FOR_NORMAL_SOLVER() - Static method in class org.apache.spark.ml.regression.LinearRegression
When using LinearRegression.solver == "normal", the solver must limit the number of features to at most this number.
MAX_INT_DIGITS() - Static method in class org.apache.spark.sql.types.Decimal
Maximum number of decimal digits an Int can represent
MAX_LONG_DIGITS() - Static method in class org.apache.spark.sql.types.Decimal
Maximum number of decimal digits a Long can represent
MAX_PRECISION() - Static method in class org.apache.spark.sql.types.DecimalType
 
MAX_RETAINED_DEAD_EXECUTORS() - Static method in class org.apache.spark.status.config
 
MAX_RETAINED_JOBS() - Static method in class org.apache.spark.status.config
 
MAX_RETAINED_ROOT_NODES() - Static method in class org.apache.spark.status.config
 
MAX_RETAINED_STAGES() - Static method in class org.apache.spark.status.config
 
MAX_RETAINED_TASKS_PER_STAGE() - Static method in class org.apache.spark.status.config
 
MAX_SCALE() - Static method in class org.apache.spark.sql.types.DecimalType
 
maxAbs() - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
MaxAbsScaler - Class in org.apache.spark.ml.feature
Rescale each feature individually to range [-1, 1] by dividing through the largest maximum absolute value in each feature.
MaxAbsScaler(String) - Constructor for class org.apache.spark.ml.feature.MaxAbsScaler
 
MaxAbsScaler() - Constructor for class org.apache.spark.ml.feature.MaxAbsScaler
 
MaxAbsScalerModel - Class in org.apache.spark.ml.feature
Model fitted by MaxAbsScaler.
maxBins() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
maxBins() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
maxBins() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
maxBins() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
maxBins() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
maxBins() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
maxBins() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
maxBins() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
maxBins() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
maxBins() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
maxBins() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
maxBins() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
maxBins() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
maxBufferSizeMb() - Method in class org.apache.spark.serializer.KryoSerializer
 
maxBy(Function1<A, B>, Ordering<B>) - Static method in class org.apache.spark.sql.types.StructType
 
maxCategories() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
maxCategories() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
maxCores() - Method in class org.apache.spark.status.api.v1.ApplicationInfo
 
maxDepth() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
maxDepth() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
maxDepth() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
maxDepth() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
maxDepth() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
maxDepth() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
maxDepth() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
maxDepth() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
maxDepth() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
maxDepth() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
maxDepth() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
maxDepth() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
maxDepth() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
maxId() - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
maxId() - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
maxId() - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
maxId() - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
maxId() - Static method in class org.apache.spark.rdd.CheckpointState
 
maxId() - Static method in class org.apache.spark.rdd.DeterministicLevel
 
maxId() - Static method in class org.apache.spark.scheduler.SchedulingMode
 
maxId() - Static method in class org.apache.spark.scheduler.TaskLocality
 
maxId() - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
maxId() - Static method in class org.apache.spark.TaskState
 
maxIter() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
maxIter() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
maxIter() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
maxIter() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
maxIter() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
maxIter() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
maxIter() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
maxIter() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
maxIter() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
maxIter() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
maxIter() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
maxIter() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
maxIter() - Static method in class org.apache.spark.ml.clustering.KMeans
 
maxIter() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
maxIter() - Static method in class org.apache.spark.ml.clustering.LDA
 
maxIter() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
maxIter() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
maxIter() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
maxIter() - Method in interface org.apache.spark.ml.param.shared.HasMaxIter
Param for maximum number of iterations (&gt;= 0).
maxIter() - Static method in class org.apache.spark.ml.recommendation.ALS
 
maxIter() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
maxIter() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
maxIter() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
maxIter() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
maxIter() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
maxIter() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
maxIter() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
maxIter() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
maxIters() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
maxMem() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
maxMem() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the max memory can be used by this block manager.
maxMemory() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
maxMemory() - Method in class org.apache.spark.status.LiveExecutor
 
maxMemory() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
maxMemoryInMB() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
maxMemoryInMB() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
maxMessageSizeBytes(SparkConf) - Static method in class org.apache.spark.util.RpcUtils
Returns the configured max message size for messages in bytes.
maxNodesInLevel(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Return the maximum number of nodes which can be in the given level of the tree.
maxOffHeapMem() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
maxOffHeapMem() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
 
maxOffHeapMemSize() - Method in class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager
 
maxOnHeapMem() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
maxOnHeapMem() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
 
maxOnHeapMemSize() - Method in class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager
 
maxReplicas() - Method in class org.apache.spark.storage.BlockManagerMessages.ReplicateBlock
 
maxRows() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
maxRows() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
maxRows() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
maxRowsPerPartition() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
maxRowsPerPartition() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
maxRowsPerPartition() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
maxSentenceLength() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
maxSentenceLength() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
maxTasks() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
maxTasks() - Method in class org.apache.spark.status.LiveExecutor
 
maxVal() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
maybeUpdateOutputMetrics(OutputMetrics, Function0<Object>, long) - Static method in class org.apache.spark.internal.io.SparkHadoopWriterUtils
 
md5(Column) - Static method in class org.apache.spark.sql.functions
Calculates the MD5 digest of a binary column and returns the value as a 32 character hex string.
mean() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute the mean of this RDD's elements.
mean() - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
mean() - Method in class org.apache.spark.ml.stat.distribution.MultivariateGaussian
 
mean(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
mean(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
mean() - Method in class org.apache.spark.mllib.feature.StandardScalerModel
 
mean() - Method in class org.apache.spark.mllib.random.ExponentialGenerator
 
mean() - Method in class org.apache.spark.mllib.random.LogNormalGenerator
 
mean() - Method in class org.apache.spark.mllib.random.PoissonGenerator
 
mean() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Sample mean of each dimension.
mean() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Sample mean vector.
mean() - Method in class org.apache.spark.partial.BoundedDouble
 
mean() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute the mean of this RDD's elements.
mean(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the average of the values in a group.
mean(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the average of the values in a group.
mean(String...) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the average value for each numeric columns for each group.
mean(Seq<String>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the average value for each numeric columns for each group.
mean() - Method in class org.apache.spark.util.StatCounter
 
meanAbsoluteError() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Returns the mean absolute error, which is a risk function corresponding to the expected value of the absolute error loss or l1-norm loss.
meanAbsoluteError() - Method in class org.apache.spark.mllib.evaluation.RegressionMetrics
Returns the mean absolute error, which is a risk function corresponding to the expected value of the absolute error loss or l1-norm loss.
meanApprox(long, Double) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return the approximate mean of the elements in this RDD.
meanApprox(long) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Approximate operation to return the mean within a timeout.
meanApprox(long, double) - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Approximate operation to return the mean within a timeout.
meanAveragePrecision() - Method in class org.apache.spark.mllib.evaluation.RankingMetrics
Returns the mean average precision (MAP) of all the queries.
means() - Method in class org.apache.spark.ml.clustering.ExpectationAggregator
 
means() - Method in class org.apache.spark.mllib.clustering.ExpectationSum
 
meanSquaredError() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Returns the mean squared error, which is a risk function corresponding to the expected value of the squared error loss or quadratic loss.
meanSquaredError() - Method in class org.apache.spark.mllib.evaluation.RegressionMetrics
Returns the mean squared error, which is a risk function corresponding to the expected value of the squared error loss or quadratic loss.
megabytesToString(long) - Static method in class org.apache.spark.util.Utils
Convert a quantity in megabytes to a human-readable string such as "4.0 MB".
MEM_SPILL() - Static method in class org.apache.spark.status.TaskIndexNames
 
MEMORY_AND_DISK - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_AND_DISK() - Static method in class org.apache.spark.storage.StorageLevel
 
MEMORY_AND_DISK_2 - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_AND_DISK_2() - Static method in class org.apache.spark.storage.StorageLevel
 
MEMORY_AND_DISK_SER - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_AND_DISK_SER() - Static method in class org.apache.spark.storage.StorageLevel
 
MEMORY_AND_DISK_SER_2 - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_AND_DISK_SER_2() - Static method in class org.apache.spark.storage.StorageLevel
 
MEMORY_BYTES_SPILLED() - Static method in class org.apache.spark.InternalAccumulator
 
MEMORY_ONLY - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_ONLY() - Static method in class org.apache.spark.storage.StorageLevel
 
MEMORY_ONLY_2 - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_ONLY_2() - Static method in class org.apache.spark.storage.StorageLevel
 
MEMORY_ONLY_SER - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_ONLY_SER() - Static method in class org.apache.spark.storage.StorageLevel
 
MEMORY_ONLY_SER_2 - Static variable in class org.apache.spark.api.java.StorageLevels
 
MEMORY_ONLY_SER_2() - Static method in class org.apache.spark.storage.StorageLevel
 
memoryBytesSpilled() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
memoryBytesSpilled() - Method in class org.apache.spark.status.api.v1.StageData
 
memoryBytesSpilled() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
memoryBytesSpilled() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
memoryCost(int, int) - Static method in class org.apache.spark.mllib.feature.PCAUtil
 
MemoryEntry<T> - Interface in org.apache.spark.storage.memory
 
memoryManager() - Method in class org.apache.spark.SparkEnv
 
memoryMetrics() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
MemoryMetrics - Class in org.apache.spark.status.api.v1
 
memoryMode() - Method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
memoryMode() - Method in interface org.apache.spark.storage.memory.MemoryEntry
 
memoryMode() - Method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
MemoryParam - Class in org.apache.spark.util
An extractor object for parsing JVM memory strings, such as "10g", into an Int representing the number of megabytes.
MemoryParam() - Constructor for class org.apache.spark.util.MemoryParam
 
memoryPerExecutorMB() - Method in class org.apache.spark.status.api.v1.ApplicationInfo
 
memoryRemaining() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
memoryStringToMb(String) - Static method in class org.apache.spark.util.Utils
Convert a Java memory parameter passed to -Xmx (such as 300m or 1g) to a number of mebibytes.
memoryUsed() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
memoryUsed() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
memoryUsed() - Method in class org.apache.spark.status.api.v1.RDDPartitionInfo
 
memoryUsed() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
memoryUsed() - Method in class org.apache.spark.status.LiveExecutor
 
memoryUsed() - Method in class org.apache.spark.status.LiveRDD
 
memoryUsed() - Method in class org.apache.spark.status.LiveRDDDistribution
 
memoryUsed() - Method in class org.apache.spark.status.LiveRDDPartition
 
memoryUsedBytes() - Method in class org.apache.spark.sql.streaming.StateOperatorProgress
 
memRemaining() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the memory remaining in this block manager.
memSize() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
memSize() - Method in class org.apache.spark.storage.BlockStatus
 
memSize() - Method in class org.apache.spark.storage.BlockUpdatedInfo
 
memSize() - Method in class org.apache.spark.storage.RDDInfo
 
memUsed() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the memory used by this block manager.
memUsedByRdd(int) - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the memory used by the given RDD in this block manager in O(1) time.
merge(R) - Method in class org.apache.spark.Accumulable
Deprecated.
Merge two accumulable objects together
merge(R) - Static method in class org.apache.spark.Accumulator
Deprecated.
 
merge(ExpectationAggregator) - Method in class org.apache.spark.ml.clustering.ExpectationAggregator
Merge another ExpectationAggregator, update the weights, means and covariances for each distributions, and update the log likelihood.
merge(AFTAggregator) - Method in class org.apache.spark.ml.regression.AFTAggregator
Merge another AFTAggregator, and update the loss and gradient of the objective function.
merge(IDF.DocumentFrequencyAggregator) - Method in class org.apache.spark.mllib.feature.IDF.DocumentFrequencyAggregator
Merges another.
merge(MultivariateOnlineSummarizer) - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Merge another MultivariateOnlineSummarizer, and update the statistical summary.
merge(BUF, BUF) - Method in class org.apache.spark.sql.expressions.Aggregator
Merge two intermediate values.
merge(MutableAggregationBuffer, Row) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Merges two aggregation buffers and stores the updated buffer values back to buffer1.
merge(AccumulatorV2<IN, OUT>) - Method in class org.apache.spark.util.AccumulatorV2
Merges another same-type accumulator into this one and update its state, i.e.
merge(AccumulatorV2<T, List<T>>) - Method in class org.apache.spark.util.CollectionAccumulator
 
merge(AccumulatorV2<Double, Double>) - Method in class org.apache.spark.util.DoubleAccumulator
 
merge(AccumulatorV2<T, R>) - Method in class org.apache.spark.util.LegacyAccumulatorWrapper
 
merge(AccumulatorV2<Long, Long>) - Method in class org.apache.spark.util.LongAccumulator
 
merge(double) - Method in class org.apache.spark.util.StatCounter
Add a value into this StatCounter, updating the internal statistics.
merge(TraversableOnce<Object>) - Method in class org.apache.spark.util.StatCounter
Add multiple values into this StatCounter, updating the internal statistics.
merge(StatCounter) - Method in class org.apache.spark.util.StatCounter
Merge another StatCounter into this one, adding up the internal statistics.
mergeCombiners() - Method in class org.apache.spark.Aggregator
 
mergeInPlace(BloomFilter) - Method in class org.apache.spark.util.sketch.BloomFilter
Combines this bloom filter with another bloom filter by performing a bitwise OR of the underlying data.
mergeInPlace(CountMinSketch) - Method in class org.apache.spark.util.sketch.CountMinSketch
Merges another CountMinSketch with this one in place.
mergeOffsets(PartitionOffset[]) - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.ContinuousReader
Merge partitioned offsets coming from ContinuousDataReader instances for each partition to a single global offset.
mergeValue() - Method in class org.apache.spark.Aggregator
 
message() - Method in class org.apache.spark.FetchFailed
 
message() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutorFailed
 
message() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveWorker
 
message() - Static method in class org.apache.spark.scheduler.ExecutorKilled
 
message() - Static method in class org.apache.spark.scheduler.LossReasonPending
 
message() - Method in exception org.apache.spark.sql.AnalysisException
 
message() - Method in exception org.apache.spark.sql.streaming.StreamingQueryException
 
message() - Method in class org.apache.spark.sql.streaming.StreamingQueryStatus
 
MetaAlgorithmReadWrite - Class in org.apache.spark.ml.util
Default Meta-Algorithm read and write implementation.
MetaAlgorithmReadWrite() - Constructor for class org.apache.spark.ml.util.MetaAlgorithmReadWrite
 
Metadata - Class in org.apache.spark.sql.types
Metadata is a wrapper over Map[String, Any] that limits the value type to simple ones: Boolean, Long, Double, String, Metadata, Array[Boolean], Array[Long], Array[Double], Array[String], and Array[Metadata].
metadata() - Method in class org.apache.spark.sql.types.StructField
 
metadata() - Method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
METADATA_KEY_DESCRIPTION() - Static method in class org.apache.spark.streaming.scheduler.StreamInputInfo
The key for description in StreamInputInfo.metadata.
MetadataBuilder - Class in org.apache.spark.sql.types
Builder for Metadata.
MetadataBuilder() - Constructor for class org.apache.spark.sql.types.MetadataBuilder
 
metadataDescription() - Method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
MetadataUtils - Class in org.apache.spark.ml.util
Helper utilities for algorithms using ML metadata
MetadataUtils() - Constructor for class org.apache.spark.ml.util.MetadataUtils
 
Method(String, Function2<Object, Object, Object>) - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest.Method
 
method() - Method in class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
Method$() - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest.Method$
 
MethodIdentifier<T> - Class in org.apache.spark.util
Helper class to identify a method.
MethodIdentifier(Class<T>, String, String) - Constructor for class org.apache.spark.util.MethodIdentifier
 
methodName() - Static method in class org.apache.spark.mllib.stat.test.StudentTTest
 
methodName() - Static method in class org.apache.spark.mllib.stat.test.WelchTTest
 
METRIC_COMPILATION_TIME() - Static method in class org.apache.spark.metrics.source.CodegenMetrics
Histogram of the time it took to compile source code text (in milliseconds).
METRIC_FILE_CACHE_HITS() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
Tracks the total number of files served from the file status cache instead of discovered.
METRIC_FILES_DISCOVERED() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
Tracks the total number of files discovered off of the filesystem by InMemoryFileIndex.
METRIC_GENERATED_CLASS_BYTECODE_SIZE() - Static method in class org.apache.spark.metrics.source.CodegenMetrics
Histogram of the bytecode size of each class generated by CodeGenerator.
METRIC_GENERATED_METHOD_BYTECODE_SIZE() - Static method in class org.apache.spark.metrics.source.CodegenMetrics
Histogram of the bytecode size of each method in classes generated by CodeGenerator.
METRIC_HIVE_CLIENT_CALLS() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
Tracks the total number of Hive client calls (e.g.
METRIC_PARALLEL_LISTING_JOB_COUNT() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
Tracks the total number of Spark jobs launched for parallel file listing.
METRIC_PARTITIONS_FETCHED() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
Tracks the total number of partition metadata entries fetched via the client api.
METRIC_SOURCE_CODE_SIZE() - Static method in class org.apache.spark.metrics.source.CodegenMetrics
Histogram of the length of source code text compiled by CodeGenerator (in characters).
metricName() - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
param for metric name in evaluation (supports "areaUnderROC" (default), "areaUnderPR")
metricName() - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
param for metric name in evaluation (supports "silhouette" (default))
metricName() - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
param for metric name in evaluation (supports "f1" (default), "weightedPrecision", "weightedRecall", "accuracy")
metricName() - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
Param for metric name in evaluation.
metricRegistry() - Static method in class org.apache.spark.metrics.source.CodegenMetrics
 
metricRegistry() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
 
metrics(String...) - Static method in class org.apache.spark.ml.stat.Summarizer
Given a list of metrics, provides a builder that it turns computes metrics from a column.
metrics(Seq<String>) - Static method in class org.apache.spark.ml.stat.Summarizer
Given a list of metrics, provides a builder that it turns computes metrics from a column.
metrics() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
metrics() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
metrics() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
metrics() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
metrics() - Method in class org.apache.spark.status.LiveExecutorStageSummary
 
metrics() - Method in class org.apache.spark.status.LiveStage
 
METRICS_PREFIX() - Static method in class org.apache.spark.InternalAccumulator
 
metricsSystem() - Method in class org.apache.spark.SparkEnv
 
MFDataGenerator - Class in org.apache.spark.mllib.util
Developer API Generate RDD(s) containing data for Matrix Factorization.
MFDataGenerator() - Constructor for class org.apache.spark.mllib.util.MFDataGenerator
 
MicroBatchReader - Interface in org.apache.spark.sql.sources.v2.reader.streaming
A mix-in interface for DataSourceReader.
MicroBatchReadSupport - Interface in org.apache.spark.sql.sources.v2
A mix-in interface for DataSourceV2.
microF1Measure() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns micro-averaged label-based f1-measure (equals to micro-averaged document-based f1-measure)
microPrecision() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns micro-averaged label-based precision (equals to micro-averaged document-based precision)
microRecall() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns micro-averaged label-based recall (equals to micro-averaged document-based recall)
mightContain(Object) - Method in class org.apache.spark.util.sketch.BloomFilter
Returns true if the element might have been put in this Bloom filter, false if this is definitely not the case.
mightContainBinary(byte[]) - Method in class org.apache.spark.util.sketch.BloomFilter
A specialized variant of BloomFilter.mightContain(Object) that only tests byte array items.
mightContainLong(long) - Method in class org.apache.spark.util.sketch.BloomFilter
A specialized variant of BloomFilter.mightContain(Object) that only tests long items.
mightContainString(String) - Method in class org.apache.spark.util.sketch.BloomFilter
A specialized variant of BloomFilter.mightContain(Object) that only tests String items.
milliseconds() - Method in class org.apache.spark.streaming.Duration
 
milliseconds(long) - Static method in class org.apache.spark.streaming.Durations
 
Milliseconds - Class in org.apache.spark.streaming
Helper object that creates instance of Duration representing a given number of milliseconds.
Milliseconds() - Constructor for class org.apache.spark.streaming.Milliseconds
 
milliseconds() - Method in class org.apache.spark.streaming.Time
 
millisToString(long) - Static method in class org.apache.spark.scheduler.StatsReportListener
Reformat a time interval in milliseconds to a prettier format for output
min() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Returns the minimum element from this RDD as defined by the default comparator natural order.
min(Comparator<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
min(Comparator<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
min(Comparator<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Returns the minimum element from this RDD as defined by the specified Comparator[T].
min(Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
min(Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
min(Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
min(Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
min(Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
MIN() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
min() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
min() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
min() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
min(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
min(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
min() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Minimum value of each dimension.
min() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Minimum value of each column.
min(Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
min(Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
min(Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
min(Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
min(Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Returns the min of this RDD as defined by the implicit Ordering[T].
min(Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
min(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the minimum value of the expression in a group.
min(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the minimum value of the column in a group.
min(String...) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the min value for each numeric column for each group.
min(Seq<String>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the min value for each numeric column for each group.
min(Ordering<B>) - Static method in class org.apache.spark.sql.types.StructType
 
min(Duration) - Method in class org.apache.spark.streaming.Duration
 
min(Time) - Method in class org.apache.spark.streaming.Time
 
min() - Method in class org.apache.spark.util.StatCounter
 
minBy(Function1<A, B>, Ordering<B>) - Static method in class org.apache.spark.sql.types.StructType
 
minConfidence() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
minConfidence() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
minCount() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
minCount() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
minDF() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
minDF() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
minDivisibleClusterSize() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
minDivisibleClusterSize() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
minDocFreq() - Static method in class org.apache.spark.ml.feature.IDF
 
minDocFreq() - Static method in class org.apache.spark.ml.feature.IDFModel
 
minDocFreq() - Method in class org.apache.spark.mllib.feature.IDF.DocumentFrequencyAggregator
 
minDocFreq() - Method in class org.apache.spark.mllib.feature.IDF
 
MinHashLSH - Class in org.apache.spark.ml.feature
Experimental
MinHashLSH(String) - Constructor for class org.apache.spark.ml.feature.MinHashLSH
 
MinHashLSH() - Constructor for class org.apache.spark.ml.feature.MinHashLSH
 
MinHashLSHModel - Class in org.apache.spark.ml.feature
Experimental
MINIMUM_ADJUSTED_SCALE() - Static method in class org.apache.spark.sql.types.DecimalType
 
minInfoGain() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
minInfoGain() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
minInfoGain() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
minInfoGain() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
minInfoGain() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
minInfoGain() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
minInfoGain() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
minInfoGain() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
minInfoGain() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
minInfoGain() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
minInfoGain() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
minInfoGain() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
minInfoGain() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
minInstancesPerNode() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
minInstancesPerNode() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
MinMax() - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
MinMaxScaler - Class in org.apache.spark.ml.feature
Rescale each feature individually to a common range [min, max] linearly using column summary statistics, which is also known as min-max normalization or Rescaling.
MinMaxScaler(String) - Constructor for class org.apache.spark.ml.feature.MinMaxScaler
 
MinMaxScaler() - Constructor for class org.apache.spark.ml.feature.MinMaxScaler
 
MinMaxScalerModel - Class in org.apache.spark.ml.feature
Model fitted by MinMaxScaler.
minorVersion(String) - Static method in class org.apache.spark.util.VersionUtils
Given a Spark version string, return the minor version number.
minSamplingRate() - Static method in class org.apache.spark.util.random.BinomialBounds
 
minSupport() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
minSupport() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
minTF() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
minTF() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
minTokenLength() - Method in class org.apache.spark.ml.feature.RegexTokenizer
Minimum token length, greater than or equal to 0.
minus(RDD<Tuple2<Object, VD>>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
minus(VertexRDD<VD>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
minus(RDD<Tuple2<Object, VD>>) - Method in class org.apache.spark.graphx.VertexRDD
For each VertexId present in both this and other, minus will act as a set difference operation returning only those unique VertexId's present in this.
minus(VertexRDD<VD>) - Method in class org.apache.spark.graphx.VertexRDD
For each VertexId present in both this and other, minus will act as a set difference operation returning only those unique VertexId's present in this.
minus(Object) - Method in class org.apache.spark.sql.Column
Subtraction.
minus(Duration) - Method in class org.apache.spark.streaming.Duration
 
minus(Time) - Method in class org.apache.spark.streaming.Time
 
minus(Duration) - Method in class org.apache.spark.streaming.Time
 
minute(Column) - Static method in class org.apache.spark.sql.functions
Extracts the minutes as an integer from a given date/timestamp/string.
minutes() - Static method in class org.apache.spark.scheduler.StatsReportListener
 
minutes(long) - Static method in class org.apache.spark.streaming.Durations
 
Minutes - Class in org.apache.spark.streaming
Helper object that creates instance of Duration representing a given number of minutes.
Minutes() - Constructor for class org.apache.spark.streaming.Minutes
 
minVal() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
missingInput() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
missingInput() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
missingInput() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
missingInput() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
missingValue() - Static method in class org.apache.spark.ml.feature.Imputer
 
missingValue() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
mkList() - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
mkString() - Method in interface org.apache.spark.sql.Row
Displays all elements of this sequence in a string (without a separator).
mkString(String) - Method in interface org.apache.spark.sql.Row
Displays all elements of this sequence in a string using a separator string.
mkString(String, String, String) - Method in interface org.apache.spark.sql.Row
Displays all elements of this traversable or iterator in a string using start, end, and separator strings.
mkString(String, String, String) - Static method in class org.apache.spark.sql.types.StructType
 
mkString(String) - Static method in class org.apache.spark.sql.types.StructType
 
mkString() - Static method in class org.apache.spark.sql.types.StructType
 
ML_ATTR() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
mlDenseMatrixToMLlibDenseMatrix(DenseMatrix) - Static method in class org.apache.spark.mllib.linalg.MatrixImplicits
 
mlDenseVectorToMLlibDenseVector(DenseVector) - Static method in class org.apache.spark.mllib.linalg.VectorImplicits
 
mllibDenseMatrixToMLDenseMatrix(DenseMatrix) - Static method in class org.apache.spark.mllib.linalg.MatrixImplicits
 
mllibDenseVectorToMLDenseVector(DenseVector) - Static method in class org.apache.spark.mllib.linalg.VectorImplicits
 
mllibMatrixToMLMatrix(Matrix) - Static method in class org.apache.spark.mllib.linalg.MatrixImplicits
 
mllibSparseMatrixToMLSparseMatrix(SparseMatrix) - Static method in class org.apache.spark.mllib.linalg.MatrixImplicits
 
mllibSparseVectorToMLSparseVector(SparseVector) - Static method in class org.apache.spark.mllib.linalg.VectorImplicits
 
mllibVectorToMLVector(Vector) - Static method in class org.apache.spark.mllib.linalg.VectorImplicits
 
mlMatrixToMLlibMatrix(Matrix) - Static method in class org.apache.spark.mllib.linalg.MatrixImplicits
 
MLPairRDDFunctions<K,V> - Class in org.apache.spark.mllib.rdd
Developer API Machine learning specific Pair RDD functions.
MLPairRDDFunctions(RDD<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>) - Constructor for class org.apache.spark.mllib.rdd.MLPairRDDFunctions
 
MLReadable<T> - Interface in org.apache.spark.ml.util
Trait for objects that provide MLReader.
MLReader<T> - Class in org.apache.spark.ml.util
Abstract class for utility classes that can load ML instances.
MLReader() - Constructor for class org.apache.spark.ml.util.MLReader
 
mlSparseMatrixToMLlibSparseMatrix(SparseMatrix) - Static method in class org.apache.spark.mllib.linalg.MatrixImplicits
 
mlSparseVectorToMLlibSparseVector(SparseVector) - Static method in class org.apache.spark.mllib.linalg.VectorImplicits
 
MLUtils - Class in org.apache.spark.mllib.util
Helper methods to load, save and pre-process data used in MLLib.
MLUtils() - Constructor for class org.apache.spark.mllib.util.MLUtils
 
mlVectorToMLlibVector(Vector) - Static method in class org.apache.spark.mllib.linalg.VectorImplicits
 
MLWritable - Interface in org.apache.spark.ml.util
Trait for classes that provide MLWriter.
MLWriter - Class in org.apache.spark.ml.util
Abstract class for utility classes that can save ML instances.
MLWriter() - Constructor for class org.apache.spark.ml.util.MLWriter
 
mod(Object) - Method in class org.apache.spark.sql.Column
Modulo (a.k.a.
mode(SaveMode) - Method in class org.apache.spark.sql.DataFrameWriter
Specifies the behavior when data or table already exists.
mode(String) - Method in class org.apache.spark.sql.DataFrameWriter
Specifies the behavior when data or table already exists.
mode() - Method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
Model<M extends Model<M>> - Class in org.apache.spark.ml
Developer API A fitted model, i.e., a Transformer produced by an Estimator.
Model() - Constructor for class org.apache.spark.ml.Model
 
models() - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
modelType() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
modelType() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
modelType() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
modelType() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$.Data
 
MODULE$ - Static variable in class org.apache.spark.AccumulatorParam.DoubleAccumulatorParam$
Deprecated.
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.AccumulatorParam.FloatAccumulatorParam$
Deprecated.
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.AccumulatorParam.IntAccumulatorParam$
Deprecated.
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.AccumulatorParam.LongAccumulatorParam$
Deprecated.
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.AccumulatorParam.StringAccumulatorParam$
Deprecated.
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.graphx.PartitionStrategy.CanonicalRandomVertexCut$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.graphx.PartitionStrategy.EdgePartition1D$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.graphx.PartitionStrategy.EdgePartition2D$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.graphx.PartitionStrategy.RandomVertexCut$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.internal.io.FileCommitProtocol.EmptyTaskCommitMessage$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.InternalAccumulator.input$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.InternalAccumulator.output$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.InternalAccumulator.shuffleRead$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.InternalAccumulator.shuffleWrite$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette.ClusterStats$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.feature.Word2VecModel.Word2VecModelWriter$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.Pipeline.SharedReadWrite$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.recommendation.ALS.InBlock$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.recommendation.ALS.Rating$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.recommendation.ALS.RatingBlock$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.CLogLog$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Family$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.FamilyAndLink$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Identity$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Inverse$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Link$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Log$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Logit$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Probit$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Sqrt$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Tweedie$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ml.tree.EnsembleModelReadWrite.EnsembleNodeData$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.clustering.BisectingKMeansModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.clustering.KMeansModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.clustering.PowerIterationClustering.Assignment$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.clustering.PowerIterationClusteringModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.feature.ChiSqSelectorModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.fpm.FPGrowthModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.fpm.PrefixSpan.Postfix$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.fpm.PrefixSpan.Prefix$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.fpm.PrefixSpanModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.stat.test.ChiSqTest.Method$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.stat.test.ChiSqTest.NullHypothesis$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest.NullHypothesis$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.rdd.HadoopRDD.HadoopMapPartitionsWithSplitRDD$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.rdd.NewHadoopRDD.NewHadoopMapPartitionsWithSplitRDD$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.GetExecutorLossReason$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutors$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillExecutorsOnHost$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.LaunchTask$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterClusterManager$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisteredExecutor$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutorFailed$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveWorker$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RetrieveLastAllocatedExecutorId$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RetrieveSparkAppConfig$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.ReviveOffers$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SetupDriver$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.Shutdown$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SparkAppConfig$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopDriver$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopExecutor$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopExecutors$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.UpdateDelegationTokens$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.hive.HiveShim.HiveFunctionWrapper$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.RelationalGroupedDataset.CubeType$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.RelationalGroupedDataset.GroupByType$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.RelationalGroupedDataset.PivotType$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.RelationalGroupedDataset.RollupType$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.types.Decimal.DecimalAsIfIntegral$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.types.Decimal.DecimalIsFractional$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.types.DecimalType.Expression$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.sql.types.DecimalType.Fixed$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.BlockLocationsAndStatus$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.BlockManagerHeartbeat$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetBlockStatus$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetExecutorEndpointRef$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetLocations$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetLocationsAndStatus$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetLocationsMultipleBlockIds$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetMatchingBlockIds$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetMemoryStatus$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetPeers$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.GetStorageStatus$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.HasCachedBlocks$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.RemoveBlock$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.RemoveExecutor$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.RemoveRdd$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.RemoveShuffle$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.ReplicateBlock$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.StopBlockManagerMaster$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.TriggerThreadDump$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo$
Static reference to the singleton instance of this Scala object.
MODULE$ - Static variable in class org.apache.spark.ui.JettyUtils.ServletParams$
Static reference to the singleton instance of this Scala object.
monotonically_increasing_id() - Static method in class org.apache.spark.sql.functions
A column expression that generates monotonically increasing 64-bit integers.
monotonicallyIncreasingId() - Static method in class org.apache.spark.sql.functions
Deprecated.
Use monotonically_increasing_id(). Since 2.0.0.
month(Column) - Static method in class org.apache.spark.sql.functions
Extracts the month as an integer from a given date/timestamp/string.
months_between(Column, Column) - Static method in class org.apache.spark.sql.functions
Returns number of months between dates date1 and date2.
msDurationToString(long) - Static method in class org.apache.spark.util.Utils
Returns a human-readable string representing a duration such as "35ms"
MsSqlServerDialect - Class in org.apache.spark.sql.jdbc
 
MsSqlServerDialect() - Constructor for class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
mu() - Method in class org.apache.spark.mllib.stat.distribution.MultivariateGaussian
 
MulticlassClassificationEvaluator - Class in org.apache.spark.ml.evaluation
Experimental Evaluator for multiclass classification, which expects two input columns: prediction and label.
MulticlassClassificationEvaluator(String) - Constructor for class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
MulticlassClassificationEvaluator() - Constructor for class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
multiclassMetrics() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
 
MulticlassMetrics - Class in org.apache.spark.mllib.evaluation
Evaluator for multiclass classification.
MulticlassMetrics(RDD<Tuple2<Object, Object>>) - Constructor for class org.apache.spark.mllib.evaluation.MulticlassMetrics
 
MultilabelMetrics - Class in org.apache.spark.mllib.evaluation
Evaluator for multilabel classification.
MultilabelMetrics(RDD<Tuple2<double[], double[]>>) - Constructor for class org.apache.spark.mllib.evaluation.MultilabelMetrics
 
multiLabelValidator(int) - Static method in class org.apache.spark.mllib.util.DataValidators
Function to check if labels used for k class multi-label classification are in the range of {0, 1, ..., k - 1}.
MultilayerPerceptronClassificationModel - Class in org.apache.spark.ml.classification
Classification model based on the Multilayer Perceptron.
MultilayerPerceptronClassifier - Class in org.apache.spark.ml.classification
Classifier trainer based on the Multilayer Perceptron.
MultilayerPerceptronClassifier(String) - Constructor for class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
MultilayerPerceptronClassifier() - Constructor for class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
multiply(DenseMatrix) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
multiply(DenseVector) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
multiply(Vector) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
multiply(DenseMatrix) - Method in interface org.apache.spark.ml.linalg.Matrix
Convenience method for Matrix-DenseMatrix multiplication.
multiply(DenseVector) - Method in interface org.apache.spark.ml.linalg.Matrix
Convenience method for Matrix-DenseVector multiplication.
multiply(Vector) - Method in interface org.apache.spark.ml.linalg.Matrix
Convenience method for Matrix-Vector multiplication.
multiply(DenseMatrix) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
multiply(DenseVector) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
multiply(Vector) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
multiply(DenseMatrix) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
 
multiply(DenseVector) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
 
multiply(Vector) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
 
multiply(BlockMatrix) - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Left multiplies this BlockMatrix to other, another BlockMatrix.
multiply(BlockMatrix, int) - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Left multiplies this BlockMatrix to other, another BlockMatrix.
multiply(Matrix) - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Multiply this matrix by a local matrix on the right.
multiply(Matrix) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Multiply this matrix by a local matrix on the right.
multiply(DenseMatrix) - Method in interface org.apache.spark.mllib.linalg.Matrix
Convenience method for Matrix-DenseMatrix multiplication.
multiply(DenseVector) - Method in interface org.apache.spark.mllib.linalg.Matrix
Convenience method for Matrix-DenseVector multiplication.
multiply(Vector) - Method in interface org.apache.spark.mllib.linalg.Matrix
Convenience method for Matrix-Vector multiplication.
multiply(DenseMatrix) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
 
multiply(DenseVector) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
 
multiply(Vector) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
 
multiply(Object) - Method in class org.apache.spark.sql.Column
Multiplication of this expression and another expression.
MultivariateGaussian - Class in org.apache.spark.ml.stat.distribution
This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution.
MultivariateGaussian(Vector, Matrix) - Constructor for class org.apache.spark.ml.stat.distribution.MultivariateGaussian
 
MultivariateGaussian - Class in org.apache.spark.mllib.stat.distribution
Developer API This class provides basic functionality for a Multivariate Gaussian (Normal) Distribution.
MultivariateGaussian(Vector, Matrix) - Constructor for class org.apache.spark.mllib.stat.distribution.MultivariateGaussian
 
MultivariateOnlineSummarizer - Class in org.apache.spark.mllib.stat
Developer API MultivariateOnlineSummarizer implements MultivariateStatisticalSummary to compute the mean, variance, minimum, maximum, counts, and nonzero counts for instances in sparse or dense vector format in an online fashion.
MultivariateOnlineSummarizer() - Constructor for class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
 
MultivariateStatisticalSummary - Interface in org.apache.spark.mllib.stat
Trait for multivariate statistical summary of a data matrix.
MutableAggregationBuffer - Class in org.apache.spark.sql.expressions
A Row representing a mutable aggregation buffer.
MutableAggregationBuffer() - Constructor for class org.apache.spark.sql.expressions.MutableAggregationBuffer
 
MutablePair<T1,T2> - Class in org.apache.spark.util
Developer API A tuple of 2 elements.
MutablePair(T1, T2) - Constructor for class org.apache.spark.util.MutablePair
 
MutablePair() - Constructor for class org.apache.spark.util.MutablePair
No-arg constructor for serialization
MutableURLClassLoader - Class in org.apache.spark.util
URL class loader that exposes the `addURL` method in URLClassLoader.
MutableURLClassLoader(URL[], ClassLoader) - Constructor for class org.apache.spark.util.MutableURLClassLoader
 
myName() - Method in class org.apache.spark.util.InnerClosureFinder
 
MySQLDialect - Class in org.apache.spark.sql.jdbc
 
MySQLDialect() - Constructor for class org.apache.spark.sql.jdbc.MySQLDialect
 

N

n() - Method in class org.apache.spark.ml.feature.NGram
Minimum n-gram length, greater than or equal to 1.
n() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
na() - Method in class org.apache.spark.sql.Dataset
Returns a DataFrameNaFunctions for working with missing data.
NaiveBayes - Class in org.apache.spark.ml.classification
Naive Bayes Classifiers.
NaiveBayes(String) - Constructor for class org.apache.spark.ml.classification.NaiveBayes
 
NaiveBayes() - Constructor for class org.apache.spark.ml.classification.NaiveBayes
 
NaiveBayes - Class in org.apache.spark.mllib.classification
Trains a Naive Bayes model given an RDD of (label, features) pairs.
NaiveBayes(double) - Constructor for class org.apache.spark.mllib.classification.NaiveBayes
 
NaiveBayes() - Constructor for class org.apache.spark.mllib.classification.NaiveBayes
 
NaiveBayesModel - Class in org.apache.spark.ml.classification
Model produced by NaiveBayes param: pi log of class priors, whose dimension is C (number of classes) param: theta log of class conditional probabilities, whose dimension is C (number of classes) by D (number of features)
NaiveBayesModel - Class in org.apache.spark.mllib.classification
Model for Naive Bayes Classifiers.
NaiveBayesModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.classification
 
NaiveBayesModel.SaveLoadV1_0$.Data - Class in org.apache.spark.mllib.classification
Model data for model import/export
NaiveBayesModel.SaveLoadV2_0$ - Class in org.apache.spark.mllib.classification
 
NaiveBayesModel.SaveLoadV2_0$.Data - Class in org.apache.spark.mllib.classification
Model data for model import/export
name() - Method in class org.apache.spark.Accumulable
Deprecated.
 
name() - Static method in class org.apache.spark.Accumulator
Deprecated.
 
name() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
name() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
name() - Static method in class org.apache.spark.api.java.JavaRDD
 
name() - Method in interface org.apache.spark.api.java.JavaRDDLike
 
name() - Static method in class org.apache.spark.api.r.RRDD
 
name() - Static method in class org.apache.spark.graphx.EdgeRDD
 
name() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
name() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
name() - Static method in class org.apache.spark.graphx.VertexRDD
 
name() - Method in class org.apache.spark.ml.attribute.Attribute
Name of the attribute.
name() - Method in class org.apache.spark.ml.attribute.AttributeGroup
 
NAME() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
name() - Method in class org.apache.spark.ml.attribute.AttributeType
 
name() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
name() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
name() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
name() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
name() - Static method in class org.apache.spark.ml.param.DoubleParam
 
name() - Static method in class org.apache.spark.ml.param.FloatParam
 
name() - Method in class org.apache.spark.ml.param.Param
 
name() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
 
name() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
name() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Identity$
 
name() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Inverse$
 
name() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Log$
 
name() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
 
name() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Sqrt$
 
name() - Method in class org.apache.spark.mllib.stat.test.ChiSqTest.Method
 
name() - Static method in class org.apache.spark.rdd.HadoopRDD
 
name() - Static method in class org.apache.spark.rdd.JdbcRDD
 
name() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
name() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
name() - Method in class org.apache.spark.rdd.RDD
A friendly name for this RDD
name() - Static method in class org.apache.spark.rdd.UnionRDD
 
name() - Method in class org.apache.spark.scheduler.AccumulableInfo
 
name() - Method in class org.apache.spark.scheduler.AsyncEventQueue
 
name() - Method in class org.apache.spark.scheduler.StageInfo
 
name() - Method in interface org.apache.spark.SparkStageInfo
 
name() - Method in class org.apache.spark.SparkStageInfoImpl
 
name() - Method in class org.apache.spark.sql.catalog.Column
 
name() - Method in class org.apache.spark.sql.catalog.Database
 
name() - Method in class org.apache.spark.sql.catalog.Function
 
name() - Method in class org.apache.spark.sql.catalog.Table
 
name(String) - Method in class org.apache.spark.sql.Column
Gives the column a name (alias).
name() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns the user-specified name of the query, or null if not specified.
name() - Method in class org.apache.spark.sql.streaming.StreamingQueryListener.QueryStartedEvent
 
name() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
name(String) - Method in class org.apache.spark.sql.TypedColumn
Gives the TypedColumn a name (alias).
name() - Method in class org.apache.spark.sql.types.StructField
 
name() - Method in class org.apache.spark.status.api.v1.AccumulableInfo
 
name() - Method in class org.apache.spark.status.api.v1.ApplicationInfo
 
name() - Method in class org.apache.spark.status.api.v1.JobData
 
name() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
name() - Method in class org.apache.spark.status.api.v1.StageData
 
name() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
name() - Method in class org.apache.spark.storage.BlockId
A globally unique identifier for this Block.
name() - Method in class org.apache.spark.storage.BroadcastBlockId
 
name() - Method in class org.apache.spark.storage.RDDBlockId
 
name() - Method in class org.apache.spark.storage.RDDInfo
 
name() - Method in class org.apache.spark.storage.ShuffleBlockId
 
name() - Method in class org.apache.spark.storage.ShuffleDataBlockId
 
name() - Method in class org.apache.spark.storage.ShuffleIndexBlockId
 
name() - Method in class org.apache.spark.storage.StreamBlockId
 
name() - Method in class org.apache.spark.storage.TaskResultBlockId
 
name() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
name() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
name() - Method in class org.apache.spark.util.AccumulatorV2
Returns the name of this accumulator, can only be called after registration.
name() - Method in class org.apache.spark.util.MethodIdentifier
 
name_$eq(String) - Static method in class org.apache.spark.api.r.RRDD
 
name_$eq(String) - Static method in class org.apache.spark.graphx.EdgeRDD
 
name_$eq(String) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
name_$eq(String) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
name_$eq(String) - Static method in class org.apache.spark.graphx.VertexRDD
 
name_$eq(String) - Static method in class org.apache.spark.rdd.HadoopRDD
 
name_$eq(String) - Static method in class org.apache.spark.rdd.JdbcRDD
 
name_$eq(String) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
name_$eq(String) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
name_$eq(String) - Static method in class org.apache.spark.rdd.UnionRDD
 
namedThreadFactory(String) - Static method in class org.apache.spark.util.ThreadUtils
Create a thread factory that names threads with a prefix and also sets the threads to daemon.
names() - Method in class org.apache.spark.ml.feature.VectorSlicer
An array of feature names to select features from a vector column.
nameToObjectMap() - Static method in class org.apache.spark.mllib.stat.correlation.CorrelationNames
 
nanSafeCompareDoubles(double, double) - Static method in class org.apache.spark.util.Utils
NaN-safe version of java.lang.Double.compare() which allows NaN values to be compared according to semantics where NaN == NaN and NaN is greater than any non-NaN double.
nanSafeCompareFloats(float, float) - Static method in class org.apache.spark.util.Utils
NaN-safe version of java.lang.Float.compare() which allows NaN values to be compared according to semantics where NaN == NaN and NaN is greater than any non-NaN float.
nanvl(Column, Column) - Static method in class org.apache.spark.sql.functions
Returns col1 if it is not NaN, or col2 if col1 is NaN.
NarrowDependency<T> - Class in org.apache.spark
Developer API Base class for dependencies where each partition of the child RDD depends on a small number of partitions of the parent RDD.
NarrowDependency(RDD<T>) - Constructor for class org.apache.spark.NarrowDependency
 
ndcgAt(int) - Method in class org.apache.spark.mllib.evaluation.RankingMetrics
Compute the average NDCG value of all the queries, truncated at ranking position k.
needConversion() - Method in class org.apache.spark.sql.sources.BaseRelation
Whether does it need to convert the objects in Row to internal representation, for example: java.lang.String to UTF8String java.lang.Decimal to Decimal
needsReconfiguration() - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.ContinuousReader
The execution engine will call this method in every epoch to determine if new reader factories need to be generated, which may be required if for example the underlying source system has had partitions added or removed.
negate(Column) - Static method in class org.apache.spark.sql.functions
Unary minus, i.e.
newAccumulatorInfos(Iterable<AccumulableInfo>) - Static method in class org.apache.spark.status.LiveEntityHelpers
 
newAPIHadoopFile(String, Class<F>, Class<K>, Class<V>, Configuration) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
newAPIHadoopFile(String, ClassTag<K>, ClassTag<V>, ClassTag<F>) - Method in class org.apache.spark.SparkContext
Smarter version of newApiHadoopFile that uses class tags to figure out the classes of keys, values and the org.apache.hadoop.mapreduce.InputFormat (new MapReduce API) so that user don't need to pass them directly.
newAPIHadoopFile(String, Class<F>, Class<K>, Class<V>, Configuration) - Method in class org.apache.spark.SparkContext
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
newAPIHadoopRDD(Configuration, Class<F>, Class<K>, Class<V>) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
newAPIHadoopRDD(Configuration, Class<F>, Class<K>, Class<V>) - Method in class org.apache.spark.SparkContext
Get an RDD for a given Hadoop file with an arbitrary new API InputFormat and extra configuration options to pass to the input format.
newBooleanArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBooleanEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBooleanSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newBoxedBooleanEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBoxedByteEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBoxedDoubleEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBoxedFloatEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBoxedIntEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBoxedLongEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newBoxedShortEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newByteArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newByteEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newByteSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newDaemonCachedThreadPool(String) - Static method in class org.apache.spark.util.ThreadUtils
Wrapper over newCachedThreadPool.
newDaemonCachedThreadPool(String, int, int) - Static method in class org.apache.spark.util.ThreadUtils
Create a cached thread pool whose max number of threads is maxThreadNumber.
newDaemonFixedThreadPool(int, String) - Static method in class org.apache.spark.util.ThreadUtils
Wrapper over newFixedThreadPool.
newDaemonSingleThreadExecutor(String) - Static method in class org.apache.spark.util.ThreadUtils
Wrapper over newSingleThreadExecutor.
newDaemonSingleThreadScheduledExecutor(String) - Static method in class org.apache.spark.util.ThreadUtils
Wrapper over ScheduledThreadPoolExecutor.
newDateEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newDoubleArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newDoubleEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newDoubleSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newFloatArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newFloatEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newFloatSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newForkJoinPool(String, int) - Static method in class org.apache.spark.util.ThreadUtils
Construct a new Scala ForkJoinPool with a specified max parallelism and name prefix.
NewHadoopMapPartitionsWithSplitRDD$() - Constructor for class org.apache.spark.rdd.NewHadoopRDD.NewHadoopMapPartitionsWithSplitRDD$
 
NewHadoopRDD<K,V> - Class in org.apache.spark.rdd
Developer API An RDD that provides core functionality for reading data stored in Hadoop (e.g., files in HDFS, sources in HBase, or S3), using the new MapReduce API (org.apache.hadoop.mapreduce).
NewHadoopRDD(SparkContext, Class<? extends InputFormat<K, V>>, Class<K>, Class<V>, Configuration) - Constructor for class org.apache.spark.rdd.NewHadoopRDD
 
NewHadoopRDD.NewHadoopMapPartitionsWithSplitRDD$ - Class in org.apache.spark.rdd
 
newId() - Static method in class org.apache.spark.util.AccumulatorContext
Returns a globally unique ID for a new AccumulatorV2.
newInstance() - Method in class org.apache.spark.serializer.JavaSerializer
 
newInstance() - Method in class org.apache.spark.serializer.KryoSerializer
 
newInstance() - Method in class org.apache.spark.serializer.Serializer
Creates a new SerializerInstance.
newIntArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newIntEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newIntSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newJavaDecimalEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newKryo() - Method in class org.apache.spark.serializer.KryoSerializer
 
newKryoOutput() - Method in class org.apache.spark.serializer.KryoSerializer
 
newLongArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newLongEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newLongSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newMapEncoder(TypeTags.TypeTag<T>) - Method in class org.apache.spark.sql.SQLImplicits
 
newProductArrayEncoder(TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SQLImplicits
 
newProductEncoder(TypeTags.TypeTag<T>) - Method in interface org.apache.spark.sql.LowPrioritySQLImplicits
 
newProductSeqEncoder(TypeTags.TypeTag<A>) - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newScalaDecimalEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newSequenceEncoder(TypeTags.TypeTag<T>) - Method in class org.apache.spark.sql.SQLImplicits
 
newSession() - Method in class org.apache.spark.sql.hive.HiveContext
Deprecated.
Returns a new HiveContext as new session, which will have separated SQLConf, UDF/UDAF, temporary tables and SessionState, but sharing the same CacheManager, IsolatedClientLoader and Hive client (both of execution and metadata) with existing HiveContext.
newSession() - Method in class org.apache.spark.sql.SparkSession
Start a new session with isolated SQL configurations, temporary tables, registered functions are isolated, but sharing the underlying SparkContext and cached data.
newSession() - Method in class org.apache.spark.sql.SQLContext
Returns a SQLContext as new session, with separated SQL configurations, temporary tables, registered functions, but sharing the same SparkContext, cached data and other things.
newSetEncoder(TypeTags.TypeTag<T>) - Method in class org.apache.spark.sql.SQLImplicits
Notice that we serialize Set to Catalyst array.
newShortArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newShortEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newShortSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newStringArrayEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newStringEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
newStringSeqEncoder() - Method in class org.apache.spark.sql.SQLImplicits
Deprecated.
use newSequenceEncoder
newTaskTempFile(TaskAttemptContext, Option<String>, String) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Notifies the commit protocol to add a new file, and gets back the full path that should be used.
newTaskTempFile(TaskAttemptContext, Option<String>, String) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
newTaskTempFileAbsPath(TaskAttemptContext, String, String) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Similar to newTaskTempFile(), but allows files to committed to an absolute output location.
newTaskTempFileAbsPath(TaskAttemptContext, String, String) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
newTemporaryConfiguration(boolean) - Static method in class org.apache.spark.sql.hive.HiveUtils
Constructs a configuration for hive, where the metastore is located in a temp directory.
newTimeStampEncoder() - Method in class org.apache.spark.sql.SQLImplicits
 
next() - Method in class org.apache.spark.InterruptibleIterator
 
next() - Method in interface org.apache.spark.mllib.clustering.LDAOptimizer
 
next() - Method in interface org.apache.spark.sql.sources.v2.reader.DataReader
Proceed to next record, returns false if there is no more records.
next() - Method in class org.apache.spark.status.LiveRDDPartition
 
next_day(Column, String) - Static method in class org.apache.spark.sql.functions
Given a date column, returns the first date which is later than the value of the date column that is on the specified day of the week.
nextValue() - Method in class org.apache.spark.mllib.random.ExponentialGenerator
 
nextValue() - Method in class org.apache.spark.mllib.random.GammaGenerator
 
nextValue() - Method in class org.apache.spark.mllib.random.LogNormalGenerator
 
nextValue() - Method in class org.apache.spark.mllib.random.PoissonGenerator
 
nextValue() - Method in interface org.apache.spark.mllib.random.RandomDataGenerator
Returns an i.i.d.
nextValue() - Method in class org.apache.spark.mllib.random.StandardNormalGenerator
 
nextValue() - Method in class org.apache.spark.mllib.random.UniformGenerator
 
nextValue() - Method in class org.apache.spark.mllib.random.WeibullGenerator
 
NGram - Class in org.apache.spark.ml.feature
A feature transformer that converts the input array of strings into an array of n-grams.
NGram(String) - Constructor for class org.apache.spark.ml.feature.NGram
 
NGram() - Constructor for class org.apache.spark.ml.feature.NGram
 
NioBufferedFileInputStream - Class in org.apache.spark.io
InputStream implementation which uses direct buffer to read a file to avoid extra copy of data between Java and native memory which happens when using BufferedInputStream.
NioBufferedFileInputStream(File, int) - Constructor for class org.apache.spark.io.NioBufferedFileInputStream
 
NioBufferedFileInputStream(File) - Constructor for class org.apache.spark.io.NioBufferedFileInputStream
 
nioByteBuffer() - Method in class org.apache.spark.storage.EncryptedManagedBuffer
 
NNLS - Class in org.apache.spark.mllib.optimization
Object used to solve nonnegative least squares problems using a modified projected gradient method.
NNLS() - Constructor for class org.apache.spark.mllib.optimization.NNLS
 
NNLS.Workspace - Class in org.apache.spark.mllib.optimization
 
NO_PREF() - Static method in class org.apache.spark.scheduler.TaskLocality
 
NO_RESOURCE - Static variable in class org.apache.spark.launcher.SparkLauncher
A special value for the resource that tells Spark to not try to process the app resource as a file.
Node - Class in org.apache.spark.ml.tree
Decision tree node interface.
Node() - Constructor for class org.apache.spark.ml.tree.Node
 
Node - Class in org.apache.spark.mllib.tree.model
Developer API Node in a decision tree.
Node(int, Predict, double, boolean, Option<Split>, Option<Node>, Option<Node>, Option<InformationGainStats>) - Constructor for class org.apache.spark.mllib.tree.model.Node
 
node() - Method in class org.apache.spark.scheduler.BlacklistedExecutor
 
NODE_LOCAL() - Static method in class org.apache.spark.scheduler.TaskLocality
 
nodeBlacklist() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors
 
NodeData(int, double, double, double[], double, int, int, DecisionTreeModelReadWrite.SplitData) - Constructor for class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
nodeData() - Method in class org.apache.spark.ml.tree.EnsembleModelReadWrite.EnsembleNodeData
 
NodeData(int, int, org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0.PredictData, double, boolean, Option<org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0.SplitData>, Option<Object>, Option<Object>, Option<Object>) - Constructor for class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
NodeData$() - Constructor for class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData$
 
nodeId() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
nodeName() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
nodeName() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
nodeName() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
nodeName() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
noLocality() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
Nominal() - Static method in class org.apache.spark.ml.attribute.AttributeType
Nominal type.
NominalAttribute - Class in org.apache.spark.ml.attribute
Developer API A nominal attribute.
NONE - Static variable in class org.apache.spark.api.java.StorageLevels
 
None - Static variable in class org.apache.spark.graphx.TripletFields
None of the triplet fields are exposed.
NONE() - Static method in class org.apache.spark.scheduler.SchedulingMode
 
NONE() - Static method in class org.apache.spark.storage.StorageLevel
 
nonEmpty() - Static method in class org.apache.spark.sql.types.StructType
 
nonLocalPaths(String, boolean) - Static method in class org.apache.spark.util.Utils
Return all non-local paths from a comma-separated list of paths.
nonnegative() - Static method in class org.apache.spark.ml.recommendation.ALS
 
nonNegativeHash(Object) - Static method in class org.apache.spark.util.Utils
 
nonNegativeMod(int, int) - Static method in class org.apache.spark.util.Utils
 
NoopDialect - Class in org.apache.spark.sql.jdbc
NOOP dialect object, always returning the neutral element.
NoopDialect() - Constructor for class org.apache.spark.sql.jdbc.NoopDialect
 
norm(Vector, double) - Static method in class org.apache.spark.ml.linalg.Vectors
Returns the p-norm of this vector.
norm(Vector, double) - Static method in class org.apache.spark.mllib.linalg.Vectors
Returns the p-norm of this vector.
normalizeDuration(long) - Static method in class org.apache.spark.streaming.ui.UIUtils
Find the best TimeUnit for converting milliseconds to a friendly string.
Normalizer - Class in org.apache.spark.ml.feature
Normalize a vector to have unit norm using the given p-norm.
Normalizer(String) - Constructor for class org.apache.spark.ml.feature.Normalizer
 
Normalizer() - Constructor for class org.apache.spark.ml.feature.Normalizer
 
Normalizer - Class in org.apache.spark.mllib.feature
Normalizes samples individually to unit L^p^ norm
Normalizer(double) - Constructor for class org.apache.spark.mllib.feature.Normalizer
 
Normalizer() - Constructor for class org.apache.spark.mllib.feature.Normalizer
 
normalizeToProbabilitiesInPlace(DenseVector) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
Normalize a vector of raw predictions to be a multinomial probability vector, in place.
normalJavaRDD(JavaSparkContext, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.normalRDD.
normalJavaRDD(JavaSparkContext, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.normalJavaRDD with the default seed.
normalJavaRDD(JavaSparkContext, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.normalJavaRDD with the default number of partitions and the default seed.
normalJavaVectorRDD(JavaSparkContext, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.normalVectorRDD.
normalJavaVectorRDD(JavaSparkContext, long, int, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.normalJavaVectorRDD with the default seed.
normalJavaVectorRDD(JavaSparkContext, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.normalJavaVectorRDD with the default number of partitions and the default seed.
normalRDD(SparkContext, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD comprised of i.i.d. samples from the standard normal distribution.
normalVectorRDD(SparkContext, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD[Vector] with vectors containing i.i.d. samples drawn from the standard normal distribution.
normL1(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
normL1(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
normL1() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
L1 norm of each dimension.
normL1() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
L1 norm of each column
normL2(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
normL2(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
normL2() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
L2 (Euclidian) norm of each dimension.
normL2() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Euclidean magnitude of each column
normPdf(double, double, double, double) - Static method in class org.apache.spark.mllib.stat.KernelDensity
Evaluates the PDF of a normal distribution.
not(Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
not(Column) - Static method in class org.apache.spark.sql.functions
Inversion of boolean expression, i.e.
Not - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff child is evaluated to false.
Not(Filter) - Constructor for class org.apache.spark.sql.sources.Not
 
notEqual(Object) - Method in class org.apache.spark.sql.Column
Inequality test.
NoTimeout() - Static method in class org.apache.spark.sql.streaming.GroupStateTimeout
No timeout.
ntile(int) - Static method in class org.apache.spark.sql.functions
Window function: returns the ntile group id (from 1 to n inclusive) in an ordered window partition.
nullable() - Method in class org.apache.spark.sql.catalog.Column
 
nullable() - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
Returns true when the UDF can return a nullable value.
nullable() - Method in class org.apache.spark.sql.types.StructField
 
nullDeviance() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
The deviance for the null model.
nullHypothesis() - Method in class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
nullHypothesis() - Method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult
 
nullHypothesis() - Static method in class org.apache.spark.mllib.stat.test.StudentTTest
 
nullHypothesis() - Method in interface org.apache.spark.mllib.stat.test.TestResult
Null hypothesis of the test.
nullHypothesis() - Static method in class org.apache.spark.mllib.stat.test.WelchTTest
 
NullHypothesis$() - Constructor for class org.apache.spark.mllib.stat.test.ChiSqTest.NullHypothesis$
 
NullHypothesis$() - Constructor for class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest.NullHypothesis$
 
NullType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the NullType object.
NullType - Class in org.apache.spark.sql.types
The data type representing NULL values.
NUM_ATTRIBUTES() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
NUM_PARTITIONS() - Static method in class org.apache.spark.ui.UIWorkloadGenerator
 
NUM_VALUES() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
numAccums() - Static method in class org.apache.spark.util.AccumulatorContext
Returns the number of accumulators registered.
numActiveBatches() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numActiveOutputOps() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
numActiveReceivers() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numActives() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
numActives() - Method in class org.apache.spark.ml.linalg.DenseVector
 
numActives() - Method in interface org.apache.spark.ml.linalg.Matrix
Find the number of values stored explicitly.
numActives() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
numActives() - Method in class org.apache.spark.ml.linalg.SparseVector
 
numActives() - Method in interface org.apache.spark.ml.linalg.Vector
Number of active entries.
numActives() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
numActives() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
numActives() - Method in interface org.apache.spark.mllib.linalg.Matrix
Find the number of values stored explicitly.
numActives() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
numActives() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
numActives() - Method in interface org.apache.spark.mllib.linalg.Vector
Number of active entries.
numActiveStages() - Method in class org.apache.spark.status.api.v1.JobData
 
numActiveTasks() - Method in interface org.apache.spark.SparkStageInfo
 
numActiveTasks() - Method in class org.apache.spark.SparkStageInfoImpl
 
numActiveTasks() - Method in class org.apache.spark.status.api.v1.JobData
 
numActiveTasks() - Method in class org.apache.spark.status.api.v1.StageData
 
numAttributes() - Method in class org.apache.spark.ml.attribute.AttributeGroup
 
numAvailableOutputs() - Method in class org.apache.spark.ShuffleStatus
Number of partitions that have shuffle outputs.
numberedTreeString() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
numberedTreeString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
numberedTreeString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
numberedTreeString() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
numBins() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
 
numBlocks() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the number of blocks stored in this block manager in O(RDDs) time.
numBuckets() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
numBucketsArray() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
numCachedPartitions() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
numCachedPartitions() - Method in class org.apache.spark.storage.RDDInfo
 
numCategories() - Method in class org.apache.spark.ml.tree.CategoricalSplit
 
numCategories() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData
 
numClasses() - Method in class org.apache.spark.ml.classification.ClassificationModel
Number of classes (values which the label can take).
numClasses() - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
numClasses() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
numClasses() - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
numClasses() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
numClasses() - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
numClasses() - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
numClasses() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
numClasses() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
numClasses() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
numClasses() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
numColBlocks() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
numCols() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
numCols() - Method in interface org.apache.spark.ml.linalg.Matrix
Number of columns.
numCols() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
numCols() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
numCols() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
numCols() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Gets or computes the number of columns.
numCols() - Method in interface org.apache.spark.mllib.linalg.distributed.DistributedMatrix
Gets or computes the number of columns.
numCols() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
 
numCols() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Gets or computes the number of columns.
numCols() - Method in interface org.apache.spark.mllib.linalg.Matrix
Number of columns.
numCols() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
numCols() - Method in class org.apache.spark.sql.vectorized.ColumnarBatch
Returns the number of columns that make up this batch.
numCompletedIndices() - Method in class org.apache.spark.status.api.v1.JobData
 
numCompletedIndices() - Method in class org.apache.spark.status.api.v1.StageData
 
numCompletedOutputOps() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
numCompletedStages() - Method in class org.apache.spark.status.api.v1.JobData
 
numCompletedTasks() - Method in interface org.apache.spark.SparkStageInfo
 
numCompletedTasks() - Method in class org.apache.spark.SparkStageInfoImpl
 
numCompletedTasks() - Method in class org.apache.spark.status.api.v1.JobData
 
numCompleteTasks() - Method in class org.apache.spark.status.api.v1.StageData
 
numEdges() - Method in class org.apache.spark.graphx.GraphOps
The number of edges in the graph.
numElements() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
numElements() - Method in class org.apache.spark.sql.vectorized.ColumnarMap
 
Numeric() - Static method in class org.apache.spark.ml.attribute.AttributeType
Numeric type.
NumericAttribute - Class in org.apache.spark.ml.attribute
Developer API A numeric attribute with optional summary statistics.
NumericParser - Class in org.apache.spark.mllib.util
Simple parser for a numeric structure consisting of three types:
NumericParser() - Constructor for class org.apache.spark.mllib.util.NumericParser
 
numericRDDToDoubleRDDFunctions(RDD<T>, Numeric<T>) - Static method in class org.apache.spark.rdd.RDD
 
NumericType - Class in org.apache.spark.sql.types
Numeric data types.
NumericType() - Constructor for class org.apache.spark.sql.types.NumericType
 
numFailedOutputOps() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
numFailedStages() - Method in class org.apache.spark.status.api.v1.JobData
 
numFailedTasks() - Method in interface org.apache.spark.SparkStageInfo
 
numFailedTasks() - Method in class org.apache.spark.SparkStageInfoImpl
 
numFailedTasks() - Method in class org.apache.spark.status.api.v1.JobData
 
numFailedTasks() - Method in class org.apache.spark.status.api.v1.StageData
 
numFeatures() - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
numFeatures() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
numFeatures() - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
numFeatures() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
numFeatures() - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
numFeatures() - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
numFeatures() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
numFeatures() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
numFeatures() - Method in class org.apache.spark.ml.feature.FeatureHasher
Number of features.
numFeatures() - Method in class org.apache.spark.ml.feature.HashingTF
Number of features.
numFeatures() - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
numFeatures() - Method in class org.apache.spark.ml.PredictionModel
Returns the number of features the model was trained on.
numFeatures() - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
numFeatures() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
 
numFeatures() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
numFeatures() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
 
numFeatures() - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
numFeatures() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
numFeatures() - Method in class org.apache.spark.mllib.feature.HashingTF
 
numFields() - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
numFolds() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
numFolds() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
numHashTables() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
numHashTables() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
numHashTables() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
numHashTables() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
numInactiveReceivers() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numInputRows() - Method in class org.apache.spark.sql.streaming.SourceProgress
 
numInputRows() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
The aggregate (across all sources) number of records processed in a trigger.
numInstances() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
Number of instances in DataFrame predictions.
numInstances() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Number of instances in DataFrame predictions
numItemBlocks() - Static method in class org.apache.spark.ml.recommendation.ALS
 
numIterations() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionTrainingSummary
 
numIterations() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
numKilledTasks() - Method in class org.apache.spark.status.api.v1.JobData
 
numKilledTasks() - Method in class org.apache.spark.status.api.v1.StageData
 
numNodes() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
numNodes() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
numNodes() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Get number of nodes in tree, including leaf nodes.
numNonzeros() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
numNonzeros() - Method in class org.apache.spark.ml.linalg.DenseVector
 
numNonzeros() - Method in interface org.apache.spark.ml.linalg.Matrix
Find the number of non-zero active values.
numNonzeros() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
numNonzeros() - Method in class org.apache.spark.ml.linalg.SparseVector
 
numNonzeros() - Method in interface org.apache.spark.ml.linalg.Vector
Number of nonzero elements.
numNonZeros(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
numNonZeros(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
numNonzeros() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
numNonzeros() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
numNonzeros() - Method in interface org.apache.spark.mllib.linalg.Matrix
Find the number of non-zero active values.
numNonzeros() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
numNonzeros() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
numNonzeros() - Method in interface org.apache.spark.mllib.linalg.Vector
Number of nonzero elements.
numNonzeros() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Number of nonzero elements in each dimension.
numNonzeros() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Number of nonzero elements (including explicitly presented zero values) in each column.
numNulls() - Method in class org.apache.spark.sql.vectorized.ArrowColumnVector
 
numNulls() - Method in class org.apache.spark.sql.vectorized.ColumnVector
Returns the number of nulls in this column vector.
numOfPoints() - Method in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette.ClusterStats
 
numPartitions() - Method in class org.apache.spark.HashPartitioner
 
numPartitions() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
numPartitions() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
numPartitions() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
numPartitions() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
numPartitions() - Method in class org.apache.spark.Partitioner
 
numPartitions() - Method in class org.apache.spark.RangePartitioner
 
numPartitions() - Method in class org.apache.spark.rdd.PartitionGroup
 
numPartitions() - Method in interface org.apache.spark.sql.sources.v2.reader.partitioning.Partitioning
Returns the number of partitions(i.e., DataReaderFactorys) the data source outputs.
numPartitions() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
numPartitions() - Method in class org.apache.spark.storage.RDDInfo
 
numPartitions(int) - Method in class org.apache.spark.streaming.StateSpec
Set the number of partitions by which the state RDDs generated by mapWithState will be partitioned.
numProcessedRecords() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numRddBlocks() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the number of RDD blocks stored in this block manager in O(RDDs) time.
numRddBlocksById(int) - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the number of blocks that belong to the given RDD in O(1) time.
numReceivedRecords() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numReceivers() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numRecords() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
The number of recorders received by the receivers in this batch.
numRecords() - Method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
numRetainedCompletedBatches() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numRetries(SparkConf) - Static method in class org.apache.spark.util.RpcUtils
Returns the configured number of times to retry connecting
numRowBlocks() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
numRows() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
numRows() - Method in interface org.apache.spark.ml.linalg.Matrix
Number of rows.
numRows() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
numRows() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
numRows() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
numRows() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Gets or computes the number of rows.
numRows() - Method in interface org.apache.spark.mllib.linalg.distributed.DistributedMatrix
Gets or computes the number of rows.
numRows() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
 
numRows() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Gets or computes the number of rows.
numRows() - Method in interface org.apache.spark.mllib.linalg.Matrix
Number of rows.
numRows() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
numRows() - Method in interface org.apache.spark.sql.sources.v2.reader.Statistics
 
numRows() - Method in class org.apache.spark.sql.vectorized.ColumnarBatch
Returns the number of rows for read, including filtered rows.
numRowsTotal() - Method in class org.apache.spark.sql.streaming.StateOperatorProgress
 
numRowsUpdated() - Method in class org.apache.spark.sql.streaming.StateOperatorProgress
 
numRunningTasks() - Method in interface org.apache.spark.SparkExecutorInfo
 
numRunningTasks() - Method in class org.apache.spark.SparkExecutorInfoImpl
 
numSkippedStages() - Method in class org.apache.spark.status.api.v1.JobData
 
numSkippedTasks() - Method in class org.apache.spark.status.api.v1.JobData
 
numSpilledStages() - Method in class org.apache.spark.SpillListener
 
numStreamBlocks() - Method in class org.apache.spark.ui.storage.ExecutorStreamSummary
 
numTasks() - Method in class org.apache.spark.scheduler.StageInfo
 
numTasks() - Method in interface org.apache.spark.SparkStageInfo
 
numTasks() - Method in class org.apache.spark.SparkStageInfoImpl
 
numTasks() - Method in class org.apache.spark.status.api.v1.JobData
 
numTasks() - Method in class org.apache.spark.status.api.v1.StageData
 
numTopFeatures() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
numTopFeatures() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
numTopFeatures() - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
numTotalCompletedBatches() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
numTotalOutputOps() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
numTrees() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
Number of trees in ensemble
numTrees() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
numTrees() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
numTrees() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
Number of trees in ensemble
numTrees() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
numTrees() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
numTrees() - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
numTrees() - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
numUserBlocks() - Static method in class org.apache.spark.ml.recommendation.ALS
 
numValues() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
numVertices() - Method in class org.apache.spark.graphx.GraphOps
The number of vertices in the graph.

O

obj() - Method in class org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage
 
objectFile(String, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
objectFile(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
objectFile(String, int, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Load an RDD saved as a SequenceFile containing serialized objects, with NullWritable keys and BytesWritable values that contain a serialized partition.
objectiveHistory() - Method in class org.apache.spark.ml.classification.BinaryLogisticRegressionTrainingSummaryImpl
 
objectiveHistory() - Method in interface org.apache.spark.ml.classification.LogisticRegressionTrainingSummary
objective function (scaled loss + regularization) at each iteration.
objectiveHistory() - Method in class org.apache.spark.ml.classification.LogisticRegressionTrainingSummaryImpl
 
objectiveHistory() - Method in class org.apache.spark.ml.regression.LinearRegressionTrainingSummary
 
ObjectStreamClassMethods(ObjectStreamClass) - Constructor for class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods
 
ObjectStreamClassMethods$() - Constructor for class org.apache.spark.serializer.SerializationDebugger.ObjectStreamClassMethods$
 
ObjectType - Class in org.apache.spark.sql.types
 
ObjectType(Class<?>) - Constructor for class org.apache.spark.sql.types.ObjectType
 
ocvTypes() - Static method in class org.apache.spark.ml.image.ImageSchema
(Scala-specific) OpenCV type mapping supported
of(T) - Static method in class org.apache.spark.api.java.Optional
 
of(RDD<Tuple2<Object, Object>>) - Static method in class org.apache.spark.mllib.evaluation.AreaUnderCurve
Returns the area under the given curve.
of(Iterable<Tuple2<Object, Object>>) - Static method in class org.apache.spark.mllib.evaluation.AreaUnderCurve
Returns the area under the given curve.
of(JavaRDD<Tuple2<T, T>>) - Static method in class org.apache.spark.mllib.evaluation.RankingMetrics
Creates a RankingMetrics instance (for Java users).
OFF_HEAP - Static variable in class org.apache.spark.api.java.StorageLevels
 
OFF_HEAP() - Static method in class org.apache.spark.storage.StorageLevel
 
offHeapCacheSize() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the memory used by off-heap caching RDDs
offHeapMemoryRemaining() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
offHeapMemoryUsed() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
offHeapMemRemaining() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the off-heap memory remaining in this block manager.
offHeapMemUsed() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the off-heap memory used by this block manager.
offHeapUsed() - Method in class org.apache.spark.status.LiveRDDDistribution
 
Offset - Class in org.apache.spark.sql.sources.v2.reader.streaming
An abstract representation of progress through a MicroBatchReader or ContinuousReader.
Offset() - Constructor for class org.apache.spark.sql.sources.v2.reader.streaming.Offset
 
offsetBytes(String, long, long, long) - Static method in class org.apache.spark.util.Utils
Return a string containing part of a file from byte 'start' to 'end'.
offsetBytes(Seq<File>, Seq<Object>, long, long) - Static method in class org.apache.spark.util.Utils
Return a string containing data across a set of files.
offsetCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
offsetCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
ofNullable(T) - Static method in class org.apache.spark.api.java.Optional
 
ofRows(SparkSession, LogicalPlan) - Static method in class org.apache.spark.sql.Dataset
 
onApplicationEnd(SparkListenerApplicationEnd) - Method in class org.apache.spark.scheduler.SparkListener
 
onApplicationEnd(SparkListenerApplicationEnd) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onApplicationEnd(SparkListenerApplicationEnd) - Method in class org.apache.spark.SparkFirehoseListener
 
onApplicationStart(SparkListenerApplicationStart) - Method in class org.apache.spark.scheduler.SparkListener
 
onApplicationStart(SparkListenerApplicationStart) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onApplicationStart(SparkListenerApplicationStart) - Method in class org.apache.spark.SparkFirehoseListener
 
onBatchCompleted(StreamingListenerBatchCompleted) - Method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onBatchCompleted(StreamingListenerBatchCompleted) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when processing of a batch of jobs has completed.
onBatchStarted(StreamingListenerBatchStarted) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onBatchStarted(StreamingListenerBatchStarted) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when processing of a batch of jobs has started.
onBatchSubmitted(StreamingListenerBatchSubmitted) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onBatchSubmitted(StreamingListenerBatchSubmitted) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when a batch of jobs has been submitted for processing.
onBlockManagerAdded(SparkListenerBlockManagerAdded) - Method in class org.apache.spark.scheduler.SparkListener
 
onBlockManagerAdded(SparkListenerBlockManagerAdded) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onBlockManagerAdded(SparkListenerBlockManagerAdded) - Method in class org.apache.spark.SparkFirehoseListener
 
onBlockManagerRemoved(SparkListenerBlockManagerRemoved) - Method in class org.apache.spark.scheduler.SparkListener
 
onBlockManagerRemoved(SparkListenerBlockManagerRemoved) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onBlockManagerRemoved(SparkListenerBlockManagerRemoved) - Method in class org.apache.spark.SparkFirehoseListener
 
onBlockUpdated(SparkListenerBlockUpdated) - Method in class org.apache.spark.scheduler.SparkListener
 
onBlockUpdated(SparkListenerBlockUpdated) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onBlockUpdated(SparkListenerBlockUpdated) - Method in class org.apache.spark.SparkFirehoseListener
 
Once() - Static method in class org.apache.spark.sql.streaming.Trigger
A trigger that process only one batch of data in a streaming query then terminates the query.
OnceParser(Function1<Reader<Object>, Parsers.ParseResult<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
onComplete(Function1<Try<T>, U>, ExecutionContext) - Method in class org.apache.spark.ComplexFutureAction
 
onComplete(Function1<Try<T>, U>, ExecutionContext) - Method in interface org.apache.spark.FutureAction
When this action is completed, either through an exception, or a value, applies the provided function.
onComplete(Function1<R, BoxedUnit>) - Method in class org.apache.spark.partial.PartialResult
Set a handler to be called when this PartialResult completes.
onComplete(Function1<Try<T>, U>, ExecutionContext) - Method in class org.apache.spark.SimpleFutureAction
 
OneHotEncoder - Class in org.apache.spark.ml.feature
Deprecated.
OneHotEncoderEstimator will be renamed OneHotEncoder and this OneHotEncoder will be removed in 3.0.0.
OneHotEncoder(String) - Constructor for class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
OneHotEncoder() - Constructor for class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
OneHotEncoderCommon - Class in org.apache.spark.ml.feature
Provides some helper methods used by both OneHotEncoder and OneHotEncoderEstimator.
OneHotEncoderCommon() - Constructor for class org.apache.spark.ml.feature.OneHotEncoderCommon
 
OneHotEncoderEstimator - Class in org.apache.spark.ml.feature
A one-hot encoder that maps a column of category indices to a column of binary vectors, with at most a single one-value per row that indicates the input category index.
OneHotEncoderEstimator(String) - Constructor for class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
OneHotEncoderEstimator() - Constructor for class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
OneHotEncoderModel - Class in org.apache.spark.ml.feature
param: categorySizes Original number of categories for each feature being encoded.
onEnvironmentUpdate(SparkListenerEnvironmentUpdate) - Method in class org.apache.spark.scheduler.SparkListener
 
onEnvironmentUpdate(SparkListenerEnvironmentUpdate) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onEnvironmentUpdate(SparkListenerEnvironmentUpdate) - Method in class org.apache.spark.SparkFirehoseListener
 
ones(int, int) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
Generate a DenseMatrix consisting of ones.
ones(int, int) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a DenseMatrix consisting of ones.
ones(int, int) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
Generate a DenseMatrix consisting of ones.
ones(int, int) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a DenseMatrix consisting of ones.
OneSampleTwoSided() - Method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest.NullHypothesis$
 
OneToOneDependency<T> - Class in org.apache.spark
Developer API Represents a one-to-one dependency between partitions of the parent and child RDDs.
OneToOneDependency(RDD<T>) - Constructor for class org.apache.spark.OneToOneDependency
 
onEvent(SparkListenerEvent) - Method in class org.apache.spark.SparkFirehoseListener
 
OneVsRest - Class in org.apache.spark.ml.classification
Reduction of Multiclass Classification to Binary Classification.
OneVsRest(String) - Constructor for class org.apache.spark.ml.classification.OneVsRest
 
OneVsRest() - Constructor for class org.apache.spark.ml.classification.OneVsRest
 
OneVsRestModel - Class in org.apache.spark.ml.classification
Model produced by OneVsRest.
onExecutorAdded(SparkListenerExecutorAdded) - Method in class org.apache.spark.scheduler.SparkListener
 
onExecutorAdded(SparkListenerExecutorAdded) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onExecutorAdded(SparkListenerExecutorAdded) - Method in class org.apache.spark.SparkFirehoseListener
 
onExecutorBlacklisted(SparkListenerExecutorBlacklisted) - Method in class org.apache.spark.scheduler.SparkListener
 
onExecutorBlacklisted(SparkListenerExecutorBlacklisted) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onExecutorBlacklisted(SparkListenerExecutorBlacklisted) - Method in class org.apache.spark.SparkFirehoseListener
 
onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate) - Method in class org.apache.spark.scheduler.SparkListener
 
onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onExecutorMetricsUpdate(SparkListenerExecutorMetricsUpdate) - Method in class org.apache.spark.SparkFirehoseListener
 
onExecutorRemoved(SparkListenerExecutorRemoved) - Method in class org.apache.spark.scheduler.SparkListener
 
onExecutorRemoved(SparkListenerExecutorRemoved) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onExecutorRemoved(SparkListenerExecutorRemoved) - Method in class org.apache.spark.SparkFirehoseListener
 
onExecutorUnblacklisted(SparkListenerExecutorUnblacklisted) - Method in class org.apache.spark.scheduler.SparkListener
 
onExecutorUnblacklisted(SparkListenerExecutorUnblacklisted) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onExecutorUnblacklisted(SparkListenerExecutorUnblacklisted) - Method in class org.apache.spark.SparkFirehoseListener
 
onFail(Function1<Exception, BoxedUnit>) - Method in class org.apache.spark.partial.PartialResult
Set a handler to be called if this PartialResult's job fails.
onFailure(String, QueryExecution, Exception) - Method in interface org.apache.spark.sql.util.QueryExecutionListener
A callback function that will be called when a query execution failed.
onHeapCacheSize() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the memory used by on-heap caching RDDs
onHeapMemoryRemaining() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
onHeapMemoryUsed() - Method in class org.apache.spark.status.api.v1.RDDDataDistribution
 
onHeapMemRemaining() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the on-heap memory remaining in this block manager.
onHeapMemUsed() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the on-heap memory used by this block manager.
onHeapUsed() - Method in class org.apache.spark.status.LiveRDDDistribution
 
onJobEnd(SparkListenerJobEnd) - Method in class org.apache.spark.scheduler.SparkListener
 
onJobEnd(SparkListenerJobEnd) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onJobEnd(SparkListenerJobEnd) - Method in class org.apache.spark.SparkFirehoseListener
 
onJobEnd(SparkListenerJobEnd) - Method in class org.apache.spark.SpillListener
 
onJobStart(SparkListenerJobStart) - Method in class org.apache.spark.scheduler.SparkListener
 
onJobStart(SparkListenerJobStart) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onJobStart(SparkListenerJobStart) - Method in class org.apache.spark.SparkFirehoseListener
 
OnlineLDAOptimizer - Class in org.apache.spark.mllib.clustering
Developer API
OnlineLDAOptimizer() - Constructor for class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
 
onNodeBlacklisted(SparkListenerNodeBlacklisted) - Method in class org.apache.spark.scheduler.SparkListener
 
onNodeBlacklisted(SparkListenerNodeBlacklisted) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onNodeBlacklisted(SparkListenerNodeBlacklisted) - Method in class org.apache.spark.SparkFirehoseListener
 
onNodeUnblacklisted(SparkListenerNodeUnblacklisted) - Method in class org.apache.spark.scheduler.SparkListener
 
onNodeUnblacklisted(SparkListenerNodeUnblacklisted) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onNodeUnblacklisted(SparkListenerNodeUnblacklisted) - Method in class org.apache.spark.SparkFirehoseListener
 
onOtherEvent(SparkListenerEvent) - Method in class org.apache.spark.scheduler.SparkListener
 
onOtherEvent(SparkListenerEvent) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onOtherEvent(SparkListenerEvent) - Method in class org.apache.spark.SparkFirehoseListener
 
onOutputOperationCompleted(StreamingListenerOutputOperationCompleted) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onOutputOperationCompleted(StreamingListenerOutputOperationCompleted) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when processing of a job of a batch has completed.
onOutputOperationStarted(StreamingListenerOutputOperationStarted) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onOutputOperationStarted(StreamingListenerOutputOperationStarted) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when processing of a job of a batch has started.
onQueryProgress(StreamingQueryListener.QueryProgressEvent) - Method in class org.apache.spark.sql.streaming.StreamingQueryListener
Called when there is some status update (ingestion rate updated, etc.)
onQueryStarted(StreamingQueryListener.QueryStartedEvent) - Method in class org.apache.spark.sql.streaming.StreamingQueryListener
Called when a query is started.
onQueryTerminated(StreamingQueryListener.QueryTerminatedEvent) - Method in class org.apache.spark.sql.streaming.StreamingQueryListener
Called when a query is stopped, with or without error.
onReceiverError(StreamingListenerReceiverError) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onReceiverError(StreamingListenerReceiverError) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when a receiver has reported an error
onReceiverStarted(StreamingListenerReceiverStarted) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onReceiverStarted(StreamingListenerReceiverStarted) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when a receiver has been started
onReceiverStopped(StreamingListenerReceiverStopped) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onReceiverStopped(StreamingListenerReceiverStopped) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when a receiver has been stopped
onSpeculativeTaskSubmitted(SparkListenerSpeculativeTaskSubmitted) - Method in class org.apache.spark.scheduler.SparkListener
 
onSpeculativeTaskSubmitted(SparkListenerSpeculativeTaskSubmitted) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onSpeculativeTaskSubmitted(SparkListenerSpeculativeTaskSubmitted) - Method in class org.apache.spark.SparkFirehoseListener
 
onStageCompleted(SparkListenerStageCompleted) - Method in class org.apache.spark.scheduler.SparkListener
 
onStageCompleted(SparkListenerStageCompleted) - Method in class org.apache.spark.scheduler.StatsReportListener
 
onStageCompleted(SparkListenerStageCompleted) - Method in class org.apache.spark.SparkFirehoseListener
 
onStageCompleted(SparkListenerStageCompleted) - Method in class org.apache.spark.SpillListener
 
onStageSubmitted(SparkListenerStageSubmitted) - Method in class org.apache.spark.scheduler.SparkListener
 
onStageSubmitted(SparkListenerStageSubmitted) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onStageSubmitted(SparkListenerStageSubmitted) - Method in class org.apache.spark.SparkFirehoseListener
 
OnStart - Class in org.apache.spark.rpc.netty
 
OnStart() - Constructor for class org.apache.spark.rpc.netty.OnStart
 
onStart() - Method in class org.apache.spark.streaming.receiver.Receiver
This method is called by the system when the receiver is started.
OnStop - Class in org.apache.spark.rpc.netty
 
OnStop() - Constructor for class org.apache.spark.rpc.netty.OnStop
 
onStop() - Method in class org.apache.spark.streaming.receiver.Receiver
This method is called by the system when the receiver is stopped.
onStreamingStarted(StreamingListenerStreamingStarted) - Static method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
onStreamingStarted(StreamingListenerStreamingStarted) - Method in interface org.apache.spark.streaming.scheduler.StreamingListener
Called when the streaming has been started
onSuccess(String, QueryExecution, long) - Method in interface org.apache.spark.sql.util.QueryExecutionListener
A callback function that will be called when a query executed successfully.
onTaskCommit(FileCommitProtocol.TaskCommitMessage) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Called on the driver after a task commits.
onTaskCommit(FileCommitProtocol.TaskCommitMessage) - Static method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
onTaskCompletion(TaskContext) - Method in interface org.apache.spark.util.TaskCompletionListener
 
onTaskEnd(SparkListenerTaskEnd) - Method in class org.apache.spark.scheduler.SparkListener
 
onTaskEnd(SparkListenerTaskEnd) - Method in class org.apache.spark.scheduler.StatsReportListener
 
onTaskEnd(SparkListenerTaskEnd) - Method in class org.apache.spark.SparkFirehoseListener
 
onTaskEnd(SparkListenerTaskEnd) - Method in class org.apache.spark.SpillListener
 
onTaskFailure(TaskContext, Throwable) - Method in interface org.apache.spark.util.TaskFailureListener
 
onTaskGettingResult(SparkListenerTaskGettingResult) - Method in class org.apache.spark.scheduler.SparkListener
 
onTaskGettingResult(SparkListenerTaskGettingResult) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onTaskGettingResult(SparkListenerTaskGettingResult) - Method in class org.apache.spark.SparkFirehoseListener
 
onTaskStart(SparkListenerTaskStart) - Method in class org.apache.spark.scheduler.SparkListener
 
onTaskStart(SparkListenerTaskStart) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onTaskStart(SparkListenerTaskStart) - Method in class org.apache.spark.SparkFirehoseListener
 
onUnpersistRDD(SparkListenerUnpersistRDD) - Method in class org.apache.spark.scheduler.SparkListener
 
onUnpersistRDD(SparkListenerUnpersistRDD) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
onUnpersistRDD(SparkListenerUnpersistRDD) - Method in class org.apache.spark.SparkFirehoseListener
 
OOM() - Static method in class org.apache.spark.util.SparkExitCode
The default uncaught exception handler was reached, and the uncaught exception was an
open() - Method in class org.apache.spark.input.PortableDataStream
Create a new DataInputStream from the split and context.
open(long, long) - Method in class org.apache.spark.sql.ForeachWriter
Called when starting to process one partition of new data in the executor.
open(File, M, ClassTag<M>) - Static method in class org.apache.spark.status.KVUtils
Open or create a LevelDB store.
ops() - Method in class org.apache.spark.graphx.Graph
The associated GraphOps object.
ops() - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
opt(Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
optimize(RDD<Tuple2<Object, Vector>>, Vector) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Developer API Runs gradient descent on the given training data.
optimize(RDD<Tuple2<Object, Vector>>, Vector) - Method in class org.apache.spark.mllib.optimization.LBFGS
 
optimize(RDD<Tuple2<Object, Vector>>, Vector) - Method in interface org.apache.spark.mllib.optimization.Optimizer
Solve the provided convex optimization problem.
optimizeDocConcentration() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
optimizeDocConcentration() - Static method in class org.apache.spark.ml.clustering.LDA
 
optimizeDocConcentration() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
optimizer() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
optimizer() - Static method in class org.apache.spark.ml.clustering.LDA
 
optimizer() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
optimizer() - Method in class org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
 
optimizer() - Method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
 
optimizer() - Method in class org.apache.spark.mllib.classification.SVMWithSGD
 
Optimizer - Interface in org.apache.spark.mllib.optimization
Developer API Trait for optimization problem solvers.
optimizer() - Method in class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
The optimizer to solve the problem.
optimizer() - Method in class org.apache.spark.mllib.regression.LassoWithSGD
 
optimizer() - Method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
 
optimizer() - Method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
 
option(String, String) - Method in class org.apache.spark.ml.util.MLWriter
Adds an option to the underlying MLWriter.
option(String, String) - Method in class org.apache.spark.sql.DataFrameReader
Adds an input option for the underlying data source.
option(String, boolean) - Method in class org.apache.spark.sql.DataFrameReader
Adds an input option for the underlying data source.
option(String, long) - Method in class org.apache.spark.sql.DataFrameReader
Adds an input option for the underlying data source.
option(String, double) - Method in class org.apache.spark.sql.DataFrameReader
Adds an input option for the underlying data source.
option(String, String) - Method in class org.apache.spark.sql.DataFrameWriter
Adds an output option for the underlying data source.
option(String, boolean) - Method in class org.apache.spark.sql.DataFrameWriter
Adds an output option for the underlying data source.
option(String, long) - Method in class org.apache.spark.sql.DataFrameWriter
Adds an output option for the underlying data source.
option(String, double) - Method in class org.apache.spark.sql.DataFrameWriter
Adds an output option for the underlying data source.
option(String, String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Adds an input option for the underlying data source.
option(String, boolean) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Adds an input option for the underlying data source.
option(String, long) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Adds an input option for the underlying data source.
option(String, double) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Adds an input option for the underlying data source.
option(String, String) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Adds an output option for the underlying data source.
option(String, boolean) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Adds an output option for the underlying data source.
option(String, long) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Adds an output option for the underlying data source.
option(String, double) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Adds an output option for the underlying data source.
Optional<T> - Class in org.apache.spark.api.java
Like java.util.Optional in Java 8, scala.Option in Scala, and com.google.common.base.Optional in Google Guava, this class represents a value of a given type that may or may not exist.
options(Map<String, String>) - Method in class org.apache.spark.sql.DataFrameReader
(Scala-specific) Adds input options for the underlying data source.
options(Map<String, String>) - Method in class org.apache.spark.sql.DataFrameReader
Adds input options for the underlying data source.
options(Map<String, String>) - Method in class org.apache.spark.sql.DataFrameWriter
(Scala-specific) Adds output options for the underlying data source.
options(Map<String, String>) - Method in class org.apache.spark.sql.DataFrameWriter
Adds output options for the underlying data source.
options(Map<String, String>) - Method in class org.apache.spark.sql.streaming.DataStreamReader
(Scala-specific) Adds input options for the underlying data source.
options(Map<String, String>) - Method in class org.apache.spark.sql.streaming.DataStreamReader
(Java-specific) Adds input options for the underlying data source.
options(Map<String, String>) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
(Scala-specific) Adds output options for the underlying data source.
options(Map<String, String>) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Adds output options for the underlying data source.
optionToOptional(Option<T>) - Static method in class org.apache.spark.api.java.JavaUtils
 
or(T) - Method in class org.apache.spark.api.java.Optional
 
or(Column) - Method in class org.apache.spark.sql.Column
Boolean OR.
Or - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff at least one of left or right evaluates to true.
Or(Filter, Filter) - Constructor for class org.apache.spark.sql.sources.Or
 
OracleDialect - Class in org.apache.spark.sql.jdbc
 
OracleDialect() - Constructor for class org.apache.spark.sql.jdbc.OracleDialect
 
orc(String...) - Method in class org.apache.spark.sql.DataFrameReader
Loads ORC files and returns the result as a DataFrame.
orc(String) - Method in class org.apache.spark.sql.DataFrameReader
Loads an ORC file and returns the result as a DataFrame.
orc(Seq<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads ORC files and returns the result as a DataFrame.
orc(String) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame in ORC format at the specified path.
orc(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads a ORC file stream, returning the result as a DataFrame.
OrcFileFormat - Class in org.apache.spark.sql.hive.orc
FileFormat for reading ORC files.
OrcFileFormat() - Constructor for class org.apache.spark.sql.hive.orc.OrcFileFormat
 
OrcFileOperator - Class in org.apache.spark.sql.hive.orc
 
OrcFileOperator() - Constructor for class org.apache.spark.sql.hive.orc.OrcFileOperator
 
OrcFilters - Class in org.apache.spark.sql.hive.orc
Helper object for building ORC SearchArguments, which are used for ORC predicate push-down.
OrcFilters() - Constructor for class org.apache.spark.sql.hive.orc.OrcFilters
 
orderBy(String, String...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the given expressions.
orderBy(Column...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the given expressions.
orderBy(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the given expressions.
orderBy(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the given expressions.
orderBy(String, String...) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the ordering defined.
orderBy(Column...) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the ordering defined.
orderBy(String, Seq<String>) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the ordering defined.
orderBy(Seq<Column>) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the ordering defined.
orderBy(String, String...) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the ordering columns in a WindowSpec.
orderBy(Column...) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the ordering columns in a WindowSpec.
orderBy(String, Seq<String>) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the ordering columns in a WindowSpec.
orderBy(Seq<Column>) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the ordering columns in a WindowSpec.
OrderedRDDFunctions<K,V,P extends scala.Product2<K,V>> - Class in org.apache.spark.rdd
Extra functions available on RDDs of (key, value) pairs where the key is sortable through an implicit conversion.
OrderedRDDFunctions(RDD<P>, Ordering<K>, ClassTag<K>, ClassTag<V>, ClassTag<P>) - Constructor for class org.apache.spark.rdd.OrderedRDDFunctions
 
ordering() - Static method in class org.apache.spark.streaming.Time
 
ORDINAL() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
orElse(T) - Method in class org.apache.spark.api.java.Optional
 
orElse(PartialFunction<A1, B1>) - Static method in class org.apache.spark.sql.types.StructType
 
org.apache.spark - package org.apache.spark
Core Spark classes in Scala.
org.apache.spark.api.java - package org.apache.spark.api.java
Spark Java programming APIs.
org.apache.spark.api.java.function - package org.apache.spark.api.java.function
Set of interfaces to represent functions in Spark's Java API.
org.apache.spark.api.r - package org.apache.spark.api.r
 
org.apache.spark.broadcast - package org.apache.spark.broadcast
Spark's broadcast variables, used to broadcast immutable datasets to all nodes.
org.apache.spark.graphx - package org.apache.spark.graphx
ALPHA COMPONENT GraphX is a graph processing framework built on top of Spark.
org.apache.spark.graphx.impl - package org.apache.spark.graphx.impl
 
org.apache.spark.graphx.lib - package org.apache.spark.graphx.lib
Various analytics functions for graphs.
org.apache.spark.graphx.util - package org.apache.spark.graphx.util
Collections of utilities used by graphx.
org.apache.spark.input - package org.apache.spark.input
 
org.apache.spark.internal - package org.apache.spark.internal
 
org.apache.spark.internal.config - package org.apache.spark.internal.config
 
org.apache.spark.internal.io - package org.apache.spark.internal.io
 
org.apache.spark.io - package org.apache.spark.io
IO codecs used for compression.
org.apache.spark.launcher - package org.apache.spark.launcher
Library for launching Spark applications programmatically.
org.apache.spark.mapred - package org.apache.spark.mapred
 
org.apache.spark.metrics.sink - package org.apache.spark.metrics.sink
 
org.apache.spark.metrics.source - package org.apache.spark.metrics.source
 
org.apache.spark.ml - package org.apache.spark.ml
DataFrame-based machine learning APIs to let users quickly assemble and configure practical machine learning pipelines.
org.apache.spark.ml.ann - package org.apache.spark.ml.ann
 
org.apache.spark.ml.attribute - package org.apache.spark.ml.attribute
ML attributes
org.apache.spark.ml.classification - package org.apache.spark.ml.classification
 
org.apache.spark.ml.clustering - package org.apache.spark.ml.clustering
 
org.apache.spark.ml.evaluation - package org.apache.spark.ml.evaluation
 
org.apache.spark.ml.feature - package org.apache.spark.ml.feature
Feature transformers The `ml.feature` package provides common feature transformers that help convert raw data or features into more suitable forms for model fitting.
org.apache.spark.ml.fpm - package org.apache.spark.ml.fpm
 
org.apache.spark.ml.image - package org.apache.spark.ml.image
 
org.apache.spark.ml.impl - package org.apache.spark.ml.impl
 
org.apache.spark.ml.linalg - package org.apache.spark.ml.linalg
 
org.apache.spark.ml.param - package org.apache.spark.ml.param
 
org.apache.spark.ml.param.shared - package org.apache.spark.ml.param.shared
 
org.apache.spark.ml.r - package org.apache.spark.ml.r
 
org.apache.spark.ml.recommendation - package org.apache.spark.ml.recommendation
 
org.apache.spark.ml.regression - package org.apache.spark.ml.regression
 
org.apache.spark.ml.source.libsvm - package org.apache.spark.ml.source.libsvm
 
org.apache.spark.ml.stat - package org.apache.spark.ml.stat
 
org.apache.spark.ml.stat.distribution - package org.apache.spark.ml.stat.distribution
 
org.apache.spark.ml.tree - package org.apache.spark.ml.tree
 
org.apache.spark.ml.tree.impl - package org.apache.spark.ml.tree.impl
 
org.apache.spark.ml.tuning - package org.apache.spark.ml.tuning
 
org.apache.spark.ml.util - package org.apache.spark.ml.util
 
org.apache.spark.mllib - package org.apache.spark.mllib
RDD-based machine learning APIs (in maintenance mode).
org.apache.spark.mllib.classification - package org.apache.spark.mllib.classification
 
org.apache.spark.mllib.classification.impl - package org.apache.spark.mllib.classification.impl
 
org.apache.spark.mllib.clustering - package org.apache.spark.mllib.clustering
 
org.apache.spark.mllib.evaluation - package org.apache.spark.mllib.evaluation
 
org.apache.spark.mllib.evaluation.binary - package org.apache.spark.mllib.evaluation.binary
 
org.apache.spark.mllib.feature - package org.apache.spark.mllib.feature
 
org.apache.spark.mllib.fpm - package org.apache.spark.mllib.fpm
 
org.apache.spark.mllib.linalg - package org.apache.spark.mllib.linalg
 
org.apache.spark.mllib.linalg.distributed - package org.apache.spark.mllib.linalg.distributed
 
org.apache.spark.mllib.optimization - package org.apache.spark.mllib.optimization
 
org.apache.spark.mllib.pmml - package org.apache.spark.mllib.pmml
 
org.apache.spark.mllib.pmml.export - package org.apache.spark.mllib.pmml.export
 
org.apache.spark.mllib.random - package org.apache.spark.mllib.random
 
org.apache.spark.mllib.rdd - package org.apache.spark.mllib.rdd
 
org.apache.spark.mllib.recommendation - package org.apache.spark.mllib.recommendation
 
org.apache.spark.mllib.regression - package org.apache.spark.mllib.regression
 
org.apache.spark.mllib.regression.impl - package org.apache.spark.mllib.regression.impl
 
org.apache.spark.mllib.stat - package org.apache.spark.mllib.stat
 
org.apache.spark.mllib.stat.correlation - package org.apache.spark.mllib.stat.correlation
 
org.apache.spark.mllib.stat.distribution - package org.apache.spark.mllib.stat.distribution
 
org.apache.spark.mllib.stat.test - package org.apache.spark.mllib.stat.test
 
org.apache.spark.mllib.tree - package org.apache.spark.mllib.tree
 
org.apache.spark.mllib.tree.configuration - package org.apache.spark.mllib.tree.configuration
 
org.apache.spark.mllib.tree.impurity - package org.apache.spark.mllib.tree.impurity
 
org.apache.spark.mllib.tree.loss - package org.apache.spark.mllib.tree.loss
 
org.apache.spark.mllib.tree.model - package org.apache.spark.mllib.tree.model
 
org.apache.spark.mllib.util - package org.apache.spark.mllib.util
 
org.apache.spark.partial - package org.apache.spark.partial
 
org.apache.spark.rdd - package org.apache.spark.rdd
Provides implementation's of various RDDs.
org.apache.spark.rpc.netty - package org.apache.spark.rpc.netty
 
org.apache.spark.scheduler - package org.apache.spark.scheduler
Spark's DAG scheduler.
org.apache.spark.scheduler.cluster - package org.apache.spark.scheduler.cluster
 
org.apache.spark.scheduler.local - package org.apache.spark.scheduler.local
 
org.apache.spark.security - package org.apache.spark.security
 
org.apache.spark.serializer - package org.apache.spark.serializer
Pluggable serializers for RDD and shuffle data.
org.apache.spark.sql - package org.apache.spark.sql
 
org.apache.spark.sql.api.java - package org.apache.spark.sql.api.java
Allows the execution of relational queries, including those expressed in SQL using Spark.
org.apache.spark.sql.api.r - package org.apache.spark.sql.api.r
 
org.apache.spark.sql.catalog - package org.apache.spark.sql.catalog
 
org.apache.spark.sql.expressions - package org.apache.spark.sql.expressions
 
org.apache.spark.sql.expressions.javalang - package org.apache.spark.sql.expressions.javalang
 
org.apache.spark.sql.expressions.scalalang - package org.apache.spark.sql.expressions.scalalang
 
org.apache.spark.sql.hive - package org.apache.spark.sql.hive
 
org.apache.spark.sql.hive.execution - package org.apache.spark.sql.hive.execution
 
org.apache.spark.sql.hive.orc - package org.apache.spark.sql.hive.orc
 
org.apache.spark.sql.jdbc - package org.apache.spark.sql.jdbc
 
org.apache.spark.sql.sources - package org.apache.spark.sql.sources
 
org.apache.spark.sql.sources.v2 - package org.apache.spark.sql.sources.v2
 
org.apache.spark.sql.sources.v2.reader - package org.apache.spark.sql.sources.v2.reader
 
org.apache.spark.sql.sources.v2.reader.partitioning - package org.apache.spark.sql.sources.v2.reader.partitioning
 
org.apache.spark.sql.sources.v2.reader.streaming - package org.apache.spark.sql.sources.v2.reader.streaming
 
org.apache.spark.sql.sources.v2.writer - package org.apache.spark.sql.sources.v2.writer
 
org.apache.spark.sql.sources.v2.writer.streaming - package org.apache.spark.sql.sources.v2.writer.streaming
 
org.apache.spark.sql.streaming - package org.apache.spark.sql.streaming
 
org.apache.spark.sql.types - package org.apache.spark.sql.types
 
org.apache.spark.sql.util - package org.apache.spark.sql.util
 
org.apache.spark.sql.vectorized - package org.apache.spark.sql.vectorized
 
org.apache.spark.status - package org.apache.spark.status
 
org.apache.spark.status.api.v1 - package org.apache.spark.status.api.v1
 
org.apache.spark.status.api.v1.streaming - package org.apache.spark.status.api.v1.streaming
 
org.apache.spark.storage - package org.apache.spark.storage
 
org.apache.spark.storage.memory - package org.apache.spark.storage.memory
 
org.apache.spark.streaming - package org.apache.spark.streaming
 
org.apache.spark.streaming.api.java - package org.apache.spark.streaming.api.java
Java APIs for spark streaming.
org.apache.spark.streaming.dstream - package org.apache.spark.streaming.dstream
Various implementations of DStreams.
org.apache.spark.streaming.kinesis - package org.apache.spark.streaming.kinesis
 
org.apache.spark.streaming.receiver - package org.apache.spark.streaming.receiver
 
org.apache.spark.streaming.scheduler - package org.apache.spark.streaming.scheduler
 
org.apache.spark.streaming.ui - package org.apache.spark.streaming.ui
 
org.apache.spark.streaming.util - package org.apache.spark.streaming.util
 
org.apache.spark.ui - package org.apache.spark.ui
 
org.apache.spark.ui.jobs - package org.apache.spark.ui.jobs
 
org.apache.spark.ui.storage - package org.apache.spark.ui.storage
 
org.apache.spark.util - package org.apache.spark.util
Spark utilities.
org.apache.spark.util.random - package org.apache.spark.util.random
Utilities for random number generation.
org.apache.spark.util.sketch - package org.apache.spark.util.sketch
 
origin() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
origin() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
origin() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
origin() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
originalMax() - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
originalMin() - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
orNull() - Method in class org.apache.spark.api.java.Optional
 
other() - Method in class org.apache.spark.scheduler.RuntimePercentage
 
otherVertexAttr(long) - Method in class org.apache.spark.graphx.EdgeTriplet
Given one vertex in the edge return the other vertex.
otherVertexId(long) - Method in class org.apache.spark.graphx.Edge
Given one vertex in the edge return the other vertex.
otherwise(Object) - Method in class org.apache.spark.sql.Column
Evaluates a list of conditions and returns one of multiple possible result expressions.
Out() - Static method in class org.apache.spark.graphx.EdgeDirection
Edges originating from a vertex.
outDegrees() - Method in class org.apache.spark.graphx.GraphOps
The out-degree of each vertex in the graph.
outerJoinVertices(RDD<Tuple2<Object, U>>, Function3<Object, VD, Option<U>, VD2>, ClassTag<U>, ClassTag<VD2>, Predef.$eq$colon$eq<VD, VD2>) - Method in class org.apache.spark.graphx.Graph
Joins the vertices with entries in the table RDD and merges the results using mapFunc.
outerJoinVertices(RDD<Tuple2<Object, U>>, Function3<Object, VD, Option<U>, VD2>, ClassTag<U>, ClassTag<VD2>, Predef.$eq$colon$eq<VD, VD2>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
outerJoinVertices$default$5(RDD<Tuple2<Object, U>>, Function3<Object, VD, Option<U>, VD2>) - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
output() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
output() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
output() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
output() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
OUTPUT() - Static method in class org.apache.spark.ui.ToolTips
 
output$() - Constructor for class org.apache.spark.InternalAccumulator.output$
 
OUTPUT_FORMAT() - Static method in class org.apache.spark.sql.hive.execution.HiveOptions
 
OUTPUT_METRICS_PREFIX() - Static method in class org.apache.spark.InternalAccumulator
 
OUTPUT_RECORDS() - Static method in class org.apache.spark.status.TaskIndexNames
 
OUTPUT_SIZE() - Static method in class org.apache.spark.status.TaskIndexNames
 
outputBytes() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
outputBytes() - Method in class org.apache.spark.status.api.v1.StageData
 
outputCol() - Static method in class org.apache.spark.ml.feature.Binarizer
 
outputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
outputCol() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
outputCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
outputCol() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
outputCol() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.DCT
 
outputCol() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
outputCol() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
outputCol() - Static method in class org.apache.spark.ml.feature.HashingTF
 
outputCol() - Static method in class org.apache.spark.ml.feature.IDF
 
outputCol() - Static method in class org.apache.spark.ml.feature.IDFModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.IndexToString
 
outputCol() - Static method in class org.apache.spark.ml.feature.Interaction
 
outputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
outputCol() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
outputCol() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
outputCol() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.NGram
 
outputCol() - Static method in class org.apache.spark.ml.feature.Normalizer
 
outputCol() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
outputCol() - Static method in class org.apache.spark.ml.feature.PCA
 
outputCol() - Static method in class org.apache.spark.ml.feature.PCAModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
outputCol() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
outputCol() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
outputCol() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
outputCol() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
outputCol() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
outputCol() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
outputCol() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
outputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
outputCol() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
outputCol() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
outputCol() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
outputCol() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
outputCol() - Method in interface org.apache.spark.ml.param.shared.HasOutputCol
Param for output column name.
outputCols() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
outputCols() - Static method in class org.apache.spark.ml.feature.Imputer
 
outputCols() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
outputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
outputCols() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
outputCols() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
outputCols() - Method in interface org.apache.spark.ml.param.shared.HasOutputCols
Param for output column names.
outputColumnNames() - Method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
outputColumnNames() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
outputColumnNames() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
outputColumns() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
outputColumns() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
outputColumns() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
OutputCommitCoordinationMessage - Interface in org.apache.spark.scheduler
 
outputCommitCoordinator() - Method in class org.apache.spark.SparkEnv
 
outputEncoder() - Method in class org.apache.spark.sql.expressions.Aggregator
Specifies the Encoder for the final output value type.
outputFormat() - Method in class org.apache.spark.sql.hive.execution.HiveOptions
 
OutputMetricDistributions - Class in org.apache.spark.status.api.v1
 
OutputMetrics - Class in org.apache.spark.status.api.v1
 
outputMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
outputMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
outputMode(OutputMode) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Specifies how data of a streaming DataFrame/Dataset is written to a streaming sink.
outputMode(String) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Specifies how data of a streaming DataFrame/Dataset is written to a streaming sink.
OutputMode - Class in org.apache.spark.sql.streaming
OutputMode describes what data will be written to a streaming sink when there is new data available in a streaming DataFrame/Dataset.
OutputMode() - Constructor for class org.apache.spark.sql.streaming.OutputMode
 
OutputOperationInfo - Class in org.apache.spark.status.api.v1.streaming
 
OutputOperationInfo - Class in org.apache.spark.streaming.scheduler
Developer API Class having information on output operations.
OutputOperationInfo(Time, int, String, String, Option<Object>, Option<Object>, Option<String>) - Constructor for class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
outputOperationInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
outputOperationInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
outputOperationInfos() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
 
outputOpId() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
outputOrdering() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
outputPartitioning() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
outputPartitioning() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsReportPartitioning
Returns the output data partitioning that this reader guarantees.
outputRecords() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
outputRecords() - Method in class org.apache.spark.status.api.v1.StageData
 
outputRowFormat() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
outputRowFormatMap() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
outputSerdeClass() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
outputSerdeProps() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
outputSet() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
outputSet() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
outputSet() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
outputSet() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
over(WindowSpec) - Method in class org.apache.spark.sql.Column
Defines a windowing column.
over() - Method in class org.apache.spark.sql.Column
Defines an empty analytic clause.
overwrite() - Method in class org.apache.spark.ml.util.MLWriter
Overwrites if the output path already exists.
overwrite() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
overwrite() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 

P

p() - Method in class org.apache.spark.ml.feature.Normalizer
Normalization in L^p^ space.
p(int) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
p(int) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
p(int) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
p(int) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
padTo(int, B, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
pageRank(double, double) - Method in class org.apache.spark.graphx.GraphOps
Run a dynamic version of PageRank returning a graph with vertex attributes containing the PageRank and edge attributes containing the normalized edge weight.
PageRank - Class in org.apache.spark.graphx.lib
PageRank algorithm implementation.
PageRank() - Constructor for class org.apache.spark.graphx.lib.PageRank
 
PairDStreamFunctions<K,V> - Class in org.apache.spark.streaming.dstream
Extra functions available on DStream of (key, value) pairs through an implicit conversion.
PairDStreamFunctions(DStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>, Ordering<K>) - Constructor for class org.apache.spark.streaming.dstream.PairDStreamFunctions
 
PairFlatMapFunction<T,K,V> - Interface in org.apache.spark.api.java.function
A function that returns zero or more key-value pair records from each input record.
PairFunction<T,K,V> - Interface in org.apache.spark.api.java.function
A function that returns key-value pairs (Tuple2<K, V>), and can be used to construct PairRDDs.
PairRDDFunctions<K,V> - Class in org.apache.spark.rdd
Extra functions available on RDDs of (key, value) pairs through an implicit conversion.
PairRDDFunctions(RDD<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>, Ordering<K>) - Constructor for class org.apache.spark.rdd.PairRDDFunctions
 
PairwiseRRDD<T> - Class in org.apache.spark.api.r
Form an RDD[(Int, Array[Byte])] from key-value pairs returned from R.
PairwiseRRDD(RDD<T>, int, byte[], String, byte[], Object[], ClassTag<T>) - Constructor for class org.apache.spark.api.r.PairwiseRRDD
 
par() - Static method in class org.apache.spark.sql.types.StructType
 
parallelism() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
parallelism() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
parallelism() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
parallelize(List<T>, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Distribute a local Scala collection to form an RDD.
parallelize(List<T>) - Method in class org.apache.spark.api.java.JavaSparkContext
Distribute a local Scala collection to form an RDD.
parallelize(Seq<T>, int, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Distribute a local Scala collection to form an RDD.
parallelizeDoubles(List<Double>, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Distribute a local Scala collection to form an RDD.
parallelizeDoubles(List<Double>) - Method in class org.apache.spark.api.java.JavaSparkContext
Distribute a local Scala collection to form an RDD.
parallelizePairs(List<Tuple2<K, V>>, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Distribute a local Scala collection to form an RDD.
parallelizePairs(List<Tuple2<K, V>>) - Method in class org.apache.spark.api.java.JavaSparkContext
Distribute a local Scala collection to form an RDD.
Param<T> - Class in org.apache.spark.ml.param
Developer API A param with self-contained documentation and optionally default value.
Param(String, String, String, Function1<T, Object>) - Constructor for class org.apache.spark.ml.param.Param
 
Param(Identifiable, String, String, Function1<T, Object>) - Constructor for class org.apache.spark.ml.param.Param
 
Param(String, String, String) - Constructor for class org.apache.spark.ml.param.Param
 
Param(Identifiable, String, String) - Constructor for class org.apache.spark.ml.param.Param
 
param() - Method in class org.apache.spark.ml.param.ParamPair
 
ParamGridBuilder - Class in org.apache.spark.ml.tuning
Builder for a param grid used in grid search-based model selection.
ParamGridBuilder() - Constructor for class org.apache.spark.ml.tuning.ParamGridBuilder
 
ParamMap - Class in org.apache.spark.ml.param
A param to value map.
ParamMap() - Constructor for class org.apache.spark.ml.param.ParamMap
Creates an empty param map.
paramMap() - Method in interface org.apache.spark.ml.param.Params
Internal param map for user-supplied values.
ParamPair<T> - Class in org.apache.spark.ml.param
A param and its value.
ParamPair(Param<T>, T) - Constructor for class org.apache.spark.ml.param.ParamPair
 
params() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
params() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
params() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
params() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
params() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
params() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
params() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
params() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
params() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
params() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
params() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
params() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
params() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
params() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
params() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
params() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
params() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
params() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
params() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
params() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
params() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
params() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
params() - Static method in class org.apache.spark.ml.clustering.KMeans
 
params() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
params() - Static method in class org.apache.spark.ml.clustering.LDA
 
params() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
params() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
params() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
params() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
params() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
params() - Static method in class org.apache.spark.ml.feature.Binarizer
 
params() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
params() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
params() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
params() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
params() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
params() - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
params() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
params() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
params() - Static method in class org.apache.spark.ml.feature.DCT
 
params() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
params() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
params() - Static method in class org.apache.spark.ml.feature.HashingTF
 
params() - Static method in class org.apache.spark.ml.feature.IDF
 
params() - Static method in class org.apache.spark.ml.feature.IDFModel
 
params() - Static method in class org.apache.spark.ml.feature.Imputer
 
params() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
params() - Static method in class org.apache.spark.ml.feature.IndexToString
 
params() - Static method in class org.apache.spark.ml.feature.Interaction
 
params() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
params() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
params() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
params() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
params() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
params() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
params() - Static method in class org.apache.spark.ml.feature.NGram
 
params() - Static method in class org.apache.spark.ml.feature.Normalizer
 
params() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
params() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
params() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
params() - Static method in class org.apache.spark.ml.feature.PCA
 
params() - Static method in class org.apache.spark.ml.feature.PCAModel
 
params() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
params() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
params() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
params() - Static method in class org.apache.spark.ml.feature.RFormula
 
params() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
params() - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
params() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
params() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
params() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
params() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
params() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
params() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
params() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
params() - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
params() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
params() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
params() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
params() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
params() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
params() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
params() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
params() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
Params - Interface in org.apache.spark.ml.param
Developer API Trait for components that take parameters.
params() - Method in interface org.apache.spark.ml.param.Params
Returns all params sorted by their names.
params() - Static method in class org.apache.spark.ml.Pipeline
 
params() - Static method in class org.apache.spark.ml.PipelineModel
 
params() - Static method in class org.apache.spark.ml.recommendation.ALS
 
params() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
params() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
params() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
params() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
params() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
params() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
params() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
params() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
params() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
params() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
params() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
params() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
params() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
params() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
params() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
params() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
params() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
params() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
params() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
ParamValidators - Class in org.apache.spark.ml.param
Developer API Factory methods for common validation functions for Param.isValid.
ParamValidators() - Constructor for class org.apache.spark.ml.param.ParamValidators
 
parent() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
parent() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
parent() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
parent() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
parent() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
parent() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
parent() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
parent() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
parent() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
parent() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
parent() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
parent() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
parent() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
parent() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
parent() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
parent() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
parent() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
parent() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
parent() - Static method in class org.apache.spark.ml.feature.IDFModel
 
parent() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
parent() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
parent() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
parent() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
parent() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
parent() - Static method in class org.apache.spark.ml.feature.PCAModel
 
parent() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
parent() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
parent() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
parent() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
parent() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
parent() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
parent() - Method in class org.apache.spark.ml.Model
The parent estimator that produced this model.
parent() - Static method in class org.apache.spark.ml.param.DoubleParam
 
parent() - Static method in class org.apache.spark.ml.param.FloatParam
 
parent() - Method in class org.apache.spark.ml.param.Param
 
parent() - Static method in class org.apache.spark.ml.PipelineModel
 
parent() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
parent() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
parent() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
parent() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
parent() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
parent() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
parent() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
parent() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
parent() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
parent() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.PipelineModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
parent_$eq(Estimator<M>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
ParentClassLoader - Class in org.apache.spark.util
A class loader which makes some protected methods in ClassLoader accessible.
ParentClassLoader(ClassLoader) - Constructor for class org.apache.spark.util.ParentClassLoader
 
parentIds() - Method in class org.apache.spark.scheduler.StageInfo
 
parentIds() - Method in class org.apache.spark.storage.RDDInfo
 
parentIndex(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Get the parent index of the given node, or 0 if it is the root.
parentState() - Static method in class org.apache.spark.sql.hive.HiveSessionStateBuilder
 
parquet(String...) - Method in class org.apache.spark.sql.DataFrameReader
Loads a Parquet file, returning the result as a DataFrame.
parquet(String) - Method in class org.apache.spark.sql.DataFrameReader
Loads a Parquet file, returning the result as a DataFrame.
parquet(Seq<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads a Parquet file, returning the result as a DataFrame.
parquet(String) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame in Parquet format at the specified path.
parquet(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads a Parquet file stream, returning the result as a DataFrame.
parquetFile(String...) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
As of 1.4.0, replaced by read().parquet().
parquetFile(Seq<String>) - Method in class org.apache.spark.sql.SQLContext
Deprecated.
Use read.parquet() instead. Since 1.4.0.
parse(String) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
parse(String) - Static method in class org.apache.spark.mllib.linalg.Vectors
Parses a string resulted from Vector.toString into a Vector.
parse(String) - Static method in class org.apache.spark.mllib.regression.LabeledPoint
Parses a string resulted from LabeledPoint#toString into an LabeledPoint.
parse(String) - Static method in class org.apache.spark.mllib.util.NumericParser
Parses a string into a Double, an Array[Double], or a Seq[Any].
parseAll(Parsers.Parser<T>, Reader<Object>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
parseAll(Parsers.Parser<T>, Reader) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
parseAll(Parsers.Parser<T>, CharSequence) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
parseHostPort(String) - Static method in class org.apache.spark.util.Utils
 
parseIgnoreCase(Class<E>, String) - Static method in class org.apache.spark.util.EnumUtil
 
Parser(Function1<Reader<Object>, Parsers.ParseResult<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
parseStandaloneMasterUrls(String) - Static method in class org.apache.spark.util.Utils
Split the comma delimited string of master URLs into a list.
PartialResult<R> - Class in org.apache.spark.partial
 
PartialResult(R, boolean) - Constructor for class org.apache.spark.partial.PartialResult
 
Partition - Interface in org.apache.spark
An identifier for a partition in an RDD.
partition() - Method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
partition() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
partition(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
partition(String) - Method in class org.apache.spark.status.LiveRDD
 
partitionBy(Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a copy of the RDD partitioned using the specified partitioner.
partitionBy(PartitionStrategy) - Method in class org.apache.spark.graphx.Graph
Repartitions the edges in the graph according to partitionStrategy.
partitionBy(PartitionStrategy, int) - Method in class org.apache.spark.graphx.Graph
Repartitions the edges in the graph according to partitionStrategy.
partitionBy(PartitionStrategy) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
partitionBy(PartitionStrategy, int) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
partitionBy(Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return a copy of the RDD partitioned using the specified partitioner.
partitionBy(String...) - Method in class org.apache.spark.sql.DataFrameWriter
Partitions the output by the given columns on the file system.
partitionBy(Seq<String>) - Method in class org.apache.spark.sql.DataFrameWriter
Partitions the output by the given columns on the file system.
partitionBy(String, String...) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the partitioning defined.
partitionBy(Column...) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the partitioning defined.
partitionBy(String, Seq<String>) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the partitioning defined.
partitionBy(Seq<Column>) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the partitioning defined.
partitionBy(String, String...) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the partitioning columns in a WindowSpec.
partitionBy(Column...) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the partitioning columns in a WindowSpec.
partitionBy(String, Seq<String>) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the partitioning columns in a WindowSpec.
partitionBy(Seq<Column>) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the partitioning columns in a WindowSpec.
partitionBy(String...) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Partitions the output by the given columns on the file system.
partitionBy(Seq<String>) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Partitions the output by the given columns on the file system.
PartitionCoalescer - Interface in org.apache.spark.rdd
::DeveloperApi:: A PartitionCoalescer defines how to coalesce the partitions of a given RDD.
partitioner() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
partitioner() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
partitioner() - Static method in class org.apache.spark.api.java.JavaRDD
 
partitioner() - Method in interface org.apache.spark.api.java.JavaRDDLike
The partitioner of this RDD.
partitioner() - Static method in class org.apache.spark.api.r.RRDD
 
partitioner() - Static method in class org.apache.spark.graphx.EdgeRDD
 
partitioner() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
If partitionsRDD already has a partitioner, use it.
partitioner() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
partitioner() - Static method in class org.apache.spark.graphx.VertexRDD
 
Partitioner - Class in org.apache.spark
An object that defines how the elements in a key-value pair RDD are partitioned by key.
Partitioner() - Constructor for class org.apache.spark.Partitioner
 
partitioner() - Method in class org.apache.spark.rdd.CoGroupedRDD
 
partitioner() - Static method in class org.apache.spark.rdd.HadoopRDD
 
partitioner() - Static method in class org.apache.spark.rdd.JdbcRDD
 
partitioner() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
partitioner() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
partitioner() - Method in class org.apache.spark.rdd.RDD
Optionally overridden by subclasses to specify how they are partitioned.
partitioner() - Method in class org.apache.spark.rdd.ShuffledRDD
 
partitioner() - Static method in class org.apache.spark.rdd.UnionRDD
 
partitioner() - Method in class org.apache.spark.ShuffleDependency
 
partitioner(Partitioner) - Method in class org.apache.spark.streaming.StateSpec
Set the partitioner by which the state RDDs generated by mapWithState will be partitioned.
PartitionGroup - Class in org.apache.spark.rdd
::DeveloperApi:: A group of Partitions param: prefLoc preferred location for the partition group
PartitionGroup(Option<String>) - Constructor for class org.apache.spark.rdd.PartitionGroup
 
partitionID() - Method in class org.apache.spark.TaskCommitDenied
 
partitionId() - Method in class org.apache.spark.TaskContext
The ID of the RDD partition that is computed by this task.
Partitioning - Interface in org.apache.spark.sql.sources.v2.reader.partitioning
An interface to represent the output data partitioning for a data source, which is returned by SupportsReportPartitioning.outputPartitioning().
PartitionLocations(RDD<?>) - Constructor for class org.apache.spark.rdd.DefaultPartitionCoalescer.PartitionLocations
 
PartitionOffset - Interface in org.apache.spark.sql.sources.v2.reader.streaming
Used for per-partition offsets in continuous processing.
PartitionPruningRDD<T> - Class in org.apache.spark.rdd
Developer API An RDD used to prune RDD partitions/partitions so we can avoid launching tasks on all partitions.
PartitionPruningRDD(RDD<T>, Function1<Object, Object>, ClassTag<T>) - Constructor for class org.apache.spark.rdd.PartitionPruningRDD
 
partitions() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
partitions() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
partitions() - Static method in class org.apache.spark.api.java.JavaRDD
 
partitions() - Method in interface org.apache.spark.api.java.JavaRDDLike
Set of partitions in this RDD.
partitions() - Static method in class org.apache.spark.api.r.RRDD
 
partitions() - Static method in class org.apache.spark.graphx.EdgeRDD
 
partitions() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
partitions() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
partitions() - Static method in class org.apache.spark.graphx.VertexRDD
 
partitions() - Static method in class org.apache.spark.rdd.HadoopRDD
 
partitions() - Static method in class org.apache.spark.rdd.JdbcRDD
 
partitions() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
partitions() - Method in class org.apache.spark.rdd.PartitionGroup
 
partitions() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
partitions() - Method in class org.apache.spark.rdd.RDD
Get the array of partitions of this RDD, taking into account whether the RDD is checkpointed or not.
partitions() - Static method in class org.apache.spark.rdd.UnionRDD
 
partitions() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
partitionsRDD() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
partitionsRDD() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
PartitionStrategy - Interface in org.apache.spark.graphx
Represents the way edges are assigned to edge partitions based on their source and destination vertex IDs.
PartitionStrategy.CanonicalRandomVertexCut$ - Class in org.apache.spark.graphx
Assigns edges to partitions by hashing the source and destination vertex IDs in a canonical direction, resulting in a random vertex cut that colocates all edges between two vertices, regardless of direction.
PartitionStrategy.EdgePartition1D$ - Class in org.apache.spark.graphx
Assigns edges to partitions using only the source vertex ID, colocating edges with the same source.
PartitionStrategy.EdgePartition2D$ - Class in org.apache.spark.graphx
Assigns edges to partitions using a 2D partitioning of the sparse edge adjacency matrix, guaranteeing a 2 * sqrt(numParts) bound on vertex replication.
PartitionStrategy.RandomVertexCut$ - Class in org.apache.spark.graphx
Assigns edges to partitions by hashing the source and destination vertex IDs, resulting in a random vertex cut that colocates all same-direction edges between two vertices.
partsWithLocs() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer.PartitionLocations
 
partsWithoutLocs() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer.PartitionLocations
 
patch(int, GenSeq<B>, int, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
path() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
path() - Method in class org.apache.spark.scheduler.SplitInfo
 
pattern() - Method in class org.apache.spark.ml.feature.RegexTokenizer
Regex pattern used to match delimiters if gaps is true or tokens if gaps is false.
pc() - Method in class org.apache.spark.ml.feature.PCAModel
 
pc() - Method in class org.apache.spark.mllib.feature.PCAModel
 
PCA - Class in org.apache.spark.ml.feature
PCA trains a model to project vectors to a lower dimensional space of the top PCA!.k principal components.
PCA(String) - Constructor for class org.apache.spark.ml.feature.PCA
 
PCA() - Constructor for class org.apache.spark.ml.feature.PCA
 
PCA - Class in org.apache.spark.mllib.feature
A feature transformer that projects vectors to a low-dimensional space using PCA.
PCA(int) - Constructor for class org.apache.spark.mllib.feature.PCA
 
PCAModel - Class in org.apache.spark.ml.feature
Model fitted by PCA.
PCAModel - Class in org.apache.spark.mllib.feature
Model fitted by PCA that can project vectors to a low-dimensional space using PCA.
PCAUtil - Class in org.apache.spark.mllib.feature
 
PCAUtil() - Constructor for class org.apache.spark.mllib.feature.PCAUtil
 
pdf(Vector) - Method in class org.apache.spark.ml.stat.distribution.MultivariateGaussian
Returns density of this multivariate Gaussian at given point, x
pdf(Vector) - Method in class org.apache.spark.mllib.stat.distribution.MultivariateGaussian
Returns density of this multivariate Gaussian at given point, x
PEAK_EXECUTION_MEMORY() - Static method in class org.apache.spark.InternalAccumulator
 
PEAK_EXECUTION_MEMORY() - Static method in class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
PEAK_EXECUTION_MEMORY() - Static method in class org.apache.spark.ui.ToolTips
 
PEAK_MEM() - Static method in class org.apache.spark.status.TaskIndexNames
 
peakExecutionMemory() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
peakExecutionMemory() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
PEARSON() - Static method in class org.apache.spark.mllib.stat.test.ChiSqTest
 
PearsonCorrelation - Class in org.apache.spark.mllib.stat.correlation
Compute Pearson correlation for two RDDs of the type RDD[Double] or the correlation matrix for an RDD of the type RDD[Vector].
PearsonCorrelation() - Constructor for class org.apache.spark.mllib.stat.correlation.PearsonCorrelation
 
percent_rank() - Static method in class org.apache.spark.sql.functions
Window function: returns the relative rank (i.e.
percentile() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
percentile() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
percentile() - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
percentiles() - Static method in class org.apache.spark.scheduler.StatsReportListener
 
percentilesHeader() - Static method in class org.apache.spark.scheduler.StatsReportListener
 
permutations() - Static method in class org.apache.spark.sql.types.StructType
 
persist(StorageLevel) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Set this RDD's storage level to persist its values across operations after the first time it is computed.
persist(StorageLevel) - Method in class org.apache.spark.api.java.JavaPairRDD
Set this RDD's storage level to persist its values across operations after the first time it is computed.
persist(StorageLevel) - Method in class org.apache.spark.api.java.JavaRDD
Set this RDD's storage level to persist its values across operations after the first time it is computed.
persist(StorageLevel) - Static method in class org.apache.spark.api.r.RRDD
 
persist() - Static method in class org.apache.spark.api.r.RRDD
 
persist(StorageLevel) - Static method in class org.apache.spark.graphx.EdgeRDD
 
persist() - Static method in class org.apache.spark.graphx.EdgeRDD
 
persist(StorageLevel) - Method in class org.apache.spark.graphx.Graph
Caches the vertices and edges associated with this graph at the specified storage level, ignoring any target storage levels previously set.
persist(StorageLevel) - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
Persists the edge partitions at the specified storage level, ignoring any existing target storage level.
persist(StorageLevel) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
persist(StorageLevel) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
Persists the vertex partitions at the specified storage level, ignoring any existing target storage level.
persist(StorageLevel) - Static method in class org.apache.spark.graphx.VertexRDD
 
persist() - Static method in class org.apache.spark.graphx.VertexRDD
 
persist(StorageLevel) - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Persists the underlying RDD with the specified storage level.
persist(StorageLevel) - Method in class org.apache.spark.rdd.HadoopRDD
 
persist(StorageLevel) - Static method in class org.apache.spark.rdd.JdbcRDD
 
persist() - Static method in class org.apache.spark.rdd.JdbcRDD
 
persist(StorageLevel) - Method in class org.apache.spark.rdd.NewHadoopRDD
 
persist(StorageLevel) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
persist() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
persist(StorageLevel) - Method in class org.apache.spark.rdd.RDD
Set this RDD's storage level to persist its values across operations after the first time it is computed.
persist() - Method in class org.apache.spark.rdd.RDD
Persist this RDD with the default storage level (MEMORY_ONLY).
persist(StorageLevel) - Static method in class org.apache.spark.rdd.UnionRDD
 
persist() - Static method in class org.apache.spark.rdd.UnionRDD
 
persist() - Method in class org.apache.spark.sql.Dataset
Persist this Dataset with the default storage level (MEMORY_AND_DISK).
persist(StorageLevel) - Method in class org.apache.spark.sql.Dataset
Persist this Dataset with the given storage level.
persist() - Method in class org.apache.spark.streaming.api.java.JavaDStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
persist(StorageLevel) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Persist the RDDs of this DStream with the given storage level
persist() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
persist(StorageLevel) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
persist() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
persist(StorageLevel) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Persist the RDDs of this DStream with the given storage level
persist() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
persist(StorageLevel) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
persist() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
persist(StorageLevel) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
persist() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
persist(StorageLevel) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
persist(StorageLevel) - Method in class org.apache.spark.streaming.dstream.DStream
Persist the RDDs of this DStream with the given storage level
persist() - Method in class org.apache.spark.streaming.dstream.DStream
Persist RDDs of this DStream with the default storage level (MEMORY_ONLY_SER)
persist$default$1() - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
personalizedPageRank(long, double, double) - Method in class org.apache.spark.graphx.GraphOps
Run personalized PageRank for a given vertex, such that all random walks are started relative to the source node.
phrase(Parsers.Parser<T>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
pi() - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
pi() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
pi() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$.Data
 
pi() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$.Data
 
pickBin(Partition, RDD<?>, double, DefaultPartitionCoalescer.PartitionLocations) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
Takes a parent RDD partition and decides which of the partition groups to put it in Takes locality into account, but also uses power of 2 choices to load balance It strikes a balance between the two using the balanceSlack variable
pickRandomVertex() - Method in class org.apache.spark.graphx.GraphOps
Picks a random vertex from the graph and returns its ID.
pipe(String) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
pipe(List<String>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
pipe(List<String>, Map<String, String>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
pipe(List<String>, Map<String, String>, boolean, int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
pipe(List<String>, Map<String, String>, boolean, int, String) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
pipe(String) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
pipe(List<String>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
pipe(List<String>, Map<String, String>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
pipe(List<String>, Map<String, String>, boolean, int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
pipe(List<String>, Map<String, String>, boolean, int, String) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
pipe(String) - Static method in class org.apache.spark.api.java.JavaRDD
 
pipe(List<String>) - Static method in class org.apache.spark.api.java.JavaRDD
 
pipe(List<String>, Map<String, String>) - Static method in class org.apache.spark.api.java.JavaRDD
 
pipe(List<String>, Map<String, String>, boolean, int) - Static method in class org.apache.spark.api.java.JavaRDD
 
pipe(List<String>, Map<String, String>, boolean, int, String) - Static method in class org.apache.spark.api.java.JavaRDD
 
pipe(String) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD created by piping elements to a forked external process.
pipe(List<String>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD created by piping elements to a forked external process.
pipe(List<String>, Map<String, String>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD created by piping elements to a forked external process.
pipe(List<String>, Map<String, String>, boolean, int) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD created by piping elements to a forked external process.
pipe(List<String>, Map<String, String>, boolean, int, String) - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an RDD created by piping elements to a forked external process.
pipe(String) - Static method in class org.apache.spark.api.r.RRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.api.r.RRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.api.r.RRDD
 
pipe(String) - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe(String) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe(String) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe(String) - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe(String) - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe(String) - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe(String) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe(String) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe(String) - Method in class org.apache.spark.rdd.RDD
Return an RDD created by piping elements to a forked external process.
pipe(String, Map<String, String>) - Method in class org.apache.spark.rdd.RDD
Return an RDD created by piping elements to a forked external process.
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Method in class org.apache.spark.rdd.RDD
Return an RDD created by piping elements to a forked external process.
pipe(String) - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe(String, Map<String, String>) - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe(Seq<String>, Map<String, String>, Function1<Function1<String, BoxedUnit>, BoxedUnit>, Function2<T, Function1<String, BoxedUnit>, BoxedUnit>, boolean, int, String) - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
pipe$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe$default$3() - Static method in class org.apache.spark.api.r.RRDD
 
pipe$default$3() - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe$default$3() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe$default$3() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe$default$3() - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe$default$3() - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe$default$3() - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe$default$3() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe$default$3() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe$default$3() - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe$default$4() - Static method in class org.apache.spark.api.r.RRDD
 
pipe$default$4() - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe$default$4() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe$default$4() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe$default$4() - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe$default$4() - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe$default$4() - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe$default$4() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe$default$4() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe$default$4() - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe$default$5() - Static method in class org.apache.spark.api.r.RRDD
 
pipe$default$5() - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe$default$5() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe$default$5() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe$default$5() - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe$default$5() - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe$default$5() - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe$default$5() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe$default$5() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe$default$5() - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe$default$6() - Static method in class org.apache.spark.api.r.RRDD
 
pipe$default$6() - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe$default$6() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe$default$6() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe$default$6() - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe$default$6() - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe$default$6() - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe$default$6() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe$default$6() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe$default$6() - Static method in class org.apache.spark.rdd.UnionRDD
 
pipe$default$7() - Static method in class org.apache.spark.api.r.RRDD
 
pipe$default$7() - Static method in class org.apache.spark.graphx.EdgeRDD
 
pipe$default$7() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
pipe$default$7() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
pipe$default$7() - Static method in class org.apache.spark.graphx.VertexRDD
 
pipe$default$7() - Static method in class org.apache.spark.rdd.HadoopRDD
 
pipe$default$7() - Static method in class org.apache.spark.rdd.JdbcRDD
 
pipe$default$7() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
pipe$default$7() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
pipe$default$7() - Static method in class org.apache.spark.rdd.UnionRDD
 
Pipeline - Class in org.apache.spark.ml
A simple pipeline, which acts as an estimator.
Pipeline(String) - Constructor for class org.apache.spark.ml.Pipeline
 
Pipeline() - Constructor for class org.apache.spark.ml.Pipeline
 
Pipeline.SharedReadWrite$ - Class in org.apache.spark.ml
Methods for MLReader and MLWriter shared between Pipeline and PipelineModel
PipelineModel - Class in org.apache.spark.ml
Represents a fitted pipeline.
PipelineStage - Class in org.apache.spark.ml
Developer API A stage in a pipeline, either an Estimator or a Transformer.
PipelineStage() - Constructor for class org.apache.spark.ml.PipelineStage
 
pivot(String) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Pivots a column of the current DataFrame and performs the specified aggregation.
pivot(String, Seq<Object>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Pivots a column of the current DataFrame and performs the specified aggregation.
pivot(String, List<Object>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
(Java-specific) Pivots a column of the current DataFrame and performs the specified aggregation.
PivotType$() - Constructor for class org.apache.spark.sql.RelationalGroupedDataset.PivotType$
 
plan() - Method in exception org.apache.spark.sql.AnalysisException
 
plus(Object) - Method in class org.apache.spark.sql.Column
Sum of this expression and another expression.
plus(Duration) - Method in class org.apache.spark.streaming.Duration
 
plus(Duration) - Method in class org.apache.spark.streaming.Time
 
PMMLExportable - Interface in org.apache.spark.mllib.pmml
Developer API Export model to the PMML format Predictive Model Markup Language (PMML) is an XML-based file format developed by the Data Mining Group (www.dmg.org).
PMMLModelExportFactory - Class in org.apache.spark.mllib.pmml.export
 
PMMLModelExportFactory() - Constructor for class org.apache.spark.mllib.pmml.export.PMMLModelExportFactory
 
pmod(Column, Column) - Static method in class org.apache.spark.sql.functions
Returns the positive value of dividend mod divisor.
point() - Method in class org.apache.spark.mllib.feature.VocabWord
 
POINTS() - Static method in class org.apache.spark.mllib.clustering.StreamingKMeans
 
POISON_PILL() - Static method in class org.apache.spark.scheduler.AsyncEventQueue
 
Poisson$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
 
PoissonBounds - Class in org.apache.spark.util.random
Utility functions that help us determine bounds on adjusted sampling rate to guarantee exact sample sizes with high confidence when sampling with replacement.
PoissonBounds() - Constructor for class org.apache.spark.util.random.PoissonBounds
 
PoissonGenerator - Class in org.apache.spark.mllib.random
Developer API Generates i.i.d.
PoissonGenerator(double) - Constructor for class org.apache.spark.mllib.random.PoissonGenerator
 
poissonJavaRDD(JavaSparkContext, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.poissonRDD.
poissonJavaRDD(JavaSparkContext, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.poissonJavaRDD with the default seed.
poissonJavaRDD(JavaSparkContext, double, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.poissonJavaRDD with the default number of partitions and the default seed.
poissonJavaVectorRDD(JavaSparkContext, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.poissonVectorRDD.
poissonJavaVectorRDD(JavaSparkContext, double, long, int, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.poissonJavaVectorRDD with the default seed.
poissonJavaVectorRDD(JavaSparkContext, double, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.poissonJavaVectorRDD with the default number of partitions and the default seed.
poissonRDD(SparkContext, double, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD comprised of i.i.d. samples from the Poisson distribution with the input mean.
PoissonSampler<T> - Class in org.apache.spark.util.random
Developer API A sampler for sampling with replacement, based on values drawn from Poisson distribution.
PoissonSampler(double, boolean) - Constructor for class org.apache.spark.util.random.PoissonSampler
 
PoissonSampler(double) - Constructor for class org.apache.spark.util.random.PoissonSampler
 
poissonVectorRDD(SparkContext, double, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD[Vector] with vectors containing i.i.d. samples drawn from the Poisson distribution with the input mean.
PolynomialExpansion - Class in org.apache.spark.ml.feature
Perform feature expansion in a polynomial space.
PolynomialExpansion(String) - Constructor for class org.apache.spark.ml.feature.PolynomialExpansion
 
PolynomialExpansion() - Constructor for class org.apache.spark.ml.feature.PolynomialExpansion
 
popStdev() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute the population standard deviation of this RDD's elements.
popStdev() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute the population standard deviation of this RDD's elements.
popStdev() - Method in class org.apache.spark.util.StatCounter
Return the population standard deviation of the values.
popVariance() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute the population variance of this RDD's elements.
popVariance() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute the population variance of this RDD's elements.
popVariance() - Method in class org.apache.spark.util.StatCounter
Return the population variance of the values.
port() - Method in interface org.apache.spark.SparkExecutorInfo
 
port() - Method in class org.apache.spark.SparkExecutorInfoImpl
 
port() - Method in class org.apache.spark.storage.BlockManagerId
 
PortableDataStream - Class in org.apache.spark.input
A class that allows DataStreams to be serialized and moved around by not creating them until they need to be read
PortableDataStream(CombineFileSplit, TaskAttemptContext, Integer) - Constructor for class org.apache.spark.input.PortableDataStream
 
portMaxRetries(SparkConf) - Static method in class org.apache.spark.util.Utils
Maximum number of retries when binding to a port before giving up.
posexplode(Column) - Static method in class org.apache.spark.sql.functions
Creates a new row for each element with position in the given array or map column.
posexplode_outer(Column) - Static method in class org.apache.spark.sql.functions
Creates a new row for each element with position in the given array or map column.
position() - Method in class org.apache.spark.storage.ReadableChannelFileRegion
 
positioned(Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
post(SparkListenerEvent) - Method in class org.apache.spark.scheduler.AsyncEventQueue
 
Postfix$() - Constructor for class org.apache.spark.mllib.fpm.PrefixSpan.Postfix$
 
PostgresDialect - Class in org.apache.spark.sql.jdbc
 
PostgresDialect() - Constructor for class org.apache.spark.sql.jdbc.PostgresDialect
 
postToAll(E) - Static method in class org.apache.spark.scheduler.AsyncEventQueue
 
pow(Column, Column) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
pow(Column, String) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
pow(String, Column) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
pow(String, String) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
pow(Column, double) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
pow(String, double) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
pow(double, Column) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
pow(double, String) - Static method in class org.apache.spark.sql.functions
Returns the value of the first argument raised to the power of the second argument.
PowerIterationClustering - Class in org.apache.spark.mllib.clustering
Power Iteration Clustering (PIC), a scalable graph clustering algorithm developed by Lin and Cohen.
PowerIterationClustering() - Constructor for class org.apache.spark.mllib.clustering.PowerIterationClustering
Constructs a PIC instance with default parameters: {k: 2, maxIterations: 100, initMode: "random"}.
PowerIterationClustering.Assignment - Class in org.apache.spark.mllib.clustering
Cluster assignment.
PowerIterationClustering.Assignment$ - Class in org.apache.spark.mllib.clustering
 
PowerIterationClusteringModel - Class in org.apache.spark.mllib.clustering
Model produced by PowerIterationClustering.
PowerIterationClusteringModel(int, RDD<PowerIterationClustering.Assignment>) - Constructor for class org.apache.spark.mllib.clustering.PowerIterationClusteringModel
 
PowerIterationClusteringModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.clustering
 
pr() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
Returns the precision-recall curve, which is a Dataframe containing two fields recall, precision with (0.0, 1.0) prepended to it.
pr() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Returns the precision-recall curve, which is an RDD of (recall, precision), NOT (precision, recall), with (0.0, p) prepended to it, where p is the precision associated with the lowest recall on the curve.
Precision - Class in org.apache.spark.mllib.evaluation.binary
Precision.
Precision() - Constructor for class org.apache.spark.mllib.evaluation.binary.Precision
 
precision(double) - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns precision for a given label (category)
precision() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Deprecated.
Use accuracy. Since 2.0.0.
precision() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns document-based precision averaged by the number of documents
precision(double) - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns precision for a given label (category)
precision() - Method in class org.apache.spark.sql.types.Decimal
 
precision() - Method in class org.apache.spark.sql.types.DecimalType
 
precisionAt(int) - Method in class org.apache.spark.mllib.evaluation.RankingMetrics
Compute the average precision of all the queries, truncated at ranking position k.
precisionByLabel() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns precision for each label (category).
precisionByThreshold() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
Returns a dataframe with two fields (threshold, precision) curve.
precisionByThreshold() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Returns the (threshold, precision) curve.
predict(Vector) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
predict(RDD<Vector>) - Method in interface org.apache.spark.mllib.classification.ClassificationModel
Predict values for the given data set using the model trained.
predict(Vector) - Method in interface org.apache.spark.mllib.classification.ClassificationModel
Predict values for a single data point using the model trained.
predict(JavaRDD<Vector>) - Method in interface org.apache.spark.mllib.classification.ClassificationModel
Predict values for examples stored in a JavaRDD.
predict(RDD<Vector>) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
predict(Vector) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
predict(JavaRDD<Vector>) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
predict(RDD<Vector>) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
predict(Vector) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
predict(RDD<Vector>) - Static method in class org.apache.spark.mllib.classification.SVMModel
 
predict(Vector) - Static method in class org.apache.spark.mllib.classification.SVMModel
 
predict(JavaRDD<Vector>) - Static method in class org.apache.spark.mllib.classification.SVMModel
 
predict(Vector) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Predicts the index of the cluster that the input point belongs to.
predict(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Predicts the indices of the clusters that the input points belong to.
predict(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
Java-friendly version of predict().
predict(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
Maps given points to their cluster indices.
predict(Vector) - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
Maps given point to its cluster index.
predict(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
Java-friendly version of predict()
predict(Vector) - Method in class org.apache.spark.mllib.clustering.KMeansModel
Returns the cluster index that a given point belongs to.
predict(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.KMeansModel
Maps given points to their cluster indices.
predict(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.clustering.KMeansModel
Maps given points to their cluster indices.
predict(int, int) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Predict the rating of one user for one product.
predict(RDD<Tuple2<Object, Object>>) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Predict the rating of many users for many products.
predict(JavaPairRDD<Integer, Integer>) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Java-friendly version of MatrixFactorizationModel.predict.
predict(RDD<Vector>) - Method in class org.apache.spark.mllib.regression.GeneralizedLinearModel
Predict values for the given data set using the model trained.
predict(Vector) - Method in class org.apache.spark.mllib.regression.GeneralizedLinearModel
Predict values for a single data point using the model trained.
predict(RDD<Object>) - Method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
Predict labels for provided features.
predict(JavaDoubleRDD) - Method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
Predict labels for provided features.
predict(double) - Method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
Predict a single label.
predict(RDD<Vector>) - Static method in class org.apache.spark.mllib.regression.LassoModel
 
predict(Vector) - Static method in class org.apache.spark.mllib.regression.LassoModel
 
predict(JavaRDD<Vector>) - Static method in class org.apache.spark.mllib.regression.LassoModel
 
predict(RDD<Vector>) - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
predict(Vector) - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
predict(JavaRDD<Vector>) - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
predict(RDD<Vector>) - Method in interface org.apache.spark.mllib.regression.RegressionModel
Predict values for the given data set using the model trained.
predict(Vector) - Method in interface org.apache.spark.mllib.regression.RegressionModel
Predict values for a single data point using the model trained.
predict(JavaRDD<Vector>) - Method in interface org.apache.spark.mllib.regression.RegressionModel
Predict values for examples stored in a JavaRDD.
predict(RDD<Vector>) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
predict(Vector) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
predict(JavaRDD<Vector>) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
predict(Vector) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Predict values for a single data point using the model trained.
predict(RDD<Vector>) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Predict values for the given data set using the model trained.
predict(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Predict values for the given data set using the model trained.
predict() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
predict() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.PredictData
 
predict(Vector) - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
predict(RDD<Vector>) - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
predict(JavaRDD<Vector>) - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
predict() - Method in class org.apache.spark.mllib.tree.model.Node
 
predict(Vector) - Method in class org.apache.spark.mllib.tree.model.Node
predict value if node is not leaf
Predict - Class in org.apache.spark.mllib.tree.model
Developer API Predicted value for a node param: predict predicted value param: prob probability of the label (classification only)
Predict(double, double) - Constructor for class org.apache.spark.mllib.tree.model.Predict
 
predict() - Method in class org.apache.spark.mllib.tree.model.Predict
 
predict(Vector) - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
predict(RDD<Vector>) - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
predict(JavaRDD<Vector>) - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
PredictData(double, double) - Constructor for class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.PredictData
 
prediction() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
prediction() - Method in class org.apache.spark.ml.tree.InternalNode
 
prediction() - Method in class org.apache.spark.ml.tree.LeafNode
 
prediction() - Method in class org.apache.spark.ml.tree.Node
Prediction a leaf node makes, or which an internal node would make if it were a leaf node
predictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
predictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
predictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
predictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
predictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
predictionCol() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Field in "predictions" which gives the prediction of each class.
predictionCol() - Method in class org.apache.spark.ml.classification.LogisticRegressionSummaryImpl
 
predictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
predictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
predictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
predictionCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
predictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
predictionCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
predictionCol() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
predictionCol() - Method in class org.apache.spark.ml.clustering.ClusteringSummary
 
predictionCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
predictionCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
predictionCol() - Static method in class org.apache.spark.ml.clustering.KMeans
 
predictionCol() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
predictionCol() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
predictionCol() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
predictionCol() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
predictionCol() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
predictionCol() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
predictionCol() - Method in interface org.apache.spark.ml.param.shared.HasPredictionCol
Param for prediction column name.
predictionCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
predictionCol() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
predictionCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
predictionCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
predictionCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
predictionCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
predictionCol() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
predictionCol() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
predictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
predictionCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
predictionCol() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
Field in "predictions" which gives the predicted value of each instance.
predictionCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
predictionCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
predictionCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
predictionCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
predictionCol() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
 
predictionCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
predictionCol() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
PredictionModel<FeaturesType,M extends PredictionModel<FeaturesType,M>> - Class in org.apache.spark.ml
Developer API Abstraction for a model for prediction tasks (regression and classification).
PredictionModel() - Constructor for class org.apache.spark.ml.PredictionModel
 
predictions() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Dataframe output by the model's transform method.
predictions() - Method in class org.apache.spark.ml.classification.LogisticRegressionSummaryImpl
 
predictions() - Method in class org.apache.spark.ml.clustering.ClusteringSummary
 
predictions() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
Predictions output by the model's transform method.
predictions() - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
Predictions associated with the boundaries at the same index, monotone because of isotonic regression.
predictions() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
 
predictions() - Method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
 
predictOn(DStream<Vector>) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Use the clustering model to make predictions on batches of data from a DStream.
predictOn(JavaDStream<Vector>) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Java-friendly version of predictOn.
predictOn(DStream<Vector>) - Method in class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
Use the model to make predictions on batches of data from a DStream
predictOn(JavaDStream<Vector>) - Method in class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
Java-friendly version of predictOn.
predictOnValues(DStream<Tuple2<K, Vector>>, ClassTag<K>) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Use the model to make predictions on the values of a DStream and carry over its keys.
predictOnValues(JavaPairDStream<K, Vector>) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Java-friendly version of predictOnValues.
predictOnValues(DStream<Tuple2<K, Vector>>, ClassTag<K>) - Method in class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
Use the model to make predictions on the values of a DStream and carry over its keys.
predictOnValues(JavaPairDStream<K, Vector>) - Method in class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
Java-friendly version of predictOnValues.
Predictor<FeaturesType,Learner extends Predictor<FeaturesType,Learner,M>,M extends PredictionModel<FeaturesType,M>> - Class in org.apache.spark.ml
Developer API Abstraction for prediction problems (regression and classification).
Predictor() - Constructor for class org.apache.spark.ml.Predictor
 
predictProbabilities(RDD<Vector>) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
Predict values for the given data set using the model trained.
predictProbabilities(Vector) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
Predict posterior class probabilities for a single data point using the model trained.
predictQuantiles(Vector) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
predictSoft(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
Given the input vectors, return the membership value of each vector to all mixture components.
predictSoft(Vector) - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
Given the input vector, return the membership values to all mixture components.
preferredLocation() - Method in class org.apache.spark.streaming.receiver.Receiver
Override this to specify a preferred location (hostname).
preferredLocations(Partition) - Static method in class org.apache.spark.api.r.RRDD
 
preferredLocations(Partition) - Static method in class org.apache.spark.graphx.EdgeRDD
 
preferredLocations(Partition) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
preferredLocations(Partition) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
preferredLocations(Partition) - Static method in class org.apache.spark.graphx.VertexRDD
 
preferredLocations(Partition) - Static method in class org.apache.spark.rdd.HadoopRDD
 
preferredLocations(Partition) - Static method in class org.apache.spark.rdd.JdbcRDD
 
preferredLocations(Partition) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
preferredLocations(Partition) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
preferredLocations(Partition) - Method in class org.apache.spark.rdd.RDD
Get the preferred locations of a partition, taking into account whether the RDD is checkpointed.
preferredLocations(Partition) - Static method in class org.apache.spark.rdd.UnionRDD
 
preferredLocations() - Method in interface org.apache.spark.sql.sources.v2.reader.DataReaderFactory
The preferred locations where the data reader returned by this reader factory can run faster, but Spark does not guarantee to run the data reader on these locations.
Prefix$() - Constructor for class org.apache.spark.mllib.fpm.PrefixSpan.Prefix$
 
prefixesToRewrite() - Method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
prefixLength(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
PrefixSpan - Class in org.apache.spark.mllib.fpm
A parallel PrefixSpan algorithm to mine frequent sequential patterns.
PrefixSpan() - Constructor for class org.apache.spark.mllib.fpm.PrefixSpan
Constructs a default instance with default parameters {minSupport: 0.1, maxPatternLength: 10, maxLocalProjDBSize: 32000000L}.
PrefixSpan.FreqSequence<Item> - Class in org.apache.spark.mllib.fpm
Represents a frequent sequence.
PrefixSpan.Postfix$ - Class in org.apache.spark.mllib.fpm
 
PrefixSpan.Prefix$ - Class in org.apache.spark.mllib.fpm
 
PrefixSpanModel<Item> - Class in org.apache.spark.mllib.fpm
Model fitted by PrefixSpan param: freqSequences frequent sequences
PrefixSpanModel(RDD<PrefixSpan.FreqSequence<Item>>) - Constructor for class org.apache.spark.mllib.fpm.PrefixSpanModel
 
PrefixSpanModel.SaveLoadV1_0$ - Class in org.apache.spark.mllib.fpm
 
prefLoc() - Method in class org.apache.spark.rdd.PartitionGroup
 
pregel(A, int, EdgeDirection, Function3<Object, VD, A, VD>, Function1<EdgeTriplet<VD, ED>, Iterator<Tuple2<Object, A>>>, Function2<A, A, A>, ClassTag<A>) - Method in class org.apache.spark.graphx.GraphOps
Execute a Pregel-like iterative vertex-parallel abstraction.
Pregel - Class in org.apache.spark.graphx
Implements a Pregel-like bulk-synchronous message-passing API.
Pregel() - Constructor for class org.apache.spark.graphx.Pregel
 
prepare() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
prepareWritable(Writable, Seq<Tuple2<String, String>>) - Static method in class org.apache.spark.sql.hive.HiveShim
 
prepareWrite(SparkSession, Job, Map<String, String>, StructType) - Method in class org.apache.spark.sql.hive.execution.HiveFileFormat
 
prepareWrite(SparkSession, Job, Map<String, String>, StructType) - Method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
prependBaseUri(String, String) - Static method in class org.apache.spark.ui.UIUtils
 
prettyJson() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
prettyJson() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
prettyJson() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
prettyJson() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
prettyJson() - Method in class org.apache.spark.sql.streaming.SinkProgress
The pretty (i.e.
prettyJson() - Method in class org.apache.spark.sql.streaming.SourceProgress
The pretty (i.e.
prettyJson() - Method in class org.apache.spark.sql.streaming.StateOperatorProgress
The pretty (i.e.
prettyJson() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
The pretty (i.e.
prettyJson() - Method in class org.apache.spark.sql.streaming.StreamingQueryStatus
The pretty (i.e.
prettyJson() - Static method in class org.apache.spark.sql.types.ArrayType
 
prettyJson() - Static method in class org.apache.spark.sql.types.BinaryType
 
prettyJson() - Static method in class org.apache.spark.sql.types.BooleanType
 
prettyJson() - Static method in class org.apache.spark.sql.types.ByteType
 
prettyJson() - Static method in class org.apache.spark.sql.types.CalendarIntervalType
 
prettyJson() - Static method in class org.apache.spark.sql.types.CharType
 
prettyJson() - Method in class org.apache.spark.sql.types.DataType
The pretty (i.e.
prettyJson() - Static method in class org.apache.spark.sql.types.DateType
 
prettyJson() - Static method in class org.apache.spark.sql.types.DecimalType
 
prettyJson() - Static method in class org.apache.spark.sql.types.DoubleType
 
prettyJson() - Static method in class org.apache.spark.sql.types.FloatType
 
prettyJson() - Static method in class org.apache.spark.sql.types.HiveStringType
 
prettyJson() - Static method in class org.apache.spark.sql.types.IntegerType
 
prettyJson() - Static method in class org.apache.spark.sql.types.LongType
 
prettyJson() - Static method in class org.apache.spark.sql.types.MapType
 
prettyJson() - Static method in class org.apache.spark.sql.types.NullType
 
prettyJson() - Static method in class org.apache.spark.sql.types.NumericType
 
prettyJson() - Static method in class org.apache.spark.sql.types.ObjectType
 
prettyJson() - Static method in class org.apache.spark.sql.types.ShortType
 
prettyJson() - Static method in class org.apache.spark.sql.types.StringType
 
prettyJson() - Static method in class org.apache.spark.sql.types.StructType
 
prettyJson() - Static method in class org.apache.spark.sql.types.TimestampType
 
prettyJson() - Static method in class org.apache.spark.sql.types.VarcharType
 
prettyPrint() - Method in class org.apache.spark.streaming.Duration
 
prev() - Method in class org.apache.spark.rdd.ShuffledRDD
 
prev() - Method in class org.apache.spark.status.LiveRDDPartition
 
print() - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
print(int) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
print() - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Print the first ten elements of each RDD generated in this DStream.
print(int) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Print the first num elements of each RDD generated in this DStream.
print() - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
print(int) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
print() - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
print(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
print() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
print(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
print() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
print(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
print() - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
print(int) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
print() - Method in class org.apache.spark.streaming.dstream.DStream
Print the first ten elements of each RDD generated in this DStream.
print(int) - Method in class org.apache.spark.streaming.dstream.DStream
Print the first num elements of each RDD generated in this DStream.
printSchema() - Method in class org.apache.spark.sql.Dataset
Prints the schema to the console in a nice tree format.
printSchema() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
printSchema() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
printSchema() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
printSchema() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
printStackTrace() - Static method in exception org.apache.spark.sql.AnalysisException
 
printStackTrace(PrintStream) - Static method in exception org.apache.spark.sql.AnalysisException
 
printStackTrace(PrintWriter) - Static method in exception org.apache.spark.sql.AnalysisException
 
printStats() - Method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
printTreeString() - Method in class org.apache.spark.sql.types.StructType
 
prioritize(BlockManagerId, Seq<BlockManagerId>, HashSet<BlockManagerId>, BlockId, int) - Method in class org.apache.spark.storage.BasicBlockReplicationPolicy
Method to prioritize a bunch of candidate peers of a block manager.
prioritize(BlockManagerId, Seq<BlockManagerId>, HashSet<BlockManagerId>, BlockId, int) - Method in interface org.apache.spark.storage.BlockReplicationPolicy
Method to prioritize a bunch of candidate peers of a block
prioritize(BlockManagerId, Seq<BlockManagerId>, HashSet<BlockManagerId>, BlockId, int) - Method in class org.apache.spark.storage.RandomBlockReplicationPolicy
Method to prioritize a bunch of candidate peers of a block.
prob() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.PredictData
 
prob() - Method in class org.apache.spark.mllib.tree.model.Predict
 
ProbabilisticClassificationModel<FeaturesType,M extends ProbabilisticClassificationModel<FeaturesType,M>> - Class in org.apache.spark.ml.classification
Developer API
ProbabilisticClassificationModel() - Constructor for class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
ProbabilisticClassifier<FeaturesType,E extends ProbabilisticClassifier<FeaturesType,E,M>,M extends ProbabilisticClassificationModel<FeaturesType,M>> - Class in org.apache.spark.ml.classification
Developer API
ProbabilisticClassifier() - Constructor for class org.apache.spark.ml.classification.ProbabilisticClassifier
 
probabilities() - Static method in class org.apache.spark.scheduler.StatsReportListener
 
probability() - Method in class org.apache.spark.ml.clustering.GaussianMixtureSummary
Probability of each cluster.
probabilityCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
probabilityCol() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Field in "predictions" which gives the probability of each class as a vector.
probabilityCol() - Method in class org.apache.spark.ml.classification.LogisticRegressionSummaryImpl
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
probabilityCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
probabilityCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
probabilityCol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
probabilityCol() - Method in class org.apache.spark.ml.clustering.GaussianMixtureSummary
 
probabilityCol() - Method in interface org.apache.spark.ml.param.shared.HasProbabilityCol
Param for Column name for predicted class conditional probabilities.
Probit$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Probit$
 
process(T) - Method in class org.apache.spark.sql.ForeachWriter
Called to process the data in the executor side.
PROCESS_LOCAL() - Static method in class org.apache.spark.scheduler.TaskLocality
 
processAllAvailable() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Blocks until all available data in the source has been processed and committed to the sink.
processedRowsPerSecond() - Method in class org.apache.spark.sql.streaming.SourceProgress
 
processedRowsPerSecond() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
The aggregate (across all sources) rate at which Spark is processing data.
processingDelay() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
Time taken for the all jobs of this batch to finish processing from the time they started processing.
processingEndTime() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
 
processingStartTime() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
 
ProcessingTime - Class in org.apache.spark.sql.streaming
Deprecated.
use Trigger.ProcessingTime(intervalMs). Since 2.2.0.
ProcessingTime(long) - Constructor for class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
ProcessingTime(long) - Static method in class org.apache.spark.sql.streaming.Trigger
A trigger policy that runs a query periodically based on an interval in processing time.
ProcessingTime(long, TimeUnit) - Static method in class org.apache.spark.sql.streaming.Trigger
(Java-friendly) A trigger policy that runs a query periodically based on an interval in processing time.
ProcessingTime(Duration) - Static method in class org.apache.spark.sql.streaming.Trigger
(Scala-friendly) A trigger policy that runs a query periodically based on an interval in processing time.
ProcessingTime(String) - Static method in class org.apache.spark.sql.streaming.Trigger
A trigger policy that runs a query periodically based on an interval in processing time.
processingTime() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
ProcessingTimeTimeout() - Static method in class org.apache.spark.sql.streaming.GroupStateTimeout
Timeout based on processing time.
processStreamByLine(String, InputStream, Function1<String, BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Return and start a daemon thread that processes the content of the input stream line by line.
producedAttributes() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
producedAttributes() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
producedAttributes() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
producedAttributes() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
product() - Method in class org.apache.spark.mllib.recommendation.Rating
 
product(TypeTags.TypeTag<T>) - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's product type (tuples, case classes, etc).
product(Numeric<B>) - Static method in class org.apache.spark.sql.types.StructType
 
productArity() - Static method in class org.apache.spark.Aggregator
 
productArity() - Static method in class org.apache.spark.CleanAccum
 
productArity() - Static method in class org.apache.spark.CleanBroadcast
 
productArity() - Static method in class org.apache.spark.CleanCheckpoint
 
productArity() - Static method in class org.apache.spark.CleanRDD
 
productArity() - Static method in class org.apache.spark.CleanShuffle
 
productArity() - Static method in class org.apache.spark.ExceptionFailure
 
productArity() - Static method in class org.apache.spark.ExecutorLostFailure
 
productArity() - Static method in class org.apache.spark.ExecutorRegistered
 
productArity() - Static method in class org.apache.spark.ExecutorRemoved
 
productArity() - Static method in class org.apache.spark.ExpireDeadHosts
 
productArity() - Static method in class org.apache.spark.FetchFailed
 
productArity() - Static method in class org.apache.spark.graphx.Edge
 
productArity() - Static method in class org.apache.spark.ml.feature.Dot
 
productArity() - Static method in class org.apache.spark.ml.feature.LabeledPoint
 
productArity() - Static method in class org.apache.spark.ml.param.ParamPair
 
productArity() - Static method in class org.apache.spark.mllib.feature.VocabWord
 
productArity() - Static method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
productArity() - Static method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
productArity() - Static method in class org.apache.spark.mllib.linalg.QRDecomposition
 
productArity() - Static method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
productArity() - Static method in class org.apache.spark.mllib.recommendation.Rating
 
productArity() - Static method in class org.apache.spark.mllib.regression.LabeledPoint
 
productArity() - Static method in class org.apache.spark.mllib.stat.test.BinarySample
 
productArity() - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
productArity() - Static method in class org.apache.spark.mllib.tree.model.Split
 
productArity() - Static method in class org.apache.spark.Resubmitted
 
productArity() - Static method in class org.apache.spark.rpc.netty.OnStart
 
productArity() - Static method in class org.apache.spark.rpc.netty.OnStop
 
productArity() - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
productArity() - Static method in class org.apache.spark.scheduler.AllJobsCancelled
 
productArity() - Static method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
productArity() - Static method in class org.apache.spark.scheduler.BlacklistedExecutor
 
productArity() - Static method in class org.apache.spark.scheduler.JobSucceeded
 
productArity() - Static method in class org.apache.spark.scheduler.local.KillTask
 
productArity() - Static method in class org.apache.spark.scheduler.local.ReviveOffers
 
productArity() - Static method in class org.apache.spark.scheduler.local.StatusUpdate
 
productArity() - Static method in class org.apache.spark.scheduler.local.StopExecutor
 
productArity() - Static method in class org.apache.spark.scheduler.ResubmitFailedStages
 
productArity() - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerJobStart
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerLogStart
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerStageCompleted
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
productArity() - Static method in class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
productArity() - Static method in class org.apache.spark.scheduler.StopCoordinator
 
productArity() - Static method in class org.apache.spark.sql.DatasetHolder
 
productArity() - Static method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
productArity() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
productArity() - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
productArity() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
productArity() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
productArity() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
productArity() - Static method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
productArity() - Static method in class org.apache.spark.sql.hive.RelationConversions
 
productArity() - Static method in class org.apache.spark.sql.jdbc.JdbcType
 
productArity() - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
productArity() - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
productArity() - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
productArity() - Static method in class org.apache.spark.sql.sources.And
 
productArity() - Static method in class org.apache.spark.sql.sources.EqualNullSafe
 
productArity() - Static method in class org.apache.spark.sql.sources.EqualTo
 
productArity() - Static method in class org.apache.spark.sql.sources.GreaterThan
 
productArity() - Static method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
productArity() - Static method in class org.apache.spark.sql.sources.In
 
productArity() - Static method in class org.apache.spark.sql.sources.IsNotNull
 
productArity() - Static method in class org.apache.spark.sql.sources.IsNull
 
productArity() - Static method in class org.apache.spark.sql.sources.LessThan
 
productArity() - Static method in class org.apache.spark.sql.sources.LessThanOrEqual
 
productArity() - Static method in class org.apache.spark.sql.sources.Not
 
productArity() - Static method in class org.apache.spark.sql.sources.Or
 
productArity() - Static method in class org.apache.spark.sql.sources.StringContains
 
productArity() - Static method in class org.apache.spark.sql.sources.StringEndsWith
 
productArity() - Static method in class org.apache.spark.sql.sources.StringStartsWith
 
productArity() - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
productArity() - Static method in class org.apache.spark.sql.types.ArrayType
 
productArity() - Static method in class org.apache.spark.sql.types.CharType
 
productArity() - Static method in class org.apache.spark.sql.types.DecimalType
 
productArity() - Static method in class org.apache.spark.sql.types.MapType
 
productArity() - Static method in class org.apache.spark.sql.types.ObjectType
 
productArity() - Static method in class org.apache.spark.sql.types.StructField
 
productArity() - Static method in class org.apache.spark.sql.types.StructType
 
productArity() - Static method in class org.apache.spark.sql.types.VarcharType
 
productArity() - Static method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
productArity() - Static method in class org.apache.spark.status.api.v1.ApplicationInfo
 
productArity() - Static method in class org.apache.spark.StopMapOutputTracker
 
productArity() - Static method in class org.apache.spark.storage.BlockStatus
 
productArity() - Static method in class org.apache.spark.storage.BlockUpdatedInfo
 
productArity() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
productArity() - Static method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
productArity() - Static method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
productArity() - Static method in class org.apache.spark.storage.RDDBlockId
 
productArity() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
productArity() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
productArity() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
productArity() - Static method in class org.apache.spark.storage.StreamBlockId
 
productArity() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
productArity() - Static method in class org.apache.spark.streaming.Duration
 
productArity() - Static method in class org.apache.spark.streaming.kinesis.DefaultCredentials
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.AllReceiverIds
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.BatchInfo
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.GetAllReceiverInfo
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StopAllReceivers
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
productArity() - Static method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
productArity() - Static method in class org.apache.spark.streaming.Time
 
productArity() - Static method in class org.apache.spark.Success
 
productArity() - Static method in class org.apache.spark.TaskCommitDenied
 
productArity() - Static method in class org.apache.spark.TaskKilled
 
productArity() - Static method in class org.apache.spark.TaskResultLost
 
productArity() - Static method in class org.apache.spark.TaskSchedulerIsSet
 
productArity() - Static method in class org.apache.spark.UnknownReason
 
productArity() - Static method in class org.apache.spark.util.MethodIdentifier
 
productArity() - Static method in class org.apache.spark.util.MutablePair
 
productElement(int) - Static method in class org.apache.spark.Aggregator
 
productElement(int) - Static method in class org.apache.spark.CleanAccum
 
productElement(int) - Static method in class org.apache.spark.CleanBroadcast
 
productElement(int) - Static method in class org.apache.spark.CleanCheckpoint
 
productElement(int) - Static method in class org.apache.spark.CleanRDD
 
productElement(int) - Static method in class org.apache.spark.CleanShuffle
 
productElement(int) - Static method in class org.apache.spark.ExceptionFailure
 
productElement(int) - Static method in class org.apache.spark.ExecutorLostFailure
 
productElement(int) - Static method in class org.apache.spark.ExecutorRegistered
 
productElement(int) - Static method in class org.apache.spark.ExecutorRemoved
 
productElement(int) - Static method in class org.apache.spark.ExpireDeadHosts
 
productElement(int) - Static method in class org.apache.spark.FetchFailed
 
productElement(int) - Static method in class org.apache.spark.graphx.Edge
 
productElement(int) - Static method in class org.apache.spark.ml.feature.Dot
 
productElement(int) - Static method in class org.apache.spark.ml.feature.LabeledPoint
 
productElement(int) - Static method in class org.apache.spark.ml.param.ParamPair
 
productElement(int) - Static method in class org.apache.spark.mllib.feature.VocabWord
 
productElement(int) - Static method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
productElement(int) - Static method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
productElement(int) - Static method in class org.apache.spark.mllib.linalg.QRDecomposition
 
productElement(int) - Static method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
productElement(int) - Static method in class org.apache.spark.mllib.recommendation.Rating
 
productElement(int) - Static method in class org.apache.spark.mllib.regression.LabeledPoint
 
productElement(int) - Static method in class org.apache.spark.mllib.stat.test.BinarySample
 
productElement(int) - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
productElement(int) - Static method in class org.apache.spark.mllib.tree.model.Split
 
productElement(int) - Static method in class org.apache.spark.Resubmitted
 
productElement(int) - Static method in class org.apache.spark.rpc.netty.OnStart
 
productElement(int) - Static method in class org.apache.spark.rpc.netty.OnStop
 
productElement(int) - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
productElement(int) - Static method in class org.apache.spark.scheduler.AllJobsCancelled
 
productElement(int) - Static method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
productElement(int) - Static method in class org.apache.spark.scheduler.BlacklistedExecutor
 
productElement(int) - Static method in class org.apache.spark.scheduler.JobSucceeded
 
productElement(int) - Static method in class org.apache.spark.scheduler.local.KillTask
 
productElement(int) - Static method in class org.apache.spark.scheduler.local.ReviveOffers
 
productElement(int) - Static method in class org.apache.spark.scheduler.local.StatusUpdate
 
productElement(int) - Static method in class org.apache.spark.scheduler.local.StopExecutor
 
productElement(int) - Static method in class org.apache.spark.scheduler.ResubmitFailedStages
 
productElement(int) - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerJobStart
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerLogStart
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerStageCompleted
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
productElement(int) - Static method in class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
productElement(int) - Static method in class org.apache.spark.scheduler.StopCoordinator
 
productElement(int) - Static method in class org.apache.spark.sql.DatasetHolder
 
productElement(int) - Static method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
productElement(int) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
productElement(int) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
productElement(int) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
productElement(int) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
productElement(int) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
productElement(int) - Static method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
productElement(int) - Static method in class org.apache.spark.sql.hive.RelationConversions
 
productElement(int) - Static method in class org.apache.spark.sql.jdbc.JdbcType
 
productElement(int) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
productElement(int) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
productElement(int) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
productElement(int) - Static method in class org.apache.spark.sql.sources.And
 
productElement(int) - Static method in class org.apache.spark.sql.sources.EqualNullSafe
 
productElement(int) - Static method in class org.apache.spark.sql.sources.EqualTo
 
productElement(int) - Static method in class org.apache.spark.sql.sources.GreaterThan
 
productElement(int) - Static method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
productElement(int) - Static method in class org.apache.spark.sql.sources.In
 
productElement(int) - Static method in class org.apache.spark.sql.sources.IsNotNull
 
productElement(int) - Static method in class org.apache.spark.sql.sources.IsNull
 
productElement(int) - Static method in class org.apache.spark.sql.sources.LessThan
 
productElement(int) - Static method in class org.apache.spark.sql.sources.LessThanOrEqual
 
productElement(int) - Static method in class org.apache.spark.sql.sources.Not
 
productElement(int) - Static method in class org.apache.spark.sql.sources.Or
 
productElement(int) - Static method in class org.apache.spark.sql.sources.StringContains
 
productElement(int) - Static method in class org.apache.spark.sql.sources.StringEndsWith
 
productElement(int) - Static method in class org.apache.spark.sql.sources.StringStartsWith
 
productElement(int) - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
productElement(int) - Static method in class org.apache.spark.sql.types.ArrayType
 
productElement(int) - Static method in class org.apache.spark.sql.types.CharType
 
productElement(int) - Static method in class org.apache.spark.sql.types.DecimalType
 
productElement(int) - Static method in class org.apache.spark.sql.types.MapType
 
productElement(int) - Static method in class org.apache.spark.sql.types.ObjectType
 
productElement(int) - Static method in class org.apache.spark.sql.types.StructField
 
productElement(int) - Static method in class org.apache.spark.sql.types.StructType
 
productElement(int) - Static method in class org.apache.spark.sql.types.VarcharType
 
productElement(int) - Static method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
productElement(int) - Static method in class org.apache.spark.status.api.v1.ApplicationInfo
 
productElement(int) - Static method in class org.apache.spark.StopMapOutputTracker
 
productElement(int) - Static method in class org.apache.spark.storage.BlockStatus
 
productElement(int) - Static method in class org.apache.spark.storage.BlockUpdatedInfo
 
productElement(int) - Static method in class org.apache.spark.storage.BroadcastBlockId
 
productElement(int) - Static method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
productElement(int) - Static method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
productElement(int) - Static method in class org.apache.spark.storage.RDDBlockId
 
productElement(int) - Static method in class org.apache.spark.storage.ShuffleBlockId
 
productElement(int) - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
productElement(int) - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
productElement(int) - Static method in class org.apache.spark.storage.StreamBlockId
 
productElement(int) - Static method in class org.apache.spark.storage.TaskResultBlockId
 
productElement(int) - Static method in class org.apache.spark.streaming.Duration
 
productElement(int) - Static method in class org.apache.spark.streaming.kinesis.DefaultCredentials
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.AllReceiverIds
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.BatchInfo
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.GetAllReceiverInfo
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StopAllReceivers
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
productElement(int) - Static method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
productElement(int) - Static method in class org.apache.spark.streaming.Time
 
productElement(int) - Static method in class org.apache.spark.Success
 
productElement(int) - Static method in class org.apache.spark.TaskCommitDenied
 
productElement(int) - Static method in class org.apache.spark.TaskKilled
 
productElement(int) - Static method in class org.apache.spark.TaskResultLost
 
productElement(int) - Static method in class org.apache.spark.TaskSchedulerIsSet
 
productElement(int) - Static method in class org.apache.spark.UnknownReason
 
productElement(int) - Static method in class org.apache.spark.util.MethodIdentifier
 
productElement(int) - Static method in class org.apache.spark.util.MutablePair
 
productFeatures() - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
 
productIterator() - Static method in class org.apache.spark.Aggregator
 
productIterator() - Static method in class org.apache.spark.CleanAccum
 
productIterator() - Static method in class org.apache.spark.CleanBroadcast
 
productIterator() - Static method in class org.apache.spark.CleanCheckpoint
 
productIterator() - Static method in class org.apache.spark.CleanRDD
 
productIterator() - Static method in class org.apache.spark.CleanShuffle
 
productIterator() - Static method in class org.apache.spark.ExceptionFailure
 
productIterator() - Static method in class org.apache.spark.ExecutorLostFailure
 
productIterator() - Static method in class org.apache.spark.ExecutorRegistered
 
productIterator() - Static method in class org.apache.spark.ExecutorRemoved
 
productIterator() - Static method in class org.apache.spark.ExpireDeadHosts
 
productIterator() - Static method in class org.apache.spark.FetchFailed
 
productIterator() - Static method in class org.apache.spark.graphx.Edge
 
productIterator() - Static method in class org.apache.spark.ml.feature.Dot
 
productIterator() - Static method in class org.apache.spark.ml.feature.LabeledPoint
 
productIterator() - Static method in class org.apache.spark.ml.param.ParamPair
 
productIterator() - Static method in class org.apache.spark.mllib.feature.VocabWord
 
productIterator() - Static method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
productIterator() - Static method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
productIterator() - Static method in class org.apache.spark.mllib.linalg.QRDecomposition
 
productIterator() - Static method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
productIterator() - Static method in class org.apache.spark.mllib.recommendation.Rating
 
productIterator() - Static method in class org.apache.spark.mllib.regression.LabeledPoint
 
productIterator() - Static method in class org.apache.spark.mllib.stat.test.BinarySample
 
productIterator() - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
productIterator() - Static method in class org.apache.spark.mllib.tree.model.Split
 
productIterator() - Static method in class org.apache.spark.Resubmitted
 
productIterator() - Static method in class org.apache.spark.rpc.netty.OnStart
 
productIterator() - Static method in class org.apache.spark.rpc.netty.OnStop
 
productIterator() - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
productIterator() - Static method in class org.apache.spark.scheduler.AllJobsCancelled
 
productIterator() - Static method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
productIterator() - Static method in class org.apache.spark.scheduler.BlacklistedExecutor
 
productIterator() - Static method in class org.apache.spark.scheduler.JobSucceeded
 
productIterator() - Static method in class org.apache.spark.scheduler.local.KillTask
 
productIterator() - Static method in class org.apache.spark.scheduler.local.ReviveOffers
 
productIterator() - Static method in class org.apache.spark.scheduler.local.StatusUpdate
 
productIterator() - Static method in class org.apache.spark.scheduler.local.StopExecutor
 
productIterator() - Static method in class org.apache.spark.scheduler.ResubmitFailedStages
 
productIterator() - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerJobStart
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerLogStart
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerStageCompleted
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
productIterator() - Static method in class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
productIterator() - Static method in class org.apache.spark.scheduler.StopCoordinator
 
productIterator() - Static method in class org.apache.spark.sql.DatasetHolder
 
productIterator() - Static method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
productIterator() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
productIterator() - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
productIterator() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
productIterator() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
productIterator() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
productIterator() - Static method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
productIterator() - Static method in class org.apache.spark.sql.hive.RelationConversions
 
productIterator() - Static method in class org.apache.spark.sql.jdbc.JdbcType
 
productIterator() - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
productIterator() - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
productIterator() - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
productIterator() - Static method in class org.apache.spark.sql.sources.And
 
productIterator() - Static method in class org.apache.spark.sql.sources.EqualNullSafe
 
productIterator() - Static method in class org.apache.spark.sql.sources.EqualTo
 
productIterator() - Static method in class org.apache.spark.sql.sources.GreaterThan
 
productIterator() - Static method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
productIterator() - Static method in class org.apache.spark.sql.sources.In
 
productIterator() - Static method in class org.apache.spark.sql.sources.IsNotNull
 
productIterator() - Static method in class org.apache.spark.sql.sources.IsNull
 
productIterator() - Static method in class org.apache.spark.sql.sources.LessThan
 
productIterator() - Static method in class org.apache.spark.sql.sources.LessThanOrEqual
 
productIterator() - Static method in class org.apache.spark.sql.sources.Not
 
productIterator() - Static method in class org.apache.spark.sql.sources.Or
 
productIterator() - Static method in class org.apache.spark.sql.sources.StringContains
 
productIterator() - Static method in class org.apache.spark.sql.sources.StringEndsWith
 
productIterator() - Static method in class org.apache.spark.sql.sources.StringStartsWith
 
productIterator() - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
productIterator() - Static method in class org.apache.spark.sql.types.ArrayType
 
productIterator() - Static method in class org.apache.spark.sql.types.CharType
 
productIterator() - Static method in class org.apache.spark.sql.types.DecimalType
 
productIterator() - Static method in class org.apache.spark.sql.types.MapType
 
productIterator() - Static method in class org.apache.spark.sql.types.ObjectType
 
productIterator() - Static method in class org.apache.spark.sql.types.StructField
 
productIterator() - Static method in class org.apache.spark.sql.types.StructType
 
productIterator() - Static method in class org.apache.spark.sql.types.VarcharType
 
productIterator() - Static method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
productIterator() - Static method in class org.apache.spark.status.api.v1.ApplicationInfo
 
productIterator() - Static method in class org.apache.spark.StopMapOutputTracker
 
productIterator() - Static method in class org.apache.spark.storage.BlockStatus
 
productIterator() - Static method in class org.apache.spark.storage.BlockUpdatedInfo
 
productIterator() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
productIterator() - Static method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
productIterator() - Static method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
productIterator() - Static method in class org.apache.spark.storage.RDDBlockId
 
productIterator() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
productIterator() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
productIterator() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
productIterator() - Static method in class org.apache.spark.storage.StreamBlockId
 
productIterator() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
productIterator() - Static method in class org.apache.spark.streaming.Duration
 
productIterator() - Static method in class org.apache.spark.streaming.kinesis.DefaultCredentials
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.AllReceiverIds
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.BatchInfo
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.GetAllReceiverInfo
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StopAllReceivers
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
productIterator() - Static method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
productIterator() - Static method in class org.apache.spark.streaming.Time
 
productIterator() - Static method in class org.apache.spark.Success
 
productIterator() - Static method in class org.apache.spark.TaskCommitDenied
 
productIterator() - Static method in class org.apache.spark.TaskKilled
 
productIterator() - Static method in class org.apache.spark.TaskResultLost
 
productIterator() - Static method in class org.apache.spark.TaskSchedulerIsSet
 
productIterator() - Static method in class org.apache.spark.UnknownReason
 
productIterator() - Static method in class org.apache.spark.util.MethodIdentifier
 
productIterator() - Static method in class org.apache.spark.util.MutablePair
 
productPrefix() - Static method in class org.apache.spark.Aggregator
 
productPrefix() - Static method in class org.apache.spark.CleanAccum
 
productPrefix() - Static method in class org.apache.spark.CleanBroadcast
 
productPrefix() - Static method in class org.apache.spark.CleanCheckpoint
 
productPrefix() - Static method in class org.apache.spark.CleanRDD
 
productPrefix() - Static method in class org.apache.spark.CleanShuffle
 
productPrefix() - Static method in class org.apache.spark.ExceptionFailure
 
productPrefix() - Static method in class org.apache.spark.ExecutorLostFailure
 
productPrefix() - Static method in class org.apache.spark.ExecutorRegistered
 
productPrefix() - Static method in class org.apache.spark.ExecutorRemoved
 
productPrefix() - Static method in class org.apache.spark.ExpireDeadHosts
 
productPrefix() - Static method in class org.apache.spark.FetchFailed
 
productPrefix() - Static method in class org.apache.spark.graphx.Edge
 
productPrefix() - Static method in class org.apache.spark.ml.feature.Dot
 
productPrefix() - Static method in class org.apache.spark.ml.feature.LabeledPoint
 
productPrefix() - Static method in class org.apache.spark.ml.param.ParamPair
 
productPrefix() - Static method in class org.apache.spark.mllib.feature.VocabWord
 
productPrefix() - Static method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
productPrefix() - Static method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
productPrefix() - Static method in class org.apache.spark.mllib.linalg.QRDecomposition
 
productPrefix() - Static method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
productPrefix() - Static method in class org.apache.spark.mllib.recommendation.Rating
 
productPrefix() - Static method in class org.apache.spark.mllib.regression.LabeledPoint
 
productPrefix() - Static method in class org.apache.spark.mllib.stat.test.BinarySample
 
productPrefix() - Static method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
productPrefix() - Static method in class org.apache.spark.mllib.tree.model.Split
 
productPrefix() - Static method in class org.apache.spark.Resubmitted
 
productPrefix() - Static method in class org.apache.spark.rpc.netty.OnStart
 
productPrefix() - Static method in class org.apache.spark.rpc.netty.OnStop
 
productPrefix() - Static method in class org.apache.spark.scheduler.AccumulableInfo
 
productPrefix() - Static method in class org.apache.spark.scheduler.AllJobsCancelled
 
productPrefix() - Static method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
productPrefix() - Static method in class org.apache.spark.scheduler.BlacklistedExecutor
 
productPrefix() - Static method in class org.apache.spark.scheduler.JobSucceeded
 
productPrefix() - Static method in class org.apache.spark.scheduler.local.KillTask
 
productPrefix() - Static method in class org.apache.spark.scheduler.local.ReviveOffers
 
productPrefix() - Static method in class org.apache.spark.scheduler.local.StatusUpdate
 
productPrefix() - Static method in class org.apache.spark.scheduler.local.StopExecutor
 
productPrefix() - Static method in class org.apache.spark.scheduler.ResubmitFailedStages
 
productPrefix() - Static method in class org.apache.spark.scheduler.RuntimePercentage
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerJobStart
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerLogStart
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerStageCompleted
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
productPrefix() - Static method in class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
productPrefix() - Static method in class org.apache.spark.scheduler.StopCoordinator
 
productPrefix() - Static method in class org.apache.spark.sql.DatasetHolder
 
productPrefix() - Static method in class org.apache.spark.sql.expressions.UserDefinedFunction
 
productPrefix() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
productPrefix() - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
productPrefix() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
productPrefix() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
productPrefix() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
productPrefix() - Static method in class org.apache.spark.sql.hive.HiveUDAFBuffer
 
productPrefix() - Static method in class org.apache.spark.sql.hive.RelationConversions
 
productPrefix() - Static method in class org.apache.spark.sql.jdbc.JdbcType
 
productPrefix() - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
productPrefix() - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
productPrefix() - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 
productPrefix() - Static method in class org.apache.spark.sql.sources.And
 
productPrefix() - Static method in class org.apache.spark.sql.sources.EqualNullSafe
 
productPrefix() - Static method in class org.apache.spark.sql.sources.EqualTo
 
productPrefix() - Static method in class org.apache.spark.sql.sources.GreaterThan
 
productPrefix() - Static method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
productPrefix() - Static method in class org.apache.spark.sql.sources.In
 
productPrefix() - Static method in class org.apache.spark.sql.sources.IsNotNull
 
productPrefix() - Static method in class org.apache.spark.sql.sources.IsNull
 
productPrefix() - Static method in class org.apache.spark.sql.sources.LessThan
 
productPrefix() - Static method in class org.apache.spark.sql.sources.LessThanOrEqual
 
productPrefix() - Static method in class org.apache.spark.sql.sources.Not
 
productPrefix() - Static method in class org.apache.spark.sql.sources.Or
 
productPrefix() - Static method in class org.apache.spark.sql.sources.StringContains
 
productPrefix() - Static method in class org.apache.spark.sql.sources.StringEndsWith
 
productPrefix() - Static method in class org.apache.spark.sql.sources.StringStartsWith
 
productPrefix() - Static method in class org.apache.spark.sql.streaming.ProcessingTime
Deprecated.
 
productPrefix() - Static method in class org.apache.spark.sql.types.ArrayType
 
productPrefix() - Static method in class org.apache.spark.sql.types.CharType
 
productPrefix() - Static method in class org.apache.spark.sql.types.DecimalType
 
productPrefix() - Static method in class org.apache.spark.sql.types.MapType
 
productPrefix() - Static method in class org.apache.spark.sql.types.ObjectType
 
productPrefix() - Static method in class org.apache.spark.sql.types.StructField
 
productPrefix() - Static method in class org.apache.spark.sql.types.StructType
 
productPrefix() - Static method in class org.apache.spark.sql.types.VarcharType
 
productPrefix() - Static method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
productPrefix() - Static method in class org.apache.spark.status.api.v1.ApplicationInfo
 
productPrefix() - Static method in class org.apache.spark.StopMapOutputTracker
 
productPrefix() - Static method in class org.apache.spark.storage.BlockStatus
 
productPrefix() - Static method in class org.apache.spark.storage.BlockUpdatedInfo
 
productPrefix() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
productPrefix() - Static method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
productPrefix() - Static method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
productPrefix() - Static method in class org.apache.spark.storage.RDDBlockId
 
productPrefix() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
productPrefix() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
productPrefix() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
productPrefix() - Static method in class org.apache.spark.storage.StreamBlockId
 
productPrefix() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
productPrefix() - Static method in class org.apache.spark.streaming.Duration
 
productPrefix() - Static method in class org.apache.spark.streaming.kinesis.DefaultCredentials
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.AllReceiverIds
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.BatchInfo
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.GetAllReceiverInfo
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StopAllReceivers
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
productPrefix() - Static method in class org.apache.spark.streaming.scheduler.StreamInputInfo
 
productPrefix() - Static method in class org.apache.spark.streaming.Time
 
productPrefix() - Static method in class org.apache.spark.Success
 
productPrefix() - Static method in class org.apache.spark.TaskCommitDenied
 
productPrefix() - Static method in class org.apache.spark.TaskKilled
 
productPrefix() - Static method in class org.apache.spark.TaskResultLost
 
productPrefix() - Static method in class org.apache.spark.TaskSchedulerIsSet
 
productPrefix() - Static method in class org.apache.spark.UnknownReason
 
productPrefix() - Static method in class org.apache.spark.util.MethodIdentifier
 
productPrefix() - Static method in class org.apache.spark.util.MutablePair
 
progress() - Method in class org.apache.spark.sql.streaming.StreamingQueryListener.QueryProgressEvent
 
project(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
 
project(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
properties() - Method in class org.apache.spark.scheduler.SparkListenerJobStart
 
properties() - Method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
propertiesFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
propertiesToJson(Properties) - Static method in class org.apache.spark.util.JsonProtocol
 
provider() - Static method in class org.apache.spark.streaming.kinesis.DefaultCredentials
 
proxyBase() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.AddWebUIFilter
 
pruneColumns(StructType) - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsPushDownRequiredColumns
Applies column pruning w.r.t.
PrunedFilteredScan - Interface in org.apache.spark.sql.sources
A BaseRelation that can eliminate unneeded columns and filter using selected predicates before producing an RDD containing all matching tuples as Row objects.
PrunedScan - Interface in org.apache.spark.sql.sources
A BaseRelation that can eliminate unneeded columns before producing an RDD containing all of its tuples as Row objects.
Pseudorandom - Interface in org.apache.spark.util.random
Developer API A class with pseudorandom behavior.
pushCatalystFilters(Expression[]) - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsPushDownCatalystFilters
Pushes down filters, and returns filters that need to be evaluated after scanning.
pushedCatalystFilters() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsPushDownCatalystFilters
Returns the catalyst filters that are pushed in SupportsPushDownCatalystFilters.pushCatalystFilters(Expression[]).
pushedFilters() - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters
Returns the filters that are pushed in SupportsPushDownFilters.pushFilters(Filter[]).
pushFilters(Filter[]) - Method in interface org.apache.spark.sql.sources.v2.reader.SupportsPushDownFilters
Pushes down filters, and returns filters that need to be evaluated after scanning.
put(ParamPair<?>...) - Method in class org.apache.spark.ml.param.ParamMap
Puts a list of param pairs (overwrites if the input params exists).
put(Param<T>, T) - Method in class org.apache.spark.ml.param.ParamMap
Puts a (param, value) pair (overwrites if the input param exists).
put(Seq<ParamPair<?>>) - Method in class org.apache.spark.ml.param.ParamMap
Puts a list of param pairs (overwrites if the input params exists).
put(Object) - Method in class org.apache.spark.util.sketch.BloomFilter
Puts an item into this BloomFilter.
putBinary(byte[]) - Method in class org.apache.spark.util.sketch.BloomFilter
A specialized variant of BloomFilter.put(Object) that only supports byte array items.
putBoolean(String, boolean) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Boolean.
putBooleanArray(String, boolean[]) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Boolean array.
putDouble(String, double) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Double.
putDoubleArray(String, double[]) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Double array.
putLong(String, long) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Long.
putLong(long) - Method in class org.apache.spark.util.sketch.BloomFilter
A specialized variant of BloomFilter.put(Object) that only supports long items.
putLongArray(String, long[]) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Long array.
putMetadata(String, Metadata) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Metadata.
putMetadataArray(String, Metadata[]) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a Metadata array.
putNull(String) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a null.
putString(String, String) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a String.
putString(String) - Method in class org.apache.spark.util.sketch.BloomFilter
A specialized variant of BloomFilter.put(Object) that only supports String items.
putStringArray(String, String[]) - Method in class org.apache.spark.sql.types.MetadataBuilder
Puts a String array.
pValue() - Method in class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
pValue() - Method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult
 
pValue() - Method in interface org.apache.spark.mllib.stat.test.TestResult
The probability of obtaining a test statistic result at least as extreme as the one that was actually observed, assuming that the null hypothesis is true.
pValues() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionTrainingSummary
Two-sided p-value of estimated coefficients and intercept.
pValues() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Two-sided p-value of estimated coefficients and intercept.
pyUDT() - Method in class org.apache.spark.mllib.linalg.VectorUDT
 

Q

Q() - Method in class org.apache.spark.mllib.linalg.QRDecomposition
 
QRDecomposition<QType,RType> - Class in org.apache.spark.mllib.linalg
Represents QR factors.
QRDecomposition(QType, RType) - Constructor for class org.apache.spark.mllib.linalg.QRDecomposition
 
quantileCalculationStrategy() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
QuantileDiscretizer - Class in org.apache.spark.ml.feature
QuantileDiscretizer takes a column with continuous features and outputs a column with binned categorical features.
QuantileDiscretizer(String) - Constructor for class org.apache.spark.ml.feature.QuantileDiscretizer
 
QuantileDiscretizer() - Constructor for class org.apache.spark.ml.feature.QuantileDiscretizer
 
quantileProbabilities() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
quantileProbabilities() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
quantiles() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
quantilesCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
quantilesCol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
QuantileStrategy - Class in org.apache.spark.mllib.tree.configuration
Enum for selecting the quantile calculation strategy
QuantileStrategy() - Constructor for class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
quarter(Column) - Static method in class org.apache.spark.sql.functions
Extracts the quarter as an integer from a given date/timestamp/string.
query() - Method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
query() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
query() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
queryExecution() - Method in class org.apache.spark.sql.Dataset
 
queryExecution() - Method in class org.apache.spark.sql.KeyValueGroupedDataset
 
QueryExecutionListener - Interface in org.apache.spark.sql.util
Experimental The interface of query execution listener that can be used to analyze execution metrics.
queryName(String) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Specifies the name of the StreamingQuery that can be started with start().
queueStream(Queue<JavaRDD<T>>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from a queue of RDDs.
queueStream(Queue<JavaRDD<T>>, boolean) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from a queue of RDDs.
queueStream(Queue<JavaRDD<T>>, boolean, JavaRDD<T>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from a queue of RDDs.
queueStream(Queue<RDD<T>>, boolean, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream from a queue of RDDs.
queueStream(Queue<RDD<T>>, boolean, RDD<T>, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream from a queue of RDDs.
quot(Decimal, Decimal) - Method in class org.apache.spark.sql.types.Decimal.DecimalAsIfIntegral$
 
quoteIdentifier(String) - Method in class org.apache.spark.sql.jdbc.AggregatedDialect
 
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.DB2Dialect
 
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.DerbyDialect
 
quoteIdentifier(String) - Method in class org.apache.spark.sql.jdbc.JdbcDialect
Quotes the identifier.
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.MsSqlServerDialect
 
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.MySQLDialect
 
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.NoopDialect
 
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.OracleDialect
 
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.PostgresDialect
 
quoteIdentifier(String) - Static method in class org.apache.spark.sql.jdbc.TeradataDialect
 

R

R() - Method in class org.apache.spark.mllib.linalg.QRDecomposition
 
r2() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Returns R^2^, the coefficient of determination.
r2() - Method in class org.apache.spark.mllib.evaluation.RegressionMetrics
Returns R^2^, the unadjusted coefficient of determination.
r2adj() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Returns Adjusted R^2^, the adjusted coefficient of determination.
RACK_LOCAL() - Static method in class org.apache.spark.scheduler.TaskLocality
 
radians(Column) - Static method in class org.apache.spark.sql.functions
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.
radians(String) - Static method in class org.apache.spark.sql.functions
Converts an angle measured in degrees to an approximately equivalent angle measured in radians.
rand(int, int, Random) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
Generate a DenseMatrix consisting of i.i.d. uniform random numbers.
rand(int, int, Random) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a DenseMatrix consisting of i.i.d. uniform random numbers.
rand(int, int, Random) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
Generate a DenseMatrix consisting of i.i.d. uniform random numbers.
rand(int, int, Random) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a DenseMatrix consisting of i.i.d. uniform random numbers.
rand(long) - Static method in class org.apache.spark.sql.functions
Generate a random column with independent and identically distributed (i.i.d.) samples from U[0.0, 1.0].
rand() - Static method in class org.apache.spark.sql.functions
Generate a random column with independent and identically distributed (i.i.d.) samples from U[0.0, 1.0].
randn(int, int, Random) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
Generate a DenseMatrix consisting of i.i.d. gaussian random numbers.
randn(int, int, Random) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a DenseMatrix consisting of i.i.d. gaussian random numbers.
randn(int, int, Random) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
Generate a DenseMatrix consisting of i.i.d. gaussian random numbers.
randn(int, int, Random) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a DenseMatrix consisting of i.i.d. gaussian random numbers.
randn(long) - Static method in class org.apache.spark.sql.functions
Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.
randn() - Static method in class org.apache.spark.sql.functions
Generate a column with independent and identically distributed (i.i.d.) samples from the standard normal distribution.
random() - Method in class org.apache.spark.ml.image.SamplePathFilter
 
RANDOM() - Static method in class org.apache.spark.mllib.clustering.KMeans
 
random() - Static method in class org.apache.spark.util.Utils
 
RandomBlockReplicationPolicy - Class in org.apache.spark.storage
 
RandomBlockReplicationPolicy() - Constructor for class org.apache.spark.storage.RandomBlockReplicationPolicy
 
RandomDataGenerator<T> - Interface in org.apache.spark.mllib.random
Developer API Trait for random data generators that generate i.i.d.
RandomForest - Class in org.apache.spark.ml.tree.impl
ALGORITHM
RandomForest() - Constructor for class org.apache.spark.ml.tree.impl.RandomForest
 
RandomForest - Class in org.apache.spark.mllib.tree
A class that implements a Random Forest learning algorithm for classification and regression.
RandomForest(Strategy, int, String, int) - Constructor for class org.apache.spark.mllib.tree.RandomForest
 
RandomForestClassificationModel - Class in org.apache.spark.ml.classification
Random Forest model for classification.
RandomForestClassifier - Class in org.apache.spark.ml.classification
Random Forest learning algorithm for classification.
RandomForestClassifier(String) - Constructor for class org.apache.spark.ml.classification.RandomForestClassifier
 
RandomForestClassifier() - Constructor for class org.apache.spark.ml.classification.RandomForestClassifier
 
RandomForestModel - Class in org.apache.spark.mllib.tree.model
Represents a random forest model.
RandomForestModel(Enumeration.Value, DecisionTreeModel[]) - Constructor for class org.apache.spark.mllib.tree.model.RandomForestModel
 
RandomForestRegressionModel - Class in org.apache.spark.ml.regression
Random Forest model for regression.
RandomForestRegressor - Class in org.apache.spark.ml.regression
Random Forest learning algorithm for regression.
RandomForestRegressor(String) - Constructor for class org.apache.spark.ml.regression.RandomForestRegressor
 
RandomForestRegressor() - Constructor for class org.apache.spark.ml.regression.RandomForestRegressor
 
randomize(TraversableOnce<T>, ClassTag<T>) - Static method in class org.apache.spark.util.Utils
Shuffle the elements of a collection into a random order, returning the result in a new collection.
randomizeInPlace(Object, Random) - Static method in class org.apache.spark.util.Utils
Shuffle the elements of an array into a random order, modifying the original array.
randomJavaRDD(JavaSparkContext, RandomDataGenerator<T>, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API Generates an RDD comprised of i.i.d. samples produced by the input RandomDataGenerator.
randomJavaRDD(JavaSparkContext, RandomDataGenerator<T>, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API RandomRDDs.randomJavaRDD with the default seed.
randomJavaRDD(JavaSparkContext, RandomDataGenerator<T>, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API RandomRDDs.randomJavaRDD with the default seed & numPartitions
randomJavaVectorRDD(JavaSparkContext, RandomDataGenerator<Object>, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API Java-friendly version of RandomRDDs.randomVectorRDD.
randomJavaVectorRDD(JavaSparkContext, RandomDataGenerator<Object>, long, int, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API RandomRDDs.randomJavaVectorRDD with the default seed.
randomJavaVectorRDD(JavaSparkContext, RandomDataGenerator<Object>, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API RandomRDDs.randomJavaVectorRDD with the default number of partitions and the default seed.
randomRDD(SparkContext, RandomDataGenerator<T>, long, int, long, ClassTag<T>) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API Generates an RDD comprised of i.i.d. samples produced by the input RandomDataGenerator.
RandomRDDs - Class in org.apache.spark.mllib.random
Generator methods for creating RDDs comprised of i.i.d. samples from some distribution.
RandomRDDs() - Constructor for class org.apache.spark.mllib.random.RandomRDDs
 
RandomSampler<T,U> - Interface in org.apache.spark.util.random
Developer API A pseudorandom sampler.
randomSplit(double[]) - Method in class org.apache.spark.api.java.JavaRDD
Randomly splits this RDD with the provided weights.
randomSplit(double[], long) - Method in class org.apache.spark.api.java.JavaRDD
Randomly splits this RDD with the provided weights.
randomSplit(double[], long) - Static method in class org.apache.spark.api.r.RRDD
 
randomSplit(double[], long) - Static method in class org.apache.spark.graphx.EdgeRDD
 
randomSplit(double[], long) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
randomSplit(double[], long) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
randomSplit(double[], long) - Static method in class org.apache.spark.graphx.VertexRDD
 
randomSplit(double[], long) - Static method in class org.apache.spark.rdd.HadoopRDD
 
randomSplit(double[], long) - Static method in class org.apache.spark.rdd.JdbcRDD
 
randomSplit(double[], long) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
randomSplit(double[], long) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
randomSplit(double[], long) - Method in class org.apache.spark.rdd.RDD
Randomly splits this RDD with the provided weights.
randomSplit(double[], long) - Static method in class org.apache.spark.rdd.UnionRDD
 
randomSplit(double[], long) - Method in class org.apache.spark.sql.Dataset
Randomly splits this Dataset with the provided weights.
randomSplit(double[]) - Method in class org.apache.spark.sql.Dataset
Randomly splits this Dataset with the provided weights.
randomSplit$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
randomSplit$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
randomSplit$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
randomSplit$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
randomSplit$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
randomSplit$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
randomSplit$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
randomSplit$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
randomSplit$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
randomSplit$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
randomSplitAsList(double[], long) - Method in class org.apache.spark.sql.Dataset
Returns a Java list that contains randomly split Dataset with the provided weights.
randomVectorRDD(SparkContext, RandomDataGenerator<Object>, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Developer API Generates an RDD[Vector] with vectors containing i.i.d. samples produced by the input RandomDataGenerator.
RandomVertexCut$() - Constructor for class org.apache.spark.graphx.PartitionStrategy.RandomVertexCut$
 
range(long, long, long, int) - Method in class org.apache.spark.SparkContext
Creates a new RDD[Long] containing elements from start to end(exclusive), increased by step every element.
range(long) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a Dataset with a single LongType column named id, containing elements in a range from 0 to end (exclusive) with step value 1.
range(long, long) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with step value 1.
range(long, long, long) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value.
range(long, long, long, int) - Method in class org.apache.spark.sql.SparkSession
Experimental Creates a Dataset with a single LongType column named id, containing elements in a range from start to end (exclusive) with a step value, with partition number specified.
range(long) - Method in class org.apache.spark.sql.SQLContext
 
range(long, long) - Method in class org.apache.spark.sql.SQLContext
 
range(long, long, long) - Method in class org.apache.spark.sql.SQLContext
 
range(long, long, long, int) - Method in class org.apache.spark.sql.SQLContext
 
rangeBetween(long, long) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the frame boundaries defined, from start (inclusive) to end (inclusive).
rangeBetween(Column, Column) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the frame boundaries defined, from start (inclusive) to end (inclusive).
rangeBetween(long, long) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the frame boundaries, from start (inclusive) to end (inclusive).
rangeBetween(Column, Column) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the frame boundaries, from start (inclusive) to end (inclusive).
RangeDependency<T> - Class in org.apache.spark
Developer API Represents a one-to-one dependency between ranges of partitions in the parent and child RDDs.
RangeDependency(RDD<T>, int, int, int) - Constructor for class org.apache.spark.RangeDependency
 
RangePartitioner<K,V> - Class in org.apache.spark
A Partitioner that partitions sortable records by range into roughly equal ranges.
RangePartitioner(int, RDD<? extends Product2<K, V>>, boolean, int, Ordering<K>, ClassTag<K>) - Constructor for class org.apache.spark.RangePartitioner
 
RangePartitioner(int, RDD<? extends Product2<K, V>>, boolean, Ordering<K>, ClassTag<K>) - Constructor for class org.apache.spark.RangePartitioner
 
rank() - Method in class org.apache.spark.graphx.lib.SVDPlusPlus.Conf
 
rank() - Static method in class org.apache.spark.ml.recommendation.ALS
 
rank() - Method in class org.apache.spark.ml.recommendation.ALSModel
 
rank() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
The numeric rank of the fitted linear model.
rank() - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
 
rank() - Static method in class org.apache.spark.sql.functions
Window function: returns the rank of rows within a window partition.
RankingMetrics<T> - Class in org.apache.spark.mllib.evaluation
Evaluator for ranking algorithms.
RankingMetrics(RDD<Tuple2<Object, Object>>, ClassTag<T>) - Constructor for class org.apache.spark.mllib.evaluation.RankingMetrics
 
Rating(ID, ID, float) - Constructor for class org.apache.spark.ml.recommendation.ALS.Rating
 
rating() - Method in class org.apache.spark.ml.recommendation.ALS.Rating
 
Rating - Class in org.apache.spark.mllib.recommendation
A more compact class to represent a rating than Tuple3[Int, Int, Double].
Rating(int, int, double) - Constructor for class org.apache.spark.mllib.recommendation.Rating
 
rating() - Method in class org.apache.spark.mllib.recommendation.Rating
 
Rating$() - Constructor for class org.apache.spark.ml.recommendation.ALS.Rating$
 
RatingBlock$() - Constructor for class org.apache.spark.ml.recommendation.ALS.RatingBlock$
 
ratingCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
ratioParam() - Static method in class org.apache.spark.ml.image.SamplePathFilter
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
rawPredictionCol() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
rawPredictionCol() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
rawPredictionCol() - Method in interface org.apache.spark.ml.param.shared.HasRawPredictionCol
Param for raw prediction (a.k.a.
rawSocketStream(String, int, StorageLevel) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them.
rawSocketStream(String, int) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them.
rawSocketStream(String, int, StorageLevel, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream from network source hostname:port, where data is received as serialized blocks (serialized using the Spark's serializer) that can be directly pushed into the block manager without deserializing them.
RawTextHelper - Class in org.apache.spark.streaming.util
 
RawTextHelper() - Constructor for class org.apache.spark.streaming.util.RawTextHelper
 
RawTextSender - Class in org.apache.spark.streaming.util
A helper program that sends blocks of Kryo-serialized text strings out on a socket at a specified rate.
RawTextSender() - Constructor for class org.apache.spark.streaming.util.RawTextSender
 
RBackendAuthHandler - Class in org.apache.spark.api.r
Authentication handler for connections from the R process.
RBackendAuthHandler(String) - Constructor for class org.apache.spark.api.r.RBackendAuthHandler
 
rdd() - Method in class org.apache.spark.api.java.JavaDoubleRDD
 
rdd() - Method in class org.apache.spark.api.java.JavaPairRDD
 
rdd() - Method in class org.apache.spark.api.java.JavaRDD
 
rdd() - Method in interface org.apache.spark.api.java.JavaRDDLike
 
RDD() - Static method in class org.apache.spark.api.r.RRunnerModes
 
rdd() - Method in class org.apache.spark.Dependency
 
rdd() - Method in class org.apache.spark.NarrowDependency
 
RDD<T> - Class in org.apache.spark.rdd
A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
RDD(SparkContext, Seq<Dependency<?>>, ClassTag<T>) - Constructor for class org.apache.spark.rdd.RDD
 
RDD(RDD<?>, ClassTag<T>) - Constructor for class org.apache.spark.rdd.RDD
Construct an RDD with just a one-to-one dependency on one parent
rdd() - Method in class org.apache.spark.ShuffleDependency
 
rdd() - Method in class org.apache.spark.sql.Dataset
Represents the content of the Dataset as an RDD of T.
RDD() - Static method in class org.apache.spark.storage.BlockId
 
RDDBlockId - Class in org.apache.spark.storage
 
RDDBlockId(int, int) - Constructor for class org.apache.spark.storage.RDDBlockId
 
rddBlocks() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
rddBlocks() - Method in class org.apache.spark.status.LiveExecutor
 
rddBlocks() - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the RDD blocks stored in this block manager.
rddBlocksById(int) - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the blocks that belong to the given RDD stored in this block manager.
RDDDataDistribution - Class in org.apache.spark.status.api.v1
 
RDDFunctions<T> - Class in org.apache.spark.mllib.rdd
Developer API Machine learning specific RDD functions.
RDDFunctions(RDD<T>, ClassTag<T>) - Constructor for class org.apache.spark.mllib.rdd.RDDFunctions
 
rddId() - Method in class org.apache.spark.CleanCheckpoint
 
rddId() - Method in class org.apache.spark.CleanRDD
 
rddId() - Method in class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
rddId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveRdd
 
rddId() - Method in class org.apache.spark.storage.RDDBlockId
 
rddIds() - Method in class org.apache.spark.status.api.v1.StageData
 
RDDInfo - Class in org.apache.spark.storage
 
RDDInfo(int, String, int, StorageLevel, Seq<Object>, String, Option<org.apache.spark.rdd.RDDOperationScope>) - Constructor for class org.apache.spark.storage.RDDInfo
 
rddInfoFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
rddInfos() - Method in class org.apache.spark.scheduler.StageInfo
 
rddInfoToJson(RDDInfo) - Static method in class org.apache.spark.util.JsonProtocol
 
RDDPartitionInfo - Class in org.apache.spark.status.api.v1
 
RDDPartitionSeq - Class in org.apache.spark.status
A custom sequence of partitions based on a mutable linked list.
RDDPartitionSeq() - Constructor for class org.apache.spark.status.RDDPartitionSeq
 
rdds() - Method in class org.apache.spark.rdd.CoGroupedRDD
 
rdds() - Method in class org.apache.spark.rdd.UnionRDD
 
RDDStorageInfo - Class in org.apache.spark.status.api.v1
 
rddStorageLevel(int) - Method in class org.apache.spark.storage.StorageStatus
Deprecated.
Return the storage level, if any, used by the given RDD in this block manager.
rddToAsyncRDDActions(RDD<T>, ClassTag<T>) - Static method in class org.apache.spark.rdd.RDD
 
rddToDatasetHolder(RDD<T>, Encoder<T>) - Method in class org.apache.spark.sql.SQLImplicits
Creates a Dataset from an RDD.
rddToOrderedRDDFunctions(RDD<Tuple2<K, V>>, Ordering<K>, ClassTag<K>, ClassTag<V>) - Static method in class org.apache.spark.rdd.RDD
 
rddToPairRDDFunctions(RDD<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>, Ordering<K>) - Static method in class org.apache.spark.rdd.RDD
 
rddToSequenceFileRDDFunctions(RDD<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>, <any>, <any>) - Static method in class org.apache.spark.rdd.RDD
 
read() - Method in class org.apache.spark.io.NioBufferedFileInputStream
 
read(byte[], int, int) - Method in class org.apache.spark.io.NioBufferedFileInputStream
 
read() - Method in class org.apache.spark.io.ReadAheadInputStream
 
read(byte[], int, int) - Method in class org.apache.spark.io.ReadAheadInputStream
 
read() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
read() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
read() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
read() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
read() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
read() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
read() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
read() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
read() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
read() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
read() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
read() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
read() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
read() - Static method in class org.apache.spark.ml.clustering.LDA
 
read() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
read() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
read() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
read() - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
read() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
read() - Static method in class org.apache.spark.ml.feature.IDFModel
 
read() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
read() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
read() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
read() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
read() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
read() - Static method in class org.apache.spark.ml.feature.PCAModel
 
read() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
read() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
read() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
read() - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
read() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
read() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
read() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
read() - Static method in class org.apache.spark.ml.Pipeline
 
read() - Static method in class org.apache.spark.ml.PipelineModel
 
read() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
read() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
read() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
read() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
read() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
read() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
read() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
read() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
read() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
read() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
read() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
read() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
read() - Method in interface org.apache.spark.ml.util.DefaultParamsReadable
 
read() - Method in interface org.apache.spark.ml.util.MLReadable
Returns an MLReader instance for this class.
read(Kryo, Input, Class<Iterable<?>>) - Method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
read() - Method in class org.apache.spark.sql.SparkSession
Returns a DataFrameReader that can be used to read non-streaming data in as a DataFrame.
read() - Method in class org.apache.spark.sql.SQLContext
 
read() - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
read(byte[]) - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
read(byte[], int, int) - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
read(String) - Static method in class org.apache.spark.streaming.CheckpointReader
Read checkpoint files present in the given checkpoint directory.
read(String, SparkConf, Configuration, boolean) - Static method in class org.apache.spark.streaming.CheckpointReader
Read checkpoint files present in the given checkpoint directory.
read(WriteAheadLogRecordHandle) - Method in class org.apache.spark.streaming.util.WriteAheadLog
Read a written record based on the given record handle.
ReadableChannelFileRegion - Class in org.apache.spark.storage
 
ReadableChannelFileRegion(ReadableByteChannel, long) - Constructor for class org.apache.spark.storage.ReadableChannelFileRegion
 
ReadAheadInputStream - Class in org.apache.spark.io
InputStream implementation which asynchronously reads ahead from the underlying input stream when specified amount of data has been read from the current buffer.
ReadAheadInputStream(InputStream, int, int) - Constructor for class org.apache.spark.io.ReadAheadInputStream
Creates a ReadAheadInputStream with the specified buffer size and read-ahead threshold
readAll() - Method in class org.apache.spark.streaming.util.WriteAheadLog
Read and return an iterator of all the records that have been written but not yet cleaned up.
readArray(DataInputStream, JVMObjectTracker) - Static method in class org.apache.spark.api.r.SerDe
 
readBoolean(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readBooleanArr(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readBytes(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readBytes() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
readBytesArr(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readDate(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readDouble(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readDoubleArr(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readExternal(ObjectInput) - Method in class org.apache.spark.serializer.JavaSerializer
 
readExternal(ObjectInput) - Method in class org.apache.spark.storage.BlockManagerId
 
readExternal(ObjectInput) - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
readExternal(ObjectInput) - Method in class org.apache.spark.storage.StorageLevel
 
readFrom(ConfigReader) - Method in class org.apache.spark.internal.config.ConfigEntryWithDefault
 
readFrom(ConfigReader) - Method in class org.apache.spark.internal.config.ConfigEntryWithDefaultFunction
 
readFrom(ConfigReader) - Method in class org.apache.spark.internal.config.ConfigEntryWithDefaultString
 
readFrom(InputStream) - Static method in class org.apache.spark.util.sketch.BloomFilter
Reads in a BloomFilter from an input stream.
readFrom(InputStream) - Static method in class org.apache.spark.util.sketch.CountMinSketch
Reads in a CountMinSketch from an input stream.
readFrom(byte[]) - Static method in class org.apache.spark.util.sketch.CountMinSketch
Reads in a CountMinSketch from a byte array.
readImages(String) - Static method in class org.apache.spark.ml.image.ImageSchema
Read the directory of images from the local or remote source
readImages(String, SparkSession, boolean, int, boolean, double, long) - Static method in class org.apache.spark.ml.image.ImageSchema
Read the directory of images from the local or remote source
readInt(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readIntArr(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readKey(ClassTag<T>) - Method in class org.apache.spark.serializer.DeserializationStream
Reads the object representing the key of a key-value pair.
readList(DataInputStream, JVMObjectTracker) - Static method in class org.apache.spark.api.r.SerDe
 
readMap(DataInputStream, JVMObjectTracker) - Static method in class org.apache.spark.api.r.SerDe
 
readObject(DataInputStream, JVMObjectTracker) - Static method in class org.apache.spark.api.r.SerDe
 
readObject(ClassTag<T>) - Method in class org.apache.spark.serializer.DeserializationStream
The most general-purpose method to read an object.
readObjectType(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readRecords() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
readSchema(Seq<String>, Option<Configuration>, boolean) - Static method in class org.apache.spark.sql.hive.orc.OrcFileOperator
 
readSchema() - Method in interface org.apache.spark.sql.sources.v2.reader.DataSourceReader
Returns the actual schema of this data source reader, which may be different from the physical schema of the underlying storage, as column pruning or other optimizations may happen.
readSqlObject(DataInputStream, char) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
readStream() - Method in class org.apache.spark.sql.SparkSession
Returns a DataStreamReader that can be used to read streaming data in as a DataFrame.
readStream() - Method in class org.apache.spark.sql.SQLContext
 
readString(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readStringArr(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readStringBytes(DataInputStream, int) - Static method in class org.apache.spark.api.r.SerDe
 
ReadSupport - Interface in org.apache.spark.sql.sources.v2
A mix-in interface for DataSourceV2.
ReadSupportWithSchema - Interface in org.apache.spark.sql.sources.v2
A mix-in interface for DataSourceV2.
readTime(DataInputStream) - Static method in class org.apache.spark.api.r.SerDe
 
readTypedObject(DataInputStream, char, JVMObjectTracker) - Static method in class org.apache.spark.api.r.SerDe
 
readValue(ClassTag<T>) - Method in class org.apache.spark.serializer.DeserializationStream
Reads the object representing the value of a key-value pair.
ready(Duration, CanAwait) - Method in class org.apache.spark.ComplexFutureAction
 
ready(Duration, CanAwait) - Method in interface org.apache.spark.FutureAction
Blocks until this action completes.
ready(Duration, CanAwait) - Method in class org.apache.spark.SimpleFutureAction
 
reason() - Method in class org.apache.spark.ExecutorLostFailure
 
reason() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask
 
reason() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
 
reason() - Method in class org.apache.spark.scheduler.local.KillTask
 
reason() - Method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
reason() - Method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
reason() - Method in class org.apache.spark.TaskKilled
 
reason() - Method in exception org.apache.spark.TaskKilledException
 
Recall - Class in org.apache.spark.mllib.evaluation.binary
Recall.
Recall() - Constructor for class org.apache.spark.mllib.evaluation.binary.Recall
 
recall(double) - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns recall for a given label (category)
recall() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Deprecated.
Use accuracy. Since 2.0.0.
recall() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns document-based recall averaged by the number of documents
recall(double) - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns recall for a given label (category)
recallByLabel() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns recall for each label (category).
recallByThreshold() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
Returns a dataframe with two fields (threshold, recall) curve.
recallByThreshold() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Returns the (threshold, recall) curve.
Receiver<T> - Class in org.apache.spark.streaming.receiver
Developer API Abstract class of a receiver that can be run on worker nodes to receive external data.
Receiver(StorageLevel) - Constructor for class org.apache.spark.streaming.receiver.Receiver
 
RECEIVER_WAL_CLASS_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
RECEIVER_WAL_CLOSE_AFTER_WRITE_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
RECEIVER_WAL_ENABLE_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
RECEIVER_WAL_MAX_FAILURES_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
RECEIVER_WAL_ROLLING_INTERVAL_CONF_KEY() - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
ReceiverInfo - Class in org.apache.spark.status.api.v1.streaming
 
ReceiverInfo - Class in org.apache.spark.streaming.scheduler
Developer API Class having information about a receiver
ReceiverInfo(int, String, boolean, String, String, String, String, long) - Constructor for class org.apache.spark.streaming.scheduler.ReceiverInfo
 
receiverInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
receiverInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
receiverInfo() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
receiverInputDStream() - Method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
receiverInputDStream() - Method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
ReceiverInputDStream<T> - Class in org.apache.spark.streaming.dstream
Abstract class for defining any InputDStream that has to start a receiver on worker nodes to receive external data.
ReceiverInputDStream(StreamingContext, ClassTag<T>) - Constructor for class org.apache.spark.streaming.dstream.ReceiverInputDStream
 
ReceiverState - Class in org.apache.spark.streaming.scheduler
Enumeration to identify current state of a Receiver
ReceiverState() - Constructor for class org.apache.spark.streaming.scheduler.ReceiverState
 
receiverStream(Receiver<T>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream with any arbitrary user implemented receiver.
receiverStream(Receiver<T>, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream with any arbitrary user implemented receiver.
recentProgress() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns an array of the most recent StreamingQueryProgress updates for this query.
recommendForAllItems(int) - Method in class org.apache.spark.ml.recommendation.ALSModel
Returns top numUsers users recommended for each item, for all items.
recommendForAllUsers(int) - Method in class org.apache.spark.ml.recommendation.ALSModel
Returns top numItems items recommended for each user, for all users.
recommendForItemSubset(Dataset<?>, int) - Method in class org.apache.spark.ml.recommendation.ALSModel
Returns top numUsers users recommended for each item id in the input data set.
recommendForUserSubset(Dataset<?>, int) - Method in class org.apache.spark.ml.recommendation.ALSModel
Returns top numItems items recommended for each user id in the input data set.
recommendProducts(int, int) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Recommends products to a user.
recommendProductsForUsers(int) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Recommends top products for all users.
recommendUsers(int, int) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Recommends users to a product.
recommendUsersForProducts(int) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Recommends top users for all products.
recordReader(InputStream, Configuration) - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
recordReaderClass() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
RECORDS_BETWEEN_BYTES_READ_METRIC_UPDATES() - Static method in class org.apache.spark.rdd.HadoopRDD
Update the input bytes read metric each time this number of records has been read
RECORDS_READ() - Method in class org.apache.spark.InternalAccumulator.input$
 
RECORDS_READ() - Method in class org.apache.spark.InternalAccumulator.shuffleRead$
 
RECORDS_WRITTEN() - Method in class org.apache.spark.InternalAccumulator.output$
 
RECORDS_WRITTEN() - Method in class org.apache.spark.InternalAccumulator.shuffleWrite$
 
recordsRead() - Method in class org.apache.spark.status.api.v1.InputMetricDistributions
 
recordsRead() - Method in class org.apache.spark.status.api.v1.InputMetrics
 
recordsRead() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetrics
 
recordsWritten() - Method in class org.apache.spark.status.api.v1.OutputMetricDistributions
 
recordsWritten() - Method in class org.apache.spark.status.api.v1.OutputMetrics
 
recordsWritten() - Method in class org.apache.spark.status.api.v1.ShuffleWriteMetrics
 
recordWriter(OutputStream, Configuration) - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
recordWriterClass() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
recoverPartitions(String) - Method in class org.apache.spark.sql.catalog.Catalog
Recovers all the partitions in the directory of a table and update the catalog.
RecursiveFlag - Class in org.apache.spark.ml.image
 
RecursiveFlag() - Constructor for class org.apache.spark.ml.image.RecursiveFlag
 
recursiveList(File) - Static method in class org.apache.spark.util.Utils
Lists files recursively.
redact(SparkConf, Seq<Tuple2<String, String>>) - Static method in class org.apache.spark.util.Utils
Redact the sensitive values in the given map.
redact(Option<Regex>, Seq<Tuple2<String, String>>) - Static method in class org.apache.spark.util.Utils
Redact the sensitive values in the given map.
redact(Option<Regex>, String) - Static method in class org.apache.spark.util.Utils
Redact the sensitive information in the given string.
redact(Map<String, String>) - Static method in class org.apache.spark.util.Utils
Looks up the redaction regex from within the key value pairs and uses it to redact the rest of the key value pairs.
REDIRECT_CONNECTOR_NAME() - Static method in class org.apache.spark.ui.JettyUtils
 
redirectError() - Method in class org.apache.spark.launcher.SparkLauncher
Specifies that stderr in spark-submit should be redirected to stdout.
redirectError(ProcessBuilder.Redirect) - Method in class org.apache.spark.launcher.SparkLauncher
Redirects error output to the specified Redirect.
redirectError(File) - Method in class org.apache.spark.launcher.SparkLauncher
Redirects error output to the specified File.
redirectOutput(ProcessBuilder.Redirect) - Method in class org.apache.spark.launcher.SparkLauncher
Redirects standard output to the specified Redirect.
redirectOutput(File) - Method in class org.apache.spark.launcher.SparkLauncher
Redirects error output to the specified File.
redirectToLog(String) - Method in class org.apache.spark.launcher.SparkLauncher
Sets all output to be logged and redirected to a logger with the specified name.
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
reduce(Function2<T, T, T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Reduces the elements of this RDD using the specified commutative and associative binary operator.
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.api.r.RRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
reduce(Function2<T, T, T>) - Method in class org.apache.spark.rdd.RDD
Reduces the elements of this RDD using the specified commutative and associative binary operator.
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
reduce(Function2<T, T, T>) - Method in class org.apache.spark.sql.Dataset
Experimental (Scala-specific) Reduces the elements of this Dataset using the specified binary function.
reduce(ReduceFunction<T>) - Method in class org.apache.spark.sql.Dataset
Experimental (Java-specific) Reduces the elements of this Dataset using the specified binary function.
reduce(BUF, IN) - Method in class org.apache.spark.sql.expressions.Aggregator
Combine two values to produce a new value.
reduce(Function2<A1, A1, A1>) - Static method in class org.apache.spark.sql.types.StructType
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
reduce(Function2<T, T, T>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD has a single element generated by reducing each RDD of this DStream.
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduce(Function2<T, T, T>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
reduce(Function2<T, T, T>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD has a single element generated by reducing each RDD of this DStream.
reduceByKey(Partitioner, Function2<V, V, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Merge the values for each key using an associative and commutative reduce function.
reduceByKey(Function2<V, V, V>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Merge the values for each key using an associative and commutative reduce function.
reduceByKey(Function2<V, V, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Merge the values for each key using an associative and commutative reduce function.
reduceByKey(Partitioner, Function2<V, V, V>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Merge the values for each key using an associative and commutative reduce function.
reduceByKey(Function2<V, V, V>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Merge the values for each key using an associative and commutative reduce function.
reduceByKey(Function2<V, V, V>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Merge the values for each key using an associative and commutative reduce function.
reduceByKey(Function2<V, V, V>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying reduceByKey to each RDD.
reduceByKey(Function2<V, V, V>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying reduceByKey to each RDD.
reduceByKey(Function2<V, V, V>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying reduceByKey to each RDD.
reduceByKey(Function2<V, V, V>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKey(Function2<V, V, V>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKey(Function2<V, V, V>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKey(Function2<V, V, V>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKey(Function2<V, V, V>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKey(Function2<V, V, V>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKey(Function2<V, V, V>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying reduceByKey to each RDD.
reduceByKey(Function2<V, V, V>, int) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying reduceByKey to each RDD.
reduceByKey(Function2<V, V, V>, Partitioner) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying reduceByKey to each RDD.
reduceByKeyAndWindow(Function2<V, V, V>, Duration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Create a new DStream by applying reduceByKey over a sliding window on this DStream.
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by reducing over a using incremental computation.
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, int, Function<Tuple2<K, V>, Boolean>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying incremental reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, Partitioner, Function<Tuple2<K, V>, Boolean>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying incremental reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, int, Function<Tuple2<K, V>, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, Partitioner, Function<Tuple2<K, V>, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, int, Function<Tuple2<K, V>, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, Partitioner, Function<Tuple2<K, V>, Boolean>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByKeyAndWindow(Function2<V, V, V>, Duration) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying reduceByKey over a sliding window on this DStream.
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, int) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Duration, Duration, Partitioner) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, int, Function1<Tuple2<K, V>, Object>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying incremental reduceByKey over a sliding window.
reduceByKeyAndWindow(Function2<V, V, V>, Function2<V, V, V>, Duration, Duration, Partitioner, Function1<Tuple2<K, V>, Object>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying incremental reduceByKey over a sliding window.
reduceByKeyLocally(Function2<V, V, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Merge the values for each key using an associative and commutative reduce function, but return the result immediately to the master as a Map.
reduceByKeyLocally(Function2<V, V, V>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Merge the values for each key using an associative and commutative reduce function, but return the results immediately to the master as a Map.
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD has a single element generated by reducing all elements in a sliding window over this DStream.
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD has a single element generated by reducing all elements in a sliding window over this DStream.
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
reduceByWindow(Function2<T, T, T>, Duration, Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD has a single element generated by reducing all elements in a sliding window over this DStream.
reduceByWindow(Function2<T, T, T>, Function2<T, T, T>, Duration, Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD has a single element generated by reducing all elements in a sliding window over this DStream.
ReduceFunction<T> - Interface in org.apache.spark.api.java.function
Base interface for function used in Dataset's reduce.
reduceGroups(Function2<V, V, V>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Scala-specific) Reduces the elements of each group of data using the specified binary function.
reduceGroups(ReduceFunction<V>) - Method in class org.apache.spark.sql.KeyValueGroupedDataset
(Java-specific) Reduces the elements of each group of data using the specified binary function.
reduceId() - Method in class org.apache.spark.FetchFailed
 
reduceId() - Method in class org.apache.spark.storage.ShuffleBlockId
 
reduceId() - Method in class org.apache.spark.storage.ShuffleDataBlockId
 
reduceId() - Method in class org.apache.spark.storage.ShuffleIndexBlockId
 
reduceLeft(Function2<B, A, B>) - Static method in class org.apache.spark.sql.types.StructType
 
reduceLeftOption(Function2<B, A, B>) - Static method in class org.apache.spark.sql.types.StructType
 
reduceOption(Function2<A1, A1, A1>) - Static method in class org.apache.spark.sql.types.StructType
 
reduceRight(Function2<A, B, B>) - Static method in class org.apache.spark.sql.types.StructType
 
reduceRightOption(Function2<A, B, B>) - Static method in class org.apache.spark.sql.types.StructType
 
references() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
references() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
references() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
references() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
references() - Method in class org.apache.spark.sql.sources.And
 
references() - Method in class org.apache.spark.sql.sources.EqualNullSafe
 
references() - Method in class org.apache.spark.sql.sources.EqualTo
 
references() - Method in class org.apache.spark.sql.sources.Filter
List of columns that are referenced by this filter.
references() - Method in class org.apache.spark.sql.sources.GreaterThan
 
references() - Method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
references() - Method in class org.apache.spark.sql.sources.In
 
references() - Method in class org.apache.spark.sql.sources.IsNotNull
 
references() - Method in class org.apache.spark.sql.sources.IsNull
 
references() - Method in class org.apache.spark.sql.sources.LessThan
 
references() - Method in class org.apache.spark.sql.sources.LessThanOrEqual
 
references() - Method in class org.apache.spark.sql.sources.Not
 
references() - Method in class org.apache.spark.sql.sources.Or
 
references() - Method in class org.apache.spark.sql.sources.StringContains
 
references() - Method in class org.apache.spark.sql.sources.StringEndsWith
 
references() - Method in class org.apache.spark.sql.sources.StringStartsWith
 
refresh() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
refresh() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
refresh() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
refreshByPath(String) - Method in class org.apache.spark.sql.catalog.Catalog
Invalidates and refreshes all the cached data (and the associated metadata) for any Dataset that contains the given data source path.
refreshTable(String) - Method in class org.apache.spark.sql.catalog.Catalog
Invalidates and refreshes all the cached data and metadata of the given table.
refreshTable(String) - Method in class org.apache.spark.sql.hive.HiveContext
Deprecated.
Invalidate and refresh all the cached the metadata of the given table.
regex(Regex) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
regexFromString(String, String) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
regexp_extract(Column, String, int) - Static method in class org.apache.spark.sql.functions
Extract a specific group matched by a Java regex, from the specified string column.
regexp_replace(Column, String, String) - Static method in class org.apache.spark.sql.functions
Replace all substrings of the specified string value that match regexp with rep.
regexp_replace(Column, Column, Column) - Static method in class org.apache.spark.sql.functions
Replace all substrings of the specified string value that match regexp with rep.
RegexTokenizer - Class in org.apache.spark.ml.feature
A regex based tokenizer that extracts tokens either by using the provided regex pattern to split the text (default) or repeatedly matching the regex (if gaps is false).
RegexTokenizer(String) - Constructor for class org.apache.spark.ml.feature.RegexTokenizer
 
RegexTokenizer() - Constructor for class org.apache.spark.ml.feature.RegexTokenizer
 
register(AccumulatorV2<?, ?>) - Method in class org.apache.spark.SparkContext
Register the given accumulator.
register(AccumulatorV2<?, ?>, String) - Method in class org.apache.spark.SparkContext
Register the given accumulator with given name.
register(String, String) - Static method in class org.apache.spark.sql.types.UDTRegistration
Registers an UserDefinedType to an user class.
register(String, UserDefinedAggregateFunction) - Method in class org.apache.spark.sql.UDFRegistration
Registers a user-defined aggregate function (UDAF).
register(String, UserDefinedFunction) - Method in class org.apache.spark.sql.UDFRegistration
Registers a user-defined function (UDF), for a UDF that's already defined using the Dataset API (i.e.
register(String, Function0<RT>, TypeTags.TypeTag<RT>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 0 arguments as user-defined function (UDF).
register(String, Function1<A1, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 1 arguments as user-defined function (UDF).
register(String, Function2<A1, A2, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 2 arguments as user-defined function (UDF).
register(String, Function3<A1, A2, A3, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 3 arguments as user-defined function (UDF).
register(String, Function4<A1, A2, A3, A4, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 4 arguments as user-defined function (UDF).
register(String, Function5<A1, A2, A3, A4, A5, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 5 arguments as user-defined function (UDF).
register(String, Function6<A1, A2, A3, A4, A5, A6, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 6 arguments as user-defined function (UDF).
register(String, Function7<A1, A2, A3, A4, A5, A6, A7, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 7 arguments as user-defined function (UDF).
register(String, Function8<A1, A2, A3, A4, A5, A6, A7, A8, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 8 arguments as user-defined function (UDF).
register(String, Function9<A1, A2, A3, A4, A5, A6, A7, A8, A9, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 9 arguments as user-defined function (UDF).
register(String, Function10<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 10 arguments as user-defined function (UDF).
register(String, Function11<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 11 arguments as user-defined function (UDF).
register(String, Function12<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 12 arguments as user-defined function (UDF).
register(String, Function13<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 13 arguments as user-defined function (UDF).
register(String, Function14<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 14 arguments as user-defined function (UDF).
register(String, Function15<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 15 arguments as user-defined function (UDF).
register(String, Function16<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>, TypeTags.TypeTag<A16>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 16 arguments as user-defined function (UDF).
register(String, Function17<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>, TypeTags.TypeTag<A16>, TypeTags.TypeTag<A17>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 17 arguments as user-defined function (UDF).
register(String, Function18<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>, TypeTags.TypeTag<A16>, TypeTags.TypeTag<A17>, TypeTags.TypeTag<A18>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 18 arguments as user-defined function (UDF).
register(String, Function19<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>, TypeTags.TypeTag<A16>, TypeTags.TypeTag<A17>, TypeTags.TypeTag<A18>, TypeTags.TypeTag<A19>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 19 arguments as user-defined function (UDF).
register(String, Function20<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>, TypeTags.TypeTag<A16>, TypeTags.TypeTag<A17>, TypeTags.TypeTag<A18>, TypeTags.TypeTag<A19>, TypeTags.TypeTag<A20>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 20 arguments as user-defined function (UDF).
register(String, Function21<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, A21, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>, TypeTags.TypeTag<A16>, TypeTags.TypeTag<A17>, TypeTags.TypeTag<A18>, TypeTags.TypeTag<A19>, TypeTags.TypeTag<A20>, TypeTags.TypeTag<A21>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 21 arguments as user-defined function (UDF).
register(String, Function22<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, A11, A12, A13, A14, A15, A16, A17, A18, A19, A20, A21, A22, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>, TypeTags.TypeTag<A11>, TypeTags.TypeTag<A12>, TypeTags.TypeTag<A13>, TypeTags.TypeTag<A14>, TypeTags.TypeTag<A15>, TypeTags.TypeTag<A16>, TypeTags.TypeTag<A17>, TypeTags.TypeTag<A18>, TypeTags.TypeTag<A19>, TypeTags.TypeTag<A20>, TypeTags.TypeTag<A21>, TypeTags.TypeTag<A22>) - Method in class org.apache.spark.sql.UDFRegistration
Registers a deterministic Scala closure of 22 arguments as user-defined function (UDF).
register(String, UDF0<?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF0 instance as user-defined function (UDF).
register(String, UDF1<?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF1 instance as user-defined function (UDF).
register(String, UDF2<?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF2 instance as user-defined function (UDF).
register(String, UDF3<?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF3 instance as user-defined function (UDF).
register(String, UDF4<?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF4 instance as user-defined function (UDF).
register(String, UDF5<?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF5 instance as user-defined function (UDF).
register(String, UDF6<?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF6 instance as user-defined function (UDF).
register(String, UDF7<?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF7 instance as user-defined function (UDF).
register(String, UDF8<?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF8 instance as user-defined function (UDF).
register(String, UDF9<?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF9 instance as user-defined function (UDF).
register(String, UDF10<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF10 instance as user-defined function (UDF).
register(String, UDF11<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF11 instance as user-defined function (UDF).
register(String, UDF12<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF12 instance as user-defined function (UDF).
register(String, UDF13<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF13 instance as user-defined function (UDF).
register(String, UDF14<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF14 instance as user-defined function (UDF).
register(String, UDF15<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF15 instance as user-defined function (UDF).
register(String, UDF16<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF16 instance as user-defined function (UDF).
register(String, UDF17<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF17 instance as user-defined function (UDF).
register(String, UDF18<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF18 instance as user-defined function (UDF).
register(String, UDF19<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF19 instance as user-defined function (UDF).
register(String, UDF20<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF20 instance as user-defined function (UDF).
register(String, UDF21<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF21 instance as user-defined function (UDF).
register(String, UDF22<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Method in class org.apache.spark.sql.UDFRegistration
Register a deterministic Java UDF22 instance as user-defined function (UDF).
register(QueryExecutionListener) - Method in class org.apache.spark.sql.util.ExecutionListenerManager
Registers the specified QueryExecutionListener.
register(AccumulatorV2<?, ?>) - Static method in class org.apache.spark.util.AccumulatorContext
Registers an AccumulatorV2 created on the driver such that it can be used on the executors.
register(String, Function0<Object>) - Static method in class org.apache.spark.util.SignalUtils
Adds an action to be run when a given signal is received by this process.
registerAvroSchemas(Seq<Schema>) - Method in class org.apache.spark.SparkConf
Use Kryo serialization and register the given set of Avro schemas so that the generic record serializer can decrease network IO
RegisterBlockManager(BlockManagerId, long, long, org.apache.spark.rpc.RpcEndpointRef) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager
 
RegisterBlockManager$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager$
 
registerClasses(Kryo) - Method in interface org.apache.spark.serializer.KryoRegistrator
 
RegisterClusterManager(org.apache.spark.rpc.RpcEndpointRef) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterClusterManager
 
RegisterClusterManager$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterClusterManager$
 
registerDialect(JdbcDialect) - Static method in class org.apache.spark.sql.jdbc.JdbcDialects
Register a dialect for use on all new matching jdbc org.apache.spark.sql.DataFrame.
RegisteredExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisteredExecutor$
 
RegisterExecutor(String, org.apache.spark.rpc.RpcEndpointRef, String, int, Map<String, String>) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor
 
RegisterExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutor$
 
RegisterExecutorFailed(String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutorFailed
 
RegisterExecutorFailed$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RegisterExecutorFailed$
 
registerKryoClasses(SparkConf) - Static method in class org.apache.spark.graphx.GraphXUtils
Registers classes that GraphX uses with Kryo.
registerKryoClasses(SparkContext) - Static method in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette
This method registers the class SquaredEuclideanSilhouette.ClusterStats for kryo serialization.
registerKryoClasses(Class<?>[]) - Method in class org.apache.spark.SparkConf
Use Kryo serialization and register the given set of classes with Kryo.
registerLogger(Logger) - Static method in class org.apache.spark.util.SignalUtils
Register a signal handler to log signals on UNIX-like systems.
registerShutdownDeleteDir(File) - Static method in class org.apache.spark.util.ShutdownHookManager
 
registerStream(DStream<BinarySample>) - Method in class org.apache.spark.mllib.stat.test.StreamingTest
Register a DStream of values for significance testing.
registerStream(JavaDStream<BinarySample>) - Method in class org.apache.spark.mllib.stat.test.StreamingTest
Register a JavaDStream of values for significance testing.
registerTempTable(String) - Method in class org.apache.spark.sql.Dataset
Deprecated.
Use createOrReplaceTempView(viewName) instead. Since 2.0.0.
regParam() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
regParam() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
regParam() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
regParam() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
regParam() - Method in interface org.apache.spark.ml.param.shared.HasRegParam
Param for regularization parameter (&gt;= 0).
regParam() - Static method in class org.apache.spark.ml.recommendation.ALS
 
regParam() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
regParam() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
regParam() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
regParam() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
Regression() - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
RegressionEvaluator - Class in org.apache.spark.ml.evaluation
Experimental Evaluator for regression, which expects two input columns: prediction and label.
RegressionEvaluator(String) - Constructor for class org.apache.spark.ml.evaluation.RegressionEvaluator
 
RegressionEvaluator() - Constructor for class org.apache.spark.ml.evaluation.RegressionEvaluator
 
RegressionMetrics - Class in org.apache.spark.mllib.evaluation
Evaluator for regression.
RegressionMetrics(RDD<Tuple2<Object, Object>>, boolean) - Constructor for class org.apache.spark.mllib.evaluation.RegressionMetrics
 
RegressionMetrics(RDD<Tuple2<Object, Object>>) - Constructor for class org.apache.spark.mllib.evaluation.RegressionMetrics
 
RegressionModel<FeaturesType,M extends RegressionModel<FeaturesType,M>> - Class in org.apache.spark.ml.regression
Developer API
RegressionModel() - Constructor for class org.apache.spark.ml.regression.RegressionModel
 
RegressionModel - Interface in org.apache.spark.mllib.regression
 
reindex() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
reindex() - Method in class org.apache.spark.graphx.VertexRDD
Construct a new VertexRDD that is indexed by only the visible vertices.
RelationalGroupedDataset - Class in org.apache.spark.sql
A set of methods for aggregations on a DataFrame, created by groupBy, cube or rollup (and also pivot).
RelationalGroupedDataset.CubeType$ - Class in org.apache.spark.sql
To indicate it's the CUBE
RelationalGroupedDataset.GroupByType$ - Class in org.apache.spark.sql
To indicate it's the GroupBy
RelationalGroupedDataset.PivotType$ - Class in org.apache.spark.sql
 
RelationalGroupedDataset.RollupType$ - Class in org.apache.spark.sql
To indicate it's the ROLLUP
RelationConversions - Class in org.apache.spark.sql.hive
Relation conversion from metastore relations to data source relations for better performance
RelationConversions(SQLConf, HiveSessionCatalog) - Constructor for class org.apache.spark.sql.hive.RelationConversions
 
RelationProvider - Interface in org.apache.spark.sql.sources
Implemented by objects that produce relations for a specific kind of data source.
relativeDirection(long) - Method in class org.apache.spark.graphx.Edge
Return the relative direction of the edge to the corresponding vertex.
relativeError() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
relativeError() - Method in class org.apache.spark.util.sketch.CountMinSketch
Returns the relative error (or eps) of this CountMinSketch.
release() - Method in class org.apache.spark.storage.EncryptedManagedBuffer
 
rem(Decimal, Decimal) - Method in class org.apache.spark.sql.types.Decimal.DecimalAsIfIntegral$
 
remainder(Decimal) - Method in class org.apache.spark.sql.types.Decimal
 
remember(Duration) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Sets each DStreams in this context to remember RDDs it generated in the last given duration.
remember(Duration) - Method in class org.apache.spark.streaming.StreamingContext
Set each DStream in this context to remember RDDs it generated in the last given duration.
REMOTE_BLOCKS_FETCHED() - Method in class org.apache.spark.InternalAccumulator.shuffleRead$
 
REMOTE_BYTES_READ() - Method in class org.apache.spark.InternalAccumulator.shuffleRead$
 
REMOTE_BYTES_READ_TO_DISK() - Method in class org.apache.spark.InternalAccumulator.shuffleRead$
 
remoteBlocksFetched() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
remoteBlocksFetched() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetrics
 
remoteBytesRead() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
remoteBytesRead() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetrics
 
remoteBytesReadToDisk() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
remoteBytesReadToDisk() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetrics
 
remove(Param<T>) - Method in class org.apache.spark.ml.param.ParamMap
Removes a key from this map and returns its value associated previously as an option.
remove(String) - Method in class org.apache.spark.SparkConf
Remove a parameter from the configuration
remove() - Method in interface org.apache.spark.sql.streaming.GroupState
Remove this state.
remove(String) - Method in class org.apache.spark.sql.types.MetadataBuilder
 
remove() - Method in class org.apache.spark.streaming.State
Remove the state if it exists.
remove(long) - Static method in class org.apache.spark.util.AccumulatorContext
Unregisters the AccumulatorV2 with the given ID, if any.
RemoveBlock(BlockId) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBlock
 
RemoveBlock$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBlock$
 
RemoveBroadcast(long, boolean) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast
 
RemoveBroadcast$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast$
 
removeDistribution(LiveExecutor) - Method in class org.apache.spark.status.LiveRDD
 
RemoveExecutor(String, org.apache.spark.scheduler.ExecutorLossReason) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor
 
RemoveExecutor(String) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveExecutor
 
RemoveExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveExecutor$
 
RemoveExecutor$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveExecutor$
 
removeFromDriver() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveBroadcast
 
removeListener(L) - Static method in class org.apache.spark.scheduler.AsyncEventQueue
 
removeListener(StreamingQueryListener) - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
removeListenerOnError(SparkListenerInterface) - Method in class org.apache.spark.scheduler.AsyncEventQueue
 
removeMapOutput(int, BlockManagerId) - Method in class org.apache.spark.ShuffleStatus
Remove the map output which was served by the specified block manager.
removeOutputsByFilter(Function1<BlockManagerId, Object>) - Method in class org.apache.spark.ShuffleStatus
Removes all shuffle outputs which satisfies the filter.
removeOutputsOnExecutor(String) - Method in class org.apache.spark.ShuffleStatus
Removes all map outputs associated with the specified executor.
removeOutputsOnHost(String) - Method in class org.apache.spark.ShuffleStatus
Removes all shuffle outputs associated with this host.
removePartition(String) - Method in class org.apache.spark.status.LiveRDD
 
removePartition(LiveRDDPartition) - Method in class org.apache.spark.status.RDDPartitionSeq
 
RemoveRdd(int) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveRdd
 
RemoveRdd$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveRdd$
 
removeReason() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
removeReason() - Method in class org.apache.spark.status.LiveExecutor
 
removeSelfEdges() - Method in class org.apache.spark.graphx.GraphOps
Remove self edges.
RemoveShuffle(int) - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveShuffle
 
RemoveShuffle$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.RemoveShuffle$
 
removeShutdownDeleteDir(File) - Static method in class org.apache.spark.util.ShutdownHookManager
 
removeShutdownHook(Object) - Static method in class org.apache.spark.util.ShutdownHookManager
Remove a previously installed shutdown hook.
removeSparkListener(SparkListenerInterface) - Method in class org.apache.spark.SparkContext
Developer API Deregister the listener from Spark's listener bus.
removeTime() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
removeTime() - Method in class org.apache.spark.status.LiveExecutor
 
RemoveWorker(String, String, String) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveWorker
 
RemoveWorker$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveWorker$
 
rep(Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
rep1(Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
rep1(Function0<Parsers.Parser<T>>, Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
rep1sep(Function0<Parsers.Parser<T>>, Function0<Parsers.Parser<Object>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
repartition(int) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a new RDD that has exactly numPartitions partitions.
repartition(int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a new RDD that has exactly numPartitions partitions.
repartition(int) - Method in class org.apache.spark.api.java.JavaRDD
Return a new RDD that has exactly numPartitions partitions.
repartition(int, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
repartition(int, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
repartition(int, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return a new RDD that has exactly numPartitions partitions.
repartition(int, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
repartition(int, Column...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions into numPartitions.
repartition(Column...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions.
repartition(int) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset that has exactly numPartitions partitions.
repartition(int, Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions into numPartitions.
repartition(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions.
repartition(int) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Return a new DStream with an increased or decreased level of parallelism.
repartition(int) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
repartition(int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream with an increased or decreased level of parallelism.
repartition(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
repartition(int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
repartition(int) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
repartition(int) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream with an increased or decreased level of parallelism.
repartition$default$2(int) - Static method in class org.apache.spark.api.r.RRDD
 
repartition$default$2(int) - Static method in class org.apache.spark.graphx.EdgeRDD
 
repartition$default$2(int) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
repartition$default$2(int) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
repartition$default$2(int) - Static method in class org.apache.spark.graphx.VertexRDD
 
repartition$default$2(int) - Static method in class org.apache.spark.rdd.HadoopRDD
 
repartition$default$2(int) - Static method in class org.apache.spark.rdd.JdbcRDD
 
repartition$default$2(int) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
repartition$default$2(int) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
repartition$default$2(int) - Static method in class org.apache.spark.rdd.UnionRDD
 
repartitionAndSortWithinPartitions(Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Repartition the RDD according to the given partitioner and, within each resulting partition, sort records by their keys.
repartitionAndSortWithinPartitions(Partitioner, Comparator<K>) - Method in class org.apache.spark.api.java.JavaPairRDD
Repartition the RDD according to the given partitioner and, within each resulting partition, sort records by their keys.
repartitionAndSortWithinPartitions(Partitioner) - Method in class org.apache.spark.rdd.OrderedRDDFunctions
Repartition the RDD according to the given partitioner and, within each resulting partition, sort records by their keys.
repartitionByRange(int, Column...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions into numPartitions.
repartitionByRange(Column...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions.
repartitionByRange(int, Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions into numPartitions.
repartitionByRange(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset partitioned by the given partitioning expressions, using spark.sql.shuffle.partitions as number of partitions.
repeat(Column, int) - Static method in class org.apache.spark.sql.functions
Repeats a string column n times, and returns it as a new string column.
replace(String, Map<T, T>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Replaces values matching keys in replacement map with the corresponding values.
replace(String[], Map<T, T>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
Replaces values matching keys in replacement map with the corresponding values.
replace(String, Map<T, T>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Replaces values matching keys in replacement map.
replace(Seq<String>, Map<T, T>) - Method in class org.apache.spark.sql.DataFrameNaFunctions
(Scala-specific) Replaces values matching keys in replacement map.
replaceCharType(DataType) - Static method in class org.apache.spark.sql.types.HiveStringType
 
replicas() - Method in class org.apache.spark.storage.BlockManagerMessages.ReplicateBlock
 
ReplicateBlock(BlockId, Seq<BlockManagerId>, int) - Constructor for class org.apache.spark.storage.BlockManagerMessages.ReplicateBlock
 
ReplicateBlock$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.ReplicateBlock$
 
replicatedVertexView() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
replication() - Method in class org.apache.spark.storage.StorageLevel
 
repN(int, Function0<Parsers.Parser<T>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
reportError(String, Throwable) - Method in class org.apache.spark.streaming.receiver.Receiver
Report exceptions in receiving data.
repr() - Static method in class org.apache.spark.sql.types.StructType
 
repsep(Function0<Parsers.Parser<T>>, Function0<Parsers.Parser<Object>>) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
requestedTotal() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors
 
RequestExecutors(int, int, Map<String, Object>, Set<String>) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors
 
requestExecutors(int) - Method in class org.apache.spark.SparkContext
Developer API Request an additional number of executors from the cluster manager.
RequestExecutors$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RequestExecutors$
 
requestTotalExecutors(int, int, Map<String, Object>) - Method in class org.apache.spark.SparkContext
Update the cluster manager on our scheduling needs.
requiredChildDistribution() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
requiredChildOrdering() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
res() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
reservoirSampleAndCount(Iterator<T>, int, long, ClassTag<T>) - Static method in class org.apache.spark.util.random.SamplingUtils
Reservoir sampling implementation that also returns the input size.
reset() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
Resets the values of all metrics to zero.
reset() - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
reset() - Method in class org.apache.spark.util.AccumulatorV2
Resets this accumulator, which is zero value.
reset() - Method in class org.apache.spark.util.CollectionAccumulator
 
reset() - Method in class org.apache.spark.util.DoubleAccumulator
 
reset() - Method in class org.apache.spark.util.LegacyAccumulatorWrapper
 
reset() - Method in class org.apache.spark.util.LongAccumulator
 
resetMetrics() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
resetTerminated() - Method in class org.apache.spark.sql.streaming.StreamingQueryManager
Forget about past terminated queries so that awaitAnyTermination() can be used again to wait for new terminations.
residualDegreeOfFreedom() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
The residual degrees of freedom.
residualDegreeOfFreedomNull() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
The residual degrees of freedom for the null model.
residuals() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
Get the default residuals (deviance residuals) of the fitted model.
residuals(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionSummary
Get the residuals of the fitted model by type.
residuals() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Residuals (label - predicted value)
resolve(StructType, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
resolve(Seq<String>, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
resolve(StructType, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
resolve(Seq<String>, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
resolve(StructType, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
resolve(Seq<String>, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
resolveChildren(Seq<String>, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
resolveChildren(Seq<String>, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
resolveChildren(Seq<String>, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
resolved() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
resolved() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
resolved() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
ResolveHiveSerdeTable - Class in org.apache.spark.sql.hive
Determine the database, serde/format and schema of the Hive serde table, according to the storage properties.
ResolveHiveSerdeTable(SparkSession) - Constructor for class org.apache.spark.sql.hive.ResolveHiveSerdeTable
 
resolveQuoted(String, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
resolveQuoted(String, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
resolveQuoted(String, Function2<String, String, Object>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
resolveURI(String) - Static method in class org.apache.spark.util.Utils
Return a well-formed URI for the file described by a user input string.
resolveURIs(String) - Static method in class org.apache.spark.util.Utils
Resolve a comma-separated list of paths.
responder() - Method in class org.apache.spark.ui.JettyUtils.ServletParams
 
responseFromBackup(String) - Static method in class org.apache.spark.util.Utils
Return true if the response message is sent from a backup Master on standby.
restart(String) - Method in class org.apache.spark.streaming.receiver.Receiver
Restart the receiver.
restart(String, Throwable) - Method in class org.apache.spark.streaming.receiver.Receiver
Restart the receiver.
restart(String, Throwable, int) - Method in class org.apache.spark.streaming.receiver.Receiver
Restart the receiver.
ResubmitFailedStages - Class in org.apache.spark.scheduler
 
ResubmitFailedStages() - Constructor for class org.apache.spark.scheduler.ResubmitFailedStages
 
Resubmitted - Class in org.apache.spark
Developer API A org.apache.spark.scheduler.ShuffleMapTask that completed successfully earlier, but we lost the executor before the stage completed.
Resubmitted() - Constructor for class org.apache.spark.Resubmitted
 
result(Duration, CanAwait) - Method in class org.apache.spark.ComplexFutureAction
 
result(Duration, CanAwait) - Method in interface org.apache.spark.FutureAction
Awaits and returns the result (of type T) of this action.
result(Duration, CanAwait) - Method in class org.apache.spark.SimpleFutureAction
 
RESULT_SERIALIZATION_TIME() - Static method in class org.apache.spark.InternalAccumulator
 
RESULT_SERIALIZATION_TIME() - Static method in class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
RESULT_SERIALIZATION_TIME() - Static method in class org.apache.spark.ui.ToolTips
 
RESULT_SIZE() - Static method in class org.apache.spark.InternalAccumulator
 
RESULT_SIZE() - Static method in class org.apache.spark.status.TaskIndexNames
 
resultFetchStart() - Method in class org.apache.spark.status.api.v1.TaskData
 
resultSerializationTime() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
resultSerializationTime() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
resultSetToObjectArray(ResultSet) - Static method in class org.apache.spark.rdd.JdbcRDD
 
resultSize() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
resultSize() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
retain() - Method in class org.apache.spark.storage.EncryptedManagedBuffer
 
RetrieveLastAllocatedExecutorId$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RetrieveLastAllocatedExecutorId$
 
RetrieveSparkAppConfig$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RetrieveSparkAppConfig$
 
retryWaitMs(SparkConf) - Static method in class org.apache.spark.util.RpcUtils
Returns the configured number of milliseconds to wait on each retry
ReturnStatementFinder - Class in org.apache.spark.util
 
ReturnStatementFinder() - Constructor for class org.apache.spark.util.ReturnStatementFinder
 
reverse() - Method in class org.apache.spark.graphx.EdgeDirection
Reverse the direction of an edge.
reverse() - Method in class org.apache.spark.graphx.EdgeRDD
Reverse all the edges in this RDD.
reverse() - Method in class org.apache.spark.graphx.Graph
Reverses all edges in the graph.
reverse() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
reverse() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
reverse(Column) - Static method in class org.apache.spark.sql.functions
Reverses the string column and returns it as a new string column.
reverse() - Static method in class org.apache.spark.sql.types.StructType
 
reverseIterator() - Static method in class org.apache.spark.sql.types.StructType
 
reverseMap(Function1<A, B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
reverseRoutingTables() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
reverseRoutingTables() - Method in class org.apache.spark.graphx.VertexRDD
Returns a new VertexRDD reflecting a reversal of all edge directions in the corresponding EdgeRDD.
ReviveOffers - Class in org.apache.spark.scheduler.local
 
ReviveOffers() - Constructor for class org.apache.spark.scheduler.local.ReviveOffers
 
ReviveOffers$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.ReviveOffers$
 
RFormula - Class in org.apache.spark.ml.feature
Experimental Implements the transforms required for fitting a dataset against an R model formula.
RFormula(String) - Constructor for class org.apache.spark.ml.feature.RFormula
 
RFormula() - Constructor for class org.apache.spark.ml.feature.RFormula
 
RFormulaModel - Class in org.apache.spark.ml.feature
Experimental Model fitted by RFormula.
RFormulaParser - Class in org.apache.spark.ml.feature
Limited implementation of R formula parsing.
RFormulaParser() - Constructor for class org.apache.spark.ml.feature.RFormulaParser
 
RidgeRegressionModel - Class in org.apache.spark.mllib.regression
Regression model trained using RidgeRegression.
RidgeRegressionModel(Vector, double) - Constructor for class org.apache.spark.mllib.regression.RidgeRegressionModel
 
RidgeRegressionWithSGD - Class in org.apache.spark.mllib.regression
Train a regression model with L2-regularization using Stochastic Gradient Descent.
RidgeRegressionWithSGD() - Constructor for class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
Deprecated.
Use ml.regression.LinearRegression with elasticNetParam = 0.0. Note the default regParam is 0.01 for RidgeRegressionWithSGD, but is 0.0 for LinearRegression. Since 2.0.0.
right() - Method in class org.apache.spark.sql.sources.And
 
right() - Method in class org.apache.spark.sql.sources.Or
 
rightCategories() - Method in class org.apache.spark.ml.tree.CategoricalSplit
Get sorted categories which split to the right
rightChild() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
rightChild() - Method in class org.apache.spark.ml.tree.InternalNode
 
rightChildIndex(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Return the index of the right child of this node.
rightImpurity() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
rightNode() - Method in class org.apache.spark.mllib.tree.model.Node
 
rightNodeId() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
rightOuterJoin(JavaPairRDD<K, W>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a right outer join of this and other.
rightOuterJoin(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a right outer join of this and other.
rightOuterJoin(JavaPairRDD<K, W>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Perform a right outer join of this and other.
rightOuterJoin(RDD<Tuple2<K, W>>, Partitioner) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a right outer join of this and other.
rightOuterJoin(RDD<Tuple2<K, W>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a right outer join of this and other.
rightOuterJoin(RDD<Tuple2<K, W>>, int) - Method in class org.apache.spark.rdd.PairRDDFunctions
Perform a right outer join of this and other.
rightOuterJoin(JavaPairDStream<K, W>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'right outer join' between RDDs of this DStream and other DStream.
rightOuterJoin(JavaPairDStream<K, W>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'right outer join' between RDDs of this DStream and other DStream.
rightOuterJoin(JavaPairDStream<K, W>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by applying 'right outer join' between RDDs of this DStream and other DStream.
rightOuterJoin(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
rightOuterJoin(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
rightOuterJoin(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
rightOuterJoin(JavaPairDStream<K, W>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
rightOuterJoin(JavaPairDStream<K, W>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
rightOuterJoin(JavaPairDStream<K, W>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
rightOuterJoin(DStream<Tuple2<K, W>>, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'right outer join' between RDDs of this DStream and other DStream.
rightOuterJoin(DStream<Tuple2<K, W>>, int, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'right outer join' between RDDs of this DStream and other DStream.
rightOuterJoin(DStream<Tuple2<K, W>>, Partitioner, ClassTag<W>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new DStream by applying 'right outer join' between RDDs of this DStream and other DStream.
rightPredict() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
rint(Column) - Static method in class org.apache.spark.sql.functions
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.
rint(String) - Static method in class org.apache.spark.sql.functions
Returns the double value that is closest in value to the argument and is equal to a mathematical integer.
rlike(String) - Method in class org.apache.spark.sql.Column
SQL RLIKE expression (LIKE with Regex).
RMATa() - Static method in class org.apache.spark.graphx.util.GraphGenerators
 
RMATb() - Static method in class org.apache.spark.graphx.util.GraphGenerators
 
RMATc() - Static method in class org.apache.spark.graphx.util.GraphGenerators
 
RMATd() - Static method in class org.apache.spark.graphx.util.GraphGenerators
 
rmatGraph(SparkContext, int, int) - Static method in class org.apache.spark.graphx.util.GraphGenerators
A random graph generator using the R-MAT model, proposed in "R-MAT: A Recursive Model for Graph Mining" by Chakrabarti et al.
rnd() - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
roc() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
Returns the receiver operating characteristic (ROC) curve, which is a Dataframe having two fields (FPR, TPR) with (0.0, 0.0) prepended and (1.0, 1.0) appended to it.
roc() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Returns the receiver operating characteristic (ROC) curve, which is an RDD of (false positive rate, true positive rate) with (0.0, 0.0) prepended and (1.0, 1.0) appended to it.
rollup(Column...) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them.
rollup(String, String...) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them.
rollup(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them.
rollup(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Create a multi-dimensional rollup for the current Dataset using the specified columns, so we can run aggregation on them.
RollupType$() - Constructor for class org.apache.spark.sql.RelationalGroupedDataset.RollupType$
 
rootMeanSquaredError() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
Returns the root mean squared error, which is defined as the square root of the mean squared error.
rootMeanSquaredError() - Method in class org.apache.spark.mllib.evaluation.RegressionMetrics
Returns the root mean squared error, which is defined as the square root of the mean squared error.
rootNode() - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
rootNode() - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
round(Column) - Static method in class org.apache.spark.sql.functions
Returns the value of the column e rounded to 0 decimal places with HALF_UP round mode.
round(Column, int) - Static method in class org.apache.spark.sql.functions
Round the value of e to scale decimal places with HALF_UP round mode if scale is greater than or equal to 0 or at integral part when scale is less than 0.
ROUND_CEILING() - Static method in class org.apache.spark.sql.types.Decimal
 
ROUND_FLOOR() - Static method in class org.apache.spark.sql.types.Decimal
 
ROUND_HALF_EVEN() - Static method in class org.apache.spark.sql.types.Decimal
 
ROUND_HALF_UP() - Static method in class org.apache.spark.sql.types.Decimal
 
ROW() - Static method in class org.apache.spark.api.r.SerializationFormats
 
Row - Interface in org.apache.spark.sql
Represents one row of output from a relational operator.
row_number() - Static method in class org.apache.spark.sql.functions
Window function: returns a sequential number starting at 1 within a window partition.
RowFactory - Class in org.apache.spark.sql
A factory class used to construct Row objects.
RowFactory() - Constructor for class org.apache.spark.sql.RowFactory
 
rowIndices() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
rowIndices() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
rowIter() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
rowIter() - Method in interface org.apache.spark.ml.linalg.Matrix
Returns an iterator of row vectors.
rowIter() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
rowIter() - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
 
rowIter() - Method in interface org.apache.spark.mllib.linalg.Matrix
Returns an iterator of row vectors.
rowIter() - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
 
rowIterator() - Method in class org.apache.spark.sql.vectorized.ColumnarBatch
Returns an iterator over the rows in this batch.
RowMatrix - Class in org.apache.spark.mllib.linalg.distributed
Represents a row-oriented distributed Matrix with no meaningful row indices.
RowMatrix(RDD<Vector>, long, int) - Constructor for class org.apache.spark.mllib.linalg.distributed.RowMatrix
 
RowMatrix(RDD<Vector>) - Constructor for class org.apache.spark.mllib.linalg.distributed.RowMatrix
Alternative constructor leaving matrix dimensions to be determined automatically.
rows() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
 
rows() - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
 
rowsBetween(long, long) - Static method in class org.apache.spark.sql.expressions.Window
Creates a WindowSpec with the frame boundaries defined, from start (inclusive) to end (inclusive).
rowsBetween(long, long) - Method in class org.apache.spark.sql.expressions.WindowSpec
Defines the frame boundaries, from start (inclusive) to end (inclusive).
rowsPerBlock() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
 
rPackages() - Static method in class org.apache.spark.api.r.RUtils
 
rpad(Column, int, String) - Static method in class org.apache.spark.sql.functions
Right-pad the string column with pad to a length of len.
RpcUtils - Class in org.apache.spark.util
 
RpcUtils() - Constructor for class org.apache.spark.util.RpcUtils
 
RRDD<T> - Class in org.apache.spark.api.r
An RDD that stores serialized R objects as Array[Byte].
RRDD(RDD<T>, byte[], String, String, byte[], Object[], ClassTag<T>) - Constructor for class org.apache.spark.api.r.RRDD
 
RRunnerModes - Class in org.apache.spark.api.r
 
RRunnerModes() - Constructor for class org.apache.spark.api.r.RRunnerModes
 
rtrim(Column) - Static method in class org.apache.spark.sql.functions
Trim the spaces from right end for the specified string value.
rtrim(Column, String) - Static method in class org.apache.spark.sql.functions
Trim the specified character string from right end for the specified string column.
ruleName() - Static method in class org.apache.spark.sql.hive.HiveAnalysis
 
ruleName() - Static method in class org.apache.spark.sql.hive.RelationConversions
 
run(Graph<VD, ED>, int, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.ConnectedComponents
Compute the connected component membership of each vertex and return a graph with the vertex value containing the lowest vertex id in the connected component containing that vertex.
run(Graph<VD, ED>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.ConnectedComponents
Compute the connected component membership of each vertex and return a graph with the vertex value containing the lowest vertex id in the connected component containing that vertex.
run(Graph<VD, ED>, int, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.LabelPropagation
Run static Label Propagation for detecting communities in networks.
run(Graph<VD, ED>, int, double, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.PageRank
Run PageRank for a fixed number of iterations returning a graph with vertex attributes containing the PageRank and edge attributes the normalized edge weight.
run(Graph<VD, ED>, Seq<Object>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.ShortestPaths
Computes shortest paths to the given set of landmark vertices.
run(Graph<VD, ED>, int, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.StronglyConnectedComponents
Compute the strongly connected component (SCC) of each vertex and return a graph with the vertex value containing the lowest vertex id in the SCC containing that vertex.
run(RDD<Edge<Object>>, SVDPlusPlus.Conf) - Static method in class org.apache.spark.graphx.lib.SVDPlusPlus
Implement SVD++ based on "Factorization Meets the Neighborhood: a Multifaceted Collaborative Filtering Model", available at here.
run(Graph<VD, ED>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.TriangleCount
 
run(RDD<LabeledPoint>, BoostingStrategy, long, String) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Method to train a gradient boosting model
run(RDD<LabeledPoint>, Strategy, int, String, long, Option<<any>>, Option<String>) - Static method in class org.apache.spark.ml.tree.impl.RandomForest
Train a random forest.
run(RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
Run Logistic Regression with the configured parameters on an input RDD of LabeledPoint entries.
run(RDD<LabeledPoint>, Vector) - Method in class org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
Run Logistic Regression with the configured parameters on an input RDD of LabeledPoint entries starting from the initial weights provided.
run(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
 
run(RDD<LabeledPoint>, Vector) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
 
run(RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.classification.NaiveBayes
Run the algorithm with the configured parameters on an input RDD of LabeledPoint entries.
run(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
 
run(RDD<LabeledPoint>, Vector) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
 
run(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Runs the bisecting k-means algorithm.
run(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Java-friendly version of run().
run(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Perform expectation maximization
run(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Java-friendly version of run()
run(RDD<Vector>) - Method in class org.apache.spark.mllib.clustering.KMeans
Train a K-means model on the given set of points; data should be cached for high performance, because this is an iterative algorithm.
run(RDD<Tuple2<Object, Vector>>) - Method in class org.apache.spark.mllib.clustering.LDA
Learn an LDA model using the given dataset.
run(JavaPairRDD<Long, Vector>) - Method in class org.apache.spark.mllib.clustering.LDA
Java-friendly version of run()
run(Graph<Object, Object>) - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering
Run the PIC algorithm on Graph.
run(RDD<Tuple3<Object, Object, Object>>) - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering
Run the PIC algorithm.
run(JavaRDD<Tuple3<Long, Long, Double>>) - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering
A Java-friendly version of PowerIterationClustering.run.
run(RDD<FPGrowth.FreqItemset<Item>>, ClassTag<Item>) - Method in class org.apache.spark.mllib.fpm.AssociationRules
Computes the association rules with confidence above minConfidence.
run(JavaRDD<FPGrowth.FreqItemset<Item>>) - Method in class org.apache.spark.mllib.fpm.AssociationRules
Java-friendly version of run.
run(RDD<Object>, ClassTag<Item>) - Method in class org.apache.spark.mllib.fpm.FPGrowth
Computes an FP-Growth model that contains frequent itemsets.
run(JavaRDD<Basket>) - Method in class org.apache.spark.mllib.fpm.FPGrowth
Java-friendly version of run.
run(RDD<Object[]>, ClassTag<Item>) - Method in class org.apache.spark.mllib.fpm.PrefixSpan
Finds the complete set of frequent sequential patterns in the input sequences of itemsets.
run(JavaRDD<Sequence>) - Method in class org.apache.spark.mllib.fpm.PrefixSpan
A Java-friendly version of run() that reads sequences from a JavaRDD and returns frequent sequences in a PrefixSpanModel.
run(RDD<Rating>) - Method in class org.apache.spark.mllib.recommendation.ALS
Run ALS with the configured parameters on an input RDD of Rating objects.
run(JavaRDD<Rating>) - Method in class org.apache.spark.mllib.recommendation.ALS
Java-friendly version of ALS.run.
run(RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
Run the algorithm with the configured parameters on an input RDD of LabeledPoint entries.
run(RDD<LabeledPoint>, Vector) - Method in class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
Run the algorithm with the configured parameters on an input RDD of LabeledPoint entries starting from the initial weights provided.
run(RDD<Tuple3<Object, Object, Object>>) - Method in class org.apache.spark.mllib.regression.IsotonicRegression
Run IsotonicRegression algorithm to obtain isotonic regression model.
run(JavaRDD<Tuple3<Double, Double, Double>>) - Method in class org.apache.spark.mllib.regression.IsotonicRegression
Run pool adjacent violators algorithm to obtain isotonic regression model.
run(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
 
run(RDD<LabeledPoint>, Vector) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
 
run(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
 
run(RDD<LabeledPoint>, Vector) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
 
run(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
 
run(RDD<LabeledPoint>, Vector) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
 
run(RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.tree.DecisionTree
Method to train a decision tree model over an RDD
run(RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.tree.GradientBoostedTrees
Method to train a gradient boosting model
run(JavaRDD<LabeledPoint>) - Method in class org.apache.spark.mllib.tree.GradientBoostedTrees
Java-friendly API for org.apache.spark.mllib.tree.GradientBoostedTrees.run.
run(RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.tree.RandomForest
Method to train a decision tree model over an RDD
run(SparkSession, SparkPlan) - Method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
run(SparkSession, SparkPlan) - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
run(SparkSession, SparkPlan) - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
Inserts all the rows in the table into Hive.
run() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread
 
run() - Method in class org.apache.spark.util.SparkShutdownHook
 
runApproximateJob(RDD<T>, Function2<TaskContext, Iterator<T>, U>, <any>, long) - Method in class org.apache.spark.SparkContext
Developer API Run a job that can return approximate results.
runId() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns the unique id of this run of the query.
runId() - Method in class org.apache.spark.sql.streaming.StreamingQueryListener.QueryStartedEvent
 
runId() - Method in class org.apache.spark.sql.streaming.StreamingQueryListener.QueryTerminatedEvent
 
runId() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
runInNewThread(String, boolean, Function0<T>) - Static method in class org.apache.spark.util.ThreadUtils
Run a piece of code in a new thread and return the result.
runJob(RDD<T>, Function2<TaskContext, Iterator<T>, U>, Seq<Object>, Function2<Object, U, BoxedUnit>, ClassTag<U>) - Method in class org.apache.spark.SparkContext
Run a function on a given set of partitions in an RDD and pass the results to the given handler function.
runJob(RDD<T>, Function2<TaskContext, Iterator<T>, U>, Seq<Object>, ClassTag<U>) - Method in class org.apache.spark.SparkContext
Run a function on a given set of partitions in an RDD and return the results as an array.
runJob(RDD<T>, Function1<Iterator<T>, U>, Seq<Object>, ClassTag<U>) - Method in class org.apache.spark.SparkContext
Run a function on a given set of partitions in an RDD and return the results as an array.
runJob(RDD<T>, Function2<TaskContext, Iterator<T>, U>, ClassTag<U>) - Method in class org.apache.spark.SparkContext
Run a job on all partitions in an RDD and return the results in an array.
runJob(RDD<T>, Function1<Iterator<T>, U>, ClassTag<U>) - Method in class org.apache.spark.SparkContext
Run a job on all partitions in an RDD and return the results in an array.
runJob(RDD<T>, Function2<TaskContext, Iterator<T>, U>, Function2<Object, U, BoxedUnit>, ClassTag<U>) - Method in class org.apache.spark.SparkContext
Run a job on all partitions in an RDD and pass the results to a handler function.
runJob(RDD<T>, Function1<Iterator<T>, U>, Function2<Object, U, BoxedUnit>, ClassTag<U>) - Method in class org.apache.spark.SparkContext
Run a job on all partitions in an RDD and pass the results to a handler function.
runLBFGS(RDD<Tuple2<Object, Vector>>, Gradient, Updater, int, double, int, double, Vector) - Static method in class org.apache.spark.mllib.optimization.LBFGS
Run Limited-memory BFGS (L-BFGS) in parallel.
runMiniBatchSGD(RDD<Tuple2<Object, Vector>>, Gradient, Updater, double, int, double, double, Vector, double) - Static method in class org.apache.spark.mllib.optimization.GradientDescent
Run stochastic gradient descent (SGD) in parallel using mini batches.
runMiniBatchSGD(RDD<Tuple2<Object, Vector>>, Gradient, Updater, double, int, double, double, Vector) - Static method in class org.apache.spark.mllib.optimization.GradientDescent
Alias of runMiniBatchSGD with convergenceTol set to default value of 0.001.
running() - Method in class org.apache.spark.scheduler.TaskInfo
 
RUNNING() - Static method in class org.apache.spark.TaskState
 
runParallelPersonalizedPageRank(Graph<VD, ED>, int, double, long[], ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.PageRank
Run Personalized PageRank for a fixed number of iterations, for a set of starting nodes in parallel.
runPreCanonicalized(Graph<VD, ED>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.TriangleCount
 
runtime() - Method in class org.apache.spark.status.api.v1.ApplicationEnvironmentInfo
 
RuntimeConfig - Class in org.apache.spark.sql
Runtime configuration interface for Spark.
RuntimeInfo - Class in org.apache.spark.status.api.v1
 
RuntimePercentage - Class in org.apache.spark.scheduler
 
RuntimePercentage(double, Option<Object>, double) - Constructor for class org.apache.spark.scheduler.RuntimePercentage
 
runUntilConvergence(Graph<VD, ED>, double, double, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.PageRank
Run a dynamic version of PageRank returning a graph with vertex attributes containing the PageRank and edge attributes containing the normalized edge weight.
runUntilConvergenceWithOptions(Graph<VD, ED>, double, double, Option<Object>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.PageRank
Run a dynamic version of PageRank returning a graph with vertex attributes containing the PageRank and edge attributes containing the normalized edge weight.
runWith(Function1<B, U>) - Static method in class org.apache.spark.sql.types.StructType
 
runWithOptions(Graph<VD, ED>, int, double, Option<Object>, ClassTag<VD>, ClassTag<ED>) - Static method in class org.apache.spark.graphx.lib.PageRank
Run PageRank for a fixed number of iterations returning a graph with vertex attributes containing the PageRank and edge attributes the normalized edge weight.
runWithValidation(RDD<LabeledPoint>, RDD<LabeledPoint>, BoostingStrategy, long, String) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Method to validate a gradient boosting model
runWithValidation(RDD<LabeledPoint>, RDD<LabeledPoint>) - Method in class org.apache.spark.mllib.tree.GradientBoostedTrees
Method to validate a gradient boosting model
runWithValidation(JavaRDD<LabeledPoint>, JavaRDD<LabeledPoint>) - Method in class org.apache.spark.mllib.tree.GradientBoostedTrees
Java-friendly API for org.apache.spark.mllib.tree.GradientBoostedTrees.runWithValidation.
RUtils - Class in org.apache.spark.api.r
 
RUtils() - Constructor for class org.apache.spark.api.r.RUtils
 
RWrappers - Class in org.apache.spark.ml.r
This is the Scala stub of SparkR read.ml.
RWrappers() - Constructor for class org.apache.spark.ml.r.RWrappers
 
RWrapperUtils - Class in org.apache.spark.ml.r
 
RWrapperUtils() - Constructor for class org.apache.spark.ml.r.RWrapperUtils
 

S

s() - Method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
sameElements(GenIterable<B>) - Static method in class org.apache.spark.sql.types.StructType
 
sameResult(PlanType) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
sameResult(PlanType) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
sameResult(PlanType) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
sameResult(PlanType) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
sameThread() - Static method in class org.apache.spark.util.ThreadUtils
An ExecutionContextExecutor that runs each task in the thread that invokes execute/submit.
sample(boolean, Double) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a sampled subset of this RDD.
sample(boolean, Double, long) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a sampled subset of this RDD.
sample(boolean, double) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a sampled subset of this RDD.
sample(boolean, double, long) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a sampled subset of this RDD.
sample(boolean, double) - Method in class org.apache.spark.api.java.JavaRDD
Return a sampled subset of this RDD with a random seed.
sample(boolean, double, long) - Method in class org.apache.spark.api.java.JavaRDD
Return a sampled subset of this RDD, with a user-supplied seed.
sample(boolean, double, long) - Static method in class org.apache.spark.api.r.RRDD
 
sample(boolean, double, long) - Static method in class org.apache.spark.graphx.EdgeRDD
 
sample(boolean, double, long) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
sample(boolean, double, long) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
sample(boolean, double, long) - Static method in class org.apache.spark.graphx.VertexRDD
 
sample(boolean, double, long) - Static method in class org.apache.spark.rdd.HadoopRDD
 
sample(boolean, double, long) - Static method in class org.apache.spark.rdd.JdbcRDD
 
sample(boolean, double, long) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
sample(boolean, double, long) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
sample(boolean, double, long) - Method in class org.apache.spark.rdd.RDD
Return a sampled subset of this RDD.
sample(boolean, double, long) - Static method in class org.apache.spark.rdd.UnionRDD
 
sample(double, long) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset by sampling a fraction of rows (without replacement), using a user-supplied seed.
sample(double) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset by sampling a fraction of rows (without replacement), using a random seed.
sample(boolean, double, long) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset by sampling a fraction of rows, using a user-supplied seed.
sample(boolean, double) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset by sampling a fraction of rows, using a random seed.
sample() - Method in class org.apache.spark.util.random.BernoulliCellSampler
 
sample() - Method in class org.apache.spark.util.random.BernoulliSampler
 
sample() - Method in class org.apache.spark.util.random.PoissonSampler
 
sample(Iterator<T>) - Method in class org.apache.spark.util.random.PoissonSampler
 
sample(Iterator<T>) - Method in interface org.apache.spark.util.random.RandomSampler
take a random sample
sample() - Method in interface org.apache.spark.util.random.RandomSampler
Whether to sample the next item or not.
sample$default$3() - Static method in class org.apache.spark.api.r.RRDD
 
sample$default$3() - Static method in class org.apache.spark.graphx.EdgeRDD
 
sample$default$3() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
sample$default$3() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
sample$default$3() - Static method in class org.apache.spark.graphx.VertexRDD
 
sample$default$3() - Static method in class org.apache.spark.rdd.HadoopRDD
 
sample$default$3() - Static method in class org.apache.spark.rdd.JdbcRDD
 
sample$default$3() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
sample$default$3() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
sample$default$3() - Static method in class org.apache.spark.rdd.UnionRDD
 
sampleBy(String, Map<T, Object>, long) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Returns a stratified sample without replacement based on the fraction given on each stratum.
sampleBy(String, Map<T, Double>, long) - Method in class org.apache.spark.sql.DataFrameStatFunctions
Returns a stratified sample without replacement based on the fraction given on each stratum.
sampleByKey(boolean, Map<K, Double>, long) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a subset of this RDD sampled by key (via stratified sampling).
sampleByKey(boolean, Map<K, Double>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a subset of this RDD sampled by key (via stratified sampling).
sampleByKey(boolean, Map<K, Object>, long) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return a subset of this RDD sampled by key (via stratified sampling).
sampleByKeyExact(boolean, Map<K, Double>, long) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a subset of this RDD sampled by key (via stratified sampling) containing exactly math.ceil(numItems * samplingRate) for each stratum (group of pairs with the same key).
sampleByKeyExact(boolean, Map<K, Double>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return a subset of this RDD sampled by key (via stratified sampling) containing exactly math.ceil(numItems * samplingRate) for each stratum (group of pairs with the same key).
sampleByKeyExact(boolean, Map<K, Object>, long) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return a subset of this RDD sampled by key (via stratified sampling) containing exactly math.ceil(numItems * samplingRate) for each stratum (group of pairs with the same key).
SamplePathFilter - Class in org.apache.spark.ml.image
Filter that allows loading a fraction of HDFS files.
SamplePathFilter() - Constructor for class org.apache.spark.ml.image.SamplePathFilter
 
samplePointsPerPartitionHint() - Method in class org.apache.spark.RangePartitioner
 
sampleRatio() - Method in class org.apache.spark.ml.image.SamplePathFilter
 
sampleStdev() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute the sample standard deviation of this RDD's elements (which corrects for bias in estimating the standard deviation by dividing by N-1 instead of N).
sampleStdev() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute the sample standard deviation of this RDD's elements (which corrects for bias in estimating the standard deviation by dividing by N-1 instead of N).
sampleStdev() - Method in class org.apache.spark.util.StatCounter
Return the sample standard deviation of the values, which corrects for bias in estimating the variance by dividing by N-1 instead of N.
sampleVariance() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute the sample variance of this RDD's elements (which corrects for bias in estimating the standard variance by dividing by N-1 instead of N).
sampleVariance() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute the sample variance of this RDD's elements (which corrects for bias in estimating the variance by dividing by N-1 instead of N).
sampleVariance() - Method in class org.apache.spark.util.StatCounter
Return the sample variance, which corrects for bias in estimating the variance by dividing by N-1 instead of N.
SamplingUtils - Class in org.apache.spark.util.random
 
SamplingUtils() - Constructor for class org.apache.spark.util.random.SamplingUtils
 
satisfy(Distribution) - Method in interface org.apache.spark.sql.sources.v2.reader.partitioning.Partitioning
Returns true if this partitioning can satisfy the given distribution, which means Spark does not need to shuffle the output data of this data source for some certain operations.
save(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
save(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
save(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
save(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
save(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
save(String) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
save(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
save(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
save(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
save(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
save(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
save(String) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
save(String) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
save(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
save(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
save(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
save(String) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
save(String) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
save(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
save(String) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
save(String) - Static method in class org.apache.spark.ml.clustering.KMeans
 
save(String) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
save(String) - Static method in class org.apache.spark.ml.clustering.LDA
 
save(String) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
save(String) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
save(String) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
save(String) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
save(String) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
save(String) - Static method in class org.apache.spark.ml.feature.Binarizer
 
save(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
save(String) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
save(String) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
save(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
save(String) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
save(String) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
save(String) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
save(String) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
save(String) - Static method in class org.apache.spark.ml.feature.DCT
 
save(String) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
save(String) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
save(String) - Static method in class org.apache.spark.ml.feature.HashingTF
 
save(String) - Static method in class org.apache.spark.ml.feature.IDF
 
save(String) - Static method in class org.apache.spark.ml.feature.IDFModel
 
save(String) - Static method in class org.apache.spark.ml.feature.Imputer
 
save(String) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
save(String) - Static method in class org.apache.spark.ml.feature.IndexToString
 
save(String) - Static method in class org.apache.spark.ml.feature.Interaction
 
save(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
save(String) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
save(String) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
save(String) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
save(String) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
save(String) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
save(String) - Static method in class org.apache.spark.ml.feature.NGram
 
save(String) - Static method in class org.apache.spark.ml.feature.Normalizer
 
save(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
save(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
save(String) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
save(String) - Static method in class org.apache.spark.ml.feature.PCA
 
save(String) - Static method in class org.apache.spark.ml.feature.PCAModel
 
save(String) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
save(String) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
save(String) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
save(String) - Static method in class org.apache.spark.ml.feature.RFormula
 
save(String) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
save(String) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
save(String) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
save(String) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
save(String) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
save(String) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
save(String) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
save(String) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
save(String) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
save(String) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
save(String) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
save(String) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
save(String) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
save(String) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
save(String) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
save(String) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
save(String) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
save(String) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
save(String) - Static method in class org.apache.spark.ml.Pipeline
 
save(String) - Static method in class org.apache.spark.ml.PipelineModel
 
save(String) - Static method in class org.apache.spark.ml.recommendation.ALS
 
save(String) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
save(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
save(String) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
save(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
save(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
save(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
save(String) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
save(String) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
save(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
save(String) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
save(String) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
save(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
save(String) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
save(String) - Method in interface org.apache.spark.ml.util.MLWritable
Saves this ML instance to the input path, a shortcut of write.save(path).
save(String) - Method in class org.apache.spark.ml.util.MLWriter
Saves the ML instances to the input path.
save(SparkContext, String, String, int, int, Vector, double, Option<Object>) - Method in class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$
Helper method for saving GLM classification model metadata and data.
save(SparkContext, String) - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
save(SparkContext, String, org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0.Data) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$
 
save(SparkContext, String, org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0.Data) - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.classification.SVMModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel
 
save(SparkContext, BisectingKMeansModel, String) - Method in class org.apache.spark.mllib.clustering.BisectingKMeansModel.SaveLoadV1_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.KMeansModel
 
save(SparkContext, KMeansModel, String) - Method in class org.apache.spark.mllib.clustering.KMeansModel.SaveLoadV1_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.clustering.PowerIterationClusteringModel
 
save(SparkContext, PowerIterationClusteringModel, String) - Method in class org.apache.spark.mllib.clustering.PowerIterationClusteringModel.SaveLoadV1_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.feature.ChiSqSelectorModel
 
save(SparkContext, ChiSqSelectorModel, String) - Method in class org.apache.spark.mllib.feature.ChiSqSelectorModel.SaveLoadV1_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.feature.Word2VecModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.fpm.FPGrowthModel
Save this model to the given path.
save(FPGrowthModel<?>, String) - Method in class org.apache.spark.mllib.fpm.FPGrowthModel.SaveLoadV1_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.fpm.PrefixSpanModel
Save this model to the given path.
save(PrefixSpanModel<?>, String) - Method in class org.apache.spark.mllib.fpm.PrefixSpanModel.SaveLoadV1_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
Save this model to the given path.
save(MatrixFactorizationModel, String) - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel.SaveLoadV1_0$
Saves a MatrixFactorizationModel, where user features are saved under data/users and product features are saved under data/products.
save(SparkContext, String, String, Vector, double) - Method in class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$
Helper method for saving GLM regression model metadata and data.
save(SparkContext, String) - Method in class org.apache.spark.mllib.regression.IsotonicRegressionModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.regression.LassoModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
 
save(SparkContext, String, DecisionTreeModel) - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
save(SparkContext, String) - Method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
save(SparkContext, String) - Method in interface org.apache.spark.mllib.util.Saveable
Save this model to the given path.
save(String) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame at the specified path.
save() - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame as the specified table.
Saveable - Interface in org.apache.spark.mllib.util
Developer API
saveAsHadoopDataset(JobConf) - Method in class org.apache.spark.api.java.JavaPairRDD
Output the RDD to any Hadoop-supported storage system, using a Hadoop JobConf object for that storage system.
saveAsHadoopDataset(JobConf) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported storage system, using a Hadoop JobConf object for that storage system.
saveAsHadoopFile(String, Class<?>, Class<?>, Class<F>, JobConf) - Method in class org.apache.spark.api.java.JavaPairRDD
Output the RDD to any Hadoop-supported file system.
saveAsHadoopFile(String, Class<?>, Class<?>, Class<F>) - Method in class org.apache.spark.api.java.JavaPairRDD
Output the RDD to any Hadoop-supported file system.
saveAsHadoopFile(String, Class<?>, Class<?>, Class<F>, Class<? extends CompressionCodec>) - Method in class org.apache.spark.api.java.JavaPairRDD
Output the RDD to any Hadoop-supported file system, compressing with the supplied codec.
saveAsHadoopFile(String, ClassTag<F>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.
saveAsHadoopFile(String, Class<? extends CompressionCodec>, ClassTag<F>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.
saveAsHadoopFile(String, Class<?>, Class<?>, Class<? extends OutputFormat<?, ?>>, Class<? extends CompressionCodec>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.
saveAsHadoopFile(String, Class<?>, Class<?>, Class<? extends OutputFormat<?, ?>>, JobConf, Option<Class<? extends CompressionCodec>>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported file system, using a Hadoop OutputFormat class supporting the key and value types K and V in this RDD.
saveAsHadoopFiles(String, String) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Save each RDD in this DStream as a Hadoop file.
saveAsHadoopFiles(String, String, Class<?>, Class<?>, Class<F>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Save each RDD in this DStream as a Hadoop file.
saveAsHadoopFiles(String, String, Class<?>, Class<?>, Class<F>, JobConf) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Save each RDD in this DStream as a Hadoop file.
saveAsHadoopFiles(String, String) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
saveAsHadoopFiles(String, String, Class<?>, Class<?>, Class<F>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
saveAsHadoopFiles(String, String, Class<?>, Class<?>, Class<F>, JobConf) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
saveAsHadoopFiles(String, String) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
saveAsHadoopFiles(String, String, Class<?>, Class<?>, Class<F>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
saveAsHadoopFiles(String, String, Class<?>, Class<?>, Class<F>, JobConf) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
saveAsHadoopFiles(String, String, ClassTag<F>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Save each RDD in this DStream as a Hadoop file.
saveAsHadoopFiles(String, String, Class<?>, Class<?>, Class<? extends OutputFormat<?, ?>>, JobConf) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Save each RDD in this DStream as a Hadoop file.
saveAsLibSVMFile(RDD<LabeledPoint>, String) - Static method in class org.apache.spark.mllib.util.MLUtils
Save labeled data in LIBSVM format.
saveAsNewAPIHadoopDataset(Configuration) - Method in class org.apache.spark.api.java.JavaPairRDD
Output the RDD to any Hadoop-supported storage system, using a Configuration object for that storage system.
saveAsNewAPIHadoopDataset(Configuration) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported storage system with new Hadoop API, using a Hadoop Configuration object for that storage system.
saveAsNewAPIHadoopFile(String, Class<?>, Class<?>, Class<F>, Configuration) - Method in class org.apache.spark.api.java.JavaPairRDD
Output the RDD to any Hadoop-supported file system.
saveAsNewAPIHadoopFile(String, Class<?>, Class<?>, Class<F>) - Method in class org.apache.spark.api.java.JavaPairRDD
Output the RDD to any Hadoop-supported file system.
saveAsNewAPIHadoopFile(String, ClassTag<F>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported file system, using a new Hadoop API OutputFormat (mapreduce.OutputFormat) object supporting the key and value types K and V in this RDD.
saveAsNewAPIHadoopFile(String, Class<?>, Class<?>, Class<? extends OutputFormat<?, ?>>, Configuration) - Method in class org.apache.spark.rdd.PairRDDFunctions
Output the RDD to any Hadoop-supported file system, using a new Hadoop API OutputFormat (mapreduce.OutputFormat) object supporting the key and value types K and V in this RDD.
saveAsNewAPIHadoopFiles(String, String) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Save each RDD in this DStream as a Hadoop file.
saveAsNewAPIHadoopFiles(String, String, Class<?>, Class<?>, Class<F>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Save each RDD in this DStream as a Hadoop file.
saveAsNewAPIHadoopFiles(String, String, Class<?>, Class<?>, Class<F>, Configuration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Save each RDD in this DStream as a Hadoop file.
saveAsNewAPIHadoopFiles(String, String) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
saveAsNewAPIHadoopFiles(String, String, Class<?>, Class<?>, Class<F>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
saveAsNewAPIHadoopFiles(String, String, Class<?>, Class<?>, Class<F>, Configuration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
saveAsNewAPIHadoopFiles(String, String) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
saveAsNewAPIHadoopFiles(String, String, Class<?>, Class<?>, Class<F>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
saveAsNewAPIHadoopFiles(String, String, Class<?>, Class<?>, Class<F>, Configuration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
saveAsNewAPIHadoopFiles(String, String, ClassTag<F>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Save each RDD in this DStream as a Hadoop file.
saveAsNewAPIHadoopFiles(String, String, Class<?>, Class<?>, Class<? extends OutputFormat<?, ?>>, Configuration) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Save each RDD in this DStream as a Hadoop file.
saveAsNewAPIHadoopFiles$default$6() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
saveAsNewAPIHadoopFiles$default$6() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
saveAsObjectFile(String) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.api.java.JavaRDD
 
saveAsObjectFile(String) - Method in interface org.apache.spark.api.java.JavaRDDLike
Save this RDD as a SequenceFile of serialized objects.
saveAsObjectFile(String) - Static method in class org.apache.spark.api.r.RRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.graphx.EdgeRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
saveAsObjectFile(String) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
saveAsObjectFile(String) - Static method in class org.apache.spark.graphx.VertexRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.rdd.HadoopRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.rdd.JdbcRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
saveAsObjectFile(String) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
saveAsObjectFile(String) - Method in class org.apache.spark.rdd.RDD
Save this RDD as a SequenceFile of serialized objects.
saveAsObjectFile(String) - Static method in class org.apache.spark.rdd.UnionRDD
 
saveAsObjectFiles(String, String) - Method in class org.apache.spark.streaming.dstream.DStream
Save each RDD in this DStream as a Sequence file of serialized objects.
saveAsSequenceFile(String, Option<Class<? extends CompressionCodec>>) - Method in class org.apache.spark.rdd.SequenceFileRDDFunctions
Output the RDD as a Hadoop SequenceFile using the Writable types we infer from the RDD's key and value types.
saveAsTable(String) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame as the specified table.
saveAsTextFile(String) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.api.java.JavaRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.api.java.JavaRDD
 
saveAsTextFile(String) - Method in interface org.apache.spark.api.java.JavaRDDLike
Save this RDD as a text file, using string representations of elements.
saveAsTextFile(String, Class<? extends CompressionCodec>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Save this RDD as a compressed text file, using string representations of elements.
saveAsTextFile(String) - Static method in class org.apache.spark.api.r.RRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.api.r.RRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.graphx.EdgeRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
saveAsTextFile(String) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
saveAsTextFile(String) - Static method in class org.apache.spark.graphx.VertexRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.graphx.VertexRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.rdd.HadoopRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.rdd.JdbcRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
saveAsTextFile(String) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
saveAsTextFile(String) - Method in class org.apache.spark.rdd.RDD
Save this RDD as a text file, using string representations of elements.
saveAsTextFile(String, Class<? extends CompressionCodec>) - Method in class org.apache.spark.rdd.RDD
Save this RDD as a compressed text file, using string representations of elements.
saveAsTextFile(String) - Static method in class org.apache.spark.rdd.UnionRDD
 
saveAsTextFile(String, Class<? extends CompressionCodec>) - Static method in class org.apache.spark.rdd.UnionRDD
 
saveAsTextFiles(String, String) - Method in class org.apache.spark.streaming.dstream.DStream
Save each RDD in this DStream as at text file, using string representation of elements.
savedTasks() - Method in class org.apache.spark.status.LiveStage
 
saveImpl(Params, PipelineStage[], SparkContext, String) - Method in class org.apache.spark.ml.Pipeline.SharedReadWrite$
Save metadata and stages for a Pipeline or PipelineModel - save metadata to path/metadata - save stages to stages/IDX_UID
saveImpl(M, String, SparkSession, JsonAST.JObject) - Static method in class org.apache.spark.ml.tree.EnsembleModelReadWrite
Helper method for saving a tree ensemble to disk.
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.clustering.BisectingKMeansModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.clustering.KMeansModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.clustering.PowerIterationClusteringModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.feature.ChiSqSelectorModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.fpm.FPGrowthModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.fpm.PrefixSpanModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.recommendation.MatrixFactorizationModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$
 
SaveLoadV1_0$() - Constructor for class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
 
SaveLoadV2_0$() - Constructor for class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$
 
SaveMode - Enum in org.apache.spark.sql
SaveMode is used to specify the expected behavior of saving a DataFrame to a data source.
sc() - Method in class org.apache.spark.api.java.JavaSparkContext
 
sc() - Method in class org.apache.spark.sql.SQLImplicits.StringToColumn
 
scal(double, Vector) - Static method in class org.apache.spark.ml.linalg.BLAS
x = a * x
scal(double, Vector) - Static method in class org.apache.spark.mllib.linalg.BLAS
x = a * x
scalaBoolean() - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's primitive boolean type.
scalaByte() - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's primitive byte type.
scalaDouble() - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's primitive double type.
scalaFloat() - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's primitive float type.
scalaInt() - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's primitive int type.
scalaIntToJavaLong(DStream<Object>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
scalaIntToJavaLong(DStream<Object>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
 
scalaIntToJavaLong(DStream<Object>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
scalaIntToJavaLong(DStream<Object>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
scalaIntToJavaLong(DStream<Object>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
scalaIntToJavaLong(DStream<Object>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
scalaIntToJavaLong(DStream<Object>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
scalaLong() - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's primitive long type.
scalaShort() - Static method in class org.apache.spark.sql.Encoders
An encoder for Scala's primitive short type.
scalaToJavaLong(JavaPairDStream<K, Object>, ClassTag<K>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
scalaVersion() - Method in class org.apache.spark.status.api.v1.RuntimeInfo
 
scale() - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
scale() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
 
scale() - Method in class org.apache.spark.mllib.random.GammaGenerator
 
scale() - Method in class org.apache.spark.sql.types.Decimal
 
scale() - Method in class org.apache.spark.sql.types.DecimalType
 
scalingVec() - Method in class org.apache.spark.ml.feature.ElementwiseProduct
the vector to multiply with input vectors
scalingVec() - Method in class org.apache.spark.mllib.feature.ElementwiseProduct
 
scan(B, Function2<B, B, B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
scanLeft(B, Function2<B, A, B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
scanRight(B, Function2<A, B, B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
SCHEDULED() - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
SCHEDULER_DELAY() - Static method in class org.apache.spark.status.TaskIndexNames
 
SCHEDULER_DELAY() - Static method in class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
SCHEDULER_DELAY() - Static method in class org.apache.spark.ui.ToolTips
 
SchedulerBackendUtils - Class in org.apache.spark.scheduler.cluster
 
SchedulerBackendUtils() - Constructor for class org.apache.spark.scheduler.cluster.SchedulerBackendUtils
 
schedulerDelay() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
schedulerDelay(TaskData) - Static method in class org.apache.spark.status.AppStatusUtils
 
schedulerDelay(long, long, long, long, long, long) - Static method in class org.apache.spark.status.AppStatusUtils
 
SchedulerPool - Class in org.apache.spark.status
 
SchedulerPool(String) - Constructor for class org.apache.spark.status.SchedulerPool
 
schedulingDelay() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
schedulingDelay() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
Time taken for the first job of this batch to start processing from the time this batch was submitted to the streaming scheduler.
SchedulingMode - Class in org.apache.spark.scheduler
"FAIR" and "FIFO" determines which policy is used to order tasks amongst a Schedulable's sub-queues "NONE" is used when the a Schedulable has no sub-queues.
SchedulingMode() - Constructor for class org.apache.spark.scheduler.SchedulingMode
 
schedulingPool() - Method in class org.apache.spark.status.api.v1.StageData
 
schedulingPool() - Method in class org.apache.spark.status.LiveStage
 
schema(StructType) - Method in class org.apache.spark.sql.DataFrameReader
Specifies the input schema.
schema(String) - Method in class org.apache.spark.sql.DataFrameReader
Specifies the schema by using the input DDL-formatted string.
schema() - Method in class org.apache.spark.sql.Dataset
Returns the schema of this Dataset.
schema() - Method in interface org.apache.spark.sql.Encoder
Returns the schema of encoding this type of object as a Row.
schema() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
schema() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
schema() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
schema() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
schema() - Method in interface org.apache.spark.sql.Row
Schema for the row.
schema() - Method in class org.apache.spark.sql.sources.BaseRelation
 
schema(StructType) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Specifies the input schema.
schema(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Specifies the schema by using the input DDL-formatted string.
schemaLess() - Method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
SchemaRelationProvider - Interface in org.apache.spark.sql.sources
Implemented by objects that produce relations for a specific kind of data source with a given schema.
schemaString() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
schemaString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
schemaString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
schemaString() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
SchemaUtils - Class in org.apache.spark.ml.util
Utils for handling schemas.
SchemaUtils() - Constructor for class org.apache.spark.ml.util.SchemaUtils
 
SchemaUtils - Class in org.apache.spark.sql.util
Utils for handling schemas.
SchemaUtils() - Constructor for class org.apache.spark.sql.util.SchemaUtils
 
scope() - Method in class org.apache.spark.storage.RDDInfo
 
scoreAndLabels() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
 
scratch() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
script() - Method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
ScriptTransformationExec - Class in org.apache.spark.sql.hive.execution
Transforms the input by forking and running the specified script.
ScriptTransformationExec(Seq<Expression>, String, Seq<Attribute>, SparkPlan, HiveScriptIOSchema) - Constructor for class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
ScriptTransformationWriterThread - Class in org.apache.spark.sql.hive.execution
 
ScriptTransformationWriterThread(Iterator<InternalRow>, Seq<DataType>, org.apache.spark.sql.catalyst.expressions.Projection, AbstractSerDe, ObjectInspector, HiveScriptIOSchema, OutputStream, Process, org.apache.spark.util.CircularBuffer, TaskContext, Configuration) - Constructor for class org.apache.spark.sql.hive.execution.ScriptTransformationWriterThread
 
second(Column) - Static method in class org.apache.spark.sql.functions
Extracts the seconds as an integer from a given date/timestamp/string.
seconds() - Static method in class org.apache.spark.scheduler.StatsReportListener
 
seconds(long) - Static method in class org.apache.spark.streaming.Durations
 
Seconds - Class in org.apache.spark.streaming
Helper object that creates instance of Duration representing a given number of seconds.
Seconds() - Constructor for class org.apache.spark.streaming.Seconds
 
securityManager() - Method in class org.apache.spark.SparkEnv
 
seed() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
seed() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
seed() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
seed() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
seed() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
seed() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
seed() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
seed() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
seed() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
seed() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
seed() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
seed() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
seed() - Static method in class org.apache.spark.ml.clustering.KMeans
 
seed() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
seed() - Static method in class org.apache.spark.ml.clustering.LDA
 
seed() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
seed() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
seed() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
seed() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
seed() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
seed() - Method in interface org.apache.spark.ml.param.shared.HasSeed
Param for random seed.
seed() - Static method in class org.apache.spark.ml.recommendation.ALS
 
seed() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
seed() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
seed() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
seed() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
seed() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
seed() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
seed() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
seed() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
seed() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
seed() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
seedParam() - Static method in class org.apache.spark.ml.image.SamplePathFilter
 
segmentLength(Function1<A, Object>, int) - Static method in class org.apache.spark.sql.types.StructType
 
select(Column...) - Method in class org.apache.spark.sql.Dataset
Selects a set of column based expressions.
select(String, String...) - Method in class org.apache.spark.sql.Dataset
Selects a set of columns.
select(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Selects a set of column based expressions.
select(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Selects a set of columns.
select(TypedColumn<T, U1>) - Method in class org.apache.spark.sql.Dataset
Experimental Returns a new Dataset by computing the given Column expression for each element.
select(TypedColumn<T, U1>, TypedColumn<T, U2>) - Method in class org.apache.spark.sql.Dataset
Experimental Returns a new Dataset by computing the given Column expressions for each element.
select(TypedColumn<T, U1>, TypedColumn<T, U2>, TypedColumn<T, U3>) - Method in class org.apache.spark.sql.Dataset
Experimental Returns a new Dataset by computing the given Column expressions for each element.
select(TypedColumn<T, U1>, TypedColumn<T, U2>, TypedColumn<T, U3>, TypedColumn<T, U4>) - Method in class org.apache.spark.sql.Dataset
Experimental Returns a new Dataset by computing the given Column expressions for each element.
select(TypedColumn<T, U1>, TypedColumn<T, U2>, TypedColumn<T, U3>, TypedColumn<T, U4>, TypedColumn<T, U5>) - Method in class org.apache.spark.sql.Dataset
Experimental Returns a new Dataset by computing the given Column expressions for each element.
selectedFeatures() - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
list of indices to select (filter).
selectedFeatures() - Method in class org.apache.spark.mllib.feature.ChiSqSelectorModel
 
selectExpr(String...) - Method in class org.apache.spark.sql.Dataset
Selects a set of SQL expressions.
selectExpr(Seq<String>) - Method in class org.apache.spark.sql.Dataset
Selects a set of SQL expressions.
selectorType() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
selectorType() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
selectorType() - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
semanticHash() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
semanticHash() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
semanticHash() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
semanticHash() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
sender() - Method in class org.apache.spark.storage.BlockManagerMessages.RegisterBlockManager
 
sendToDst(A) - Method in class org.apache.spark.graphx.EdgeContext
Sends a message to the destination vertex.
sendToDst(A) - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
sendToSrc(A) - Method in class org.apache.spark.graphx.EdgeContext
Sends a message to the source vertex.
sendToSrc(A) - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
seq() - Static method in class org.apache.spark.sql.types.StructType
 
seqToString(Seq<T>, Function1<T, String>) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
sequence() - Method in class org.apache.spark.mllib.fpm.PrefixSpan.FreqSequence
 
sequenceFile(String, Class<K>, Class<V>, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a Hadoop SequenceFile with given key and value types.
sequenceFile(String, Class<K>, Class<V>) - Method in class org.apache.spark.api.java.JavaSparkContext
Get an RDD for a Hadoop SequenceFile.
sequenceFile(String, Class<K>, Class<V>, int) - Method in class org.apache.spark.SparkContext
Get an RDD for a Hadoop SequenceFile with given key and value types.
sequenceFile(String, Class<K>, Class<V>) - Method in class org.apache.spark.SparkContext
Get an RDD for a Hadoop SequenceFile with given key and value types.
sequenceFile(String, int, ClassTag<K>, ClassTag<V>, Function0<WritableConverter<K>>, Function0<WritableConverter<V>>) - Method in class org.apache.spark.SparkContext
Version of sequenceFile() for types implicitly convertible to Writables through a WritableConverter.
SequenceFileRDDFunctions<K,V> - Class in org.apache.spark.rdd
Extra functions available on RDDs of (key, value) pairs to create a Hadoop SequenceFile, through an implicit conversion.
SequenceFileRDDFunctions(RDD<Tuple2<K, V>>, Class<? extends Writable>, Class<? extends Writable>, Function1<K, Writable>, ClassTag<K>, Function1<V, Writable>, ClassTag<V>) - Constructor for class org.apache.spark.rdd.SequenceFileRDDFunctions
 
SER_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
SerDe - Class in org.apache.spark.api.r
Utility functions to serialize, deserialize objects to / from R
SerDe() - Constructor for class org.apache.spark.api.r.SerDe
 
SERDE() - Static method in class org.apache.spark.sql.hive.execution.HiveOptions
 
serde() - Method in class org.apache.spark.sql.hive.execution.HiveOptions
 
serdeProperties() - Method in class org.apache.spark.sql.hive.execution.HiveOptions
 
SerializableMapWrapper(Map<A, B>) - Constructor for class org.apache.spark.api.java.JavaUtils.SerializableMapWrapper
 
SerializableWritable<T extends org.apache.hadoop.io.Writable> - Class in org.apache.spark
 
SerializableWritable(T) - Constructor for class org.apache.spark.SerializableWritable
 
SerializationDebugger - Class in org.apache.spark.serializer
 
SerializationDebugger() - Constructor for class org.apache.spark.serializer.SerializationDebugger
 
SerializationDebugger.ObjectStreamClassMethods - Class in org.apache.spark.serializer
An implicit class that allows us to call private methods of ObjectStreamClass.
SerializationDebugger.ObjectStreamClassMethods$ - Class in org.apache.spark.serializer
 
SerializationFormats - Class in org.apache.spark.api.r
 
SerializationFormats() - Constructor for class org.apache.spark.api.r.SerializationFormats
 
SerializationStream - Class in org.apache.spark.serializer
Developer API A stream for writing serialized objects.
SerializationStream() - Constructor for class org.apache.spark.serializer.SerializationStream
 
serialize(Vector) - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
serialize(T, ClassTag<T>) - Method in class org.apache.spark.serializer.DummySerializerInstance
 
serialize(T, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializerInstance
 
serialize(T) - Static method in class org.apache.spark.util.Utils
Serialize an object using Java serialization
SERIALIZED_R_DATA_SCHEMA() - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
serializedData() - Method in class org.apache.spark.scheduler.local.StatusUpdate
 
serializedMapStatus(org.apache.spark.broadcast.BroadcastManager, boolean, int) - Method in class org.apache.spark.ShuffleStatus
Serializes the mapStatuses array into an efficient compressed format.
SerializedMemoryEntry<T> - Class in org.apache.spark.storage.memory
 
SerializedMemoryEntry(org.apache.spark.util.io.ChunkedByteBuffer, MemoryMode, ClassTag<T>) - Constructor for class org.apache.spark.storage.memory.SerializedMemoryEntry
 
Serializer - Class in org.apache.spark.serializer
Developer API A serializer.
Serializer() - Constructor for class org.apache.spark.serializer.Serializer
 
serializer() - Method in class org.apache.spark.ShuffleDependency
 
serializer() - Method in class org.apache.spark.SparkEnv
 
SerializerInstance - Class in org.apache.spark.serializer
Developer API An instance of a serializer, for use by one thread at a time.
SerializerInstance() - Constructor for class org.apache.spark.serializer.SerializerInstance
 
serializerManager() - Method in class org.apache.spark.SparkEnv
 
serializeStream(OutputStream) - Method in class org.apache.spark.serializer.DummySerializerInstance
 
serializeStream(OutputStream) - Method in class org.apache.spark.serializer.SerializerInstance
 
serializeViaNestedStream(OutputStream, SerializerInstance, Function1<SerializationStream, BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Serialize via nested stream using specific serializer
ServletParams(Function1<HttpServletRequest, T>, String, Function1<T, String>) - Constructor for class org.apache.spark.ui.JettyUtils.ServletParams
 
ServletParams$() - Constructor for class org.apache.spark.ui.JettyUtils.ServletParams$
 
session(SparkSession) - Static method in class org.apache.spark.ml.r.RWrappers
 
session(SparkSession) - Method in class org.apache.spark.ml.util.MLReader
 
session(SparkSession) - Method in class org.apache.spark.ml.util.MLWriter
 
session() - Static method in class org.apache.spark.sql.hive.HiveSessionStateBuilder
 
sessionCatalog() - Method in class org.apache.spark.sql.hive.RelationConversions
 
SessionConfigSupport - Interface in org.apache.spark.sql.sources.v2
A mix-in interface for DataSourceV2.
sessionState() - Method in class org.apache.spark.sql.SparkSession
State isolated across sessions, including SQL configurations, temporary tables, registered functions, and everything else that accepts a SQLConf.
set(long, long, int, int, VD, VD, ED) - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
Set() - Static method in class org.apache.spark.metrics.sink.StatsdMetricType
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.OneVsRest
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.KMeans
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.LDA
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Binarizer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.DCT
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.HashingTF
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.IDF
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.IDFModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Imputer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.IndexToString
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Interaction
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.NGram
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Normalizer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.PCA
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.PCAModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.RFormula
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.StandardScaler
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.StringIndexer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Word2Vec
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
set(Param<T>, T) - Method in interface org.apache.spark.ml.param.Params
Sets a parameter in the embedded param map.
set(String, Object) - Method in interface org.apache.spark.ml.param.Params
Sets a parameter (by name) in the embedded param map.
set(ParamPair<?>) - Method in interface org.apache.spark.ml.param.Params
Sets a parameter in the embedded param map.
set(Param<T>, T) - Static method in class org.apache.spark.ml.Pipeline
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.PipelineModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.recommendation.ALS
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
set(Param<T>, T) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
set(String, long, long) - Static method in class org.apache.spark.rdd.InputFileBlockHolder
Sets the thread-local input block.
set(String, String) - Method in class org.apache.spark.SparkConf
Set a configuration variable.
set(SparkEnv) - Static method in class org.apache.spark.SparkEnv
 
set(String, String) - Method in class org.apache.spark.sql.RuntimeConfig
Sets the given Spark runtime configuration property.
set(String, boolean) - Method in class org.apache.spark.sql.RuntimeConfig
Sets the given Spark runtime configuration property.
set(String, long) - Method in class org.apache.spark.sql.RuntimeConfig
Sets the given Spark runtime configuration property.
set(long) - Method in class org.apache.spark.sql.types.Decimal
Set this Decimal to the given Long.
set(int) - Method in class org.apache.spark.sql.types.Decimal
Set this Decimal to the given Int.
set(long, int, int) - Method in class org.apache.spark.sql.types.Decimal
Set this Decimal to the given unscaled Long, with a given precision and scale.
set(BigDecimal, int, int) - Method in class org.apache.spark.sql.types.Decimal
Set this Decimal to the given BigDecimal value, with a given precision and scale.
set(BigDecimal) - Method in class org.apache.spark.sql.types.Decimal
Set this Decimal to the given BigDecimal value, inheriting its precision and scale.
set(BigInteger) - Method in class org.apache.spark.sql.types.Decimal
If the value is not in the range of long, convert it to BigDecimal and the precision and scale are based on the converted value.
set(Decimal) - Method in class org.apache.spark.sql.types.Decimal
Set this Decimal to the given Decimal value.
setAcceptsNull(boolean) - Static method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
setActive(SQLContext) - Static method in class org.apache.spark.sql.SQLContext
Deprecated.
Use SparkSession.setActiveSession instead. Since 2.0.0.
setActiveSession(SparkSession) - Static method in class org.apache.spark.sql.SparkSession
Changes the SparkSession that will be returned in this thread and its children when SparkSession.getOrCreate() is called.
setAggregationDepth(int) - Method in class org.apache.spark.ml.classification.LinearSVC
Suggested depth for treeAggregate (greater than or equal to 2).
setAggregationDepth(int) - Method in class org.apache.spark.ml.classification.LogisticRegression
Suggested depth for treeAggregate (greater than or equal to 2).
setAggregationDepth(int) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
Suggested depth for treeAggregate (greater than or equal to 2).
setAggregationDepth(int) - Method in class org.apache.spark.ml.regression.LinearRegression
Suggested depth for treeAggregate (greater than or equal to 2).
setAggregator(Aggregator<K, V, C>) - Method in class org.apache.spark.rdd.ShuffledRDD
Set aggregator for RDD's shuffle.
setAlgo(String) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
Sets Algorithm using a String.
setAll(Traversable<Tuple2<String, String>>) - Method in class org.apache.spark.SparkConf
Set multiple parameters together
setAlpha(double) - Method in class org.apache.spark.ml.recommendation.ALS
 
setAlpha(Vector) - Method in class org.apache.spark.mllib.clustering.LDA
Alias for setDocConcentration()
setAlpha(double) - Method in class org.apache.spark.mllib.clustering.LDA
Alias for setDocConcentration()
setAlpha(double) - Method in class org.apache.spark.mllib.recommendation.ALS
Sets the constant used in computing confidence in implicit ALS.
setAppName(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Set the application name.
setAppName(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
setAppName(String) - Method in class org.apache.spark.SparkConf
Set a name for your application.
setAppResource(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Set the main application resource.
setAppResource(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
setBandwidth(double) - Method in class org.apache.spark.mllib.stat.KernelDensity
Sets the bandwidth (standard deviation) of the Gaussian kernel (default: 1.0).
setBeta(double) - Method in class org.apache.spark.mllib.clustering.LDA
Alias for setTopicConcentration()
setBinary(boolean) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
setBinary(boolean) - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
setBinary(boolean) - Method in class org.apache.spark.ml.feature.HashingTF
 
setBinary(boolean) - Method in class org.apache.spark.mllib.feature.HashingTF
If true, term frequency vector will be binary such that non-zero term counts will be set to 1 (default: false)
setBlocks(int) - Method in class org.apache.spark.mllib.recommendation.ALS
Set the number of blocks for both user blocks and product blocks to parallelize the computation into; pass -1 for an auto-configured number of blocks.
setBlockSize(int) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Sets the value of param blockSize.
setBucketLength(double) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
setCacheNodeIds(boolean) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setCacheNodeIds(boolean) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setCacheNodeIds(boolean) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setCacheNodeIds(boolean) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setCacheNodeIds(boolean) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setCacheNodeIds(boolean) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setCacheNodeIds(boolean) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setCacheNodeIds(boolean) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setCacheNodeIds(boolean) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setCacheNodeIds(boolean) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setCacheNodeIds(boolean) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setCacheNodeIds(boolean) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setCallSite(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Pass-through to SparkContext.setCallSite.
setCallSite(String) - Method in class org.apache.spark.SparkContext
Set the thread-local property for overriding the call sites of actions and RDDs.
setCaseSensitive(boolean) - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
setCategoricalCols(String[]) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
setCategoricalFeaturesInfo(Map<Integer, Integer>) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
Sets categoricalFeaturesInfo using a Java Map.
setCensorCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
setCheckpointDir(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Set the directory under which RDDs are going to be checkpointed.
setCheckpointDir(String) - Method in class org.apache.spark.SparkContext
Set the directory under which RDDs are going to be checkpointed.
setCheckpointInterval(int) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setCheckpointInterval(int) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
Specifies how often to checkpoint the cached node IDs.
setCheckpointInterval(int) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setCheckpointInterval(int) - Method in class org.apache.spark.ml.classification.GBTClassifier
Specifies how often to checkpoint the cached node IDs.
setCheckpointInterval(int) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setCheckpointInterval(int) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
Specifies how often to checkpoint the cached node IDs.
setCheckpointInterval(int) - Method in class org.apache.spark.ml.clustering.LDA
 
setCheckpointInterval(int) - Method in class org.apache.spark.ml.recommendation.ALS
 
setCheckpointInterval(int) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setCheckpointInterval(int) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
Specifies how often to checkpoint the cached node IDs.
setCheckpointInterval(int) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setCheckpointInterval(int) - Method in class org.apache.spark.ml.regression.GBTRegressor
Specifies how often to checkpoint the cached node IDs.
setCheckpointInterval(int) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setCheckpointInterval(int) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
Specifies how often to checkpoint the cached node IDs.
setCheckpointInterval(int) - Method in class org.apache.spark.mllib.clustering.LDA
Parameter for set checkpoint interval (greater than or equal to 1) or disable checkpoint (-1).
setCheckpointInterval(int) - Method in class org.apache.spark.mllib.recommendation.ALS
Developer API Set period (in iterations) between checkpoints (default = 10).
setCheckpointInterval(int) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setClassifier(Classifier<?, ?, ?>) - Method in class org.apache.spark.ml.classification.OneVsRest
 
setColdStartStrategy(String) - Method in class org.apache.spark.ml.recommendation.ALS
 
setColdStartStrategy(String) - Method in class org.apache.spark.ml.recommendation.ALSModel
 
setCollectSubModels(boolean) - Method in class org.apache.spark.ml.tuning.CrossValidator
Whether to collect submodels when fitting.
setCollectSubModels(boolean) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
Whether to collect submodels when fitting.
setConf(String, String) - Method in class org.apache.spark.launcher.AbstractLauncher
Set a single configuration value for the application.
setConf(String, String) - Method in class org.apache.spark.launcher.SparkLauncher
 
setConf(Configuration) - Method in class org.apache.spark.ml.image.SamplePathFilter
 
setConf(Properties) - Method in class org.apache.spark.sql.SQLContext
Set Spark SQL configuration properties.
setConf(String, String) - Method in class org.apache.spark.sql.SQLContext
Set the given Spark SQL configuration property.
setConfig(String, String) - Static method in class org.apache.spark.launcher.SparkLauncher
Set a configuration value for the launcher library.
setConvergenceTol(double) - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Set the largest change in log-likelihood at which convergence is considered to have occurred.
setConvergenceTol(double) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Set the convergence tolerance.
setConvergenceTol(double) - Method in class org.apache.spark.mllib.optimization.LBFGS
Set the convergence tolerance of iterations for L-BFGS.
setConvergenceTol(double) - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
Set the convergence tolerance.
setCurrentDatabase(String) - Method in class org.apache.spark.sql.catalog.Catalog
Sets the current default database in this session.
setCustomHostname(String) - Static method in class org.apache.spark.util.Utils
Allow setting a custom host name because when we run on Mesos we need to use the same hostname it reports to the master.
setDecayFactor(double) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Set the forgetfulness of the previous centroids.
setDefault(Param<T>, T) - Method in interface org.apache.spark.ml.param.Params
Sets a default value for a param.
setDefault(Seq<ParamPair<?>>) - Method in interface org.apache.spark.ml.param.Params
Sets default values for a list of params.
setDefaultClassLoader(ClassLoader) - Static method in class org.apache.spark.serializer.KryoSerializer
 
setDefaultClassLoader(ClassLoader) - Method in class org.apache.spark.serializer.Serializer
Sets a class loader for the serializer to use in deserialization.
setDefaultSession(SparkSession) - Static method in class org.apache.spark.sql.SparkSession
Sets the default SparkSession that is returned by the builder.
setDegree(int) - Method in class org.apache.spark.ml.feature.PolynomialExpansion
 
setDeployMode(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Set the deploy mode for the application.
setDeployMode(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
setDocConcentration(double[]) - Method in class org.apache.spark.ml.clustering.LDA
 
setDocConcentration(double) - Method in class org.apache.spark.ml.clustering.LDA
 
setDocConcentration(Vector) - Method in class org.apache.spark.mllib.clustering.LDA
Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").
setDocConcentration(double) - Method in class org.apache.spark.mllib.clustering.LDA
Replicates a Double docConcentration to create a symmetric prior.
setDropLast(boolean) - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
setDropLast(boolean) - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
setDropLast(boolean) - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
setElasticNetParam(double) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the ElasticNet mixing parameter.
setElasticNetParam(double) - Method in class org.apache.spark.ml.regression.LinearRegression
Set the ElasticNet mixing parameter.
setEpsilon(double) - Method in class org.apache.spark.ml.regression.LinearRegression
Sets the value of param epsilon.
setEpsilon(double) - Method in class org.apache.spark.mllib.clustering.KMeans
Set the distance threshold within which we've consider centers to have converged.
setEstimator(Estimator<?>) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
setEstimator(Estimator<?>) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
setEstimatorParamMaps(ParamMap[]) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
setEstimatorParamMaps(ParamMap[]) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
setEvaluator(Evaluator) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
setEvaluator(Evaluator) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
setExecutorEnv(String, String) - Method in class org.apache.spark.SparkConf
Set an environment variable to be used when launching executors for this application.
setExecutorEnv(Seq<Tuple2<String, String>>) - Method in class org.apache.spark.SparkConf
Set multiple environment variables to be used when launching executors.
setExecutorEnv(Tuple2<String, String>[]) - Method in class org.apache.spark.SparkConf
Set multiple environment variables to be used when launching executors.
setFamily(String) - Method in class org.apache.spark.ml.classification.LogisticRegression
Sets the value of param family.
setFamily(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the value of param family.
setFdr(double) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setFdr(double) - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
setFeatureIndex(int) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
setFeatureIndex(int) - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.classification.OneVsRest
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.KMeans
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.KMeansModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.LDA
The features for LDA should be a Vector representing the word counts in a document.
setFeaturesCol(String) - Method in class org.apache.spark.ml.clustering.LDAModel
The features for LDA should be a Vector representing the word counts in a document.
setFeaturesCol(String) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.feature.RFormula
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.PredictionModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.Predictor
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
setFeaturesCol(String) - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setFeaturesCol(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setFeatureSubsetStrategy(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setFeatureSubsetStrategy(String) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setFeatureSubsetStrategy(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setFeatureSubsetStrategy(String) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setFeatureSubsetStrategy(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setFeatureSubsetStrategy(String) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setFeatureSubsetStrategy(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setFeatureSubsetStrategy(String) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setFinalRDDStorageLevel(StorageLevel) - Method in class org.apache.spark.mllib.recommendation.ALS
Developer API Sets storage level for final RDDs (user/product used in MatrixFactorizationModel).
setFinalStorageLevel(String) - Method in class org.apache.spark.ml.recommendation.ALS
 
setFitIntercept(boolean) - Method in class org.apache.spark.ml.classification.LinearSVC
Whether to fit an intercept term.
setFitIntercept(boolean) - Method in class org.apache.spark.ml.classification.LogisticRegression
Whether to fit an intercept term.
setFitIntercept(boolean) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
Set if we should fit the intercept Default is true.
setFitIntercept(boolean) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets if we should fit the intercept.
setFitIntercept(boolean) - Method in class org.apache.spark.ml.regression.LinearRegression
Set if we should fit the intercept.
setForceIndexLabel(boolean) - Method in class org.apache.spark.ml.feature.RFormula
 
setFormula(String) - Method in class org.apache.spark.ml.feature.RFormula
Sets the formula to use for this transformer.
setFpr(double) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setFpr(double) - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
setFwe(double) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setFwe(double) - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
setGaps(boolean) - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
setGenerics(Kryo, Class<?>[]) - Static method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
setGradient(Gradient) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Set the gradient function (of the loss function of one single data example) to be used for SGD.
setGradient(Gradient) - Method in class org.apache.spark.mllib.optimization.LBFGS
Set the gradient function (of the loss function of one single data example) to be used for L-BFGS.
setHalfLife(double, String) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Set the half life and time unit ("batches" or "points").
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.Bucketizer
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.RFormula
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.StringIndexer
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.VectorIndexer
 
setHandleInvalid(String) - Method in class org.apache.spark.ml.feature.VectorSizeHint
 
setHashAlgorithm(String) - Method in class org.apache.spark.mllib.feature.HashingTF
Set the hash algorithm used when mapping term to integer.
setIfMissing(String, String) - Method in class org.apache.spark.SparkConf
Set a parameter if it isn't already configured
setImmutable(boolean) - Static method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
setImplicitPrefs(boolean) - Method in class org.apache.spark.ml.recommendation.ALS
 
setImplicitPrefs(boolean) - Method in class org.apache.spark.mllib.recommendation.ALS
Sets whether to use implicit preference.
setImpurity(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setImpurity(String) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setImpurity(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setImpurity(String) - Method in class org.apache.spark.ml.classification.GBTClassifier
The impurity setting is ignored for GBT models.
setImpurity(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setImpurity(String) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setImpurity(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setImpurity(String) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setImpurity(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setImpurity(String) - Method in class org.apache.spark.ml.regression.GBTRegressor
The impurity setting is ignored for GBT models.
setImpurity(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setImpurity(String) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setImpurity(Impurity) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setIndices(int[]) - Method in class org.apache.spark.ml.feature.VectorSlicer
 
setInitialCenters(Vector[], double[]) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Specify initial centers directly.
setInitializationMode(String) - Method in class org.apache.spark.mllib.clustering.KMeans
Set the initialization algorithm.
setInitializationMode(String) - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering
Set the initialization mode.
setInitializationSteps(int) - Method in class org.apache.spark.mllib.clustering.KMeans
Set the number of steps for the k-means|| initialization mode.
setInitialModel(GaussianMixtureModel) - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Set the initial GMM starting point, bypassing the random initialization.
setInitialModel(KMeansModel) - Method in class org.apache.spark.mllib.clustering.KMeans
Set the initial starting point, bypassing the random initialization or k-means|| The condition model.k == this.k must be met, failure results in an IllegalArgumentException.
setInitialWeights(Vector) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Sets the value of param initialWeights.
setInitialWeights(Vector) - Method in class org.apache.spark.mllib.classification.StreamingLogisticRegressionWithSGD
Set the initial weights.
setInitialWeights(Vector) - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
Set the initial weights.
setInitMode(String) - Method in class org.apache.spark.ml.clustering.KMeans
 
setInitSteps(int) - Method in class org.apache.spark.ml.clustering.KMeans
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.Binarizer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.Bucketizer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
setInputCol(String) - Static method in class org.apache.spark.ml.feature.DCT
 
setInputCol(String) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.HashingTF
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.IDF
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.IDFModel
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.IndexToString
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.MaxAbsScaler
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.MinHashLSH
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
setInputCol(String) - Static method in class org.apache.spark.ml.feature.NGram
 
setInputCol(String) - Static method in class org.apache.spark.ml.feature.Normalizer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.PCA
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.PCAModel
 
setInputCol(String) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setInputCol(String) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.StandardScaler
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.StringIndexer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
setInputCol(String) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.VectorIndexer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.VectorSizeHint
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.VectorSlicer
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setInputCol(String) - Method in class org.apache.spark.ml.feature.Word2VecModel
 
setInputCol(String) - Method in class org.apache.spark.ml.UnaryTransformer
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.Bucketizer
 
setInputCols(Seq<String>) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.Imputer
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.ImputerModel
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.Interaction
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setInputCols(String[]) - Method in class org.apache.spark.ml.feature.VectorAssembler
 
setIntercept(boolean) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
 
setIntercept(boolean) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
 
setIntercept(boolean) - Method in class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
Set if the algorithm should add an intercept.
setIntercept(boolean) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
 
setIntercept(boolean) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
 
setIntercept(boolean) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
 
setIntermediateRDDStorageLevel(StorageLevel) - Method in class org.apache.spark.mllib.recommendation.ALS
Developer API Sets storage level for intermediate RDDs (user/product in/out links).
setIntermediateStorageLevel(String) - Method in class org.apache.spark.ml.recommendation.ALS
 
setInverse(boolean) - Method in class org.apache.spark.ml.feature.DCT
 
setIsotonic(boolean) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
setIsotonic(boolean) - Method in class org.apache.spark.mllib.regression.IsotonicRegression
Sets the isotonic parameter.
setItemCol(String) - Method in class org.apache.spark.ml.recommendation.ALS
 
setItemCol(String) - Method in class org.apache.spark.ml.recommendation.ALSModel
 
setItemsCol(String) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
setItemsCol(String) - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
setIterations(int) - Method in class org.apache.spark.mllib.recommendation.ALS
Set the number of iterations to run.
setJars(Seq<String>) - Method in class org.apache.spark.SparkConf
Set JAR files to distribute to the cluster.
setJars(String[]) - Method in class org.apache.spark.SparkConf
Set JAR files to distribute to the cluster.
setJavaHome(String) - Method in class org.apache.spark.launcher.SparkLauncher
Set a custom JAVA_HOME for launching the Spark application.
setJobDescription(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Set a human readable description of the current job.
setJobDescription(String) - Method in class org.apache.spark.SparkContext
Set a human readable description of the current job.
setJobGroup(String, String, boolean) - Method in class org.apache.spark.api.java.JavaSparkContext
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.
setJobGroup(String, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.
setJobGroup(String, String, boolean) - Method in class org.apache.spark.SparkContext
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value or cleared.
setK(int) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
setK(int) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
setK(int) - Method in class org.apache.spark.ml.clustering.KMeans
 
setK(int) - Method in class org.apache.spark.ml.clustering.LDA
 
setK(int) - Method in class org.apache.spark.ml.feature.PCA
 
setK(int) - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Sets the desired number of leaf clusters (default: 4).
setK(int) - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Set the number of Gaussians in the mixture model.
setK(int) - Method in class org.apache.spark.mllib.clustering.KMeans
Set the number of clusters to create (k).
setK(int) - Method in class org.apache.spark.mllib.clustering.LDA
Set the number of topics to infer, i.e., the number of soft cluster centers.
setK(int) - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering
Set the number of clusters.
setK(int) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Set the number of clusters.
setKappa(double) - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
Learning rate: exponential decay rate---should be between (0.5, 1.0] to guarantee asymptotic convergence.
setKeepLastCheckpoint(boolean) - Method in class org.apache.spark.ml.clustering.LDA
 
setKeepLastCheckpoint(boolean) - Method in class org.apache.spark.mllib.clustering.EMLDAOptimizer
If using checkpointing, this indicates whether to keep the last checkpoint (vs clean up).
setKeyOrdering(Ordering<K>) - Method in class org.apache.spark.rdd.ShuffledRDD
Set key ordering for RDD's shuffle.
setLabelCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setLabelCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
setLabelCol(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
setLabelCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
setLabelCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
setLabelCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
setLabelCol(String) - Method in class org.apache.spark.ml.classification.OneVsRest
 
setLabelCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setLabelCol(String) - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
setLabelCol(String) - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
setLabelCol(String) - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
setLabelCol(String) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setLabelCol(String) - Method in class org.apache.spark.ml.feature.RFormula
 
setLabelCol(String) - Method in class org.apache.spark.ml.Predictor
 
setLabelCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
setLabelCol(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setLabelCol(String) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
setLabelCol(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
setLabelCol(String) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
setLabelCol(String) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
setLabelCol(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setLabels(String[]) - Method in class org.apache.spark.ml.feature.IndexToString
 
setLambda(double) - Method in class org.apache.spark.mllib.classification.NaiveBayes
Set the smoothing parameter.
setLambda(double) - Method in class org.apache.spark.mllib.recommendation.ALS
Set the regularization parameter, lambda.
setLayers(int[]) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Sets the value of param layers.
setLearningDecay(double) - Method in class org.apache.spark.ml.clustering.LDA
 
setLearningOffset(double) - Method in class org.apache.spark.ml.clustering.LDA
 
setLearningRate(double) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets initial learning rate (default: 0.025).
setLearningRate(double) - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
setLink(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the value of param link.
setLinkPower(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the value of param linkPower.
setLinkPredictionCol(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the link prediction (linear predictor) column name.
setLinkPredictionCol(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
Sets the link prediction (linear predictor) column name.
setLocalProperty(String, String) - Method in class org.apache.spark.api.java.JavaSparkContext
Set a local property that affects jobs submitted from this thread, and all child threads, such as the Spark fair scheduler pool.
setLocalProperty(String, String) - Method in class org.apache.spark.SparkContext
Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler pool.
setLogLevel(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Control our logLevel.
setLogLevel(String) - Method in class org.apache.spark.SparkContext
Control our logLevel.
setLogLevel(Level) - Static method in class org.apache.spark.util.Utils
configure a new log4j level
setLoss(String) - Method in class org.apache.spark.ml.regression.LinearRegression
Sets the value of param loss.
setLoss(Loss) - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
setLossType(String) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setLossType(String) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setLowerBoundsOnCoefficients(Matrix) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the lower bounds on coefficients if fitting under bound constrained optimization.
setLowerBoundsOnIntercepts(Vector) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the lower bounds on intercepts if fitting under bound constrained optimization.
setMainClass(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Sets the application class name for Java/Scala applications.
setMainClass(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
setMapSideCombine(boolean) - Method in class org.apache.spark.rdd.ShuffledRDD
Set mapSideCombine flag for RDD's shuffle.
setMaster(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Set the Spark master for the application.
setMaster(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
setMaster(String) - Method in class org.apache.spark.SparkConf
The master URL to connect to, such as "local" to run locally with one thread, "local[4]" to run locally with 4 cores, or "spark://master:7077" to run on a Spark standalone cluster.
setMax(double) - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
setMax(double) - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
setMaxBins(int) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setMaxBins(int) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setMaxBins(int) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setMaxBins(int) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setMaxBins(int) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setMaxBins(int) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setMaxBins(int) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setMaxBins(int) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setMaxBins(int) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setMaxBins(int) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setMaxBins(int) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setMaxBins(int) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setMaxBins(int) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setMaxCategories(int) - Method in class org.apache.spark.ml.feature.VectorIndexer
 
setMaxDepth(int) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setMaxDepth(int) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setMaxDepth(int) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setMaxDepth(int) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setMaxDepth(int) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setMaxDepth(int) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setMaxDepth(int) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setMaxDepth(int) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setMaxDepth(int) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setMaxDepth(int) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setMaxDepth(int) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setMaxDepth(int) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setMaxDepth(int) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setMaxIter(int) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setMaxIter(int) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setMaxIter(int) - Method in class org.apache.spark.ml.classification.LinearSVC
Set the maximum number of iterations.
setMaxIter(int) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the maximum number of iterations.
setMaxIter(int) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Set the maximum number of iterations.
setMaxIter(int) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
setMaxIter(int) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
setMaxIter(int) - Method in class org.apache.spark.ml.clustering.KMeans
 
setMaxIter(int) - Method in class org.apache.spark.ml.clustering.LDA
 
setMaxIter(int) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setMaxIter(int) - Method in class org.apache.spark.ml.recommendation.ALS
 
setMaxIter(int) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
Set the maximum number of iterations.
setMaxIter(int) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setMaxIter(int) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setMaxIter(int) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the maximum number of iterations (applicable for solver "irls").
setMaxIter(int) - Method in class org.apache.spark.ml.regression.LinearRegression
Set the maximum number of iterations.
setMaxIterations(int) - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Sets the max number of k-means iterations to split clusters (default: 20).
setMaxIterations(int) - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Set the maximum number of iterations allowed.
setMaxIterations(int) - Method in class org.apache.spark.mllib.clustering.KMeans
Set maximum number of iterations allowed.
setMaxIterations(int) - Method in class org.apache.spark.mllib.clustering.LDA
Set the maximum number of iterations allowed.
setMaxIterations(int) - Method in class org.apache.spark.mllib.clustering.PowerIterationClustering
Set maximum number of iterations of the power iteration loop
setMaxLocalProjDBSize(long) - Method in class org.apache.spark.mllib.fpm.PrefixSpan
Sets the maximum number of items (including delimiters used in the internal storage format) allowed in a projected database before local processing (default: 32000000L).
setMaxMemoryInMB(int) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setMaxMemoryInMB(int) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setMaxMemoryInMB(int) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setMaxMemoryInMB(int) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setMaxMemoryInMB(int) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setMaxMemoryInMB(int) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setMaxMemoryInMB(int) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setMaxMemoryInMB(int) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setMaxMemoryInMB(int) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setMaxMemoryInMB(int) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setMaxMemoryInMB(int) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setMaxMemoryInMB(int) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setMaxMemoryInMB(int) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setMaxPatternLength(int) - Method in class org.apache.spark.mllib.fpm.PrefixSpan
Sets maximal pattern length (default: 10).
setMaxSentenceLength(int) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setMaxSentenceLength(int) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets the maximum length (in words) of each sentence in the input data.
setMetricName(String) - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
setMetricName(String) - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
setMetricName(String) - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
setMetricName(String) - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
setMin(double) - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
setMin(double) - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
setMinConfidence(double) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
setMinConfidence(double) - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
setMinConfidence(double) - Method in class org.apache.spark.mllib.fpm.AssociationRules
Sets the minimal confidence (default: 0.8).
setMinCount(int) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setMinCount(int) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets minCount, the minimum number of times a token must appear to be included in the word2vec model's vocabulary (default: 5).
setMinDF(double) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
setMinDivisibleClusterSize(double) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
setMinDivisibleClusterSize(double) - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Sets the minimum number of points (if greater than or equal to 1.0) or the minimum proportion of points (if less than 1.0) of a divisible cluster (default: 1).
setMinDocFreq(int) - Method in class org.apache.spark.ml.feature.IDF
 
setMiniBatchFraction(double) - Method in class org.apache.spark.mllib.classification.StreamingLogisticRegressionWithSGD
Set the fraction of each batch to use for updates.
setMiniBatchFraction(double) - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
Mini-batch fraction in (0, 1], which sets the fraction of document sampled and used in each iteration.
setMiniBatchFraction(double) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Set fraction of data to be used for each SGD iteration.
setMiniBatchFraction(double) - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
Set the fraction of each batch to use for updates.
setMinInfoGain(double) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setMinInfoGain(double) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setMinInfoGain(double) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setMinInfoGain(double) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setMinInfoGain(double) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setMinInfoGain(double) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setMinInfoGain(double) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setMinInfoGain(double) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setMinInfoGain(double) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setMinInfoGain(double) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setMinInfoGain(double) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setMinInfoGain(double) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setMinInfoGain(double) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setMinInstancesPerNode(int) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setMinInstancesPerNode(int) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setMinInstancesPerNode(int) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setMinInstancesPerNode(int) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setMinInstancesPerNode(int) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setMinInstancesPerNode(int) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setMinInstancesPerNode(int) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setMinInstancesPerNode(int) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setMinInstancesPerNode(int) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setMinInstancesPerNode(int) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setMinInstancesPerNode(int) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setMinInstancesPerNode(int) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setMinInstancesPerNode(int) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setMinSupport(double) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
setMinSupport(double) - Method in class org.apache.spark.mllib.fpm.FPGrowth
Sets the minimal support level (default: 0.3).
setMinSupport(double) - Method in class org.apache.spark.mllib.fpm.PrefixSpan
Sets the minimal support level (default: 0.1).
setMinTF(double) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
setMinTF(double) - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
setMinTokenLength(int) - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
setMissingValue(double) - Method in class org.apache.spark.ml.feature.Imputer
 
setModelType(String) - Method in class org.apache.spark.ml.classification.NaiveBayes
Set the model type using a string (case-sensitive).
setModelType(String) - Method in class org.apache.spark.mllib.classification.NaiveBayes
Set the model type using a string (case-sensitive).
setN(int) - Method in class org.apache.spark.ml.feature.NGram
 
setName(String) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Assign a name to this RDD
setName(String) - Method in class org.apache.spark.api.java.JavaPairRDD
Assign a name to this RDD
setName(String) - Method in class org.apache.spark.api.java.JavaRDD
Assign a name to this RDD
setName(String) - Static method in class org.apache.spark.api.r.RRDD
 
setName(String) - Static method in class org.apache.spark.graphx.EdgeRDD
 
setName(String) - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
setName(String) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
setName(String) - Static method in class org.apache.spark.graphx.VertexRDD
 
setName(String) - Static method in class org.apache.spark.rdd.HadoopRDD
 
setName(String) - Static method in class org.apache.spark.rdd.JdbcRDD
 
setName(String) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
setName(String) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
setName(String) - Method in class org.apache.spark.rdd.RDD
Assign a name to this RDD
setName(String) - Static method in class org.apache.spark.rdd.UnionRDD
 
setNames(String[]) - Method in class org.apache.spark.ml.feature.VectorSlicer
 
setNonnegative(boolean) - Method in class org.apache.spark.ml.recommendation.ALS
 
setNonnegative(boolean) - Method in class org.apache.spark.mllib.recommendation.ALS
Set whether the least-squares problems solved at each iteration should have nonnegativity constraints.
setNullAt(int) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
setNullAt(int) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
setNumBlocks(int) - Method in class org.apache.spark.ml.recommendation.ALS
Sets both numUserBlocks and numItemBlocks to the specific value.
setNumBuckets(int) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setNumBucketsArray(int[]) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setNumClasses(int) - Method in class org.apache.spark.mllib.classification.LogisticRegressionWithLBFGS
Set the number of possible outcomes for k classes classification problem in Multinomial Logistic Regression.
setNumClasses(int) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setNumCorrections(int) - Method in class org.apache.spark.mllib.optimization.LBFGS
Set the number of corrections used in the LBFGS update.
setNumFeatures(int) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
setNumFeatures(int) - Method in class org.apache.spark.ml.feature.HashingTF
 
setNumFolds(int) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
setNumHashTables(int) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
setNumHashTables(int) - Method in class org.apache.spark.ml.feature.MinHashLSH
 
setNumItemBlocks(int) - Method in class org.apache.spark.ml.recommendation.ALS
 
setNumIterations(int) - Method in class org.apache.spark.mllib.classification.StreamingLogisticRegressionWithSGD
Set the number of iterations of gradient descent to run per update.
setNumIterations(int) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets number of iterations (default: 1), which should be smaller than or equal to number of partitions.
setNumIterations(int) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Set the number of iterations for SGD.
setNumIterations(int) - Method in class org.apache.spark.mllib.optimization.LBFGS
Set the maximal number of iterations for L-BFGS.
setNumIterations(int) - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
Set the number of iterations of gradient descent to run per update.
setNumIterations(int) - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
setNumPartitions(int) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setNumPartitions(int) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
setNumPartitions(int) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets number of partitions (default: 1).
setNumPartitions(int) - Method in class org.apache.spark.mllib.fpm.FPGrowth
Sets the number of partitions used by parallel FP-growth (default: same as input data).
setNumRows(int) - Method in class org.apache.spark.sql.vectorized.ColumnarBatch
Sets the number of rows in this batch.
setNumTopFeatures(int) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setNumTopFeatures(int) - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
setNumTrees(int) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setNumTrees(int) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setNumTrees(int) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setNumTrees(int) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setNumUserBlocks(int) - Method in class org.apache.spark.ml.recommendation.ALS
 
setOffsetCol(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the value of param offsetCol.
setOffsetRange(Optional<Offset>, Optional<Offset>) - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.MicroBatchReader
Set the desired offset range for reader factories created from this reader.
setOptimizeDocConcentration(boolean) - Method in class org.apache.spark.ml.clustering.LDA
 
setOptimizeDocConcentration(boolean) - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
Sets whether to optimize docConcentration parameter during training.
setOptimizer(String) - Method in class org.apache.spark.ml.clustering.LDA
 
setOptimizer(LDAOptimizer) - Method in class org.apache.spark.mllib.clustering.LDA
Developer API
setOptimizer(String) - Method in class org.apache.spark.mllib.clustering.LDA
Set the LDAOptimizer used to perform the actual calculation by algorithm name.
setOrNull(long, int, int) - Method in class org.apache.spark.sql.types.Decimal
Set this Decimal to the given unscaled Long, with a given precision and scale, and return it, or return null if it cannot be set due to overflow.
setOutputCol(String) - Method in class org.apache.spark.ml.feature.Binarizer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.Bucketizer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
setOutputCol(String) - Static method in class org.apache.spark.ml.feature.DCT
 
setOutputCol(String) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.HashingTF
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.IDF
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.IDFModel
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.IndexToString
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.Interaction
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.MaxAbsScaler
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.MinHashLSH
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
setOutputCol(String) - Static method in class org.apache.spark.ml.feature.NGram
 
setOutputCol(String) - Static method in class org.apache.spark.ml.feature.Normalizer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.PCA
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.PCAModel
 
setOutputCol(String) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setOutputCol(String) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.StandardScaler
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.StringIndexer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
setOutputCol(String) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.VectorAssembler
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.VectorIndexer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.VectorSlicer
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setOutputCol(String) - Method in class org.apache.spark.ml.feature.Word2VecModel
 
setOutputCol(String) - Method in class org.apache.spark.ml.UnaryTransformer
 
setOutputCols(String[]) - Method in class org.apache.spark.ml.feature.Bucketizer
 
setOutputCols(String[]) - Method in class org.apache.spark.ml.feature.Imputer
 
setOutputCols(String[]) - Method in class org.apache.spark.ml.feature.ImputerModel
 
setOutputCols(String[]) - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
setOutputCols(String[]) - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
setOutputCols(String[]) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setP(double) - Method in class org.apache.spark.ml.feature.Normalizer
 
setParallelism(int) - Method in class org.apache.spark.ml.classification.OneVsRest
The implementation of parallel one vs.
setParallelism(int) - Method in class org.apache.spark.ml.tuning.CrossValidator
Set the maximum level of parallelism to evaluate models in parallel.
setParallelism(int) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
Set the mamixum level of parallelism to evaluate models in parallel.
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.Bucketizer
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.IDFModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.ImputerModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.PCAModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
setParent(Estimator<M>) - Method in class org.apache.spark.ml.Model
Sets the parent of this model (Java API).
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.PipelineModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
setParent(Estimator<M>) - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
setPattern(String) - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
setPeacePeriod(int) - Method in class org.apache.spark.mllib.stat.test.StreamingTest
Set the number of initial batches to ignore.
setPercentile(double) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setPercentile(double) - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.classification.OneVsRest
 
setPredictionCol(String) - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setPredictionCol(String) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
setPredictionCol(String) - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
setPredictionCol(String) - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.clustering.KMeans
 
setPredictionCol(String) - Method in class org.apache.spark.ml.clustering.KMeansModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
setPredictionCol(String) - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
setPredictionCol(String) - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
setPredictionCol(String) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
setPredictionCol(String) - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.PredictionModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.Predictor
 
setPredictionCol(String) - Method in class org.apache.spark.ml.recommendation.ALS
 
setPredictionCol(String) - Method in class org.apache.spark.ml.recommendation.ALSModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
setPredictionCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
setPredictionCol(String) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
setPredictionCol(String) - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setPredictionCol(String) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
setProbabilityCol(String) - Method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
setProbabilityCol(String) - Method in class org.apache.spark.ml.classification.ProbabilisticClassifier
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setProbabilityCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setProbabilityCol(String) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
setProbabilityCol(String) - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
setProductBlocks(int) - Method in class org.apache.spark.mllib.recommendation.ALS
Set the number of product blocks to parallelize the computation.
setPropertiesFile(String) - Method in class org.apache.spark.launcher.AbstractLauncher
Set a custom properties file with Spark configuration for the application.
setPropertiesFile(String) - Method in class org.apache.spark.launcher.SparkLauncher
 
setQuantileCalculationStrategy(Enumeration.Value) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setQuantileProbabilities(double[]) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
setQuantileProbabilities(double[]) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
setQuantilesCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
setQuantilesCol(String) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
setRandomCenters(int, double, long) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Initialize random centers, requiring only the number of dimensions.
setRank(int) - Method in class org.apache.spark.ml.recommendation.ALS
 
setRank(int) - Method in class org.apache.spark.mllib.recommendation.ALS
Set the rank of the feature matrices computed (number of features).
setRatingCol(String) - Method in class org.apache.spark.ml.recommendation.ALS
 
setRawPredictionCol(String) - Method in class org.apache.spark.ml.classification.ClassificationModel
 
setRawPredictionCol(String) - Method in class org.apache.spark.ml.classification.Classifier
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setRawPredictionCol(String) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setRawPredictionCol(String) - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
setRegParam(double) - Method in class org.apache.spark.ml.classification.LinearSVC
Set the regularization parameter.
setRegParam(double) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the regularization parameter.
setRegParam(double) - Method in class org.apache.spark.ml.recommendation.ALS
 
setRegParam(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the regularization parameter for L2 regularization.
setRegParam(double) - Method in class org.apache.spark.ml.regression.LinearRegression
Set the regularization parameter.
setRegParam(double) - Method in class org.apache.spark.mllib.classification.StreamingLogisticRegressionWithSGD
Set the regularization parameter.
setRegParam(double) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Set the regularization parameter.
setRegParam(double) - Method in class org.apache.spark.mllib.optimization.LBFGS
Set the regularization parameter.
setRegParam(double) - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
Set the regularization parameter.
setRelativeError(double) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
setRequiredColumns(Configuration, StructType, StructType) - Static method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
setRest(long, int, VD, ED) - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
setRuns(int) - Method in class org.apache.spark.mllib.clustering.KMeans
Deprecated.
This has no effect. Since 2.1.0.
setSample(RDD<Object>) - Method in class org.apache.spark.mllib.stat.KernelDensity
Sets the sample to use for density estimation.
setSample(JavaRDD<Double>) - Method in class org.apache.spark.mllib.stat.KernelDensity
Sets the sample to use for density estimation (for Java users).
setScalingVec(Vector) - Method in class org.apache.spark.ml.feature.ElementwiseProduct
 
setSeed(long) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setSeed(long) - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setSeed(long) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setSeed(long) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setSeed(long) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Set the seed for weights initialization if weights are not set
setSeed(long) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setSeed(long) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setSeed(long) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
setSeed(long) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
setSeed(long) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
setSeed(long) - Method in class org.apache.spark.ml.clustering.KMeans
 
setSeed(long) - Method in class org.apache.spark.ml.clustering.LDA
 
setSeed(long) - Method in class org.apache.spark.ml.clustering.LDAModel
 
setSeed(long) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
setSeed(long) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
setSeed(long) - Method in class org.apache.spark.ml.feature.MinHashLSH
 
setSeed(long) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setSeed(long) - Method in class org.apache.spark.ml.recommendation.ALS
 
setSeed(long) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setSeed(long) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setSeed(long) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setSeed(long) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setSeed(long) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setSeed(long) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setSeed(long) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
setSeed(long) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
setSeed(long) - Method in class org.apache.spark.mllib.clustering.BisectingKMeans
Sets the random seed (default: hash value of the class name).
setSeed(long) - Method in class org.apache.spark.mllib.clustering.GaussianMixture
Set the random seed
setSeed(long) - Method in class org.apache.spark.mllib.clustering.KMeans
Set the random seed for cluster initialization.
setSeed(long) - Method in class org.apache.spark.mllib.clustering.LDA
Set the random seed for cluster initialization.
setSeed(long) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets random seed (default: a random long integer).
setSeed(long) - Method in class org.apache.spark.mllib.random.ExponentialGenerator
 
setSeed(long) - Method in class org.apache.spark.mllib.random.GammaGenerator
 
setSeed(long) - Method in class org.apache.spark.mllib.random.LogNormalGenerator
 
setSeed(long) - Method in class org.apache.spark.mllib.random.PoissonGenerator
 
setSeed(long) - Method in class org.apache.spark.mllib.random.StandardNormalGenerator
 
setSeed(long) - Method in class org.apache.spark.mllib.random.UniformGenerator
 
setSeed(long) - Method in class org.apache.spark.mllib.random.WeibullGenerator
 
setSeed(long) - Method in class org.apache.spark.mllib.recommendation.ALS
Sets a random seed to have deterministic results.
setSeed(long) - Method in class org.apache.spark.util.random.BernoulliCellSampler
 
setSeed(long) - Method in class org.apache.spark.util.random.BernoulliSampler
 
setSeed(long) - Method in class org.apache.spark.util.random.PoissonSampler
 
setSeed(long) - Method in interface org.apache.spark.util.random.Pseudorandom
Set random seed.
setSelectorType(String) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
setSelectorType(String) - Method in class org.apache.spark.mllib.feature.ChiSqSelector
 
setSerializer(Serializer) - Method in class org.apache.spark.rdd.CoGroupedRDD
Set a serializer for this RDD's shuffle, or null to use the default (spark.serializer)
setSerializer(Serializer) - Method in class org.apache.spark.rdd.ShuffledRDD
Set a serializer for this RDD's shuffle, or null to use the default (spark.serializer)
setSize(int) - Method in class org.apache.spark.ml.feature.VectorSizeHint
 
setSmoothing(double) - Method in class org.apache.spark.ml.classification.NaiveBayes
Set the smoothing parameter.
setSolver(String) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Sets the value of param solver.
setSolver(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the solver algorithm used for optimization.
setSolver(String) - Method in class org.apache.spark.ml.regression.LinearRegression
Set the solver algorithm used for optimization.
setSparkContextSessionConf(SparkSession, Map<Object, Object>) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
setSparkHome(String) - Method in class org.apache.spark.launcher.SparkLauncher
Set a custom Spark installation location for the application.
setSparkHome(String) - Method in class org.apache.spark.SparkConf
Set the location where Spark is installed on worker nodes.
setSplits(double[]) - Method in class org.apache.spark.ml.feature.Bucketizer
 
setSplitsArray(double[][]) - Method in class org.apache.spark.ml.feature.Bucketizer
 
setSQLReadObject(Function2<DataInputStream, Object, Object>) - Static method in class org.apache.spark.api.r.SerDe
 
setSQLWriteObject(Function2<DataOutputStream, Object, Object>) - Static method in class org.apache.spark.api.r.SerDe
 
setSrcOnly(long, int, VD) - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
setStackTrace(StackTraceElement[]) - Static method in exception org.apache.spark.sql.AnalysisException
 
setStages(PipelineStage[]) - Method in class org.apache.spark.ml.Pipeline
 
setStandardization(boolean) - Method in class org.apache.spark.ml.classification.LinearSVC
Whether to standardize the training features before fitting the model.
setStandardization(boolean) - Method in class org.apache.spark.ml.classification.LogisticRegression
Whether to standardize the training features before fitting the model.
setStandardization(boolean) - Method in class org.apache.spark.ml.regression.LinearRegression
Whether to standardize the training features before fitting the model.
setStartOffset(Optional<Offset>) - Method in interface org.apache.spark.sql.sources.v2.reader.streaming.ContinuousReader
Set the desired start offset for reader factories created from this reader.
setStatement(String) - Method in class org.apache.spark.ml.feature.SQLTransformer
 
setStepSize(double) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setStepSize(double) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setStepSize(double) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Sets the value of param stepSize (applicable only for solver "gd").
setStepSize(double) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setStepSize(double) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setStepSize(double) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setStepSize(double) - Method in class org.apache.spark.mllib.classification.StreamingLogisticRegressionWithSGD
Set the step size for gradient descent.
setStepSize(double) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Set the initial step size of SGD for the first step.
setStepSize(double) - Method in class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
Set the step size for gradient descent.
setStopWords(String[]) - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
setStorageLevel(String) - Method in class org.apache.spark.status.LiveRDD
 
setStrategy(String) - Method in class org.apache.spark.ml.feature.Imputer
Imputation strategy.
setStringIndexerOrderType(String) - Method in class org.apache.spark.ml.feature.RFormula
 
setStringOrderType(String) - Method in class org.apache.spark.ml.feature.StringIndexer
 
setSubsamplingRate(double) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setSubsamplingRate(double) - Method in class org.apache.spark.ml.classification.GBTClassifier
 
setSubsamplingRate(double) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setSubsamplingRate(double) - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setSubsamplingRate(double) - Method in class org.apache.spark.ml.clustering.LDA
 
setSubsamplingRate(double) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
setSubsamplingRate(double) - Method in class org.apache.spark.ml.regression.GBTRegressor
 
setSubsamplingRate(double) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
setSubsamplingRate(double) - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
setSubsamplingRate(double) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setTau0(double) - Method in class org.apache.spark.mllib.clustering.OnlineLDAOptimizer
A (positive) learning parameter that downweights early iterations.
setTestMethod(String) - Method in class org.apache.spark.mllib.stat.test.StreamingTest
Set the statistical method used for significance testing.
setThreshold(double) - Method in class org.apache.spark.ml.classification.LinearSVC
Set threshold in binary classification.
setThreshold(double) - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
setThreshold(double) - Method in class org.apache.spark.ml.classification.LogisticRegression
 
setThreshold(double) - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
setThreshold(double) - Method in class org.apache.spark.ml.feature.Binarizer
 
setThreshold(double) - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
Sets the threshold that separates positive predictions from negative predictions in Binary Logistic Regression.
setThreshold(double) - Method in class org.apache.spark.mllib.classification.SVMModel
Sets the threshold that separates positive predictions from negative predictions.
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
setThresholds(double[]) - Method in class org.apache.spark.ml.classification.LogisticRegression
 
setThresholds(double[]) - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
setThresholds(double[]) - Method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
setThresholds(double[]) - Method in class org.apache.spark.ml.classification.ProbabilisticClassifier
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
setThresholds(double[]) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
setTimeoutDuration(long) - Method in interface org.apache.spark.sql.streaming.GroupState
Set the timeout duration in ms for this key.
setTimeoutDuration(String) - Method in interface org.apache.spark.sql.streaming.GroupState
Set the timeout duration for this key as a string.
setTimeoutTimestamp(long) - Method in interface org.apache.spark.sql.streaming.GroupState
Set the timeout timestamp for this key as milliseconds in epoch time.
setTimeoutTimestamp(long, String) - Method in interface org.apache.spark.sql.streaming.GroupState
Set the timeout timestamp for this key as milliseconds in epoch time and an additional duration as a string (e.g.
setTimeoutTimestamp(Date) - Method in interface org.apache.spark.sql.streaming.GroupState
Set the timeout timestamp for this key as a java.sql.Date.
setTimeoutTimestamp(Date, String) - Method in interface org.apache.spark.sql.streaming.GroupState
Set the timeout timestamp for this key as a java.sql.Date and an additional duration as a string (e.g.
setTol(double) - Method in class org.apache.spark.ml.classification.LinearSVC
Set the convergence tolerance of iterations.
setTol(double) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the convergence tolerance of iterations.
setTol(double) - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
Set the convergence tolerance of iterations.
setTol(double) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
setTol(double) - Method in class org.apache.spark.ml.clustering.KMeans
 
setTol(double) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
Set the convergence tolerance of iterations.
setTol(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the convergence tolerance of iterations.
setTol(double) - Method in class org.apache.spark.ml.regression.LinearRegression
Set the convergence tolerance of iterations.
setToLowercase(boolean) - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
setTopicConcentration(double) - Method in class org.apache.spark.ml.clustering.LDA
 
setTopicConcentration(double) - Method in class org.apache.spark.mllib.clustering.LDA
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.
setTopicDistributionCol(String) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
setTopicDistributionCol(String) - Method in class org.apache.spark.ml.clustering.LDA
 
setTopicDistributionCol(String) - Method in class org.apache.spark.ml.clustering.LDAModel
 
setTopicDistributionCol(String) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
setTrainRatio(double) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
setTreeStrategy(Strategy) - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
setUiRoot(ContextHandler, UIRoot) - Static method in class org.apache.spark.status.api.v1.UIRootFromServletContext
 
setupCommitter(TaskAttemptContext) - Method in class org.apache.spark.internal.io.HadoopMapRedCommitProtocol
 
setUpdater(Updater) - Method in class org.apache.spark.mllib.optimization.GradientDescent
Set the updater function to actually perform a gradient step in a given direction.
setUpdater(Updater) - Method in class org.apache.spark.mllib.optimization.LBFGS
Set the updater function to actually perform a gradient step in a given direction.
SetupDriver(org.apache.spark.rpc.RpcEndpointRef) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SetupDriver
 
SetupDriver$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SetupDriver$
 
setupGroups(int, DefaultPartitionCoalescer.PartitionLocations) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
Initializes targetLen partition groups.
setupJob(JobContext) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Setups up a job.
setupJob(JobContext) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
setUpperBoundsOnCoefficients(Matrix) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the upper bounds on coefficients if fitting under bound constrained optimization.
setUpperBoundsOnIntercepts(Vector) - Method in class org.apache.spark.ml.classification.LogisticRegression
Set the upper bounds on intercepts if fitting under bound constrained optimization.
setupSecureURLConnection(URLConnection, org.apache.spark.SecurityManager) - Static method in class org.apache.spark.util.Utils
If the given URL connection is HttpsURLConnection, it sets the SSL socket factory and the host verifier from the given security manager.
setupTask(TaskAttemptContext) - Method in class org.apache.spark.internal.io.FileCommitProtocol
Sets up a task within a job.
setupTask(TaskAttemptContext) - Method in class org.apache.spark.internal.io.HadoopMapReduceCommitProtocol
 
setUseNodeIdCache(boolean) - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
setUserBlocks(int) - Method in class org.apache.spark.mllib.recommendation.ALS
Set the number of user blocks to parallelize the computation.
setUserCol(String) - Method in class org.apache.spark.ml.recommendation.ALS
 
setUserCol(String) - Method in class org.apache.spark.ml.recommendation.ALSModel
 
setValidateData(boolean) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
 
setValidateData(boolean) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
 
setValidateData(boolean) - Method in class org.apache.spark.mllib.regression.GeneralizedLinearAlgorithm
Set if the algorithm should validate data before training.
setValidateData(boolean) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
 
setValidateData(boolean) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
 
setValidateData(boolean) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
 
setValidationTol(double) - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
setValue(R) - Method in class org.apache.spark.Accumulable
Deprecated.
Set the accumulator's value.
setValue(R) - Static method in class org.apache.spark.Accumulator
Deprecated.
 
setVarianceCol(String) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
setVarianceCol(String) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
setVariancePower(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the value of param variancePower.
setVectorSize(int) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setVectorSize(int) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets vector size (default: 100).
setVerbose(boolean) - Method in class org.apache.spark.launcher.AbstractLauncher
Enables verbose reporting for SparkSubmit.
setVerbose(boolean) - Method in class org.apache.spark.launcher.SparkLauncher
 
setVocabSize(int) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
setWeightCol(String) - Method in class org.apache.spark.ml.classification.LinearSVC
Set the value of param weightCol.
setWeightCol(double) - Method in class org.apache.spark.ml.classification.LinearSVCModel
Deprecated.
This method is deprecated and will be removed in 3.0.0. Since 2.3.4.
setWeightCol(String) - Method in class org.apache.spark.ml.classification.LogisticRegression
Sets the value of param weightCol.
setWeightCol(String) - Method in class org.apache.spark.ml.classification.NaiveBayes
Sets the value of param weightCol.
setWeightCol(String) - Method in class org.apache.spark.ml.classification.OneVsRest
Sets the value of param weightCol.
setWeightCol(String) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
Sets the value of param weightCol.
setWeightCol(String) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
setWeightCol(String) - Method in class org.apache.spark.ml.regression.LinearRegression
Whether to over-/under-sample training instances according to the given weights in weightCol.
setWindowSize(int) - Method in class org.apache.spark.ml.feature.Word2Vec
 
setWindowSize(int) - Method in class org.apache.spark.mllib.feature.Word2Vec
Sets the window of words (default: 5)
setWindowSize(int) - Method in class org.apache.spark.mllib.stat.test.StreamingTest
Set the number of batches to compute significance tests over.
setWithMean(boolean) - Method in class org.apache.spark.ml.feature.StandardScaler
 
setWithMean(boolean) - Method in class org.apache.spark.mllib.feature.StandardScalerModel
Developer API
setWithStd(boolean) - Method in class org.apache.spark.ml.feature.StandardScaler
 
setWithStd(boolean) - Method in class org.apache.spark.mllib.feature.StandardScalerModel
Developer API
sha1(Column) - Static method in class org.apache.spark.sql.functions
Calculates the SHA-1 digest of a binary column and returns the value as a 40 character hex string.
sha2(Column, int) - Static method in class org.apache.spark.sql.functions
Calculates the SHA-2 family of hash functions of a binary column and returns the value as a hex string.
shape() - Method in class org.apache.spark.mllib.random.GammaGenerator
 
SharedParamsCodeGen - Class in org.apache.spark.ml.param.shared
Code generator for shared params (sharedParams.scala).
SharedParamsCodeGen() - Constructor for class org.apache.spark.ml.param.shared.SharedParamsCodeGen
 
SharedReadWrite$() - Constructor for class org.apache.spark.ml.Pipeline.SharedReadWrite$
 
sharedState() - Method in class org.apache.spark.sql.SparkSession
State shared across sessions, including the SparkContext, cached data, listener, and a catalog that interacts with external systems.
shiftLeft(Column, int) - Static method in class org.apache.spark.sql.functions
Shift the given value numBits left.
shiftRight(Column, int) - Static method in class org.apache.spark.sql.functions
(Signed) shift the given value numBits right.
shiftRightUnsigned(Column, int) - Static method in class org.apache.spark.sql.functions
Unsigned shift the given value numBits right.
SHORT() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable short type.
ShortestPaths - Class in org.apache.spark.graphx.lib
Computes shortest paths to the given set of landmark vertices, returning a graph where each vertex attribute is a map containing the shortest-path distance to each reachable landmark.
ShortestPaths() - Constructor for class org.apache.spark.graphx.lib.ShortestPaths
 
shortName() - Method in class org.apache.spark.sql.hive.execution.HiveFileFormat
 
shortName() - Method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
shortName() - Method in interface org.apache.spark.sql.sources.DataSourceRegister
The string that represents the format that this data source provider uses.
shortTimeUnitString(TimeUnit) - Static method in class org.apache.spark.streaming.ui.UIUtils
Return the short string for a TimeUnit.
ShortType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the ShortType object.
ShortType - Class in org.apache.spark.sql.types
The data type representing Short values.
shouldCloseFileAfterWrite(SparkConf, boolean) - Static method in class org.apache.spark.streaming.util.WriteAheadLogUtils
 
shouldDistributeGaussians(int, int) - Static method in class org.apache.spark.mllib.clustering.GaussianMixture
Heuristic to distribute the computation of the MultivariateGaussians, approximately when d is greater than 25 except for when k is very small.
shouldGoLeft(Vector) - Method in interface org.apache.spark.ml.tree.Split
Return true (split to left) or false (split to right).
shouldGoLeft(int, Split[]) - Method in interface org.apache.spark.ml.tree.Split
Return true (split to left) or false (split to right).
shouldOwn(Param<?>) - Method in interface org.apache.spark.ml.param.Params
Validates that the input param belongs to this instance.
show(int) - Method in class org.apache.spark.sql.Dataset
Displays the Dataset in a tabular form.
show() - Method in class org.apache.spark.sql.Dataset
Displays the top 20 rows of Dataset in a tabular form.
show(boolean) - Method in class org.apache.spark.sql.Dataset
Displays the top 20 rows of Dataset in a tabular form.
show(int, boolean) - Method in class org.apache.spark.sql.Dataset
Displays the Dataset in a tabular form.
show(int, int) - Method in class org.apache.spark.sql.Dataset
Displays the Dataset in a tabular form.
show(int, int, boolean) - Method in class org.apache.spark.sql.Dataset
Displays the Dataset in a tabular form.
showBytesDistribution(String, Function2<TaskInfo, TaskMetrics, Object>, Seq<Tuple2<TaskInfo, TaskMetrics>>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showBytesDistribution(String, Option<org.apache.spark.util.Distribution>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showBytesDistribution(String, org.apache.spark.util.Distribution) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showDagVizForJob(int, Seq<org.apache.spark.ui.scope.RDDOperationGraph>) - Static method in class org.apache.spark.ui.UIUtils
Return a "DAG visualization" DOM element that expands into a visualization for a job.
showDagVizForStage(int, Option<org.apache.spark.ui.scope.RDDOperationGraph>) - Static method in class org.apache.spark.ui.UIUtils
Return a "DAG visualization" DOM element that expands into a visualization for a stage.
showDistribution(String, org.apache.spark.util.Distribution, Function1<Object, String>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showDistribution(String, Option<org.apache.spark.util.Distribution>, Function1<Object, String>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showDistribution(String, Option<org.apache.spark.util.Distribution>, String) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showDistribution(String, String, Function2<TaskInfo, TaskMetrics, Object>, Seq<Tuple2<TaskInfo, TaskMetrics>>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showMillisDistribution(String, Option<org.apache.spark.util.Distribution>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showMillisDistribution(String, Function2<TaskInfo, TaskMetrics, Object>, Seq<Tuple2<TaskInfo, TaskMetrics>>) - Static method in class org.apache.spark.scheduler.StatsReportListener
 
showMillisDistribution(String, Function1<BatchInfo, Option<Object>>) - Method in class org.apache.spark.streaming.scheduler.StatsReportListener
 
SHUFFLE() - Static method in class org.apache.spark.storage.BlockId
 
SHUFFLE_DATA() - Static method in class org.apache.spark.storage.BlockId
 
SHUFFLE_INDEX() - Static method in class org.apache.spark.storage.BlockId
 
SHUFFLE_LOCAL_BLOCKS() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_READ() - Static method in class org.apache.spark.ui.ToolTips
 
SHUFFLE_READ_BLOCKED_TIME() - Static method in class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
SHUFFLE_READ_BLOCKED_TIME() - Static method in class org.apache.spark.ui.ToolTips
 
SHUFFLE_READ_METRICS_PREFIX() - Static method in class org.apache.spark.InternalAccumulator
 
SHUFFLE_READ_RECORDS() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_READ_REMOTE_SIZE() - Static method in class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
SHUFFLE_READ_REMOTE_SIZE() - Static method in class org.apache.spark.ui.ToolTips
 
SHUFFLE_READ_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_REMOTE_BLOCKS() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_REMOTE_READS() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_REMOTE_READS_TO_DISK() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_TOTAL_BLOCKS() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_TOTAL_READS() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_WRITE() - Static method in class org.apache.spark.ui.ToolTips
 
SHUFFLE_WRITE_METRICS_PREFIX() - Static method in class org.apache.spark.InternalAccumulator
 
SHUFFLE_WRITE_RECORDS() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_WRITE_SIZE() - Static method in class org.apache.spark.status.TaskIndexNames
 
SHUFFLE_WRITE_TIME() - Static method in class org.apache.spark.status.TaskIndexNames
 
ShuffleBlockId - Class in org.apache.spark.storage
 
ShuffleBlockId(int, int, int) - Constructor for class org.apache.spark.storage.ShuffleBlockId
 
ShuffleDataBlockId - Class in org.apache.spark.storage
 
ShuffleDataBlockId(int, int, int) - Constructor for class org.apache.spark.storage.ShuffleDataBlockId
 
ShuffleDependency<K,V,C> - Class in org.apache.spark
Developer API Represents a dependency on the output of a shuffle stage.
ShuffleDependency(RDD<? extends Product2<K, V>>, Partitioner, Serializer, Option<Ordering<K>>, Option<Aggregator<K, V, C>>, boolean, ClassTag<K>, ClassTag<V>, ClassTag<C>) - Constructor for class org.apache.spark.ShuffleDependency
 
ShuffledRDD<K,V,C> - Class in org.apache.spark.rdd
Developer API The resulting RDD from a shuffle (e.g.
ShuffledRDD(RDD<? extends Product2<K, V>>, Partitioner, ClassTag<K>, ClassTag<V>, ClassTag<C>) - Constructor for class org.apache.spark.rdd.ShuffledRDD
 
shuffleHandle() - Method in class org.apache.spark.ShuffleDependency
 
shuffleId() - Method in class org.apache.spark.CleanShuffle
 
shuffleId() - Method in class org.apache.spark.FetchFailed
 
shuffleId() - Method in class org.apache.spark.ShuffleDependency
 
shuffleId() - Method in class org.apache.spark.storage.BlockManagerMessages.RemoveShuffle
 
shuffleId() - Method in class org.apache.spark.storage.ShuffleBlockId
 
shuffleId() - Method in class org.apache.spark.storage.ShuffleDataBlockId
 
shuffleId() - Method in class org.apache.spark.storage.ShuffleIndexBlockId
 
ShuffleIndexBlockId - Class in org.apache.spark.storage
 
ShuffleIndexBlockId(int, int, int) - Constructor for class org.apache.spark.storage.ShuffleIndexBlockId
 
shuffleManager() - Method in class org.apache.spark.SparkEnv
 
shuffleRead() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
shuffleRead$() - Constructor for class org.apache.spark.InternalAccumulator.shuffleRead$
 
shuffleReadBytes() - Method in class org.apache.spark.status.api.v1.StageData
 
ShuffleReadMetricDistributions - Class in org.apache.spark.status.api.v1
 
ShuffleReadMetrics - Class in org.apache.spark.status.api.v1
 
shuffleReadMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
shuffleReadMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
shuffleReadRecords() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
shuffleReadRecords() - Method in class org.apache.spark.status.api.v1.StageData
 
ShuffleStatus - Class in org.apache.spark
Helper class used by the MapOutputTrackerMaster to perform bookkeeping for a single ShuffleMapStage.
ShuffleStatus(int) - Constructor for class org.apache.spark.ShuffleStatus
 
shuffleWrite() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
shuffleWrite$() - Constructor for class org.apache.spark.InternalAccumulator.shuffleWrite$
 
shuffleWriteBytes() - Method in class org.apache.spark.status.api.v1.StageData
 
ShuffleWriteMetricDistributions - Class in org.apache.spark.status.api.v1
 
ShuffleWriteMetrics - Class in org.apache.spark.status.api.v1
 
shuffleWriteMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetricDistributions
 
shuffleWriteMetrics() - Method in class org.apache.spark.status.api.v1.TaskMetrics
 
shuffleWriteRecords() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
shuffleWriteRecords() - Method in class org.apache.spark.status.api.v1.StageData
 
Shutdown$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.Shutdown$
 
ShutdownHookManager - Class in org.apache.spark.util
Various utility methods used by Spark.
ShutdownHookManager() - Constructor for class org.apache.spark.util.ShutdownHookManager
 
sigma() - Method in class org.apache.spark.mllib.stat.distribution.MultivariateGaussian
 
sigmas() - Method in class org.apache.spark.mllib.clustering.ExpectationSum
 
SignalUtils - Class in org.apache.spark.util
Contains utilities for working with posix signals.
SignalUtils() - Constructor for class org.apache.spark.util.SignalUtils
 
signum(Column) - Static method in class org.apache.spark.sql.functions
Computes the signum of the given value.
signum(String) - Static method in class org.apache.spark.sql.functions
Computes the signum of the given column.
SimpleFutureAction<T> - Class in org.apache.spark
A FutureAction holding the result of an action that triggers a single job.
simpleString() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
simpleString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
simpleString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
simpleString() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
simpleString() - Method in class org.apache.spark.sql.types.ArrayType
 
simpleString() - Static method in class org.apache.spark.sql.types.BinaryType
 
simpleString() - Static method in class org.apache.spark.sql.types.BooleanType
 
simpleString() - Method in class org.apache.spark.sql.types.ByteType
 
simpleString() - Static method in class org.apache.spark.sql.types.CalendarIntervalType
 
simpleString() - Method in class org.apache.spark.sql.types.CharType
 
simpleString() - Method in class org.apache.spark.sql.types.DataType
Readable string representation for the type.
simpleString() - Static method in class org.apache.spark.sql.types.DateType
 
simpleString() - Method in class org.apache.spark.sql.types.DecimalType
 
simpleString() - Static method in class org.apache.spark.sql.types.DoubleType
 
simpleString() - Static method in class org.apache.spark.sql.types.FloatType
 
simpleString() - Static method in class org.apache.spark.sql.types.HiveStringType
 
simpleString() - Method in class org.apache.spark.sql.types.IntegerType
 
simpleString() - Method in class org.apache.spark.sql.types.LongType
 
simpleString() - Method in class org.apache.spark.sql.types.MapType
 
simpleString() - Static method in class org.apache.spark.sql.types.NullType
 
simpleString() - Method in class org.apache.spark.sql.types.ObjectType
 
simpleString() - Method in class org.apache.spark.sql.types.ShortType
 
simpleString() - Static method in class org.apache.spark.sql.types.StringType
 
simpleString() - Method in class org.apache.spark.sql.types.StructType
 
simpleString() - Static method in class org.apache.spark.sql.types.TimestampType
 
simpleString() - Method in class org.apache.spark.sql.types.VarcharType
 
SimpleUpdater - Class in org.apache.spark.mllib.optimization
Developer API A simple updater for gradient descent *without* any regularization.
SimpleUpdater() - Constructor for class org.apache.spark.mllib.optimization.SimpleUpdater
 
sin(Column) - Static method in class org.apache.spark.sql.functions
 
sin(String) - Static method in class org.apache.spark.sql.functions
 
SingularValueDecomposition<UType,VType> - Class in org.apache.spark.mllib.linalg
Represents singular value decomposition (SVD) factors.
SingularValueDecomposition(UType, Vector, VType) - Constructor for class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
sinh(Column) - Static method in class org.apache.spark.sql.functions
 
sinh(String) - Static method in class org.apache.spark.sql.functions
 
sink() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
SinkProgress - Class in org.apache.spark.sql.streaming
Information about progress made for a sink in the execution of a StreamingQuery during a trigger.
size() - Method in class org.apache.spark.api.java.JavaUtils.SerializableMapWrapper
 
size() - Method in class org.apache.spark.ml.attribute.AttributeGroup
Size of the attribute group.
size() - Method in class org.apache.spark.ml.feature.VectorSizeHint
The size of Vectors in inputCol.
size() - Method in class org.apache.spark.ml.linalg.DenseVector
 
size() - Method in class org.apache.spark.ml.linalg.SparseVector
 
size() - Method in interface org.apache.spark.ml.linalg.Vector
Size of the vector.
size() - Method in class org.apache.spark.ml.param.ParamMap
Number of param pairs in this map.
size() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
size() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
size() - Method in interface org.apache.spark.mllib.linalg.Vector
Size of the vector.
size(Column) - Static method in class org.apache.spark.sql.functions
Returns length of array or map.
size() - Method in interface org.apache.spark.sql.Row
Number of elements in the Row.
size() - Static method in class org.apache.spark.sql.types.StructType
 
size() - Method in class org.apache.spark.storage.DiskBlockData
 
size() - Method in class org.apache.spark.storage.EncryptedBlockData
 
size() - Method in class org.apache.spark.storage.EncryptedManagedBuffer
 
size() - Method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
size() - Method in interface org.apache.spark.storage.memory.MemoryEntry
 
size() - Method in class org.apache.spark.storage.memory.SerializedMemoryEntry
 
SizeEstimator - Class in org.apache.spark.util
Developer API Estimates the sizes of Java objects (number of bytes of memory they occupy), for use in memory-aware caches.
SizeEstimator() - Constructor for class org.apache.spark.util.SizeEstimator
 
sizeInBytes() - Method in class org.apache.spark.sql.sources.BaseRelation
Returns an estimated size of this relation in bytes.
sizeInBytes() - Method in interface org.apache.spark.sql.sources.v2.reader.Statistics
 
sketch(RDD<K>, int, ClassTag<K>) - Static method in class org.apache.spark.RangePartitioner
Sketches the input RDD via reservoir sampling on each partition.
skewness(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the skewness of the values in a group.
skewness(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the skewness of the values in a group.
skip(long) - Method in class org.apache.spark.io.NioBufferedFileInputStream
 
skip(long) - Method in class org.apache.spark.io.ReadAheadInputStream
 
skip(long) - Method in class org.apache.spark.storage.BufferReleasingInputStream
 
skippedStages() - Method in class org.apache.spark.status.LiveJob
 
skippedTasks() - Method in class org.apache.spark.status.LiveJob
 
skipWhitespace() - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
slice(int, int) - Static method in class org.apache.spark.sql.types.StructType
 
slice(Time, Time) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
slice(Time, Time) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return all the RDDs between 'fromDuration' to 'toDuration' (both included)
slice(Time, Time) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
slice(Time, Time) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
slice(Time, Time) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
slice(Time, Time) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
slice(Time, Time) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
slice(org.apache.spark.streaming.Interval) - Method in class org.apache.spark.streaming.dstream.DStream
Return all the RDDs defined by the Interval object (both end times included)
slice(Time, Time) - Method in class org.apache.spark.streaming.dstream.DStream
Return all the RDDs between 'fromTime' to 'toTime' (both included)
slideDuration() - Method in class org.apache.spark.streaming.dstream.DStream
Time interval after which the DStream generates an RDD
slideDuration() - Method in class org.apache.spark.streaming.dstream.InputDStream
 
sliding(int, int) - Method in class org.apache.spark.mllib.rdd.RDDFunctions
Returns an RDD from grouping items of its parent RDD in fixed size blocks by passing a sliding window over them.
sliding(int) - Method in class org.apache.spark.mllib.rdd.RDDFunctions
sliding(Int, Int)* with step = 1.
sliding(int) - Static method in class org.apache.spark.sql.types.StructType
 
sliding(int, int) - Static method in class org.apache.spark.sql.types.StructType
 
smoothing() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
smoothing() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
SnappyCompressionCodec - Class in org.apache.spark.io
Developer API Snappy implementation of CompressionCodec.
SnappyCompressionCodec(SparkConf) - Constructor for class org.apache.spark.io.SnappyCompressionCodec
 
SnappyOutputStreamWrapper - Class in org.apache.spark.io
Wrapper over SnappyOutputStream which guards against write-after-close and double-close issues.
SnappyOutputStreamWrapper(SnappyOutputStream) - Constructor for class org.apache.spark.io.SnappyOutputStreamWrapper
 
socketStream(String, int, Function<InputStream, Iterable<T>>, StorageLevel) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from network source hostname:port.
socketStream(String, int, Function1<InputStream, Iterator<T>>, StorageLevel, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Creates an input stream from TCP source hostname:port.
socketTextStream(String, int, StorageLevel) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from network source hostname:port.
socketTextStream(String, int) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream from network source hostname:port.
socketTextStream(String, int, StorageLevel) - Method in class org.apache.spark.streaming.StreamingContext
Creates an input stream from TCP source hostname:port.
solve(double[], double[]) - Static method in class org.apache.spark.mllib.linalg.CholeskyDecomposition
Solves a symmetric positive definite linear system via Cholesky factorization.
solve(double[], double[], NNLS.Workspace) - Static method in class org.apache.spark.mllib.optimization.NNLS
Solve a least squares problem, possibly with nonnegativity constraints, by a modified projected gradient method.
solver() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
solver() - Method in interface org.apache.spark.ml.param.shared.HasSolver
Param for the solver algorithm for optimization.
solver() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
solver() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
solver() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionTrainingSummary
 
solver() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
solver() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
Sort() - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
sort(String, String...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the specified column, all in ascending order.
sort(Column...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the given expressions.
sort(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the specified column, all in ascending order.
sort(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset sorted by the given expressions.
sort_array(Column) - Static method in class org.apache.spark.sql.functions
Sorts the input array for the given column in ascending order, according to the natural ordering of the array elements.
sort_array(Column, boolean) - Static method in class org.apache.spark.sql.functions
Sorts the input array for the given column in ascending or descending order, according to the natural ordering of the array elements.
sortBy(Function<T, S>, boolean, int) - Method in class org.apache.spark.api.java.JavaRDD
Return this RDD sorted by the given key function.
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.api.r.RRDD
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.graphx.VertexRDD
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Method in class org.apache.spark.rdd.RDD
Return this RDD sorted by the given key function.
sortBy(Function1<T, K>, boolean, int, Ordering<K>, ClassTag<K>) - Static method in class org.apache.spark.rdd.UnionRDD
 
sortBy(String, String...) - Method in class org.apache.spark.sql.DataFrameWriter
Sorts the output in each bucket by the given columns.
sortBy(String, Seq<String>) - Method in class org.apache.spark.sql.DataFrameWriter
Sorts the output in each bucket by the given columns.
sortBy(Function1<A, B>, Ordering<B>) - Static method in class org.apache.spark.sql.types.StructType
 
sortBy$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
sortBy$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
sortBy$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
sortBy$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
sortBy$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
sortBy$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
sortBy$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
sortBy$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
sortBy$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
sortBy$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
sortBy$default$3() - Static method in class org.apache.spark.api.r.RRDD
 
sortBy$default$3() - Static method in class org.apache.spark.graphx.EdgeRDD
 
sortBy$default$3() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
sortBy$default$3() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
sortBy$default$3() - Static method in class org.apache.spark.graphx.VertexRDD
 
sortBy$default$3() - Static method in class org.apache.spark.rdd.HadoopRDD
 
sortBy$default$3() - Static method in class org.apache.spark.rdd.JdbcRDD
 
sortBy$default$3() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
sortBy$default$3() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
sortBy$default$3() - Static method in class org.apache.spark.rdd.UnionRDD
 
sortByKey() - Method in class org.apache.spark.api.java.JavaPairRDD
Sort the RDD by key, so that each partition contains a sorted range of the elements in ascending order.
sortByKey(boolean) - Method in class org.apache.spark.api.java.JavaPairRDD
Sort the RDD by key, so that each partition contains a sorted range of the elements.
sortByKey(boolean, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Sort the RDD by key, so that each partition contains a sorted range of the elements.
sortByKey(Comparator<K>) - Method in class org.apache.spark.api.java.JavaPairRDD
Sort the RDD by key, so that each partition contains a sorted range of the elements.
sortByKey(Comparator<K>, boolean) - Method in class org.apache.spark.api.java.JavaPairRDD
Sort the RDD by key, so that each partition contains a sorted range of the elements.
sortByKey(Comparator<K>, boolean, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Sort the RDD by key, so that each partition contains a sorted range of the elements.
sortByKey(boolean, int) - Method in class org.apache.spark.rdd.OrderedRDDFunctions
Sort the RDD by key, so that each partition contains a sorted range of the elements.
sorted(Ordering<B>) - Static method in class org.apache.spark.sql.types.StructType
 
sortWith(Function2<A, A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
sortWithinPartitions(String, String...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with each partition sorted by the given expressions.
sortWithinPartitions(Column...) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with each partition sorted by the given expressions.
sortWithinPartitions(String, Seq<String>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with each partition sorted by the given expressions.
sortWithinPartitions(Seq<Column>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with each partition sorted by the given expressions.
soundex(Column) - Static method in class org.apache.spark.sql.functions
Returns the soundex code for the specified expression.
sourceName() - Static method in class org.apache.spark.metrics.source.CodegenMetrics
 
sourceName() - Static method in class org.apache.spark.metrics.source.HiveCatalogMetrics
 
SourceProgress - Class in org.apache.spark.sql.streaming
Information about progress made for a source in the execution of a StreamingQuery during a trigger.
sources() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
sourceSchema(SQLContext, Option<StructType>, String, Map<String, String>) - Method in interface org.apache.spark.sql.sources.StreamSourceProvider
Returns the name and schema of the source that can be used to continually read data.
span(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
spark() - Method in class org.apache.spark.status.api.v1.VersionInfo
 
SPARK_CONNECTOR_NAME() - Static method in class org.apache.spark.ui.JettyUtils
 
SPARK_CONTEXT_SHUTDOWN_PRIORITY() - Static method in class org.apache.spark.util.ShutdownHookManager
The shutdown priority of the SparkContext instance.
SPARK_IO_ENCRYPTION_COMMONS_CONFIG_PREFIX() - Static method in class org.apache.spark.security.CryptoStreamUtils
 
SPARK_MASTER - Static variable in class org.apache.spark.launcher.SparkLauncher
The Spark master.
spark_partition_id() - Static method in class org.apache.spark.sql.functions
Partition ID.
SPARK_REGEX() - Static method in class org.apache.spark.SparkMasterRegex
 
SparkAppConfig(Seq<Tuple2<String, String>>, Option<byte[]>, Option<byte[]>) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SparkAppConfig
 
SparkAppConfig$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SparkAppConfig$
 
SparkAppHandle - Interface in org.apache.spark.launcher
A handle to a running Spark application.
SparkAppHandle.Listener - Interface in org.apache.spark.launcher
Listener for updates to a handle's state.
SparkAppHandle.State - Enum in org.apache.spark.launcher
Represents the application's state.
SparkConf - Class in org.apache.spark
Configuration for a Spark application.
SparkConf(boolean) - Constructor for class org.apache.spark.SparkConf
 
SparkConf() - Constructor for class org.apache.spark.SparkConf
Create a SparkConf that loads defaults from system properties and the classpath
sparkContext() - Static method in class org.apache.spark.api.r.RRDD
 
sparkContext() - Static method in class org.apache.spark.graphx.EdgeRDD
 
sparkContext() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
sparkContext() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
sparkContext() - Static method in class org.apache.spark.graphx.VertexRDD
 
sparkContext() - Static method in class org.apache.spark.rdd.HadoopRDD
 
sparkContext() - Static method in class org.apache.spark.rdd.JdbcRDD
 
sparkContext() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
sparkContext() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
sparkContext() - Method in class org.apache.spark.rdd.RDD
The SparkContext that created this RDD.
sparkContext() - Static method in class org.apache.spark.rdd.UnionRDD
 
SparkContext - Class in org.apache.spark
Main entry point for Spark functionality.
SparkContext(SparkConf) - Constructor for class org.apache.spark.SparkContext
 
SparkContext() - Constructor for class org.apache.spark.SparkContext
Create a SparkContext that loads settings from system properties (for instance, when launching with ./bin/spark-submit).
SparkContext(String, String, SparkConf) - Constructor for class org.apache.spark.SparkContext
Alternative constructor that allows setting common Spark properties directly
SparkContext(String, String, String, Seq<String>, Map<String, String>) - Constructor for class org.apache.spark.SparkContext
Alternative constructor that allows setting common Spark properties directly
sparkContext() - Method in class org.apache.spark.sql.SparkSession
 
sparkContext() - Method in class org.apache.spark.sql.SQLContext
 
sparkContext() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
The underlying SparkContext
sparkContext() - Method in class org.apache.spark.streaming.StreamingContext
Return the associated Spark context
SparkEnv - Class in org.apache.spark
Developer API Holds all the runtime environment objects for a running Spark instance (either master or worker), including the serializer, RpcEnv, block manager, map output tracker, etc.
SparkEnv(String, org.apache.spark.rpc.RpcEnv, Serializer, Serializer, SerializerManager, MapOutputTracker, ShuffleManager, org.apache.spark.broadcast.BroadcastManager, BlockManager, SecurityManager, org.apache.spark.metrics.MetricsSystem, MemoryManager, OutputCommitCoordinator, SparkConf) - Constructor for class org.apache.spark.SparkEnv
 
sparkEventFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
sparkEventToJson(SparkListenerEvent) - Static method in class org.apache.spark.util.JsonProtocol
------------------------------------------------- * JSON serialization methods for SparkListenerEvents |
SparkException - Exception in org.apache.spark
 
SparkException(String, Throwable) - Constructor for exception org.apache.spark.SparkException
 
SparkException(String) - Constructor for exception org.apache.spark.SparkException
 
SparkExecutorInfo - Interface in org.apache.spark
Exposes information about Spark Executors.
SparkExecutorInfoImpl - Class in org.apache.spark
 
SparkExecutorInfoImpl(String, int, long, int) - Constructor for class org.apache.spark.SparkExecutorInfoImpl
 
SparkExitCode - Class in org.apache.spark.util
 
SparkExitCode() - Constructor for class org.apache.spark.util.SparkExitCode
 
SparkFiles - Class in org.apache.spark
Resolves paths to files added through SparkContext.addFile().
SparkFiles() - Constructor for class org.apache.spark.SparkFiles
 
SparkFirehoseListener - Class in org.apache.spark
Class that allows users to receive all SparkListener events.
SparkFirehoseListener() - Constructor for class org.apache.spark.SparkFirehoseListener
 
SparkHadoopMapRedUtil - Class in org.apache.spark.mapred
 
SparkHadoopMapRedUtil() - Constructor for class org.apache.spark.mapred.SparkHadoopMapRedUtil
 
SparkHadoopWriter - Class in org.apache.spark.internal.io
A helper object that saves an RDD using a Hadoop OutputFormat.
SparkHadoopWriter() - Constructor for class org.apache.spark.internal.io.SparkHadoopWriter
 
SparkHadoopWriterUtils - Class in org.apache.spark.internal.io
A helper object that provide common utils used during saving an RDD using a Hadoop OutputFormat (both from the old mapred API and the new mapreduce API)
SparkHadoopWriterUtils() - Constructor for class org.apache.spark.internal.io.SparkHadoopWriterUtils
 
sparkJavaOpts(SparkConf, Function1<String, Object>) - Static method in class org.apache.spark.util.Utils
Convert all spark properties set in the given SparkConf to a sequence of java options.
SparkJobInfo - Interface in org.apache.spark
Exposes information about Spark Jobs.
SparkJobInfoImpl - Class in org.apache.spark
 
SparkJobInfoImpl(int, int[], JobExecutionStatus) - Constructor for class org.apache.spark.SparkJobInfoImpl
 
SparkLauncher - Class in org.apache.spark.launcher
Launcher for Spark applications.
SparkLauncher() - Constructor for class org.apache.spark.launcher.SparkLauncher
 
SparkLauncher(Map<String, String>) - Constructor for class org.apache.spark.launcher.SparkLauncher
Creates a launcher that will set the given environment variables in the child.
SparkListener - Class in org.apache.spark.scheduler
Developer API A default implementation for SparkListenerInterface that has no-op implementations for all callbacks.
SparkListener() - Constructor for class org.apache.spark.scheduler.SparkListener
 
SparkListenerApplicationEnd - Class in org.apache.spark.scheduler
 
SparkListenerApplicationEnd(long) - Constructor for class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
SparkListenerApplicationStart - Class in org.apache.spark.scheduler
 
SparkListenerApplicationStart(String, Option<String>, long, String, Option<String>, Option<Map<String, String>>) - Constructor for class org.apache.spark.scheduler.SparkListenerApplicationStart
 
SparkListenerBlockManagerAdded - Class in org.apache.spark.scheduler
 
SparkListenerBlockManagerAdded(long, BlockManagerId, long, Option<Object>, Option<Object>) - Constructor for class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
SparkListenerBlockManagerRemoved - Class in org.apache.spark.scheduler
 
SparkListenerBlockManagerRemoved(long, BlockManagerId) - Constructor for class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
SparkListenerBlockUpdated - Class in org.apache.spark.scheduler
 
SparkListenerBlockUpdated(BlockUpdatedInfo) - Constructor for class org.apache.spark.scheduler.SparkListenerBlockUpdated
 
SparkListenerEnvironmentUpdate - Class in org.apache.spark.scheduler
 
SparkListenerEnvironmentUpdate(Map<String, Seq<Tuple2<String, String>>>) - Constructor for class org.apache.spark.scheduler.SparkListenerEnvironmentUpdate
 
SparkListenerEvent - Interface in org.apache.spark.scheduler
 
SparkListenerExecutorAdded - Class in org.apache.spark.scheduler
 
SparkListenerExecutorAdded(long, String, ExecutorInfo) - Constructor for class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
SparkListenerExecutorBlacklisted - Class in org.apache.spark.scheduler
 
SparkListenerExecutorBlacklisted(long, String, int) - Constructor for class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
SparkListenerExecutorMetricsUpdate - Class in org.apache.spark.scheduler
Periodic updates from executors.
SparkListenerExecutorMetricsUpdate(String, Seq<Tuple4<Object, Object, Object, Seq<AccumulableInfo>>>) - Constructor for class org.apache.spark.scheduler.SparkListenerExecutorMetricsUpdate
 
SparkListenerExecutorRemoved - Class in org.apache.spark.scheduler
 
SparkListenerExecutorRemoved(long, String, String) - Constructor for class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
SparkListenerExecutorUnblacklisted - Class in org.apache.spark.scheduler
 
SparkListenerExecutorUnblacklisted(long, String) - Constructor for class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
SparkListenerJobEnd - Class in org.apache.spark.scheduler
 
SparkListenerJobEnd(int, long, JobResult) - Constructor for class org.apache.spark.scheduler.SparkListenerJobEnd
 
SparkListenerJobStart - Class in org.apache.spark.scheduler
 
SparkListenerJobStart(int, long, Seq<StageInfo>, Properties) - Constructor for class org.apache.spark.scheduler.SparkListenerJobStart
 
SparkListenerLogStart - Class in org.apache.spark.scheduler
An internal class that describes the metadata of an event log.
SparkListenerLogStart(String) - Constructor for class org.apache.spark.scheduler.SparkListenerLogStart
 
SparkListenerNodeBlacklisted - Class in org.apache.spark.scheduler
 
SparkListenerNodeBlacklisted(long, String, int) - Constructor for class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
SparkListenerNodeUnblacklisted - Class in org.apache.spark.scheduler
 
SparkListenerNodeUnblacklisted(long, String) - Constructor for class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
SparkListenerSpeculativeTaskSubmitted - Class in org.apache.spark.scheduler
 
SparkListenerSpeculativeTaskSubmitted(int) - Constructor for class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
SparkListenerStageCompleted - Class in org.apache.spark.scheduler
 
SparkListenerStageCompleted(StageInfo) - Constructor for class org.apache.spark.scheduler.SparkListenerStageCompleted
 
SparkListenerStageSubmitted - Class in org.apache.spark.scheduler
 
SparkListenerStageSubmitted(StageInfo, Properties) - Constructor for class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
SparkListenerTaskEnd - Class in org.apache.spark.scheduler
 
SparkListenerTaskEnd(int, int, String, TaskEndReason, TaskInfo, TaskMetrics) - Constructor for class org.apache.spark.scheduler.SparkListenerTaskEnd
 
SparkListenerTaskGettingResult - Class in org.apache.spark.scheduler
 
SparkListenerTaskGettingResult(TaskInfo) - Constructor for class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
SparkListenerTaskStart - Class in org.apache.spark.scheduler
 
SparkListenerTaskStart(int, int, TaskInfo) - Constructor for class org.apache.spark.scheduler.SparkListenerTaskStart
 
SparkListenerUnpersistRDD - Class in org.apache.spark.scheduler
 
SparkListenerUnpersistRDD(int) - Constructor for class org.apache.spark.scheduler.SparkListenerUnpersistRDD
 
SparkMasterRegex - Class in org.apache.spark
A collection of regexes for extracting information from the master string.
SparkMasterRegex() - Constructor for class org.apache.spark.SparkMasterRegex
 
sparkProperties() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.SparkAppConfig
 
sparkProperties() - Method in class org.apache.spark.status.api.v1.ApplicationEnvironmentInfo
 
SparkRDefaults - Class in org.apache.spark.api.r
 
SparkRDefaults() - Constructor for class org.apache.spark.api.r.SparkRDefaults
 
sparkRPackagePath(boolean) - Static method in class org.apache.spark.api.r.RUtils
Get the list of paths for R packages in various deployment modes, of which the first path is for the SparkR package itself.
sparkSession() - Method in interface org.apache.spark.ml.classification.BinaryLogisticRegressionSummary
 
sparkSession() - Method in class org.apache.spark.sql.Dataset
 
SparkSession - Class in org.apache.spark.sql
The entry point to programming Spark with the Dataset and DataFrame API.
sparkSession() - Method in class org.apache.spark.sql.SQLContext
 
sparkSession() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns the SparkSession associated with this.
SparkSession.Builder - Class in org.apache.spark.sql
Builder for SparkSession.
SparkSession.implicits$ - Class in org.apache.spark.sql
Experimental (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
SparkSessionExtensions - Class in org.apache.spark.sql
Experimental Holder for injection points to the SparkSession.
SparkSessionExtensions() - Constructor for class org.apache.spark.sql.SparkSessionExtensions
 
SparkShutdownHook - Class in org.apache.spark.util
 
SparkShutdownHook(int, Function0<BoxedUnit>) - Constructor for class org.apache.spark.util.SparkShutdownHook
 
SparkStageInfo - Interface in org.apache.spark
Exposes information about Spark Stages.
SparkStageInfoImpl - Class in org.apache.spark
 
SparkStageInfoImpl(int, int, long, String, int, int, int, int) - Constructor for class org.apache.spark.SparkStageInfoImpl
 
SparkStatusTracker - Class in org.apache.spark
Low-level status reporting APIs for monitoring job and stage progress.
sparkUser() - Method in class org.apache.spark.api.java.JavaSparkContext
 
sparkUser() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
sparkUser() - Method in class org.apache.spark.SparkContext
 
sparkUser() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
sparkVersion() - Method in class org.apache.spark.scheduler.SparkListenerLogStart
 
sparse(int, int, int[], int[], double[]) - Static method in class org.apache.spark.ml.linalg.Matrices
Creates a column-major sparse matrix in Compressed Sparse Column (CSC) format.
sparse(int, int[], double[]) - Static method in class org.apache.spark.ml.linalg.Vectors
Creates a sparse vector providing its index array and value array.
sparse(int, Seq<Tuple2<Object, Object>>) - Static method in class org.apache.spark.ml.linalg.Vectors
Creates a sparse vector using unordered (index, value) pairs.
sparse(int, Iterable<Tuple2<Integer, Double>>) - Static method in class org.apache.spark.ml.linalg.Vectors
Creates a sparse vector using unordered (index, value) pairs in a Java friendly way.
sparse(int, int, int[], int[], double[]) - Static method in class org.apache.spark.mllib.linalg.Matrices
Creates a column-major sparse matrix in Compressed Sparse Column (CSC) format.
sparse(int, int[], double[]) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a sparse vector providing its index array and value array.
sparse(int, Seq<Tuple2<Object, Object>>) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a sparse vector using unordered (index, value) pairs.
sparse(int, Iterable<Tuple2<Integer, Double>>) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a sparse vector using unordered (index, value) pairs in a Java friendly way.
SparseMatrix - Class in org.apache.spark.ml.linalg
Column-major sparse matrix.
SparseMatrix(int, int, int[], int[], double[], boolean) - Constructor for class org.apache.spark.ml.linalg.SparseMatrix
 
SparseMatrix(int, int, int[], int[], double[]) - Constructor for class org.apache.spark.ml.linalg.SparseMatrix
Column-major sparse matrix.
SparseMatrix - Class in org.apache.spark.mllib.linalg
Column-major sparse matrix.
SparseMatrix(int, int, int[], int[], double[], boolean) - Constructor for class org.apache.spark.mllib.linalg.SparseMatrix
 
SparseMatrix(int, int, int[], int[], double[]) - Constructor for class org.apache.spark.mllib.linalg.SparseMatrix
Column-major sparse matrix.
SparseVector - Class in org.apache.spark.ml.linalg
A sparse vector represented by an index array and a value array.
SparseVector(int, int[], double[]) - Constructor for class org.apache.spark.ml.linalg.SparseVector
 
SparseVector - Class in org.apache.spark.mllib.linalg
A sparse vector represented by an index array and a value array.
SparseVector(int, int[], double[]) - Constructor for class org.apache.spark.mllib.linalg.SparseVector
 
SPARSITY() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
sparsity() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
spdiag(Vector) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
Generate a diagonal matrix in SparseMatrix format from the supplied values.
spdiag(Vector) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
Generate a diagonal matrix in SparseMatrix format from the supplied values.
SpearmanCorrelation - Class in org.apache.spark.mllib.stat.correlation
Compute Spearman's correlation for two RDDs of the type RDD[Double] or the correlation matrix for an RDD of the type RDD[Vector].
SpearmanCorrelation() - Constructor for class org.apache.spark.mllib.stat.correlation.SpearmanCorrelation
 
SpecialLengths - Class in org.apache.spark.api.r
 
SpecialLengths() - Constructor for class org.apache.spark.api.r.SpecialLengths
 
speculative() - Method in class org.apache.spark.scheduler.TaskInfo
 
speculative() - Method in class org.apache.spark.status.api.v1.TaskData
 
speye(int) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a sparse Identity Matrix in Matrix format.
speye(int) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
Generate an Identity Matrix in SparseMatrix format.
speye(int) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a sparse Identity Matrix in Matrix format.
speye(int) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
Generate an Identity Matrix in SparseMatrix format.
SpillListener - Class in org.apache.spark
A SparkListener that detects whether spills have occurred in Spark jobs.
SpillListener() - Constructor for class org.apache.spark.SpillListener
 
split() - Method in class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.NodeData
 
split() - Method in class org.apache.spark.ml.tree.InternalNode
 
Split - Interface in org.apache.spark.ml.tree
Interface for a "Split," which specifies a test made at a decision tree node to choose the left or right path.
split() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
split() - Method in class org.apache.spark.mllib.tree.model.Node
 
Split - Class in org.apache.spark.mllib.tree.model
Developer API Split applied to a feature param: feature feature index param: threshold Threshold for continuous feature.
Split(int, double, Enumeration.Value, List<Object>) - Constructor for class org.apache.spark.mllib.tree.model.Split
 
split(Column, String) - Static method in class org.apache.spark.sql.functions
Splits str around pattern (pattern is a regular expression).
splitAndCountPartitions(Iterator<String>) - Static method in class org.apache.spark.streaming.util.RawTextHelper
Splits lines and counts the words.
splitAt(int) - Static method in class org.apache.spark.sql.types.StructType
 
splitCommandString(String) - Static method in class org.apache.spark.util.Utils
Split a string of potentially quoted arguments from the command line the way that a shell would do it to determine arguments to a command.
SplitData(int, double[], int) - Constructor for class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData
 
SplitData(int, double, int, Seq<Object>) - Constructor for class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.SplitData
 
SplitData$() - Constructor for class org.apache.spark.ml.tree.DecisionTreeModelReadWrite.SplitData$
 
splitIndex() - Method in class org.apache.spark.storage.RDDBlockId
 
SplitInfo - Class in org.apache.spark.scheduler
 
SplitInfo(Class<?>, String, String, long, Object) - Constructor for class org.apache.spark.scheduler.SplitInfo
 
splits() - Method in class org.apache.spark.ml.feature.Bucketizer
Parameter for mapping continuous features into buckets.
splitsArray() - Method in class org.apache.spark.ml.feature.Bucketizer
Parameter for specifying multiple splits parameters.
spr(double, Vector, DenseVector) - Static method in class org.apache.spark.ml.linalg.BLAS
Adds alpha * x * x.t to a matrix in-place.
spr(double, Vector, double[]) - Static method in class org.apache.spark.ml.linalg.BLAS
Adds alpha * x * x.t to a matrix in-place.
spr(double, Vector, DenseVector) - Static method in class org.apache.spark.mllib.linalg.BLAS
Adds alpha * v * v.t to a matrix in-place.
spr(double, Vector, double[]) - Static method in class org.apache.spark.mllib.linalg.BLAS
Adds alpha * v * v.t to a matrix in-place.
sprand(int, int, double, Random) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a SparseMatrix consisting of i.i.d. gaussian random numbers.
sprand(int, int, double, Random) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
Generate a SparseMatrix consisting of i.i.d.
sprand(int, int, double, Random) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a SparseMatrix consisting of i.i.d. gaussian random numbers.
sprand(int, int, double, Random) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
Generate a SparseMatrix consisting of i.i.d.
sprandn(int, int, double, Random) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a SparseMatrix consisting of i.i.d. gaussian random numbers.
sprandn(int, int, double, Random) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
Generate a SparseMatrix consisting of i.i.d.
sprandn(int, int, double, Random) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a SparseMatrix consisting of i.i.d. gaussian random numbers.
sprandn(int, int, double, Random) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
Generate a SparseMatrix consisting of i.i.d.
sqdist(Vector, Vector) - Static method in class org.apache.spark.ml.linalg.Vectors
Returns the squared distance between two Vectors.
sqdist(Vector, Vector) - Static method in class org.apache.spark.mllib.linalg.Vectors
Returns the squared distance between two Vectors.
sql(String) - Method in class org.apache.spark.sql.SparkSession
Executes a SQL query using Spark, returning the result as a DataFrame.
sql(String) - Method in class org.apache.spark.sql.SQLContext
 
sql() - Method in class org.apache.spark.sql.types.ArrayType
 
sql() - Static method in class org.apache.spark.sql.types.BinaryType
 
sql() - Static method in class org.apache.spark.sql.types.BooleanType
 
sql() - Static method in class org.apache.spark.sql.types.ByteType
 
sql() - Static method in class org.apache.spark.sql.types.CalendarIntervalType
 
sql() - Static method in class org.apache.spark.sql.types.CharType
 
sql() - Method in class org.apache.spark.sql.types.DataType
 
sql() - Static method in class org.apache.spark.sql.types.DateType
 
sql() - Method in class org.apache.spark.sql.types.DecimalType
 
sql() - Static method in class org.apache.spark.sql.types.DoubleType
 
sql() - Static method in class org.apache.spark.sql.types.FloatType
 
sql() - Static method in class org.apache.spark.sql.types.HiveStringType
 
sql() - Static method in class org.apache.spark.sql.types.IntegerType
 
sql() - Static method in class org.apache.spark.sql.types.LongType
 
sql() - Method in class org.apache.spark.sql.types.MapType
 
sql() - Static method in class org.apache.spark.sql.types.NullType
 
sql() - Static method in class org.apache.spark.sql.types.NumericType
 
sql() - Static method in class org.apache.spark.sql.types.ObjectType
 
sql() - Static method in class org.apache.spark.sql.types.ShortType
 
sql() - Static method in class org.apache.spark.sql.types.StringType
 
sql() - Method in class org.apache.spark.sql.types.StructType
 
sql() - Static method in class org.apache.spark.sql.types.TimestampType
 
sql() - Static method in class org.apache.spark.sql.types.VarcharType
 
sqlContext() - Method in class org.apache.spark.sql.Dataset
 
sqlContext() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
sqlContext() - Method in class org.apache.spark.sql.sources.BaseRelation
 
sqlContext() - Method in class org.apache.spark.sql.SparkSession
A wrapped version of this session in the form of a SQLContext, for backward compatibility.
SQLContext - Class in org.apache.spark.sql
The entry point for working with structured data (rows and columns) in Spark 1.x.
SQLContext(SparkContext) - Constructor for class org.apache.spark.sql.SQLContext
Deprecated.
Use SparkSession.builder instead. Since 2.0.0.
SQLContext(JavaSparkContext) - Constructor for class org.apache.spark.sql.SQLContext
Deprecated.
Use SparkSession.builder instead. Since 2.0.0.
SQLContext.implicits$ - Class in org.apache.spark.sql
Experimental (Scala-specific) Implicit methods available in Scala for converting common Scala objects into DataFrames.
SQLDataTypes - Class in org.apache.spark.ml.linalg
Developer API SQL data types for vectors and matrices.
SQLDataTypes() - Constructor for class org.apache.spark.ml.linalg.SQLDataTypes
 
SQLImplicits - Class in org.apache.spark.sql
A collection of implicit methods for converting common Scala objects into Datasets.
SQLImplicits() - Constructor for class org.apache.spark.sql.SQLImplicits
 
SQLImplicits.StringToColumn - Class in org.apache.spark.sql
Converts $"col name" into a Column.
SQLTransformer - Class in org.apache.spark.ml.feature
Implements the transformations which are defined by SQL statement.
SQLTransformer(String) - Constructor for class org.apache.spark.ml.feature.SQLTransformer
 
SQLTransformer() - Constructor for class org.apache.spark.ml.feature.SQLTransformer
 
sqlType() - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
SQLUserDefinedType - Annotation Type in org.apache.spark.sql.types
::DeveloperApi:: A user-defined type which can be automatically recognized by a SQLContext and registered.
SQLUtils - Class in org.apache.spark.sql.api.r
 
SQLUtils() - Constructor for class org.apache.spark.sql.api.r.SQLUtils
 
sqrt(Column) - Static method in class org.apache.spark.sql.functions
Computes the square root of the specified float value.
sqrt(String) - Static method in class org.apache.spark.sql.functions
Computes the square root of the specified float value.
Sqrt$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Sqrt$
 
SquaredError - Class in org.apache.spark.mllib.tree.loss
Developer API Class for squared error loss calculation.
SquaredError() - Constructor for class org.apache.spark.mllib.tree.loss.SquaredError
 
SquaredEuclideanSilhouette - Class in org.apache.spark.ml.evaluation
SquaredEuclideanSilhouette computes the average of the Silhouette over all the data of the dataset, which is a measure of how appropriately the data have been clustered.
SquaredEuclideanSilhouette() - Constructor for class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette
 
SquaredEuclideanSilhouette.ClusterStats - Class in org.apache.spark.ml.evaluation
 
SquaredEuclideanSilhouette.ClusterStats$ - Class in org.apache.spark.ml.evaluation
 
SquaredL2Updater - Class in org.apache.spark.mllib.optimization
Developer API Updater for L2 regularized problems.
SquaredL2Updater() - Constructor for class org.apache.spark.mllib.optimization.SquaredL2Updater
 
squaredNormSum() - Method in class org.apache.spark.ml.evaluation.SquaredEuclideanSilhouette.ClusterStats
 
Src - Static variable in class org.apache.spark.graphx.TripletFields
Expose the source and edge fields but not the destination field.
srcAttr() - Method in class org.apache.spark.graphx.EdgeContext
The vertex attribute of the edge's source vertex.
srcAttr() - Method in class org.apache.spark.graphx.EdgeTriplet
The source vertex attribute
srcAttr() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
srcId() - Method in class org.apache.spark.graphx.Edge
 
srcId() - Method in class org.apache.spark.graphx.EdgeContext
The vertex id of the edge's source vertex.
srcId() - Method in class org.apache.spark.graphx.impl.AggregatingEdgeContext
 
srdd() - Method in class org.apache.spark.api.java.JavaDoubleRDD
 
ssc() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
 
stackTrace() - Method in class org.apache.spark.ExceptionFailure
 
stackTraceFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
stackTraceToJson(StackTraceElement[]) - Static method in class org.apache.spark.util.JsonProtocol
 
stage() - Method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
STAGE() - Static method in class org.apache.spark.status.TaskIndexNames
 
STAGE_DAG() - Static method in class org.apache.spark.ui.ToolTips
 
STAGE_TIMELINE() - Static method in class org.apache.spark.ui.ToolTips
 
stageAttempt() - Method in class org.apache.spark.scheduler.AskPermissionToCommitOutput
 
stageAttemptId() - Method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
stageAttemptId() - Method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
stageAttemptNumber() - Method in class org.apache.spark.TaskContext
How many times the stage that this task belongs to has been attempted.
stageCompletedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
stageCompletedToJson(SparkListenerStageCompleted) - Static method in class org.apache.spark.util.JsonProtocol
 
StageData - Class in org.apache.spark.status.api.v1
 
stageFailed(String) - Method in class org.apache.spark.scheduler.StageInfo
 
stageId() - Method in class org.apache.spark.scheduler.SparkListenerSpeculativeTaskSubmitted
 
stageId() - Method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
stageId() - Method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
stageId() - Method in class org.apache.spark.scheduler.StageInfo
 
stageId() - Method in interface org.apache.spark.SparkStageInfo
 
stageId() - Method in class org.apache.spark.SparkStageInfoImpl
 
stageId() - Method in class org.apache.spark.status.api.v1.StageData
 
stageId() - Method in class org.apache.spark.TaskContext
The ID of the stage that this task belong to.
stageIds() - Method in class org.apache.spark.scheduler.SparkListenerJobStart
 
stageIds() - Method in interface org.apache.spark.SparkJobInfo
 
stageIds() - Method in class org.apache.spark.SparkJobInfoImpl
 
stageIds() - Method in class org.apache.spark.status.api.v1.JobData
 
stageIds() - Method in class org.apache.spark.status.LiveJob
 
stageIds() - Method in class org.apache.spark.status.SchedulerPool
 
stageInfo() - Method in class org.apache.spark.scheduler.SparkListenerStageCompleted
 
stageInfo() - Method in class org.apache.spark.scheduler.SparkListenerStageSubmitted
 
StageInfo - Class in org.apache.spark.scheduler
Developer API Stores information about a stage to pass from the scheduler to SparkListeners.
StageInfo(int, int, String, int, Seq<RDDInfo>, Seq<Object>, String, TaskMetrics, Seq<Seq<TaskLocation>>) - Constructor for class org.apache.spark.scheduler.StageInfo
 
stageInfoFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
--------------------------------------------------------------------- * JSON deserialization methods for classes SparkListenerEvents depend on |
stageInfos() - Method in class org.apache.spark.scheduler.SparkListenerJobStart
 
stageInfoToJson(StageInfo) - Static method in class org.apache.spark.util.JsonProtocol
------------------------------------------------------------------- * JSON serialization methods for classes SparkListenerEvents depend on |
stages() - Method in class org.apache.spark.ml.Pipeline
param for pipeline stages
stages() - Method in class org.apache.spark.ml.PipelineModel
 
StageStatus - Enum in org.apache.spark.status.api.v1
 
stageSubmittedFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
stageSubmittedToJson(SparkListenerStageSubmitted) - Static method in class org.apache.spark.util.JsonProtocol
 
standardization() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
standardization() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
standardization() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
standardization() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
standardization() - Method in interface org.apache.spark.ml.param.shared.HasStandardization
Param for whether to standardize the training features before fitting the model.
standardization() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
standardization() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
StandardNormalGenerator - Class in org.apache.spark.mllib.random
Developer API Generates i.i.d.
StandardNormalGenerator() - Constructor for class org.apache.spark.mllib.random.StandardNormalGenerator
 
StandardScaler - Class in org.apache.spark.ml.feature
Standardizes features by removing the mean and scaling to unit variance using column summary statistics on the samples in the training set.
StandardScaler(String) - Constructor for class org.apache.spark.ml.feature.StandardScaler
 
StandardScaler() - Constructor for class org.apache.spark.ml.feature.StandardScaler
 
StandardScaler - Class in org.apache.spark.mllib.feature
Standardizes features by removing the mean and scaling to unit std using column summary statistics on the samples in the training set.
StandardScaler(boolean, boolean) - Constructor for class org.apache.spark.mllib.feature.StandardScaler
 
StandardScaler() - Constructor for class org.apache.spark.mllib.feature.StandardScaler
 
StandardScalerModel - Class in org.apache.spark.ml.feature
Model fitted by StandardScaler.
StandardScalerModel - Class in org.apache.spark.mllib.feature
Represents a StandardScaler model that can transform vectors.
StandardScalerModel(Vector, Vector, boolean, boolean) - Constructor for class org.apache.spark.mllib.feature.StandardScalerModel
 
StandardScalerModel(Vector, Vector) - Constructor for class org.apache.spark.mllib.feature.StandardScalerModel
 
StandardScalerModel(Vector) - Constructor for class org.apache.spark.mllib.feature.StandardScalerModel
 
starGraph(SparkContext, int) - Static method in class org.apache.spark.graphx.util.GraphGenerators
Create a star graph with vertex 0 being the center.
start(String) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Starts the execution of the streaming query, which will continually output results to the given path as new data arrives.
start() - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Starts the execution of the streaming query, which will continually output results to the given path as new data arrives.
start() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Start the execution of the streams.
start() - Method in class org.apache.spark.streaming.dstream.ConstantInputDStream
 
start() - Method in class org.apache.spark.streaming.dstream.InputDStream
Method called to start receiving data.
start() - Method in class org.apache.spark.streaming.dstream.ReceiverInputDStream
 
start() - Method in class org.apache.spark.streaming.StreamingContext
Start the execution of the streams.
startApplication(SparkAppHandle.Listener...) - Method in class org.apache.spark.launcher.AbstractLauncher
Starts a Spark application.
startApplication(SparkAppHandle.Listener...) - Method in class org.apache.spark.launcher.InProcessLauncher
Starts a Spark application.
startApplication(SparkAppHandle.Listener...) - Method in class org.apache.spark.launcher.SparkLauncher
Starts a Spark application.
startIndexInLevel(int) - Static method in class org.apache.spark.mllib.tree.model.Node
Return the index of the first node in the given level.
startJettyServer(String, int, org.apache.spark.SSLOptions, Seq<ServletContextHandler>, SparkConf, String) - Static method in class org.apache.spark.ui.JettyUtils
Attempt to start a Jetty server bound to the supplied hostName:port using the given context handlers.
startOffset() - Method in class org.apache.spark.sql.streaming.SourceProgress
 
startOffset() - Method in exception org.apache.spark.sql.streaming.StreamingQueryException
 
startPosition() - Method in exception org.apache.spark.sql.AnalysisException
 
startServiceOnPort(int, Function1<Object, Tuple2<T, Object>>, SparkConf, String) - Static method in class org.apache.spark.util.Utils
Attempt to start a service on the given port, or fail after a number of attempts.
startsWith(Column) - Method in class org.apache.spark.sql.Column
String starts with.
startsWith(String) - Method in class org.apache.spark.sql.Column
String starts with another string literal.
startsWith(GenSeq<B>) - Static method in class org.apache.spark.sql.types.StructType
 
startsWith(GenSeq<B>, int) - Static method in class org.apache.spark.sql.types.StructType
 
startTime() - Method in class org.apache.spark.api.java.JavaSparkContext
 
startTime() - Method in class org.apache.spark.SparkContext
 
startTime() - Method in class org.apache.spark.status.api.v1.ApplicationAttemptInfo
 
startTime() - Method in class org.apache.spark.status.api.v1.streaming.OutputOperationInfo
 
startTime() - Method in class org.apache.spark.status.api.v1.streaming.StreamingStatistics
 
startTime() - Method in class org.apache.spark.streaming.scheduler.OutputOperationInfo
 
stat() - Method in class org.apache.spark.sql.Dataset
Returns a DataFrameStatFunctions for working statistic functions support.
StatCounter - Class in org.apache.spark.util
A class for tracking the statistics of a set of numbers (count, mean and variance) in a numerically robust way.
StatCounter(TraversableOnce<Object>) - Constructor for class org.apache.spark.util.StatCounter
 
StatCounter() - Constructor for class org.apache.spark.util.StatCounter
Initialize the StatCounter with no values.
state() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
state() - Method in class org.apache.spark.scheduler.local.StatusUpdate
 
State<S> - Class in org.apache.spark.streaming
Experimental Abstract class for getting and updating the state in mapping function used in the mapWithState operation of a pair DStream (Scala) or a JavaPairDStream (Java).
State() - Constructor for class org.apache.spark.streaming.State
 
stateChanged(SparkAppHandle) - Method in interface org.apache.spark.launcher.SparkAppHandle.Listener
Callback for changes in the handle's state.
statement() - Method in class org.apache.spark.ml.feature.SQLTransformer
SQL statement parameter.
StateOperatorProgress - Class in org.apache.spark.sql.streaming
Information about updates made to stateful operators in a StreamingQuery during a trigger.
stateOperators() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
stateSnapshots() - Method in class org.apache.spark.streaming.api.java.JavaMapWithStateDStream
 
stateSnapshots() - Method in class org.apache.spark.streaming.dstream.MapWithStateDStream
Return a pair DStream where each RDD is the snapshot of the state of all the keys.
StateSpec<KeyType,ValueType,StateType,MappedType> - Class in org.apache.spark.streaming
Experimental Abstract class representing all the specifications of the DStream transformation mapWithState operation of a pair DStream (Scala) or a JavaPairDStream (Java).
StateSpec() - Constructor for class org.apache.spark.streaming.StateSpec
 
staticPageRank(int, double) - Method in class org.apache.spark.graphx.GraphOps
Run PageRank for a fixed number of iterations returning a graph with vertex attributes containing the PageRank and edge attributes the normalized edge weight.
staticParallelPersonalizedPageRank(long[], int, double) - Method in class org.apache.spark.graphx.GraphOps
Run parallel personalized PageRank for a given array of source vertices, such that all random walks are started relative to the source vertices
staticPersonalizedPageRank(long, int, double) - Method in class org.apache.spark.graphx.GraphOps
Run Personalized PageRank for a fixed number of iterations with with all iterations originating at the source node returning a graph with vertex attributes containing the PageRank and edge attributes the normalized edge weight.
StaticSources - Class in org.apache.spark.metrics.source
 
StaticSources() - Constructor for class org.apache.spark.metrics.source.StaticSources
 
statistic() - Method in class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
statistic() - Method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult
 
statistic() - Method in interface org.apache.spark.mllib.stat.test.TestResult
Test statistic.
Statistics - Class in org.apache.spark.mllib.stat
API for statistical functions in MLlib.
Statistics() - Constructor for class org.apache.spark.mllib.stat.Statistics
 
Statistics - Interface in org.apache.spark.sql.sources.v2.reader
An interface to represent statistics for a data source, which is returned by SupportsReportStatistics.getStatistics().
stats() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return a StatCounter object that captures the mean, variance and count of the RDD's elements in one operation.
stats() - Method in class org.apache.spark.mllib.tree.model.Node
 
stats() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Return a StatCounter object that captures the mean, variance and count of the RDD's elements in one operation.
stats() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
stats() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
stats() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
StatsdMetricType - Class in org.apache.spark.metrics.sink
 
StatsdMetricType() - Constructor for class org.apache.spark.metrics.sink.StatsdMetricType
 
StatsReportListener - Class in org.apache.spark.scheduler
Developer API Simple SparkListener that logs a few summary statistics when each stage completes.
StatsReportListener() - Constructor for class org.apache.spark.scheduler.StatsReportListener
 
StatsReportListener - Class in org.apache.spark.streaming.scheduler
Developer API A simple StreamingListener that logs summary statistics across Spark Streaming batches param: numBatchInfos Number of last batches to consider for generating statistics (default: 10)
StatsReportListener(int) - Constructor for class org.apache.spark.streaming.scheduler.StatsReportListener
 
status() - Method in class org.apache.spark.scheduler.TaskInfo
 
status() - Method in interface org.apache.spark.SparkJobInfo
 
status() - Method in class org.apache.spark.SparkJobInfoImpl
 
status() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Returns the current status of the query.
status() - Method in class org.apache.spark.status.api.v1.JobData
 
status() - Method in class org.apache.spark.status.api.v1.StageData
 
status() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
status() - Method in class org.apache.spark.status.api.v1.TaskData
 
status() - Method in class org.apache.spark.status.LiveJob
 
status() - Method in class org.apache.spark.status.LiveStage
 
STATUS() - Static method in class org.apache.spark.status.TaskIndexNames
 
status() - Method in class org.apache.spark.storage.BlockManagerMessages.BlockLocationsAndStatus
 
statusTracker() - Method in class org.apache.spark.api.java.JavaSparkContext
 
statusTracker() - Method in class org.apache.spark.SparkContext
 
StatusUpdate(String, long, Enumeration.Value, org.apache.spark.util.SerializableBuffer) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
StatusUpdate - Class in org.apache.spark.scheduler.local
 
StatusUpdate(long, Enumeration.Value, ByteBuffer) - Constructor for class org.apache.spark.scheduler.local.StatusUpdate
 
StatusUpdate$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate$
 
STD() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
std() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
std() - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
std() - Method in class org.apache.spark.mllib.feature.StandardScalerModel
 
std() - Method in class org.apache.spark.mllib.random.LogNormalGenerator
 
stddev(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: alias for stddev_samp.
stddev(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: alias for stddev_samp.
stddev_pop(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the population standard deviation of the expression in a group.
stddev_pop(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the population standard deviation of the expression in a group.
stddev_samp(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sample standard deviation of the expression in a group.
stddev_samp(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sample standard deviation of the expression in a group.
stdev() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute the population standard deviation of this RDD's elements.
stdev() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute the population standard deviation of this RDD's elements.
stdev() - Method in class org.apache.spark.util.StatCounter
Return the population standard deviation of the values.
stepSize() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
stepSize() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
stepSize() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
stepSize() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
stepSize() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
stepSize() - Method in interface org.apache.spark.ml.param.shared.HasStepSize
Param for Step size to be used for each iteration of optimization (&gt; 0).
stepSize() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
stepSize() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
stop() - Method in class org.apache.spark.api.java.JavaSparkContext
Shut down the SparkContext.
stop() - Method in interface org.apache.spark.launcher.SparkAppHandle
Asks the application to stop.
stop() - Method in class org.apache.spark.SparkContext
Shut down the SparkContext.
stop() - Method in class org.apache.spark.sql.SparkSession
Stop the underlying SparkContext.
stop() - Method in interface org.apache.spark.sql.streaming.StreamingQuery
Stops the execution of this query if it is running.
stop() - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Stop the execution of the streams.
stop(boolean) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Stop the execution of the streams.
stop(boolean, boolean) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Stop the execution of the streams.
stop() - Method in class org.apache.spark.streaming.dstream.ConstantInputDStream
 
stop() - Method in class org.apache.spark.streaming.dstream.InputDStream
Method called to stop receiving data.
stop() - Method in class org.apache.spark.streaming.dstream.ReceiverInputDStream
 
stop(String) - Method in class org.apache.spark.streaming.receiver.Receiver
Stop the receiver completely.
stop(String, Throwable) - Method in class org.apache.spark.streaming.receiver.Receiver
Stop the receiver completely due to an exception
stop(boolean) - Method in class org.apache.spark.streaming.StreamingContext
Stop the execution of the streams immediately (does not wait for all received data to be processed).
stop(boolean, boolean) - Method in class org.apache.spark.streaming.StreamingContext
Stop the execution of the streams, with option of ensuring all received data has been processed.
StopAllReceivers - Class in org.apache.spark.streaming.scheduler
This message will trigger ReceiverTrackerEndpoint to send stop signals to all registered receivers.
StopAllReceivers() - Constructor for class org.apache.spark.streaming.scheduler.StopAllReceivers
 
StopBlockManagerMaster$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.StopBlockManagerMaster$
 
StopCoordinator - Class in org.apache.spark.scheduler
 
StopCoordinator() - Constructor for class org.apache.spark.scheduler.StopCoordinator
 
StopDriver$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopDriver$
 
StopExecutor - Class in org.apache.spark.scheduler.local
 
StopExecutor() - Constructor for class org.apache.spark.scheduler.local.StopExecutor
 
StopExecutor$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopExecutor$
 
StopExecutors$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StopExecutors$
 
StopMapOutputTracker - Class in org.apache.spark
 
StopMapOutputTracker() - Constructor for class org.apache.spark.StopMapOutputTracker
 
StopReceiver - Class in org.apache.spark.streaming.receiver
 
StopReceiver() - Constructor for class org.apache.spark.streaming.receiver.StopReceiver
 
stopWords() - Method in class org.apache.spark.ml.feature.StopWordsRemover
The words to be filtered out.
StopWordsRemover - Class in org.apache.spark.ml.feature
A feature transformer that filters out stop words from input.
StopWordsRemover(String) - Constructor for class org.apache.spark.ml.feature.StopWordsRemover
 
StopWordsRemover() - Constructor for class org.apache.spark.ml.feature.StopWordsRemover
 
storage() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
STORAGE_MEMORY() - Static method in class org.apache.spark.ui.ToolTips
 
storageLevel() - Method in class org.apache.spark.sql.Dataset
Get the Dataset's current storage level, or StorageLevel.NONE if not persisted.
storageLevel() - Method in class org.apache.spark.status.api.v1.RDDPartitionInfo
 
storageLevel() - Method in class org.apache.spark.status.api.v1.RDDStorageInfo
 
storageLevel() - Method in class org.apache.spark.status.LiveRDD
 
storageLevel() - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
storageLevel() - Method in class org.apache.spark.storage.BlockStatus
 
storageLevel() - Method in class org.apache.spark.storage.BlockUpdatedInfo
 
storageLevel() - Method in class org.apache.spark.storage.RDDInfo
 
StorageLevel - Class in org.apache.spark.storage
Developer API Flags for controlling the storage of an RDD.
StorageLevel() - Constructor for class org.apache.spark.storage.StorageLevel
 
storageLevel() - Method in class org.apache.spark.streaming.receiver.Receiver
 
storageLevelFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
StorageLevels - Class in org.apache.spark.api.java
Expose some commonly useful storage level constants.
StorageLevels() - Constructor for class org.apache.spark.api.java.StorageLevels
 
storageLevelToJson(StorageLevel) - Static method in class org.apache.spark.util.JsonProtocol
 
StorageStatus - Class in org.apache.spark.storage
Deprecated.
This class may be removed or made private in a future release. Since 2.2.0.
StorageStatus(BlockManagerId, long, Option<Object>, Option<Object>) - Constructor for class org.apache.spark.storage.StorageStatus
Deprecated.
 
StorageStatus(BlockManagerId, long, Option<Object>, Option<Object>, Map<BlockId, BlockStatus>) - Constructor for class org.apache.spark.storage.StorageStatus
Deprecated.
Create a storage status with an initial set of blocks, leaving the source unmodified.
StorageUtils - Class in org.apache.spark.storage
Helper methods for storage-related objects.
StorageUtils() - Constructor for class org.apache.spark.storage.StorageUtils
 
store(T) - Method in class org.apache.spark.streaming.receiver.Receiver
Store a single item of received data to Spark's memory.
store(ArrayBuffer<T>) - Method in class org.apache.spark.streaming.receiver.Receiver
Store an ArrayBuffer of received data as a data block into Spark's memory.
store(ArrayBuffer<T>, Object) - Method in class org.apache.spark.streaming.receiver.Receiver
Store an ArrayBuffer of received data as a data block into Spark's memory.
store(Iterator<T>) - Method in class org.apache.spark.streaming.receiver.Receiver
Store an iterator of received data as a data block into Spark's memory.
store(Iterator<T>, Object) - Method in class org.apache.spark.streaming.receiver.Receiver
Store an iterator of received data as a data block into Spark's memory.
store(Iterator<T>) - Method in class org.apache.spark.streaming.receiver.Receiver
Store an iterator of received data as a data block into Spark's memory.
store(Iterator<T>, Object) - Method in class org.apache.spark.streaming.receiver.Receiver
Store an iterator of received data as a data block into Spark's memory.
store(ByteBuffer) - Method in class org.apache.spark.streaming.receiver.Receiver
Store the bytes of received data as a data block into Spark's memory.
store(ByteBuffer, Object) - Method in class org.apache.spark.streaming.receiver.Receiver
Store the bytes of received data as a data block into Spark's memory.
strategy() - Static method in class org.apache.spark.ml.feature.Imputer
 
strategy() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
Strategy - Class in org.apache.spark.mllib.tree.configuration
Stores all the configuration options for tree construction param: algo Learning goal.
Strategy(Enumeration.Value, Impurity, int, int, int, Enumeration.Value, Map<Object, Object>, int, double, int, double, boolean, int) - Constructor for class org.apache.spark.mllib.tree.configuration.Strategy
 
Strategy(Enumeration.Value, Impurity, int, int, int, Map<Integer, Integer>) - Constructor for class org.apache.spark.mllib.tree.configuration.Strategy
Java-friendly constructor for Strategy
StratifiedSamplingUtils - Class in org.apache.spark.util.random
Auxiliary functions and data structures for the sampleByKey method in PairRDDFunctions.
StratifiedSamplingUtils() - Constructor for class org.apache.spark.util.random.StratifiedSamplingUtils
 
STREAM() - Static method in class org.apache.spark.storage.BlockId
 
StreamBlockId - Class in org.apache.spark.storage
 
StreamBlockId(int, long) - Constructor for class org.apache.spark.storage.StreamBlockId
 
streamId() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
streamId() - Method in class org.apache.spark.storage.StreamBlockId
 
streamId() - Method in class org.apache.spark.streaming.receiver.Receiver
Get the unique identifier the receiver input stream that this receiver is associated with.
streamId() - Method in class org.apache.spark.streaming.scheduler.ReceiverInfo
 
streamIdToInputInfo() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
 
StreamingContext - Class in org.apache.spark.streaming
Main entry point for Spark Streaming functionality.
StreamingContext(SparkContext, Duration) - Constructor for class org.apache.spark.streaming.StreamingContext
Create a StreamingContext using an existing SparkContext.
StreamingContext(SparkConf, Duration) - Constructor for class org.apache.spark.streaming.StreamingContext
Create a StreamingContext by providing the configuration necessary for a new SparkContext.
StreamingContext(String, String, Duration, String, Seq<String>, Map<String, String>) - Constructor for class org.apache.spark.streaming.StreamingContext
Create a StreamingContext by providing the details necessary for creating a new SparkContext.
StreamingContext(String, Configuration) - Constructor for class org.apache.spark.streaming.StreamingContext
Recreate a StreamingContext from a checkpoint file.
StreamingContext(String) - Constructor for class org.apache.spark.streaming.StreamingContext
Recreate a StreamingContext from a checkpoint file.
StreamingContext(String, SparkContext) - Constructor for class org.apache.spark.streaming.StreamingContext
Recreate a StreamingContext from a checkpoint file using an existing SparkContext.
StreamingContextPythonHelper - Class in org.apache.spark.streaming
 
StreamingContextPythonHelper() - Constructor for class org.apache.spark.streaming.StreamingContextPythonHelper
 
StreamingContextState - Enum in org.apache.spark.streaming
Developer API Represents the state of a StreamingContext.
StreamingKMeans - Class in org.apache.spark.mllib.clustering
StreamingKMeans provides methods for configuring a streaming k-means analysis, training the model on streaming, and using the model to make predictions on streaming data.
StreamingKMeans(int, double, String) - Constructor for class org.apache.spark.mllib.clustering.StreamingKMeans
 
StreamingKMeans() - Constructor for class org.apache.spark.mllib.clustering.StreamingKMeans
 
StreamingKMeansModel - Class in org.apache.spark.mllib.clustering
StreamingKMeansModel extends MLlib's KMeansModel for streaming algorithms, so it can keep track of a continuously updated weight associated with each cluster, and also update the model by doing a single iteration of the standard k-means algorithm.
StreamingKMeansModel(Vector[], double[]) - Constructor for class org.apache.spark.mllib.clustering.StreamingKMeansModel
 
StreamingLinearAlgorithm<M extends GeneralizedLinearModel,A extends GeneralizedLinearAlgorithm<M>> - Class in org.apache.spark.mllib.regression
Developer API StreamingLinearAlgorithm implements methods for continuously training a generalized linear model on streaming data, and using it for prediction on (possibly different) streaming data.
StreamingLinearAlgorithm() - Constructor for class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
 
StreamingLinearRegressionWithSGD - Class in org.apache.spark.mllib.regression
Train or predict a linear regression model on streaming data.
StreamingLinearRegressionWithSGD() - Constructor for class org.apache.spark.mllib.regression.StreamingLinearRegressionWithSGD
Construct a StreamingLinearRegression object with default parameters: {stepSize: 0.1, numIterations: 50, miniBatchFraction: 1.0}.
StreamingListener - Interface in org.apache.spark.streaming.scheduler
Developer API A listener interface for receiving information about an ongoing streaming computation.
StreamingListenerBatchCompleted - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerBatchCompleted(BatchInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted
 
StreamingListenerBatchStarted - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerBatchStarted(BatchInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted
 
StreamingListenerBatchSubmitted - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerBatchSubmitted(BatchInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted
 
StreamingListenerEvent - Interface in org.apache.spark.streaming.scheduler
Developer API Base trait for events related to StreamingListener
StreamingListenerOutputOperationCompleted - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerOutputOperationCompleted(OutputOperationInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationCompleted
 
StreamingListenerOutputOperationStarted - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerOutputOperationStarted(OutputOperationInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerOutputOperationStarted
 
StreamingListenerReceiverError - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerReceiverError(ReceiverInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerReceiverError
 
StreamingListenerReceiverStarted - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerReceiverStarted(ReceiverInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStarted
 
StreamingListenerReceiverStopped - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerReceiverStopped(ReceiverInfo) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerReceiverStopped
 
StreamingListenerStreamingStarted - Class in org.apache.spark.streaming.scheduler
 
StreamingListenerStreamingStarted(long) - Constructor for class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
StreamingLogisticRegressionWithSGD - Class in org.apache.spark.mllib.classification
Train or predict a logistic regression model on streaming data.
StreamingLogisticRegressionWithSGD() - Constructor for class org.apache.spark.mllib.classification.StreamingLogisticRegressionWithSGD
Construct a StreamingLogisticRegression object with default parameters: {stepSize: 0.1, numIterations: 50, miniBatchFraction: 1.0, regParam: 0.0}.
StreamingQuery - Interface in org.apache.spark.sql.streaming
A handle to a query that is executing continuously in the background as new data arrives.
StreamingQueryException - Exception in org.apache.spark.sql.streaming
Exception that stopped a StreamingQuery.
StreamingQueryListener - Class in org.apache.spark.sql.streaming
Interface for listening to events related to StreamingQueries.
StreamingQueryListener() - Constructor for class org.apache.spark.sql.streaming.StreamingQueryListener
 
StreamingQueryListener.Event - Interface in org.apache.spark.sql.streaming
Base type of StreamingQueryListener events
StreamingQueryListener.QueryProgressEvent - Class in org.apache.spark.sql.streaming
Event representing any progress updates in a query.
StreamingQueryListener.QueryStartedEvent - Class in org.apache.spark.sql.streaming
Event representing the start of a query param: id An unique query id that persists across restarts.
StreamingQueryListener.QueryTerminatedEvent - Class in org.apache.spark.sql.streaming
Event representing that termination of a query.
StreamingQueryManager - Class in org.apache.spark.sql.streaming
A class to manage all the StreamingQuery active in a SparkSession.
StreamingQueryProgress - Class in org.apache.spark.sql.streaming
Information about progress made in the execution of a StreamingQuery during a trigger.
StreamingQueryStatus - Class in org.apache.spark.sql.streaming
Reports information about the instantaneous status of a streaming query.
StreamingStatistics - Class in org.apache.spark.status.api.v1.streaming
 
StreamingTest - Class in org.apache.spark.mllib.stat.test
Performs online 2-sample significance testing for a stream of (Boolean, Double) pairs.
StreamingTest() - Constructor for class org.apache.spark.mllib.stat.test.StreamingTest
 
StreamInputInfo - Class in org.apache.spark.streaming.scheduler
Developer API Track the information of input stream at specified batch time.
StreamInputInfo(int, long, Map<String, Object>) - Constructor for class org.apache.spark.streaming.scheduler.StreamInputInfo
 
streamName() - Method in class org.apache.spark.status.api.v1.streaming.ReceiverInfo
 
streams() - Method in class org.apache.spark.sql.SparkSession
Experimental Returns a StreamingQueryManager that allows managing all the StreamingQuerys active on this.
streams() - Method in class org.apache.spark.sql.SQLContext
 
StreamSinkProvider - Interface in org.apache.spark.sql.sources
::Experimental:: Implemented by objects that can produce a streaming Sink for a specific format or system.
StreamSourceProvider - Interface in org.apache.spark.sql.sources
::Experimental:: Implemented by objects that can produce a streaming Source for a specific format or system.
StreamWriter - Interface in org.apache.spark.sql.sources.v2.writer.streaming
A DataSourceWriter for use with structured streaming.
StreamWriteSupport - Interface in org.apache.spark.sql.sources.v2
A mix-in interface for DataSourceV2.
STRING() - Static method in class org.apache.spark.api.r.SerializationFormats
 
string() - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type string.
STRING() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable string type.
StringAccumulatorParam$() - Constructor for class org.apache.spark.AccumulatorParam.StringAccumulatorParam$
Deprecated.
 
StringArrayParam - Class in org.apache.spark.ml.param
Developer API Specialized version of Param[Array[String} for Java.
StringArrayParam(Params, String, String, Function1<String[], Object>) - Constructor for class org.apache.spark.ml.param.StringArrayParam
 
StringArrayParam(Params, String, String) - Constructor for class org.apache.spark.ml.param.StringArrayParam
 
StringContains - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a string that contains the string value.
StringContains(String, String) - Constructor for class org.apache.spark.sql.sources.StringContains
 
StringEndsWith - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a string that starts with value.
StringEndsWith(String, String) - Constructor for class org.apache.spark.sql.sources.StringEndsWith
 
StringIndexer - Class in org.apache.spark.ml.feature
A label indexer that maps a string column of labels to an ML column of label indices.
StringIndexer(String) - Constructor for class org.apache.spark.ml.feature.StringIndexer
 
StringIndexer() - Constructor for class org.apache.spark.ml.feature.StringIndexer
 
StringIndexerModel - Class in org.apache.spark.ml.feature
Model fitted by StringIndexer.
StringIndexerModel(String, String[]) - Constructor for class org.apache.spark.ml.feature.StringIndexerModel
 
StringIndexerModel(String[]) - Constructor for class org.apache.spark.ml.feature.StringIndexerModel
 
stringIndexerOrderType() - Static method in class org.apache.spark.ml.feature.RFormula
 
stringIndexerOrderType() - Static method in class org.apache.spark.ml.feature.RFormulaModel
 
stringOrderType() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
stringOrderType() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
stringPrefix() - Static method in class org.apache.spark.sql.types.StructType
 
StringRRDD<T> - Class in org.apache.spark.api.r
An RDD that stores R objects as Array[String].
StringRRDD(RDD<T>, byte[], String, byte[], Object[], ClassTag<T>) - Constructor for class org.apache.spark.api.r.StringRRDD
 
StringStartsWith - Class in org.apache.spark.sql.sources
A filter that evaluates to true iff the attribute evaluates to a string that starts with value.
StringStartsWith(String, String) - Constructor for class org.apache.spark.sql.sources.StringStartsWith
 
StringToColumn(StringContext) - Constructor for class org.apache.spark.sql.SQLImplicits.StringToColumn
 
stringToSeq(String, Function1<String, T>) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
stringToSeq(String) - Static method in class org.apache.spark.util.Utils
 
StringType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the StringType object.
StringType - Class in org.apache.spark.sql.types
The data type representing String values.
stripDirectory(String) - Static method in class org.apache.spark.util.Utils
Strip the directory from a path name
stripXSS(String) - Static method in class org.apache.spark.ui.UIUtils
Remove suspicious characters of user input to prevent Cross-Site scripting (XSS) attacks
stronglyConnectedComponents(int) - Method in class org.apache.spark.graphx.GraphOps
Compute the strongly connected component (SCC) of each vertex and return a graph with the vertex value containing the lowest vertex id in the SCC containing that vertex.
StronglyConnectedComponents - Class in org.apache.spark.graphx.lib
Strongly connected components algorithm implementation.
StronglyConnectedComponents() - Constructor for class org.apache.spark.graphx.lib.StronglyConnectedComponents
 
struct(Seq<StructField>) - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type struct.
struct(StructType) - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type struct.
struct(Column...) - Static method in class org.apache.spark.sql.functions
Creates a new struct column.
struct(String, String...) - Static method in class org.apache.spark.sql.functions
Creates a new struct column that composes multiple input columns.
struct(Seq<Column>) - Static method in class org.apache.spark.sql.functions
Creates a new struct column.
struct(String, Seq<String>) - Static method in class org.apache.spark.sql.functions
Creates a new struct column that composes multiple input columns.
StructField - Class in org.apache.spark.sql.types
A field inside a StructType.
StructField(String, DataType, boolean, Metadata) - Constructor for class org.apache.spark.sql.types.StructField
 
StructType - Class in org.apache.spark.sql.types
A StructType object can be constructed by
StructType(StructField[]) - Constructor for class org.apache.spark.sql.types.StructType
 
StructType() - Constructor for class org.apache.spark.sql.types.StructType
No-arg constructor for kryo.
StudentTTest - Class in org.apache.spark.mllib.stat.test
Performs Students's 2-sample t-test.
StudentTTest() - Constructor for class org.apache.spark.mllib.stat.test.StudentTTest
 
subexpressionEliminationEnabled() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
subgraph(Function1<EdgeTriplet<VD, ED>, Object>, Function2<Object, VD, Object>) - Method in class org.apache.spark.graphx.Graph
Restricts the graph to only the vertices and edges satisfying the predicates.
subgraph(Function1<EdgeTriplet<VD, ED>, Object>, Function2<Object, VD, Object>) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
subgraph$default$1() - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
subgraph$default$2() - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
submissionTime() - Method in class org.apache.spark.scheduler.StageInfo
When this stage was submitted from the DAGScheduler to a TaskScheduler.
submissionTime() - Method in interface org.apache.spark.SparkStageInfo
 
submissionTime() - Method in class org.apache.spark.SparkStageInfoImpl
 
submissionTime() - Method in class org.apache.spark.status.api.v1.JobData
 
submissionTime() - Method in class org.apache.spark.status.api.v1.StageData
 
submissionTime() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
 
submitJob(RDD<T>, Function1<Iterator<T>, U>, Seq<Object>, Function2<Object, U, BoxedUnit>, Function0<R>) - Method in interface org.apache.spark.JobSubmitter
Submit a job for execution and return a FutureAction holding the result.
submitJob(RDD<T>, Function1<Iterator<T>, U>, Seq<Object>, Function2<Object, U, BoxedUnit>, Function0<R>) - Method in class org.apache.spark.SparkContext
Submit a job for execution and return a FutureJob holding the result.
subModels() - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
subModels() - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
subqueries() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
subqueries() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
subqueries() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
subqueries() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
subsamplingRate() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
subsamplingRate() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
subsamplingRate() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
subsamplingRate() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
subsamplingRate() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
subsamplingRate() - Static method in class org.apache.spark.ml.clustering.LDA
 
subsamplingRate() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
subsamplingRate() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
subsamplingRate() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
subsamplingRate() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
subsamplingRate() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
subsamplingRate() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
subsetAccuracy() - Method in class org.apache.spark.mllib.evaluation.MultilabelMetrics
Returns subset accuracy (for equal sets of labels)
substr(Column, Column) - Method in class org.apache.spark.sql.Column
An expression that returns a substring.
substr(int, int) - Method in class org.apache.spark.sql.Column
An expression that returns a substring.
substring(Column, int, int) - Static method in class org.apache.spark.sql.functions
Substring starts at pos and is of length len when str is String type or returns the slice of byte array that starts at pos in byte and is of length len when str is Binary type
substring_index(Column, String, int) - Static method in class org.apache.spark.sql.functions
Returns the substring from string str before count occurrences of the delimiter delim.
subtract(JavaDoubleRDD) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaDoubleRDD, int) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaDoubleRDD, Partitioner) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaPairRDD<K, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaPairRDD<K, V>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaPairRDD<K, V>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaRDD<T>) - Method in class org.apache.spark.api.java.JavaRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaRDD<T>, int) - Method in class org.apache.spark.api.java.JavaRDD
Return an RDD with the elements from this that are not in other.
subtract(JavaRDD<T>, Partitioner) - Method in class org.apache.spark.api.java.JavaRDD
Return an RDD with the elements from this that are not in other.
subtract(RDD<T>) - Static method in class org.apache.spark.api.r.RRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.api.r.RRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
subtract(RDD<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.graphx.EdgeRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
subtract(RDD<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
subtract(RDD<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
subtract(RDD<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.graphx.VertexRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
subtract(BlockMatrix) - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Subtracts the given block matrix other from this block matrix: this - other.
subtract(RDD<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.rdd.HadoopRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
subtract(RDD<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.rdd.JdbcRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
subtract(RDD<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
subtract(RDD<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
subtract(RDD<T>) - Method in class org.apache.spark.rdd.RDD
Return an RDD with the elements from this that are not in other.
subtract(RDD<T>, int) - Method in class org.apache.spark.rdd.RDD
Return an RDD with the elements from this that are not in other.
subtract(RDD<T>, Partitioner, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Return an RDD with the elements from this that are not in other.
subtract(RDD<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
subtract(RDD<T>, int) - Static method in class org.apache.spark.rdd.UnionRDD
 
subtract(RDD<T>, Partitioner, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
subtract(long, long) - Static method in class org.apache.spark.streaming.util.RawTextHelper
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.api.r.RRDD
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.EdgeRDD
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.graphx.VertexRDD
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.HadoopRDD
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.JdbcRDD
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
subtract$default$3(RDD<T>, Partitioner) - Static method in class org.apache.spark.rdd.UnionRDD
 
subtractByKey(JavaPairRDD<K, W>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the pairs from this whose keys are not in other.
subtractByKey(JavaPairRDD<K, W>, int) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the pairs from this whose keys are not in other.
subtractByKey(JavaPairRDD<K, W>, Partitioner) - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the pairs from this whose keys are not in other.
subtractByKey(RDD<Tuple2<K, W>>, ClassTag<W>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD with the pairs from this whose keys are not in other.
subtractByKey(RDD<Tuple2<K, W>>, int, ClassTag<W>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD with the pairs from this whose keys are not in other.
subtractByKey(RDD<Tuple2<K, W>>, Partitioner, ClassTag<W>) - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD with the pairs from this whose keys are not in other.
subtractMetrics(TaskMetrics, TaskMetrics) - Static method in class org.apache.spark.status.LiveEntityHelpers
Subtract m2 values from m1.
subtreeToString$default$1() - Static method in class org.apache.spark.ml.tree.InternalNode
 
succeededTasks() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
succeededTasks() - Method in class org.apache.spark.status.LiveExecutorStageSummary
 
success(T) - Static method in class org.apache.spark.ml.feature.RFormulaParser
 
Success - Class in org.apache.spark
Developer API Task succeeded.
Success() - Constructor for class org.apache.spark.Success
 
successful() - Method in class org.apache.spark.scheduler.TaskInfo
 
sum() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Add up the elements in this RDD.
Sum() - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
sum() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Add up the elements in this RDD.
sum(MapFunction<T, Double>) - Static method in class org.apache.spark.sql.expressions.javalang.typed
Sum aggregate function for floating point (double) type.
sum(Function1<IN, Object>) - Static method in class org.apache.spark.sql.expressions.scalalang.typed
Sum aggregate function for floating point (double) type.
sum(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sum of all values in the expression.
sum(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sum of all values in the given column.
sum(String...) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the sum for each numeric columns for each group.
sum(Seq<String>) - Method in class org.apache.spark.sql.RelationalGroupedDataset
Compute the sum for each numeric columns for each group.
sum(Numeric<B>) - Static method in class org.apache.spark.sql.types.StructType
 
sum() - Method in class org.apache.spark.util.DoubleAccumulator
Returns the sum of elements added to the accumulator.
sum() - Method in class org.apache.spark.util.LongAccumulator
Returns the sum of elements added to the accumulator.
sum() - Method in class org.apache.spark.util.StatCounter
 
sumApprox(long, Double) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Approximate operation to return the sum within a timeout.
sumApprox(long) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Approximate operation to return the sum within a timeout.
sumApprox(long, double) - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Approximate operation to return the sum within a timeout.
sumDistinct(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sum of distinct values in the expression.
sumDistinct(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the sum of distinct values in the expression.
sumLong(MapFunction<T, Long>) - Static method in class org.apache.spark.sql.expressions.javalang.typed
Sum aggregate function for integral (long, i.e.
sumLong(Function1<IN, Object>) - Static method in class org.apache.spark.sql.expressions.scalalang.typed
Sum aggregate function for integral (long, i.e.
Summarizer - Class in org.apache.spark.ml.stat
Tools for vectorized statistics on MLlib Vectors.
Summarizer() - Constructor for class org.apache.spark.ml.stat.Summarizer
 
summary() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
Gets summary of model on training set.
summary() - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
Gets summary of model on training set.
summary() - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
Gets summary of model on training set.
summary() - Method in class org.apache.spark.ml.clustering.KMeansModel
Gets summary of model on training set.
summary() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
Gets R-like summary of model on training set.
summary() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
Gets summary (e.g.
summary(Column, Column) - Method in class org.apache.spark.ml.stat.SummaryBuilder
Returns an aggregate object that contains the summary of the column with the requested metrics.
summary(Column) - Method in class org.apache.spark.ml.stat.SummaryBuilder
 
summary(String...) - Method in class org.apache.spark.sql.Dataset
Computes specified statistics for numeric and string columns.
summary(Seq<String>) - Method in class org.apache.spark.sql.Dataset
Computes specified statistics for numeric and string columns.
SummaryBuilder - Class in org.apache.spark.ml.stat
A builder object that provides summary statistics about a given column.
SummaryBuilder() - Constructor for class org.apache.spark.ml.stat.SummaryBuilder
 
supportBatch(SparkSession, StructType) - Static method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
supportedFeatureSubsetStrategies() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
Accessor for supported featureSubsetStrategy settings: auto, all, onethird, sqrt, log2
supportedFeatureSubsetStrategies() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
Accessor for supported featureSubsetStrategy settings: auto, all, onethird, sqrt, log2
supportedFeatureSubsetStrategies() - Static method in class org.apache.spark.mllib.tree.RandomForest
List of supported feature subset sampling strategies.
supportedImpurities() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
Accessor for supported impurities: entropy, gini
supportedImpurities() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
Accessor for supported impurity settings: entropy, gini
supportedImpurities() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
Accessor for supported impurities: variance
supportedImpurities() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
Accessor for supported impurity settings: variance
supportedLossTypes() - Static method in class org.apache.spark.ml.classification.GBTClassifier
Accessor for supported loss settings: logistic
supportedLossTypes() - Static method in class org.apache.spark.ml.regression.GBTRegressor
Accessor for supported loss settings: squared (L2), absolute (L1)
supportedOptimizers() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
supportedOptimizers() - Static method in class org.apache.spark.ml.clustering.LDA
 
supportedOptimizers() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
supportedSelectorTypes() - Static method in class org.apache.spark.mllib.feature.ChiSqSelector
Set of selector types that ChiSqSelector supports.
SupportsPushDownCatalystFilters - Interface in org.apache.spark.sql.sources.v2.reader
A mix-in interface for DataSourceReader.
SupportsPushDownFilters - Interface in org.apache.spark.sql.sources.v2.reader
A mix-in interface for DataSourceReader.
SupportsPushDownRequiredColumns - Interface in org.apache.spark.sql.sources.v2.reader
A mix-in interface for DataSourceReader.
SupportsReportPartitioning - Interface in org.apache.spark.sql.sources.v2.reader
A mix in interface for DataSourceReader.
SupportsReportStatistics - Interface in org.apache.spark.sql.sources.v2.reader
A mix in interface for DataSourceReader.
SupportsScanColumnarBatch - Interface in org.apache.spark.sql.sources.v2.reader
A mix-in interface for DataSourceReader.
SupportsScanUnsafeRow - Interface in org.apache.spark.sql.sources.v2.reader
A mix-in interface for DataSourceReader.
SupportsWriteInternalRow - Interface in org.apache.spark.sql.sources.v2.writer
A mix-in interface for DataSourceWriter.
surrogateDF() - Method in class org.apache.spark.ml.feature.ImputerModel
 
SVDPlusPlus - Class in org.apache.spark.graphx.lib
Implementation of SVD++ algorithm.
SVDPlusPlus() - Constructor for class org.apache.spark.graphx.lib.SVDPlusPlus
 
SVDPlusPlus.Conf - Class in org.apache.spark.graphx.lib
Configuration parameters for SVDPlusPlus.
SVMDataGenerator - Class in org.apache.spark.mllib.util
Developer API Generate sample data used for SVM.
SVMDataGenerator() - Constructor for class org.apache.spark.mllib.util.SVMDataGenerator
 
SVMModel - Class in org.apache.spark.mllib.classification
Model for Support Vector Machines (SVMs).
SVMModel(Vector, double) - Constructor for class org.apache.spark.mllib.classification.SVMModel
 
SVMWithSGD - Class in org.apache.spark.mllib.classification
Train a Support Vector Machine (SVM) using Stochastic Gradient Descent.
SVMWithSGD() - Constructor for class org.apache.spark.mllib.classification.SVMWithSGD
Construct a SVM object with default parameters: {stepSize: 1.0, numIterations: 100, regParm: 0.01, miniBatchFraction: 1.0}.
symbolToColumn(Symbol) - Method in class org.apache.spark.sql.SQLImplicits
An implicit conversion that turns a Scala Symbol into a Column.
symlink(File, File) - Static method in class org.apache.spark.util.Utils
Creates a symlink.
symmetricEigs(Function1<DenseVector<Object>, DenseVector<Object>>, int, int, double, int) - Static method in class org.apache.spark.mllib.linalg.EigenValueDecomposition
Compute the leading k eigenvalues and eigenvectors on a symmetric square matrix using ARPACK.
syr(double, Vector, DenseMatrix) - Static method in class org.apache.spark.ml.linalg.BLAS
A := alpha * x * x^T^ + A
syr(double, Vector, DenseMatrix) - Static method in class org.apache.spark.mllib.linalg.BLAS
A := alpha * x * x^T^ + A
SYSTEM_DEFAULT() - Static method in class org.apache.spark.sql.types.DecimalType
 
systemProperties() - Method in class org.apache.spark.status.api.v1.ApplicationEnvironmentInfo
 

T

t() - Method in class org.apache.spark.SerializableWritable
 
Table - Class in org.apache.spark.sql.catalog
A table in Spark, as returned by the listTables method in Catalog.
Table(String, String, String, String, boolean) - Constructor for class org.apache.spark.sql.catalog.Table
 
table(String) - Method in class org.apache.spark.sql.DataFrameReader
Returns the specified table as a DataFrame.
table() - Method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
table(String) - Method in class org.apache.spark.sql.SparkSession
Returns the specified table/view as a DataFrame.
table(String) - Method in class org.apache.spark.sql.SQLContext
 
TABLE_CLASS_NOT_STRIPED() - Static method in class org.apache.spark.ui.UIUtils
 
TABLE_CLASS_STRIPED() - Static method in class org.apache.spark.ui.UIUtils
 
TABLE_CLASS_STRIPED_SORTABLE() - Static method in class org.apache.spark.ui.UIUtils
 
tableDesc() - Method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
tableExists(String) - Method in class org.apache.spark.sql.catalog.Catalog
Check if the table or view with the specified name exists.
tableExists(String, String) - Method in class org.apache.spark.sql.catalog.Catalog
Check if the table or view with the specified name exists in the specified database.
tableNames() - Method in class org.apache.spark.sql.SQLContext
 
tableNames(String) - Method in class org.apache.spark.sql.SQLContext
 
tables() - Method in class org.apache.spark.sql.SQLContext
 
tables(String) - Method in class org.apache.spark.sql.SQLContext
 
TableScan - Interface in org.apache.spark.sql.sources
A BaseRelation that can produce all of its tuples as an RDD of Row objects.
tableType() - Method in class org.apache.spark.sql.catalog.Table
 
tail() - Static method in class org.apache.spark.sql.types.StructType
 
tails() - Static method in class org.apache.spark.sql.types.StructType
 
take(int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
take(int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
take(int) - Static method in class org.apache.spark.api.java.JavaRDD
 
take(int) - Method in interface org.apache.spark.api.java.JavaRDDLike
Take the first num elements of the RDD.
take(int) - Static method in class org.apache.spark.api.r.RRDD
 
take(int) - Static method in class org.apache.spark.graphx.EdgeRDD
 
take(int) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
take(int) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
take(int) - Static method in class org.apache.spark.graphx.VertexRDD
 
take(int) - Static method in class org.apache.spark.rdd.HadoopRDD
 
take(int) - Static method in class org.apache.spark.rdd.JdbcRDD
 
take(int) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
take(int) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
take(int) - Method in class org.apache.spark.rdd.RDD
Take the first num elements of the RDD.
take(int) - Static method in class org.apache.spark.rdd.UnionRDD
 
take(int) - Method in class org.apache.spark.sql.Dataset
Returns the first n rows in the Dataset.
take(int) - Static method in class org.apache.spark.sql.types.StructType
 
takeAsList(int) - Method in class org.apache.spark.sql.Dataset
Returns the first n rows in the Dataset as a list.
takeAsync(int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
takeAsync(int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
takeAsync(int) - Static method in class org.apache.spark.api.java.JavaRDD
 
takeAsync(int) - Method in interface org.apache.spark.api.java.JavaRDDLike
The asynchronous version of the take action, which returns a future for retrieving the first num elements of this RDD.
takeAsync(int) - Method in class org.apache.spark.rdd.AsyncRDDActions
Returns a future for retrieving the first num elements of the RDD.
takeOrdered(int, Comparator<T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
takeOrdered(int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
takeOrdered(int, Comparator<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
takeOrdered(int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
takeOrdered(int, Comparator<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
takeOrdered(int) - Static method in class org.apache.spark.api.java.JavaRDD
 
takeOrdered(int, Comparator<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Returns the first k (smallest) elements from this RDD as defined by the specified Comparator[T] and maintains the order.
takeOrdered(int) - Method in interface org.apache.spark.api.java.JavaRDDLike
Returns the first k (smallest) elements from this RDD using the natural ordering for T while maintain the order.
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
takeOrdered(int, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Returns the first k (smallest) elements from this RDD as defined by the specified implicit Ordering[T] and maintains the ordering.
takeOrdered(int, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
takeRight(int) - Static method in class org.apache.spark.sql.types.StructType
 
takeSample(boolean, int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
takeSample(boolean, int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
takeSample(boolean, int) - Static method in class org.apache.spark.api.java.JavaRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.api.java.JavaRDD
 
takeSample(boolean, int) - Method in interface org.apache.spark.api.java.JavaRDDLike
 
takeSample(boolean, int, long) - Method in interface org.apache.spark.api.java.JavaRDDLike
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.api.r.RRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.graphx.EdgeRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.graphx.VertexRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.rdd.HadoopRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.rdd.JdbcRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
takeSample(boolean, int, long) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
takeSample(boolean, int, long) - Method in class org.apache.spark.rdd.RDD
Return a fixed-size sampled subset of this RDD in an array
takeSample(boolean, int, long) - Static method in class org.apache.spark.rdd.UnionRDD
 
takeSample$default$3() - Static method in class org.apache.spark.api.r.RRDD
 
takeSample$default$3() - Static method in class org.apache.spark.graphx.EdgeRDD
 
takeSample$default$3() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
takeSample$default$3() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
takeSample$default$3() - Static method in class org.apache.spark.graphx.VertexRDD
 
takeSample$default$3() - Static method in class org.apache.spark.rdd.HadoopRDD
 
takeSample$default$3() - Static method in class org.apache.spark.rdd.JdbcRDD
 
takeSample$default$3() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
takeSample$default$3() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
takeSample$default$3() - Static method in class org.apache.spark.rdd.UnionRDD
 
takeWhile(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
tallSkinnyQR(boolean) - Method in class org.apache.spark.mllib.linalg.distributed.RowMatrix
Compute QR decomposition for RowMatrix.
tan(Column) - Static method in class org.apache.spark.sql.functions
 
tan(String) - Static method in class org.apache.spark.sql.functions
 
tanh(Column) - Static method in class org.apache.spark.sql.functions
 
tanh(String) - Static method in class org.apache.spark.sql.functions
 
targetStorageLevel() - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
targetStorageLevel() - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
task() - Method in class org.apache.spark.CleanupTaskWeakReference
 
TASK_DESERIALIZATION_TIME() - Static method in class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
TASK_DESERIALIZATION_TIME() - Static method in class org.apache.spark.ui.ToolTips
 
TASK_INDEX() - Static method in class org.apache.spark.status.TaskIndexNames
 
TASK_TIME() - Static method in class org.apache.spark.ui.ToolTips
 
taskAttemptId() - Method in class org.apache.spark.TaskContext
An ID that is unique to this task attempt (within the same SparkContext, no two task attempts will share the same attempt ID).
TaskCommitDenied - Class in org.apache.spark
Developer API Task requested the driver to commit, but was denied.
TaskCommitDenied(int, int, int) - Constructor for class org.apache.spark.TaskCommitDenied
 
TaskCommitMessage(Object) - Constructor for class org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage
 
TaskCompletionListener - Interface in org.apache.spark.util
Developer API
TaskContext - Class in org.apache.spark
Contextual information about a task which can be read or mutated during execution.
TaskContext() - Constructor for class org.apache.spark.TaskContext
 
TaskData - Class in org.apache.spark.status.api.v1
 
TaskDetailsClassNames - Class in org.apache.spark.ui.jobs
Names of the CSS classes corresponding to each type of task detail.
TaskDetailsClassNames() - Constructor for class org.apache.spark.ui.jobs.TaskDetailsClassNames
 
taskEndFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
TaskEndReason - Interface in org.apache.spark
Developer API Various possible reasons why a task ended.
taskEndReasonFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
taskEndReasonToJson(TaskEndReason) - Static method in class org.apache.spark.util.JsonProtocol
 
taskEndToJson(SparkListenerTaskEnd) - Static method in class org.apache.spark.util.JsonProtocol
 
TaskFailedReason - Interface in org.apache.spark
Developer API Various possible reasons why a task failed.
TaskFailureListener - Interface in org.apache.spark.util
Developer API
taskFailures() - Method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
taskGettingResultFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
taskGettingResultToJson(SparkListenerTaskGettingResult) - Static method in class org.apache.spark.util.JsonProtocol
 
taskId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.KillTask
 
taskId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.StatusUpdate
 
taskId() - Method in class org.apache.spark.scheduler.local.KillTask
 
taskId() - Method in class org.apache.spark.scheduler.local.StatusUpdate
 
taskId() - Method in class org.apache.spark.scheduler.TaskInfo
 
taskId() - Method in class org.apache.spark.status.api.v1.TaskData
 
taskId() - Method in class org.apache.spark.storage.TaskResultBlockId
 
TaskIndexNames - Class in org.apache.spark.status
Tasks have a lot of indices that are used in a few different places.
TaskIndexNames() - Constructor for class org.apache.spark.status.TaskIndexNames
 
taskInfo() - Method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
taskInfo() - Method in class org.apache.spark.scheduler.SparkListenerTaskGettingResult
 
taskInfo() - Method in class org.apache.spark.scheduler.SparkListenerTaskStart
 
TaskInfo - Class in org.apache.spark.scheduler
Developer API Information about a running task attempt inside a TaskSet.
TaskInfo(long, int, int, long, String, String, Enumeration.Value, boolean) - Constructor for class org.apache.spark.scheduler.TaskInfo
 
taskInfoFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
taskInfoToJson(TaskInfo) - Static method in class org.apache.spark.util.JsonProtocol
 
TaskKilled - Class in org.apache.spark
Developer API Task was killed intentionally and needs to be rescheduled.
TaskKilled(String) - Constructor for class org.apache.spark.TaskKilled
 
TaskKilledException - Exception in org.apache.spark
Developer API Exception thrown when a task is explicitly killed (i.e., task failure is expected).
TaskKilledException(String) - Constructor for exception org.apache.spark.TaskKilledException
 
TaskKilledException() - Constructor for exception org.apache.spark.TaskKilledException
 
taskLocality() - Method in class org.apache.spark.scheduler.TaskInfo
 
TaskLocality - Class in org.apache.spark.scheduler
 
TaskLocality() - Constructor for class org.apache.spark.scheduler.TaskLocality
 
taskLocality() - Method in class org.apache.spark.status.api.v1.TaskData
 
TaskMetricDistributions - Class in org.apache.spark.status.api.v1
 
taskMetrics() - Method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
taskMetrics() - Method in class org.apache.spark.scheduler.StageInfo
 
taskMetrics() - Method in class org.apache.spark.status.api.v1.TaskData
 
TaskMetrics - Class in org.apache.spark.status.api.v1
 
taskMetrics() - Method in class org.apache.spark.TaskContext
 
taskMetricsFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
taskMetricsToJson(TaskMetrics) - Static method in class org.apache.spark.util.JsonProtocol
 
TASKRESULT() - Static method in class org.apache.spark.storage.BlockId
 
TaskResultBlockId - Class in org.apache.spark.storage
 
TaskResultBlockId(long) - Constructor for class org.apache.spark.storage.TaskResultBlockId
 
TaskResultLost - Class in org.apache.spark
Developer API The task finished successfully, but the result was lost from the executor's block manager before it was fetched.
TaskResultLost() - Constructor for class org.apache.spark.TaskResultLost
 
tasks() - Method in class org.apache.spark.status.api.v1.StageData
 
TaskSchedulerIsSet - Class in org.apache.spark
An event that SparkContext uses to notify HeartbeatReceiver that SparkContext.taskScheduler is created.
TaskSchedulerIsSet() - Constructor for class org.apache.spark.TaskSchedulerIsSet
 
TaskSorting - Enum in org.apache.spark.status.api.v1
 
taskStartFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
taskStartToJson(SparkListenerTaskStart) - Static method in class org.apache.spark.util.JsonProtocol
 
TaskState - Class in org.apache.spark
 
TaskState() - Constructor for class org.apache.spark.TaskState
 
taskTime() - Method in class org.apache.spark.status.api.v1.ExecutorStageSummary
 
taskTime() - Method in class org.apache.spark.status.LiveExecutorStageSummary
 
taskType() - Method in class org.apache.spark.scheduler.SparkListenerTaskEnd
 
TEMP_DIR_SHUTDOWN_PRIORITY() - Static method in class org.apache.spark.util.ShutdownHookManager
The shutdown priority of temp directory must be lower than the SparkContext shutdown priority.
TEMP_LOCAL() - Static method in class org.apache.spark.storage.BlockId
 
TEMP_SHUFFLE() - Static method in class org.apache.spark.storage.BlockId
 
tempFileWith(File) - Static method in class org.apache.spark.util.Utils
Returns a path of temporary file which is in the same directory with path.
TeradataDialect - Class in org.apache.spark.sql.jdbc
 
TeradataDialect() - Constructor for class org.apache.spark.sql.jdbc.TeradataDialect
 
terminateProcess(Process, long) - Static method in class org.apache.spark.util.Utils
Terminates a process waiting for at most the specified duration.
test(Dataset<Row>, String, String) - Static method in class org.apache.spark.ml.stat.ChiSquareTest
Conduct Pearson's independence test for every feature against the label.
TEST() - Static method in class org.apache.spark.storage.BlockId
 
TEST_ACCUM() - Static method in class org.apache.spark.InternalAccumulator
 
testCommandAvailable(String) - Static method in class org.apache.spark.TestUtils
Test if a command is available.
testOneSample(RDD<Object>, String, double...) - Static method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest
A convenience function that allows running the KS test for 1 set of sample data against a named distribution
testOneSample(RDD<Object>, Function1<Object, Object>) - Static method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest
 
testOneSample(RDD<Object>, RealDistribution) - Static method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest
 
testOneSample(RDD<Object>, String, Seq<Object>) - Static method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTest
 
TestResult<DF> - Interface in org.apache.spark.mllib.stat.test
Trait for hypothesis test results.
TestUtils - Class in org.apache.spark
Utilities for tests.
TestUtils() - Constructor for class org.apache.spark.TestUtils
 
text(String...) - Method in class org.apache.spark.sql.DataFrameReader
Loads text files and returns a DataFrame whose schema starts with a string column named "value", and followed by partitioned columns if there are any.
text(String) - Method in class org.apache.spark.sql.DataFrameReader
Loads text files and returns a DataFrame whose schema starts with a string column named "value", and followed by partitioned columns if there are any.
text(Seq<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads text files and returns a DataFrame whose schema starts with a string column named "value", and followed by partitioned columns if there are any.
text(String) - Method in class org.apache.spark.sql.DataFrameWriter
Saves the content of the DataFrame in a text file at the specified path.
text(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads text files and returns a DataFrame whose schema starts with a string column named "value", and followed by partitioned columns if there are any.
textFile(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
textFile(String, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
textFile(String, int) - Method in class org.apache.spark.SparkContext
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI, and return it as an RDD of Strings.
textFile(String...) - Method in class org.apache.spark.sql.DataFrameReader
Loads text files and returns a Dataset of String.
textFile(String) - Method in class org.apache.spark.sql.DataFrameReader
Loads text files and returns a Dataset of String.
textFile(Seq<String>) - Method in class org.apache.spark.sql.DataFrameReader
Loads text files and returns a Dataset of String.
textFile(String) - Method in class org.apache.spark.sql.streaming.DataStreamReader
Loads text file(s) and returns a Dataset of String.
textFileStream(String) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as text files (using key as LongWritable, value as Text and input format as TextInputFormat).
textFileStream(String) - Method in class org.apache.spark.streaming.StreamingContext
Create an input stream that monitors a Hadoop-compatible filesystem for new files and reads them as text files (using key as LongWritable, value as Text and input format as TextInputFormat).
textResponderToServlet(Function1<HttpServletRequest, String>) - Static method in class org.apache.spark.ui.JettyUtils
 
theta() - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
theta() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$.Data
 
theta() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$.Data
 
theta() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel
 
thisClassName() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$
Hard-code class name string in case it changes in the future
thisClassName() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$
Hard-code class name string in case it changes in the future
thisClassName() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
 
thisFormatVersion() - Method in class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$
 
thisFormatVersion() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV1_0$
 
thisFormatVersion() - Method in class org.apache.spark.mllib.classification.NaiveBayesModel.SaveLoadV2_0$
 
thisFormatVersion() - Method in class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$
 
thisFormatVersion() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$
 
ThreadUtils - Class in org.apache.spark.util
 
ThreadUtils() - Constructor for class org.apache.spark.util.ThreadUtils
 
threshold() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
threshold() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
threshold() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
threshold() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
threshold() - Method in class org.apache.spark.ml.feature.Binarizer
Param for threshold used to binarize continuous features.
threshold() - Method in interface org.apache.spark.ml.param.shared.HasThreshold
Param for threshold in binary classification prediction, in range [0, 1].
threshold() - Method in class org.apache.spark.ml.tree.ContinuousSplit
 
threshold() - Method in class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$.Data
 
threshold() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.SplitData
 
threshold() - Method in class org.apache.spark.mllib.tree.model.Split
 
thresholds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
thresholds() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
thresholds() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
thresholds() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
thresholds() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
thresholds() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
thresholds() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
thresholds() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
thresholds() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
thresholds() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
thresholds() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
thresholds() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
thresholds() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
thresholds() - Method in interface org.apache.spark.ml.param.shared.HasThresholds
Param for Thresholds in multi-class classification to adjust the probability of predicting each class.
thresholds() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Returns thresholds in descending order.
throwBalls(int, RDD<?>, double, DefaultPartitionCoalescer.PartitionLocations) - Method in class org.apache.spark.rdd.DefaultPartitionCoalescer
 
time() - Method in class org.apache.spark.scheduler.SparkListenerApplicationEnd
 
time() - Method in class org.apache.spark.scheduler.SparkListenerApplicationStart
 
time() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerAdded
 
time() - Method in class org.apache.spark.scheduler.SparkListenerBlockManagerRemoved
 
time() - Method in class org.apache.spark.scheduler.SparkListenerExecutorAdded
 
time() - Method in class org.apache.spark.scheduler.SparkListenerExecutorBlacklisted
 
time() - Method in class org.apache.spark.scheduler.SparkListenerExecutorRemoved
 
time() - Method in class org.apache.spark.scheduler.SparkListenerExecutorUnblacklisted
 
time() - Method in class org.apache.spark.scheduler.SparkListenerJobEnd
 
time() - Method in class org.apache.spark.scheduler.SparkListenerJobStart
 
time() - Method in class org.apache.spark.scheduler.SparkListenerNodeBlacklisted
 
time() - Method in class org.apache.spark.scheduler.SparkListenerNodeUnblacklisted
 
time(Function0<T>) - Method in class org.apache.spark.sql.SparkSession
Executes some code block and prints to stdout the time taken to execute the block.
time() - Method in exception org.apache.spark.sql.streaming.StreamingQueryException
Time when the exception occurred
time() - Method in class org.apache.spark.streaming.scheduler.StreamingListenerStreamingStarted
 
Time - Class in org.apache.spark.streaming
This is a simple class that represents an absolute instant of time.
Time(long) - Constructor for class org.apache.spark.streaming.Time
 
timeFromString(String, TimeUnit) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
timeIt(int, Function0<BoxedUnit>, Option<Function0<BoxedUnit>>) - Static method in class org.apache.spark.util.Utils
Timing method based on iterations that permit JVM JIT optimization.
timeout(Duration) - Method in class org.apache.spark.streaming.StateSpec
Set the duration after which the state of an idle key will be removed.
TIMER() - Static method in class org.apache.spark.metrics.sink.StatsdMetricType
 
times(int) - Method in class org.apache.spark.streaming.Duration
 
times(int, Function0<BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Method executed for repeating a task for side effects.
timestamp() - Method in class org.apache.spark.sql.ColumnName
Creates a new StructField of type timestamp.
TIMESTAMP() - Static method in class org.apache.spark.sql.Encoders
An encoder for nullable timestamp type.
timestamp() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
TimestampType - Static variable in class org.apache.spark.sql.types.DataTypes
Gets the TimestampType object.
TimestampType - Class in org.apache.spark.sql.types
The data type representing java.sql.Timestamp values.
timeStringAsMs(String) - Static method in class org.apache.spark.util.Utils
Convert a time parameter such as (50s, 100ms, or 250us) to milliseconds for internal use.
timeStringAsSeconds(String) - Static method in class org.apache.spark.util.Utils
Convert a time parameter such as (50s, 100ms, or 250us) to seconds for internal use.
timeToString(long, TimeUnit) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
TimeTrackingOutputStream - Class in org.apache.spark.storage
Intercepts write calls and tracks total time spent writing in order to update shuffle write metrics.
TimeTrackingOutputStream(ShuffleWriteMetrics, OutputStream) - Constructor for class org.apache.spark.storage.TimeTrackingOutputStream
 
timeUnit() - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
 
TIMING_DATA() - Static method in class org.apache.spark.api.r.SpecialLengths
 
to(CanBuildFrom<Nothing$, A, Col>) - Static method in class org.apache.spark.sql.types.StructType
 
to(Time, Duration) - Method in class org.apache.spark.streaming.Time
 
to_date(Column) - Static method in class org.apache.spark.sql.functions
Converts the column into DateType by casting rules to DateType.
to_date(Column, String) - Static method in class org.apache.spark.sql.functions
Converts the column into a DateType with a specified format (see [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]) return null if fail.
to_json(Column, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Scala-specific) Converts a column containing a StructType, ArrayType of StructTypes, a MapType or ArrayType of MapTypes into a JSON string with the specified schema.
to_json(Column, Map<String, String>) - Static method in class org.apache.spark.sql.functions
(Java-specific) Converts a column containing a StructType, ArrayType of StructTypes, a MapType or ArrayType of MapTypes into a JSON string with the specified schema.
to_json(Column) - Static method in class org.apache.spark.sql.functions
Converts a column containing a StructType, ArrayType of StructTypes, a MapType or ArrayType of MapTypes into a JSON string with the specified schema.
to_timestamp(Column) - Static method in class org.apache.spark.sql.functions
Convert time string to a Unix timestamp (in seconds) by casting rules to TimestampType.
to_timestamp(Column, String) - Static method in class org.apache.spark.sql.functions
Convert time string to a Unix timestamp (in seconds) with a specified format (see [http://docs.oracle.com/javase/tutorial/i18n/format/simpleDateFormat.html]) to Unix timestamp (in seconds), return null if fail.
to_utc_timestamp(Column, String) - Static method in class org.apache.spark.sql.functions
Given a timestamp like '2017-07-14 02:40:00.0', interprets it as a time in the given time zone, and renders that time as a timestamp in UTC.
toApi() - Method in class org.apache.spark.status.LiveRDDDistribution
 
toApi() - Method in class org.apache.spark.status.LiveStage
 
toArray() - Method in class org.apache.spark.input.PortableDataStream
Read the file as a byte array
toArray() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toArray() - Method in class org.apache.spark.ml.linalg.DenseVector
 
toArray() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts to a dense array in column major.
toArray() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toArray() - Method in class org.apache.spark.ml.linalg.SparseVector
 
toArray() - Method in interface org.apache.spark.ml.linalg.Vector
Converts the instance to a double array.
toArray() - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
 
toArray() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
toArray() - Method in interface org.apache.spark.mllib.linalg.Matrix
Converts to a dense array in column major.
toArray() - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
 
toArray() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
toArray() - Method in interface org.apache.spark.mllib.linalg.Vector
Converts the instance to a double array.
toArray(ClassTag<B>) - Static method in class org.apache.spark.sql.types.StructType
 
toBigDecimal() - Method in class org.apache.spark.sql.types.Decimal
 
toBlockMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Converts to BlockMatrix.
toBlockMatrix(int, int) - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Converts to BlockMatrix.
toBlockMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Converts to BlockMatrix.
toBlockMatrix(int, int) - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Converts to BlockMatrix.
toBoolean(String, String) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
toBooleanArray() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
toBreeze() - Method in interface org.apache.spark.mllib.linalg.distributed.DistributedMatrix
Collects data and assembles a local dense breeze matrix (for test only).
toBuffer() - Static method in class org.apache.spark.sql.types.StructType
 
toByte() - Method in class org.apache.spark.sql.types.Decimal
 
toByteArray() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
toByteArray() - Method in class org.apache.spark.util.sketch.CountMinSketch
Serializes this CountMinSketch and returns the serialized form.
toByteBuffer() - Method in class org.apache.spark.storage.DiskBlockData
 
toByteBuffer() - Method in class org.apache.spark.storage.EncryptedBlockData
 
toCatalystDecimal(HiveDecimalObjectInspector, Object) - Static method in class org.apache.spark.sql.hive.HiveShim
 
toChunkedByteBuffer(Function1<Object, ByteBuffer>) - Method in class org.apache.spark.storage.DiskBlockData
 
toChunkedByteBuffer(Function1<Object, ByteBuffer>) - Method in class org.apache.spark.storage.EncryptedBlockData
 
toColumn() - Method in class org.apache.spark.sql.expressions.Aggregator
Returns this Aggregator as a TypedColumn that can be used in Dataset.
toCoordinateMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Converts to CoordinateMatrix.
toCoordinateMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Converts this matrix to a CoordinateMatrix.
toCryptoConf(SparkConf) - Static method in class org.apache.spark.security.CryptoStreamUtils
 
toDebugString() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
toDebugString() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
toDebugString() - Static method in class org.apache.spark.api.java.JavaRDD
 
toDebugString() - Method in interface org.apache.spark.api.java.JavaRDDLike
A description of this RDD and its recursive dependencies for debugging.
toDebugString() - Static method in class org.apache.spark.api.r.RRDD
 
toDebugString() - Static method in class org.apache.spark.graphx.EdgeRDD
 
toDebugString() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
toDebugString() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
toDebugString() - Static method in class org.apache.spark.graphx.VertexRDD
 
toDebugString() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
toDebugString() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
toDebugString() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
toDebugString() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
toDebugString() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
toDebugString() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
toDebugString() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Print the full model to a string.
toDebugString() - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
toDebugString() - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
toDebugString() - Static method in class org.apache.spark.rdd.HadoopRDD
 
toDebugString() - Static method in class org.apache.spark.rdd.JdbcRDD
 
toDebugString() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
toDebugString() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
toDebugString() - Method in class org.apache.spark.rdd.RDD
A description of this RDD and its recursive dependencies for debugging.
toDebugString() - Static method in class org.apache.spark.rdd.UnionRDD
 
toDebugString() - Method in class org.apache.spark.SparkConf
Return a string listing all keys and values, one per line.
toDebugString() - Method in class org.apache.spark.sql.types.Decimal
 
toDegrees(Column) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use degrees. Since 2.1.0.
toDegrees(String) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use degrees. Since 2.1.0.
toDense() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toDense() - Static method in class org.apache.spark.ml.linalg.DenseVector
 
toDense() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a dense matrix while maintaining the layout of the current matrix.
toDense() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toDense() - Static method in class org.apache.spark.ml.linalg.SparseVector
 
toDense() - Method in interface org.apache.spark.ml.linalg.Vector
Converts this vector to a dense vector.
toDense() - Static method in class org.apache.spark.mllib.linalg.DenseVector
 
toDense() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
Generate a DenseMatrix from the given SparseMatrix.
toDense() - Static method in class org.apache.spark.mllib.linalg.SparseVector
 
toDense() - Method in interface org.apache.spark.mllib.linalg.Vector
Converts this vector to a dense vector.
toDenseColMajor() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toDenseColMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a dense matrix in column major order.
toDenseColMajor() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toDenseMatrix(boolean) - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a dense matrix.
toDenseRowMajor() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toDenseRowMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a dense matrix in row major order.
toDenseRowMajor() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toDF(String...) - Method in class org.apache.spark.sql.Dataset
Converts this strongly typed collection of data to generic DataFrame with columns renamed.
toDF() - Method in class org.apache.spark.sql.Dataset
Converts this strongly typed collection of data to generic Dataframe.
toDF(Seq<String>) - Method in class org.apache.spark.sql.Dataset
Converts this strongly typed collection of data to generic DataFrame with columns renamed.
toDF() - Method in class org.apache.spark.sql.DatasetHolder
 
toDF(Seq<String>) - Method in class org.apache.spark.sql.DatasetHolder
 
toDouble() - Method in class org.apache.spark.sql.types.Decimal
 
toDoubleArray() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
toDS() - Method in class org.apache.spark.sql.DatasetHolder
 
toEdgeTriplet() - Method in class org.apache.spark.graphx.EdgeContext
Converts the edge and vertex properties into an EdgeTriplet for convenience.
toErrorString() - Method in class org.apache.spark.ExceptionFailure
 
toErrorString() - Method in class org.apache.spark.ExecutorLostFailure
 
toErrorString() - Method in class org.apache.spark.FetchFailed
 
toErrorString() - Static method in class org.apache.spark.Resubmitted
 
toErrorString() - Method in class org.apache.spark.TaskCommitDenied
 
toErrorString() - Method in interface org.apache.spark.TaskFailedReason
Error message displayed in the web UI.
toErrorString() - Method in class org.apache.spark.TaskKilled
 
toErrorString() - Static method in class org.apache.spark.TaskResultLost
 
toErrorString() - Static method in class org.apache.spark.UnknownReason
 
toFloat() - Method in class org.apache.spark.sql.types.Decimal
 
toFloatArray() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
toFormattedString() - Method in class org.apache.spark.streaming.Duration
 
toIndexedRowMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Converts to IndexedRowMatrix.
toIndexedRowMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Converts to IndexedRowMatrix.
toIndexedSeq() - Static method in class org.apache.spark.sql.types.StructType
 
toInputStream() - Method in class org.apache.spark.storage.DiskBlockData
 
toInputStream() - Method in class org.apache.spark.storage.EncryptedBlockData
 
toInspector(DataType) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
toInspector(Expression) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
toInt() - Method in class org.apache.spark.sql.types.Decimal
 
toInt() - Method in class org.apache.spark.storage.StorageLevel
 
toIntArray() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
toIterable() - Static method in class org.apache.spark.sql.types.StructType
 
toIterator() - Static method in class org.apache.spark.sql.types.StructType
 
toJavaBigDecimal() - Method in class org.apache.spark.sql.types.Decimal
 
toJavaBigInteger() - Method in class org.apache.spark.sql.types.Decimal
 
toJavaDStream() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Convert to a JavaDStream
toJavaDStream() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
toJavaDStream() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
toJavaRDD() - Static method in class org.apache.spark.api.r.RRDD
 
toJavaRDD() - Static method in class org.apache.spark.graphx.EdgeRDD
 
toJavaRDD() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
toJavaRDD() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
toJavaRDD() - Static method in class org.apache.spark.graphx.VertexRDD
 
toJavaRDD() - Static method in class org.apache.spark.rdd.HadoopRDD
 
toJavaRDD() - Static method in class org.apache.spark.rdd.JdbcRDD
 
toJavaRDD() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
toJavaRDD() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
toJavaRDD() - Method in class org.apache.spark.rdd.RDD
 
toJavaRDD() - Static method in class org.apache.spark.rdd.UnionRDD
 
toJavaRDD() - Method in class org.apache.spark.sql.Dataset
Returns the content of the Dataset as a JavaRDD of Ts.
toJson(Matrix) - Static method in class org.apache.spark.ml.linalg.JsonMatrixConverter
Coverts the Matrix to a JSON string.
toJson(Vector) - Static method in class org.apache.spark.ml.linalg.JsonVectorConverter
Coverts the vector to a JSON string.
toJson() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
toJson() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
toJson() - Method in interface org.apache.spark.mllib.linalg.Vector
Converts the vector to a JSON string.
toJSON() - Method in class org.apache.spark.sql.Dataset
Returns the content of the Dataset as a Dataset of JSON strings.
toJSON() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
toJSON() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
toJSON() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
toJSON() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
Tokenizer - Class in org.apache.spark.ml.feature
A tokenizer that converts the input string to lowercase and then splits it by white spaces.
Tokenizer(String) - Constructor for class org.apache.spark.ml.feature.Tokenizer
 
Tokenizer() - Constructor for class org.apache.spark.ml.feature.Tokenizer
 
tokens() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.UpdateDelegationTokens
 
tol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
tol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
tol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
tol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
tol() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
tol() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
tol() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
tol() - Static method in class org.apache.spark.ml.clustering.KMeans
 
tol() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
tol() - Method in interface org.apache.spark.ml.param.shared.HasTol
Param for the convergence tolerance for iterative algorithms (&gt;= 0).
tol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
tol() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
tol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
tol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
tol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
tol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
toList() - Static method in class org.apache.spark.sql.types.StructType
 
toLocal() - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
Convert this distributed model to a local representation.
toLocal() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Convert model to a local model.
toLocalIterator() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
toLocalIterator() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
toLocalIterator() - Static method in class org.apache.spark.api.java.JavaRDD
 
toLocalIterator() - Method in interface org.apache.spark.api.java.JavaRDDLike
Return an iterator that contains all of the elements in this RDD.
toLocalIterator() - Static method in class org.apache.spark.api.r.RRDD
 
toLocalIterator() - Static method in class org.apache.spark.graphx.EdgeRDD
 
toLocalIterator() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
toLocalIterator() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
toLocalIterator() - Static method in class org.apache.spark.graphx.VertexRDD
 
toLocalIterator() - Static method in class org.apache.spark.rdd.HadoopRDD
 
toLocalIterator() - Static method in class org.apache.spark.rdd.JdbcRDD
 
toLocalIterator() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
toLocalIterator() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
toLocalIterator() - Method in class org.apache.spark.rdd.RDD
Return an iterator that contains all of the elements in this RDD.
toLocalIterator() - Static method in class org.apache.spark.rdd.UnionRDD
 
toLocalIterator() - Method in class org.apache.spark.sql.Dataset
Returns an iterator that contains all rows in this Dataset.
toLocalMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Collect the distributed matrix on the driver as a DenseMatrix.
toLong() - Method in class org.apache.spark.sql.types.Decimal
 
toLongArray() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
toLowercase() - Method in class org.apache.spark.ml.feature.RegexTokenizer
Indicates whether to convert all characters to lowercase before tokenizing.
toMap(Predef.$less$colon$less<A, Tuple2<T, U>>) - Static method in class org.apache.spark.sql.types.StructType
 
toMetadata(Metadata) - Method in class org.apache.spark.ml.attribute.Attribute
Converts to ML metadata with some existing metadata.
toMetadata() - Method in class org.apache.spark.ml.attribute.Attribute
Converts to ML metadata
toMetadata(Metadata) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Converts to ML metadata with some existing metadata.
toMetadata() - Method in class org.apache.spark.ml.attribute.AttributeGroup
Converts to ML metadata
toMetadata(Metadata) - Static method in class org.apache.spark.ml.attribute.BinaryAttribute
 
toMetadata() - Static method in class org.apache.spark.ml.attribute.BinaryAttribute
 
toMetadata(Metadata) - Static method in class org.apache.spark.ml.attribute.NominalAttribute
 
toMetadata() - Static method in class org.apache.spark.ml.attribute.NominalAttribute
 
toMetadata(Metadata) - Static method in class org.apache.spark.ml.attribute.NumericAttribute
 
toMetadata() - Static method in class org.apache.spark.ml.attribute.NumericAttribute
 
toMetadata(Metadata) - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
toMetadata() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
toNetty() - Method in class org.apache.spark.storage.DiskBlockData
Returns a Netty-friendly wrapper for the block's data.
toNetty() - Method in class org.apache.spark.storage.EncryptedBlockData
 
toNumber(String, Function1<String, T>, String, String) - Static method in class org.apache.spark.internal.config.ConfigHelpers
 
toOld() - Method in interface org.apache.spark.ml.tree.Split
Convert to old Split format
tooltip(String, String) - Static method in class org.apache.spark.ui.UIUtils
 
ToolTips - Class in org.apache.spark.ui
 
ToolTips() - Constructor for class org.apache.spark.ui.ToolTips
 
top(int, Comparator<T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
top(int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
top(int, Comparator<T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
top(int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
top(int, Comparator<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
top(int) - Static method in class org.apache.spark.api.java.JavaRDD
 
top(int, Comparator<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Returns the top k (largest) elements from this RDD as defined by the specified Comparator[T] and maintains the order.
top(int) - Method in interface org.apache.spark.api.java.JavaRDDLike
Returns the top k (largest) elements from this RDD using the natural ordering for T and maintains the order.
top(int, Ordering<T>) - Static method in class org.apache.spark.api.r.RRDD
 
top(int, Ordering<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
top(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
top(int, Ordering<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
top(int, Ordering<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
top(int, Ordering<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
top(int, Ordering<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
top(int, Ordering<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
top(int, Ordering<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
top(int, Ordering<T>) - Method in class org.apache.spark.rdd.RDD
Returns the top k (largest) elements from this RDD as defined by the specified implicit Ordering[T] and maintains the ordering.
top(int, Ordering<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
toPairDStreamFunctions(DStream<Tuple2<K, V>>, ClassTag<K>, ClassTag<V>, Ordering<K>) - Static method in class org.apache.spark.streaming.dstream.DStream
 
topByKey(int, Ordering<V>) - Method in class org.apache.spark.mllib.rdd.MLPairRDDFunctions
Returns the top k (largest) elements for each key from this RDD as defined by the specified implicit Ordering[T].
topDocumentsPerTopic(int) - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Return the top documents for each topic
topicAssignments() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Return the top topic for each (doc, term) pair.
topicConcentration() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
topicConcentration() - Static method in class org.apache.spark.ml.clustering.LDA
 
topicConcentration() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
topicConcentration() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
 
topicConcentration() - Method in class org.apache.spark.mllib.clustering.LDAModel
Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.
topicConcentration() - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
topicDistribution(Vector) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
Predicts the topic mixture distribution for a document (often called "theta" in the literature).
topicDistributionCol() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
topicDistributionCol() - Static method in class org.apache.spark.ml.clustering.LDA
 
topicDistributionCol() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
topicDistributions() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
For each document in the training set, return the distribution over topics for that document ("theta_doc").
topicDistributions(RDD<Tuple2<Object, Vector>>) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
Predicts the topic mixture distribution for each document (often called "theta" in the literature).
topicDistributions(JavaPairRDD<Long, Vector>) - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
Java-friendly version of topicDistributions
topics() - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
topicsMatrix() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
topicsMatrix() - Method in class org.apache.spark.ml.clustering.LDAModel
Inferred topics, where each topic is represented by a distribution over terms.
topicsMatrix() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
topicsMatrix() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
Inferred topics, where each topic is represented by a distribution over terms.
topicsMatrix() - Method in class org.apache.spark.mllib.clustering.LDAModel
Inferred topics, where each topic is represented by a distribution over terms.
topicsMatrix() - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
topK(Iterator<Tuple2<String, Object>>, int) - Static method in class org.apache.spark.streaming.util.RawTextHelper
Gets the top k words in terms of word counts.
toPMML(String) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
toPMML(SparkContext, String) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
toPMML(OutputStream) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
toPMML() - Static method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
toPMML(String) - Static method in class org.apache.spark.mllib.classification.SVMModel
 
toPMML(SparkContext, String) - Static method in class org.apache.spark.mllib.classification.SVMModel
 
toPMML(OutputStream) - Static method in class org.apache.spark.mllib.classification.SVMModel
 
toPMML() - Static method in class org.apache.spark.mllib.classification.SVMModel
 
toPMML(String) - Static method in class org.apache.spark.mllib.clustering.KMeansModel
 
toPMML(SparkContext, String) - Static method in class org.apache.spark.mllib.clustering.KMeansModel
 
toPMML(OutputStream) - Static method in class org.apache.spark.mllib.clustering.KMeansModel
 
toPMML() - Static method in class org.apache.spark.mllib.clustering.KMeansModel
 
toPMML(StreamResult) - Method in interface org.apache.spark.mllib.pmml.PMMLExportable
Export the model to the stream result in PMML format
toPMML(String) - Method in interface org.apache.spark.mllib.pmml.PMMLExportable
Export the model to a local file in PMML format
toPMML(SparkContext, String) - Method in interface org.apache.spark.mllib.pmml.PMMLExportable
Export the model to a directory on a distributed file system in PMML format
toPMML(OutputStream) - Method in interface org.apache.spark.mllib.pmml.PMMLExportable
Export the model to the OutputStream in PMML format
toPMML() - Method in interface org.apache.spark.mllib.pmml.PMMLExportable
Export the model to a String in PMML format
toPMML(String) - Static method in class org.apache.spark.mllib.regression.LassoModel
 
toPMML(SparkContext, String) - Static method in class org.apache.spark.mllib.regression.LassoModel
 
toPMML(OutputStream) - Static method in class org.apache.spark.mllib.regression.LassoModel
 
toPMML() - Static method in class org.apache.spark.mllib.regression.LassoModel
 
toPMML(String) - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
toPMML(SparkContext, String) - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
toPMML(OutputStream) - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
toPMML() - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
toPMML(String) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
toPMML(SparkContext, String) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
toPMML(OutputStream) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
toPMML() - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
topNode() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
 
topologyFile() - Method in class org.apache.spark.storage.FileBasedTopologyMapper
 
topologyInfo() - Method in class org.apache.spark.storage.BlockManagerId
 
topologyMap() - Method in class org.apache.spark.storage.FileBasedTopologyMapper
 
TopologyMapper - Class in org.apache.spark.storage
::DeveloperApi:: TopologyMapper provides topology information for a given host param: conf SparkConf to get required properties, if needed
TopologyMapper(SparkConf) - Constructor for class org.apache.spark.storage.TopologyMapper
 
toPredict() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.PredictData
 
topTopicsPerDocument(int) - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
For each document, return the top k weighted topics for that document and their weights.
toRadians(Column) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use radians. Since 2.1.0.
toRadians(String) - Static method in class org.apache.spark.sql.functions
Deprecated.
Use radians. Since 2.1.0.
toRDD(JavaDoubleRDD) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
toRDD(JavaPairRDD<K, V>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
toRDD(JavaRDD<T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
toRowMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Converts to RowMatrix, dropping row indices after grouping by row index.
toRowMatrix() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRowMatrix
Drops row indices and converts this matrix to a RowMatrix.
toScalaBigInt() - Method in class org.apache.spark.sql.types.Decimal
 
toSeq() - Method in class org.apache.spark.ml.param.ParamMap
Converts this param map to a sequence of param pairs.
toSeq() - Method in interface org.apache.spark.sql.Row
Return a Scala Seq representing the row.
toSeq() - Static method in class org.apache.spark.sql.types.StructType
 
toSet() - Static method in class org.apache.spark.sql.types.StructType
 
toShort() - Method in class org.apache.spark.sql.types.Decimal
 
toShortArray() - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
toSparkContext(JavaSparkContext) - Static method in class org.apache.spark.api.java.JavaSparkContext
 
toSparse() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toSparse() - Static method in class org.apache.spark.ml.linalg.DenseVector
 
toSparse() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a sparse matrix while maintaining the layout of the current matrix.
toSparse() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toSparse() - Static method in class org.apache.spark.ml.linalg.SparseVector
 
toSparse() - Method in interface org.apache.spark.ml.linalg.Vector
Converts this vector to a sparse vector with all explicit zeros removed.
toSparse() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
Generate a SparseMatrix from the given DenseMatrix.
toSparse() - Static method in class org.apache.spark.mllib.linalg.DenseVector
 
toSparse() - Static method in class org.apache.spark.mllib.linalg.SparseVector
 
toSparse() - Method in interface org.apache.spark.mllib.linalg.Vector
Converts this vector to a sparse vector with all explicit zeros removed.
toSparseColMajor() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toSparseColMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a sparse matrix in column major order.
toSparseColMajor() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toSparseMatrix(boolean) - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a sparse matrix.
toSparseRowMajor() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toSparseRowMajor() - Method in interface org.apache.spark.ml.linalg.Matrix
Converts this matrix to a sparse matrix in row major order.
toSparseRowMajor() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toSparseWithSize(int) - Method in interface org.apache.spark.ml.linalg.Vector
Converts this vector to a sparse vector with all explicit zeros removed when the size is known.
toSparseWithSize(int) - Method in interface org.apache.spark.mllib.linalg.Vector
Converts this vector to a sparse vector with all explicit zeros removed when the size is known.
toSplit() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.SplitData
 
toSplitInfo(Class<?>, String, InputSplit) - Static method in class org.apache.spark.scheduler.SplitInfo
 
toSplitInfo(Class<?>, String, InputSplit) - Static method in class org.apache.spark.scheduler.SplitInfo
 
toStream() - Static method in class org.apache.spark.sql.types.StructType
 
toString() - Method in class org.apache.spark.Accumulable
Deprecated.
 
toString() - Static method in class org.apache.spark.Accumulator
Deprecated.
 
toString() - Method in class org.apache.spark.api.java.JavaRDD
 
toString() - Method in class org.apache.spark.api.java.Optional
 
toString() - Static method in class org.apache.spark.api.r.RRDD
 
toString() - Method in class org.apache.spark.broadcast.Broadcast
 
toString() - Method in class org.apache.spark.graphx.EdgeDirection
 
toString() - Static method in class org.apache.spark.graphx.EdgeRDD
 
toString() - Method in class org.apache.spark.graphx.EdgeTriplet
 
toString() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
toString() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
toString() - Static method in class org.apache.spark.graphx.VertexRDD
 
toString() - Method in class org.apache.spark.ml.attribute.Attribute
 
toString() - Method in class org.apache.spark.ml.attribute.AttributeGroup
 
toString() - Static method in class org.apache.spark.ml.attribute.BinaryAttribute
 
toString() - Static method in class org.apache.spark.ml.attribute.NominalAttribute
 
toString() - Static method in class org.apache.spark.ml.attribute.NumericAttribute
 
toString() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
toString() - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
toString() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
toString() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
toString() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
toString() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
toString() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
toString() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
toString() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
toString() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
toString() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
toString() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
toString() - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
toString() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
toString() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
toString() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
toString() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
toString() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
toString() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
toString() - Static method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
toString() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
toString() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
toString() - Static method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
toString() - Static method in class org.apache.spark.ml.clustering.KMeans
 
toString() - Static method in class org.apache.spark.ml.clustering.KMeansModel
 
toString() - Static method in class org.apache.spark.ml.clustering.LDA
 
toString() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
toString() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
toString() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
toString() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
toString() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
toString() - Static method in class org.apache.spark.ml.feature.Binarizer
 
toString() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
toString() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
toString() - Static method in class org.apache.spark.ml.feature.Bucketizer
 
toString() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
toString() - Static method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
toString() - Static method in class org.apache.spark.ml.feature.ColumnPruner
 
toString() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
toString() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
toString() - Static method in class org.apache.spark.ml.feature.DCT
 
toString() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
toString() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
toString() - Static method in class org.apache.spark.ml.feature.HashingTF
 
toString() - Static method in class org.apache.spark.ml.feature.IDF
 
toString() - Static method in class org.apache.spark.ml.feature.IDFModel
 
toString() - Static method in class org.apache.spark.ml.feature.Imputer
 
toString() - Static method in class org.apache.spark.ml.feature.ImputerModel
 
toString() - Static method in class org.apache.spark.ml.feature.IndexToString
 
toString() - Static method in class org.apache.spark.ml.feature.Interaction
 
toString() - Method in class org.apache.spark.ml.feature.LabeledPoint
 
toString() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
toString() - Static method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
toString() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
toString() - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
toString() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
toString() - Static method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
toString() - Static method in class org.apache.spark.ml.feature.NGram
 
toString() - Static method in class org.apache.spark.ml.feature.Normalizer
 
toString() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
toString() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
toString() - Static method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
toString() - Static method in class org.apache.spark.ml.feature.PCA
 
toString() - Static method in class org.apache.spark.ml.feature.PCAModel
 
toString() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
toString() - Static method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
toString() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
toString() - Method in class org.apache.spark.ml.feature.RFormula
 
toString() - Method in class org.apache.spark.ml.feature.RFormulaModel
 
toString() - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
toString() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
toString() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
toString() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
toString() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
toString() - Static method in class org.apache.spark.ml.feature.StringIndexerModel
 
toString() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
toString() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
toString() - Static method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
toString() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
toString() - Static method in class org.apache.spark.ml.feature.VectorIndexerModel
 
toString() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
toString() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
toString() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
toString() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
toString() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
toString() - Static method in class org.apache.spark.ml.fpm.FPGrowthModel
 
toString() - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toString(int, int) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
 
toString() - Method in class org.apache.spark.ml.linalg.DenseVector
 
toString() - Method in interface org.apache.spark.ml.linalg.Matrix
A human readable representation of the matrix
toString(int, int) - Method in interface org.apache.spark.ml.linalg.Matrix
A human readable representation of the matrix with maximum lines and width
toString() - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toString(int, int) - Static method in class org.apache.spark.ml.linalg.SparseMatrix
 
toString() - Method in class org.apache.spark.ml.linalg.SparseVector
 
toString() - Static method in class org.apache.spark.ml.param.DoubleParam
 
toString() - Static method in class org.apache.spark.ml.param.FloatParam
 
toString() - Method in class org.apache.spark.ml.param.Param
 
toString() - Method in class org.apache.spark.ml.param.ParamMap
 
toString() - Static method in class org.apache.spark.ml.Pipeline
 
toString() - Static method in class org.apache.spark.ml.PipelineModel
 
toString() - Static method in class org.apache.spark.ml.recommendation.ALS
 
toString() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
toString() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
toString() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
toString() - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
toString() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
toString() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
 
toString() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
toString() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
toString() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
toString() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionTrainingSummary
 
toString() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
toString() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
toString() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
toString() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
toString() - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
toString() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
toString() - Method in class org.apache.spark.ml.tree.InternalNode
 
toString() - Method in class org.apache.spark.ml.tree.LeafNode
 
toString() - Static method in class org.apache.spark.ml.tuning.CrossValidator
 
toString() - Static method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
toString() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
toString() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
toString() - Method in interface org.apache.spark.ml.util.Identifiable
 
toString() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
toString() - Method in class org.apache.spark.mllib.classification.SVMModel
 
toString() - Method in class org.apache.spark.mllib.fpm.AssociationRules.Rule
 
toString() - Method in class org.apache.spark.mllib.fpm.FPGrowth.FreqItemset
 
toString() - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
 
toString(int, int) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
 
toString() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
toString() - Method in interface org.apache.spark.mllib.linalg.Matrix
A human readable representation of the matrix
toString(int, int) - Method in interface org.apache.spark.mllib.linalg.Matrix
A human readable representation of the matrix with maximum lines and width
toString() - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
 
toString(int, int) - Static method in class org.apache.spark.mllib.linalg.SparseMatrix
 
toString() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
toString() - Method in class org.apache.spark.mllib.regression.GeneralizedLinearModel
Print a summary of the model.
toString() - Method in class org.apache.spark.mllib.regression.LabeledPoint
 
toString() - Static method in class org.apache.spark.mllib.regression.LassoModel
 
toString() - Static method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
toString() - Static method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
toString() - Method in class org.apache.spark.mllib.stat.test.BinarySample
 
toString() - Method in class org.apache.spark.mllib.stat.test.ChiSqTestResult
 
toString() - Method in class org.apache.spark.mllib.stat.test.KolmogorovSmirnovTestResult
 
toString() - Method in interface org.apache.spark.mllib.stat.test.TestResult
String explaining the hypothesis test result.
toString() - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
toString() - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
toString() - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
toString() - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
toString() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel
Print a summary of the model.
toString() - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
toString() - Method in class org.apache.spark.mllib.tree.model.InformationGainStats
 
toString() - Method in class org.apache.spark.mllib.tree.model.Node
 
toString() - Method in class org.apache.spark.mllib.tree.model.Predict
 
toString() - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
toString() - Method in class org.apache.spark.mllib.tree.model.Split
 
toString() - Method in class org.apache.spark.partial.BoundedDouble
 
toString() - Method in class org.apache.spark.partial.PartialResult
 
toString() - Static method in class org.apache.spark.rdd.CheckpointState
 
toString() - Static method in class org.apache.spark.rdd.DeterministicLevel
 
toString() - Static method in class org.apache.spark.rdd.HadoopRDD
 
toString() - Static method in class org.apache.spark.rdd.JdbcRDD
 
toString() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
toString() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
toString() - Method in class org.apache.spark.rdd.RDD
 
toString() - Static method in class org.apache.spark.rdd.UnionRDD
 
toString() - Static method in class org.apache.spark.scheduler.ExecutorKilled
 
toString() - Method in class org.apache.spark.scheduler.InputFormatInfo
 
toString() - Static method in class org.apache.spark.scheduler.LossReasonPending
 
toString() - Static method in class org.apache.spark.scheduler.SchedulingMode
 
toString() - Method in class org.apache.spark.scheduler.SplitInfo
 
toString() - Static method in class org.apache.spark.scheduler.TaskLocality
 
toString() - Method in class org.apache.spark.SerializableWritable
 
toString() - Static method in exception org.apache.spark.sql.AnalysisException
 
toString() - Method in class org.apache.spark.sql.catalog.Column
 
toString() - Method in class org.apache.spark.sql.catalog.Database
 
toString() - Method in class org.apache.spark.sql.catalog.Function
 
toString() - Method in class org.apache.spark.sql.catalog.Table
 
toString() - Method in class org.apache.spark.sql.Column
 
toString() - Method in class org.apache.spark.sql.Dataset
 
toString() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
toString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
toString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
toString() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
toString() - Method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
toString() - Method in class org.apache.spark.sql.KeyValueGroupedDataset
 
toString() - Method in class org.apache.spark.sql.RelationalGroupedDataset
 
toString() - Method in interface org.apache.spark.sql.Row
 
toString() - Method in class org.apache.spark.sql.sources.In
 
toString() - Method in class org.apache.spark.sql.sources.v2.reader.streaming.Offset
 
toString() - Method in class org.apache.spark.sql.streaming.SinkProgress
 
toString() - Method in class org.apache.spark.sql.streaming.SourceProgress
 
toString() - Method in class org.apache.spark.sql.streaming.StateOperatorProgress
 
toString() - Method in exception org.apache.spark.sql.streaming.StreamingQueryException
 
toString() - Method in class org.apache.spark.sql.streaming.StreamingQueryProgress
 
toString() - Method in class org.apache.spark.sql.streaming.StreamingQueryStatus
 
toString() - Method in class org.apache.spark.sql.types.Decimal
 
toString() - Method in class org.apache.spark.sql.types.DecimalType
 
toString() - Method in class org.apache.spark.sql.types.Metadata
 
toString() - Method in class org.apache.spark.sql.types.StructField
 
toString() - Static method in class org.apache.spark.sql.types.StructType
 
toString() - Method in class org.apache.spark.storage.BlockId
 
toString() - Method in class org.apache.spark.storage.BlockManagerId
 
toString() - Static method in class org.apache.spark.storage.BroadcastBlockId
 
toString() - Static method in class org.apache.spark.storage.RDDBlockId
 
toString() - Method in class org.apache.spark.storage.RDDInfo
 
toString() - Static method in class org.apache.spark.storage.ShuffleBlockId
 
toString() - Static method in class org.apache.spark.storage.ShuffleDataBlockId
 
toString() - Static method in class org.apache.spark.storage.ShuffleIndexBlockId
 
toString() - Method in class org.apache.spark.storage.StorageLevel
 
toString() - Static method in class org.apache.spark.storage.StreamBlockId
 
toString() - Static method in class org.apache.spark.storage.TaskResultBlockId
 
toString() - Method in class org.apache.spark.streaming.Duration
 
toString() - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
toString() - Method in class org.apache.spark.streaming.State
 
toString() - Method in class org.apache.spark.streaming.Time
 
toString() - Static method in class org.apache.spark.TaskState
 
toString() - Method in class org.apache.spark.util.AccumulatorV2
 
toString() - Method in class org.apache.spark.util.MutablePair
 
toString() - Method in class org.apache.spark.util.StatCounter
 
toStructField(Metadata) - Method in class org.apache.spark.ml.attribute.Attribute
Converts to a StructField with some existing metadata.
toStructField() - Method in class org.apache.spark.ml.attribute.Attribute
Converts to a StructField.
toStructField(Metadata) - Method in class org.apache.spark.ml.attribute.AttributeGroup
Converts to a StructField with some existing metadata.
toStructField() - Method in class org.apache.spark.ml.attribute.AttributeGroup
Converts to a StructField.
toStructField(Metadata) - Static method in class org.apache.spark.ml.attribute.BinaryAttribute
 
toStructField() - Static method in class org.apache.spark.ml.attribute.BinaryAttribute
 
toStructField(Metadata) - Static method in class org.apache.spark.ml.attribute.NominalAttribute
 
toStructField() - Static method in class org.apache.spark.ml.attribute.NominalAttribute
 
toStructField(Metadata) - Static method in class org.apache.spark.ml.attribute.NumericAttribute
 
toStructField() - Static method in class org.apache.spark.ml.attribute.NumericAttribute
 
toStructField(Metadata) - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
toStructField() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
totalBlocksFetched() - Method in class org.apache.spark.status.api.v1.ShuffleReadMetricDistributions
 
totalBytesRead(ShuffleReadMetrics) - Static method in class org.apache.spark.ui.jobs.ApiHelper
 
totalCores() - Method in class org.apache.spark.scheduler.cluster.ExecutorInfo
 
totalCores() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
totalCores() - Method in class org.apache.spark.status.LiveExecutor
 
totalCount() - Method in class org.apache.spark.util.sketch.CountMinSketch
Total count of items added to this CountMinSketch so far.
totalDelay() - Method in class org.apache.spark.status.api.v1.streaming.BatchInfo
 
totalDelay() - Method in class org.apache.spark.streaming.scheduler.BatchInfo
Time taken for all the jobs of this batch to finish processing from the time they were submitted.
totalDiskSize() - Method in class org.apache.spark.ui.storage.ExecutorStreamSummary
 
totalDuration() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
totalDuration() - Method in class org.apache.spark.status.LiveExecutor
 
totalGCTime() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
totalGcTime() - Method in class org.apache.spark.status.LiveExecutor
 
totalInputBytes() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
totalInputBytes() - Method in class org.apache.spark.status.LiveExecutor
 
totalIterations() - Method in interface org.apache.spark.ml.classification.LogisticRegressionTrainingSummary
Number of training iterations.
totalIterations() - Method in class org.apache.spark.ml.regression.LinearRegressionTrainingSummary
Number of training iterations until termination
totalMemSize() - Method in class org.apache.spark.ui.storage.ExecutorStreamSummary
 
totalNumNodes() - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
totalNumNodes() - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
totalNumNodes() - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
totalNumNodes() - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
totalNumNodes() - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
totalNumNodes() - Static method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
totalOffHeap() - Method in class org.apache.spark.status.LiveExecutor
 
totalOffHeapStorageMemory() - Method in class org.apache.spark.status.api.v1.MemoryMetrics
 
totalOnHeap() - Method in class org.apache.spark.status.LiveExecutor
 
totalOnHeapStorageMemory() - Method in class org.apache.spark.status.api.v1.MemoryMetrics
 
totalShuffleRead() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
totalShuffleRead() - Method in class org.apache.spark.status.LiveExecutor
 
totalShuffleWrite() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
totalShuffleWrite() - Method in class org.apache.spark.status.LiveExecutor
 
totalTasks() - Method in class org.apache.spark.status.api.v1.ExecutorSummary
 
totalTasks() - Method in class org.apache.spark.status.LiveExecutor
 
toTraversable() - Static method in class org.apache.spark.sql.types.StructType
 
toTuple() - Method in class org.apache.spark.graphx.EdgeTriplet
 
toUnscaledLong() - Method in class org.apache.spark.sql.types.Decimal
 
toVector() - Static method in class org.apache.spark.sql.types.StructType
 
toVirtualHosts(Seq<String>) - Static method in class org.apache.spark.ui.JettyUtils
 
train(RDD<ALS.Rating<ID>>, int, int, int, int, double, boolean, double, boolean, StorageLevel, StorageLevel, int, long, ClassTag<ID>, Ordering<ID>) - Static method in class org.apache.spark.ml.recommendation.ALS
Developer API Implementation of the ALS algorithm.
train(RDD<LabeledPoint>, int, double, double, Vector) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
Train a logistic regression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
Train a logistic regression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
Train a logistic regression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int) - Static method in class org.apache.spark.mllib.classification.LogisticRegressionWithSGD
Train a logistic regression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>) - Static method in class org.apache.spark.mllib.classification.NaiveBayes
Trains a Naive Bayes model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, double) - Static method in class org.apache.spark.mllib.classification.NaiveBayes
Trains a Naive Bayes model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, double, String) - Static method in class org.apache.spark.mllib.classification.NaiveBayes
Trains a Naive Bayes model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double, double, Vector) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
Train a SVM model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double, double) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
Train a SVM model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
Train a SVM model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int) - Static method in class org.apache.spark.mllib.classification.SVMWithSGD
Train a SVM model given an RDD of (label, features) pairs.
train(RDD<Vector>, int, int, String, long) - Static method in class org.apache.spark.mllib.clustering.KMeans
Trains a k-means model using the given set of parameters.
train(RDD<Vector>, int, int, String) - Static method in class org.apache.spark.mllib.clustering.KMeans
Trains a k-means model using the given set of parameters.
train(RDD<Vector>, int, int, int, String, long) - Static method in class org.apache.spark.mllib.clustering.KMeans
Deprecated.
Use train method without 'runs'. Since 2.1.0.
train(RDD<Vector>, int, int, int, String) - Static method in class org.apache.spark.mllib.clustering.KMeans
Deprecated.
Use train method without 'runs'. Since 2.1.0.
train(RDD<Vector>, int, int) - Static method in class org.apache.spark.mllib.clustering.KMeans
Trains a k-means model using specified parameters and the default values for unspecified.
train(RDD<Vector>, int, int, int) - Static method in class org.apache.spark.mllib.clustering.KMeans
Deprecated.
Use train method without 'runs'. Since 2.1.0.
train(RDD<Rating>, int, int, double, int, long) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of ratings by users for a subset of products.
train(RDD<Rating>, int, int, double, int) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of ratings by users for a subset of products.
train(RDD<Rating>, int, int, double) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of ratings by users for a subset of products.
train(RDD<Rating>, int, int) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of ratings by users for a subset of products.
train(RDD<LabeledPoint>, int, double, double, double, Vector) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
Train a Lasso model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double, double) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
Train a Lasso model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
Train a Lasso model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int) - Static method in class org.apache.spark.mllib.regression.LassoWithSGD
Train a Lasso model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double, Vector) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
Train a Linear Regression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
Train a LinearRegression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
Train a LinearRegression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int) - Static method in class org.apache.spark.mllib.regression.LinearRegressionWithSGD
Train a LinearRegression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double, double, Vector) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
Train a RidgeRegression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double, double) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
Train a RidgeRegression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int, double, double) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
Train a RidgeRegression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, int) - Static method in class org.apache.spark.mllib.regression.RidgeRegressionWithSGD
Train a RidgeRegression model given an RDD of (label, features) pairs.
train(RDD<LabeledPoint>, Strategy) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Method to train a decision tree model.
train(RDD<LabeledPoint>, Enumeration.Value, Impurity, int) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Method to train a decision tree model.
train(RDD<LabeledPoint>, Enumeration.Value, Impurity, int, int) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Method to train a decision tree model.
train(RDD<LabeledPoint>, Enumeration.Value, Impurity, int, int, int, Enumeration.Value, Map<Object, Object>) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Method to train a decision tree model.
train(RDD<LabeledPoint>, BoostingStrategy) - Static method in class org.apache.spark.mllib.tree.GradientBoostedTrees
Method to train a gradient boosting model.
train(JavaRDD<LabeledPoint>, BoostingStrategy) - Static method in class org.apache.spark.mllib.tree.GradientBoostedTrees
Java-friendly API for org.apache.spark.mllib.tree.GradientBoostedTrees.train
trainClassifier(RDD<LabeledPoint>, int, Map<Object, Object>, String, int, int) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Method to train a decision tree model for binary or multiclass classification.
trainClassifier(JavaRDD<LabeledPoint>, int, Map<Integer, Integer>, String, int, int) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Java-friendly API for org.apache.spark.mllib.tree.DecisionTree.trainClassifier
trainClassifier(RDD<LabeledPoint>, Strategy, int, String, int) - Static method in class org.apache.spark.mllib.tree.RandomForest
Method to train a decision tree model for binary or multiclass classification.
trainClassifier(RDD<LabeledPoint>, int, Map<Object, Object>, int, String, String, int, int, int) - Static method in class org.apache.spark.mllib.tree.RandomForest
Method to train a decision tree model for binary or multiclass classification.
trainClassifier(JavaRDD<LabeledPoint>, int, Map<Integer, Integer>, int, String, String, int, int, int) - Static method in class org.apache.spark.mllib.tree.RandomForest
Java-friendly API for org.apache.spark.mllib.tree.RandomForest.trainClassifier
trainImplicit(RDD<Rating>, int, int, double, int, double, long) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of 'implicit preferences' given by users to some products, in the form of (userID, productID, preference) pairs.
trainImplicit(RDD<Rating>, int, int, double, int, double) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of 'implicit preferences' of users for a subset of products.
trainImplicit(RDD<Rating>, int, int, double, double) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of 'implicit preferences' of users for a subset of products.
trainImplicit(RDD<Rating>, int, int) - Static method in class org.apache.spark.mllib.recommendation.ALS
Train a matrix factorization model given an RDD of 'implicit preferences' of users for a subset of products.
trainingLogLikelihood() - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
Log likelihood of the observed tokens in the training set, given the current parameter estimates: log P(docs | topics, topic distributions for docs, Dirichlet hyperparameters)
trainOn(DStream<Vector>) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Update the clustering model by training on batches of data from a DStream.
trainOn(JavaDStream<Vector>) - Method in class org.apache.spark.mllib.clustering.StreamingKMeans
Java-friendly version of trainOn.
trainOn(DStream<LabeledPoint>) - Method in class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
Update the model by training on batches of data from a DStream.
trainOn(JavaDStream<LabeledPoint>) - Method in class org.apache.spark.mllib.regression.StreamingLinearAlgorithm
Java-friendly version of trainOn.
trainRatio() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
trainRatio() - Static method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
trainRegressor(RDD<LabeledPoint>, Map<Object, Object>, String, int, int) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Method to train a decision tree model for regression.
trainRegressor(JavaRDD<LabeledPoint>, Map<Integer, Integer>, String, int, int) - Static method in class org.apache.spark.mllib.tree.DecisionTree
Java-friendly API for org.apache.spark.mllib.tree.DecisionTree.trainRegressor
trainRegressor(RDD<LabeledPoint>, Strategy, int, String, int) - Static method in class org.apache.spark.mllib.tree.RandomForest
Method to train a decision tree model for regression.
trainRegressor(RDD<LabeledPoint>, Map<Object, Object>, int, String, String, int, int, int) - Static method in class org.apache.spark.mllib.tree.RandomForest
Method to train a decision tree model for regression.
trainRegressor(JavaRDD<LabeledPoint>, Map<Integer, Integer>, int, String, String, int, int, int) - Static method in class org.apache.spark.mllib.tree.RandomForest
Java-friendly API for org.apache.spark.mllib.tree.RandomForest.trainRegressor
TrainValidationSplit - Class in org.apache.spark.ml.tuning
Validation for hyper-parameter tuning.
TrainValidationSplit(String) - Constructor for class org.apache.spark.ml.tuning.TrainValidationSplit
 
TrainValidationSplit() - Constructor for class org.apache.spark.ml.tuning.TrainValidationSplit
 
TrainValidationSplitModel - Class in org.apache.spark.ml.tuning
Model from train validation split.
TrainValidationSplitModel.TrainValidationSplitModelWriter - Class in org.apache.spark.ml.tuning
Writer for TrainValidationSplitModel.
transferred() - Method in class org.apache.spark.storage.ReadableChannelFileRegion
 
transferTo(WritableByteChannel, long) - Method in class org.apache.spark.storage.ReadableChannelFileRegion
 
transform(Function1<Try<T>, Try<S>>, ExecutionContext) - Method in class org.apache.spark.ComplexFutureAction
 
transform(Function1<Try<T>, Try<S>>, ExecutionContext) - Method in interface org.apache.spark.FutureAction
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.classification.ClassificationModel
Transforms dataset by reading from featuresCol, and appending new columns as specified by parameters: - predicted labels as predictionCol of type Double - raw predictions (confidences) as rawPredictionCol of type Vector.
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
Transforms dataset by reading from featuresCol, and appending new columns as specified by parameters: - predicted labels as predictionCol of type Double - raw predictions (confidences) as rawPredictionCol of type Vector - probability of each class as probabilityCol of type Vector.
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.clustering.KMeansModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.clustering.LDAModel
Transforms the input dataset.
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.Binarizer
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.Bucketizer
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.ColumnPruner
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.DCT
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.DCT
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.DCT
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.DCT
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.HashingTF
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.IDFModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.ImputerModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.IndexToString
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.Interaction
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.NGram
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.NGram
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.NGram
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.NGram
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.Normalizer
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.Normalizer
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.Normalizer
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.PCAModel
Transform a vector by computed Principal Components.
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.RFormulaModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.SQLTransformer
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.VectorAssembler
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.VectorSizeHint
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.VectorSlicer
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.feature.Word2VecModel
Transform a sentence column to a vector column to represent the whole sentence.
transform(Dataset<?>) - Method in class org.apache.spark.ml.fpm.FPGrowthModel
The transform method first generates the association rules according to the frequent itemsets.
transform(Dataset<?>) - Method in class org.apache.spark.ml.PipelineModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.PredictionModel
Transforms dataset by reading from featuresCol, calling predict, and storing the predictions as a new column predictionCol.
transform(Dataset<?>) - Method in class org.apache.spark.ml.recommendation.ALSModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
transform(Dataset<?>, ParamMap) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
transform(Dataset<?>) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
transform(Dataset<?>, ParamPair<?>, ParamPair<?>...) - Method in class org.apache.spark.ml.Transformer
Transforms the dataset with optional parameters
transform(Dataset<?>, ParamPair<?>, Seq<ParamPair<?>>) - Method in class org.apache.spark.ml.Transformer
Transforms the dataset with optional parameters
transform(Dataset<?>, ParamMap) - Method in class org.apache.spark.ml.Transformer
Transforms the dataset with provided parameter map as additional parameters.
transform(Dataset<?>) - Method in class org.apache.spark.ml.Transformer
Transforms the input dataset.
transform(Dataset<?>) - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
transform(Dataset<?>) - Method in class org.apache.spark.ml.UnaryTransformer
 
transform(Vector) - Method in class org.apache.spark.mllib.feature.ChiSqSelectorModel
Applies transformation on a vector.
transform(Vector) - Method in class org.apache.spark.mllib.feature.ElementwiseProduct
Does the hadamard product transformation.
transform(Iterable<?>) - Method in class org.apache.spark.mllib.feature.HashingTF
Transforms the input document into a sparse term frequency vector.
transform(Iterable<?>) - Method in class org.apache.spark.mllib.feature.HashingTF
Transforms the input document into a sparse term frequency vector (Java version).
transform(RDD<D>) - Method in class org.apache.spark.mllib.feature.HashingTF
Transforms the input document to term frequency vectors.
transform(JavaRDD<D>) - Method in class org.apache.spark.mllib.feature.HashingTF
Transforms the input document to term frequency vectors (Java version).
transform(RDD<Vector>) - Method in class org.apache.spark.mllib.feature.IDFModel
Transforms term frequency (TF) vectors to TF-IDF vectors.
transform(Vector) - Method in class org.apache.spark.mllib.feature.IDFModel
Transforms a term frequency (TF) vector to a TF-IDF vector
transform(JavaRDD<Vector>) - Method in class org.apache.spark.mllib.feature.IDFModel
Transforms term frequency (TF) vectors to TF-IDF vectors (Java version).
transform(Vector) - Method in class org.apache.spark.mllib.feature.Normalizer
Applies unit length normalization on a vector.
transform(Vector) - Method in class org.apache.spark.mllib.feature.PCAModel
Transform a vector by computed Principal Components.
transform(Vector) - Method in class org.apache.spark.mllib.feature.StandardScalerModel
Applies standardization transformation on a vector.
transform(Vector) - Method in interface org.apache.spark.mllib.feature.VectorTransformer
Applies transformation on a vector.
transform(RDD<Vector>) - Method in interface org.apache.spark.mllib.feature.VectorTransformer
Applies transformation on an RDD[Vector].
transform(JavaRDD<Vector>) - Method in interface org.apache.spark.mllib.feature.VectorTransformer
Applies transformation on a JavaRDD[Vector].
transform(String) - Method in class org.apache.spark.mllib.feature.Word2VecModel
Transforms a word to its vector representation
transform(Function1<Try<T>, Try<S>>, ExecutionContext) - Method in class org.apache.spark.SimpleFutureAction
 
transform(Function1<Dataset<T>, Dataset<U>>) - Method in class org.apache.spark.sql.Dataset
Concise syntax for chaining custom transformations.
transform(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
transform(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
transform(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
transform(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
transform(Function<R, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transform(Function2<R, Time, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transform(Function<R, JavaRDD<U>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream.
transform(Function2<R, Time, JavaRDD<U>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream.
transform(Function<R, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transform(Function2<R, Time, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transform(Function<R, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transform(Function2<R, Time, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transform(Function<R, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transform(Function2<R, Time, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transform(Function<R, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transform(Function2<R, Time, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transform(Function<R, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
transform(Function2<R, Time, JavaRDD<U>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
transform(List<JavaDStream<?>>, Function2<List<JavaRDD<?>>, Time, JavaRDD<T>>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams.
transform(Function1<RDD<T>, RDD<U>>, ClassTag<U>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream.
transform(Function2<RDD<T>, Time, RDD<U>>, ClassTag<U>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream.
transform(Seq<DStream<?>>, Function2<Seq<RDD<?>>, Time, RDD<T>>, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams.
transformAllExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
transformAllExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
transformAllExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
transformAllExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
transformDown(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
transformDown(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
transformDown(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
transformDown(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
Transformer - Class in org.apache.spark.ml
Developer API Abstract class for transformers that transform one dataset into another.
Transformer() - Constructor for class org.apache.spark.ml.Transformer
 
transformExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
transformExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
transformExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
transformExpressions(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
transformExpressionsDown(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
transformExpressionsDown(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
transformExpressionsDown(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
transformExpressionsDown(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
transformExpressionsUp(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
transformExpressionsUp(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
transformExpressionsUp(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
transformExpressionsUp(PartialFunction<Expression, Expression>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
transformOutputColumnSchema(StructField, String, boolean, boolean) - Static method in class org.apache.spark.ml.feature.OneHotEncoderCommon
Prepares the StructField with proper metadata for OneHotEncoder's output column.
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.GBTClassificationModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.LinearSVC
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.classification.OneVsRest
 
transformSchema(StructType) - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.KMeans
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.KMeansModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.LDA
 
transformSchema(StructType) - Method in class org.apache.spark.ml.clustering.LDAModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.Binarizer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.Bucketizer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.ColumnPruner
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.CountVectorizer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.DCT
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.FeatureHasher
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.HashingTF
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.IDF
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.IDFModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.Imputer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.ImputerModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.IndexToString
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.Interaction
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.MaxAbsScaler
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.MinHashLSH
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.MinHashLSHModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.NGram
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.Normalizer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.PCA
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.PCAModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.RFormula
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.RFormulaModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.SQLTransformer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.StandardScaler
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.StringIndexer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.feature.Tokenizer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.VectorAssembler
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.VectorIndexer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.VectorSizeHint
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.VectorSlicer
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.Word2Vec
 
transformSchema(StructType) - Method in class org.apache.spark.ml.feature.Word2VecModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.fpm.FPGrowth
 
transformSchema(StructType) - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.Pipeline
 
transformSchema(StructType) - Method in class org.apache.spark.ml.PipelineModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.PipelineStage
Developer API
transformSchema(StructType) - Method in class org.apache.spark.ml.PredictionModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.Predictor
 
transformSchema(StructType) - Method in class org.apache.spark.ml.recommendation.ALS
 
transformSchema(StructType) - Method in class org.apache.spark.ml.recommendation.ALSModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
transformSchema(StructType) - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.GBTRegressionModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
transformSchema(StructType) - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.LinearRegression
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
transformSchema(StructType) - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
transformSchema(StructType) - Method in class org.apache.spark.ml.tuning.CrossValidator
 
transformSchema(StructType) - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
transformSchema(StructType) - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
transformSchema(StructType) - Method in class org.apache.spark.ml.UnaryTransformer
 
transformToPair(Function<R, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transformToPair(Function2<R, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transformToPair(Function<R, JavaPairRDD<K2, V2>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream.
transformToPair(Function2<R, Time, JavaPairRDD<K2, V2>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream.
transformToPair(Function<R, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transformToPair(Function2<R, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transformToPair(Function<R, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transformToPair(Function2<R, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transformToPair(Function<R, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transformToPair(Function2<R, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transformToPair(Function<R, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transformToPair(Function2<R, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transformToPair(Function<R, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
transformToPair(Function2<R, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
transformToPair(List<JavaDStream<?>>, Function2<List<JavaRDD<?>>, Time, JavaPairRDD<K, V>>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a new DStream in which each RDD is generated by applying a function on RDDs of the DStreams.
transformUp(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
transformUp(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
transformUp(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
transformUp(PartialFunction<BaseType, BaseType>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
transformWith(Function1<Try<T>, Future<S>>, ExecutionContext) - Method in class org.apache.spark.ComplexFutureAction
 
transformWith(Function1<Try<T>, Future<S>>, ExecutionContext) - Method in interface org.apache.spark.FutureAction
 
transformWith(Function1<Try<T>, Future<S>>, ExecutionContext) - Method in class org.apache.spark.SimpleFutureAction
 
transformWith(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transformWith(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transformWith(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaRDD<W>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.
transformWith(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaRDD<W>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.
transformWith(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transformWith(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transformWith(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transformWith(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transformWith(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transformWith(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transformWith(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transformWith(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transformWith(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
transformWith(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaRDD<W>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
transformWith(DStream<U>, Function2<RDD<T>, RDD<U>, RDD<V>>, ClassTag<U>, ClassTag<V>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.
transformWith(DStream<U>, Function3<RDD<T>, RDD<U>, Time, RDD<V>>, ClassTag<U>, ClassTag<V>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.
transformWithToPair(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transformWithToPair(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaPairRDD<K3, V3>>) - Static method in class org.apache.spark.streaming.api.java.JavaDStream
 
transformWithToPair(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaPairRDD<K2, V2>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.
transformWithToPair(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaPairRDD<K3, V3>>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
Return a new DStream in which each RDD is generated by applying a function on each RDD of 'this' DStream and 'other' DStream.
transformWithToPair(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transformWithToPair(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaPairRDD<K3, V3>>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
transformWithToPair(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transformWithToPair(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaPairRDD<K3, V3>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
transformWithToPair(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transformWithToPair(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaPairRDD<K3, V3>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
transformWithToPair(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transformWithToPair(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaPairRDD<K3, V3>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
transformWithToPair(JavaDStream<U>, Function3<R, JavaRDD<U>, Time, JavaPairRDD<K2, V2>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
transformWithToPair(JavaPairDStream<K2, V2>, Function3<R, JavaPairRDD<K2, V2>, Time, JavaPairRDD<K3, V3>>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
translate(Column, String, String) - Static method in class org.apache.spark.sql.functions
Translate any character in the src by a character in replaceString.
transpose() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
transpose() - Method in interface org.apache.spark.ml.linalg.Matrix
Transpose the Matrix.
transpose() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
transpose() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
transpose() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Transpose this BlockMatrix.
transpose() - Method in class org.apache.spark.mllib.linalg.distributed.CoordinateMatrix
Transposes this CoordinateMatrix.
transpose() - Method in interface org.apache.spark.mllib.linalg.Matrix
Transpose the Matrix.
transpose() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
transpose(Function1<A, GenTraversableOnce<B>>) - Static method in class org.apache.spark.sql.types.StructType
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int) - Static method in class org.apache.spark.api.java.JavaRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Static method in class org.apache.spark.api.java.JavaRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int) - Method in interface org.apache.spark.api.java.JavaRDDLike
Aggregates the elements of this RDD in a multi-level tree pattern.
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>) - Method in interface org.apache.spark.api.java.JavaRDDLike
org.apache.spark.api.java.JavaRDDLike.treeAggregate with suggested depth 2.
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Aggregates the elements of this RDD in a multi-level tree pattern.
treeAggregate(U, Function2<U, T, U>, Function2<U, U, U>, int, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.api.r.RRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.graphx.EdgeRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.graphx.VertexRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.rdd.HadoopRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.rdd.JdbcRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
treeAggregate$default$4(U) - Static method in class org.apache.spark.rdd.UnionRDD
 
treeID() - Method in class org.apache.spark.ml.tree.EnsembleModelReadWrite.EnsembleNodeData
 
treeId() - Method in class org.apache.spark.mllib.tree.model.DecisionTreeModel.SaveLoadV1_0$.NodeData
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
treeReduce(Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
treeReduce(Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.api.java.JavaRDD
 
treeReduce(Function2<T, T, T>) - Static method in class org.apache.spark.api.java.JavaRDD
 
treeReduce(Function2<T, T, T>, int) - Method in interface org.apache.spark.api.java.JavaRDDLike
Reduces the elements of this RDD in a multi-level tree pattern.
treeReduce(Function2<T, T, T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
org.apache.spark.api.java.JavaRDDLike.treeReduce with suggested depth 2.
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.api.r.RRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.graphx.EdgeRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.graphx.VertexRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.rdd.HadoopRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.rdd.JdbcRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
treeReduce(Function2<T, T, T>, int) - Method in class org.apache.spark.rdd.RDD
Reduces the elements of this RDD in a multi-level tree pattern.
treeReduce(Function2<T, T, T>, int) - Static method in class org.apache.spark.rdd.UnionRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.api.r.RRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.graphx.EdgeRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
treeReduce$default$2() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
treeReduce$default$2() - Static method in class org.apache.spark.graphx.VertexRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.rdd.HadoopRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.rdd.JdbcRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
treeReduce$default$2() - Static method in class org.apache.spark.rdd.UnionRDD
 
trees() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
trees() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
trees() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
 
trees() - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
trees() - Method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
trees() - Method in class org.apache.spark.mllib.tree.model.RandomForestModel
 
treeStrategy() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
treeString() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
treeString(boolean, boolean) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
treeString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
treeString(boolean, boolean) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
treeString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
treeString(boolean, boolean) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
treeString() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
treeString(boolean, boolean) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
treeString() - Method in class org.apache.spark.sql.types.StructType
 
treeString$default$2() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
treeString$default$2() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
treeString$default$2() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
treeString$default$2() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
treeWeights() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
treeWeights() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
treeWeights() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
 
treeWeights() - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
treeWeights() - Method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
 
triangleCount() - Method in class org.apache.spark.graphx.GraphOps
Compute the number of triangles passing through each vertex.
TriangleCount - Class in org.apache.spark.graphx.lib
Compute the number of triangles passing through each vertex.
TriangleCount() - Constructor for class org.apache.spark.graphx.lib.TriangleCount
 
trigger(Trigger) - Method in class org.apache.spark.sql.streaming.DataStreamWriter
Set the trigger for the stream query.
Trigger - Class in org.apache.spark.sql.streaming
Policy used to indicate how often results should be produced by a [[StreamingQuery]].
Trigger() - Constructor for class org.apache.spark.sql.streaming.Trigger
 
TriggerThreadDump$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.TriggerThreadDump$
 
trim(Column) - Static method in class org.apache.spark.sql.functions
Trim the spaces from both ends for the specified string column.
trim(Column, String) - Static method in class org.apache.spark.sql.functions
Trim the specified character from both ends for the specified string column.
TrimHorizon() - Constructor for class org.apache.spark.streaming.kinesis.KinesisInitialPositions.TrimHorizon
 
TripletFields - Class in org.apache.spark.graphx
Represents a subset of the fields of an [[EdgeTriplet]] or [[EdgeContext]].
TripletFields() - Constructor for class org.apache.spark.graphx.TripletFields
Constructs a default TripletFields in which all fields are included.
TripletFields(boolean, boolean, boolean) - Constructor for class org.apache.spark.graphx.TripletFields
 
triplets() - Method in class org.apache.spark.graphx.Graph
An RDD containing the edge triplets, which are edges along with the vertex data associated with the adjacent vertices.
triplets() - Method in class org.apache.spark.graphx.impl.GraphImpl
Return an RDD that brings edges together with their source and destination vertices.
truePositiveRate(double) - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns true positive rate for a given label (category)
truePositiveRateByLabel() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns true positive rate for each label (category).
trunc(Column, String) - Static method in class org.apache.spark.sql.functions
Returns date truncated to the unit specified by the format.
truncatedString(Seq<T>, String, String, String, int) - Static method in class org.apache.spark.util.Utils
Format a sequence with semantics similar to calling .mkString().
truncatedString(Seq<T>, String) - Static method in class org.apache.spark.util.Utils
Shorthand for calling truncatedString() without start or end strings.
tryLog(Function0<T>) - Static method in class org.apache.spark.util.Utils
Executes the given block in a Try, logging any uncaught exceptions.
tryLogNonFatalError(Function0<BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Executes the given block.
tryOrExit(Function0<BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Execute a block of code that evaluates to Unit, forwarding any uncaught exceptions to the default UncaughtExceptionHandler
tryOrIOException(Function0<T>) - Static method in class org.apache.spark.util.Utils
Execute a block of code that returns a value, re-throwing any non-fatal uncaught exceptions as IOException.
tryOrStopSparkContext(SparkContext, Function0<BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Execute a block of code that evaluates to Unit, stop SparkContext if there is any uncaught exception
tryRecoverFromCheckpoint(String) - Method in class org.apache.spark.streaming.StreamingContextPythonHelper
This is a private method only for Python to implement getOrCreate.
tryWithResource(Function0<R>, Function1<R, T>) - Static method in class org.apache.spark.util.Utils
 
tryWithSafeFinally(Function0<T>, Function0<BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Execute a block of code, then a finally block, but if exceptions happen in the finally block, do not suppress the original exception.
tryWithSafeFinallyAndFailureCallbacks(Function0<T>, Function0<BoxedUnit>, Function0<BoxedUnit>) - Static method in class org.apache.spark.util.Utils
Execute a block of code and call the failure callbacks in the catch block.
tuple(Encoder<T1>, Encoder<T2>) - Static method in class org.apache.spark.sql.Encoders
An encoder for 2-ary tuples.
tuple(Encoder<T1>, Encoder<T2>, Encoder<T3>) - Static method in class org.apache.spark.sql.Encoders
An encoder for 3-ary tuples.
tuple(Encoder<T1>, Encoder<T2>, Encoder<T3>, Encoder<T4>) - Static method in class org.apache.spark.sql.Encoders
An encoder for 4-ary tuples.
tuple(Encoder<T1>, Encoder<T2>, Encoder<T3>, Encoder<T4>, Encoder<T5>) - Static method in class org.apache.spark.sql.Encoders
An encoder for 5-ary tuples.
tValues() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionTrainingSummary
T-statistic of estimated coefficients and intercept.
tValues() - Method in class org.apache.spark.ml.regression.LinearRegressionSummary
T-statistic of estimated coefficients and intercept.
Tweedie$() - Constructor for class org.apache.spark.ml.regression.GeneralizedLinearRegression.Tweedie$
 
TYPE() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
typed - Class in org.apache.spark.sql.expressions.javalang
Experimental Type-safe functions available for Dataset operations in Java.
typed() - Constructor for class org.apache.spark.sql.expressions.javalang.typed
 
typed - Class in org.apache.spark.sql.expressions.scalalang
Experimental Type-safe functions available for Dataset operations in Scala.
typed() - Constructor for class org.apache.spark.sql.expressions.scalalang.typed
 
TypedColumn<T,U> - Class in org.apache.spark.sql
A Column where an Encoder has been given for the expected input and return type.
TypedColumn(Expression, ExpressionEncoder<U>) - Constructor for class org.apache.spark.sql.TypedColumn
 
typedLit(T, TypeTags.TypeTag<T>) - Static method in class org.apache.spark.sql.functions
Creates a Column of literal value.
typeInfoConversions(DataType) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
typeName() - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
typeName() - Static method in class org.apache.spark.sql.types.ArrayType
 
typeName() - Static method in class org.apache.spark.sql.types.BinaryType
 
typeName() - Static method in class org.apache.spark.sql.types.BooleanType
 
typeName() - Static method in class org.apache.spark.sql.types.ByteType
 
typeName() - Static method in class org.apache.spark.sql.types.CalendarIntervalType
 
typeName() - Static method in class org.apache.spark.sql.types.CharType
 
typeName() - Method in class org.apache.spark.sql.types.DataType
Name of the type used in JSON serialization.
typeName() - Static method in class org.apache.spark.sql.types.DateType
 
typeName() - Method in class org.apache.spark.sql.types.DecimalType
 
typeName() - Static method in class org.apache.spark.sql.types.DoubleType
 
typeName() - Static method in class org.apache.spark.sql.types.FloatType
 
typeName() - Static method in class org.apache.spark.sql.types.HiveStringType
 
typeName() - Static method in class org.apache.spark.sql.types.IntegerType
 
typeName() - Static method in class org.apache.spark.sql.types.LongType
 
typeName() - Static method in class org.apache.spark.sql.types.MapType
 
typeName() - Static method in class org.apache.spark.sql.types.NullType
 
typeName() - Static method in class org.apache.spark.sql.types.NumericType
 
typeName() - Static method in class org.apache.spark.sql.types.ObjectType
 
typeName() - Static method in class org.apache.spark.sql.types.ShortType
 
typeName() - Static method in class org.apache.spark.sql.types.StringType
 
typeName() - Static method in class org.apache.spark.sql.types.StructType
 
typeName() - Static method in class org.apache.spark.sql.types.TimestampType
 
typeName() - Static method in class org.apache.spark.sql.types.VarcharType
 

U

U() - Method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
udf(Function0<RT>, TypeTags.TypeTag<RT>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 0 arguments as user-defined function (UDF).
udf(Function1<A1, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 1 arguments as user-defined function (UDF).
udf(Function2<A1, A2, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 2 arguments as user-defined function (UDF).
udf(Function3<A1, A2, A3, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 3 arguments as user-defined function (UDF).
udf(Function4<A1, A2, A3, A4, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 4 arguments as user-defined function (UDF).
udf(Function5<A1, A2, A3, A4, A5, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 5 arguments as user-defined function (UDF).
udf(Function6<A1, A2, A3, A4, A5, A6, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 6 arguments as user-defined function (UDF).
udf(Function7<A1, A2, A3, A4, A5, A6, A7, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 7 arguments as user-defined function (UDF).
udf(Function8<A1, A2, A3, A4, A5, A6, A7, A8, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 8 arguments as user-defined function (UDF).
udf(Function9<A1, A2, A3, A4, A5, A6, A7, A8, A9, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 9 arguments as user-defined function (UDF).
udf(Function10<A1, A2, A3, A4, A5, A6, A7, A8, A9, A10, RT>, TypeTags.TypeTag<RT>, TypeTags.TypeTag<A1>, TypeTags.TypeTag<A2>, TypeTags.TypeTag<A3>, TypeTags.TypeTag<A4>, TypeTags.TypeTag<A5>, TypeTags.TypeTag<A6>, TypeTags.TypeTag<A7>, TypeTags.TypeTag<A8>, TypeTags.TypeTag<A9>, TypeTags.TypeTag<A10>) - Static method in class org.apache.spark.sql.functions
Defines a Scala closure of 10 arguments as user-defined function (UDF).
udf(UDF0<?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF0 instance as user-defined function (UDF).
udf(UDF1<?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF1 instance as user-defined function (UDF).
udf(UDF2<?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF2 instance as user-defined function (UDF).
udf(UDF3<?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF3 instance as user-defined function (UDF).
udf(UDF4<?, ?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF4 instance as user-defined function (UDF).
udf(UDF5<?, ?, ?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF5 instance as user-defined function (UDF).
udf(UDF6<?, ?, ?, ?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF6 instance as user-defined function (UDF).
udf(UDF7<?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF7 instance as user-defined function (UDF).
udf(UDF8<?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF8 instance as user-defined function (UDF).
udf(UDF9<?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF9 instance as user-defined function (UDF).
udf(UDF10<?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?>, DataType) - Static method in class org.apache.spark.sql.functions
Defines a Java UDF10 instance as user-defined function (UDF).
udf(Object, DataType) - Static method in class org.apache.spark.sql.functions
Defines a deterministic user-defined function (UDF) using a Scala closure.
udf() - Method in class org.apache.spark.sql.SparkSession
A collection of methods for registering user-defined functions (UDF).
udf() - Method in class org.apache.spark.sql.SQLContext
A collection of methods for registering user-defined functions (UDF).
UDF0<R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 0 arguments.
UDF1<T1,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 1 arguments.
UDF10<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 10 arguments.
UDF11<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 11 arguments.
UDF12<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 12 arguments.
UDF13<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 13 arguments.
UDF14<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 14 arguments.
UDF15<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 15 arguments.
UDF16<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,T16,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 16 arguments.
UDF17<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,T16,T17,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 17 arguments.
UDF18<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,T16,T17,T18,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 18 arguments.
UDF19<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,T16,T17,T18,T19,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 19 arguments.
UDF2<T1,T2,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 2 arguments.
UDF20<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,T16,T17,T18,T19,T20,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 20 arguments.
UDF21<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,T16,T17,T18,T19,T20,T21,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 21 arguments.
UDF22<T1,T2,T3,T4,T5,T6,T7,T8,T9,T10,T11,T12,T13,T14,T15,T16,T17,T18,T19,T20,T21,T22,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 22 arguments.
UDF3<T1,T2,T3,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 3 arguments.
UDF4<T1,T2,T3,T4,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 4 arguments.
UDF5<T1,T2,T3,T4,T5,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 5 arguments.
UDF6<T1,T2,T3,T4,T5,T6,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 6 arguments.
UDF7<T1,T2,T3,T4,T5,T6,T7,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 7 arguments.
UDF8<T1,T2,T3,T4,T5,T6,T7,T8,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 8 arguments.
UDF9<T1,T2,T3,T4,T5,T6,T7,T8,T9,R> - Interface in org.apache.spark.sql.api.java
A Spark SQL UDF that has 9 arguments.
UDFRegistration - Class in org.apache.spark.sql
Functions for registering user-defined functions.
UDTRegistration - Class in org.apache.spark.sql.types
This object keeps the mappings between user classes and their User Defined Types (UDTs).
UDTRegistration() - Constructor for class org.apache.spark.sql.types.UDTRegistration
 
uid() - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
uid() - Method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
uid() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
uid() - Method in class org.apache.spark.ml.classification.GBTClassifier
 
uid() - Method in class org.apache.spark.ml.classification.LinearSVC
 
uid() - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
uid() - Method in class org.apache.spark.ml.classification.LogisticRegression
 
uid() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
uid() - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
uid() - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
uid() - Method in class org.apache.spark.ml.classification.NaiveBayes
 
uid() - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
uid() - Method in class org.apache.spark.ml.classification.OneVsRest
 
uid() - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
uid() - Static method in class org.apache.spark.ml.classification.ProbabilisticClassificationModel
 
uid() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
uid() - Method in class org.apache.spark.ml.classification.RandomForestClassifier
 
uid() - Method in class org.apache.spark.ml.clustering.BisectingKMeans
 
uid() - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
uid() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
uid() - Method in class org.apache.spark.ml.clustering.GaussianMixture
 
uid() - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
uid() - Method in class org.apache.spark.ml.clustering.KMeans
 
uid() - Method in class org.apache.spark.ml.clustering.KMeansModel
 
uid() - Method in class org.apache.spark.ml.clustering.LDA
 
uid() - Method in class org.apache.spark.ml.clustering.LDAModel
 
uid() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
uid() - Method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
uid() - Method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
uid() - Method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
uid() - Method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
uid() - Method in class org.apache.spark.ml.feature.Binarizer
 
uid() - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
uid() - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
uid() - Method in class org.apache.spark.ml.feature.Bucketizer
 
uid() - Method in class org.apache.spark.ml.feature.ChiSqSelector
 
uid() - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
uid() - Method in class org.apache.spark.ml.feature.ColumnPruner
 
uid() - Method in class org.apache.spark.ml.feature.CountVectorizer
 
uid() - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
uid() - Method in class org.apache.spark.ml.feature.DCT
 
uid() - Method in class org.apache.spark.ml.feature.ElementwiseProduct
 
uid() - Method in class org.apache.spark.ml.feature.FeatureHasher
 
uid() - Method in class org.apache.spark.ml.feature.HashingTF
 
uid() - Method in class org.apache.spark.ml.feature.IDF
 
uid() - Method in class org.apache.spark.ml.feature.IDFModel
 
uid() - Method in class org.apache.spark.ml.feature.Imputer
 
uid() - Method in class org.apache.spark.ml.feature.ImputerModel
 
uid() - Method in class org.apache.spark.ml.feature.IndexToString
 
uid() - Method in class org.apache.spark.ml.feature.Interaction
 
uid() - Method in class org.apache.spark.ml.feature.MaxAbsScaler
 
uid() - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
uid() - Method in class org.apache.spark.ml.feature.MinHashLSH
 
uid() - Method in class org.apache.spark.ml.feature.MinHashLSHModel
 
uid() - Method in class org.apache.spark.ml.feature.MinMaxScaler
 
uid() - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
uid() - Method in class org.apache.spark.ml.feature.NGram
 
uid() - Method in class org.apache.spark.ml.feature.Normalizer
 
uid() - Method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
uid() - Method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
uid() - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
uid() - Method in class org.apache.spark.ml.feature.PCA
 
uid() - Method in class org.apache.spark.ml.feature.PCAModel
 
uid() - Method in class org.apache.spark.ml.feature.PolynomialExpansion
 
uid() - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
uid() - Method in class org.apache.spark.ml.feature.RegexTokenizer
 
uid() - Method in class org.apache.spark.ml.feature.RFormula
 
uid() - Method in class org.apache.spark.ml.feature.RFormulaModel
 
uid() - Method in class org.apache.spark.ml.feature.SQLTransformer
 
uid() - Method in class org.apache.spark.ml.feature.StandardScaler
 
uid() - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
uid() - Method in class org.apache.spark.ml.feature.StopWordsRemover
 
uid() - Method in class org.apache.spark.ml.feature.StringIndexer
 
uid() - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
uid() - Method in class org.apache.spark.ml.feature.Tokenizer
 
uid() - Method in class org.apache.spark.ml.feature.VectorAssembler
 
uid() - Method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
uid() - Method in class org.apache.spark.ml.feature.VectorIndexer
 
uid() - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
uid() - Method in class org.apache.spark.ml.feature.VectorSizeHint
 
uid() - Method in class org.apache.spark.ml.feature.VectorSlicer
 
uid() - Method in class org.apache.spark.ml.feature.Word2Vec
 
uid() - Method in class org.apache.spark.ml.feature.Word2VecModel
 
uid() - Method in class org.apache.spark.ml.fpm.FPGrowth
 
uid() - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
uid() - Method in class org.apache.spark.ml.Pipeline
 
uid() - Method in class org.apache.spark.ml.PipelineModel
 
uid() - Method in class org.apache.spark.ml.recommendation.ALS
 
uid() - Method in class org.apache.spark.ml.recommendation.ALSModel
 
uid() - Method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
uid() - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
uid() - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
uid() - Method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
uid() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
 
uid() - Method in class org.apache.spark.ml.regression.GBTRegressor
 
uid() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
uid() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
uid() - Method in class org.apache.spark.ml.regression.IsotonicRegression
 
uid() - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
uid() - Method in class org.apache.spark.ml.regression.LinearRegression
 
uid() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
 
uid() - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
uid() - Method in class org.apache.spark.ml.regression.RandomForestRegressor
 
uid() - Method in class org.apache.spark.ml.tuning.CrossValidator
 
uid() - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
uid() - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
uid() - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
uid() - Method in interface org.apache.spark.ml.util.Identifiable
An immutable unique ID for the object and its derivatives.
uiRoot() - Static method in class org.apache.spark.ui.UIUtils
 
UIRootFromServletContext - Class in org.apache.spark.status.api.v1
 
UIRootFromServletContext() - Constructor for class org.apache.spark.status.api.v1.UIRootFromServletContext
 
UIUtils - Class in org.apache.spark.streaming.ui
 
UIUtils() - Constructor for class org.apache.spark.streaming.ui.UIUtils
 
UIUtils - Class in org.apache.spark.ui
Utility functions for generating XML pages with spark content.
UIUtils() - Constructor for class org.apache.spark.ui.UIUtils
 
uiWebUrl() - Method in class org.apache.spark.SparkContext
 
UIWorkloadGenerator - Class in org.apache.spark.ui
Continuously generates jobs that expose various features of the WebUI (internal testing tool).
UIWorkloadGenerator() - Constructor for class org.apache.spark.ui.UIWorkloadGenerator
 
unapply(EdgeContext<VD, ED, A>) - Static method in class org.apache.spark.graphx.EdgeContext
Extractor mainly used for Graph#aggregateMessages*.
unapply(DenseVector) - Static method in class org.apache.spark.ml.linalg.DenseVector
Extracts the value array from a dense vector.
unapply(SparseVector) - Static method in class org.apache.spark.ml.linalg.SparseVector
 
unapply(DenseVector) - Static method in class org.apache.spark.mllib.linalg.DenseVector
Extracts the value array from a dense vector.
unapply(SparseVector) - Static method in class org.apache.spark.mllib.linalg.SparseVector
 
unapply(Column) - Static method in class org.apache.spark.sql.Column
 
unapply(Expression) - Method in class org.apache.spark.sql.types.DecimalType.Expression$
 
unapply(DecimalType) - Method in class org.apache.spark.sql.types.DecimalType.Fixed$
 
unapply(DataType) - Static method in class org.apache.spark.sql.types.DecimalType
 
unapply(Expression) - Static method in class org.apache.spark.sql.types.DecimalType
 
unapply(Expression) - Static method in class org.apache.spark.sql.types.NumericType
Enables matching against NumericType for expressions:
unapply(Throwable) - Static method in class org.apache.spark.util.CausedBy
 
unapply(String) - Static method in class org.apache.spark.util.IntParam
 
unapply(String) - Static method in class org.apache.spark.util.MemoryParam
 
UnaryTransformer<IN,OUT,T extends UnaryTransformer<IN,OUT,T>> - Class in org.apache.spark.ml
Developer API Abstract class for transformers that take one input column, apply transformation, and output the result as a new column.
UnaryTransformer() - Constructor for class org.apache.spark.ml.UnaryTransformer
 
unbase64(Column) - Static method in class org.apache.spark.sql.functions
Decodes a BASE64 encoded string column and returns it as a binary column.
unboundedFollowing() - Static method in class org.apache.spark.sql.expressions.Window
Value representing the last row in the partition, equivalent to "UNBOUNDED FOLLOWING" in SQL.
unboundedFollowing() - Static method in class org.apache.spark.sql.functions
Window function: returns the special frame boundary that represents the last row in the window partition.
unboundedPreceding() - Static method in class org.apache.spark.sql.expressions.Window
Value representing the first row in the partition, equivalent to "UNBOUNDED PRECEDING" in SQL.
unboundedPreceding() - Static method in class org.apache.spark.sql.functions
Window function: returns the special frame boundary that represents the first row in the window partition.
uncacheTable(String) - Method in class org.apache.spark.sql.catalog.Catalog
Removes the specified table from the in-memory cache.
uncacheTable(String) - Method in class org.apache.spark.sql.SQLContext
Removes the specified table from the in-memory cache.
UNCAUGHT_EXCEPTION() - Static method in class org.apache.spark.util.SparkExitCode
The default uncaught exception handler was reached.
UNCAUGHT_EXCEPTION_TWICE() - Static method in class org.apache.spark.util.SparkExitCode
The default uncaught exception handler was called and an exception was encountered while
undefinedImageType() - Static method in class org.apache.spark.ml.image.ImageSchema
 
underlyingSplit() - Method in class org.apache.spark.scheduler.SplitInfo
 
unhandledFilters(Filter[]) - Method in class org.apache.spark.sql.sources.BaseRelation
Returns the list of Filters that this datasource may not be able to handle.
unhex(Column) - Static method in class org.apache.spark.sql.functions
Inverse of hex.
UniformGenerator - Class in org.apache.spark.mllib.random
Developer API Generates i.i.d.
UniformGenerator() - Constructor for class org.apache.spark.mllib.random.UniformGenerator
 
uniformJavaRDD(JavaSparkContext, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.uniformRDD.
uniformJavaRDD(JavaSparkContext, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.uniformJavaRDD with the default seed.
uniformJavaRDD(JavaSparkContext, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.uniformJavaRDD with the default number of partitions and the default seed.
uniformJavaVectorRDD(JavaSparkContext, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Java-friendly version of RandomRDDs.uniformVectorRDD.
uniformJavaVectorRDD(JavaSparkContext, long, int, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.uniformJavaVectorRDD with the default seed.
uniformJavaVectorRDD(JavaSparkContext, long, int) - Static method in class org.apache.spark.mllib.random.RandomRDDs
RandomRDDs.uniformJavaVectorRDD with the default number of partitions and the default seed.
uniformRDD(SparkContext, long, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD comprised of i.i.d. samples from the uniform distribution U(0.0, 1.0).
uniformVectorRDD(SparkContext, long, int, int, long) - Static method in class org.apache.spark.mllib.random.RandomRDDs
Generates an RDD[Vector] with vectors containing i.i.d. samples drawn from the uniform distribution on U(0.0, 1.0).
union(JavaDoubleRDD) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Return the union of this RDD and another one.
union(JavaPairRDD<K, V>) - Method in class org.apache.spark.api.java.JavaPairRDD
Return the union of this RDD and another one.
union(JavaRDD<T>) - Method in class org.apache.spark.api.java.JavaRDD
Return the union of this RDD and another one.
union(JavaRDD<T>, List<JavaRDD<T>>) - Method in class org.apache.spark.api.java.JavaSparkContext
Build the union of two or more RDDs.
union(JavaPairRDD<K, V>, List<JavaPairRDD<K, V>>) - Method in class org.apache.spark.api.java.JavaSparkContext
Build the union of two or more RDDs.
union(JavaDoubleRDD, List<JavaDoubleRDD>) - Method in class org.apache.spark.api.java.JavaSparkContext
Build the union of two or more RDDs.
union(RDD<T>) - Static method in class org.apache.spark.api.r.RRDD
 
union(RDD<T>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
union(RDD<T>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
union(RDD<T>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
union(RDD<T>) - Static method in class org.apache.spark.graphx.VertexRDD
 
union(RDD<T>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
union(RDD<T>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
union(RDD<T>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
union(RDD<T>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
union(RDD<T>) - Method in class org.apache.spark.rdd.RDD
Return the union of this RDD and another one.
union(RDD<T>) - Static method in class org.apache.spark.rdd.UnionRDD
 
union(Seq<RDD<T>>, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Build the union of a list of RDDs.
union(RDD<T>, Seq<RDD<T>>, ClassTag<T>) - Method in class org.apache.spark.SparkContext
Build the union of a list of RDDs passed as variable-length arguments.
union(Dataset<T>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset containing union of rows in this Dataset and another Dataset.
union(GenSeq<B>, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
union(JavaDStream<T>) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Return a new DStream by unifying data of another DStream with this DStream.
union(JavaDStream<T>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
union(JavaPairDStream<K, V>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream by unifying data of another DStream with this DStream.
union(JavaPairDStream<K, V>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
union(JavaPairDStream<K, V>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
union(JavaDStream<T>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
union(JavaDStream<T>, List<JavaDStream<T>>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a unified DStream from multiple DStreams of the same type and same slide duration.
union(JavaPairDStream<K, V>, List<JavaPairDStream<K, V>>) - Method in class org.apache.spark.streaming.api.java.JavaStreamingContext
Create a unified DStream from multiple DStreams of the same type and same slide duration.
union(DStream<T>) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream by unifying data of another DStream with this DStream.
union(Seq<DStream<T>>, ClassTag<T>) - Method in class org.apache.spark.streaming.StreamingContext
Create a unified DStream from multiple DStreams of the same type and same slide duration.
unionAll(Dataset<T>) - Method in class org.apache.spark.sql.Dataset
Deprecated.
use union(). Since 2.0.0.
unionByName(Dataset<T>) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset containing union of rows in this Dataset and another Dataset.
unionFileLists(Option<String>, Option<String>) - Static method in class org.apache.spark.util.Utils
Unions two comma-separated lists of files and filters out empty strings.
UnionRDD<T> - Class in org.apache.spark.rdd
 
UnionRDD(SparkContext, Seq<RDD<T>>, ClassTag<T>) - Constructor for class org.apache.spark.rdd.UnionRDD
 
uniqueId() - Method in class org.apache.spark.storage.StreamBlockId
 
unix_timestamp() - Static method in class org.apache.spark.sql.functions
Returns the current Unix timestamp (in seconds).
unix_timestamp(Column) - Static method in class org.apache.spark.sql.functions
Converts time string in format yyyy-MM-dd HH:mm:ss to Unix timestamp (in seconds), using the default timezone and the default locale.
unix_timestamp(Column, String) - Static method in class org.apache.spark.sql.functions
Converts time string with given pattern to Unix timestamp (in seconds).
UnknownReason - Class in org.apache.spark
Developer API We don't know why the task ended -- for example, because of a ClassNotFound exception when deserializing the task result.
UnknownReason() - Constructor for class org.apache.spark.UnknownReason
 
UNLIMITED_DECIMAL_PRECISION() - Static method in class org.apache.spark.sql.hive.HiveShim
 
UNLIMITED_DECIMAL_SCALE() - Static method in class org.apache.spark.sql.hive.HiveShim
 
unlink(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.CLogLog$
 
unlink(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Identity$
 
unlink(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Inverse$
 
unlink(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Log$
 
unlink(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Logit$
 
unlink(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Probit$
 
unlink(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Sqrt$
 
UNORDERED() - Static method in class org.apache.spark.rdd.DeterministicLevel
 
unpersist() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
unpersist(boolean) - Method in class org.apache.spark.api.java.JavaDoubleRDD
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
unpersist() - Method in class org.apache.spark.api.java.JavaPairRDD
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
unpersist(boolean) - Method in class org.apache.spark.api.java.JavaPairRDD
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
unpersist() - Method in class org.apache.spark.api.java.JavaRDD
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
unpersist(boolean) - Method in class org.apache.spark.api.java.JavaRDD
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
unpersist(boolean) - Static method in class org.apache.spark.api.r.RRDD
 
unpersist() - Method in class org.apache.spark.broadcast.Broadcast
Asynchronously delete cached copies of this broadcast on the executors.
unpersist(boolean) - Method in class org.apache.spark.broadcast.Broadcast
Delete cached copies of this broadcast on the executors.
unpersist(boolean) - Static method in class org.apache.spark.graphx.EdgeRDD
 
unpersist(boolean) - Method in class org.apache.spark.graphx.Graph
Uncaches both vertices and edges of this graph.
unpersist(boolean) - Method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
unpersist(boolean) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
unpersist(boolean) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
unpersist(boolean) - Static method in class org.apache.spark.graphx.VertexRDD
 
unpersist() - Method in class org.apache.spark.mllib.evaluation.BinaryClassificationMetrics
Unpersist intermediate RDDs used in the computation.
unpersist(boolean) - Static method in class org.apache.spark.rdd.HadoopRDD
 
unpersist(boolean) - Static method in class org.apache.spark.rdd.JdbcRDD
 
unpersist(boolean) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
unpersist(boolean) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
unpersist(boolean) - Method in class org.apache.spark.rdd.RDD
Mark the RDD as non-persistent, and remove all blocks for it from memory and disk.
unpersist(boolean) - Static method in class org.apache.spark.rdd.UnionRDD
 
unpersist(boolean) - Method in class org.apache.spark.sql.Dataset
Mark the Dataset as non-persistent, and remove all blocks for it from memory and disk.
unpersist() - Method in class org.apache.spark.sql.Dataset
Mark the Dataset as non-persistent, and remove all blocks for it from memory and disk.
unpersist$default$1() - Static method in class org.apache.spark.api.r.RRDD
 
unpersist$default$1() - Static method in class org.apache.spark.graphx.EdgeRDD
 
unpersist$default$1() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
unpersist$default$1() - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
unpersist$default$1() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
unpersist$default$1() - Static method in class org.apache.spark.graphx.VertexRDD
 
unpersist$default$1() - Static method in class org.apache.spark.rdd.HadoopRDD
 
unpersist$default$1() - Static method in class org.apache.spark.rdd.JdbcRDD
 
unpersist$default$1() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
unpersist$default$1() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
unpersist$default$1() - Static method in class org.apache.spark.rdd.UnionRDD
 
unpersistRDDFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
unpersistRDDToJson(SparkListenerUnpersistRDD) - Static method in class org.apache.spark.util.JsonProtocol
 
unpersistVertices(boolean) - Method in class org.apache.spark.graphx.Graph
Uncaches only the vertices of this graph, leaving the edges alone.
unpersistVertices(boolean) - Method in class org.apache.spark.graphx.impl.GraphImpl
 
unpersistVertices$default$1() - Static method in class org.apache.spark.graphx.impl.GraphImpl
 
UnrecognizedBlockId - Exception in org.apache.spark.storage
 
UnrecognizedBlockId(String) - Constructor for exception org.apache.spark.storage.UnrecognizedBlockId
 
unregister(QueryExecutionListener) - Method in class org.apache.spark.sql.util.ExecutionListenerManager
Unregisters the specified QueryExecutionListener.
unregisterDialect(JdbcDialect) - Static method in class org.apache.spark.sql.jdbc.JdbcDialects
Unregister a dialect.
Unresolved() - Static method in class org.apache.spark.ml.attribute.AttributeType
Unresolved type.
UnresolvedAttribute - Class in org.apache.spark.ml.attribute
Developer API An unresolved attribute.
UnresolvedAttribute() - Constructor for class org.apache.spark.ml.attribute.UnresolvedAttribute
 
unset() - Static method in class org.apache.spark.rdd.InputFileBlockHolder
Clears the input file block to default value.
unset(String) - Method in class org.apache.spark.sql.RuntimeConfig
Resets the configuration property for the given key.
until(Time, Duration) - Method in class org.apache.spark.streaming.Time
 
unwrapOrcStructs(Configuration, StructType, StructType, Option<StructObjectInspector>, Iterator<Writable>) - Static method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
unwrapperFor(ObjectInspector) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
unwrapperFor(StructField) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
unzip(Function1<A, Tuple2<A1, A2>>) - Static method in class org.apache.spark.sql.types.StructType
 
unzip3(Function1<A, Tuple3<A1, A2, A3>>) - Static method in class org.apache.spark.sql.types.StructType
 
update(int, int, double) - Method in interface org.apache.spark.ml.linalg.Matrix
Update element at (i, j)
update(Function1<Object, Object>) - Method in interface org.apache.spark.ml.linalg.Matrix
Update all the values of this matrix using the function f.
update(RDD<Vector>, double, String) - Method in class org.apache.spark.mllib.clustering.StreamingKMeansModel
Perform a k-means update on a batch of data.
update(int, int, double) - Method in interface org.apache.spark.mllib.linalg.Matrix
Update element at (i, j)
update(Function1<Object, Object>) - Method in interface org.apache.spark.mllib.linalg.Matrix
Update all the values of this matrix using the function f.
update() - Method in class org.apache.spark.scheduler.AccumulableInfo
 
update(int, Object) - Method in class org.apache.spark.sql.expressions.MutableAggregationBuffer
Update the ith value of this buffer.
update(MutableAggregationBuffer, Row) - Method in class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
Updates the given aggregation buffer buffer with new input data from input.
update(S) - Method in interface org.apache.spark.sql.streaming.GroupState
Update the value of the state.
Update() - Static method in class org.apache.spark.sql.streaming.OutputMode
OutputMode in which only the rows that were updated in the streaming DataFrame/Dataset will be written to the sink every time there are some updates.
update(int, Object) - Method in class org.apache.spark.sql.vectorized.ColumnarArray
 
update(int, Object) - Method in class org.apache.spark.sql.vectorized.ColumnarRow
 
update() - Method in class org.apache.spark.status.api.v1.AccumulableInfo
 
update(Seq<String>, String, long, long) - Method in class org.apache.spark.status.LiveRDDPartition
 
update(S) - Method in class org.apache.spark.streaming.State
Update the state with a new value.
update(T1, T2) - Method in class org.apache.spark.util.MutablePair
Updates this pair with new values and returns itself
UpdateBlockInfo(BlockManagerId, BlockId, StorageLevel, long, long) - Constructor for class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
UpdateBlockInfo() - Constructor for class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
UpdateBlockInfo$() - Constructor for class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo$
 
updated(int, B, CanBuildFrom<Repr, B, That>) - Static method in class org.apache.spark.sql.types.StructType
 
UPDATED_BLOCK_STATUSES() - Static method in class org.apache.spark.InternalAccumulator
 
UpdateDelegationTokens(byte[]) - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.UpdateDelegationTokens
 
UpdateDelegationTokens$() - Constructor for class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.UpdateDelegationTokens$
 
updateMetrics(TaskMetrics) - Method in class org.apache.spark.status.LiveTask
Update the metrics for the task and return the difference between the previous and new values.
updatePrediction(Vector, double, DecisionTreeRegressionModel, double) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Add prediction from a new boosting iteration to an existing prediction.
updatePredictionError(RDD<LabeledPoint>, RDD<Tuple2<Object, Object>>, double, DecisionTreeRegressionModel, Loss) - Static method in class org.apache.spark.ml.tree.impl.GradientBoostedTrees
Update a zipped predictionError RDD (as obtained with computeInitialPredictionAndError)
updatePredictionError(RDD<LabeledPoint>, RDD<Tuple2<Object, Object>>, double, DecisionTreeModel, Loss) - Static method in class org.apache.spark.mllib.tree.model.GradientBoostedTreesModel
Developer API Update a zipped predictionError RDD (as obtained with computeInitialPredictionAndError)
Updater - Class in org.apache.spark.mllib.optimization
Developer API Class used to perform steps (weight update) using Gradient Descent methods.
Updater() - Constructor for class org.apache.spark.mllib.optimization.Updater
 
updateRddInfo(Seq<RDDInfo>, Seq<StorageStatus>) - Static method in class org.apache.spark.storage.StorageUtils
Update the given list of RDDInfo with the given list of storage statuses.
updateSparkConfigFromProperties(SparkConf, Map<String, String>) - Static method in class org.apache.spark.util.Utils
Updates Spark config with properties from a set of Properties.
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of each key.
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, int) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of each key.
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, Partitioner) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of the key.
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, Partitioner, JavaPairRDD<K, S>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of the key.
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, Partitioner, JavaPairRDD<K, S>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, int) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, Partitioner) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
updateStateByKey(Function2<List<V>, Optional<S>, Optional<S>>, Partitioner, JavaPairRDD<K, S>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
updateStateByKey(Function2<Seq<V>, Option<S>, Option<S>>, ClassTag<S>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of each key.
updateStateByKey(Function2<Seq<V>, Option<S>, Option<S>>, int, ClassTag<S>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of each key.
updateStateByKey(Function2<Seq<V>, Option<S>, Option<S>>, Partitioner, ClassTag<S>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of the key.
updateStateByKey(Function1<Iterator<Tuple3<K, Seq<V>, Option<S>>>, Iterator<Tuple2<K, S>>>, Partitioner, boolean, ClassTag<S>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of each key.
updateStateByKey(Function2<Seq<V>, Option<S>, Option<S>>, Partitioner, RDD<Tuple2<K, S>>, ClassTag<S>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of the key.
updateStateByKey(Function1<Iterator<Tuple3<K, Seq<V>, Option<S>>>, Iterator<Tuple2<K, S>>>, Partitioner, boolean, RDD<Tuple2<K, S>>, ClassTag<S>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of each key.
updateStateByKey(Function4<Time, K, Seq<V>, Option<S>, Option<S>>, Partitioner, boolean, Option<RDD<Tuple2<K, S>>>, ClassTag<S>) - Method in class org.apache.spark.streaming.dstream.PairDStreamFunctions
Return a new "state" DStream where the state for each key is updated by applying the given function on the previous state of the key and the new values of the key.
upper(Column) - Static method in class org.apache.spark.sql.functions
Converts a string column to upper case.
upperBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
upperBoundsOnCoefficients() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
upperBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
upperBoundsOnIntercepts() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
useDisk() - Method in class org.apache.spark.storage.StorageLevel
 
usedOffHeap() - Method in class org.apache.spark.status.LiveExecutor
 
usedOffHeapStorageMemory() - Method in class org.apache.spark.status.api.v1.MemoryMetrics
 
usedOnHeap() - Method in class org.apache.spark.status.LiveExecutor
 
usedOnHeapStorageMemory() - Method in class org.apache.spark.status.api.v1.MemoryMetrics
 
useDst - Variable in class org.apache.spark.graphx.TripletFields
Indicates whether the destination vertex attribute is included.
useEdge - Variable in class org.apache.spark.graphx.TripletFields
Indicates whether the edge attribute is included.
useMemory() - Method in class org.apache.spark.storage.StorageLevel
 
useNodeIdCache() - Method in class org.apache.spark.mllib.tree.configuration.Strategy
 
useOffHeap() - Method in class org.apache.spark.storage.StorageLevel
 
user() - Method in class org.apache.spark.ml.recommendation.ALS.Rating
 
user() - Method in class org.apache.spark.mllib.recommendation.Rating
 
USER_DEFAULT() - Static method in class org.apache.spark.sql.types.DecimalType
 
userClass() - Method in class org.apache.spark.mllib.linalg.VectorUDT
 
userCol() - Static method in class org.apache.spark.ml.recommendation.ALS
 
userCol() - Static method in class org.apache.spark.ml.recommendation.ALSModel
 
UserDefinedAggregateFunction - Class in org.apache.spark.sql.expressions
The base class for implementing user-defined aggregate functions (UDAF).
UserDefinedAggregateFunction() - Constructor for class org.apache.spark.sql.expressions.UserDefinedAggregateFunction
 
UserDefinedFunction - Class in org.apache.spark.sql.expressions
A user-defined function.
userFactors() - Method in class org.apache.spark.ml.recommendation.ALSModel
 
userFeatures() - Method in class org.apache.spark.mllib.recommendation.MatrixFactorizationModel
 
userPort(int, int) - Static method in class org.apache.spark.util.Utils
Returns the user port to try when trying to bind a service.
useSrc - Variable in class org.apache.spark.graphx.TripletFields
Indicates whether the source vertex attribute is included.
Utils - Class in org.apache.spark.ml.impl
 
Utils() - Constructor for class org.apache.spark.ml.impl.Utils
 
Utils - Class in org.apache.spark.util
Various utility methods used by Spark.
Utils() - Constructor for class org.apache.spark.util.Utils
 
UUIDFromJson(JsonAST.JValue) - Static method in class org.apache.spark.util.JsonProtocol
 
UUIDToJson(UUID) - Static method in class org.apache.spark.util.JsonProtocol
 

V

V() - Method in class org.apache.spark.mllib.linalg.SingularValueDecomposition
 
validate() - Method in class org.apache.spark.mllib.linalg.distributed.BlockMatrix
Validates the block matrix info against the matrix data (blocks) and throws an exception if any error is found.
validateAndTransformSchema(StructType, boolean, DataType) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
validateAndTransformSchema(StructType, boolean, DataType) - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
validateStages(PipelineStage[]) - Method in class org.apache.spark.ml.Pipeline.SharedReadWrite$
Check that all stages are Writable
validateURL(URI) - Static method in class org.apache.spark.util.Utils
Validate that a given URI is actually a valid URL as well.
validationMetrics() - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
validationTol() - Method in class org.apache.spark.mllib.tree.configuration.BoostingStrategy
 
value() - Method in class org.apache.spark.Accumulable
Deprecated.
Access the accumulator's current value; only allowed on driver.
value() - Static method in class org.apache.spark.Accumulator
Deprecated.
 
value() - Method in class org.apache.spark.broadcast.Broadcast
Get the broadcasted value.
value() - Method in class org.apache.spark.ComplexFutureAction
 
value() - Method in interface org.apache.spark.FutureAction
The value of this Future.
value() - Method in class org.apache.spark.ml.param.ParamPair
 
value() - Method in class org.apache.spark.mllib.linalg.distributed.MatrixEntry
 
value() - Method in class org.apache.spark.mllib.stat.test.BinarySample
 
value() - Method in class org.apache.spark.scheduler.AccumulableInfo
 
value() - Method in class org.apache.spark.SerializableWritable
 
value() - Method in class org.apache.spark.SimpleFutureAction
 
value() - Method in class org.apache.spark.sql.sources.EqualNullSafe
 
value() - Method in class org.apache.spark.sql.sources.EqualTo
 
value() - Method in class org.apache.spark.sql.sources.GreaterThan
 
value() - Method in class org.apache.spark.sql.sources.GreaterThanOrEqual
 
value() - Method in class org.apache.spark.sql.sources.LessThan
 
value() - Method in class org.apache.spark.sql.sources.LessThanOrEqual
 
value() - Method in class org.apache.spark.sql.sources.StringContains
 
value() - Method in class org.apache.spark.sql.sources.StringEndsWith
 
value() - Method in class org.apache.spark.sql.sources.StringStartsWith
 
value() - Method in class org.apache.spark.status.api.v1.AccumulableInfo
 
value() - Method in class org.apache.spark.status.LiveRDDPartition
 
value() - Method in class org.apache.spark.storage.memory.DeserializedMemoryEntry
 
value() - Method in class org.apache.spark.util.AccumulatorV2
Defines the current value of this accumulator
value() - Method in class org.apache.spark.util.CollectionAccumulator
 
value() - Method in class org.apache.spark.util.DoubleAccumulator
 
value() - Method in class org.apache.spark.util.LegacyAccumulatorWrapper
 
value() - Method in class org.apache.spark.util.LongAccumulator
 
value_$eq(R) - Static method in class org.apache.spark.Accumulator
Deprecated.
 
valueArray() - Method in class org.apache.spark.sql.vectorized.ColumnarMap
 
valueContainsNull() - Method in class org.apache.spark.sql.types.MapType
 
valueOf(String) - Static method in enum org.apache.spark.graphx.impl.EdgeActiveness
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.JobExecutionStatus
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.launcher.SparkAppHandle.State
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.sql.SaveMode
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.status.api.v1.ApplicationStatus
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.status.api.v1.StageStatus
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.status.api.v1.streaming.BatchStatus
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.status.api.v1.TaskSorting
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.streaming.StreamingContextState
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.util.sketch.BloomFilter.Version
Returns the enum constant of this type with the specified name.
valueOf(String) - Static method in enum org.apache.spark.util.sketch.CountMinSketch.Version
Returns the enum constant of this type with the specified name.
values() - Method in class org.apache.spark.api.java.JavaPairRDD
Return an RDD with the values of each tuple.
values() - Static method in enum org.apache.spark.graphx.impl.EdgeActiveness
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.spark.JobExecutionStatus
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.spark.launcher.SparkAppHandle.State
Returns an array containing the constants of this enum type, in the order they are declared.
VALUES() - Static method in class org.apache.spark.ml.attribute.AttributeKeys
 
values() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
values() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
values() - Method in class org.apache.spark.ml.linalg.DenseMatrix
 
values() - Method in class org.apache.spark.ml.linalg.DenseVector
 
values() - Method in class org.apache.spark.ml.linalg.SparseMatrix
 
values() - Method in class org.apache.spark.ml.linalg.SparseVector
 
values() - Method in class org.apache.spark.mllib.linalg.DenseMatrix
 
values() - Method in class org.apache.spark.mllib.linalg.DenseVector
 
values() - Method in class org.apache.spark.mllib.linalg.SparseMatrix
 
values() - Method in class org.apache.spark.mllib.linalg.SparseVector
 
values() - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
values() - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
values() - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
values() - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
values() - Static method in class org.apache.spark.rdd.CheckpointState
 
values() - Static method in class org.apache.spark.rdd.DeterministicLevel
 
values() - Method in class org.apache.spark.rdd.PairRDDFunctions
Return an RDD with the values of each tuple.
values() - Static method in class org.apache.spark.scheduler.SchedulingMode
 
values() - Static method in class org.apache.spark.scheduler.TaskLocality
 
values() - Static method in enum org.apache.spark.sql.SaveMode
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Method in class org.apache.spark.sql.sources.In
 
values() - Static method in enum org.apache.spark.status.api.v1.ApplicationStatus
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.spark.status.api.v1.StageStatus
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.spark.status.api.v1.streaming.BatchStatus
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.spark.status.api.v1.TaskSorting
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
values() - Static method in enum org.apache.spark.streaming.StreamingContextState
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in class org.apache.spark.TaskState
 
values() - Static method in enum org.apache.spark.util.sketch.BloomFilter.Version
Returns an array containing the constants of this enum type, in the order they are declared.
values() - Static method in enum org.apache.spark.util.sketch.CountMinSketch.Version
Returns an array containing the constants of this enum type, in the order they are declared.
valueType() - Method in class org.apache.spark.sql.types.MapType
 
var_pop(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the population variance of the values in a group.
var_pop(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the population variance of the values in a group.
var_samp(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the unbiased variance of the values in a group.
var_samp(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: returns the unbiased variance of the values in a group.
VarcharType - Class in org.apache.spark.sql.types
Hive varchar type.
VarcharType(int) - Constructor for class org.apache.spark.sql.types.VarcharType
 
variance() - Method in class org.apache.spark.api.java.JavaDoubleRDD
Compute the population variance of this RDD's elements.
variance(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Binomial$
 
variance(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gamma$
 
variance(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Gaussian$
 
variance(double) - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegression.Poisson$
 
variance(Column, Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
variance(Column) - Static method in class org.apache.spark.ml.stat.Summarizer
 
variance() - Method in class org.apache.spark.mllib.stat.MultivariateOnlineSummarizer
Unbiased estimate of sample variance of each dimension.
variance() - Method in interface org.apache.spark.mllib.stat.MultivariateStatisticalSummary
Sample variance vector.
Variance - Class in org.apache.spark.mllib.tree.impurity
Class for calculating variance during regression
Variance() - Constructor for class org.apache.spark.mllib.tree.impurity.Variance
 
variance() - Method in class org.apache.spark.rdd.DoubleRDDFunctions
Compute the population variance of this RDD's elements.
variance(Column) - Static method in class org.apache.spark.sql.functions
Aggregate function: alias for var_samp.
variance(String) - Static method in class org.apache.spark.sql.functions
Aggregate function: alias for var_samp.
variance() - Method in class org.apache.spark.util.StatCounter
Return the population variance of the values.
varianceCol() - Method in interface org.apache.spark.ml.param.shared.HasVarianceCol
Param for Column name for the biased sample variance of prediction.
varianceCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
varianceCol() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
variancePower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
variancePower() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
vClassTag() - Method in class org.apache.spark.api.java.JavaHadoopRDD
 
vClassTag() - Method in class org.apache.spark.api.java.JavaNewHadoopRDD
 
vClassTag() - Method in class org.apache.spark.api.java.JavaPairRDD
 
vClassTag() - Method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
vClassTag() - Method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
Vector - Interface in org.apache.spark.ml.linalg
Represents a numeric vector, whose index type is Int and value type is Double.
vector() - Method in class org.apache.spark.mllib.linalg.distributed.IndexedRow
 
Vector - Interface in org.apache.spark.mllib.linalg
Represents a numeric vector, whose index type is Int and value type is Double.
VectorAssembler - Class in org.apache.spark.ml.feature
A feature transformer that merges multiple columns into a vector column.
VectorAssembler(String) - Constructor for class org.apache.spark.ml.feature.VectorAssembler
 
VectorAssembler() - Constructor for class org.apache.spark.ml.feature.VectorAssembler
 
VectorAttributeRewriter - Class in org.apache.spark.ml.feature
Utility transformer that rewrites Vector attribute names via prefix replacement.
VectorAttributeRewriter(String, String, Map<String, String>) - Constructor for class org.apache.spark.ml.feature.VectorAttributeRewriter
 
VectorAttributeRewriter(String, Map<String, String>) - Constructor for class org.apache.spark.ml.feature.VectorAttributeRewriter
 
vectorCol() - Method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
VectorImplicits - Class in org.apache.spark.mllib.linalg
Implicit methods available in Scala for converting Vector to Vector and vice versa.
VectorImplicits() - Constructor for class org.apache.spark.mllib.linalg.VectorImplicits
 
VectorIndexer - Class in org.apache.spark.ml.feature
Class for indexing categorical feature columns in a dataset of Vector.
VectorIndexer(String) - Constructor for class org.apache.spark.ml.feature.VectorIndexer
 
VectorIndexer() - Constructor for class org.apache.spark.ml.feature.VectorIndexer
 
VectorIndexerModel - Class in org.apache.spark.ml.feature
Model fitted by VectorIndexer.
Vectors - Class in org.apache.spark.ml.linalg
Factory methods for Vector.
Vectors() - Constructor for class org.apache.spark.ml.linalg.Vectors
 
Vectors - Class in org.apache.spark.mllib.linalg
Factory methods for Vector.
Vectors() - Constructor for class org.apache.spark.mllib.linalg.Vectors
 
vectorSize() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
vectorSize() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
VectorSizeHint - Class in org.apache.spark.ml.feature
Experimental A feature transformer that adds size information to the metadata of a vector column.
VectorSizeHint(String) - Constructor for class org.apache.spark.ml.feature.VectorSizeHint
 
VectorSizeHint() - Constructor for class org.apache.spark.ml.feature.VectorSizeHint
 
VectorSlicer - Class in org.apache.spark.ml.feature
This class takes a feature vector and outputs a new feature vector with a subarray of the original features.
VectorSlicer(String) - Constructor for class org.apache.spark.ml.feature.VectorSlicer
 
VectorSlicer() - Constructor for class org.apache.spark.ml.feature.VectorSlicer
 
VectorTransformer - Interface in org.apache.spark.mllib.feature
Developer API Trait for transformation of a vector
VectorType() - Static method in class org.apache.spark.ml.linalg.SQLDataTypes
Data type for Vector.
vectorTypes(StructType, StructType, SQLConf) - Static method in class org.apache.spark.sql.hive.orc.OrcFileFormat
 
VectorUDT - Class in org.apache.spark.mllib.linalg
Alpha Component
VectorUDT() - Constructor for class org.apache.spark.mllib.linalg.VectorUDT
 
verboseString() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
verboseString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
verboseString() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
verboseString() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
verboseStringWithSuffix() - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
verboseStringWithSuffix() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
verboseStringWithSuffix() - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
verboseStringWithSuffix() - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
version() - Method in class org.apache.spark.api.java.JavaSparkContext
The version of Spark on which this application is running.
version() - Method in class org.apache.spark.io.SnappyCompressionCodec
 
version() - Method in class org.apache.spark.SparkContext
The version of Spark on which this application is running.
version() - Method in class org.apache.spark.sql.SparkSession
The version of Spark on which this application is running.
VersionInfo - Class in org.apache.spark.status.api.v1
 
VersionUtils - Class in org.apache.spark.util
Utilities for working with Spark version strings
VersionUtils() - Constructor for class org.apache.spark.util.VersionUtils
 
vertcat(Matrix[]) - Static method in class org.apache.spark.ml.linalg.Matrices
Vertically concatenate a sequence of matrices.
vertcat(Matrix[]) - Static method in class org.apache.spark.mllib.linalg.Matrices
Vertically concatenate a sequence of matrices.
vertexAttr(long) - Method in class org.apache.spark.graphx.EdgeTriplet
Get the vertex object for the given vertex in the edge.
VertexRDD<VD> - Class in org.apache.spark.graphx
Extends RDD[(VertexId, VD)] by ensuring that there is only one entry for each vertex and by pre-indexing the entries for fast, efficient joins.
VertexRDD(SparkContext, Seq<Dependency<?>>) - Constructor for class org.apache.spark.graphx.VertexRDD
 
VertexRDDImpl<VD> - Class in org.apache.spark.graphx.impl
 
vertices() - Method in class org.apache.spark.graphx.Graph
An RDD containing the vertices and their associated attributes.
vertices() - Method in class org.apache.spark.graphx.impl.GraphImpl
 
view() - Static method in class org.apache.spark.sql.types.StructType
 
view(int, int) - Static method in class org.apache.spark.sql.types.StructType
 
viewToSeq(KVStoreView<T>, int, Function1<T, Object>) - Static method in class org.apache.spark.status.KVUtils
Turns a KVStoreView into a Scala sequence, applying a filter.
visit(int, int, String, String, String, String[]) - Method in class org.apache.spark.util.InnerClosureFinder
 
visitMethod(int, String, String, String, String[]) - Method in class org.apache.spark.util.InnerClosureFinder
 
visitMethod(int, String, String, String, String[]) - Method in class org.apache.spark.util.ReturnStatementFinder
 
vizHeaderNodes() - Static method in class org.apache.spark.ui.UIUtils
 
vManifest() - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
vManifest() - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
vManifest() - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
vocabSize() - Static method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
vocabSize() - Method in class org.apache.spark.ml.clustering.LDAModel
 
vocabSize() - Static method in class org.apache.spark.ml.clustering.LocalLDAModel
 
vocabSize() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
vocabSize() - Static method in class org.apache.spark.ml.feature.CountVectorizerModel
 
vocabSize() - Method in class org.apache.spark.mllib.clustering.DistributedLDAModel
 
vocabSize() - Method in class org.apache.spark.mllib.clustering.LDAModel
Vocabulary size (number of terms or terms in the vocabulary)
vocabSize() - Method in class org.apache.spark.mllib.clustering.LocalLDAModel
 
vocabulary() - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
VocabWord - Class in org.apache.spark.mllib.feature
Entry in vocabulary
VocabWord(String, long, int[], int[], int) - Constructor for class org.apache.spark.mllib.feature.VocabWord
 
VoidFunction<T> - Interface in org.apache.spark.api.java.function
A function with no return value.
VoidFunction2<T1,T2> - Interface in org.apache.spark.api.java.function
A two-argument function that takes arguments of type T1 and T2 with no return value.
Vote() - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 

W

w(boolean) - Method in class org.apache.spark.ml.param.BooleanParam
Creates a param pair with the given value (for Java).
w(List<List<Double>>) - Method in class org.apache.spark.ml.param.DoubleArrayArrayParam
Creates a param pair with a `java.util.List` of values (for Java and Python).
w(List<Double>) - Method in class org.apache.spark.ml.param.DoubleArrayParam
Creates a param pair with a `java.util.List` of values (for Java and Python).
w(double) - Method in class org.apache.spark.ml.param.DoubleParam
Creates a param pair with the given value (for Java).
w(float) - Method in class org.apache.spark.ml.param.FloatParam
Creates a param pair with the given value (for Java).
w(List<Integer>) - Method in class org.apache.spark.ml.param.IntArrayParam
Creates a param pair with a `java.util.List` of values (for Java and Python).
w(int) - Method in class org.apache.spark.ml.param.IntParam
Creates a param pair with the given value (for Java).
w(long) - Method in class org.apache.spark.ml.param.LongParam
Creates a param pair with the given value (for Java).
w(T) - Method in class org.apache.spark.ml.param.Param
Creates a param pair with the given value (for Java).
w(List<String>) - Method in class org.apache.spark.ml.param.StringArrayParam
Creates a param pair with a `java.util.List` of values (for Java and Python).
waitUntilEmpty(long) - Method in class org.apache.spark.scheduler.AsyncEventQueue
For testing only.
warmUp(SparkContext) - Static method in class org.apache.spark.streaming.util.RawTextHelper
Warms up the SparkContext in master and slave by running tasks to force JIT kick in before real workload starts.
weakIntern(String) - Static method in class org.apache.spark.status.LiveEntityHelpers
String interning to reduce the memory usage.
weekofyear(Column) - Static method in class org.apache.spark.sql.functions
Extracts the week number as an integer from a given date/timestamp/string.
WeibullGenerator - Class in org.apache.spark.mllib.random
Developer API Generates i.i.d.
WeibullGenerator(double, double) - Constructor for class org.apache.spark.mllib.random.WeibullGenerator
 
weightCol() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
weightCol() - Static method in class org.apache.spark.ml.classification.LinearSVCModel
 
weightCol() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
weightCol() - Static method in class org.apache.spark.ml.classification.LogisticRegressionModel
 
weightCol() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
weightCol() - Static method in class org.apache.spark.ml.classification.NaiveBayesModel
 
weightCol() - Static method in class org.apache.spark.ml.classification.OneVsRest
 
weightCol() - Static method in class org.apache.spark.ml.classification.OneVsRestModel
 
weightCol() - Method in interface org.apache.spark.ml.param.shared.HasWeightCol
Param for weight column name.
weightCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
weightCol() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
 
weightCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
weightCol() - Static method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
weightCol() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
weightCol() - Static method in class org.apache.spark.ml.regression.LinearRegressionModel
 
weightedFalsePositiveRate() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns weighted false positive rate.
weightedFalsePositiveRate() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns weighted false positive rate
weightedFMeasure(double) - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns weighted averaged f-measure.
weightedFMeasure() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns weighted averaged f1-measure.
weightedFMeasure(double) - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns weighted averaged f-measure
weightedFMeasure() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns weighted averaged f1-measure
weightedPrecision() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns weighted averaged precision.
weightedPrecision() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns weighted averaged precision
weightedRecall() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns weighted averaged recall.
weightedRecall() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns weighted averaged recall (equals to precision, recall and f-measure)
weightedTruePositiveRate() - Method in interface org.apache.spark.ml.classification.LogisticRegressionSummary
Returns weighted true positive rate.
weightedTruePositiveRate() - Method in class org.apache.spark.mllib.evaluation.MulticlassMetrics
Returns weighted true positive rate (equals to precision, recall and f-measure)
weights() - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
weights() - Method in class org.apache.spark.ml.clustering.ExpectationAggregator
 
weights() - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
 
weights() - Method in class org.apache.spark.mllib.classification.impl.GLMClassificationModel.SaveLoadV1_0$.Data
 
weights() - Method in class org.apache.spark.mllib.classification.LogisticRegressionModel
 
weights() - Method in class org.apache.spark.mllib.classification.SVMModel
 
weights() - Method in class org.apache.spark.mllib.clustering.ExpectationSum
 
weights() - Method in class org.apache.spark.mllib.clustering.GaussianMixtureModel
 
weights() - Method in class org.apache.spark.mllib.regression.GeneralizedLinearModel
 
weights() - Method in class org.apache.spark.mllib.regression.impl.GLMRegressionModel.SaveLoadV1_0$.Data
 
weights() - Method in class org.apache.spark.mllib.regression.LassoModel
 
weights() - Method in class org.apache.spark.mllib.regression.LinearRegressionModel
 
weights() - Method in class org.apache.spark.mllib.regression.RidgeRegressionModel
 
WelchTTest - Class in org.apache.spark.mllib.stat.test
Performs Welch's 2-sample t-test.
WelchTTest() - Constructor for class org.apache.spark.mllib.stat.test.WelchTTest
 
when(Column, Object) - Method in class org.apache.spark.sql.Column
Evaluates a list of conditions and returns one of multiple possible result expressions.
when(Column, Object) - Static method in class org.apache.spark.sql.functions
Evaluates a list of conditions and returns one of multiple possible result expressions.
where(Column) - Method in class org.apache.spark.sql.Dataset
Filters rows using the given condition.
where(String) - Method in class org.apache.spark.sql.Dataset
Filters rows using the given SQL expression.
wholeTextFiles(String, int) - Method in class org.apache.spark.api.java.JavaSparkContext
Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI.
wholeTextFiles(String) - Method in class org.apache.spark.api.java.JavaSparkContext
Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI.
wholeTextFiles(String, int) - Method in class org.apache.spark.SparkContext
Read a directory of text files from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI.
width() - Method in class org.apache.spark.util.sketch.CountMinSketch
Width of this CountMinSketch.
Window - Class in org.apache.spark.sql.expressions
Utility functions for defining window in DataFrames.
window(Column, String, String, String) - Static method in class org.apache.spark.sql.functions
Bucketize rows into one or more time windows given a timestamp specifying column.
window(Column, String, String) - Static method in class org.apache.spark.sql.functions
Bucketize rows into one or more time windows given a timestamp specifying column.
window(Column, String) - Static method in class org.apache.spark.sql.functions
Generates tumbling time windows given a timestamp specifying column.
window(Duration) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.
window(Duration, Duration) - Method in class org.apache.spark.streaming.api.java.JavaDStream
Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.
window(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
window(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
window(Duration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream which is computed based on windowed batches of this DStream.
window(Duration, Duration) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
Return a new DStream which is computed based on windowed batches of this DStream.
window(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
window(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
window(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
window(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
window(Duration) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
window(Duration, Duration) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
window(Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.
window(Duration, Duration) - Method in class org.apache.spark.streaming.dstream.DStream
Return a new DStream in which each RDD contains all the elements in seen in a sliding window of time over this DStream.
windowsDrive() - Static method in class org.apache.spark.util.Utils
Pattern for matching a Windows drive, which contains only a single alphabet character.
windowSize() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
windowSize() - Static method in class org.apache.spark.ml.feature.Word2VecModel
 
WindowSpec - Class in org.apache.spark.sql.expressions
A window specification that defines the partitioning, ordering, and frame boundaries.
wipe() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 
withColumn(String, Column) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset by adding a column or replacing the existing column that has the same name.
withColumnRenamed(String, String) - Method in class org.apache.spark.sql.Dataset
Returns a new Dataset with a column renamed.
withComment(String) - Method in class org.apache.spark.sql.types.StructField
Updates the StructField with a new comment value.
withDummyCallSite(SparkContext, Function0<T>) - Static method in class org.apache.spark.util.Utils
To avoid calling Utils.getCallSite for every single RDD we create in the body, set a dummy call site that RDDs use instead.
withEdges(EdgeRDD<?>) - Method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
withEdges(EdgeRDD<?>) - Method in class org.apache.spark.graphx.VertexRDD
Prepares this VertexRDD for efficient joins with the given EdgeRDD.
withExtensions(Function1<SparkSessionExtensions, BoxedUnit>) - Method in class org.apache.spark.sql.SparkSession.Builder
Inject extensions into the SparkSession.
withFilter(Function1<A, Object>) - Static method in class org.apache.spark.sql.types.StructType
 
withHiveExternalCatalog(SparkContext) - Static method in class org.apache.spark.sql.hive.HiveUtils
 
withIndex(int) - Method in class org.apache.spark.ml.attribute.Attribute
Copy with a new index.
withIndex(int) - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
withIndex(int) - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
withIndex(int) - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
withIndex(int) - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
withMapStatuses(Function1<MapStatus[], T>) - Method in class org.apache.spark.ShuffleStatus
Helper function which provides thread-safe access to the mapStatuses array.
withMax(double) - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy with a new max value.
withMean() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
withMean() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
withMean() - Method in class org.apache.spark.mllib.feature.StandardScalerModel
 
withMetadata(Metadata) - Method in class org.apache.spark.sql.types.MetadataBuilder
Include the content of an existing Metadata instance.
withMin(double) - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy with a new min value.
withName(String) - Method in class org.apache.spark.ml.attribute.Attribute
Copy with a new name.
withName(String) - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
withName(String) - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
withName(String) - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
withName(String) - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
withName(String) - Static method in class org.apache.spark.mllib.tree.configuration.Algo
 
withName(String) - Static method in class org.apache.spark.mllib.tree.configuration.EnsembleCombiningStrategy
 
withName(String) - Static method in class org.apache.spark.mllib.tree.configuration.FeatureType
 
withName(String) - Static method in class org.apache.spark.mllib.tree.configuration.QuantileStrategy
 
withName(String) - Static method in class org.apache.spark.rdd.CheckpointState
 
withName(String) - Static method in class org.apache.spark.rdd.DeterministicLevel
 
withName(String) - Static method in class org.apache.spark.scheduler.SchedulingMode
 
withName(String) - Static method in class org.apache.spark.scheduler.TaskLocality
 
withName(String) - Method in class org.apache.spark.sql.expressions.UserDefinedFunction
Updates UserDefinedFunction with a given name.
withName(String) - Static method in class org.apache.spark.streaming.scheduler.ReceiverState
 
withName(String) - Static method in class org.apache.spark.TaskState
 
withNewChildren(Seq<BaseType>) - Static method in class org.apache.spark.sql.hive.execution.CreateHiveTableAsSelectCommand
 
withNewChildren(Seq<BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveDirCommand
 
withNewChildren(Seq<BaseType>) - Static method in class org.apache.spark.sql.hive.execution.InsertIntoHiveTable
 
withNewChildren(Seq<BaseType>) - Static method in class org.apache.spark.sql.hive.execution.ScriptTransformationExec
 
withNumValues(int) - Method in class org.apache.spark.ml.attribute.NominalAttribute
Copy with a new numValues and empty values.
withoutIndex() - Method in class org.apache.spark.ml.attribute.Attribute
Copy without the index.
withoutIndex() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
withoutIndex() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
withoutIndex() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
withoutIndex() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
withoutMax() - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy without the max value.
withoutMin() - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy without the min value.
withoutName() - Method in class org.apache.spark.ml.attribute.Attribute
Copy without the name.
withoutName() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
 
withoutName() - Method in class org.apache.spark.ml.attribute.NominalAttribute
 
withoutName() - Method in class org.apache.spark.ml.attribute.NumericAttribute
 
withoutName() - Static method in class org.apache.spark.ml.attribute.UnresolvedAttribute
 
withoutNumValues() - Method in class org.apache.spark.ml.attribute.NominalAttribute
Copy without the numValues.
withoutSparsity() - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy without the sparsity.
withoutStd() - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy without the standard deviation.
withoutSummary() - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy without summary statistics.
withoutValues() - Method in class org.apache.spark.ml.attribute.BinaryAttribute
Copy without the values.
withoutValues() - Method in class org.apache.spark.ml.attribute.NominalAttribute
Copy without the values.
withPathFilter(double, SparkSession, long, Function0<T>) - Static method in class org.apache.spark.ml.image.SamplePathFilter
Sets the HDFS PathFilter flag and then restores it.
withPosition(Option<Object>, Option<Object>) - Method in exception org.apache.spark.sql.AnalysisException
 
withRecursiveFlag(boolean, SparkSession, Function0<T>) - Static method in class org.apache.spark.ml.image.RecursiveFlag
Sets the spark recursive flag and then restores it.
withSparsity(double) - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy with a new sparsity.
withStd(double) - Method in class org.apache.spark.ml.attribute.NumericAttribute
Copy with a new standard deviation.
withStd() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
withStd() - Static method in class org.apache.spark.ml.feature.StandardScalerModel
 
withStd() - Method in class org.apache.spark.mllib.feature.StandardScalerModel
 
withValues(String, String) - Method in class org.apache.spark.ml.attribute.BinaryAttribute
Copy with new values.
withValues(String, String...) - Method in class org.apache.spark.ml.attribute.NominalAttribute
Copy with new values and empty numValues.
withValues(String[]) - Method in class org.apache.spark.ml.attribute.NominalAttribute
Copy with new values and empty numValues.
withValues(String, Seq<String>) - Method in class org.apache.spark.ml.attribute.NominalAttribute
Copy with new values and empty numValues.
withWatermark(String, String) - Method in class org.apache.spark.sql.Dataset
Defines an event time watermark for this Dataset.
word() - Method in class org.apache.spark.mllib.feature.VocabWord
 
Word2Vec - Class in org.apache.spark.ml.feature
Word2Vec trains a model of Map(String, Vector), i.e.
Word2Vec(String) - Constructor for class org.apache.spark.ml.feature.Word2Vec
 
Word2Vec() - Constructor for class org.apache.spark.ml.feature.Word2Vec
 
Word2Vec - Class in org.apache.spark.mllib.feature
Word2Vec creates vector representation of words in a text corpus.
Word2Vec() - Constructor for class org.apache.spark.mllib.feature.Word2Vec
 
Word2VecModel - Class in org.apache.spark.ml.feature
Model fitted by Word2Vec.
Word2VecModel - Class in org.apache.spark.mllib.feature
Word2Vec model param: wordIndex maps each word to an index, which can retrieve the corresponding vector from wordVectors param: wordVectors array of length numWords * vectorSize, vector corresponding to the word mapped with index i can be retrieved by the slice (i * vectorSize, i * vectorSize + vectorSize)
Word2VecModel(Map<String, float[]>) - Constructor for class org.apache.spark.mllib.feature.Word2VecModel
 
Word2VecModel.Word2VecModelWriter$ - Class in org.apache.spark.ml.feature
 
Word2VecModelWriter$() - Constructor for class org.apache.spark.ml.feature.Word2VecModel.Word2VecModelWriter$
 
workerId() - Method in class org.apache.spark.scheduler.cluster.CoarseGrainedClusterMessages.RemoveWorker
 
Workspace(int) - Constructor for class org.apache.spark.mllib.optimization.NNLS.Workspace
 
wrap(Object, ObjectInspector, DataType) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
wrap(InternalRow, Function1<Object, Object>[], Object[], DataType[]) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
wrap(Seq<Object>, Function1<Object, Object>[], Object[], DataType[]) - Static method in class org.apache.spark.sql.hive.execution.HiveScriptIOSchema
 
wrapperClass() - Static method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
wrapperToFileSinkDesc(HiveShim.ShimFileSinkDesc) - Static method in class org.apache.spark.sql.hive.HiveShim
 
wrapRDD(RDD<Double>) - Method in class org.apache.spark.api.java.JavaDoubleRDD
 
wrapRDD(RDD<Tuple2<K, V>>) - Method in class org.apache.spark.api.java.JavaPairRDD
 
wrapRDD(RDD<T>) - Method in class org.apache.spark.api.java.JavaRDD
 
wrapRDD(RDD<T>) - Method in interface org.apache.spark.api.java.JavaRDDLike
 
wrapRDD(RDD<T>) - Method in class org.apache.spark.streaming.api.java.JavaDStream
 
wrapRDD(RDD<T>) - Method in interface org.apache.spark.streaming.api.java.JavaDStreamLike
 
wrapRDD(RDD<T>) - Static method in class org.apache.spark.streaming.api.java.JavaInputDStream
 
wrapRDD(RDD<Tuple2<K, V>>) - Method in class org.apache.spark.streaming.api.java.JavaPairDStream
 
wrapRDD(RDD<Tuple2<K, V>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairInputDStream
 
wrapRDD(RDD<Tuple2<K, V>>) - Static method in class org.apache.spark.streaming.api.java.JavaPairReceiverInputDStream
 
wrapRDD(RDD<T>) - Static method in class org.apache.spark.streaming.api.java.JavaReceiverInputDStream
 
write(Tuple2<K, V>) - Method in class org.apache.spark.internal.io.HadoopWriteConfigUtil
 
write(RDD<Tuple2<K, V>>, HadoopWriteConfigUtil<K, V>, ClassTag<V>) - Static method in class org.apache.spark.internal.io.SparkHadoopWriter
Basic work flow of this command is: 1.
write(int) - Method in class org.apache.spark.io.SnappyOutputStreamWrapper
 
write(byte[]) - Method in class org.apache.spark.io.SnappyOutputStreamWrapper
 
write(byte[], int, int) - Method in class org.apache.spark.io.SnappyOutputStreamWrapper
 
write() - Method in class org.apache.spark.ml.classification.DecisionTreeClassificationModel
 
write() - Static method in class org.apache.spark.ml.classification.DecisionTreeClassifier
 
write() - Method in class org.apache.spark.ml.classification.GBTClassificationModel
 
write() - Static method in class org.apache.spark.ml.classification.GBTClassifier
 
write() - Static method in class org.apache.spark.ml.classification.LinearSVC
 
write() - Method in class org.apache.spark.ml.classification.LinearSVCModel
 
write() - Static method in class org.apache.spark.ml.classification.LogisticRegression
 
write() - Method in class org.apache.spark.ml.classification.LogisticRegressionModel
Returns a MLWriter instance for this ML instance.
write() - Method in class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
 
write() - Static method in class org.apache.spark.ml.classification.MultilayerPerceptronClassifier
 
write() - Static method in class org.apache.spark.ml.classification.NaiveBayes
 
write() - Method in class org.apache.spark.ml.classification.NaiveBayesModel
 
write() - Method in class org.apache.spark.ml.classification.OneVsRest
 
write() - Method in class org.apache.spark.ml.classification.OneVsRestModel
 
write() - Method in class org.apache.spark.ml.classification.RandomForestClassificationModel
 
write() - Static method in class org.apache.spark.ml.classification.RandomForestClassifier
 
write() - Static method in class org.apache.spark.ml.clustering.BisectingKMeans
 
write() - Method in class org.apache.spark.ml.clustering.BisectingKMeansModel
 
write() - Method in class org.apache.spark.ml.clustering.DistributedLDAModel
 
write() - Static method in class org.apache.spark.ml.clustering.GaussianMixture
 
write() - Method in class org.apache.spark.ml.clustering.GaussianMixtureModel
Returns a MLWriter instance for this ML instance.
write() - Static method in class org.apache.spark.ml.clustering.KMeans
 
write() - Method in class org.apache.spark.ml.clustering.KMeansModel
Returns a MLWriter instance for this ML instance.
write() - Static method in class org.apache.spark.ml.clustering.LDA
 
write() - Method in class org.apache.spark.ml.clustering.LocalLDAModel
 
write() - Static method in class org.apache.spark.ml.evaluation.BinaryClassificationEvaluator
 
write() - Static method in class org.apache.spark.ml.evaluation.ClusteringEvaluator
 
write() - Static method in class org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator
 
write() - Static method in class org.apache.spark.ml.evaluation.RegressionEvaluator
 
write() - Static method in class org.apache.spark.ml.feature.Binarizer
 
write() - Static method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSH
 
write() - Method in class org.apache.spark.ml.feature.BucketedRandomProjectionLSHModel
 
write() - Method in class org.apache.spark.ml.feature.Bucketizer
 
write() - Static method in class org.apache.spark.ml.feature.ChiSqSelector
 
write() - Method in class org.apache.spark.ml.feature.ChiSqSelectorModel
 
write() - Method in class org.apache.spark.ml.feature.ColumnPruner
 
write() - Static method in class org.apache.spark.ml.feature.CountVectorizer
 
write() - Method in class org.apache.spark.ml.feature.CountVectorizerModel
 
write() - Static method in class org.apache.spark.ml.feature.DCT
 
write() - Static method in class org.apache.spark.ml.feature.ElementwiseProduct
 
write() - Static method in class org.apache.spark.ml.feature.FeatureHasher
 
write() - Static method in class org.apache.spark.ml.feature.HashingTF
 
write() - Static method in class org.apache.spark.ml.feature.IDF
 
write() - Method in class org.apache.spark.ml.feature.IDFModel
 
write() - Static method in class org.apache.spark.ml.feature.Imputer
 
write() - Method in class org.apache.spark.ml.feature.ImputerModel
 
write() - Static method in class org.apache.spark.ml.feature.IndexToString
 
write() - Static method in class org.apache.spark.ml.feature.Interaction
 
write() - Static method in class org.apache.spark.ml.feature.MaxAbsScaler
 
write() - Method in class org.apache.spark.ml.feature.MaxAbsScalerModel
 
write() - Static method in class org.apache.spark.ml.feature.MinHashLSH
 
write() - Method in class org.apache.spark.ml.feature.MinHashLSHModel
 
write() - Static method in class org.apache.spark.ml.feature.MinMaxScaler
 
write() - Method in class org.apache.spark.ml.feature.MinMaxScalerModel
 
write() - Static method in class org.apache.spark.ml.feature.NGram
 
write() - Static method in class org.apache.spark.ml.feature.Normalizer
 
write() - Static method in class org.apache.spark.ml.feature.OneHotEncoder
Deprecated.
 
write() - Static method in class org.apache.spark.ml.feature.OneHotEncoderEstimator
 
write() - Method in class org.apache.spark.ml.feature.OneHotEncoderModel
 
write() - Static method in class org.apache.spark.ml.feature.PCA
 
write() - Method in class org.apache.spark.ml.feature.PCAModel
 
write() - Static method in class org.apache.spark.ml.feature.PolynomialExpansion
 
write() - Method in class org.apache.spark.ml.feature.QuantileDiscretizer
 
write() - Static method in class org.apache.spark.ml.feature.RegexTokenizer
 
write() - Static method in class org.apache.spark.ml.feature.RFormula
 
write() - Method in class org.apache.spark.ml.feature.RFormulaModel
 
write() - Static method in class org.apache.spark.ml.feature.SQLTransformer
 
write() - Static method in class org.apache.spark.ml.feature.StandardScaler
 
write() - Method in class org.apache.spark.ml.feature.StandardScalerModel
 
write() - Static method in class org.apache.spark.ml.feature.StopWordsRemover
 
write() - Static method in class org.apache.spark.ml.feature.StringIndexer
 
write() - Method in class org.apache.spark.ml.feature.StringIndexerModel
 
write() - Static method in class org.apache.spark.ml.feature.Tokenizer
 
write() - Static method in class org.apache.spark.ml.feature.VectorAssembler
 
write() - Method in class org.apache.spark.ml.feature.VectorAttributeRewriter
 
write() - Static method in class org.apache.spark.ml.feature.VectorIndexer
 
write() - Method in class org.apache.spark.ml.feature.VectorIndexerModel
 
write() - Static method in class org.apache.spark.ml.feature.VectorSizeHint
 
write() - Static method in class org.apache.spark.ml.feature.VectorSlicer
 
write() - Static method in class org.apache.spark.ml.feature.Word2Vec
 
write() - Method in class org.apache.spark.ml.feature.Word2VecModel
 
write() - Static method in class org.apache.spark.ml.fpm.FPGrowth
 
write() - Method in class org.apache.spark.ml.fpm.FPGrowthModel
 
write() - Method in class org.apache.spark.ml.Pipeline
 
write() - Method in class org.apache.spark.ml.PipelineModel
 
write() - Static method in class org.apache.spark.ml.recommendation.ALS
 
write() - Method in class org.apache.spark.ml.recommendation.ALSModel
 
write() - Static method in class org.apache.spark.ml.regression.AFTSurvivalRegression
 
write() - Method in class org.apache.spark.ml.regression.AFTSurvivalRegressionModel
 
write() - Method in class org.apache.spark.ml.regression.DecisionTreeRegressionModel
 
write() - Static method in class org.apache.spark.ml.regression.DecisionTreeRegressor
 
write() - Method in class org.apache.spark.ml.regression.GBTRegressionModel
 
write() - Static method in class org.apache.spark.ml.regression.GBTRegressor
 
write() - Static method in class org.apache.spark.ml.regression.GeneralizedLinearRegression
 
write() - Method in class org.apache.spark.ml.regression.GeneralizedLinearRegressionModel
Returns a MLWriter instance for this ML instance.
write() - Static method in class org.apache.spark.ml.regression.IsotonicRegression
 
write() - Method in class org.apache.spark.ml.regression.IsotonicRegressionModel
 
write() - Static method in class org.apache.spark.ml.regression.LinearRegression
 
write() - Method in class org.apache.spark.ml.regression.LinearRegressionModel
Returns a MLWriter instance for this ML instance.
write() - Method in class org.apache.spark.ml.regression.RandomForestRegressionModel
 
write() - Static method in class org.apache.spark.ml.regression.RandomForestRegressor
 
write() - Method in class org.apache.spark.ml.tuning.CrossValidator
 
write() - Method in class org.apache.spark.ml.tuning.CrossValidatorModel
 
write() - Method in class org.apache.spark.ml.tuning.TrainValidationSplit
 
write() - Method in class org.apache.spark.ml.tuning.TrainValidationSplitModel
 
write() - Method in interface org.apache.spark.ml.util.DefaultParamsWritable
 
write() - Method in interface org.apache.spark.ml.util.MLWritable
Returns an MLWriter instance for this ML instance.
write(Kryo, Output, Iterable<?>) - Method in class org.apache.spark.serializer.JavaIterableWrapperSerializer
 
write() - Method in class org.apache.spark.sql.Dataset
Interface for saving the content of the non-streaming Dataset out into external storage.
write(InternalRow) - Method in class org.apache.spark.sql.hive.execution.HiveOutputWriter
 
write(T) - Method in interface org.apache.spark.sql.sources.v2.writer.DataWriter
Writes one record.
write(ByteBuffer) - Method in class org.apache.spark.storage.CountingWritableChannel
 
write(int) - Method in class org.apache.spark.storage.TimeTrackingOutputStream
 
write(byte[]) - Method in class org.apache.spark.storage.TimeTrackingOutputStream
 
write(byte[], int, int) - Method in class org.apache.spark.storage.TimeTrackingOutputStream
 
write(ByteBuffer, long) - Method in class org.apache.spark.streaming.util.WriteAheadLog
Write the record to the log and return a record handle, which contains all the information necessary to read back the written record.
WRITE_TIME() - Method in class org.apache.spark.InternalAccumulator.shuffleWrite$
 
WriteAheadLog - Class in org.apache.spark.streaming.util
Developer API This abstract class represents a write ahead log (aka journal) that is used by Spark Streaming to save the received data (by receivers) and associated metadata to a reliable storage, so that they can be recovered after driver failures.
WriteAheadLog() - Constructor for class org.apache.spark.streaming.util.WriteAheadLog
 
WriteAheadLogRecordHandle - Class in org.apache.spark.streaming.util
Developer API This abstract class represents a handle that refers to a record written in a WriteAheadLog.
WriteAheadLogRecordHandle() - Constructor for class org.apache.spark.streaming.util.WriteAheadLogRecordHandle
 
WriteAheadLogUtils - Class in org.apache.spark.streaming.util
A helper class with utility functions related to the WriteAheadLog interface
WriteAheadLogUtils() - Constructor for class org.apache.spark.streaming.util.WriteAheadLogUtils
 
writeAll(Iterator<T>, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializationStream
 
writeBoolean(DataOutputStream, boolean) - Static method in class org.apache.spark.api.r.SerDe
 
writeBooleanArr(DataOutputStream, boolean[]) - Static method in class org.apache.spark.api.r.SerDe
 
writeByteBuffer(ByteBuffer, DataOutput) - Static method in class org.apache.spark.util.Utils
Primitive often used when writing ByteBuffer to DataOutput
writeByteBuffer(ByteBuffer, OutputStream) - Static method in class org.apache.spark.util.Utils
Primitive often used when writing ByteBuffer to OutputStream
writeBytes(DataOutputStream, byte[]) - Static method in class org.apache.spark.api.r.SerDe
 
writeBytes() - Method in class org.apache.spark.status.api.v1.ShuffleWriteMetricDistributions
 
writeDate(DataOutputStream, Date) - Static method in class org.apache.spark.api.r.SerDe
 
writeDouble(DataOutputStream, double) - Static method in class org.apache.spark.api.r.SerDe
 
writeDoubleArr(DataOutputStream, double[]) - Static method in class org.apache.spark.api.r.SerDe
 
writeExternal(ObjectOutput) - Method in class org.apache.spark.serializer.JavaSerializer
 
writeExternal(ObjectOutput) - Method in class org.apache.spark.storage.BlockManagerId
 
writeExternal(ObjectOutput) - Method in class org.apache.spark.storage.BlockManagerMessages.UpdateBlockInfo
 
writeExternal(ObjectOutput) - Method in class org.apache.spark.storage.StorageLevel
 
writeInt(DataOutputStream, int) - Static method in class org.apache.spark.api.r.SerDe
 
writeIntArr(DataOutputStream, int[]) - Static method in class org.apache.spark.api.r.SerDe
 
writeJObj(DataOutputStream, Object, JVMObjectTracker) - Static method in class org.apache.spark.api.r.SerDe
 
writeKey(T, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializationStream
Writes the object representing the key of a key-value pair.
writeObject(DataOutputStream, Object, JVMObjectTracker) - Static method in class org.apache.spark.api.r.SerDe
 
writeObject(T, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializationStream
The most general-purpose method to write an object.
WriterCommitMessage - Interface in org.apache.spark.sql.sources.v2.writer
A commit message returned by DataWriter.commit() and will be sent back to the driver side as the input parameter of DataSourceWriter.commit(WriterCommitMessage[]).
writeRecords() - Method in class org.apache.spark.status.api.v1.ShuffleWriteMetricDistributions
 
writeSqlObject(DataOutputStream, Object) - Static method in class org.apache.spark.sql.api.r.SQLUtils
 
writeStream() - Method in class org.apache.spark.sql.Dataset
Interface for saving the content of the streaming Dataset out into external storage.
writeString(DataOutputStream, String) - Static method in class org.apache.spark.api.r.SerDe
 
writeStringArr(DataOutputStream, String[]) - Static method in class org.apache.spark.api.r.SerDe
 
WriteSupport - Interface in org.apache.spark.sql.sources.v2
A mix-in interface for DataSourceV2.
writeTime(DataOutputStream, Time) - Static method in class org.apache.spark.api.r.SerDe
 
writeTime(DataOutputStream, Timestamp) - Static method in class org.apache.spark.api.r.SerDe
 
writeTime() - Method in class org.apache.spark.status.api.v1.ShuffleWriteMetricDistributions
 
writeTime() - Method in class org.apache.spark.status.api.v1.ShuffleWriteMetrics
 
writeTo(OutputStream) - Method in class org.apache.spark.util.sketch.BloomFilter
Writes out this BloomFilter to an output stream in binary format.
writeTo(OutputStream) - Method in class org.apache.spark.util.sketch.CountMinSketch
Writes out this CountMinSketch to an output stream in binary format.
writeType(DataOutputStream, String) - Static method in class org.apache.spark.api.r.SerDe
 
writeValue(T, ClassTag<T>) - Method in class org.apache.spark.serializer.SerializationStream
Writes the object representing the value of a key-value pair.

X

x() - Method in class org.apache.spark.mllib.optimization.NNLS.Workspace
 

Y

year(Column) - Static method in class org.apache.spark.sql.functions
Extracts the year as an integer from a given date/timestamp/string.

Z

zero() - Method in class org.apache.spark.Accumulable
Deprecated.
 
zero(R) - Method in interface org.apache.spark.AccumulableParam
Deprecated.
Return the "zero" (identity) value for an accumulator type, given its initial value.
zero() - Static method in class org.apache.spark.Accumulator
Deprecated.
 
zero(double) - Method in class org.apache.spark.AccumulatorParam.DoubleAccumulatorParam$
Deprecated.
 
zero(float) - Method in class org.apache.spark.AccumulatorParam.FloatAccumulatorParam$
Deprecated.
 
zero(int) - Method in class org.apache.spark.AccumulatorParam.IntAccumulatorParam$
Deprecated.
 
zero(long) - Method in class org.apache.spark.AccumulatorParam.LongAccumulatorParam$
Deprecated.
 
zero(String) - Method in class org.apache.spark.AccumulatorParam.StringAccumulatorParam$
Deprecated.
 
zero(int, int) - Static method in class org.apache.spark.mllib.clustering.ExpectationSum
 
zero() - Method in class org.apache.spark.sql.expressions.Aggregator
A zero value for this aggregation.
zeros(int, int) - Static method in class org.apache.spark.ml.linalg.DenseMatrix
Generate a DenseMatrix consisting of zeros.
zeros(int, int) - Static method in class org.apache.spark.ml.linalg.Matrices
Generate a Matrix consisting of zeros.
zeros(int) - Static method in class org.apache.spark.ml.linalg.Vectors
Creates a vector of all zeros.
zeros(int, int) - Static method in class org.apache.spark.mllib.linalg.DenseMatrix
Generate a DenseMatrix consisting of zeros.
zeros(int, int) - Static method in class org.apache.spark.mllib.linalg.Matrices
Generate a Matrix consisting of zeros.
zeros(int) - Static method in class org.apache.spark.mllib.linalg.Vectors
Creates a vector of all zeros.
zip(JavaRDDLike<U, ?>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
zip(JavaRDDLike<U, ?>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
zip(JavaRDDLike<U, ?>) - Static method in class org.apache.spark.api.java.JavaRDD
 
zip(JavaRDDLike<U, ?>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Zips this RDD with another one, returning key-value pairs with the first element in each RDD, second element in each RDD, etc.
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.api.r.RRDD
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.graphx.VertexRDD
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zip(RDD<U>, ClassTag<U>) - Method in class org.apache.spark.rdd.RDD
Zips this RDD with another one, returning key-value pairs with the first element in each RDD, second element in each RDD, etc.
zip(RDD<U>, ClassTag<U>) - Static method in class org.apache.spark.rdd.UnionRDD
 
zip(GenIterable<B>, CanBuildFrom<Repr, Tuple2<A1, B>, That>) - Static method in class org.apache.spark.sql.types.StructType
 
zipAll(GenIterable<B>, A1, B, CanBuildFrom<Repr, Tuple2<A1, B>, That>) - Static method in class org.apache.spark.sql.types.StructType
 
zipPartitions(JavaRDDLike<U, ?>, FlatMapFunction2<Iterator<T>, Iterator<U>, V>) - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
zipPartitions(JavaRDDLike<U, ?>, FlatMapFunction2<Iterator<T>, Iterator<U>, V>) - Static method in class org.apache.spark.api.java.JavaPairRDD
 
zipPartitions(JavaRDDLike<U, ?>, FlatMapFunction2<Iterator<T>, Iterator<U>, V>) - Static method in class org.apache.spark.api.java.JavaRDD
 
zipPartitions(JavaRDDLike<U, ?>, FlatMapFunction2<Iterator<T>, Iterator<U>, V>) - Method in interface org.apache.spark.api.java.JavaRDDLike
Zip this RDD's partitions with one (or more) RDD(s) and return a new RDD by applying a function to the zipped partitions.
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.api.r.RRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.api.r.RRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.api.r.RRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.api.r.RRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.api.r.RRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.api.r.RRDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.VertexRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.graphx.VertexRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.VertexRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.graphx.VertexRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.VertexRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.graphx.VertexRDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Method in class org.apache.spark.rdd.RDD
Zip this RDD's partitions with one (or more) RDD(s) and return a new RDD by applying a function to the zipped partitions.
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Method in class org.apache.spark.rdd.RDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Method in class org.apache.spark.rdd.RDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Method in class org.apache.spark.rdd.RDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Method in class org.apache.spark.rdd.RDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Method in class org.apache.spark.rdd.RDD
 
zipPartitions(RDD<B>, boolean, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.UnionRDD
 
zipPartitions(RDD<B>, Function2<Iterator<T>, Iterator<B>, Iterator<V>>, ClassTag<B>, ClassTag<V>) - Static method in class org.apache.spark.rdd.UnionRDD
 
zipPartitions(RDD<B>, RDD<C>, boolean, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.UnionRDD
 
zipPartitions(RDD<B>, RDD<C>, Function3<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<V>) - Static method in class org.apache.spark.rdd.UnionRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, boolean, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.UnionRDD
 
zipPartitions(RDD<B>, RDD<C>, RDD<D>, Function4<Iterator<T>, Iterator<B>, Iterator<C>, Iterator<D>, Iterator<V>>, ClassTag<B>, ClassTag<C>, ClassTag<D>, ClassTag<V>) - Static method in class org.apache.spark.rdd.UnionRDD
 
zipWithIndex() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
zipWithIndex() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
zipWithIndex() - Static method in class org.apache.spark.api.java.JavaRDD
 
zipWithIndex() - Method in interface org.apache.spark.api.java.JavaRDDLike
Zips this RDD with its element indices.
zipWithIndex() - Static method in class org.apache.spark.api.r.RRDD
 
zipWithIndex() - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipWithIndex() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipWithIndex() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipWithIndex() - Static method in class org.apache.spark.graphx.VertexRDD
 
zipWithIndex() - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipWithIndex() - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipWithIndex() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipWithIndex() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipWithIndex() - Method in class org.apache.spark.rdd.RDD
Zips this RDD with its element indices.
zipWithIndex() - Static method in class org.apache.spark.rdd.UnionRDD
 
zipWithIndex(CanBuildFrom<Repr, Tuple2<A1, Object>, That>) - Static method in class org.apache.spark.sql.types.StructType
 
zipWithUniqueId() - Static method in class org.apache.spark.api.java.JavaDoubleRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.api.java.JavaPairRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.api.java.JavaRDD
 
zipWithUniqueId() - Method in interface org.apache.spark.api.java.JavaRDDLike
Zips this RDD with generated unique Long ids.
zipWithUniqueId() - Static method in class org.apache.spark.api.r.RRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.graphx.EdgeRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.graphx.impl.EdgeRDDImpl
 
zipWithUniqueId() - Static method in class org.apache.spark.graphx.impl.VertexRDDImpl
 
zipWithUniqueId() - Static method in class org.apache.spark.graphx.VertexRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.rdd.HadoopRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.rdd.JdbcRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.rdd.NewHadoopRDD
 
zipWithUniqueId() - Static method in class org.apache.spark.rdd.PartitionPruningRDD
 
zipWithUniqueId() - Method in class org.apache.spark.rdd.RDD
Zips this RDD with generated unique Long ids.
zipWithUniqueId() - Static method in class org.apache.spark.rdd.UnionRDD
 
ZStdCompressionCodec - Class in org.apache.spark.io
Developer API ZStandard implementation of CompressionCodec.
ZStdCompressionCodec(SparkConf) - Constructor for class org.apache.spark.io.ZStdCompressionCodec
 

_

_1() - Method in class org.apache.spark.util.MutablePair
 
_2() - Method in class org.apache.spark.util.MutablePair
 
$ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ 
Skip navigation links