Package org.apache.spark.mllib.clustering

Class LocalLDAModel

Object
org.apache.spark.mllib.clustering.LDAModel
org.apache.spark.mllib.clustering.LocalLDAModel
All Implemented Interfaces:
Serializable, Saveable
public class LocalLDAModel extends LDAModel implements Serializable
Local LDA model. This model stores only the inferred topics.

param: topics Inferred topics (vocabSize x k matrix).

See Also:

  • Method Details

    • load

      public static LocalLDAModel load(SparkContext sc, String path)

    • topics

      public Matrix topics()

    • docConcentration

      public Vector docConcentration()
      Description copied from class: LDAModel
      Concentration parameter (commonly named "alpha") for the prior placed on documents' distributions over topics ("theta").

      This is the parameter to a Dirichlet distribution.

      Specified by:
      docConcentration in class LDAModel
      Returns:
      (undocumented)

    • topicConcentration

      public double topicConcentration()
      Description copied from class: LDAModel
      Concentration parameter (commonly named "beta" or "eta") for the prior placed on topics' distributions over terms.

      This is the parameter to a symmetric Dirichlet distribution.

      Specified by:
      topicConcentration in class LDAModel
      Returns:
      (undocumented)

    • k

      public int k()
      Description copied from class: LDAModel
      Number of topics
      Specified by:
      k in class LDAModel

    • vocabSize

      public int vocabSize()
      Description copied from class: LDAModel
      Vocabulary size (number of terms or terms in the vocabulary)
      Specified by:
      vocabSize in class LDAModel

    • topicsMatrix

      public Matrix topicsMatrix()
      Description copied from class: LDAModel
      Inferred topics, where each topic is represented by a distribution over terms. This is a matrix of size vocabSize x k, where each column is a topic. No guarantees are given about the ordering of the topics.
      Specified by:
      topicsMatrix in class LDAModel
      Returns:
      (undocumented)

    • describeTopics

      public scala.Tuple2<int[],double[]>[] describeTopics(int maxTermsPerTopic)
      Description copied from class: LDAModel
      Return the topics described by weighted terms.

      Specified by:
      describeTopics in class LDAModel
      Parameters:
      maxTermsPerTopic - Maximum number of terms to collect for each topic.
      Returns:
      Array over topics. Each topic is represented as a pair of matching arrays: (term indices, term weights in topic). Each topic's terms are sorted in order of decreasing weight.

    • getSeed

      public long getSeed()
      Random seed for cluster initialization.
      Returns:
      (undocumented)

    • setSeed

      public LocalLDAModel setSeed(long seed)
      Set the random seed for cluster initialization.
      Parameters:
      seed - (undocumented)
      Returns:
      (undocumented)

    • save

      public void save(SparkContext sc, String path)
      Description copied from interface: Saveable
      Save this model to the given path.

      This saves: - human-readable (JSON) model metadata to path/metadata/ - Parquet formatted data to path/data/

      The model may be loaded using Loader.load.

      Specified by:
      save in interface Saveable
      Parameters:
      sc - Spark context used to save model data.
      path - Path specifying the directory in which to save this model. If the directory already exists, this method throws an exception.

    • logLikelihood

      public double logLikelihood(RDD<scala.Tuple2<Object,Vector>> documents)
      Calculates a lower bound on the log likelihood of the entire corpus.

      See Equation (16) in original Online LDA paper.

      Parameters:
      documents - test corpus to use for calculating log likelihood
      Returns:
      variational lower bound on the log likelihood of the entire corpus

    • logLikelihood

      public double logLikelihood(JavaPairRDD<Long,Vector> documents)
      Java-friendly version of logLikelihood
      Parameters:
      documents - (undocumented)
      Returns:
      (undocumented)

    • logPerplexity

      public double logPerplexity(RDD<scala.Tuple2<Object,Vector>> documents)
      Calculate an upper bound on perplexity. (Lower is better.) See Equation (16) in original Online LDA paper.

      Parameters:
      documents - test corpus to use for calculating perplexity
      Returns:
      Variational upper bound on log perplexity per token.

    • logPerplexity

      public double logPerplexity(JavaPairRDD<Long,Vector> documents)
      Java-friendly version of logPerplexity
      Parameters:
      documents - (undocumented)
      Returns:
      (undocumented)

    • topicDistributions

      public RDD<scala.Tuple2<Object,Vector>> topicDistributions(RDD<scala.Tuple2<Object,Vector>> documents)
      Predicts the topic mixture distribution for each document (often called "theta" in the literature). Returns a vector of zeros for an empty document.

      This uses a variational approximation following Hoffman et al. (2010), where the approximate distribution is called "gamma." Technically, this method returns this approximation "gamma" for each document.

      Parameters:
      documents - documents to predict topic mixture distributions for
      Returns:
      An RDD of (document ID, topic mixture distribution for document)

    • topicDistribution

      public Vector topicDistribution(Vector document)
      Predicts the topic mixture distribution for a document (often called "theta" in the literature). Returns a vector of zeros for an empty document.

      Note this means to allow quick query for single document. For batch documents, please refer to topicDistributions() to avoid overhead.

      Parameters:
      document - document to predict topic mixture distributions for
      Returns:
      topic mixture distribution for the document

    • topicDistributions

      public JavaPairRDD<Long,Vector> topicDistributions(JavaPairRDD<Long,Vector> documents)
      Java-friendly version of topicDistributions
      Parameters:
      documents - (undocumented)
      Returns:
      (undocumented)