Jekyll20221201T16:53:4708:00https://joshnguyen.net/feed.xmlJosh Nguyenpersonal descriptionJosh Nguyenjosh.nguyen@anu.edu.auPageRank, Stochastic Matrices and the Power Iteration20221026T00:00:0007:0020221026T00:00:0007:00https://joshnguyen.net/posts/pagerank<p>In this post, we will revisit a popular algorithm called <a href="https://en.wikipedia.org/wiki/PageRank">PageRank</a>, which is used by Google to rank webpages for its search engine. Surprising to some but not so to others, PageRank is simple enough that only a level of firstyear undergraduate linear algebra is required to understand it.</p>
<h2 id="thewebasagraph">The Web as a Graph</h2>
<p>Consider the web as a collection of <em>pages</em>, some of which are connected to each other using <em>hyperlinks</em>. For example, the Wikipedia article on <a href="https://en.wikipedia.org/wiki/General_relativity">general relativity</a> contains a hyperlink to another article on <a href="https://en.wikipedia.org/wiki/Albert_Einstein">Albert Einstein</a>. By clicking the link, we move from the former webpage to the latter.</p>
<p>We can model this as a graph \(G = (V, E)\), where the set of nodes (or vertices) \(V\) contains the webpages and the set of edges \(E\) contains binary relations \((v_i, v_j)\), indicating that the page \(v_i \in V\) contains a hyperlink to \(v_j \in E\). Since \(v_i\) may lead to \(v_j\) but not the other way around, the edge \((v_i, v_j)\) may be in \(E\) while \((v_j, v_i)\) may not. In this case we call the graph \(G\) a <em>directed</em> graph.</p>
<h2 id="theimportanceofapage">The Importance of a Page</h2>
<p>PageRank defines a score for each webpage where more “important” pages have high scores. This is particularly useful in <em>information retrieval</em>, where a system is asked to return pages relevant to a query. An assumption is that higherranked pages should be returned first, as they are more important and therefore have a higher chance of being what the user wants. Some other intuitions on building a ranking system are:</p>
<ul>
<li>If many pages have a hyperlink to page \(i\), then \(i\) should be important.</li>
<li>If a highly ranked page links to page \(i\), then \(i\) should also be highly ranked.</li>
</ul>
<p>Let \(r \in \mathbb{R}^n\) be the rank vector—that is, \(r_i\) is the numerical value denoting the importance of page \(i\). We will propose a method for finding \(r\) such that if \(r_i > r_j\), then page \(i\) is more important than page \(j\). Note that since the rankings are ordinal, we can just scale \(r\) by a positive number and the relative ordering of the pages based on importance will not change at all.</p>
<h3 id="theimportancematrix">The importance matrix</h3>
<p>To find the importance of every page, we will need to exploit the structure of the graph, specifically the in and outlinks of every node. Suppose that page \(i\) with importance \(r_i\) has \(d_i\) outneighbors—that is, pages that \(i\) links to. In graph theory, \(d_i\) is also called the <em>outdegree</em> of \(i\). Based on the intuitions above, we want these outneighbors to enjoy \(i\)’s importance. To do so, we assume that each outneighbor of \(i\) will get an equal amount of importance from \(i\). In other words, each outneighbor will get an amount \(\frac{r_i}{d_i}\) of importance from \(i\).</p>
<p>In this setting, the importance of a page \(j\) will be the sum of all importance flowing into it from its inneighbors:</p>
\[\begin{align} \label{eq:importance_flow}
r_j = \sum_{i \rightarrow j} \frac{r_i}{d_i}.
\end{align}\]
<p>Notice that we have a recursive structure: Every page influences the pages it leads to. But the importance of that page is flows from the pages leading to it.</p>
<p>Define a matrix \(A\), called the <em>importance matrix</em>, where \(A_{j, i} = \frac{1}{d_i}\) if page \(i\) leads to page \(j\). In other words, each column of \(i\) of \(A\) is a vector containing either \(0\) (where there is no outgoing edge) or \(\frac{1}{d_i}\) (when there is). Since the outdegree of \(i\) is exactly \(d_i\), it must be the case that every column of \(i\) sums to \(1\).</p>
<p>The product \(A r\) gives us the importance flowing into every page. To see why it is, consider the \(j\)th component of this product:</p>
\[\begin{align*}
(A r)_j = \sum_{i=1}^{n} A_{j, i} r_{i} = \sum_{i \rightarrow j} \frac{r_i}{d_i},
\end{align*}\]
<p>where we have the last inequality because \(A_{j, i}\) is nonzero (and equal to \(\frac{1}{d_i}\)) when there is an edge from \(i\) to \(j\). This equation exactly matches \(\eqref{eq:importance_flow}\).</p>
<h3 id="therandomsurfer">The random surfer</h3>
<p>One can think of \(A\) as an adjacency matrix of \(G\), but instead of \(A_{j, i} = 1\) when there is an edge from \(i\) to \(j\), we have \(A_{j, i} = \frac{1}{d_i}\). There is a nice interpretation of \(A\) called the random surfer model.</p>
<p>Suppose we have a web surfer who is currently on page \(i\). To visit a new page, the surfer will randomly choose one of the outneighbors of \(i\). Since the outdegree of \(i\) is \(d_i\), if we assume that all outneighbors are equally likely to be chosen, the probability that the surfer will choose a neighbor is \(\frac{1}{d_i}\). This is exactly captured in the matrix \(A\).</p>
<h3 id="pagerankasafixedpointproblem">PageRank as a fixedpoint problem</h3>
<p>Since \((A r)_j\) gives us the importance of page \(j\), which is also equal to \(r_j\), we have:</p>
\[\begin{align} \label{eq:fixed_point}
A r = r.
\end{align}\]
<p>The solution \(r\) to this linear system is the vector containing the ranks of our webpages. Note that we can scale \(r\) by a positive number and it would still satisfy this equation, achieving our goal of preserving the order from positive scaling stated above.</p>
<p>Such an \(r\) satisfying \(\eqref{eq:fixed_point}\) is called a <em>fixed point</em> of \(A\), because applying \(A\) to \(r\) (that is, multiplying \(A\) by \(r\)) will not change the values of \(r\) at all. I have another post on solving for a fixed point in the context of machine learing, which can be found <a href="/posts/andersonacceleration">here</a>. In this post, we will revisit a method to solve for \(r\).</p>
<h2 id="solvingpagerank">Solving PageRank</h2>
<p>If we look again at equation \(\eqref{eq:fixed_point}\), we can recognize that this is an eigenvector problem. Specifically, if \(\eqref{eq:fixed_point}\) holds, then \(r\) must be an eigenvector of \(A\) corresponding to an eigenvalue of \(1\). There are two important questions to answer.</p>
<p>First, is it guaranteed that \(A\) has \(1\) as an eigenvalue? After all, \(A\) is just a nonnegative matrix with each column summing to \(1\). It turns out that this is true, and we will see the proof below.</p>
<p>Second, given that \(1\) is an eigenvalue, then we can solve \(A r = r\) using a rowreduction algorithm such as <a href="https://en.wikipedia.org/wiki/Gaussian_elimination">Gaussian elimination</a>. Is that it? The answer is no, because Gaussian elimination has the time complexity of \(O(n^3)\), where \(n\) is the number of pages. This does not scale well with our page collection, because \(n\) could be in the billions, if not more. Therefore, we need to find another way to solve \(\eqref{eq:fixed_point}\).</p>
<h3 id="stochasticmatrices">Stochastic matrices</h3>
<p>To answer the first question above, notice that the matrix \(A\) is an example of a <em>stochastic matrix</em>, which is a square matrix with nonnegative entries and having every column sum to 1. In the context of PageRank, \(A\) is also called the <em>stochastic adjacency matrix</em>.</p>
<p>What is interesting about a stochastic matrix is that it accepts \(1\) as an eigenvalue, and all other eigenvalues (real or complex) of \(A\) are less than or equal to \(1\) in absolute value.</p>
<div style="paddingleft:2em; paddingright:2em">
<b>Proof.</b>
<br />
Since $A$ is a square matrix, $A$ and $A^\top$ share the same eigenvalues. We need to prove that $1$ is an eigenvalue of $A^\top$. Because every row of $A^\top$ sums to $1$, we have $A^\top \mathbf{1}_n = \mathbf{1}_n$, where $\mathbf{1}_n$ is a column vector of $n$ ones. So, $1$ is an eigenvalue of $A^\top$ and, therefore, of $A$.
<br /><br />
To show why all other eigenvalues of $A$ are less than or equal to $1$ in absolute value, let $\lambda$ be an eigenvalue of $A$. So $\lambda$ is also an eigenvalue of $A^\top$, associated with an eigenvector $x = [x_1,\ldots,x_n]^\top$. In other words, $A^\top x = \lambda x$. Let $j$ be index of the largest element in absolute value of $x$, that is, $x_i \leq x_j ~ \text{for all} ~ i=1,\ldots,n$. We have
$$
\begin{align*}
\lambda x_j = \lambda x_j = \left \sum_{i=1}^{n} A_{i, j} x_i \right \leq \sum_{i=1}^{n} A_{i, j} x_j = x_j \sum_{i=1}^{n} A_{i, j} = x_j,
\end{align*}
$$
where the first inequality uses the triangle inequality and the definition of $x_j$, and the last equality uses the fact the column $j$ of $A$ sums to 1. Since $x_j \neq 0$, this implies that $\lambda \leq 1$.
</div>
<h3 id="thefixedpointiteration">The fixedpoint iteration</h3>
<p>To answer the second question, we use the fact we just proved above, which is that \(1\) is the largest eigenvalue of \(A\) in absolute value. In linear algebra, it is also called the <a href="https://en.wikipedia.org/wiki/Spectral_radius"><em>spectral radius</em></a> of \(A\). As an alternative to Gaussian elimination, a popular algorithm to find the spectral radius and its corresponding eigenvector is the <a href="https://en.wikipedia.org/wiki/Power_iteration">power iteration</a>.</p>
<div style="paddingleft:2em; paddingbottom:1em;">
Procedure: Power Iteration
<br />
Input: A diagonalizable $n \times n$ matrix $A$
<br />
Let $b_0$ some nonzero vector
<br />
For $k = 0, \ldots, K1$ do
<br />
Apply $A$ to $b_k$: $\tilde{b}_{k+1} = A b_{k}$
<br />
Normalize: $b_{k+1} = \frac{\tilde{b}_{k+1}}{\lVert \tilde{b}_{k+1} \rVert}$
<br />
Output: $b_{K}$
</div>
<p>The sequence \(\left(\frac{\lVert A b_k \rVert}{\lVert b_k \rVert}\right)_k\) is guaranteed to converge to the spectral radius of \(A\) (which is \(1\) in our case), and the sequence \((b_k)_k\) converges to the corresponding eigenvector with unit norm.</p>
<p>The convergence rate of this sequence can be found <a href="https://en.wikipedia.org/wiki/Power_iteration#Analysis">here</a>. In practice, one would run the power iteration until the difference between two iterates falls below some predefined tolerance \(\epsilon\). For example, we can run until \(\lVert b_{k+1}  b_{k} \rVert \leq 10^{3}\).</p>
<h3 id="conclusiontemporary">Conclusion (temporary)</h3>
<p>We have learned how to find the importance scores of webpages in order to rank them. First, we construct the importance matrix from the structure of the graph. Then, we use the power iteration to solve for the fixed point of this matrix, which is the eigenvector corresponding to the largest eigenvalue in absolute value. This solution \(r\) now contains the importance of the pages, and we are ready to use \(r\) to rank them!</p>
<p>However, there are two potential problems with this approach. We will explore it and propose a solution in the below.</p>
<h2 id="twoproblemswithpagerank">Two Problems with PageRank</h2>
<h3 id="problem1deadends">Problem 1: dead ends</h3>
<p>In the previous section, we have learned to use the power iteration to solve for the importance vector \(r\). However, the power iteration works under an assumption that the matrix \(A\) is <a href="https://en.wikipedia.org/wiki/Diagonalizable_matrix"><em>diagonalizable</em></a>. This will not hold if a column of \(A\) contains all zeros. This case happens when a webpage has no outgoing links. In other words, the page is a <em>dead end</em>.</p>
<p>How do we solve this? Let’s go back to the random surfer model above. If the surfer is at a dead end, meaning there is no hyperlink on the page the surfer can click to go to, we will assume that they will randomly jump to any other page in our collection. In addition, all pages are assumed to be equally likely to be chosen. So, if a page \(i\) is a dead end, we will replace the allzeros column for \(i\) with a column of all \(\frac{1}{n}\)’s, where \(n\) is the number of pages in our collection.</p>
<p>Therefore, we can transform the matrix \(A\) into one without dead ends. Let us call this matrix \(A'\). Every column of \(A'\) now sums to 1.</p>
<h3 id="problem2spidertraps">Problem 2: spider traps</h3>
<p>The matrix \(A'\) is now guaranteed to be a stochastic matrix, and we are ready to use the power iteration to find its fixed point. However, the result might not be what we want. Consider the following scenario: In our web graph, there is a set of at least one node such that there are no links coming out of this set. There can be links between nodes in this set, but there are no link to any other outside node.</p>
<p>We call such a set of nodes a <em>spider trap</em>. But what is the problem? If we use the power iteration for a graph with a spider trap, the algorithm will cause all importance scores to be captured within the nodes in this spider trap, and the rest of the nodes will have zero importance. This kind of pages can be constructed intentionally or unintentionally, but their existence will cause PageRank to output an undesirable result.</p>
<p>So how do we deal with spider traps? Once the random surfer is in a spider trap, they will never be able to leave it. We will assume that, when the surfer is at page \(i\), they will flip a coin. If the coin comes up heads, the surfer will follow a link at random, and the probability of choosing a page is found by looking up the \(i\)th column of \(A'\). If the coin comes up tails, the surfer will jump to a page in our collection uniformly at random. So, if page \(i\) is in a spider trap, the surfer has a some chance of jumping outside the trap when the coin comes up tails.</p>
<p>To formalize this, let \(p\) be the probability of the coin coming up heads. The probability that the surfer, currently at page \(i\), will go to page \(j\) is</p>
\[\begin{align*}
p A'_{j, i} + (1  p) \frac{1}{n}.
\end{align*}\]
<h3 id="thegooglematrix">The Google matrix</h3>
<p>In 1998, Larry Page and Sergey Brin, the founders of Google, proposed a matrix combining the solutions to these two problems. It is now widely called the <em>Google matrix</em>:</p>
\[\begin{align*}
\mathscr{G} = pA' + (1p) \frac{1}{n} \mathbf{1}_n \mathbf{1}_n^\top.
\end{align*}\]
<p>By using the power iteration on \(\mathscr{G}\), we can find the importance scores of the pages in our collection. This is the algorithm that Google uses to rank webpages.</p>
<h2 id="resources">Resources</h2>
<ol>
<li><a href="https://textbooks.math.gatech.edu/ila">Interactive Linear Algebra</a> by Dan Margalit and Joseph Rabinoff. Specifically Chapter 5.</li>
<li><a href="http://www.mmds.org/">Mining of Massive Datasets</a> by Jure Leskovec, Anand Rajaraman, and Jeffrey D. Ullman. Specifically Chapter 5.</li>
<li><a href="http://web.stanford.edu/class/cs224w/">CS224W  Machine Learning with Graphs</a> by Jure Leskovec, Fall 2021 edition. Specifically Lecture 4.</li>
<li><a href="https://research.google/pubs/pub334.pdf">The Anatomy of a LargeScale Hypertextual Web Search Engine</a> by Sergey Brin and Lawrence Page.</li>
</ol>Josh Nguyenjosh.nguyen@anu.edu.auIn this post, we will revisit a popular algorithm called PageRank, which is used by Google to rank webpages for its search engine. Surprising to some but not so to others, PageRank is simple enough that only a level of firstyear undergraduate linear algebra is required to understand it.A Gentle Introduction to Computational Optimal Transport20221018T00:00:0007:0020221018T00:00:0007:00https://joshnguyen.net/posts/otanuaml<p>This is a lecture I gave to the <a href="/teaching/2022AML">COMP4680/8650 Advanced Topics in Machine Learning</a> 2022S2 class at ANU.</p>
<p>The slides can be found <a href="/files/ANU_OT_Slides.pdf">here</a>. The lecture recording is on <a href="https://www.youtube.com/watch?v=3YFmaoCYSlc">YouTube</a>.</p>Josh Nguyenjosh.nguyen@anu.edu.auThis is a lecture I gave to the COMP4680/8650 Advanced Topics in Machine Learning 2022S2 class at ANU.Variational Bayes for Latent Dirichlet Allocation20220801T00:00:0007:0020220801T00:00:0007:00https://joshnguyen.net/posts/lda<p>In this post we will learn about a widelyused topic model called Latent Dirichlet Allocation (LDA), proposed by <a href="https://www.jmlr.org/papers/volume3/blei03a/blei03a.pdf">Blei, Ng and Jordan in 2003</a>. Although research in probabilistic topic modeling has been longstanding, approaching it from a perspective of a newcomer can be quite challenging. Also, there is a lot of literature on the applications of topic models, especially LDA and in many disciplines; I therefore would need to dedicate at least a series of 10 posts to reasonably cover these applications. As such, I constrain myself to the following desiderata when writing this post:</p>
<ul>
<li>Explain what probabilistic topic modeling is, and what assumptions it makes.</li>
<li>Recognize the observable and latent variables in a topic model, and specifically in LDA.</li>
<li>Explain the generative process of LDA, and derive the complete probability.</li>
<li>Explain what inference means in a mixture model, and why it is hard in LDA.</li>
<li>Find the approximate posterior distribution of LDA using variational inference, and explain the procedure to find the optimal variational parameters.</li>
<li>Explain what it means to “fit” an LDA model to a corpus, and describe how this procedure works.</li>
<li>Be able to write code for an LDA model, including training and inference.</li>
</ul>
<h2 id="introduction">Introduction</h2>
<p>Being able to describe a large collection of documents is an important task in many disciplines. This task is often called “describe the haystack,” and the idea is to find the common <em>themes</em> that appear in the documents. For example, given a corpus of abstracts from papers published to <a href="https://www.pnas.org/">PNAS</a>, can we find the common scientific topics—such as “cellular biology,” “genetics” or “evolution”—that are covered in these abstracts? Another example is when you collect many tweets in a specific period, and want to find out what common topics people tweet about during this period, in the hope of predicting what topics will be trending in the near future. To help us approach this, there are three discussions worth noting here.</p>
<p>First, identifying topics by manually reading a collection of documents is probably the best way to characterize its themes, but the mere size of a corpus makes it impossible to perform this; we are looking at tens of thousands of abstracts, hudreds of thousands of Reddit posts, millions of Wikipedia articles, and tens of millions of tweets. Coming up with a way in which a computer can help us <em>automatically</em> identify the topics is much more desirable.</p>
<p>Second, what do we mean by <em>topics</em>, or themes? Put simply, a topic is a probability distribution over the vocabulary. For example, a topic about natural language processing is a distribution, with (much) higher probabilities for words such as “machine,” “token,” “vector” and “likelihood” than for words such as “mechanic,” “torts,” “cell” and “chemical.” Typically, we describe a topic by a list of mostlikely words of size, say, 10 or 15. A human can look at this list and give the topic a representative name if necessary.</p>
<p>Third, it is quite evident that a document is rarely exclusively about one topic. (Well, this depends on how finegrained you define each topic to be, but note that the more finegrained, the harder it is to generalize.) In fact, we often associate a document with a <em>mixture</em> of topics, perhaps with a higher weight to some than others. For example, a research paper in machine learning can be a mixture of topics such as optimization, statistics, statistical physics, and so on, and a human reader can probably tell which topic is weighed higher than others after reading the paper. A solution to modeling this is to have a probability distribution over topics, given a document.</p>
<h3 id="probabilistictopicmodels">Probabilistic topic models</h3>
<p>The two types probability distribution described above are the main ingredients of probabilistic topic models such as LDA. If we are able to model them, we can do many useful things. First, using the topicword distributions allows us to characterize the topics present in a corpus, thereby summarizing it in a meaningful way. And using the documenttopic distributions allows us to draw inference on the topics that a document is about, also helping with summarization. The applications of these models are quite boundless, which is why they are so popular in many fields such as computational social science, psychology, cognitive science, and so on.</p>
<p>However, in order to use them correctly as well as identifying the pros and cons to make good decisions while modeling, one should not stop at only calling <code class="languageplaintext highlighterrouge">sklearn.decomposition.LatentDirichletAllocation</code> arbitrarily, but should be able to understand the model, its assumptions, and how to tune its hyperparameters. To demonstrate this, let us dive into the details of the model.</p>
<h2 id="latentdirichletallocation">Latent Dirichlet Allocation</h2>
<p>A probabilistic topic model, LDA still remains one of the most popular choices for topic modeling today. It is an example of a <em>mixture model</em> whose structure contains two types of random variables:</p>
<ul>
<li>The <em>observable variables</em> are the words you observe in each document.</li>
<li>The <em>latent variables</em> are those you do not observe, but which describe some internal <em>structure</em> of your data, in particular, the “topics”.</li>
</ul>
<p>You can readily see the assumption here, which is that there there <em>is</em> some internal structure to your data, and our job is to model that structure using the latent variables.</p>
<h3 id="generativeprocess">Generative process</h3>
<p>In specifying a mixture model like LDA, we need to describe how data can be generated using this model. Before we do that, let us set up the notation carefully. Note that in this blog post, I have chosen the notation used in Hoffmann, Blei and Bach’s <a href="https://papers.nips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0Paper.pdf">paper on online learning for LDA</a>. This blog is intended to follow the “batch variational Bayes” part of the paper, with some more detail to help you read more easily.</p>
<p>Suppose we have a collection of $D$ documents, where each document $d$ is of length $N_d$. Also suppose that we have a fixed vocabulary of $W$ words. We wish to discover $K$ topics in this collection, where each topic $k$ is specified by the probability $\beta_k$ over all words. The generative process works as follows. For document $d$, sample probability distribution $\theta_d$ over the topics $1, \ldots, K$. For each word $w_{di}$ in document $d$, sample a topic $z_{di}$ from the distribution $\theta_d$. With the chosen topic $z_{di}$, sample a word $w_{di}$ from the probability distribution $\beta_{z_{di}}$. In other words,</p>
<ul>
<li>Draw a topicword distribution $\beta_k \sim \text{Dir}(\eta)$ for $k = 1, \ldots, K$.</li>
<li>For each document $d = 1, \ldots, D$:
<ul>
<li>Draw documenttopic distribution for document $d$: $\theta_d \sim \text{Dir}(\alpha)$.</li>
<li>For each word $i$ in document $d$:
<ul>
<li>Draw a topic $z_{di} \sim \theta_d$.</li>
<li>Draw a word $w_{di} \sim \beta_{z_{di}}$.</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>The notation is summarized in the following table.</p>
<table>
<thead>
<tr>
<th style="textalign: center">Notation</th>
<th style="textalign: center">Dimensionality</th>
<th style="textalign: left">Meaning</th>
<th style="textalign: left">Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td style="textalign: center">$D$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Number of documents</td>
<td style="textalign: left">Positive integer</td>
</tr>
<tr>
<td style="textalign: center">$W$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Number of words in the vocabulary</td>
<td style="textalign: left">Positive integer</td>
</tr>
<tr>
<td style="textalign: center">$K$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Number of topics</td>
<td style="textalign: left">Positive integer,typically much smaller than $D$</td>
</tr>
<tr>
<td style="textalign: center">$N_d$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Number of words in document $d$</td>
<td style="textalign: left">Positive integer</td>
</tr>
<tr>
<td style="textalign: center">$\beta_k$</td>
<td style="textalign: center">$W$</td>
<td style="textalign: left">Word distribution for topic $k$</td>
<td style="textalign: left">$\beta_k$ ($k = 1, \ldots, K)$ are mutually independent. Each $\beta_k$ is a nonnegative vector and $\sum_{w=1}^{W} \beta_{kw} = 1$.</td>
</tr>
<tr>
<td style="textalign: center">$\eta$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Dirichlet prior parameter for $\beta_k$</td>
<td style="textalign: left">All $\beta_k$ share the same parameter $\eta$.</td>
</tr>
<tr>
<td style="textalign: center">$\theta_d$</td>
<td style="textalign: center">$K$</td>
<td style="textalign: left">Topic distribution for document $d$</td>
<td style="textalign: left">$\theta_d$ ($d = 1, \ldots, D$) are mutually independent. Each $\theta_d$ is a nonnegative vector and $\sum_{k=1}^{K} \theta_{dk} = 1$.</td>
</tr>
<tr>
<td style="textalign: center">$\alpha$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Dirichlet prior parameter for $\theta_d$</td>
<td style="textalign: left">All $\theta_d$ share the same parameter $\alpha$.</td>
</tr>
<tr>
<td style="textalign: center">$w_{di}$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Word $i$ in document $d$</td>
<td style="textalign: left">$w_{di} \in \{1, 2, \ldots, W\}$</td>
</tr>
<tr>
<td style="textalign: center">$z_{di}$</td>
<td style="textalign: center">Scalar</td>
<td style="textalign: left">Topic assignment for word $w_{di}$</td>
<td style="textalign: left">$z_{di} \in \{1, 2, \ldots, K\}$</td>
</tr>
</tbody>
</table>
<h3 id="completemodel">Complete model</h3>
<p>The types of variables should be clear to us now. The only observables we have are $w$, the words in the documents. On the other hand, the latent variables are $z$, $\theta$ and $\beta$. The generative process allows us to specify the complete model—i.e., the joint distribution of both observable and latent variables—as follows</p>
\[\begin{align}
p(w, z, \theta, \beta \mid \alpha, \eta) & = p(\beta \mid \eta) \prod_{d=1}^{D} p(\theta_d \mid \alpha) p(z_d \mid \theta_d) p(w_d \mid \theta_d, z_i, \beta) \label{eq:joint_prob}\\
& = \prod_{k=1}^{K} p(\beta_k \mid \eta) \prod_{d=1}^{D} p(\theta_d \mid \alpha) \prod_{i=1}^{N_d} p(z_{di} \mid \theta_d) p(w_{di} \mid \theta_d, z_{di}, \beta). \nonumber
\end{align}\]
<h3 id="dirichletandcategoricaldistributions">Dirichlet and categorical distributions</h3>
<p>Note that there are two probability distributions used in this process. The first is the <a href="https://en.wikipedia.org/wiki/Dirichlet_distribution">Dirichlet</a> used to sample $\beta_k$ and $\theta_d$. For example, the probability of the topic distribution for document $d$ is</p>
\[p(\theta_d \mid \alpha) = \frac{\Gamma\left( K \alpha \right)}{\Gamma(\alpha)^K} \prod_{k=1}^K \theta_{dk}^{\alpha1}.\]
<p>The second is the <a href="https://en.wikipedia.org/wiki/Categorical_distribution">categorical distribution</a>, used to sample $z_{di}$ and $w_{di}$. For example, to find the probablity that the word $w_{di}$ given all other variables, we first need to find the value of $z_{di}$. Suppose $z_{d_i} = 2$. Then the distribution we need to use is $\beta_2$, or the second topic. Then the probability that $w_{di}$ equals some $w$ is</p>
\[p(w_{di}  z_{di} = 2, \beta, \theta_d) = \beta_{2, w},\]
<p>that is, the $w$th entry of $\beta_{2}$.</p>
<h2 id="inferenceapproximateinferenceandparameterestimation">Inference, Approximate Inference and Parameter Estimation</h2>
<p>Inference refers to the task of finding the probability of latent varibles given observable variables. In our LDA example, the quantity we want to calculate is</p>
\[\begin{align}
p(z, \theta, \beta \mid w, \alpha, \eta) = \frac{p(w, z, \theta, \beta \mid \alpha, \eta)}{\int_{z, \theta, \beta} p(w, z, \theta, \beta \mid \alpha, \eta) dz d\theta d\beta}. \label{eq:bayesinfr}
\end{align}\]
<p>What is this quantity? Imagine you see new document, how do you know what topics it belongs to, along with the topic weights? The probability in $\eqref{eq:bayesinfr}$ helps us do just that: use the Bayes theorem to find the <em>posterior</em> distribution on the latent variables, enabling us to draw inference on the structure of the document.</p>
<p>But there is a catch. The integral in the denominator $\eqref{eq:bayesinfr}$, which is equal to $p(w \mid \alpha, \eta)$ and often called the <em>evidence</em>, is very hard to evaluate. This is mainly because of the coupling of the latent variables, and exactly calulating this will take exponential time. Instead, we will use an method called <em>variational inference</em> to approximate it.</p>
<h3 id="variationalinferencevi">Variational inference (VI)</h3>
<p>(To keep this blog post short enough, I will not explain the details of VI. You are encourage to check out Chapter 10 in <a href="https://probml.github.io/pmlbook/book2.html">Kevin Murphy’s textbook on probabilistic machine learning</a> for an introduction to VI.)</p>
<p>Basically, the goal of VI is to approximate the distribution $p(z, \theta, \beta \mid w, \alpha, \eta)$ using a simpler distribution $q(z, \theta, \beta)$ that is “the closest” to $p$. Here “closeness” is defined by the KullbackLeibler divergence between $q$ and $p$. In other words, we aim to solve the following optimization problem:</p>
\[\min_{q} \left\{ \text{KL}(q(z, \theta, \beta) \ p(z, \theta, \beta \mid w, \alpha, \eta)) = \mathbb{E}_q \left[ \log \frac{q(z, \theta, \beta)}{p(z, \theta, \beta \mid w, \alpha, \eta)} \right] \right\}.\]
<h3 id="evidencelowerboundelboandvariationalbayesvb">Evidence lower bound (ELBO) and variational Bayes (VB)</h3>
<p>Interestingly, minimizing this KL divergence is equivalent to maximizing the <em>evidence lower bound</em> (ELBO) of the data, where the ELBO $\mathcal{L}(w, z, \theta, \beta)$ is defined as</p>
\[\begin{align}
\mathcal{L}(w, \phi, \gamma, \lambda) = \mathbb{E}_q\left[ \log p(w, z, \theta, \beta \mid \alpha, \eta) \right]  \mathbb{E}_q\left[ \log q(z, \theta, \beta) \right]. \label{eq:elbo:def}
\end{align}\]
<p>As the name suggests, the ELBO is a lower bound on the loglikelihood of our data. The maximum ELBO gives us the “closest” approximation to the likelihood. Check Section 10.1.2 in <a href="https://probml.github.io/pmlbook/book2.html">Murphy’s textbook</a> for a full derivation.</p>
<p>To “fit” the data in the Bayesian sense, we will aim to approximate the true posterior as well as possible. Applying VI to this task is called <em>variational Bayes</em> (VB).</p>
<h3 id="choosingvariationalparameters">Choosing variational parameters</h3>
<p>We have mentioned the “simpler” distribution $q(z, \theta, \beta)$ above, but what exactly is it? In using VI for LDA inference, we assume that $q(z, \theta, \beta)$ factorizes to three marginal distributions:</p>
<ul>
<li>$q(z_{di}) = \phi_{d w_{di} k}$. The dimensionality of $\phi$ is $D \times W \times K$, and $\sum_{k=1}^{K} \phi_{d w k} = 1, \forall d, w$;</li>
<li>$\theta_d \sim \text{Dir}(\gamma_d)$, where $\gamma_d$ is a vector of length $K$. Note that $\gamma_d$ is <em>not</em> symmetric;</li>
<li>$\beta_k \sim \text{Dir}(\lambda_k)$, where $\lambda_k$ is a vector of length $W$. Similarly, $\beta_k$ is <em>not</em> symmetric.</li>
</ul>
<p>This is an application of the <em>meanfield assumption</em>, which says that variational distributions for each set of latent variables are mutually independent, allowing the joint to be factorized into marginals.</p>
<p>In summary,</p>
\[\begin{align}
q(z_d, \theta_d,\beta) = q(z_d) q(\theta_d)q(\beta), \label{eq:mean_field}
\end{align}\]
<p>and we have three types of variational parameters: $\phi$ of size $D \times W \times K$; $\gamma_d$ of size $K$, for $d = 1, \ldots, D$; and $\lambda_k$ of size $W$, for $k = 1, \ldots, K$.</p>
<h3 id="factorizingelbo">Factorizing ELBO</h3>
<! $\log p(w, z, \theta, \beta \mid \alpha, \eta) = \log p(\beta \mid \eta) + \sum_{d=1}^{D} \left[ \log p(\theta_d \mid \alpha) + \log p(z_d \mid \theta_d) + \log p(w_d \mid z_d, \theta_d, \beta) \right]$ >
<! $\log q(z_d, \theta_d,\beta) = \log q(z_d) + \log q(\theta_d) + \log q(\beta)$. >
<p>Given the complete model in $\eqref{eq:joint_prob}$ and the variational distribution in $\eqref{eq:mean_field}$, we can decompose the ELBO as follows:
\(\begin{align}
\mathcal{L}(w, \phi, \gamma, \lambda) & =
\sum_{d=1}^{D}
\left\{
\mathbb{E}_q\left[ \log p(w_d \mid \theta_d, z_d, \beta) \right] +
\mathbb{E}_q\left[ \log p(z_d \mid \theta_d) \right] 
\mathbb{E}_q\left[ \log p(\theta_d \mid \alpha) \right]
\right\} \nonumber \\
&~~~~

\sum_{d=1}^{D}
\left\{
\mathbb{E}_q\left[ \log q(z_d \mid \theta_d) \right] +
\mathbb{E}_q\left[ \log q(\theta_d) \right]
\right\} \nonumber \\
&~~~~ + \mathbb{E}_q\left[ \log p(\beta \mid \eta) \right]  \mathbb{E}_q\left[ \log q(\beta) \right] \nonumber \\
& = \sum_{d=1}^{D} \left\{ \mathbb{E}_q\left[ \log p(w_d \mid \theta_d, z_d, \beta) \right] + \mathbb{E}_q\left[ \log p(z_d \mid \theta_d) \right]  \mathbb{E}_q\left[ \log q(z_d \mid \theta_d) \right] \right. \nonumber\\
&\quad \quad \quad ~
+\left.\mathbb{E}_q\left[ \log p(\theta_d \mid \alpha) \right]  \mathbb{E}_q\left[ \log q(\theta_d) \right]
\right\} \nonumber \\
& ~~~~ + (\mathbb{E}_q\left[ \log p(\beta \mid \eta) \right]  \mathbb{E}_q\left[ \log q(\beta) \right]). \label{eq:elbo} \\
\end{align}\)</p>
<h3 id="elboasafunctionofvariationalparameters">ELBO as a function of variational parameters</h3>
<p>Analyzing each term in the sum.
\(\begin{align}
\mathbb{E}_q\left[ \log p(w_d \mid \theta_d, z_d, \beta) \right] & = \sum_{i=1}^{N_d} \mathbb{E}_q\left[ \log p(w_{di} \mid \theta_d, z_{di}, \beta) \right] \nonumber \\
& = \sum_{i=1}^{N_d} \sum_{k=1}^{K} q(z_{di} = k) \mathbb{E}_q\left[ \log p(w_{di} \mid \theta_d, z_{di}, \beta) \right] \nonumber \\
& = \sum_{i=1}^{N_d} \sum_{k=1}^{K} \phi_{d w_{di} k} \mathbb{E}_q\left[ \log \beta_{k w_{di}} \right], \nonumber
\end{align}\)</p>
<p>where the expectation on the last row is with respect to $q(\beta_k)$. We can see that in this formula, the contribution of each word $w$ to the term is $\sum_{k=1}^{K} \phi_{d w k} \mathbb{E} \left[ \log \beta_{k w} \right]$, which is the same for regardless of the position of word $w$ in document $d$. Therefore, we can simply count the number of times $w$ appears in $d$, and then multiply it with this contribution to get the contribution of all occurrences of $w$. This gives us the equivalent expression:
\(\begin{align}
\mathbb{E}_q\left[ \log p(w_d \mid \theta_d, z_d, \beta) \right] = \sum_{w=1}^{W} n_{dw} \sum_{k=1}^{K} \phi_{d w k} \mathbb{E}_q\left[ \log \beta_{k w} \right], \label{eq:elbo:1}
\end{align}\)</p>
<p>where $n_{dw}$ is the number of occurrences of word $w$ in document $d$. Using the same trick, we have
\(\begin{align}
\mathbb{E}_q\left[ \log p(z_d \mid \theta_d) \right] & = \sum_{w=1}^{W} n_{dw} \sum_{k=1}^{K} \phi_{d w k} \mathbb{E}_q\left[ \log \theta_{dk} \right], \text{and} \label{eq:elbo:2} \\
\mathbb{E}_q\left[ \log q(z_d) \right] & = \sum_{w=1}^{W} n_{dw} \sum_{k=1}^{K} \phi_{d w k} \log \phi_{d w k}. \label{eq:elbo:3}
\end{align}\)</p>
<p>For the last two terms inside the sum, first note that $p(\theta_d \mid \alpha)$ is a Dirichlet distribution with symmetric parameter $\alpha$, i.e., $q(\theta_d \mid \alpha) = \frac{\Gamma(K \alpha)}{\Gamma(\alpha)^K} \prod_{k=1}^{K} \theta_{dk}^{\alpha1}$. Therefore,
\(\begin{align}
\mathbb{E}_q\left[ \log p(\theta_d \mid \alpha) \right] = \log \Gamma(K \alpha)  K \log \Gamma(\alpha) + (\alpha  1) \sum_{k=1}^{K} \log \theta_{dk}. \label{eq:elbo:4}
\end{align}\)</p>
<p>Similarly, because $q(\theta_d)$ is a Dirichlet distribution with asymmetric parameter $\gamma_d$, we have
\(\begin{align}
\mathbb{E}_q\left[ \log q(\theta_d) \right] = \log \Gamma\left(\sum_{k=1}^{K} \gamma_{dk} \right)  \sum_{k=1}^{K} \log \Gamma(\gamma_{dk}) + \sum_{k=1}^{K} (\theta_{dk}  1) \log \theta_{dk}. \label{eq:elbo:5}
\end{align}\)</p>
<p>Now for the last two terms, also note that $p(\beta_k \mid \eta)$ is Dirichlet with symmetric $\eta$. Therefore,
\(\begin{align}
\mathbb{E}_q\left[ \log p(\beta \mid \eta) \right] &= \sum_{k=1}^{K} \mathbb{E}_q\left[ \log p(\beta_k \mid \eta) \right] \nonumber \\
&= K [\log \Gamma(W \eta)  W \log \Gamma(\eta)] + \sum_{k=1}^{K} \sum_{w=1}^{W} (\eta  1) \mathbb{E}_q\left[ \log \beta_{k w} \right]. \label{eq:elbo:6}
\end{align}\)</p>
<p>Simlarly, the final term is
\(\begin{align}
\mathbb{E}_q\left[ \log q(\beta) \right] &= \sum_{k=1}^{K} \mathbb{E}_q\left[ \log q(\beta_k) \right] \nonumber \\
&= \sum_{k=1}^{K} \left( \log \Gamma \left( \sum_{w=1}^{W} \lambda_{kw} \right)  \sum_{w=1}^{W} \Gamma(\lambda_{kw}) + \sum_{w=1}^{W} (\lambda_{kw}  1) \mathbb{E}_q\left[ \log \beta_{k w} \right] \right). \label{eq:elbo:7}
\end{align}\)</p>
<p>Plugging $\eqref{eq:elbo:1}, \eqref{eq:elbo:2}, \eqref{eq:elbo:3}, \eqref{eq:elbo:4}, \eqref{eq:elbo:5}, \eqref{eq:elbo:6}, \eqref{eq:elbo:7}$ into $\eqref{eq:elbo}$, we have the ELBO as a function of variational parameters:</p>
\[\begin{align}
\mathcal{L} &= \sum_{d=1}^{D} \left\{ \sum_{w=1}^{W} n_{dw} \sum_{k=1}^{K} \phi_{dwk} \left( \mathbb{E}_q\left[ \log \theta_{dk} \right] + \mathbb{E}_q\left[ \log \beta_{k w} \right]  \log \phi_{dwk} \right) \right. \nonumber\\
& \left. \quad \quad \quad ~  \log \Gamma\left( \sum_{k=1}^{K} \gamma_{dk} \right) + \sum_{k=1}^{K}\left( \log \Gamma(\gamma_{dk}) + (\alpha  \gamma_{dk}) \mathbb{E}_q\left[ \log \theta_{dk} \right] \right) \right\} \nonumber \\
&~~~~ + \sum_{k=1}^{K} \left(  \log \Gamma\left( \sum_{w}^{W} \lambda_{kw} \right) + \sum_{w=1}^{W} \left( \log \Gamma(\lambda_{kw}) + (\eta  \lambda_{kw}) \mathbb{E}_q\left[ \log \beta_{k w} \right] \right) \right) \nonumber \\
&~~~~ + D [\log \Gamma(K \alpha)  K \log \Gamma(\alpha)] + K [\log \Gamma(W \eta)  W \log \Gamma(\eta)]. \label{eq:elbo:var}
\end{align}\]
<h2 id="variationalbayesforlda">Variational Bayes for LDA</h2>
<p>The main objective here is to maximize the ELBO $\mathcal{L}$ with respect to the variational parameters $\phi$, $\gamma$ and $\lambda$. To do so, we will use a procedure called <em>coordinate ascent</em>, in which we maximize $\mathcal{L}$ with respect to one set of parameters, keeping the others fixed. We will then alternate to another set of variables, keeping others fixed, and so on. In our LDA example, we first keep $\gamma$ and $\lambda$ fixed, and maximize $\mathcal{L}$ as a function of $\phi$ only. Then we do the same for $\gamma$ and $\lambda$.</p>
<h3 id="maximizingwithrespecttophi">Maximizing with respect to $\phi$</h3>
<p>Only keeping the terms involving $\phi_{dwk}$ in $\eqref{eq:elbo:var}$, and treating everything else as constants, we have the objective function w.r.t. $\phi_{dwk}$ as</p>
\[\mathcal{L}_{[\phi_{dwk}]} = \phi_{dwk} \left( \mathbb{E}_q\left[ \log \theta_{dk} \right] + \mathbb{E}_q\left[ \log \beta_{k w} \right]  \log \phi_{dwk} \right) + \text{const},\]
<p>which gives the gradient:</p>
\[\frac{\partial \mathcal{L}}{\partial \phi_{dwk}} = \mathbb{E}_q\left[ \log \theta_{dk} \right] + \mathbb{E}_q\left[ \log \beta_{k w} \right]  \log \phi_{dwk}  1.\]
<p>Setting the gradient to zero and solving for $\phi_{dwk}$, we get the update rule for $\phi_{dwk}$:</p>
\[\begin{align}
\phi_{dwk} \propto \exp \left\{ \mathbb{E}_q\left[ \log \theta_{dk} \right] + \mathbb{E}_q\left[ \log \beta_{k w} \right] \right\}. \label{eq:update:phi}
\end{align}\]
<p>Where we have suppressed all multiplicative constants by using $\propto$. After this update for all $\phi_{dwk}$, we can simply rescale them so that $\sum_{k=1}^{K} \phi_{dwk} = 1, \forall d, w$.</p>
<p>The final thing to handle is the expectations inside $\exp$. How do we calculate them exactly? Lucklily, both of them can be calculated using the <a href="https://en.wikipedia.org/wiki/Digamma_function"><em>digamma function</em></a> $\Psi$—the first derivative of the logarithm of the gamma function—as follows:</p>
\[\begin{align*}
\mathbb{E}_q\left[ \log \theta_{dk} \right] & = \Psi(\gamma_{dk})  \Psi\left(\sum_{i=1}^{K} \gamma_{di}\right), \\
\mathbb{E}_q\left[ \log \beta_{k w} \right] & = \Psi(\lambda_{kw})  \Psi\left(\sum_{i=1}^{W} \lambda_{ki}\right).
\end{align*}\]
<h3 id="maximizingwithrespecttogamma">Maximizing with respect to $\gamma$</h3>
<p>Similarly, the objective function w.r.t. $\gamma_{dk}$ is</p>
\[\begin{align*}
\mathcal{L}_{[\gamma_{dk}]} & = \sum_{w=1}^{W} n_{dw} \phi_{dwk} \mathbb{E}_q \left[ \log \theta_{dk} \right]  \log \Gamma\left( \sum_{i=1}^{K} \gamma_{d_i} \right) \\
& ~~~~+ \log \Gamma(\gamma_{dk}) + (\alpha  \gamma_{dk}) \mathbb{E}_q \left[ \log \theta_{dk} \right] + \text{const} \\
& = \left( \alpha + \sum_{w=1}^{W} n_{dw} \phi_{dwk}  \gamma_{dk} \right) \left( \Psi(\gamma_{dk})  \Psi\left(\sum_{i=1}^{K} \gamma_{di}\right) \right) \\
& ~~~~  \log \Gamma\left( \sum_{i=1}^{K} \gamma_{d_i} \right) + \log \Gamma(\gamma_{dk}) + \text{const},
\end{align*}\]
<p>where we have used the digamma function $\Psi$ similarly to the previous section. A simple manipulation gives the gradient:</p>
\[\begin{align*}
\frac{\partial \mathcal{L}}{\partial \gamma_{dk}} = \left( \Psi'(\gamma_{dk})  \Psi'\left(\sum_{i=1}^{K} \gamma_{di}\right) \right) \left( \alpha + \sum_{w=1}^{W} n_{dw} \phi_{dwk}  \gamma_{dk} \right).
\end{align*}\]
<p>Setting this gradient to zero and solving for $\gamma_{dk}$, we get the update rule for $\gamma_{dk}$:</p>
\[\begin{align}
\gamma_{dk} = \alpha + \sum_{w=1}^{W} n_{dw} \phi_{dwk}. \label{eq:update:gamma}
\end{align}\]
<p>The variational Bayes estimate of $\gamma$ has an intuitive explanation. The number of times document $d$ is assigned to topic $k$ is the weighted sum of the times each word in $d$ is assigned to topic $k$, where the weight $\phi_{dwk}$ is the probability that word $w$ in document $d$ belongs to topic $k$—plus the Dirichlet prior $\eta$.</p>
<h3 id="maximizingwithrespecttolambda">Maximizing with respect to $\lambda$</h3>
<p>Similar to $\gamma$, we can use the digamma function $\Psi$ in the objective functin w.r.t. $\lambda_{kw}$ as follows</p>
\[\begin{align*}
\mathcal{L}_{[\lambda_{kw}]} & = \left( \eta + \sum_{d=1}^{D} n_{dw} \phi_{dwk}  \lambda_{kw} \right) \left( \Psi(\lambda_{kw})  \Psi\left(\sum_{i=1}^{W} \lambda_{ki} \right) \right) \\
& ~~~~  \log \Gamma\left(\sum_{i=1}^{W} \lambda_{ki} \right) + \log \Gamma(\lambda_{kw}) + \text{const},
\end{align*}\]
<p>which gives the gradient:</p>
\[\begin{align*}
\frac{\partial \mathcal{L}}{\partial \lambda_{kw}} = \left( \Psi'(\lambda_{kw})  \Psi'\left(\sum_{i=1}^{W} \lambda_{ki} \right) \right) \left( \eta + \sum_{d=1}^{D} n_{dw} \phi_{dwk}  \lambda_{kw} \right).
\end{align*}\]
<p>Setting the gradient to zero and solving for $\lambda_{kw}$, we get the update estimate:</p>
\[\begin{align}
\lambda_{kw} = \eta + \sum_{d=1}^{D} n_{dw} \phi_{dwk}. \label{eq:update:lambda}
\end{align}\]
<p>Similar to $\gamma_{dk}$, the variational Bayes estimate of $\lambda$ has an intuitive explanation. The count of word $w$ in topic $k$ the weighted sum of word count for $w$ in each document $d$, where the weight $\phi_{dwk}$ is the probability that word $w$ in document $d$ belongs to topic $k$—plus the Dirichlet prior $\eta$.</p>
<h3 id="puttingeverythingtogether">Putting everything together</h3>
<p>We have shown the update rules for the variational parameters: $\phi_{dwk}$ in $\eqref{eq:update:phi}$, $\gamma_{dk}$ in $\eqref{eq:update:gamma}$, and $\lambda_{kw}$ in $\eqref{eq:update:lambda}$. The variational Bayes algorithm is complete. There is one final thing to note, taken from the Section 2.1 of the <a href="https://papers.nips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0Paper.pdf">original paper</a>.</p>
<p>We can actually partition these updates into two steps, analogous to the two steps in the EM algorithm. In the “E”step, we keep updating $\gamma$ and $\phi$ until convergence, keeping $\lambda$ fixed. In the “M”step, iteratively update $\lambda$ holding $\gamma$ and $\phi$ fixed.</p>
<p>Now you can understand the paper’s <a href="https://papers.nips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0Paper.pdf">Algorithm 1</a> fully and can start implementing it in your favorite language.</p>Josh Nguyenjosh.nguyen@anu.edu.auIn this post we will learn about a widelyused topic model called Latent Dirichlet Allocation (LDA), proposed by Blei, Ng and Jordan in 2003. Although research in probabilistic topic modeling has been longstanding, approaching it from a perspective of a newcomer can be quite challenging. Also, there is a lot of literature on the applications of topic models, especially LDA and in many disciplines; I therefore would need to dedicate at least a series of 10 posts to reasonably cover these applications. As such, I constrain myself to the following desiderata when writing this post: Explain what probabilistic topic modeling is, and what assumptions it makes. Recognize the observable and latent variables in a topic model, and specifically in LDA. Explain the generative process of LDA, and derive the complete probability. Explain what inference means in a mixture model, and why it is hard in LDA. Find the approximate posterior distribution of LDA using variational inference, and explain the procedure to find the optimal variational parameters. Explain what it means to “fit” an LDA model to a corpus, and describe how this procedure works. Be able to write code for an LDA model, including training and inference.Anderson Acceleration for FixedPoint Iteration20220413T00:00:0007:0020220413T00:00:0007:00https://joshnguyen.net/posts/andersonacceleration<p>In the first ever post on my website (yay!), I will introduce you to the Anderson acceleration method in fixedpoint iteration. It accompanies our paper <a href="https://joshnguyen.net/publication/202108FedAA"><em>“Accelerating Federated Edge Learning”</em></a>. The code can be found <a href="https://github.com/joshnguyen99/anderson_acceleration">this repository</a>.</p>
<h2 id="fixedpointiteration">FixedPoint Iteration</h2>
<p>Let $g: \mathbb{R}^d \rightarrow \mathbb{R}^d$ be an affine function of the form $g(x) = Ax + b$, where $A \in \mathbb{R}^{d \times d}$ and $b \in \mathbb{R}^d$. We would like to find a <em>fixed point</em> of $g$, which is a vector $x^\ast$ such that $g(x^\ast) = x^\ast$. The reason why $x^\ast$ is called a fixed point is because applying $g$ to $x^\ast$ doesn’t change itself.</p>
<p>The analytical solution to this problem is $x^\ast = (A  I)^{1} b$, but there may be several issues to this. First, $A  I$ may not be invertible, in which case we need to use least squares to find its pseudoinverse. Second, even if it is invertible, the cost of solving for $x^\ast$ is $O(d^3)$, where $d$ is the dimensionality $d$, which is very costly in high dimensions.</p>
<p>The common numerical method to solve for a fixed point of $g$ is the <em>fixedpoint iteration</em>. Start with a randomly chosen $x_0$ and iteratively apply $g$ to it:</p>
\[\label{eqn:fixed_point}
x_{t+1} = g(x_t),\]
<p>until $\lVert g(x_{t+1})  x_{t+1} \rVert \lt \epsilon$ for some predetermined precision $\epsilon$. In order for this to converge, we want to ensure that $g$ is a contraction mapping, that is, there exists an $L \in [0, 1)$ such that $\forall x, x’ \in \mathbb{R}^d, \lVert g(x)  g(x’) \rVert \leq L \lVert x  x’ \rVert$. This can be achieved when the <em>spectral radius</em> of $A$ is less than $1$.</p>
<p>We can prove that to achieve a precision of $\epsilon$, we need to apply $O\left(\kappa \log \frac{1}{\epsilon} \right)$ iterations, where $\kappa$ is the <em>condition number</em> of $A$, which is the ratio between $A$’s largest and smallest singular values.</p>
<h2 id="andersonacceleration">Anderson Acceleration</h2>
<p>Fixedpoint iteration could converge very slowly. The reason is that the condition number of $A$ could be large. (In real datasets, $\kappa$ could be greater than $10^6$.) Anderson acceleration (AA) can speed up convergence considerably. Here’s how it works.</p>
<p>Define $f_t = g(x_t)  x_t$ to be the <em>residual</em> at iteration $t$. To find $x_{t+1}$, consider the space spanned by the previous $m_t+1$ iterates $\{x_{t  m_t}, x_{t  m_t + 1}, \ldots, x_t \}$, where $m_t$ is the <em>window size</em> you can choose. To find the next iterate, we consider a linear combination of these previous vectors:</p>
\[\label{eqn:linear_comb}
\bar{x}_t = \sum_{i=1}^{m_t} \alpha_i^{(t)} x_{t  m_t + i},\]
<p>and find $\alpha^{(t)} \in \mathbb{R}^{m_t + 1}$ such that \(\ g(\bar{x}_t)  \bar{x}_t \\) is minimized. So what are doing here to use the previous iterates to better guide us to the solution. You can check the paper for a full derivation, but the $\alpha^{(t)}$ we should choose is</p>
\[\label{eqn:alpha}
\alpha^{(t)} = \frac{(F_t^\top F_t)^{1}\boldsymbol{1}}{\boldsymbol{1}^\top (F_t^\top F_t)^{1} \boldsymbol{1}},\]
<p>where $F_t = \left[ f_{t m_t},\ldots, f_{t} \right] \in \mathbb{R}^{d \times (m_t + 1)}$ is the matrix of all residuals, $\boldsymbol{1}$ is the $(m_t + 1)$dimensional column vector of all ones.</p>
<p>After finding $\alpha^{(t)}$, we set the new iterate to</p>
\[\label{eqn:extrapolate}
x_{t+1} = \beta \sum_{i=0}^{m_t} \alpha_i^{(t)} g(x_{t  m_t + i}) + (1  \beta) \sum_{i=0}^{m_t} \alpha_i^{(t)} x_{t  m_t + i},\]
<p>where $\beta \in [0, 1]$ is a predetermined <em>mixing parameter</em>.</p>
<h3 id="regularization">“Regularization”</h3>
<p>You can see in the paper that in Algorithm 1, we actually set $\alpha^{(t)}$ as</p>
\[\label{eqn:alpha_reg}
\alpha^{(t)} = \frac{(F_t^\top F_t + \lambda I)^{1}\boldsymbol{1}}{\boldsymbol{1}^\top (F_t^\top F_t + \lambda I)^{1} \boldsymbol{1}},\]
<p>which is slightly different from \eqref{eqn:alpha}. The reason is we want to solve the regularized version of the problem</p>
\[\underset{\alpha^{(t)}: \boldsymbol{1}^\top \alpha^{(t)} = 1}{\min} \ g(\bar{x}_t)  \bar{x}_t \^2 + \lambda \ \alpha^{(t)} \^2\]
<p>for stability (Section II). Without regularization ($\lambda = 0$), we recover \eqref{eqn:alpha}.</p>
<h3 id="thealgorithm">The algorithm</h3>
<p>Anderson acceleration is very similar to the vanilla fixedpoint iteration: start with some $x_0$. In each iteration, find $\alpha^{(t)}$ like above, and <em>extrapolate</em> from the $m_t + 1$ previous iterates to find the next iterate $x_{t+1}$. In other words, in each iteration $t$:</p>
<ul>
<li>Calculate $g(x_t)$.</li>
<li>Compute the residual: $f_t = g(x_t)  x_t$.</li>
<li>Form the residual matrix: $F_t = \left[ f_{t m_t},\ldots, f_{t} \right]$.</li>
<li>Solve for $\alpha^{(t)}$ according to \eqref{eqn:alpha_reg}.</li>
<li>Extrapolate from $m_t + 1$ previous iterates according to \eqref{eqn:extrapolate}.</li>
</ul>
<h2 id="pythonimplementationofaa">Python Implementation of AA</h2>
<p>You can find the implementation in the <a href="https://github.com/joshnguyen99/anderson_acceleration/blob/main/aa.py">aa.py</a> file. The <code class="languageplaintext highlighterrouge">AndersonAcceleration</code> class should be in instantiated with the <code class="languageplaintext highlighterrouge">window_size</code> ($m_t$, defaulted to $5$) and <code class="languageplaintext highlighterrouge">reg</code> ($\lambda$, defaulted to 0). Here’s an example.</p>
<div class="languagepython highlighterrouge"><div class="highlight"><pre class="highlight"><code><span class="o">>>></span> <span class="kn">import</span> <span class="nn">numpy</span> <span class="k">as</span> <span class="n">np</span>
<span class="o">>>></span> <span class="kn">from</span> <span class="nn">aa</span> <span class="kn">import</span> <span class="n">AndersonAcceleration</span>
<span class="o">>>></span> <span class="n">acc</span> <span class="o">=</span> <span class="n">AndersonAcceleration</span><span class="p">(</span><span class="n">window_size</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">reg</span><span class="o">=</span><span class="mi">0</span><span class="p">)</span>
<span class="o">>>></span> <span class="n">x</span> <span class="o">=</span> <span class="n">np</span><span class="p">.</span><span class="n">random</span><span class="p">.</span><span class="n">rand</span><span class="p">(</span><span class="mi">100</span><span class="p">)</span> <span class="c1"># some iterate
</span><span class="o">>>></span> <span class="n">x_acc</span> <span class="o">=</span> <span class="n">acc</span><span class="p">.</span><span class="nb">apply</span><span class="p">(</span><span class="n">x</span><span class="p">)</span> <span class="c1"># accelerated from x
</span></code></pre></div></div>
<p>You will need to apply $g$ to $x_t$ first. The result $g(x_t)$ should be the input to <code class="languageplaintext highlighterrouge">acc.apply</code>, which will solve for $\alpha^{(t)}$ and extrapolate to find $x_{t+1}$. See <a href="https://github.com/joshnguyen99/anderson_acceleration">the repository</a> for more detail.</p>
<h2 id="somenumericalexamples">Some numerical examples</h2>
<h3 id="minimizingaconvexquadraticobjective">Minimizing a convex quadratic objective</h3>
<p>We will minimize a strictly convex quadratic objective. Check <a href="https://github.com/joshnguyen99/anderson_acceleration/blob/main/quadratic_example.ipynb"><code class="languageplaintext highlighterrouge">quadratic_example.ipynb</code></a> for more detail. The below plot shows the <em>optimality gap</em> between $f(x_t)$ and $f(x^\ast)$ over $t$. AA with a window size of 2 converges much faster than the vanilla gradient descent (GD).</p>
<p align="center">
<img src="https://github.com/joshnguyen99/anderson_acceleration/raw/main/AA_GD_quadratic.png" title="Comparing GD to AA on a quadratic objective with very high condition number" />
</p>
<h3 id="minimizingaconvexnonquadraticobjective">Minimizing a convex nonquadratic objective</h3>
<p>We will minimize the $\ell_2$regularized cross entropy loss function for logistic regression. Check <a href="logistic_regression_example.ipynb"><code class="languageplaintext highlighterrouge">logistic_regression_example.ipynb</code></a> for more detail. Similarly, AA is much more favorable than the vanilla GD when optimizing this objective.</p>
<p align="center">
<img src="https://github.com/joshnguyen99/anderson_acceleration/raw/main/AA_GD_logistic_regression.png" title="Comparing GD to AA on a nonquadratic objective with very high condition number" />
</p>Josh Nguyenjosh.nguyen@anu.edu.auIn the first ever post on my website (yay!), I will introduce you to the Anderson acceleration method in fixedpoint iteration. It accompanies our paper “Accelerating Federated Edge Learning”. The code can be found this repository. FixedPoint Iteration Let $g: \mathbb{R}^d \rightarrow \mathbb{R}^d$ be an affine function of the form $g(x) = Ax + b$, where $A \in \mathbb{R}^{d \times d}$ and $b \in \mathbb{R}^d$. We would like to find a fixed point of $g$, which is a vector $x^\ast$ such that $g(x^\ast) = x^\ast$. The reason why $x^\ast$ is called a fixed point is because applying $g$ to $x^\ast$ doesn’t change itself. The analytical solution to this problem is $x^\ast = (A  I)^{1} b$, but there may be several issues to this. First, $A  I$ may not be invertible, in which case we need to use least squares to find its pseudoinverse. Second, even if it is invertible, the cost of solving for $x^\ast$ is $O(d^3)$, where $d$ is the dimensionality $d$, which is very costly in high dimensions. The common numerical method to solve for a fixed point of $g$ is the fixedpoint iteration. Start with a randomly chosen $x_0$ and iteratively apply $g$ to it: \[\label{eqn:fixed_point} x_{t+1} = g(x_t),\] until $\lVert g(x_{t+1})  x_{t+1} \rVert \lt \epsilon$ for some predetermined precision $\epsilon$. In order for this to converge, we want to ensure that $g$ is a contraction mapping, that is, there exists an $L \in [0, 1)$ such that $\forall x, x’ \in \mathbb{R}^d, \lVert g(x)  g(x’) \rVert \leq L \lVert x  x’ \rVert$. This can be achieved when the spectral radius of $A$ is less than $1$. We can prove that to achieve a precision of $\epsilon$, we need to apply $O\left(\kappa \log \frac{1}{\epsilon} \right)$ iterations, where $\kappa$ is the condition number of $A$, which is the ratio between $A$’s largest and smallest singular values. Anderson Acceleration Fixedpoint iteration could converge very slowly. The reason is that the condition number of $A$ could be large. (In real datasets, $\kappa$ could be greater than $10^6$.) Anderson acceleration (AA) can speed up convergence considerably. Here’s how it works. Define $f_t = g(x_t)  x_t$ to be the residual at iteration $t$. To find $x_{t+1}$, consider the space spanned by the previous $m_t+1$ iterates $\{x_{t  m_t}, x_{t  m_t + 1}, \ldots, x_t \}$, where $m_t$ is the window size you can choose. To find the next iterate, we consider a linear combination of these previous vectors: \[\label{eqn:linear_comb} \bar{x}_t = \sum_{i=1}^{m_t} \alpha_i^{(t)} x_{t  m_t + i},\] and find $\alpha^{(t)} \in \mathbb{R}^{m_t + 1}$ such that \(\ g(\bar{x}_t)  \bar{x}_t \\) is minimized. So what are doing here to use the previous iterates to better guide us to the solution. You can check the paper for a full derivation, but the $\alpha^{(t)}$ we should choose is \[\label{eqn:alpha} \alpha^{(t)} = \frac{(F_t^\top F_t)^{1}\boldsymbol{1}}{\boldsymbol{1}^\top (F_t^\top F_t)^{1} \boldsymbol{1}},\] where $F_t = \left[ f_{t m_t},\ldots, f_{t} \right] \in \mathbb{R}^{d \times (m_t + 1)}$ is the matrix of all residuals, $\boldsymbol{1}$ is the $(m_t + 1)$dimensional column vector of all ones. After finding $\alpha^{(t)}$, we set the new iterate to \[\label{eqn:extrapolate} x_{t+1} = \beta \sum_{i=0}^{m_t} \alpha_i^{(t)} g(x_{t  m_t + i}) + (1  \beta) \sum_{i=0}^{m_t} \alpha_i^{(t)} x_{t  m_t + i},\] where $\beta \in [0, 1]$ is a predetermined mixing parameter. “Regularization” You can see in the paper that in Algorithm 1, we actually set $\alpha^{(t)}$ as \[\label{eqn:alpha_reg} \alpha^{(t)} = \frac{(F_t^\top F_t + \lambda I)^{1}\boldsymbol{1}}{\boldsymbol{1}^\top (F_t^\top F_t + \lambda I)^{1} \boldsymbol{1}},\] which is slightly different from \eqref{eqn:alpha}. The reason is we want to solve the regularized version of the problem \[\underset{\alpha^{(t)}: \boldsymbol{1}^\top \alpha^{(t)} = 1}{\min} \ g(\bar{x}_t)  \bar{x}_t \^2 + \lambda \ \alpha^{(t)} \^2\] for stability (Section II). Without regularization ($\lambda = 0$), we recover \eqref{eqn:alpha}. The algorithm Anderson acceleration is very similar to the vanilla fixedpoint iteration: start with some $x_0$. In each iteration, find $\alpha^{(t)}$ like above, and extrapolate from the $m_t + 1$ previous iterates to find the next iterate $x_{t+1}$. In other words, in each iteration $t$: Calculate $g(x_t)$. Compute the residual: $f_t = g(x_t)  x_t$. Form the residual matrix: $F_t = \left[ f_{t m_t},\ldots, f_{t} \right]$. Solve for $\alpha^{(t)}$ according to \eqref{eqn:alpha_reg}. Extrapolate from $m_t + 1$ previous iterates according to \eqref{eqn:extrapolate}. Python Implementation of AA You can find the implementation in the aa.py file. The AndersonAcceleration class should be in instantiated with the window_size ($m_t$, defaulted to $5$) and reg ($\lambda$, defaulted to 0). Here’s an example. >>> import numpy as np >>> from aa import AndersonAcceleration >>> acc = AndersonAcceleration(window_size=2, reg=0) >>> x = np.random.rand(100) # some iterate >>> x_acc = acc.apply(x) # accelerated from x You will need to apply $g$ to $x_t$ first. The result $g(x_t)$ should be the input to acc.apply, which will solve for $\alpha^{(t)}$ and extrapolate to find $x_{t+1}$. See the repository for more detail. Some numerical examples Minimizing a convex quadratic objective We will minimize a strictly convex quadratic objective. Check quadratic_example.ipynb for more detail. The below plot shows the optimality gap between $f(x_t)$ and $f(x^\ast)$ over $t$. AA with a window size of 2 converges much faster than the vanilla gradient descent (GD). Minimizing a convex nonquadratic objective We will minimize the $\ell_2$regularized cross entropy loss function for logistic regression. Check logistic_regression_example.ipynb for more detail. Similarly, AA is much more favorable than the vanilla GD when optimizing this objective.