{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Dimensionality reduction of text data\n",
"\n",
"In the Clustering Chapter we briefly introduced how to deal with text data.\n",
"There, we presented the concept of vectorization, where we treat each word as\n",
"a feature (a column), and represent each document as a vector (a row). As this\n",
"process creates as many features as unique words occurring in the data, the\n",
"dimension of the feature space can be very large.\n",
"\n",
"In this notebook we use the Wikinews dataset to explore how to reduce that\n",
"dimensionality for visualization and analysis. This turns out to be trickier\n",
"than in the tabular setting. The heuristics we used before, such as the 90%\n",
"variance threshold and the Kaiser criterion, behave very differently on text\n",
"data and can lead to impractical choices.\n",
"\n",
"We also compare linear and non-linear reduction techniques. Each technique\n",
"tells a different story about the same data, and knowing when to use which is\n",
"one of the goals of this notebook."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"\n",
"data = pd.read_csv(\"../datasets/wiki_news.csv\")\n",
"data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the Clustering Chapter we encoded the text using `CountVectorizer` first,\n",
"then we just mentioned that `skrub.StringEncoder` encodes text similarly to\n",
"`CountVectorizer` but it additionally reduces the dimension of the feature\n",
"space.\n",
"\n",
"In this notebook, we use `TfidfVectorizer` to vectorize the \"text\" column. The\n",
"`min_df` and `max_df` hyperparameters discard any word that appears in fewer\n",
"than 5 documents, or in more than 80% of documents, respectively. The logic is\n",
"that very rare terms may just be typos, proper nouns, or highly specific terms\n",
"that won't generalize across the corpus; whereas words appearing in almost\n",
"every document may not help distinguish one document from another: articles,\n",
"conjunctions, auxiliary verbs, etc."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.feature_extraction.text import TfidfVectorizer\n",
"\n",
"vectorizer = TfidfVectorizer(min_df=5, max_df=0.8)\n",
"data_encoded = vectorizer.fit_transform(data[\"text\"])\n",
"data_encoded"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The output from the cell above tells us that, after discarding terms according\n",
"to the `min_df` and `max_df`, we are left with 6,678 unique terms distributed\n",
"in the 1,250 documents. Most entries are zero, since any given document uses\n",
"only a small fraction of all possible terms; out of the 1250 \u00d7 6678 \u2248 8.3\n",
"million possible entries, only ~190,000 are non-zero. Working directly in this\n",
"6678-dimensional space is computationally expensive and unnecessary, as most\n",
"dimensions carry little information and many terms are correlated (synonyms,\n",
"verb conjugations, etc). Dimensionality reduction, such as PCA, can compress\n",
"this into a much smaller set of directions that capture the dominant patterns\n",
"across documents.\n",
"\n",
"Let's now use `PCA` to keep just 2 dimensions. But first we define a helper\n",
"function that allows us to plot the different categories and explore the data\n",
"structure at a glance."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import textwrap\n",
"import plotly.graph_objects as go\n",
"from sklearn.decomposition import PCA\n",
"\n",
"\n",
"def wrap(text, width=80, max_lines=3):\n",
" lines = textwrap.wrap(text, width)\n",
" if len(lines) > max_lines:\n",
" return \"
\".join(lines[:max_lines]) + \"...\"\n",
" return \"
\".join(lines)\n",
"\n",
"\n",
"def plot_2d_projection(estimator, data, categories_to_plot):\n",
" X_2d = estimator.fit_transform(data_encoded)\n",
"\n",
" fig = go.Figure()\n",
"\n",
" for cat in categories_to_plot:\n",
" idx = data[\"category\"] == cat\n",
" fig.add_trace(\n",
" go.Scatter(\n",
" x=X_2d[idx, 0],\n",
" y=X_2d[idx, 1],\n",
" mode=\"markers\",\n",
" name=cat,\n",
" marker=dict(size=5, opacity=0.6),\n",
" text=data.loc[idx, \"text\"].apply(wrap),\n",
" hovertemplate=\"%{text}\",\n",
" )\n",
" )\n",
"\n",
" estimator_name = type(estimator).__name__\n",
" fig.update_layout(\n",
" title=f\"TF-IDF + {estimator_name} (2D projection)\",\n",
" xaxis_title=\"PC1\",\n",
" yaxis_title=\"PC2\",\n",
" yaxis=dict(scaleanchor=\"x\", scaleratio=1), # set equal axes\n",
" )\n",
" fig.show(renderer=\"notebook\")\n",
"\n",
"\n",
"all_categories = data[\"category\"].unique()\n",
"pca = PCA(n_components=2) # no need to set the random seed, can you tell why?\n",
"plot_2d_projection(pca, data, all_categories)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"All categories crowd near the origin, but \"sport\" and \"tech\" extend into\n",
"distinct regions of the PC space. Their characteristic vocabulary is\n",
"distinctive enough to pull them apart with 2 components.\n",
"\n",
"Remember also that the First Principal Component carries more variance than\n",
"the Second Principal Component, and so on. Because of this, the fact that\n",
"\"sport\" extends largely in the negative PC1 direction suggests sports\n",
"vocabulary is the most distinctive across this corpus of documents and is\n",
"well-captured by that direction. Similarly, \"tech\" is spread in the positive\n",
"PC2 direction. This suggests that the words that define PC2 are\n",
"disproportionately tech-related terms.\n",
"\n",
"\"Business\" and \"entertainment\" stay near the center, suggesting their\n",
"vocabulary is spread across many directions rather than concentrated along the\n",
"first two components. Let's focus on \"entertainment\" and use a pairplot to\n",
"explore whether higher components better capture the vocabulary specific to\n",
"this category."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import pandas as pd\n",
"import seaborn as sns\n",
"import matplotlib.pyplot as plt\n",
"\n",
"\n",
"def set_equal_axes(*args, **kwargs):\n",
" plt.xlim(-lim, lim)\n",
" plt.ylim(-lim, lim)\n",
" plt.gca().set_aspect(\"equal\")\n",
" plt.xticks(rotation=45)\n",
"\n",
"\n",
"n_components = 4\n",
"category_of_interest = \"entertainment\"\n",
"pca.set_params(n_components=n_components)\n",
"X_pca = pca.fit_transform(data_encoded)\n",
"entertainment = pd.DataFrame(\n",
" X_pca[data[\"category\"] == category_of_interest],\n",
" columns=[f\"PC{i + 1}\" for i in range(n_components)],\n",
")\n",
"lim = max(abs(entertainment.values.min()), abs(entertainment.values.max()))\n",
"g = sns.PairGrid(entertainment, corner=True, aspect=1.2)\n",
"g.map_offdiag(sns.scatterplot, alpha=0.6, s=20)\n",
"g.map_diag(sns.histplot)\n",
"g.map_offdiag(set_equal_axes)\n",
"_ = g.figure.suptitle(\n",
" f\"TF-IDF + PCA on {category_of_interest}\\n(first {n_components} components)\"\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The first panel shows PC1 vs PC2, which corresponds to the 2D scatter plot we\n",
"explored earlier with all categories. The cloud is roughly isotropic, with a\n",
"few outliers pulling away from the center. As neither component dominates, PC1\n",
"and PC2 are driven by other categories' vocabulary, and entertainment articles\n",
"are essentially scattered at random along those first directions.\n",
"\n",
"From PC3 onward, the scatter panels show a clear diagonal elongation, meaning\n",
"entertainment articles tend to land on defined regions of these components\n",
"rather than randomly. This correlation between components suggests they\n",
"capture coherent vocabulary patterns within the category. The panels involving\n",
"PC4 show elongation along the PC4 axis, suggesting it is the first component\n",
"to capture variance specific to entertainment.\n",
"\n",
"Now let's get back to our 2D scatter plot to visualize \"sport\" and\n",
"\"entertainment\" together in the first two components."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"categories_to_plot = [\"entertainment\", \"sport\"]\n",
"plot_2d_projection(pca, data, categories_to_plot)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Observe that near the center of the overlap region between \"entertainment\" and\n",
"\"sport\", articles reference the Olympic Games, which naturally belongs to both\n",
"categories. Other articles in that region simply share common terms such as\n",
"city names, that frequently host prominent figures from both categories.\n",
"\n",
"## Choice of `n_components` for text data\n",
"\n",
"So far we have used 2 or just a few components for easier visualization. But\n",
"for a downstream task like clustering, we would want to keep more, as\n",
"additional components better capture the variance coming from the specific\n",
"vocabulary of different categories. In a previous notebook we used the 90%\n",
"variance threshold and the Kaiser criterion to guide the choice of\n",
"`n_components`. Let's see how they behave on text data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"\n",
"n_components = 900\n",
"pca = PCA(n_components=n_components).fit(data_encoded)\n",
"cumvar = np.cumsum(pca.explained_variance_ratio_)\n",
"n_90 = np.searchsorted(cumvar, 0.90) + 1\n",
"n_kaiser = np.sum(pca.explained_variance_ratio_ > 1 / data_encoded.shape[1])\n",
"\n",
"print(\n",
" f\"Cumulated explained variance with {n_components} \"\n",
" f\"components: {cumvar[-1] * 100:.1f}%\"\n",
")\n",
"print(f\"90% variance threshold: {n_90} components\")\n",
"print(f\"Kaiser criterion: {n_kaiser} components\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Both heuristics suggest keeping far more components than is practical. The 90%\n",
"variance threshold already requires on the order of thousands of components.\n",
"The Kaiser criterion, which sets the threshold at 1/6678 explained\n",
"variance, keeps even more: all 900 components we computed pass it, meaning the\n",
"true cutoff lies beyond what we measured.\n",
"\n",
"For text data, a common practice is to fix the number of components to be 100\n",
"or 300. Other values are valid, but a finer grid is unlikely to yield\n",
"meaningfully different results and would add unnecessary computational cost.\n",
"\n",
"Let's now run a K-Means as downstream task. We set `n_clusters=5` to match the\n",
"number of categories we know exist in this dataset. This is similar to what we\n",
"did in the Clustering chapter, except that now we can use PCA's inverse\n",
"transform to project the cluster centroids back into the TF-IDF vocabulary\n",
"space. The terms with the highest TF-IDF weights in each centroid are the most\n",
"characteristic words for that cluster, giving us a sanity check on whether the\n",
"clusters are meaningful. In a setting without known labels, inspecting\n",
"centroid terms across different values of `n_clusters` could be a qualitative\n",
"way to evaluate the pipeline."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.cluster import KMeans\n",
"\n",
"n_components = 300\n",
"pca = PCA(n_components=n_components)\n",
"X_pca = pca.fit_transform(data_encoded)\n",
"explained_variance = pca.explained_variance_ratio_.sum()\n",
"kmeans = KMeans(n_clusters=5, n_init=5, random_state=42).fit(X_pca)\n",
"original_space_centroids = pca.inverse_transform(kmeans.cluster_centers_)\n",
"order_centroids = original_space_centroids.argsort()[\n",
" :, ::-1\n",
"] # highest weights\n",
"terms = vectorizer.get_feature_names_out()\n",
"\n",
"print(\n",
" f\"Cumulated explained variance with {n_components} components: {explained_variance * 100:.1f}%\"\n",
")\n",
"for i in range(5):\n",
" print(f\"Cluster {i}: \", end=\"\")\n",
" for ind in order_centroids[i, :10]:\n",
" print(f\"{terms[ind]} \", end=\"\")\n",
" print()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Clusters start to be interpretable, but clusters 1 and 3 belong both to the\n",
"\"sport\" category, as we saw olympic sports have a vocabulary close to category\n",
"\"entertainment\". Clusters 2 and 4 are polluted by common function words such\n",
"as \"is\", \"that\" and \"was\", which carry little to no information. One way to\n",
"address this is to lower `max_df`. However, setting it too aggressively risks\n",
"removing words genuinely carry information. Another approach is to use the\n",
"`stop_words=\"english\"` parameter, which explicitly filters a predefined list\n",
"of common English words. This is generally discouraged (see the [documentation\n",
"for\n",
"stop_words](https://scikit-learn.org/stable/modules/feature_extraction.html#stop-words))\n",
"because the list is arbitrary and hard coded in the scikit-learn\n",
"implementation, but is still a reasonable option to explore here:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"vectorizer = TfidfVectorizer(min_df=5, max_df=0.8, stop_words=\"english\")\n",
"data_encoded = vectorizer.fit_transform(data[\"text\"])\n",
"\n",
"X_pca = pca.fit_transform(data_encoded)\n",
"explained_variance = pca.explained_variance_ratio_.sum()\n",
"kmeans_centroids = kmeans.fit(X_pca).cluster_centers_\n",
"original_space_centroids = pca.inverse_transform(kmeans_centroids)\n",
"order_centroids = original_space_centroids.argsort()[:, ::-1]\n",
"terms = vectorizer.get_feature_names_out()\n",
"\n",
"print(\n",
" f\"Cumulated explained variance with {n_components} components: {explained_variance * 100:.1f}%\"\n",
")\n",
"for i in range(5):\n",
" print(f\"Cluster {i}: \", end=\"\")\n",
" for ind in order_centroids[i, :10]:\n",
" print(f\"{terms[ind]} \", end=\"\")\n",
" print()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This time the centroids contain vocabulary that truely reflects the expected\n",
"categories.\n",
"\n",
"## Non-linear dimensionality reduction\n",
"\n",
"So far we've seen that PCA is a linear method for dimensionality reduction. As\n",
"such, it does not distort the original feature space. It finds straight\n",
"directions in it and projects all samples onto them. We now introduce a\n",
"non-linear technique for dimensionality reduction, known as t-SNE\n",
"(t-distributed Stochastic Neighbour Embedding) which is neighbour-based\n",
"instead. It arranges points in 2D so that each point's nearest neighbours in\n",
"the original space remain its nearest neighbours in the projection. This makes\n",
"it better suited for visualizing local cluster structure, at the cost of\n",
"losing the interpretability of the axes.\n",
"\n",
"In t-SNE, distances between clusters in a t-SNE plot are not meaningful. Two\n",
"clusters appearing far apart in a t-SNE plot does not mean they are dissimilar\n",
"in the original space, and two clusters appearing close does not mean they are\n",
"similar. t-SNE optimizes local neighbourhood structure within clusters but\n",
"does not preserve the global geometry between them."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.manifold import TSNE\n",
"\n",
"tsne = TSNE(n_components=2)\n",
"plot_2d_projection(tsne, data, all_categories)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The clusters are much more clearly separated than in the PCA projection. Sport\n",
"and politics form tight, compact groups, while entertainment and business\n",
"remain more diffuse, consistent with what we observed earlier. Crucially, the\n",
"model had no access to the category labels: this structure emerges purely from\n",
"word co-occurrence patterns. A neighbour-based approach like a news\n",
"recommender system could work directly in this space, since documents that\n",
"land close together are genuinely similar in vocabulary.\n",
"\n",
"Two caveats are worth keeping in mind:\n",
"\n",
"- First, t-SNE cannot be inverted. Unlike PCA, there is no way to map a point\n",
" in the 2D projection back to the original document space. This means t-SNE\n",
" is useful for visualization but cannot serve as a preprocessing step for a\n",
" classifier that needs to generalize to new documents.\n",
"- Second, the layout depends on the initialization. The scikit-learn\n",
" implementation of `TSNE` uses PCA initialization by default, which is more\n",
" globally stable than a random start. The interesting consequence **for this\n",
" particular dataset** is that the PCA initialization is fully deterministic\n",
" regardless of the `random_state` you pass to `TSNE`. This is because the\n",
" automatic solver selection for this data shape (1,250 samples, 6,678\n",
" features, 2 components) follows a deterministic path, so the starting\n",
" configuration does not change across runs. A random initialization with\n",
" different seeds can produce a noticeably different global layout, making\n",
" comparisons across runs harder to interpret.\n",
"\n",
"Finally, some dimensionality reduction algorithms are not included in\n",
"scikit-learn. One of them is [UMAP (Uniform Manifold Approximation and\n",
"Projection)](https://umap-learn.readthedocs.io), which is also neighbour-based\n",
"like t-SNE, but represents the data as a graph, where each document is a node\n",
"connected to its nearest neighbours in the TF-IDF space. UMAP then finds a 2D\n",
"arrangement of those nodes that best preserves the connections in that graph,\n",
"including connections between distant nodes. This means it optimizes for both\n",
"local and global structure simultaneously. In practice, similar documents end\n",
"up placed close together, but clusters that are far apart in the original\n",
"space tend to remain far apart in the projection as well.\n",
"\n",
"Note: UMAP is not available in JupyterLite. To run the following cell, execute\n",
"the notebook locally."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# %pip install umap-learn\n",
"# from umap import UMAP\n",
"\n",
"# umap = UMAP(random_state=42)\n",
"# plot_2d_projection(umap, data, all_categories)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Categories \"sport\" and \"tech\" form the most distinct clusters, while business,\n",
"politics and entertainment overlap more, reflecting their shared vocabulary.\n",
"For this specific random seed, a large space-related blob appears in the\n",
"lower-left of the plot, and a small isolated blob can be distinguished in the\n",
"upper-right corresponds to cycling articles. Its position may vary across\n",
"random seeds, but it tends to remain isolated, as its vocabulary is specific\n",
"enough to pull it away from the rest of the sport articles. It also tends to\n",
"sit close to the Olympic and Paralympic vocabulary cluster, which makes sense\n",
"given how prominently cycling features in those events.\n",
"\n",
"## Take home messages\n",
"\n",
"In this notebook we saw that standard heuristics for choosing the number of\n",
"components break down for text data, and that a fixed budget of 100 to 300\n",
"components is a more practical choice. We also saw that the right\n",
"visualization technique depends on what question you are asking: PCA is\n",
"interpretable and preserves global variance structure, t-SNE reveals local\n",
"cluster structure, and UMAP strikes a balance between the two. Finally,\n",
"inspecting cluster centroids in the original vocabulary space, made possible\n",
"by PCA's inverse transform, gives a useful sanity check on whether the learned\n",
"representation is meaningful. Together, these tools form a practical toolkit\n",
"for exploring and understanding large text corpora."
]
}
],
"metadata": {
"jupytext": {
"main_language": "python"
},
"kernelspec": {
"display_name": "Python 3",
"name": "python3"
}
},
"nbformat": 4,
"nbformat_minor": 5
}