TF-IDF Weight & Cosine Similarity Calculator: Free TF-IDF Weight & Cosine Similarity Calculator tool: calculate Term Frequency-Inverse Document Frequency (TF-IDF) weights and document cosine.
The Foundations of Vector Space Modeling & TF-IDF
In Natural Language Processing (NLP) and Information Retrieval (IR), converting unstructured human text into structured mathematical vectors is a fundamental prerequisite for search engines, document clustering, and machine learning classifiers. First introduced by British computer scientist Karen Spärck Jones in 1972 and later popularized by Gerard Salton in the SMART Information Retrieval System, TF-IDF remains one of the most effective and computationally efficient feature extraction algorithms in computer science.
TF-IDF strikes an elegant balance between two competing statistical properties:
- Term Frequency ($text{TF}$): Measures how frequently a word occurs inside a given document. A document mentioning "quantum" five times is more likely to be about quantum physics than a document mentioning it once.
- Inverse Document Frequency ($text{IDF}$): Measures how informative or rare that word is across the broader corpus. Words like "the", "system", or "is" appear in virtually every document and therefore carry virtually zero discriminative meaning. Conversely, rare domain-specific words carry immense semantic value.
The Mathematical Formulas: TF, IDF, and Cosine Similarity
Given a collection of $N$ documents and a vocabulary of terms, the components are defined as:
Relative Term Frequency: $text{TF}(t, d) = frac{f(t, d)}{sum_{t' in d} f(t', d)}$
scikit-learn Smooth IDF: $text{IDF}(t) = lnleft(frac{1 + N}{1 + text{DF}(t)}right) + 1$
Standard Classic IDF: $text{IDF}(t) = lnleft(frac{N}{text{DF}(t)}right)$
TF-IDF Weight: $W(t, d) = text{TF}(t, d) times text{IDF}(t)$
Vector Cosine Similarity: $text{Cosine}(A, B) = frac{A cdot B}{|A| |B|} = frac{sum_{i=1}^V A_i B_i}{sqrt{sum_{i=1}^V A_i^2} sqrt{sum_{i=1}^V B_i^2}}$
Step-by-Step Worked Numerical Example
Consider a minimal corpus of $N = 3$ documents:
- Doc 1: "deep learning algorithms train neural networks" (5 words)
- Doc 2: "neural networks require high performance gpu clusters" (6 words)
- Doc 3: "cloud gpu clusters provide scalable computing" (6 words)
Let us compute the TF-IDF weight of the term "learning" in Doc 1 using scikit-learn smooth IDF:
Step 1: Compute Term Frequency ($text{TF}$):
Frequency in Doc 1 = 1 count
Total words in Doc 1 = 5
$text{TF}(text{"learning"}, D_1) = frac{1}{5} = mathbf{0.20}$
Step 2: Compute Inverse Document Frequency ($text{IDF}$):
Corpus size ($N$) = 3
Document Frequency ($text{DF}$) = 1 (appears only in Doc 1)
$text{IDF} = lnleft(frac{1 + 3}{1 + 1}right) + 1 = lnleft(frac{4}{2}right) + 1 = ln(2) + 1 approx 0.6931 + 1 = mathbf{1.6931}$
Step 3: Calculate Final TF-IDF Weight:
$W(text{"learning"}, D_1) = 0.20 times 1.6931 = mathbf{0.3386}$
Now consider the term "gpu", which appears in both Doc 2 and Doc 3 ($text{DF} = 2$):
$text{IDF}(text{"gpu"}) = lnleft(frac{1 + 3}{1 + 2}right) + 1 = ln(1.333) + 1 approx 0.2877 + 1 = mathbf{1.2877}$
Notice that "learning" (which appears in only 1 document) receives an IDF weight of 1.6931, while "gpu" (which appears in 2 documents) receives a lower weight of 1.2877. The algorithm automatically determines that "learning" is a more selective and distinctive keyword for Doc 1.
The Geometric Beauty of Cosine Similarity
Why do search engines use Cosine Similarity rather than Euclidean distance ($L_2$ distance)?
In vector space, a document's vector length grows with its word count. If an article mentions a keyword 10 times simply because it is 5,000 words long, its Euclidean distance from a short search query will be enormous.
Cosine similarity projects both the search query and the document into $V$-dimensional space and computes the cosine of the angle $theta$ between their vectors. Because $cos(0^circ) = 1.0$, identical topic distributions achieve a perfect score of 1.0, regardless of whether one document is a tweet and the other is a textbook.