On this page
Word Frequency Matrix Sort List
Compare word and phrase counts across texts. Your text is processed in this browser and is not uploaded.
1. Add texts to compare
Each text becomes a matrix column. Add up to 8 documents or drop plain-text files below.
2. Analysis options
3. Frequency matrix
Enter text to see results.Share is each row's percentage of all counted occurrences. Document frequency is the number of non-empty documents containing the term.
Privacy: analysis happens locally in your browser tab. Text is not sent to a server or saved by this page. Closing or refreshing the tab clears the current inputs.
Introduction: what a frequency list can reveal
A text contains more than its sentences. It also contains a pattern of repetition: some terms return often, while others appear once or not at all. A carefully built frequency list makes that pattern visible. It can help an editor notice an overused phrase, a researcher compare vocabulary between interviews, a student examine a novel, or a language learner see which words deserve attention first. Counts are simple, but they become useful only when the collection, token rules, and question are clear.
This guide explains how a word-frequency matrix works, what it can and cannot tell you, and how to interpret results without mistaking a count for meaning. It covers text preparation, comparison across documents, phrase counts, common analytical pitfalls, and practical examples. The central idea is modest: frequency describes observed occurrences in a defined sample. It does not by itself prove importance, quality, intent, or causation.
What is a word frequency matrix sort list?
A word frequency matrix is a table that records how often each selected term appears in one or more documents. A single-text list has a row for each term and one count column. A multi-document matrix adds one count column per text, then often includes a corpus total, the number of documents containing the term, and a percentage. “Corpus” simply means the collection of texts being analyzed. This format is also commonly called a document-term matrix or term-document matrix; the orientation of rows and columns can vary.
For example, imagine two short product reviews. The word “battery” might occur four times in the first review and once in the second. The matrix records those separate counts as well as the combined total. A sortable list puts the most frequent terms first or lets the reader inspect terms alphabetically. The table is a representation of counts, not a summary of the documents: it forgets sentence order, syntax, speaker identity unless documents are separated, and the context around every occurrence.
In text analysis, the unit being counted is called a token, while a distinct normalized form is often called a type. If “green green grass” is analyzed, it contains three word tokens and two distinct types. A frequency distribution maps each type to its count. The NLTK reference describes this as recording how often each sample, such as a word type in a document, occurs. That distinction matters: total tokens and vocabulary size answer different questions.
How word counting works
A basic analysis follows a sequence. First, gather the documents and decide what belongs in the sample. Second, split text into tokens according to a rule. Third, optionally normalize tokens, for example by converting case or handling diacritics. Fourth, count each resulting term in each document. Finally, calculate totals or proportions and sort or filter the rows. The arithmetic is straightforward; choices made before counting often have a larger effect on the final result.
Tokenization is the act of determining where one word or feature ends and another begins. Spaces alone are not enough for every language or situation. Punctuation, apostrophes, hyphens, abbreviations, emoji, writing systems without spaces, and mixed scripts all raise questions. The Unicode Standard Annex on text segmentation specifies default boundaries for words and other text units, while noting that language-specific rules may be needed. Even a standards-based default is a practical convention, not a universal theory of what a word is.
Normalization decides whether two spellings should be treated as the same counting key. Lowercasing merges “Tree” and “tree,” which is helpful for many broad summaries but may erase meaningful capitalization in names or acronyms. Unicode also allows some visually equivalent text to be represented by different sequences of code points. Unicode Normalization Forms define ways to represent canonically or compatibly equivalent strings in consistent forms. Accent removal is a stronger analytical choice: it can make searching easier, but in languages where accents distinguish words, it can collapse meaningful contrasts. Keep the original data and document your choice.
For each term t in document d, a raw count is the number of times that term is observed in d. A relative frequency divides that count by a denominator, commonly the number of counted tokens in the document. Multiplying by 100 expresses it as a percentage. If “river” occurs 12 times in 1,200 counted tokens, its relative frequency is 1 percent. A percentage helps compare documents of different lengths, although the underlying counts should remain available because normalization changes what the number means.
When terms from several texts are combined, the result is often called a document-term matrix. The scikit-learn documentation defines its CountVectorizer as converting a collection of text documents to a matrix of token counts. Its documented options illustrate how choices such as lowercasing, token patterns, n-grams, stop words, and binary presence versus raw counts change the feature representation. The particular choices in any browser tool may differ, so read its rules rather than assuming that every frequency list uses identical tokenization.
Benefits and useful features
The first benefit is visibility. A count list replaces an impression such as “I use this phrase too much” with a checkable observation. It can direct attention to recurring vocabulary, reveal terms that differ between drafts, and help prioritize passages for human review. A matrix adds comparison: a term may be common in one document and absent in another, or appear in every text at a low rate. These are different patterns that a single corpus-wide total can hide.
Useful capabilities include case handling, number inclusion, optional stop-word filtering, minimum-count thresholds, sorting, search, and export. Multi-document columns make side-by-side inspection possible. Percentages provide a length-aware view, while document frequency answers how widely a term is distributed across the collection. Phrase counts add local context by counting adjacent sequences such as “climate change.” Downloadable CSV files can be opened in spreadsheet software; JSON can preserve a structured representation for later processing.
Phrase counts are usually called n-grams. A unigram is one token, a bigram is a sequence of two adjacent tokens, and a trigram is a sequence of three. The feature extraction documentation for scikit-learn describes n-gram ranges as a way to include sequences of different lengths. N-grams can surface collocations or repeated expressions, but they multiply the number of features quickly. “Adjacent” is important: the phrase “bright blue sky” contains the bigrams “bright blue” and “blue sky,” not “bright sky.”
Simple counts are interpretable and useful for first-pass exploration. For more specialized tasks, analysts may use measures such as term frequency–inverse document frequency (TF-IDF), which downweights terms that appear in many documents and emphasizes terms more concentrated in a smaller subset. TF-IDF is not the same as a raw frequency matrix. It answers a different question, depends on corpus composition and formula choices, and should not be presented as an objective measure of a term's human importance.
Real-world examples
Comparing interviews
A qualitative researcher may prepare separate interview transcripts and compare vocabulary across participants. A matrix can show that “schedule” appears repeatedly in one interview while “transport” is distributed across several. This can guide a return to the transcript, where the researcher reads complete passages and considers who said the word, what question prompted it, whether a negation changed its sense, and whether a transcription error is present. Counts support interpretation; they do not replace coding, consent, or careful reading.
Editing a manuscript
An author can compare chapter drafts to find repeated technical terms or a phrase that appears much more often in one section. Before editing, the writer should distinguish intentional repetition from accidental overuse. Repetition can provide cohesion, clarify a central concept, or preserve consistent terminology. A word list may also flag names, quoted language, or scene-specific vocabulary that is expected to recur. The best next step is to inspect sentences, not blindly replace every high-ranking term with a synonym.
Studying vocabulary
A teacher can compare reading passages to identify terms shared across units and terms introduced in one lesson. Learners can build a study list from unfamiliar words, then use definitions and example sentences from the original text. A frequency ranking can help allocate attention, but it is not a complete curriculum: a rare word may be essential to the subject, and a common function word may contribute little to a learner's immediate goals. Difficulty, relevance, and prior knowledge matter too.
Reviewing customer feedback
A small team can separate feedback by product version or month and look for changing vocabulary. A rise in “setup” or “refund” may suggest topics worth reviewing in context. But a count alone does not say whether the mention is positive, negative, sarcastic, or merely quoted from a prompt. Teams should check representative comments, preserve the source text, and be cautious about privacy when feedback includes personal details.
Common mistakes and how to avoid them
- Comparing raw counts from unequal texts. A longer file naturally has more chances to contain every word. Compare relative frequencies or use an explicit normalization strategy, and still report the document sizes.
- Ignoring tokenization rules. “can't,” “can’t,” and “cant” may be treated differently. Hyphenated words can split or remain joined. State how contractions, punctuation, and compounds were handled.
- Removing stop words automatically. Common words may be central to a question. In a study of negation, removing “not” can reverse the apparent message. Use a stop list only when it serves a defined purpose, and inspect what it removes.
- Assuming more frequency means more importance. Repeated words may be generic, copied from a prompt, or artifacts of document length. Importance requires context and a defensible research question.
- Combining documents too early. A corpus total can conceal disagreement between sources. Keep document-level counts until you know that aggregation is appropriate.
- Overinterpreting tiny samples. A word appearing twice in a short note can look dominant by percentage. Report the denominator and avoid generalizing from a narrow collection.
- Using accent folding without checking the language. Diacritics may distinguish lexical items. Treat folding as a convenience for a specific comparison, not as harmless cleanup.
- Treating a frequency list as sentiment analysis. Positive and negative meaning depend on context, negation, target, and irony. A word count does not classify opinions reliably.
- Forgetting duplicate or boilerplate text. Headers, disclaimers, signatures, and repeated survey questions can dominate counts. Decide whether these parts belong in the analysis and record any exclusions.
Professional tips for better results
Start with a question that can be answered by counts. “Which words are frequent in this set?” is measurable. “What do people really think?” requires more than a word matrix. Define the unit of comparison—document, speaker, date range, or section—before collecting text. Save an untouched copy of each source and create a separate analysis copy if cleaning is needed. This makes it possible to reproduce and audit decisions.
Make samples comparable. If one document is a full report and another is a short abstract, a raw-count ranking will mostly reflect length. Report token totals, compare percentages when appropriate, and consider whether the texts have similar genres and purposes. A relative rate is still not a magic fix: sparse terms can produce unstable percentages, and different vocabularies may reflect topics rather than style.
Inspect both the top and the tail. The most frequent terms often include grammatical words and topic vocabulary. Rare terms can expose names, typos, specialized concepts, or one-off quotations. A term that occurs once is sometimes called a hapax in corpus studies, but its presence should not be treated as automatically meaningful. Check spelling variants and decide whether inflected forms should remain separate or be grouped through lemmatization. Lemmatization requires language-aware linguistic tools; a simple suffix removal is not equivalent.
Use stop words thoughtfully. There is no universal list that suits every research question or language. A word frequent in ordinary prose may be analytically important in legal, medical, or conversational data. Keep a record of the stop list, apply it consistently, and compare a filtered view with the unfiltered counts. If the outcome changes substantially, explain the choice rather than hiding it.
For cross-document comparisons, inspect both the number of documents containing a term and its within-document rate. A term repeated many times in one file differs from a term mentioned once in every file. Document frequency helps distinguish concentration from spread, but it too depends on how documents were divided. Splitting one long interview into many short files can change document-frequency values without changing the underlying words.
When n-grams are enabled, set the maximum phrase length to match the question. Longer phrases create sparse matrices and many one-off items. A bigram can reveal a stable expression, while a long phrase may simply reproduce part of a sentence. Check several concordance lines—sentences or contexts where a term occurs—before deciding what a repeated sequence signifies.
Preserve provenance. Note where texts came from, when they were collected, how they were cleaned, and which token and normalization settings were used. If the analysis informs a report, include enough method details for another person to repeat it. Do not expose private or confidential text in exports or shared spreadsheets. Browser-local processing reduces transmission risk for this page, but it does not replace your organization's security requirements or the protections of the device itself.
Finally, treat the result as an exploratory aid. Test a few expected examples manually. If a known word is missing, investigate encoding, filters, token boundaries, or the search setting. If totals differ from another program, compare both programs' definitions before assuming one is wrong. Reproducibility comes from explicit choices, not from a particular software brand.
Frequently Asked Questions
1. What does a word frequency matrix show?
It shows how many times each counted term appears in each document, often with totals, percentages, and document coverage. It does not preserve sentence order or meaning.
2. What is the difference between a token and a unique word?
A token is an occurrence in the text. A unique word is a distinct form after the selected normalization rules. A word repeated ten times contributes ten tokens but one unique term.
3. How is percentage frequency calculated?
Divide a term's count by the total number of counted occurrences in the selected denominator, then multiply by 100. Always identify whether the denominator is one document or the whole collection.
4. Why do different tools report different counts?
They may tokenize punctuation, apostrophes, hyphens, numbers, Unicode, and contractions differently. They may also lowercase terms, remove stop words, or normalize accents differently.
5. Should I count uppercase and lowercase separately?
For broad vocabulary analysis, case-insensitive counting is often useful. Keep case distinctions when capitalization itself matters, such as named entities, acronyms, titles, or sentence-position studies.
6. What are stop words?
Stop words are terms an analyst chooses to exclude, often very common function words. They are task-dependent; removing them is inappropriate when those words carry the signal under study.
7. What is an n-gram?
An n-gram is a contiguous sequence of n tokens. A unigram has one token, a bigram two, and a trigram three. N-grams can capture common phrases but increase the feature count.
8. Does a high count mean a term is important?
Not necessarily. High frequency may reflect document length, boilerplate, a generic word, or a topic. Importance depends on context, purpose, and evidence beyond raw count.
9. Can I compare texts of different lengths?
Yes, but do not rely only on raw counts. Compare relative frequencies, retain the original token totals, and consider whether the texts are otherwise comparable.
10. What is document frequency?
Document frequency counts how many documents contain a term at least once. It differs from total frequency, which counts all occurrences, including repeats within one document.
11. Does this analysis understand word meaning?
No. A frequency matrix counts surface forms under chosen rules. It does not understand context, sentiment, sarcasm, topic, or whether two different words share a meaning.
12. What does accent folding do?
It can remove combining marks so some accented forms share a key with unaccented forms. This can aid broad matching, but it may merge distinct words and should be optional.
13. Is a frequency matrix the same as TF-IDF?
No. A count matrix contains term counts. TF-IDF reweights a term using its distribution across documents, emphasizing terms that are comparatively less common across the collection.
14. Is my text uploaded when I use this page?
This tool's analysis runs in the browser and does not send the input to a server. Your browser, device, extensions, or the site hosting the page have separate privacy and security considerations.
15. How can I make results reproducible?
Keep the original texts and record the document boundaries, tokenization, case and accent settings, stop words, phrase length, and any exclusions. Use the same rules when rerunning the analysis.
Conclusion
A word frequency matrix turns repeated language into a form that can be inspected and compared. Its strengths are transparency, speed, and a clear link between observed text and numeric counts. Its limitations are equally important: the result depends on how text is segmented and normalized, and a table cannot recover the context it leaves out. Define the sample, choose rules that fit the question, compare documents responsibly, and return to the original passages before drawing conclusions. Used that way, a sorted frequency list is a practical starting point for deeper reading rather than a substitute for it.
Sources and Further Reading
- Unicode Consortium: Unicode Text Segmentation (UAX #29) — default word-boundary rules and language-specific considerations.
- Unicode Consortium: Unicode Normalization Forms (UAX #15) — canonical and compatibility equivalence and normalization.
- scikit-learn: CountVectorizer documentation — token-count matrices, tokenization, n-grams, and stop-word options.
- NLTK: Frequency Distribution API — counts, total outcomes, vocabulary size, and relative frequency.
- Piantadosi (2014): “Zipf’s word frequency law in natural language” — peer-reviewed critical review of word-frequency distributions.