This article is transcoded by 简悦 SimpRead, original address fancyerii.github.io
This module introduces language models. Readers can refer to the Language Model Tutorial. For more articles in this series, please click Microsoft Edx Speech Recognition Course.
Contents
- Introduction
- N-gram Language Model
- Evaluation Methods of Language Models
- Handling N-gram Models
- Advanced Models
- Lab
Introduction
This module introduces the basic concept of language models — the part that calculates the prior probability P(W) in speech recognition systems. Recall the basic formula of speech recognition systems, where we combine the acoustic model probability P(O \vert W) and the language model probability P(W) to find the most likely hypothesis:
Therefore, a language model (abbreviated as LM) contains the prior probability of word sequences, allowing us to calculate it even without hearing the actual speech. Traditional grammars (such as regular grammars and context-free grammars) define which sentences are grammatical and which are not by hardcoded rules (similar to how a C language compiler checks syntax for C programs). Currently, a more mainstream method is to use probabilistic models, which assign higher probabilities to more likely sentences and lower probabilities to less likely ones, although even very rare sentences may have a very small probability (because no one knows exactly what a speaker will say; they may even say completely ungrammatical sentences). Moreover, these probabilities in statistical models are not assigned by linguistic experts. Like acoustic models, we estimate model parameters through large amounts of data. Thus, we estimate which word sequences are more likely under a particular language, scenario, and application based on actual training data statistics.
Note: In language models, a “sentence” refers to a sequence of words in an utterance, which may not be grammatically correct or even complete.
N-gram Language Model
Vocabulary
We need to calculate the probability of each sentence, where a sentence is a sequence of words:
$$W = w_1 w_2 \ldots w_n$$
Here, n is the number of words, theoretically unlimited. To simplify the problem, we first assume a finite number of words, meaning the LM’s vocabulary is a finite set. Note: The LM’s vocabulary is also the speech recognition system’s vocabulary — we cannot recognize words not in the language model vocabulary.
Words outside the vocabulary are called out-of-vocabulary (OOV). The presence of an OOV will cause at least one speech recognition error, so we need to choose an appropriate vocabulary to minimize OOV. A common strategy is to estimate the prior probabilities of each word’s occurrence from data, then select the high-probability words. That is, we select the most frequent words in the training data. The larger the vocabulary, the smaller the OOV and the higher the recognition accuracy (possibly, but not necessarily, because including very rare words can reduce accuracy by confusing normal words). But a too large vocabulary causes the model to be too big, slowing training and decoding.
Markov Decomposition and N-Gram
Even when the vocabulary is finite, we still face the (theoretically) infinite sentence length problem, so we cannot enumerate all possibilities (similar to a probability distribution function). And even if theoretically possible, it is ineffective since many sentences will never appear exactly in the training data.
We can use a trick similar to acoustic modeling: use the chain rule to decompose the sentence probability into a product of conditional probabilities, then apply the Markov assumption to limit the condition (history) to a finite length.
$$P(W) = P(w_1) \times P(w_2 | w_1) \times P(w_3 | w_1 w_2) \times \ldots \times P(w_n | w_1 \ldots w_{n-1})$$
If we enforce the first-order Markov assumption—that the probability of a word depends only on the previous word (history):
$$P(W) = P(w_1) \times P(w_2 | w_1) \times P(w_3 | w_2) \times \ldots \times P(w_n | w_{n-1})$$
Note that each word depends on the preceding word; this is a first-order Markov model. However, in language modeling, we prefer not to use this terminology; here, we call it a bigram model, where only adjacent two words are considered statistically. Similarly, a second-order Markov model corresponds to trigram, which predicts the current word based on the previous two words.
The above method generalizes to the N-gram model, where each word depends on the previous N-1 words. Trigram performs much better than bigram, but higher N’s improvement is less significant, and model size (parameters) grows greatly. Thus, in practice, 4-gram or 5-gram are rarely used. In this module’s experiments, we use trigram; elsewhere, for simplifying concepts, we mostly use bigram as examples. But this theory applies to all N-grams.
w_1
Sentence Start and End
To enable our N-gram language model to assign probability to all finite-length sentences, we face a small problem: how to model sentence ending? One way is to have a model for sentence length n, but a simpler method is to introduce a special sentence-end token </s> to mark sentence end. From a generative model perspective, we first predict the first word, then predict the second from the first (assuming bigram), then the third from the second, …, until we reach </s>, at which point the process ends. Such a model guarantees that the total probabilities of all possible sentences sum to 1, i.e., a valid probability distribution.
However, the problem before that is that predicting the first word has no history; although denoted as p(w_1), it is not the probability of word $w_1$’s occurrence but the probability of w_1 appearing at the first position! To unify notation, we introduce a special sentence start…Starting tag <s>， denoted as p(w_1 \vert <s>), which represents the probability of word w_1 appearing after <s>, that is, the probability of w_1 appearing in the first position.
With these two special “words”, our probability formula becomes simpler:
Probability estimation of N-gram
The conditional probability of an N-gram can be simply estimated by its relative frequency. Suppose c(w_1 \dots w_k) is the count of the k-gram w_1 \dots w_k. Then the probability of “bites” appearing after “dog” is:
$$P(bites| dog) = { c(dog\ bites) \over c(dog) }$$
The reader is asked to prove:
$$P(bites | dog ) + P(bite | dog) + P(wags | dog) + \cdots$$
sums to 1, meaning all probabilities conditioned on “dog” sum to 1, thus guaranteeing a valid probability distribution.
More generally, the probability of a k-gram can be estimated using the formula:
$$P(w_k | w_1 \ldots w_{k-1}) = { c(w_1 \ldots w_k) \over c(w_1 \ldots w_{k-1})}$$
Smoothing and discounting of N-gram
A serious problem with the above relative frequency estimation is that the probability of an N-gram that never appears in the training data is zero. The training data is finite and cannot cover all possible N-grams. Additionally, for speech recognition systems, speakers often think while speaking, and many utterances are not perfectly grammatical.
Therefore, we need to assign nonzero probability to those N-grams that never appear in the training data, which can be achieved through language model smoothing. This is an important subfield of language model research; wiki lists common smoothing methods, most of which are implemented in SRILM. Here we only discuss one method—Witten-Bell smoothing. The reasons for choosing it are twofold: first, it is relatively easy to understand and implement; second, compared to more complex methods, it does not require many additional assumptions about the data distribution, making it more robust.
The basic idea of Witten-Bell smoothing is to include unseen words in the counts. But how many “unseen” words are there? This seems difficult: since they have never appeared, how can we know? Witten-Bell smoothing handles this as follows: in the training data, before a word appears for the first time, it is unseen. Therefore, for the training data, the number of unseen words equals the number of words that appear in the training data. Thus, for 1-gram, we can estimate its probability as:
$$\hat{P}(w)={c(w) \over c(.)+V}$$
Here, c(w) is the count of word w occurrences, c(.) is the total count of all word occurrences (i.e., the length of the training data), and V is the number of distinct words that appear first, i.e., the vocabulary size. Compared to no smoothing, the denominator has an additional V, so the summed probability of all observed words is less than 1. This smoothing is called discounting, where the observed n-gram probabilities are discounted, and the remainder of probability mass is assigned to unseen words. The probability for unseen words is:
$$P(unseenword)={V \over c(.)+V}$$
Note: There are many unseen words; their summed probability is the above value, not each unseen word having that value (otherwise, the total probability would exceed one). The next section explains how unseen words share this probability.
Extension from 1-gram to k-gram is similar:
$$\hat{P}(w_k|w_1…w_{k−1}) = {c(w_1…w_k) \over c(w_1…w_{k−1})+V(w_1…w_{k−1}⋅)}$$
where c(w_1…w_{k-1}) is the count of the k-1 gram, and V(w_1…w_{k−1}⋅) is the number of distinct words following w_1…w_{k−1}.
For example, if the training data is:
a b c a b d a b d
then c(ab)=3, while V(ab⋅)=\vert \{c,d,d\} \vert=2. That is, the words appearing after “ab” are [c, d, d], and distinct words are only [c, d].
Back-off of N-gram
Previously, discounting reserves some probability mass for unseen N-grams, but there are many unseen N-grams—how should we distribute this probability? This section discusses that issue.
A common idea is equal distribution. For example, using the Witten-Bell method introduced before, consider a trigram with the history “white dog”. Suppose the training data only contains “white dog barks” once, and nowhere else is “white dog” seen. Without smoothing, \hat{P}(barked \vert white\ dog)=1, and other \hat{P}(\text{other words} \vert white\ dog)=0. After Witten-Bell discounting,
$$\hat{P}(barked|white\ dog) = {c(white\ dog\ barked) \over c(white\ dog)+V(white\ dog⋅)} = {2\over2+1} = {2\over3}$$
So 1/3 of the probability mass remains for \hat{P}(\text{other words} | white\ dog). The simplest approach is equal division; for example, besides “barks” there are 10 other words, then each gets 1/30, meaning
$$\hat{P}(\text{other words} | white\ dog)={1 \over 30}.$$
This is obviously not ideal. Another intuitive method is to allocate probability according to these 10 words’ unigram frequency, reasoning that frequently appearing words should get more probability. But this approach has a problem: “the” is a high-frequency word but may have low probability following “white dog”; “eat” might have low frequency generally but higher conditional probability after “white dog”.
Therefore, when allocating probability, the history (context) is important: if the probability of “the” after “white dog” is high, it gets higher probability. But if the training data has no “white dog the” occurrence, what to do? Here, reduced context is used: shorten the history “white dog” to “dog”, and allocate the probability based on \hat{P}(the \vert dog) and \hat{P}(eat \vert dog). If “dog the” is still unseen, then \hat{P}(\cdot \vert dog) will itself be discounted, and from this discounted probability, a smaller probability for “the” can again be distributed using unigram \hat{P}(the). This process is recursive: if the trigram is unseen, use bigram for distributing discounted probability; if the bigram is unseen, use unigram to distribute the further discounted probability. This method is called backoff.
[
{\hat{P}}{\text{bo}}\left( w{k} \right|w_{1}\ldots w_{k - 1}) = \ \left{
$$\begin{matrix} \hat{P}\left( w_{k} \right|w_{1}\ldots w_{k - 1}),\ \ \ c(w_{1}\ldots w_{k}) > 0 \ {\hat{P}}{\text{bo}}\left( w{k} \right|w_{2}\ldots w_{k - 1})\ \alpha(w_{2}\ldots w_{k - 1}),\ \ c\left( w_{1}\ldots w_{k} \right) = 0 \ \end{matrix}$$
\right.\]
]
\hat{P}_{bo} is the discounted probability for all N-grams. If an N-gram appears in training data, its probability can be estimated with discounting (including Witten-Bell). If not, shorten the history to compute \hat{P}_{bo}(\cdot \vert w_2…w_{k−1}), then multiply by \alpha(w_{2}\ldots w_{k - 1}), the extra probability mass discounted from history w_{2}\ldots w_{k - 1}. Note that \alpha is not a free parameter; after estimating all N-gram probabilities, it is fixed and serves as a normalization constant.
Evaluation method of language models
Given two language models A and B, how do we decide which is better? Intuitively, for a test dataset, if model A always assigns higher probability than B, then A is better because it predicts more accurately. Suppose a test sentence is w_{1}\ldots w_{n}, then the probability predicted by the model is:
$$P\left( w_{1}\ldots w_{n} \right) = P\left( w_{1}| < s > \right) \times P\left( w_{2} | w_{1}\right) \times P\left( w_{3} | w_{2}\right) \times \ldots \times P( < /s > | w_{n})$$
Because this probability is very small and multiplying causes underflow, we take the log:
$$\log{ P(w_{1}\ldots w_{n})} = \log{P(w_{1}| < s > )} + \log{P\left( w_{2} | w_{1}\right)} + \ldots + \log{P( < /s > | w_{n})}$$
The above log-probability is also called log-likelihood. Since probabilities are less than 1, log-likelihood is less than 0. Taking the negative and dividing by sentence length gives:
$$- \frac{1}{n}\log{P(w_{1}\ldots w_{n})}$$
This is entropy. Imagine encoding all words using bits; to minimize the encoded bit-length of a sentence, more frequent words have shorter codes and less frequent words have longer codes. Entropy equals the optimal average bit-length of encoding the sentence.
Another metric to evaluate model quality is the average inverse word probability, called perplexity (PPL). For example, if the average word probability is 1/100, PPL is 100. It is computed as:
Taking log of the above shows it exactly equals entropy. Thus,
$$PPL=2^{\text{entropy}}.$$
Summary of correlations:
likelihood ↑ ↔ entropy ↓ ↔ PPL ↓ ↔ better model
likelihood ↓ ↔ entropy ↑ ↔ PPL ↑ ↔ worse model
Handling N-gram models
Pruning
N-gram language models store statistics of N-grams from training data, each N-gram representing one model parameter. As training data grows, the number of N-grams increases rapidly. If an N-gram’s probability can be approximated well via backoff, we can remove that N-gram, reducing the number of parameters.
Which N-grams to delete? We can use entropy (or PPL) to guide this: if deleting changes entropy negligibly (below a threshold), then delete. After deleting an N-gram, the model requires renormalization (recomputing backoff coefficients).
To improve algorithm efficiency, we don’t need a separate test set to estimate entropy changes. We can use the model itself, as it contains all necessary calculation information. This enables an efficient pruning algorithm; readers can refer to Entropy-based Pruning of Backoff Language Models.
Interpolation
Suppose we have trained two language models with probabilities P_1 and P_2. How to fuse them into a better model? One way is to combine their training data and retrain a single model, which is inconvenient and may cause new issues. For instance, if the two datasets differ greatly in size, the smaller dataset is essentially drowned out. In practice, we often have large general data but limited domain-specific data. From a task perspective, domain data is more important. Clearly, we want to assign larger “weight” to domain data.
Thus, a better approach is to fuse the two models by interpolation—weighted averaging their probabilities:
$$\hat{P}\left( w_{k} \right|w_{1}\ldots w_{k - 1}) = \lambda\ {\hat{P}}{1}\left( w{k} \right|w_{1}\ldots w_{k - 1}) + (1 - \lambda)\ {\hat{P}}{2}\left( w{k} \right|w_{1}\ldots w_{k - 1})$$
Parameter \lambda controls relative importance. If it’s near 1, the first model dominates; near 0, the second model is more important. Optimal \lambda is a hyperparameter estimated using held-out data.
We can interpolate more than two models: for M models, weights \lambda_{1}, \lambda_{2}, \ldots, \lambda_{M} must satisfy \lambda_{1} + \lambda_{2} + \ldots + \lambda_{M} = 1 to ensure the combined probability distribution remains valid.
Model merging
Though interpolation avoids retraining, it has a big drawback: multiple models must be stored, requiring more disk space and memory, and prediction is slower.
Fortunately, for backoff N-gram model interpolation, we can merge them into a single model, whose probabilities approximate the interpolated model well. The merging steps are:
for k=1,...,N:
for all ngrams w1…wk
insert w1…wk into the new model (merged model)
compute P^(wk|w1…wk−1) according to the interpolation formula
end
recompute k-1 backoff coefficients for the new model
end
```Advanced Models[](#高级模型)
-------------
### Class-Based Language Models[](#类class-based语言模型)
In this section, we discuss some advanced topics of language models. Methods for language modeling are continuously evolving, making this a very active research area. According to the estimation in [Estimation of Gap Between Current Language Models and Human Performance](www.isca-speech.org/archive/Interspeech_2017/abstracts/0729.html), the perplexity (PPL) of the best current models is still two to three times worse than that of humans, indicating there is still a long way to go.
A major drawback of N-gram models is that they treat every word as entirely different entities. Therefore, for a certain word, the model needs sufficient training data to estimate its probability accurately. However, humans do not use language this way. We know that "Tuesday" and "Wednesday" share certain syntactic and semantic similarities. Even if the training data may not contain "store open Tuesday," because "store open Wednesday" has appeared, we can still assign a relatively high probability to the former. In other words, we need to consider word similarity.
Class-based language models cluster similar words into word classes, and then use word class tags instead of words when calculating N-gram statistics. For example, if we define a "WEEKDAY" word class that includes "Monday," "Tuesday," ..., "Friday," then the N-gram "store open Tuesday" will be treated as "store open WEEKDAY." Thus:
$$P(\text{Tuesday}|\text{store open})=P(\text{WEEKDAY}|\text{store open})P(\text{Tuesday}|\text{WEEKDAY})$$
The class membership probabilities $P(\text{Tuesday} \vert \text{WEEKDAY})$ can be estimated from the training data or set to an average by experience (because we priorly assume the probabilities of Monday and Sunday are the same).
So how do we obtain word classes? One method is to use prior knowledge, usually domain knowledge of the application. For example, when building a language model for a travel app, we need to recognize destination city names (such as "Tahiti," "Oslo"), flight information, and weekdays (Monday to Sunday). No matter how large the training data is, it may not contain all cities and flight information. Thus, we can assume equal class membership probability for each city based on prior knowledge or allocate these probabilities proportionally according to city popularity (rather than training data probabilities). Some entities consist of two or more words, such as "Los Angeles." This requires modifying the class-based language model; readers can refer to [Word-Phrase-Entity Language Models: Getting More Mileage out of N-grams](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/Levit_WPELM_Interspeech2014.v3.pdf).
Another approach is purely data-driven and does not require human or domain knowledge. For example, we can first define a fixed number of classes, then search all word-to-class mappings (each mapping is a clustering), and find the clustering with the minimum PPL. Of course, exhaustive search is computationally infeasible; some algorithms have been developed to address this, and readers may consult [Class-Based n-gram Models of Natural Language](anthology.aclweb.org/J/J92/J92-4003.pdf).
### Neural Network Language Models[](#神经网络语言模型)
Neural network methods have become mainstream in many fields, including the acoustic models we introduced earlier. Similarly, neural networks can be used for language modeling, and their performance surpasses that of N-gram models if provided with sufficient data.
Neural network language models can overcome two shortcomings of N-gram models. The first shortcoming is the inability to generalize to similar words. For instance, even if "store open Wednesday" occurs many times in the training data, if "store open Tuesday" never appears, its probability will be very low. We previously saw that class-based language models try to solve this problem, but they introduce a new problem: how to define word classes. The earliest neural network language model comes from the 2003 paper [A Neural Probabilistic Language Model](http://www.jmlr.org/papers/volume3/bengio03a/bengio03a.pdf). Because this network is a feedforward neural network, this language model is also called a feedforward network language model. It introduces an Embedding layer to map discrete word labels into a dense vector space, thereby enabling similar words to share similar contexts.
 _Figure: Neural Network Language Model_
As shown in the figure above, the neural network input is the previous N-1 words of the current word, with each word encoded as a one-hot vector. The output is the predicted probability of the current word. The key is that the input one-hot vectors are transformed into low-dimensional dense vectors through an Embedding matrix shared across all words. The advantage is that if two words frequently appear in similar contexts (such as "Wednesday" and "Tuesday"), their vectors will be similar. Although "store open Tuesday" rarely appears in the training data, the two words appear in similar contexts at other times, so the model learns that the vectors of "Wednesday" and "Tuesday" are similar. Therefore, the probabilities predicted for "Wednesday" and "Tuesday" following "store open" are similar. This shared Embedding matrix solves the first problem of N-gram models.
The second problem of N-gram models is their limited ability to rely on long histories. For example, the sentence "I was born in France, ………………….., So I can spoken fluent (French/English/Chinese)." If only a short history "spoken fluent" is considered, the probabilities of the three words are similar. But if a longer history including "France" is taken into account, the probability of "French" will be higher.
The earlier neural network language model is similar to N-gram models and can only look at the previous N-1 words. To solve this problem, we can use Recurrent Neural Networks. Readers may refer to [Introduction to Recurrent Neural Networks](/books/rnn-intro). We will not elaborate here.
Lab[](#lab)
-----------
### Environment[](#环境)
This lab requires building N-gram language models using SRILM in a shell (bash) environment. The language models constructed in this lab will be used later in the decoding section.
First, enter the Lab4 directory:
cd M4_Language_Modeling
$ pwd
/home/lili/codes/Speech-Recognition/M4_Language_Modeling
Set the environment variable so that SRILM commands can be used:
export PATH=$PWD/srilm/bin/i686-m64:$PWD/srilm/bin:$PATH
If using Cygwin (Windows), please use:
export PATH=$PWD/srilm/bin/cygwin64:$PWD/srilm/bin:$PATH
To test if the environment is okay, execute the following command. If there is no error message, it is good.
$ ngram-count -write-vocab -
-pau-
<```/s>
$ compute-oov-rate < /dev/null
OOV tokens: 0 / 0 (0.00%) excluding fragments: 0 / 0 (0.00%)
OOV types: 0 / 0 (0.00%) excluding fragments: 0 / 0 (0.00%)
Besides that, we also need wget, sort, head, wc, sed, gawk, and perl. These tools are generally already installed in the system; if not, please install them yourself.
### Preparing Data[](#准备数据)
We will use the transcripts from the acoustic model training data, where the dev and test sets are used as the development and test sets for training the language model. Since the recordings we ultimately want to recognize are from the dev and test sets, using their texts as the development and test sets for the language model is very appropriate. Note: training a language model does not require "annotated" data; it only requires sentences one by one. Therefore, we can easily obtain a large amount of training data, but these training data domains might differ from the domain of our speech recognition application. Usually, application domain data is relatively scarce, and later experiments will introduce some methods to address this problem.
#### Counting Lines in Files Under data[](#统计data下文件的行数)
$ ls data/
ami-dev.txt ami-test.txt ami-train.min3.vocab ami-train.txt dev.txt test.txt
$ wc -wl data/dev.txt data/test.txt
466 10841 data/dev.txt
261 5236 data/test.txt
727 16077 total
We will use data/dev.txt as the language model development set and data/test.txt as the test set.
#### Viewing These Files[](#查看这些文件)
$ head -n 3 data/*.txt
==> data/ami-dev.txt <==
uhhuh
uh do we know if there will be a lot of people coming across the hall in terms of security stuff if we can i mean my idea is to put the photocopier and the fax in the hall
um
==> data/ami-test.txt <==
you mean maybe you should break the wall between the men’s room and the women’s room sorry
==> data/ami-train.txt <==
okay
does anyone want to see uh steve’s feedback from the specification
right
==> data/dev.txt <==
a laudable regard for the honor of the first proselyte has countenanced the belief the hope the wish that the ebionites or at least the nazarenes were distinguished only by their obstinate perseverance in the practice of the mosaic rites
their churches have disappeared their books are obliterated their obscure freedom might allow a latitude of faith and the softness of their infant creed would be variously moulded by the zeal or prudence of three hundred years
yet the most charitable criticism must refuse these sectaries any knowledge of the pure and proper divinity of christ
==> data/test.txt <==
when we took our seats at the breakfast table it was with the feeling of being no longer looked upon as connected in any way with this case
instantly they absorbed all my attention though i dared not give them a direct look and continued to observe them only in the glass
yes and a very respectable one
The files are large, so we use the head command to view the first few lines.
We find that all the data is lowercase, and there is no punctuation. This is because punctuation is not spoken when talking, and the absence of capitalization is also to match the previous pronunciation dictionary. The preprocessing of these texts is called text normalization, which usually includes removing punctuation, handling case, correcting spelling errors, and standardizing some words (for example, normalizing MR. to MISTER). This dirty work takes a lot of time, and we can help with tools like sed or perl.
How English handles data depends on sources, domain habits, and tools, which we will not introduce here.
#### Downloading Training Data[](#下载训练数据)
For language model training data, we use LibriSpeech text. Here, we directly download normalized text:
wget http://www.openslr.org/resources/11/librispeech-lm-norm.txt.gz
$ du -sh librispeech-lm-norm.txt.gz
1.5G librispeech-lm-norm.txt.gz
This file is 1.5G, so the download will take some time.
#### Viewing the Downloaded Text[](#查看下载的文本)
Readers may decompress the gz file and then use head or vim to view it. However, the decompressed file is very large, and our SRILM tool can handle compressed files directly, so here we use a pipe to view the file contents:
gunzip -c librispeech-lm-norm.txt.gz | head
gunzip -c librispeech-lm-norm.txt.gz | wc -wl
40418261 803288729
The second command counts how many words and how many lines the training data contains. The above result shows a total of over 40 million lines and over 800 million words. This command takes some time. Note: the text in this file is normalized, but it is normalized to all uppercase words.
### Defining the Vocabulary[](#定义词典)
The first step in training a language model is to define the model's vocabulary. We want this vocabulary to use the smallest number of words to cover most of the tokens in the training data. Therefore, we need to select the highest frequency words from the training data.
We can use the ngram-count tool to count the occurrences of n-grams in a text file, and 1-gram is the word frequency. For example:
ngram-count -text TEXT -order 1 -write COUNTS -tolower
-text means the file to count, TEXT; -order 1 means counting 1-grams; -write COUNTS outputs to the file COUNTS; -tolower converts all words to lowercase. More detailed documentation on this tool can be found [here](http://www.speech.sri.com/projects/srilm/manpages/ngram-count.1.html).
#### Extracting the Top 10000 Highest Frequency Words from Training Data[](#抽取训练数据中最高频的10000个词)
$ ngram-count -text librispeech-lm-norm.txt.gz -order 1 -write librispeech.1grams -tolower
The above command performs 1-gram counting on the input file librispeech-lm-norm.txt.gz, converts all words to lowercase, and outputs librispeech.1grams. This command takes some time to run.
Let's look at the content of this 1grams file:
$ head librispeech.1grams
bikes 79
schiffbauerdamm 4
pluseirs 1
diega 3
intermediating 4
caplike 3
ryot 65
cernis 4
moqui’s 1
prideth 1
To select high-frequency words, we need to sort this file by the second column as a number, not as a string, or situations like "12" < "2" will occur.
$ sort -k 2,2 -n -r librispeech.1grams | head -10000 > librispeech.top10k.1grams
The above uses sort to sort librispeech.1grams; -r means reverse order (we want to select the most frequent), -n means sort numerically, and -k 2,2 means sorting key is the second column (from column 2 to 2). If it were '2,3', it would sort by columns 2 and 3. Interested readers can refer to [Linux sort command overview](/2019/06/15/sort/) for more details on sort usage.
Then the result of sort is piped into the head command to select the top 10,000 most frequent words. Let's look at the content of this file:
$ head librispeech.top10k.1grams
the 49059384
40418260 40418260
and 26362574
of 24795903
to 22052019
a 17811980
in 13524728
i 10609353
he 10203671
“The” even appears more frequently than `<s>`; this indicates that the average occurrence of "the" per sentence is greater than 1. The number of `<s>` equals the number of non-empty lines, which is 40418260. Earlier, we counted lines by “wc -l” as 40418261, one extra line is an empty line (only a newline), since we often add a newline at the file's end.
However, our vocabulary does not need word frequency; also, we want the vocabulary sorted alphabetically. So we can use the following command:
cut -f 1 librispeech.top10k.1grams | sort > librispeech.top10k.vocab
The resulting librispeech.top10k.vocab is:
$ head librispeech.top10k.vocab
a
aaron
abandon
abandoned
abbe
abbey
abbot
abe
abel
abide
#### Calculating OOV Rate[](#统计oov率)
How much data does our top 10k high-frequency words cover? We can use the [compute-oov-rate tool](http://www.speech.sri.com/projects/srilm/manpages/training-scripts.1.html) to calculate this.
$ ngram-count -text data/dev.txt -order 1 -write dev.1grams
First, we use ngram-count to calculate the 1-gram of data/dev.txt. Then we use compute-oov-rate to calculate the OOV rate:
$ compute-oov-rate librispeech.top10k.vocab dev.1grams
OOV tokens: 625 / 10841 (5.77%) excluding fragments: 625 / 10841 (5.77%)
OOV types: 556 / 2872 (19.36%) excluding fragments: 556 / 2872 (19.36%)
compute-oov-rate requires two parameters: the first is the vocabulary, the second is the ngram-count statistics. We can see that the top 10k words on the training set cover 19.36% of OOV types and 5.77% of OOV tokens. OOV types are unseen words counted once regardless of multiple occurrences, while OOV tokens count all unseen words cumulatively.
Similarly, we can do statistics on data/test.txt:
ngram-count -text data/test.txt -order 1 -write test.1grams
compute-oov-rate librispeech.top10k.vocab test.1grams
OOV tokens: 258 / 5236 (4.93%) excluding fragments: 258 / 5236 (4.93%)
OOV types: 220 / 1575 (13.97%) excluding fragments: 220 / 1575 (13.97%)
We can see that OOV type rate is higher than OOV token rate because many unseen types have very low frequency. Let's also look at the training data's statistics:
$ compute-oov-rate librispeech.top10k.vocab librispeech.1grams
OOV tokens: 52454701 / 803288729 (6.53%) excluding fragments: 52454701 / 803288729 (6.53%)
OOV types: 963675 / 973673 (98.97%) excluding fragments: 963675 / 973673 (98.97%)
The OOV type rate on training data is very high. This means that we used 1.03% of the high-frequency words to cover 93.5% of the data, while 98.97% of the low-frequency words accounted for only 6.53% of occurrences!
Note: 5% OOV tokens is a very high value because OOV words are definitely unrecognized. Here, to make the model small, we chose a relatively small vocabulary. In actual applications, the vocabulary may be 50k or more.
### Training the Language Model[](#训练语言模型)
Next, we will train the language model using the ngram-count tool. This tool can complete training in one step, but to understand the process, I separate it into two steps. The first step is to count N-gram frequencies; the second is parameter estimation (discounting, backoff, and other smoothing). Note: because the data is large, training machines must have 10GB memory, or else memory might be insufficient.
#### Counting All Trigram Counts on Training Data[](#统计训练数据上的所有trigram计数)
The following counting command takes several minutes; please be patient.
$ ngram-count -text librispeech-lm-norm.txt.gz -tolower -order 3 -write librispeech.3grams.gz
Since the output file is large, we use compressed format; ngram-count detects compression by suffix. -tolower means lowercase normalization; -order 3 means counting trigrams. Note: output is grouped by common prefix, but words themselves are not sorted. If needed, use sort to sort.
We can look at the output file contents:
$ gunzip -c librispeech.3grams.gz | less 15
bikes 79
bikes here 1
bikes here and 1
bikes triple 1
bikes triple locked 1
bikes
bikes invaded 1
bikes invaded one’s 1
bikes sir 1
bikes sir 1
bikes travelled 1
bikes travelled warily 1
bikes there 1
bikes there 1
bikes are 2
bikes are good 1
The following command takes a few minutes.
$ ngram-count -debug 1 -order 3 -vocab librispeech.top10k.vocab
-read librispeech.3grams.gz -wbdiscount -lm librispeech.3bo.gz
using WittenBell for 1-grams
using WittenBell for 2-grams
using WittenBell for 3-grams
discarded 1 1-gram probs predicting pseudo-events
warning: distributing 1.26288e-05 left-over probability mass over all 9999 words
discarded 2 2-gram contexts containing pseudo-events
discarded 9999 2-gram probs predicting pseudo-events
discarded 29987 3-gram contexts containing pseudo-events
discarded 2198401 3-gram probs predicting pseudo-events
discarded 79019478 3-gram probs discounted to zero
writing 10000 1-grams
writing 13412395 2-grams
writing 36521492 3-grams
-debug 1 makes it output some debug information, -order 3 indicates it is a trigram model, -read reads the previous 3gram statistics, -wbdiscount uses Witten-Bell smoothing, and the output is finally saved to librispeech.3bo.gz.
We can view the file content:
$ gunzip -c librispeech.3bo.gz | less
\data
ngram 1=10000
ngram 2=13412395
ngram 3=36521492
\1-grams:
-1.291743
-99 ``` -2.29457
-1.647608 a -1.707602
-5.228539 aaron -0.3972774
-4.795578 abandon -0.8257342
-4.538978 abandoned -0.6036708
-5.064918 abbe -0.3938949
-4.877338 abbey -0.546707
You can also decompress first and then open with a text editor. Note that decompression requires a large amount of space, so use a text editor capable of handling large files.
### Model Evaluation[](#模型评估)
The model we obtained above is in arpa format. The full documentation is [here](http://www.speech.sri.com/projects/srilm/manpages/ngram-format.5.html).
#### "Manual" Probability Calculation[](#手动计算概率)
Given the sentence “a model was born”, how do we calculate the conditional probability of "born"? Since it’s a trigram model, we actually need to calculate $P(born \\vert \\text{model was})$.
So we need to find the trigram statistics for “model was born”, which can be searched using the `zgrep` tool:
$ zgrep " model was born" librispeech.3bo.gz
No results are found, so this trigram does not exist in the training data. In that case, the backoff mechanism should be used to calculate the probability. First, we find the backoff probability $\\alpha(\\text{model was})$, which can be searched with `zgrep`:
$ zgrep -E “\smodel was” librispeech.3bo.gz | head -1
-2.001953\tmodel was\t0.02913048
The purpose of adding "\\s" before “model was” is to avoid matching “amodel was”. -2.001953 is the bigram log probability $P(was \\vert model)$, and 0.02913048 is the backoff probability $\\alpha(\\text{model was})$ we need.
Next, we calculate $P(born \\vert was)$ by searching:
$ zgrep -E “\swas born” librispeech.3bo.gz | head -1
-2.597636\twas born\t-0.4911189
Hence, the (log) probability $P(born \\vert \\text{model was})$ is -2.597636 because:
$$P(born \\vert \\text{model was})=\\alpha(\\text{model was}) \\times P(born \\vert was)$$
Since the above probabilities are in the log domain, multiplication becomes addition:
0.02913048 + -2.597636 = -2.568506
The log probability is -2.568506, and the actual probability is $10^{-2.568506} = 0.002700813$.
#### Using ngram for Calculation[](#使用ngram计算)
Now we use the `ngram` tool with the `-ppl` option to calculate the probability of the sentence “a model was born”, to verify our manual calculation:
$ echo “a model was born” | ngram -debug 2 -lm librispeech.3bo.gz -ppl -
reading 10000 1-grams
reading 13412395 2-grams
reading 36521492 3-grams
a model was born
\tp( a | ) \t= [2gram] 0.01653415 [ -1.781618 ] | born …) \t= [3gram] 0.1352684 [ -0.8688038 ]
\tp( model | a …) \t= [3gram] 0.0001548981 [ -3.809954 ]
\tp( was | model …) \t= [3gram] 0.002774693 [ -2.556785 ]
\tp( born | was …) \t= [2gram] 0.002700813 [ -2.568506 ]
\tp(
1 sentences, 4 words, 0 OOVs
0 zeroprobs, logprob= -11.58567 ppl= 207.555 ppl1= 787.8011
file -: 1 sentences, 4 words, 0 OOVs
0 zeroprobs, logprob= -11.58567 ppl= 207.555 ppl1= 787.8011
Explanation of the above command parameters: For `ngram`, `-debug 2` prints debug info; `-lm` specifies the arpa language model; `-ppl` computes perplexity, and the last `-` means the input is read from standard input. Otherwise, we would need to create a file containing “a model was born”, which is inconvenient for quick testing. Here we use a common Bash trick, using “-” to indicate reading from standard input, piping the sentence via `echo` into `ngram`.
Note: The `ngram` tool automatically adds start and end tokens <s> and </s>. The last line outputs the entire sentence log probability and perplexity. Look at the part for $p( born \\vert was …)$:
p( born | was …) \t= [2gram] 0.002700813 [ -2.568506 ]
It matches our manual calculation; the model backed off to bigram. We can also verify PPL and log probability relation: $10^{-(-11.58567/5)} = 207.555$.
#### Calculating PPL on the Development Set[](#计算开发集上的ppl)
$ ngram -lm librispeech.3bo.gz -ppl data/dev.txt
file data/dev.txt: 466 sentences, 10841 words, 625 OOVs
0 zeroprobs, logprob= -21939 ppl= 113.1955 ppl1= 140.4475
PPL is 113, and OOV rate is 625/10841 = 5.8%.
#### Calculating PPL on the Test Set[](#计算测试集上的ppl)
$ ngram -lm librispeech.3bo.gz -ppl data/test.txt
file data/test.txt: 261 sentences, 5236 words, 258 OOVs
0 zeroprobs, logprob= -10505.09 ppl= 101.1976 ppl1= 128.9147
### Model Adaptation[](#模型自适应)
Next, we introduce how to adapt a model for a specific domain. Usually, our domain-specific data is limited but general domain data is abundant. In this example, we use [AMI](http://www.amiproject.org/) as our target domain, which is a multi-party telephone meeting scenario, i.e., spontaneous speech among multiple people. The librispeech data is recordings of book readings, with very different speaking styles and topics.
#### Domain Data[](#领域数据)
We treat librispeech as out-of-domain data and adapt the model with a small amount of AMI domain data to fit the AMI domain. Let's first look at the AMI data:
$ wc -wl data/ami-*.txt
2500 26473 data/ami-dev.txt
2096 20613 data/ami-test.txt
86685 924896 data/ami-train.txt
91281 971982 total
AMI data is much smaller than librispeech. There is also a prepared vocabulary file containing words with frequency greater than 3:
$ wc -l data/ami-train.min3.vocab
6271 data/ami-train.min3.vocab
Earlier, librispeech’s vocabulary was 10k, here it's only 6k.
#### Training a Model Using AMI Data[](#使用ami数据训练模型)
Same as previously, but using different training data and vocabulary.
ngram-count -text data/ami-train.txt -tolower -order 3 -write ami.3grams.gz
ngram-count -debug 1 -order 3 -vocab data/ami-train.min3.vocab
-read ami.3grams.gz -wbdiscount -lm ami.3bo.gz
using WittenBell for 1-grams
using WittenBell for 2-grams
using WittenBell for 3-grams
discarded 1 1-gram probs predicting pseudo-events
warning: distributing 0.00626447 left-over probability mass over all 6270 words
discarded 2 2-gram contexts containing pseudo-events
discarded 1210 2-gram probs predicting pseudo-events
discarded 6220 3-gram contexts containing pseudo-events
discarded 3640 3-gram probs predicting pseudo-events
discarded 398917 3-gram probs discounted to zero
writing 6271 1-grams
writing 167020 2-grams
writing 91496 3-grams
Next, test this model's PPL on the development set:
$ ngram -lm ami.3bo.gz -ppl data/ami-dev.txt
file data/ami-dev.txt: 2314 sentences, 26473 words, 1264 OOVs
0 zeroprobs, logprob= -55254.39 ppl= 101.7587 ppl1= 155.5435
The PPL on the development set is 101.
Now test the out-of-domain model trained on librispeech on the AMI development set:
$ ngram -lm librispeech.3bo.gz -ppl data/ami-dev.txt
file data/ami-dev.txt: 2314 sentences, 26473 words, 3790 OOVs
0 zeroprobs, logprob= -56364.05 ppl= 179.8177 ppl1= 305.3926
OOV count is 3790, much higher than 1264 before; PPL is 179, also higher. This indicates that the model trained on large librispeech data is better than that trained on limited AMI data. However, because the domains are quite different, the model trained on more data doesn’t necessarily perform well.
#### Interpolation and Merging[](#插值和合并)
Now we fuse the librispeech model with the AMI model by interpolation. Interpolation requires a weight; below, we explain how to choose the optimal weight. Based on experience, the domain model usually gets a larger weight; here we use 0.8.
We can use `ngram` with the `-mix-lm` and `-write-lm` options for interpolation and merging:
$ ngram -debug 1 -order 3 -lm ami.3bo.gz -lambda 0.8
-mix-lm librispeech.3bo.gz -write-lm ami+librispeech.3bo.gz
reading 6271 1-grams
reading 167020 2-grams
reading 91496 3-grams
reading 10000 1-grams
reading 13412395 2-grams
reading 36521492 3-grams
writing 12819 1-grams
writing 13481632 2-grams
writing 36558250 3-grams
`-lm` specifies the main model (with `-lambda` weight), `-mix-lm` is the interpolated model, and `-write-lm` outputs the interpolation result. What if we want to interpolate multiple models? We can use `-mix-lm2`, `-mix-lambda2`, `-mix-lm3`, `-mix-lambda3`, etc., e.g.:
ngram -order 3 \
-lm lm0.gz -lambda {LAMBDAS[0]}
-mix-lm lm1.gz
-mix-lm2 lm2.gz -mix-lambda2 {LAMBDAS[2]} \
-mix-lm3 lm3.gz -mix-lambda3 {LAMBDAS[3]}
-mix-lm4 lm4.gz -mix-lambda4 {LAMBDAS[4]} \
-mix-lm5 lm5.gz -mix-lambda5 {LAMBDAS[5]}
-write-lm mixed_lm.gz
Note: The above code refers to [Building large LMs with SRILM](https://joshua.incubator.apache.org/6.0/large-lms.html).
The `-mix-lm` models don’t need separate weight specification because weights sum to 1, computed automatically by the script.
Now test the PPL of this interpolated model on the development set:
$ ngram -lm ami+librispeech.3bo.gz -ppl data/ami-dev.txt
ngram -lm ami+librispeech.3bo.gz -ppl data/ami-dev.txt
file data/ami-dev.txt: 2314 sentences, 26473 words, 783 OOVs
0 zeroprobs, logprob= -56313.77 ppl= 102.546 ppl1= 155.6145
PPL is 102, worse than only AMI. But OOV dropped from 1264 to 783 because some AMI OOVs appear in librispeech. Since the interpolated model’s vocabulary is larger, it chooses more branches, causing higher PPL. So when comparing language model PPLs, be sure to check if their vocabularies are the same; otherwise, the comparison is meaningless!
How to prove interpolation effectiveness? We can specify using AMI's vocabulary only in interpolation, discarding ngrams outside AMI vocab, thus making comparisons fair:
$ ngram -debug 1 -order 3 -lm ami.3bo.gz -lambda 0.8 -mix-lm librispeech.3bo.gz
-write-lm ami+librispeech.bo3.gz -vocab data/ami-train.min3.vocab -limit-vocab
reading 6271 1-grams
reading 167020 2-grams
reading 91496 3-grams
reading 10000 1-grams
discarded 6548 OOV 1-grams
reading 13412395 2-grams
discarded 9980751 OOV 2-grams
reading 36521492 3-grams
discarded 19683914 OOV 3-grams
warning: distributing 0.0257169 left-over probability mass over all 6270 words
writing 6271 1-grams
writing 3500881 2-grams
writing 16874336 3-grams
The above command uses `-limit-vocab` and `-vocab` to tell `ngram` to only use the vocabulary in `data/ami-train.min3.vocab` for interpolation. Now use this new interpolated model to calculate PPL:
$ ngram -lm ami+librispeech.3bo.gz -ppl data/ami-dev.txt
file data/ami-dev.txt: 2314 sentences, 26473 words, 1264 OOVs
0 zeroprobs, logprob= -53856.04 ppl= 90.52426 ppl1= 136.8931
OOV equals that of pure AMI, but PPL decreased to 90.
#### Selecting the Optimal Interpolation Parameter[](#选择最优的插值参数)
How to find the optimal interpolation parameter? One way is brute force, choosing the parameter that yields the minimal PPL. But this is slow, so we can use more efficient algorithms, e.g., EM (Expectation-Maximization) method. SRILM already implements such a method. Let's test it with a script.
The `compute-best-mix` script computes the optimal interpolation weights, but requires detailed PPL files for each model on the (development) dataset. So first, we generate detailed PPL files for the two models on the development set using `ngram`:
$ ngram -debug 2 -order 3 -lm librispeech.3bo.gz -ppl data/ami-dev.txt > lm1.ppl
reading 10000 1-grams
reading 13412395 2-grams
reading 36521492 3-grams
This `ppl` file contains detailed information about PPL calculations:
```$ head -50 lm1.ppl
uhhuh
p( <unk> | <s> ) = [OOV] 0 [ -inf ]
p( </s> | <unk> ...) = [1gram] 0.05108071 [ -1.291743 ]
1 sentences, 1 words, 1 OOVs
0 zeroprobs, logprob= -1.291743 ppl= 19.57686 ppl1= undefined
uh do we know if there will be a lot of people coming across the hall in terms of security stuff if we can i mean my idea is to put the photocopier and the fax in the hall
p( <unk> | <s> ) = [OOV] 0 [ -inf ]
p( do | <unk> ...) = [1gram] 0.001973127 [ -2.704845 ]
p( we | do ...) = [2gram] 0.00794844 [ -2.099718 ]
p( know | we ...) = [3gram] 0.07709495 [ -1.112974 ]
p( if | know ...) = [3gram] 0.001739762 [ -2.75951 ]
p( there | if ...) = [3gram] 0.03683351 [ -1.433757 ]
p( will | there ...) = [3gram] 0.001275695 [ -2.894253 ]
p( be | will ...) = [3gram] 0.7819563 [ -0.1068175 ]
p( a | be ...) = [3gram] 0.06429912 [ -1.191795 ]
p( lot | a ...) = [3gram] 0.00344327 [ -2.463029 ]
p( of | lot ...) = [3gram] 0.6903875 [ -0.1609071 ]
p( people | of ...) = [3gram] 0.02535887 [ -1.59587 ]
p( coming | people ...) = [3gram] 0.003904172 [ -2.408471 ]
p( across | coming ...) = [3gram] 0.004744955 [ -2.323768 ]
p( the | across ...) = [3gram] 0.507619 [ -0.2944621 ]
p( hall | the ...) = [3gram] 0.01331778 [ -1.875568 ]
p( in | hall ...) = [3gram] 0.0139884 [ -1.854232 ]
p( terms | in ...) = [2gram] 0.0002327063 [ -3.633192 ]
p( of | terms ...) = [3gram] 0.5794951 [ -0.2369502 ]
p( security | of ...) = [3gram] 0.0002009042 [ -3.697011 ]
p( stuff | security ...) = [2gram] 6.165597e-05 [ -4.210025 ]
p( if | stuff ...) = [2gram] 0.00305367 [ -2.515178 ]
p( we | if ...) = [3gram] 0.02380954 [ -1.623249 ]
p( can | we ...) = [3gram] 0.05685819 [ -1.245207 ]
p( i | can ...) = [3gram] 0.0008678308 [ -3.061565 ]
p( mean | i ...) = [3gram] 0.0007527435 [ -3.123353 ]
p( my | mean ...) = [3gram] 0.007983495 [ -2.097807 ]
p( idea | my ...) = [2gram] 0.0007232695 [ -3.1407 ]
p( is | idea ...) = [3gram] 0.1738275 [ -0.7598816 ]
p( to | is ...) = [3gram] 0.100922 [ -0.9960141 ]
p( put | to ...) = [3gram] 0.004835975 [ -2.315516 ]
p( the | put ...) = [3gram] 0.09036702 [ -1.04399 ]
p( <unk> | the ...) = [OOV] 0 [ -inf ]
p( and | <unk> ...) = [1gram] 0.03331709 [ -1.477333 ]
p( the | and ...) = [2gram] 0.07761291 [ -1.110066 ]
p( <unk> | the ...) = [OOV] 0 [ -inf ]
p( in | <unk> ...) = [1gram] 0.01709259 [ -1.767192 ]
p( the | in ...) = [2gram] 0.2916464 [ -0.5351433 ]
p( hall | the ...) = [3gram] 0.004312756 [ -2.365245 ]
p( </s> | hall ...) = [3gram] 0.1904368 [ -0.7202492 ]
1 sentences, 39 words, 3 OOVs
0 zeroprobs, logprob= -68.95484 ppl= 73.05407 ppl1= 82.30237
Similarly, we compute the ppl file of the ami model:
$ ngram -debug 2 -order 3 -lm ami.3bo.gz -ppl data/ami-dev.txt > lm2.ppl
reading 6271 1-grams
reading 167020 2-grams
reading 91496 3-grams
With these two ppl files, we can calculate the optimal interpolation weights:
$ compute-best-mix lm*.ppl > best-mix.ppl
iteration 1, lambda = (0.5 0.5), ppl = 109.309
iteration 2, lambda = (0.387377 0.612623), ppl = 104.984
iteration 3, lambda = (0.323519 0.676481), ppl = 103.53
iteration 4, lambda = (0.286997 0.713003), ppl = 103.017
iteration 5, lambda = (0.265635 0.734365), ppl = 102.83
iteration 6, lambda = (0.252893 0.747107), ppl = 102.76
iteration 7, lambda = (0.245184 0.754816), ppl = 102.734
iteration 8, lambda = (0.240476 0.759524), ppl = 102.724
iteration 9, lambda = (0.237583 0.762417), ppl = 102.72
iteration 10, lambda = (0.235798 0.764202), ppl = 102.719
iteration 11, lambda = (0.234694 0.765306), ppl = 102.718
Let’s check the final output:
$ cat best-mix.ppl
28004 non-oov words, best lambda (0.23401 0.76599)
pairwise cumulative lambda (1 0.76599)
That is, the optimal interpolation parameter is 0.766.
Model Pruning
We observe the relationship between model PPL and the size of training data: the more training data, the better the performance. However, with more training data, the model also grows larger. A very large model takes up a lot of memory and has slower computation speed during usage. One approach is to select an appropriate amount of training data based on constraints of computational resources (such as memory and speed), but a better approach is to use all the data to train the model and then prune it according to resource constraints.
A common pruning method comes from the paper Entropy-based Pruning of Backoff Language Models, which is also implemented in SRILM. Pruning has a -prune option that specifies that pruning can continue if the relative increase (worsening) in PPL is less than this threshold. This value is typically very small, such as 10^{-8} or 10^{-9}. The larger this value is, the more pruning is done, resulting in a smaller model but with higher PPL.
Below, we prune using 10^{-5}:
ngram -debug 1 -lm librispeech.3bo.gz -prune 1e-5 -write-lm librispeech-pruned1.3bo.gz
Similarly, pruning with 10^{-6}...10^{-10} results are as follows:
| prune Value | 10−510−5 | 10−610−6 | 10−710−7 | 10−810−8 | 10−910−9 | 10−1010−10 | No Pruning |
|---|---|---|---|---|---|---|---|
| Model Size | 336K | 2.1M | 12M | 61M | 205M | 263M | 286M |
| PPL on data/dev.txt | 258 | 178 | 136 | 118 | 113.49 | 113.21 | 113.19 |