20240926 - TA - Evaluation Metrics for Language Modeling

The video discussed evaluation metrics, smoothing techniques, backoff models, interpolation, and maximum likelihood estimation in language modeling.
Evaluation metrics are used to determine the quality of a language model's sentence and pattern generation.
Cross entropy and perplexity are common metrics for language modeling, with lower values indicating better performance.
Smoothing techniques, such as add-one smoothing, can be used to avoid zero probabilities for unseen words in the corpus.
F1 smoothing is a method to handle low probabilities by adding a constant K to the count.
Backoff models are effective for unseen words, where lower probability words are removed to decrease complexity.
Interpolation can be used to mix unigram, bigram, and trigram probabilities based on lambda values.
Maximum likelihood estimation is used to calculate probabilities based on word counts in the corpus.
Cross entropy is correlated with perplexity in language modeling.
Predictive probabilities can be used to calculate cross entropy and perplexity.