|
Abstract:
|
Given a corpus of documents purporting to be in a single language, we develop a methodology for detecting documents written in a different language. Unlike previous work in language identification, the methodology does not assume any prior knowledge specific to either language. Information theoretic methods are used to determine key terms that are highly predictive of language identity, and a language model is then built from the keywords and compared to models developed from each document using rank order statistics, cosine dissimilarity, and other approaches. We evaluate the different methods of constructing and ranking the models on a corpus of multilingual Wikipedia articles.
|
