Abstract—Tokenization is very important in natural language processing. It can be seen as a preparation stage for all other natural language processing tasks. In this paper we propose a hybrid unsupervised method for Arabic tokenization system, considered as a stand-alone problem. After getting words from sentences by segmentation, we used the author’s analyzer to produce all possible tokenizations for each word. Then, written rules and statistical methods are applied to solve the ambiguities. The output is one tokenization for each word. The statistical method was trained using 29k words, manually tokenized (data available from http://www.mimuw.edu.pl\~aliwy) from Al-Watan 2004 corpus (available from http://sites.google.com/site/mouradabbas9/corpora). The final accuracy was 98.83%.
Index Terms—Arabic Tokenization, Arabic segmentation, Arabic tagging.
H. Aliwy is with Institute of Informatics, University of Warsaw, Warsaw, Poland (ahmed_7425@yahoo.com; aliwy@mimuw.edu.pl).
Cite: Ahmed H. Aliwy, "Tokenization as Preprocessing for Arabic Tagging System," International Journal of Information and Education Technology vol. 2, no. 4, pp. 348-353, 2012.