This research is in the field of machine translation and in relation to extraction of Persian-English chunks from bilingual corpus by Spark. In this regard, the most important challenge is that the operation must be carried out on large corpus; therefore, it requires di
More
This research is in the field of machine translation and in relation to extraction of Persian-English chunks from bilingual corpus by Spark. In this regard, the most important challenge is that the operation must be carried out on large corpus; therefore, it requires distributed computing along with big data analysis techniques and tools. When translating text, we are usually confronted with chunks that we need to find the corresponding chunks of each one in the target language and insert it in our translation; this is accomplished by locating it in a corpus that contain the chunks and their corresponding translations. The existing methods, perform this operations in a non-distributed way, therefore while they run slowly, they cannot use a very large corpus. To overcome this shortcoming, in this research a distributed method has been presented, which also takes distance between the sections of chunks into account. The proposed method extracts all possible chunks from the input sentences in the monolingual corpus and uses the correlation coefficient to translate those chunks using the bilingual corpus. We implemented the proposed algorithm in a platform consisting of a computing cluster with sixty-four GB of memory and a twenty-four-core processor in Spark. The incorporated experimental data was a Persian and an English monolingual corpus along with an English-Persian bilingual corpus, each of which containing 100,000 sentences. Experimental results show that run time could greatly be reduced, and the quality of translation is also significantly improved.
Manuscript profile
