非監督式中文寫作自動評閱系統之研究與設計

Abstract

寫作能力是一種綜合訓練，既是語言文字的訓練，同時也是思維能力的訓練。藉由寫作過程，可以訓練一個學生的思考、理解、推理、及創作等能力；同時亦可檢視該學生是否已理解本國語言文字並能加以靈活應用。在日常生活中，舉凡書信、公文、簡報、履歷表、及婚喪喜慶等各類應酬文書等，皆屬作文的範疇，絕大部份的人或多或少都會接觸到作文，由此更突顯出作文教學的必要性。 有效率的評閱寫作作品對於數位學習、心理計量、教育測驗等領域是一項非常重要的研究課題。由於人工評閱的成本過高，又不易達成測驗所需的評分者信度，因此如何利用機器及人工智慧技術協助評閱寫作作品的自動寫作評閱(Automated Essay Scoring,AES)技術被視為重要的解決方案。在1990 年代，隨著自然語言處理技術的長足進步，具有高正確率的系統陸續問世，且被應用於大型入學測驗及寫作教學。 在第二期計畫中，我們在自動作文評分研究上已經有初步成果，其中一篇論文已經被 IEEE Intelligent Systems 期刊接受，一篇被 Information: An International Interdisciplinary Journal期刊接受，兩篇會議文章，一篇論文正在 IEEE Intelligent Systems 期刊 2nd revision 中，另有四篇論文正在相關的期刊審查中。然而根據我們前兩期研究成果顯示，中文AES 系統的發展仍有很多的挑戰。例如：大部分的 AES 系統都需要相當數量的人工評分訓練樣本，導致系統的導入需要事先的大量人力來評分，再經由系統學習，才能夠真正開始做評分。因此一個不需要事先評分樣本的系統，將可以更符合實際評分系統的需求。此外，惡意欺騙系統問題也是非常重要，此項要求是要讓該系統可以真正應用到實際使用的一項重要因素。 在本計畫中，我們主要針對三項方向改進：第一、系統需要有一定數量的人工評分訓練樣本。第二、系統存在被惡意欺騙的風險。第三、系統使用缺乏具備語意之特徵。我們預期透過本計畫，以非監督式學習方式解決事先大量人力評分的問題，以及透過更多種類特徵以及偏離主題偵測來解決惡意欺騙的問題。這個研究計畫的結果，可以為眾多領域例如數位學習、心理計量、教育測驗等提供可實際應用的、高效能的研究工具。在本計畫中，我們將持續針對CAES 系統的三個部份進行強化與功能提升: 第一是架構在我們於第二期計畫中之非監督式學習方式，提出如何偵測偏離主題的新方法，第二是針對中文作文，提出如何找出更具作文評分依據之語義特徵的方法以及以非監督式機器自學習方法為基礎建立預防惡意欺騙之攻擊，第三是提出更多種類的特徵，特別是目前原型系統中欠缺的結構特徵。我們預期經此計劃強化後的系統能進入實際運轉階段，也能為競爭日益激烈的海外中文學習市場提供有力的競爭工具。

Automatic essay scoring (AES) system is a very important research tool for such areas as educational testing and psychometrics because studies in these domains often rely on a large number of writings to conduct various analyses. It is, however, often very difficult to obtain a large number of graded writings due to expensive cost and time consuming process of human grading. In English, the successful development of automatic essay scoring system in the past years has overcome these limitations and largely facilitated the progress of the stated research area. By contrast, the lack of Chinese automatic essay scoring system (CAES) has limited the scale, quantities, and validity of these research areas. The linguistic differences of the languages between Chinese and English suggest the need to reconsider various issues when designing CAES. Hence it is difficult to apply the current techniques of English AES systems to Chinese writings. We have developed a Chinese automated essay scoring system in the past which is far from practical application in the real world. Two main difficulties exist: (1) The system needs a large number of training data with score. (2) There exists a risk that hostile user might trick the system so that a bad writing might score very well through the weakness of the system. In the previous project, we have some research results. One of the papers has been accepted by “IEEE Intelligent Systems journal” and one of the papers has been accepted by “Information: An International Interdisciplinary Journal”. Two conference papers have been accepted. One paper is under 2nd review by “IEEE Intelligent Systems journal” and we have submitted another 4 papers to related journals. The aim of the study is to propose and to develop second generation of Chinese automatic essay scoring system so that it can be used in the area of educational research. Our aims will focus on (1) Develop an unsupervised machine learning model for CAES system, (2) Develop a method for detecting writings with digressing topics, (3) Design versatile and various features, in particular structural and semantic features. We expect that the proposed system will become a powerful tool for learning Chinese writings.