- P-ISSN1013-0799
- E-ISSN2586-2073
- KCI
위키피디아를 이용한 분류자질 선정에 관한 연구
An Experimental Study on Feature Selection Using Wikipedia for Text Categorization
정영미 (연세대학교)
초록
텍스트 범주화에 있어서 일반적인 문제는 문헌을 표현하는 핵심적인 용어라도 학습문헌 집합에 나타나지 않으면 이 용어는 분류자질로 선정되지 않는다는 것과 형태가 다른 동의어들은 서로 다른 자질로 사용된다는 점이다. 이 연구에서는 위키피디아를 활용하여 문헌에 나타나는 동의어들을 하나의 분류자질로 변환하고, 학습문헌 집합에 출현하지 않은 입력문헌의 용어를 가장 유사한 학습문헌의 용어로 대체함으로써 범주화 성능을 향상시키고자 하였다. 분류자질 선정 실험에서는 (1) 비학습용어 추출 시 범주 정보의 사용여부, (2) 용어의 유사도 측정 방법(위키피디아 문서의 제목과 본문, 카테고리 정보, 링크 정보), (3) 유사도 척도(단순 공기빈도, 정규화된 공기빈도) 등 세 가지 조건을 결합하여 실험을 수행하였다. 비학습용어를 유사도 임계치 이상의 최고 유사도를 갖는 학습용어로 대체하여 kNN 분류기로 분류할 경우 모든 조건 결합에서 범주화 성능이 0.35%~1.85% 향상되었다. 실험 결과 범주화 성능이 크게 향상되지는 못하였지만 위키피디아를 활용하여 분류자질을 선정하는 방법이 효과적인 것으로 확인되었다.
- keywords
- text categorization, document classification, feature selection, Wikipedia, term similarity, non-learning term, text categorization, document classification, feature selection, Wikipedia, term similarity, non-learning term, 자동 분류, 자질 선정, 위키피디아, 용어 간 유사도, 비학습용어
Abstract
In text categorization, core terms of an input document are hardly selected as classification features if they do not occur in a training document set. Besides, synonymous terms with the same concept are usually treated as different features. This study aims to improve text categorization performance by integrating synonyms into a single feature and by replacing input terms not in the training document set with the most similar term occurring in training documents using Wikipedia. For the selection of classification features, experiments were performed in various settings composed of three different conditions: the use of category information of non-training terms, the part of Wikipedia used for measuring term-term similarity, and the type of similarity measures. The categorization performance of a kNN classifier was improved by 0.35~1.85% in F1 value in all the experimental settings when non-learning terms were replaced by the learning term with the highest similarity above the threshold value. Although the improvement ratio is not as high as expected, several semantic as well as structural devices of Wikipedia could be used for selecting more effective classification features.
- keywords
- text categorization, document classification, feature selection, Wikipedia, term similarity, non-learning term, text categorization, document classification, feature selection, Wikipedia, term similarity, non-learning term, 자동 분류, 자질 선정, 위키피디아, 용어 간 유사도, 비학습용어
참고문헌
Bird, S.. (2007). Natural language processing in Python:O'ReillyMedia.
Gabrilovich, E.. (2005). Feature generation for text categorization using world knowledge (1048-1053). Proceedings of the 19th international Joint Conference on Artificial intelligence.
Gabrilovich, E.. (2006). Overcoming the brittleness bottleneck using Wikipedia : enhancing text categorization with encyclopedic knowledge (1301-1306). Proceedings of the 21st National Conference on Artificial Intelligence.
Gabrilovich, E.. (2007). Computing semantic relatedness using Wikipedia-based explicit semantic analysis (1606-1611). Proceedings of the 20th International Joint Conference on Artificial Intelligence.
Huang, A.. (2009). Clustering documents using a Wikipediabased concept representation (628-636). Proceedings of the 13th Pacific-Asia Conference on Advances in Knowledge Discovery and Data Mining (LNCS 5476/2009).
Milne, D.. (2007). A knowledge-based search engine powered by Wikipedia (445-454). Proceedings of the 16th ACM Conference on Information and Knowledge Management.
Milne, D.. (2008). An effective, low-cost measure of semantic relatedness obtained from Wikipedia links (-). Proceedings of the First AAAI Workshop on Wikipedia and Artificial Intelligence, (WIKIAI 2008).
Minier, Z.. (2007). Wikipedia-based kernels for text categorization (157-164). Proceedings of the International Symposium on Symbolic and Numeric Algorithms for Scientific Computing.
Ponzetto, S. P.. (2006). Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution (192-199). Proceedings of the Main Conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics.
Ponzetto, S. P.. (2007). Knowledge derived from Wikipedia for computing semantic relatedness. Journal of Artificial Intelligence Research, 30(1), 181-212.
Strube, M.. (2006). WikiRelate! Computing semantic relatedness using Wikipedia (1419-1424). Proceedings of the 21st National Conference on Artificial Intelligence.
Wang, P.. (2007). Improving text classification by using encyclopedia knowledge (332-341). Proceedings of the 2007 Seventh IEEE International Conference on Data Mining.
Wang, P.. (2008). Building semantic kernels for text classification using wikipedia (713-721). Proceeding of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.
- 다운로드 수
- 조회수
- 0KCI 피인용수
- 0WOS 피인용수