顯示具有 traditional chinese 標籤的文章。 顯示所有文章
顯示具有 traditional chinese 標籤的文章。 顯示所有文章

2012年12月12日 星期三

Better Traditional Chinese support in Lucene / Solr

    I discover the the approach you may want to know
[ Search Engine Platform ]
    I use LucidWorks Enterprise to have quick POC and initial implementation
[ Detail Configuration / Setup ]
1. Have a new field to store/index TC
2. Configure this field by Lucence SmartChineseAnalyzer. You can refer to an instruction in detail configuration.
3. Instruct an Analyzer(solr.MappingCharFilterFactory) to do character/sentence translation from TC to SC with a customized mapping file.
4. Change default defType(query parser) from lucid to edismax because 1st filter supports English only.

    There isn't a dedicate/better Traditional Chinese(Big5 code)(we use TC in this article) analyzer/tokenizer in Lucene/Solr. There are some good Simplified Chinese(GB code)(we use SC in this article) analyzer/tokenizer. Finally, I choose SmartChineseAnalyzer due to 2 major features
1. The algorithm : It uses Hidden Markov Model and it was proved that over 98% search accuracy by another implementation(ICTCLASx).
2. It has a good integration/plug-in w/o further works.