Resolve a SolrException in Solr or LucidWorks

       Solr (or LucidWorks ) requires unique id when you import data from Oracle database via Data Import Request Handler(DIH). If not, you would experience 2 kinds of exception(depends on what a system you use, Solr or LucidWorks)
1. org.apache.solr.common.SolrException: Document is missing mandatory uniqueKey field: id
2. Document [null] missing required field: id
       Unfortunately, even you alias the field name of primary key as "id". The system still reports "Document [xx] missing required field". The root cause is, Oracle always return field names as capital. So, how do we resolve it ?

• Solr user

     You can add a field transform in data-config.xml like this

    <entity name="root" pk="id" preImportDeleteQuery="data_source:29" query="select id, description from table1" transformer="TemplateTransformer" >

       <field column="[your field name in capital]" name="id" />

    </entity>

• LucidWorks user

      PLS manual edit dataconfig_[xx].xml in [LucidWorks]\conf\solr\cores\[your collection name]\conf. There is no way to edit it through ADMIN pages in LucidWorks. Find the entity XML element and add field transform element.
       <field column="[your field name in capital]" name="id" />

Hope it could resolve the problem you experience.

Better Traditional Chinese support in Lucene / Solr

    I discover the the approach you may want to know
[ Search Engine Platform ]
    I use LucidWorks Enterprise to have quick POC and initial implementation
[ Detail Configuration / Setup ]
1. Have a new field to store/index TC
2. Configure this field by Lucence SmartChineseAnalyzer. You can refer to an instruction in detail configuration.
3. Instruct an Analyzer(solr.MappingCharFilterFactory) to do character/sentence translation from TC to SC with a customized mapping file.
4. Change default defType(query parser) from lucid to edismax because 1st filter supports English only.

    There isn't a dedicate/better Traditional Chinese(Big5 code)(we use TC in this article) analyzer/tokenizer in Lucene/Solr. There are some good Simplified Chinese(GB code)(we use SC in this article) analyzer/tokenizer. Finally, I choose SmartChineseAnalyzer due to 2 major features
1. The algorithm : It uses Hidden Markov Model and it was proved that over 98% search accuracy by another implementation(ICTCLASx).
2. It has a good integration/plug-in w/o further works.

Enterprise Search Engine 企業級搜尋引擎

Lucene / Solr / Tika / Nutch / LucidWorks

        近來預期要導入企業內部search engine(不遠的將來可以會有KM的需求), 要能夠抓DB / crawl intranet webs / 文件庫(whatever file repository or MS-SharePoint)
        馬上想到的標的就是Google Appliance / 龍捲風(Tornado) / Apache Lucene這幾個常見的解決方案. 想當然爾, "要馬兒跑又要馬兒不吃草"的潛規則就自動變成"MIS限制理論". 原本國外分公司有購買Google Appliance, 這就是最好的選擇, 好用又不用自己(台灣分公司)花錢. 不幸的是...只用在internet網站. 只好繼續尋找.
         一剛開始, 由於有先前survey CMDBuild (這個好東西改天再來介紹, 該開發團隊真是太強了),  進而知道Alfresco( CMS )可搜尋PDF & Office文件, 底層也是Apache Lucene. 試了很久, 發現並不是我想要的. 或許是殺雞焉用牛刀吧, 又或許是難以快速上手, 又或許是難以擴充功能(crawl intranet webs. 最後還是放棄了.
         初步查看Lucene相關資料後, 網路都推薦Lucene + Solr. 再加上自身的需求就變成Lucene + Solr + Tika (document parser/analyzer) + Nutch (web crawl)的套餐了. 只不過因為"偷懶", 因為還要開發或整合search UI. 研究該套餐數天後, 又轉換研究目標到LucidWorks Enterprise. 其實該system也是Lucene + Solr + Tika + Nutch + ....的集成. 加上已經設計好的UI( Admin & user ), easy to use and quick configuration to production/pilot run. 加上原本Solr的彈性配置, 其實還是具有擴充性.

Useful Reference

http://blog.building-blocks.com/building-a-search-engine-with-nutch-and-solr-in-10-minutes