顯示具有 Solr 標籤的文章。 顯示所有文章
顯示具有 Solr 標籤的文章。 顯示所有文章

2013年1月7日 星期一

Utilize TermsComponent in LucidWorksEnterprise(LWE)


1. Modify solrconfig.xml
You can find lots of samples in internet. I still keep "autocomplete" because I still want to keep primary design of LWE
<requestHandler class="org.apache.solr.handler.component.SearchHandler" name="/autocomplete">
<lst name="defaults">
   <bool name="terms">true</bool>
   <str name="terms.fl">autocomplete</str>
   <str name="terms.sort">count</str>
   <str name="terms.limit">20</str>
</lst>
   <arr name="components">
<str>terms</str>
   </arr>
</requestHandler>
<searchComponent class="solr.TermsComponent" name="terms"/>

2. Modify search.rb in [LWE]\app\webapps\lwe-ui\WEB-INF\app\models\core
a. In order to make TermsComponent works and not to alter LWE a lot, we want to add 1. terms.prefix=xxxx or change "q"  to "terms.prefix". Finally, I choose option#1. My approach is adding one parameter in the http request at line#230(keyword is when :autocomplete) in function,  solr_params(xxxx), new codes are :
params.merge!(
'terms.prefix' => params[:q]
)
b. Because SpellCheck and TermsComponent returns diff. data scheme, we need to add one parser at line#400(keyword is def autocompletions)
if @results['spellcheck'] == nil
@results['terms']['autocomplete'] rescue []    
else
@results['spellcheck']['suggestions'].at(1)['suggestion'] rescue []    
end



2013年1月3日 星期四

[Notes] Solr AutoComplete resources

http://www.medihack.org/2011/03/01/autocompletion-autosuggestion-using-solr/

http://lucene.472066.n3.nabble.com/auto-completion-search-with-solr-using-NGrams-in-SOLR-td3998559i20.html

http://stackoverflow.com/questions/2768770/how-to-structure-an-index-for-type-ahead-for-extremely-large-dataset-using-lucen


http://www.cominvent.com/2012/01/25/super-flexible-autocomplete-with-solr/

2012年12月20日 星期四

Resolve a SolrException in Solr or LucidWorks

       Solr (or LucidWorks ) requires unique id when you import data from Oracle database via Data Import Request Handler(DIH). If not, you would experience 2 kinds of exception(depends on what a system you use, Solr or LucidWorks)
1. org.apache.solr.common.SolrException: Document is missing mandatory uniqueKey field: id
2. Document [null] missing required field: id
       Unfortunately, even you alias the field name of primary key as "id". The system still reports "Document [xx] missing required field". The root cause is, Oracle always return field names as capital. So, how do we resolve it ?


  • Solr user
     You can add a field transform in data-config.xml like this
    <entity name="root" pk="id" preImportDeleteQuery="data_source:29" query="select id, description from table1" transformer="TemplateTransformer" >
       <field column="[your field name in capital]" name="id" />
    </entity>
  • LucidWorks user
      PLS manual edit dataconfig_[xx].xml in [LucidWorks]\conf\solr\cores\[your collection name]\conf. There is no way to edit it through ADMIN pages in LucidWorks. Find the entity XML element and add field transform element.
       <field column="[your field name in capital]" name="id" />

Hope it could resolve the problem you experience.

2012年12月12日 星期三

Better Traditional Chinese support in Lucene / Solr

    I discover the the approach you may want to know
[ Search Engine Platform ]
    I use LucidWorks Enterprise to have quick POC and initial implementation
[ Detail Configuration / Setup ]
1. Have a new field to store/index TC
2. Configure this field by Lucence SmartChineseAnalyzer. You can refer to an instruction in detail configuration.
3. Instruct an Analyzer(solr.MappingCharFilterFactory) to do character/sentence translation from TC to SC with a customized mapping file.
4. Change default defType(query parser) from lucid to edismax because 1st filter supports English only.

    There isn't a dedicate/better Traditional Chinese(Big5 code)(we use TC in this article) analyzer/tokenizer in Lucene/Solr. There are some good Simplified Chinese(GB code)(we use SC in this article) analyzer/tokenizer. Finally, I choose SmartChineseAnalyzer due to 2 major features
1. The algorithm : It uses Hidden Markov Model and it was proved that over 98% search accuracy by another implementation(ICTCLASx).
2. It has a good integration/plug-in w/o further works.

2012年12月11日 星期二

基於Search Engine之中文分詞

http://my.opera.com/jinglepot/blog/index.dml/tag/solr
http://kb.trisugar.com/node/9751
http://code.google.com/p/mmseg4j/ mmseg4j MMSEG for java lucene chinese analyzer, or for solr
http://chenlb.iteye.com/blog/353334 中文分词 mmseg4j
http://chenlb.iteye.com/blog/439843 讨论 mmseg4j 的现状,与改进
http://railsfun.tw/viewthread.php?tid=672 CoreSeek :Sphinx + 中文字典檔:中文分詞搜尋引擎
http://www.mobile01.com/topicdetail.php?f=62&t=339949&last=34826211 求繁體中文版的ZDic[台灣教育部國語辭典]
http://blog.fulin.org/2009/08/lucene_chinese_analyzer_compare.html 当前几个主要的Lucene中文分词器的比较
http://technology.chtsai.org/mmseg/ MMSEG: A Word Identification System for Mandarin Chinese Text Based on Two Variants of the Maximum Matching Algorithm
http://manan.org/2011/08/using_ictclas_in_java_and_lucene/ 在Java程序和Lucene中使用中科院分词系统ICTCLAS
http://ictclas.org/content_java_002.html 给Lucene加入性能更好的中文分词
http://hi.baidu.com/drkevinzhan ICTCLAS张华平博士的空间
http://kb.trisugar.com/node/9751 利用Lucene制作中文搜尋應用


2012年12月10日 星期一

Enterprise Search Engine 企業級搜尋引擎

Lucene / Solr / Tika / Nutch / LucidWorks

        近來預期要導入企業內部search engine(不遠的將來可以會有KM的需求), 要能夠抓DB / crawl intranet webs / 文件庫(whatever file repository or MS-SharePoint)
        馬上想到的標的就是Google Appliance / 龍捲風(Tornado) / Apache Lucene這幾個常見的解決方案. 想當然爾, "要馬兒跑又要馬兒不吃草"的潛規則就自動變成"MIS限制理論". 原本國外分公司有購買Google Appliance, 這就是最好的選擇, 好用又不用自己(台灣分公司)花錢. 不幸的是...只用在internet網站. 只好繼續尋找.
         一剛開始, 由於有先前survey CMDBuild (這個好東西改天再來介紹, 該開發團隊真是太強了),  進而知道Alfresco( CMS )可搜尋PDF & Office文件, 底層也是Apache Lucene. 試了很久, 發現並不是我想要的. 或許是殺雞焉用牛刀吧, 又或許是難以快速上手, 又或許是難以擴充功能(crawl intranet webs. 最後還是放棄了.
         初步查看Lucene相關資料後, 網路都推薦Lucene + Solr. 再加上自身的需求就變成Lucene + Solr + Tika (document parser/analyzer) + Nutch (web crawl)的套餐了. 只不過因為"偷懶", 因為還要開發或整合search UI. 研究該套餐數天後, 又轉換研究目標到LucidWorks Enterprise. 其實該system也是Lucene + Solr + Tika + Nutch + ....的集成. 加上已經設計好的UI( Admin & user ), easy to use and quick configuration to production/pilot run. 加上原本Solr的彈性配置, 其實還是具有擴充性.

Useful Reference

http://blog.building-blocks.com/building-a-search-engine-with-nutch-and-solr-in-10-minutes