Currently I am researching on the solution for a real-time search engine which scourge user submitted contents. The user pool consists of 250+ private institutions which translate into 10K-15K users. I/O will be relatively low, but the data size could be ranging wildly since the users are uploading Microsoft Words, Excel and PDF file.
The index is based on the user uploaded files (mostly are Word/Excel/PDF/PowerPoint, and ASCII files). The I/O is expected at only 10 IOPS -20 IOPS but it can vary depends on the date. Maximum I/O could be 100 IOPS. Current database size is reaching 10GB, it's 4 months old.
For real time search server, I'm considering Solr / Lucene and probably ElasticSearch. But the challenge is how to index these files FAST, so that search server can query the index in real time.
I have found some similar questions on how to index .doc/.xls/.pdf, but they did not mention how to ensure indexing performance:
- Search for keywords in Word documents and index them
- Index Word/PDF Documents From File System To SQL Server
- How to extract text from MS office documents in C#
- Using full-text search with PDF files in SQL Server 2005
How to build the index FAST ?
Any suggestion on the architecture ? Should I focus on building fast infrastructure (i.e. RAID, SSD, more CPU, Network bandwidth ?) or focus on the index tools & algorithm?
P/S: this question is asked on Stackoverflow as well
I can't speak for Sphinx, but choose Elasticsearch w/ attachments plugin over Solr for scalability and real-time reqs.
ReplyDeleteCool, I'll try it over the weekend and update this blog on the result.
ReplyDeleteI got Solr working already. Took me a week to get it running.
ReplyDeleteDuring this period read alot of stuff around. Books, manuals.
One thing you should notice, for your case: Apache Tika analyzers are well integrated within Solr.
I am seriously considering ElasticSearch, though, because of positive community feedbacks, and my good impression about its docs, and supplementary technologies (See their website.)
In any case, I'll have several months ahead of me to tune-up Search component for the Biz. And will make decisions as I go.
I had a look at Apache Tika API and the supported format - look promising there :)
ReplyDeleteLet me know your experience with ElasticSearch. I'm also sold by the convenience in writing JSON query for ElasticSearch - it makes the test much easier.