Challenges in maintaining a high performance search engine written in Java
You will find this on the homepage or may have to search for it. I was logged in when I accessed it.
Very much worth your time for a high level overview of issues that Lucene will face sooner rather than later.
After reviewing, think about it, make serious suggestions and if possible, contributions to the future of Lucene.
Just off the cuff, I would really like to see Lucene become a search engine framework with a default data structure that admits to either extension or replacement by other data structures. Some data structures may have higher performance costs than others, but if that is what your requirements call for, they can hardly be wrong. Yes? A “fast” search engine that doesn’t meet your requirements is no prize.