Finding long tail suggestions using Lucene’s new FreeTextSuggester by Mike McCandless.
From the post:
Lucene’s suggest module offers a number of fun auto-suggest implementations to give a user live search suggestions as they type each character into a search box.
For example,
WFSTCompletionLookupcompiles all suggestions and their weights into a compact Finite State Transducer, enabling fast prefix lookup for basic suggestions.
AnalyzingSuggesterimproves on this by using anAnalyzerto normalize both the suggestions and the user’s query so that trivial differences in whitespace, casing, stop-words, synonyms, as determined by the analyzer, do not prevent a suggestion from matching.Finally,
AnalyzingInfixSuggestergoes further by allowing infix matches so that words inside each suggestion (not just the prefix) can trigger a match. You can see this one action at the Lucene/Solr Jira search application (e.g., try “python”) that I recently created to eat our own dog food. It is also the only suggester implementation so far that supports highlighting (this has proven challenging for the other suggesters).Yet, a common limitation to all of these suggesters is that they can only suggest from a finite set of previously built suggestions. This may not be a problem if your suggestions are past user queries and have tons and tons of them (e.g., you are Google). Alternatively, if your universe of suggestions is inherently closed, such as the movie and show titles that Netflix’s search will suggest, or all product names on an e-commerce site, then a closed set of suggestions is appropriate.
….
Since you are unlikely to be Google, Mike goes on to show how FreeTextSuggester can ride to your rescue!
As always, Mike’s post is a pleasure to read.