Method GetTokenStream

Low level api. Returns a token stream generated from a . This can be used to feed the highlighter with a pre-parsed token stream. The must have offsets available.

In my tests the speeds to recreate 1000 token streams using this method are:

• with TermVector offset only data stored - 420 milliseconds
• with TermVector offset AND position data stored - 271 milliseconds (nb timings for TermVector with position data are based on a tokenizer with contiguous positions - no overlaps or gaps)
• The cost of not using TermPositionVector to store pre-parsed content and using an analyzer to re-parse the original content:
  • reanalyzing the original content - 980 milliseconds

The re-analyze timings will typically vary depending on -

1. The complexity of the analyzer code (timings above were using a stemmer/lowercaser/stopword combo)
2. The number of other fields (Lucene reads ALL fields off the disk when accessing just one document field - can cost dear!)
3. Use of compression on field storage - could be faster due to compression (less disk IO) or slower (more CPU burn) depending on the content.