Commit 072f7c96 authored by Dos Santos David's avatar Dos Santos David

add default tokenizer

parent 05d0381c
from gogole.tokenizer.abstract_tokenizer import AbstractTokenizer
class NoTokenizer(AbstractTokenizer):
def get_tokens(self, document):
return document.get_raw_content().strip().split()
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment