Improve Jaccard Similarity by filtering tokens.

[PhilipMay]

No description provided.

[PhilipMay]

Valid SoMaJo tekens:

{'regular': 1, 'symbol': 1, 'number': 1, 'ordinal': 1, 'date': 1, 'abbreviation': 1, 
'number_compound': 1, 'time': 1, 'email_address': 1, 'URL': 1, 'measurement': 1, 
'emoticon': 1, 'XML_entity': 1, 'hashtag': 1, 'amount': 1, 'XML_tag': 1, 'mention': 1, 
'action_word': 1, 'semester': 1}

[PhilipMay]

Token types:

regular :        ['Als', 'AHS', 'bieten']
symbol :         ['(', '+', ')']
number :         ['4', '4', '7']
ordinal :        ['50.', '12.', '1.']
date :   ['28.', '4.', '2012']
abbreviation :   ['z.', 'B.', 'Nr.']
number_compound :        ['15-jährige', '3sat', 'PS3']
time :   ['17:00', '14:00', '19:00']
email_address :  ['[email protected]', '[email protected]', '[email protected]']
URL :    ['www.secondred.de', 'google.com', 'https://dejure.org/1962,11095']
measurement :    ['12', 'km', '2,4']
emoticon :       ['©', ']:', ']:']
XML_entity :     ['&', '&', '&']
hashtag :        ['#8222', '#8220', '#8222']
amount :         ['10,-', '15,-', '10,-']
XML_tag :        ['<br/>', '<a href="https://all-inkl.com/umweltschutz/" target="_blank">', '<img src="https://all-inkl.com/banner/all-inkl_banner_157x54_oekostromv1.png" alt="gehostet mit 100% Ökostrom von all-inkl.com" border="0" />']
mention :        ['@teinachtal', '@pameier', '@A_Cassidy']
action_word :    ['*', 'Entdeckertipp', '*']
semester :       ['SS', '20', 'WS']