v20.04.00

Remove previously tokenized files if concat occurs (#21)

* fix concat on regex

* temp git ignore change

* remove dir

* remove check internally

* fix tokenize

* fix filesystem tweaks

* rm filesystem stuff

* remove

v19.12.00

[Feature:Lichen] Add MIPS tokenizer (#17)

* Add MIPS tokenizer

The tokenizer identifies string literals, labels, dot-types, registers,
immediate values, instructions, addresses, and comments.  If two
instructions are placed on the same line, this will identify the first as
an instruction and the second as a label, however that's not too bad since
MIPS syntax only allows one instruction per line.

* Add installation

v.18.12.00: Allow regex for files in configuration (#15)

* regex for files

* remove comment

* regrex -> regex

v.18.09.00: Update llvm version for C tokenizer (#14)

* update print statement

* update llvm version

v.18.07.04: Sort processing order & parse json config file (#13)

Sort the processing order of users/versions (helps debugging)
Moved from command line arguments for each script to parsing the config json

v.18.07.03

Double hash size (#12)

v.18.07.02

Workaround to skip utf-8 characters in plaintext tokenizer (#11)

v.18.07.01: Revision to Process all script (#10)

* fix python_tokenizer tok_name error

* remove import which is not required in python_tokenizer.py

* remove empty value tokens

* modified process_all.sh to take more argument

initial java tokenizer and bugfix for python tokenizer

fix python_tokenizer tok_name error (#9)

* fix python_tokenizer tok_name error

* remove import which is not required in python_tokenizer.py

* remove empty value tokens

v.18.06.01: Make sure the matches file does not output matches to the same user (#7)

Make sure the matches file does not output matches to the same user