Clean up Notebooks and move part of the code to Python files (#37)

* Adding SentencePiece

* Rearrange

* Adding finetuning scripts

* Making dir structure more managable

* Add Random Search

* Add majority voting

* Strip notebooks

* Add nb stripout from fastai standalone version

* Move everything to one notebook

* Change the name of the file saved

* Change the parameters

* Combine common code

* Pass "name" to methods in hinglish utils

* fix imports

* Pass name as variable to add_padding

* Remove output

* Fix typo

* Change names for ensemble models

* Change from "output" to name of the LM model

* Fix typo

* Remove hardcoding for epochs

* isort and black :)

* Move everything to hinglish

* Coffe isn't good for me

* Break everything into sensible methods

* import clash

* fix run_valid

* Change notebook according to hinglish.py

* Change logs

* Fix imports

* Change method names

* Remove tf dependency

* Remove tf from requirements.txt

* Remove hardcoding from pd columns

* Use store_attr to load variables for class

* Fix the size mismatch error by changing final_test.json file

* nb-stripout worked

* Add majority voting explanation

* extract if tarfile and run_language_modeling documentation

* Split the transformers notebook

* Something broke, I don't know what.

* Remove setup

* Things would be easier if I knew OOP or Python better

* Will fix this later is this works¿

* nan sent¿

* Print eval and test metrics

* Change the label for eval

* Change the file with empty clean_text

* Remove eval testing for now

* Logfile name

* moving the part which copies things to drive here

* Fix formatting add pathlib

* add drivepath

* Changed the file paths

* Remove additional code

Co-authored-by: meghanabhange <[email protected]>
Co-authored-by: meghanabhange <[email protected]>
Co-authored-by: meghanabhange <[email protected]>

.gitignore

@@ -102,3 +102,5 @@ venv.bak/
 
 # mypy
 .mypy_cache/
+.vscode
+data/Ensemble/*