Static Tokenizer

[sfluegel05]

This implements a new static tokenizer for SMILES (see #166).

The new tokenizer only needs 572 tokens which it achieves by splitting each atom into 5 tokens (element, charge, isotope, stereochemistry and hydrogen count). As demonstrated in #166, all SMILES strings in ChEBI and PubChem can be parsed with this tokenizer.

Also, the implementation includes a decoder that reconstructs SMILES strings as far as possible (some SMILES cannot be reconstructed perfectly since the encoding is not injective. E.g. [1*] and [2*] both get resolved to *).

Todo

• check the implications of longer inputs for the ELECTRA model
• make this reader the default for PubChem and ChEBI classes

[sfluegel05]

I changed the tokenizer to skip some default tokens. This brings down the input length to a maximum of 1,196 tokens (for a 1,674 character SMILES, CHEBI:156639).

Training with this tokenizer is successful on ChEBI:

Run	Tokenisation	Time on A100	Micro-F1	Macro-F1
1	From list	15h 36m	*	*
2	From list	**	0.89131	0.6193
3	Static	23h 0m	0.89557	0.63011

.* Not comparable, different loss function
.** Not comparable, different GPU

Overall, the static tokenization run took a bit longer, but got slightly better results (which may not be significant).