Maintain line and character number after parsing

[gregorybchris]

Evaluation has many failure cases, many of which have pretty reasonable error messages. However we do not maintain the line and character numbers associated with nodes in the AST. The interpreter can't point to the part of the code associated with a failure that occurs during evaluation.

This feature request is to make code index information available in the evaluation stage.

[tekknolagi]

This is a little finicky right now unless we want to store (invalid) line number information on all objects because the AST structures are objects themselves. Perhaps we should separate them out (ehhhh) or find another way to do a mixin or something.

[tekknolagi]

came across this out of nowhere this morning

[tekknolagi]

the reasoning is a little weird but i think we can calculate the line/col from the byte position easily and it's only one number to store

[tekknolagi]

I saw/skimmed part of a cool talk that did this for Circle, I think. Perhaps this talk https://www.youtube.com/watch?v=1m_5SVmGA4k

They have some cool tricks

[tekknolagi]

Oh, no, lmao, it was Carbon: https://www.youtube.com/watch?v=ZI198eFghJk
Starting ~30mins?

[tekknolagi]

cc @neuroevolutus maybe this is also interesting

I think we want to keep line/col on all AST objects but not on interstitial/value objects. This probably means setting them dynamically after construction and having helper functions has_sourcepos and sourcepos. Then hopefully we can use this in parse/eval/compile errors!

Maybe we can one day even emit #line directives in the compiler.

[neuroevolutus]

Neat, I'll have to take a look at the referenced article and video.

[neuroevolutus]

@tekknolagi I started working on a pull request for this, and I realised that adding additional fields to the Token class to represent column numbers and/or byte positions would cause the golden tests that check for the string contents of ParseErrors to fail. Would it be acceptable to change those tests to instead test for the presence of the relevant fields and their specific values instead so that they become more immune to changes in the string representation of ParseErrors?

[tekknolagi]

Sure! Please split that out into a separate commit (same PR) onto which you stack your lineno commit.

[tekknolagi]

In general, tests are as mutable as the rest of the code, and I like your philosophy of having stable tests.

Some tests should check end-to-end string stuff though so that we have tests for stuff users see.