You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The way the Run method works is that each token is put into ASTImpl::preprocessed_code on its own line, so that we can map from source location line number back to index in ASTImpl::tokens. This means fiddling with embedded newlines in tokens, and thus introduces flakiness. My guess is that this approach is going to fall over when we want to handle C++ multiline string literals. Regardless, an easier approach would be to append unadulterated token data into ASTImpl::preprocessed_code, and build up a vector of token begin or token end indexes, for binary searching later. Alternatively, we could try some kind of binary search inside of ASTImpl::tokens, as they have data_offset, but that will require being clever, as the data_offset has an overloaded meaning.
The text was updated successfully, but these errors were encountered:
The way the
Run
method works is that each token is put intoASTImpl::preprocessed_code
on its own line, so that we can map from source location line number back to index inASTImpl::tokens
. This means fiddling with embedded newlines in tokens, and thus introduces flakiness. My guess is that this approach is going to fall over when we want to handle C++ multiline string literals. Regardless, an easier approach would be to append unadulterated token data intoASTImpl::preprocessed_code
, and build up a vector of token begin or token end indexes, for binary searching later. Alternatively, we could try some kind of binary search inside ofASTImpl::tokens
, as they havedata_offset
, but that will require being clever, as thedata_offset
has an overloaded meaning.The text was updated successfully, but these errors were encountered: