Character line size limit

[Jacob-Eliat-Eliat]

There is a debate between using byte size and character size.

This is my preferred approach because:

• while it's easy for a customer to check the byte size of a file, it's less straightforwards with a line, because it depends on encoding, and the file won't report it to them directly.
• it avoids unnecessary processing and uses ready-made fs2 methods

here is a draft of the alternative to give you an idea of what the flow would be like:
#995

(not if the alternative is preferred, the processing code needs a heavy coat of cleanup but the main idea is there)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.30%. Comparing base (fdfe8f2) to head (996c0da).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #996      +/-   ##
==========================================
+ Coverage   83.27%   83.30%   +0.02%     
==========================================
  Files          47       47              
  Lines        3145     3150       +5     
  Branches      483      432      -51     
==========================================
+ Hits         2619     2624       +5     
  Misses        526      526              

Files with missing lines	Coverage Δ
...n/scala/cognite/spark/v1/FileContentRelation.scala	95.23% <100.00%> (+1.48%)	⬆️

[dmivankov]

there's .zipWithIndex method that can expose indexes, not sure if we can attach it to error message though 🤔
But that's only a nice to have extra error detail

[Jacob-Eliat-Eliat]

It is possible, but then we cannot use linesLimited, we'd need to do a second pass with indexes and limit there. Not very hard, how much do you think this is important?