If possbile to read CVS to a Flow instead of List

[hantsy]

Currently I can read a CSV to a List result as following, but for large data, it is obviously not a good option.

DataFrame
  .readCSV(file)
  .cast<MyCsvItemData>()
  .toList()

If possible to read Csv line by line and parse them one by one and emit the parsed result into a Flow.

[koperagen]

It looks like we don't have a solution for this problem in API, but i can suggest trying something like this. Providing that in your CSV data one line == one record. Sometimes record can span on several lines. If that's the case, then i'm afraid there's no workaround and we need to modify our reader

val file = File("data.csv")
val header = file.useLines { it.first() }
sequence {
    file.useLines { lines ->
        val sb = StringBuilder()
        lines
            .windowed(1000, 1000, partialWindows = true)
            .forEach {
                sb.appendLine(header)
                it.joinTo(sb, separator = "\n")
                val df = DataFrame.readDelim(StringReader(sb.toString()))
                sb.clear()
                yield(df)
            }
    }
}