TeslaFi Import fails if one line is mailformed

[dbcm]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

I'm trying to import teslaFi data, import process fails. Tested with latest and edge releases.

Console logs (removed coordinates):

teslamate | 2025-01-05 21:38:24.662 [error] GenStateMachine :import_Fly terminating
teslamate | ** (MatchError) no match of right hand side value: {:error, #Ecto.Changeset<action: :insert, changes: %{date: ~U[2024-11-23 20:29:43.000000Z], power: 1, latitude: Decimal.new("XXX"), longitude: Decimal.new("XXX"), odometer: 73381.499485, ideal_battery_range_km: Decimal.new("109.48"), outside_temp: Decimal.new("17.5"), rated_battery_range_km: Decimal.new("109.48"), usable_battery_level: 28, driver_temp_setting: Decimal.new("21.0"), est_battery_range_km: Decimal.new("111.37"), fan_status: 0, inside_temp: Decimal.new("19.7"), passenger_temp_setting: Decimal.new("21.0")}, errors: [battery_level: {"is invalid", [type: :integer, validation: :cast]}], data: #TeslaMate.Log.Position<>, valid?: false, ...>}
teslamate | (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:766: anonymous fn/4 in TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate | (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1400: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
teslamate | (db_connection 2.7.0) lib/db_connection.ex:1756: DBConnection.run_transaction/4
teslamate | (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:758: TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate | (stdlib 5.2.3.1) gen_statem.erl:1397: :gen_statem.loop_state_callback/11
teslamate | (stdlib 5.2.3.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
teslamate | 2025-01-05 21:38:24.664 [warning] Import failed: {{:badmatch,
teslamate | {:error,
teslamate | #Ecto.Changeset<
teslamate | action: :insert,
teslamate | changes: %{
teslamate | date: ~U[2024-11-23 20:29:43.000000Z],
teslamate | power: 1,
teslamate | latitude: Decimal.new("XXX"),
teslamate | longitude: Decimal.new("XXX"),
teslamate | odometer: 73381.499485,
teslamate | ideal_battery_range_km: Decimal.new("109.48"),
teslamate | outside_temp: Decimal.new("17.5"),
teslamate | rated_battery_range_km: Decimal.new("109.48"),
teslamate | usable_battery_level: 28,
teslamate | driver_temp_setting: Decimal.new("21.0"),
teslamate | est_battery_range_km: Decimal.new("111.37"),
teslamate | fan_status: 0,
teslamate | inside_temp: Decimal.new("19.7"),
teslamate | passenger_temp_setting: Decimal.new("21.0")
teslamate | },
teslamate | errors: [
teslamate | battery_level: {"is invalid", [type: :integer, validation: :cast]}
teslamate | ],
teslamate | data: #TeslaMate.Log.Position<>,
teslamate | valid?: false,
teslamate | ...
teslamate | >}},
teslamate | [
teslamate | {TeslaMate.Vehicles.Vehicle, :"-handle_event/4-fun-3-", 4,
teslamate | [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 766]},
teslamate | {Ecto.Adapters.SQL, :"-checkout_or_transaction/4-fun-0-", 3,
teslamate | [file: ~c"lib/ecto/adapters/sql.ex", line: 1400]},
teslamate | {DBConnection, :run_transaction, 4,
teslamate | [file: ~c"lib/db_connection.ex", line: 1756]},
teslamate | {TeslaMate.Vehicles.Vehicle, :handle_event, 4,
teslamate | [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 758]},
teslamate | {:gen_statem, :loop_state_callback, 11,
teslamate | [file: ~c"gen_statem.erl", line: 1397]},
teslamate | {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]}
teslamate | ]}
teslamate | 2025-01-05 21:42:18.409 [info] GET /import

Web error:

{{:badmatch, {:error, #Ecto.Changeset< action: :insert, changes: %{ date: ~U[2024-11-23 20:29:43.000000Z], power: 1, latitude: Decimal.new("XXX"), longitude: Decimal.new("XXX"), odometer: 73381.499485, ideal_battery_range_km: Decimal.new("109.48"), outside_temp: Decimal.new("17.5"), rated_battery_range_km: Decimal.new("109.48"), usable_battery_level: 28, driver_temp_setting: Decimal.new("21.0"), est_battery_range_km: Decimal.new("111.37"), fan_status: 0, inside_temp: Decimal.new("19.7"), passenger_temp_setting: Decimal.new("21.0") }, errors: [ battery_level: {"is invalid", [type: :integer, validation: :cast]} ], data: #TeslaMate.Log.Position<>, valid?: false, ... >}}, [ {TeslaMate.Vehicles.Vehicle, :"-handle_event/4-fun-3-", 4, [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 766]}, {Ecto.Adapters.SQL, :"-checkout_or_transaction/4-fun-0-", 3, [file: ~c"lib/ecto/adapters/sql.ex", line: 1400]}, {DBConnection, :run_transaction, 4, [file: ~c"lib/db_connection.ex", line: 1756]}, {TeslaMate.Vehicles.Vehicle, :handle_event, 4, [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 758]}, {:gen_statem, :loop_state_callback, 11, [file: ~c"gen_statem.erl", line: 1397]}, {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]} ]}

CVS line:

682739,"2024-11-23 20:29:43",,1,"1227 95 2 0 0 0 62",Fly,"1 0 0 High",,,LRWYGCFS0PC551608,IEC,"17.499999608845 17073.890",0,"3.1 3.1 3.05 3.05",online,,0.0,16,0,,,0,,68.03,0,100,16,Engaged,,0,0,,28,,,,28.01,0.0,1,,,68.03,,NoPower,0,1,,1732406400,69.2,0.0,1,16,1,0.0,,0.0,19.7,XXX,250,,XXX,,,0,,0,21.0,0,21.0,17.5,0,0,,,0,0,0,0,,,,,,,0,,0,,,0,45597.149823,,0,off,,,0,,0,,,,,,0,,0,,,,,0,,,Apollo19MetallicShad,0,,,"2024.38.7 f78dd5c34b",0,,67,,63,,4456,0,2510,0,,0,0,0,248.91,,,0,,,0,1,0,0,0,74.2

Expected Behavior

Should import the rest of the file

Steps To Reproduce

create a cvs file with that line

Relevant log output

teslamate  | 2025-01-05 21:38:24.662 [error] GenStateMachine :import_Fly terminating
teslamate  | ** (MatchError) no match of right hand side value: {:error, #Ecto.Changeset<action: :insert, changes: %{date: ~U[2024-11-23 20:29:43.000000Z], power: 1, latitude: Decimal.new("XXX"), longitude: Decimal.new("XXX"), odometer: 73381.499485, ideal_battery_range_km: Decimal.new("109.48"), outside_temp: Decimal.new("17.5"), rated_battery_range_km: Decimal.new("109.48"), usable_battery_level: 28, driver_temp_setting: Decimal.new("21.0"), est_battery_range_km: Decimal.new("111.37"), fan_status: 0, inside_temp: Decimal.new("19.7"), passenger_temp_setting: Decimal.new("21.0")}, errors: [battery_level: {"is invalid", [type: :integer, validation: :cast]}], data: #TeslaMate.Log.Position<>, valid?: false, ...>}
teslamate  |     (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:766: anonymous fn/4 in TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate  |     (ecto_sql 3.12.1) lib/ecto/adapters/sql.ex:1400: anonymous fn/3 in Ecto.Adapters.SQL.checkout_or_transaction/4
teslamate  |     (db_connection 2.7.0) lib/db_connection.ex:1756: DBConnection.run_transaction/4
teslamate  |     (teslamate 1.32.1-dev) lib/teslamate/vehicles/vehicle.ex:758: TeslaMate.Vehicles.Vehicle.handle_event/4
teslamate  |     (stdlib 5.2.3.1) gen_statem.erl:1397: :gen_statem.loop_state_callback/11
teslamate  |     (stdlib 5.2.3.1) proc_lib.erl:241: :proc_lib.init_p_do_apply/3
teslamate  | 2025-01-05 21:38:24.664 [warning] Import failed: {{:badmatch,
teslamate  |   {:error,
teslamate  |    #Ecto.Changeset<
teslamate  |      action: :insert,
teslamate  |      changes: %{
teslamate  |        date: ~U[2024-11-23 20:29:43.000000Z],
teslamate  |        power: 1,
teslamate  |        latitude: Decimal.new("XXX"),
teslamate  |        longitude: Decimal.new("XXX"),
teslamate  |        odometer: 73381.499485,
teslamate  |        ideal_battery_range_km: Decimal.new("109.48"),
teslamate  |        outside_temp: Decimal.new("17.5"),
teslamate  |        rated_battery_range_km: Decimal.new("109.48"),
teslamate  |        usable_battery_level: 28,
teslamate  |        driver_temp_setting: Decimal.new("21.0"),
teslamate  |        est_battery_range_km: Decimal.new("111.37"),
teslamate  |        fan_status: 0,
teslamate  |        inside_temp: Decimal.new("19.7"),
teslamate  |        passenger_temp_setting: Decimal.new("21.0")
teslamate  |      },
teslamate  |      errors: [
teslamate  |        battery_level: {"is invalid", [type: :integer, validation: :cast]}
teslamate  |      ],
teslamate  |      data: #TeslaMate.Log.Position<>,
teslamate  |      valid?: false,
teslamate  |      ...
teslamate  |    >}},
teslamate  |  [
teslamate  |    {TeslaMate.Vehicles.Vehicle, :"-handle_event/4-fun-3-", 4,
teslamate  |     [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 766]},
teslamate  |    {Ecto.Adapters.SQL, :"-checkout_or_transaction/4-fun-0-", 3,
teslamate  |     [file: ~c"lib/ecto/adapters/sql.ex", line: 1400]},
teslamate  |    {DBConnection, :run_transaction, 4,
teslamate  |     [file: ~c"lib/db_connection.ex", line: 1756]},
teslamate  |    {TeslaMate.Vehicles.Vehicle, :handle_event, 4,
teslamate  |     [file: ~c"lib/teslamate/vehicles/vehicle.ex", line: 758]},
teslamate  |    {:gen_statem, :loop_state_callback, 11,
teslamate  |     [file: ~c"gen_statem.erl", line: 1397]},
teslamate  |    {:proc_lib, :init_p_do_apply, 3, [file: ~c"proc_lib.erl", line: 241]}
teslamate  |  ]}
teslamate  | 2025-01-05 21:42:18.409 [info] GET /import
teslamate  | 2025-01-05 21:42:18.414 [info] Sent 200 in 4ms

Screenshots

No response

Additional data

No response

Type of installation

Docker

Version

1.32 and edge

[Xapier]

FYI: I have similar issues when migrating from TeslaFi as TeslaFi battery_level column is decimal instead of integer since 22 Nov 2024 - every monthly export file one would like to import is throwing errors.

[jaggernod]

The same happened to me.
Apparently, at that time, the battery level started to be reported as a double (number with a fraction) instead of an int (natural number).

What you can do before the import is to open the CSV file, find the battery_level column and convert all the values in that column to a natural number (all spreadsheet editors should have this option).
After saving the file it was imported properly for me.

[Xapier]

The same happened to me. Apparently, at that time, the battery level started to be reported as a double (number with a fraction) instead of an int (natural number).

What you can do before the import is to open the CSV file, find the battery_level column and convert all the values in that column to a natural number (all spreadsheet editors should have this option). After saving the file it was imported properly for me.

I tried this already but failed. Can you tell me which editor worked for you? Excel import destroys several column formats so I was struggling to fix the battery_level numbers (to integer) and import the data to teslamate after transforming it.

[jeepyboi]

Wanted to add to this. All of my TeslaFi imports work until September of 2024. Anything newer than that fails. I tried to reformat the battery_level column, but either I was not successful or it didn't solve the problem.

[TheLinuxGuy]

+1 just ran into this bug on my import of last 5 years of data. As another person mentioned, around November/Dec 2024 they changed something.

[TheLinuxGuy]

Wanted to add to this. All of my TeslaFi imports work until September of 2024. Anything newer than that fails. I tried to reformat the battery_level column, but either I was not successful or it didn't solve the problem.

My updated script to generate backups from TeslaFi so they can be imported into TeslaMate will now deal with this condition and fix it automatically.

• Did this as part of my original TeslaFi import fails with "Unsupported delimiter" when using non standard delimiter and displays correct error message #4569 then I ran into this issue and said what the heck why not fix this too so .csv contains the expected data.

TL;DR Use https://gist.github.com/TheLinuxGuy/e8c85e59226014087159c5d36c0a1272 to dump your TeslaFi data into proper .csv files that will be ingested by TeslaMate without issues (as of March 3, 2025).