Crash on big data ingestion

[SamuelGauthier]

Hi guys, I was playing around with an Influxdb 2 instance running in a podman machine this weekend and got some issues when ingesting lots of data (>14k rows). The influxdb server crashes during the ingestion and I don't know what I'm doing wrong. Can you help me out please?

Steps to reproduce:
List the minimal actions needed to reproduce the behaviour.

1. setup influxdb with podman

version: '3'

services:
  influxdb:
    image: influxdb:2
    container_name: influxdb
    ports:
      - "8086:8086"
    volumes:
      - ./data/influxdb/data:/var/lib/influxdb2
      - ./config/influxdb/:/etc/influxdb2
    restart: unless-stopped

Config:

storage-write-timeout: 3600s
log-level: debug

2. setup influx

influx setup \
  --username admin \
  --password admin \
  --org my_org \
  --bucket raw_stock_data \
  --force

3. ingest a lot of data in the form

stocks,ticker=AA open=6.5321550369262695,high=6.556184768676758,low=6.5321550369262695,close=6.5321550369262695,volume=55900 -252374400000000000
stocks,ticker=AA open=6.5321550369262695,high=6.632279872894287,low=6.524145126342773,close=6.632279872894287,volume=74500 -252288000000000000
stocks,ticker=AA open=6.632279872894287,high=6.66431999206543,low=6.632279872894287,close=6.632279872894287,volume=80500 -252201600000000000
...

with the command:

influx write \
--bucket raw_stock_data \
--format lp \
--file AA_line_protocol.txt \
--rate-limit "500kB/s"

Expected behaviour:
I expect the ingestion of my 14k entries to succeed.

Actual behaviour:
Ingestion crashes at some point, with error:

Error: failed to write data: Post "http://localhost:8086/api/v2/write?bucket=raw_stock_data&org=my_org&precision=ns": EOF

Environment info:

• MacOS 15.3.1
• MacBook Pro 2018, 2.7 GHz Quad-Core Intel Core i7, 16GB Ram

Config:

See above

AA_line_protocol.txt

Logs:

[more logs in the file]

[influxdb] | ts=2025-03-02T15:03:01.283150Z lvl=info msg="Reindexing TSM data" log_id=0v1qpExl000 service=storage-engine engine=tsm1 db_shard_id=1851
[influxdb] | ts=2025-03-02T15:03:01.286823Z lvl=info msg="Reindexing WAL data" log_id=0v1qpExl000 service=storage-engine engine=tsm1 db_shard_id=1851
[influxdb] | ts=2025-03-02T15:03:01.454380Z lvl=info msg="saving field index changes (start)" log_id=0v1qpExl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
[influxdb] | ts=2025-03-02T15:03:01.454429Z lvl=info msg="saving field index changes (start)" log_id=0v1qpExl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
[influxdb] | ts=2025-03-02T15:03:01.467619Z lvl=info msg="saving field index changes (end)" log_id=0v1qpExl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=13.964ms
[influxdb] | ts=2025-03-02T15:03:01.467397Z lvl=info msg="saving field index changes (end)" log_id=0v1qpExl000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=15.701ms

logs.txt

[philjb]

I don't see failed to write data in your log.txt file. What is the error code returned when influxd server terminated?

failed to write data is an error that influx write returns.

[SamuelGauthier]

Thanks for your answer!
Yes, indeed the error comes from the influx write command. However in the logs of the influxdb instance itself, I see nothing, it just stops at:

tail -5 logs.txt
[influxdb] | ts=2025-03-03T21:05:07.447337Z lvl=info msg="Reindexing WAL data" log_id=0v3T08XG000 service=storage-engine engine=tsm1 db_shard_id=1872
[influxdb] | ts=2025-03-03T21:05:07.525519Z lvl=info msg="saving field index changes (start)" log_id=0v3T08XG000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
[influxdb] | ts=2025-03-03T21:05:07.524975Z lvl=info msg="saving field index changes (start)" log_id=0v3T08XG000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=start
[influxdb] | ts=2025-03-03T21:05:07.556098Z lvl=info msg="saving field index changes (end)" log_id=0v3T08XG000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=31.027ms
[influxdb] | ts=2025-03-03T21:05:08.234331Z lvl=info msg="saving field index changes (end)" log_id=0v3T08XG000 service=storage-engine engine=tsm1 op_name=MeasurementFieldSet op_event=end op_elapsed=713.025ms

Here are the complete logs, debug level:

logs.txt

[davidby-influx]

Since there are no errors or crashes in the logs at debug level, it seems likely that something external to influxd is killing the process. Are you monitoring things like memory use? Do you have an OOM-killer running?

The amount of data you are ingesting is not that large; on a sufficiently provisioned system it should be quite feasible.