nvs_tool.py extracts strings wrongly in json format (IDFGH-14504)

[taliesin]

Answers checklist.

• I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
• I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
• I have searched the issue tracker for a similar issue and not found a similar issue.

General issue report

I created a CSV file containing some factory data:

key,type,encoding,value
factory_data,namespace,,
hw_version,data,string,"0.15.2"
production_date,data,u64,1737662604
laserdiode_beta,data,u16,400

and used
python3 /components/nvs_flash/nvs_partition_generator/nvs_partition_gen.py generate factory_data_0.15.2.csv factory_data.bin 0x4000
to create a NVS binary image

Then used
python3 /components/nvs_flash/nvs_partition_tool/nvs_tool.py factory_data.bin -f text

which says (emtpy stuff not shown):

Page no. 0, Status: Full, Version: 2, CRC32: b9ba2d84, Page address: 0x0
 Entry state bitmap: aa fe ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
 000. Written, Namespace Index: 000, Type: uint8_t   , Span: 001, Chunk Index: 255, CRC32: a5762796 | factory_data : 1
 001. Written, Namespace Index: 001, Type: string    , Span: 002, Chunk Index: 255, CRC32: bee86f42 | hw_version : Size=7, CRC32=ee2d98d5
      0x000  30 2e 31 35 2e 32 00 ff ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff  0.15.2.ÿÿÿÿÿÿÿÿÿ ÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿÿ
 003. Written, Namespace Index: 001, Type: uint64_t  , Span: 001, Chunk Index: 255, CRC32: cdf2ce5a | production_date : 1737662604
 004. Written, Namespace Index: 001, Type: uint16_t  , Span: 001, Chunk Index: 255, CRC32:  c5e5c90 | laserdiode_beta : 400
 005. Empty

Then I tried using the JSON output, which I piped to jq to filter stuff:

python3 nvs_tool.py -f json factory_data.bin | jq '.pages[].entries[]|select(.key == "hw_version")'

which returns:

{
  "raw": "ASEC/0Jv6L5od192ZXJzaW9uAAAAAAAABwD//9WYLe4=",
  "state": "Written",
  "is_empty": false,
  "index": 1,
  "metadata": {
    "namespace": 1,
    "type": "string",
    "span": 2,
    "chunk_index": 255,
    "crc": {
      "original": 3202903874,
      "computed": 3202903874,
      "data_original": 3995965653,
      "data_computed": 3995965653
    }
  },
  "children": [
    {
      "raw": "MC4xNS4yAP////////////////////////////////8=",
      "state": "Written",
      "is_empty": false,
      "index": 2,
      "metadata": {
        "namespace": 48,
        "type": "0x2e",
        "span": 49,
        "chunk_index": 53,
        "crc": {
          "original": 4278202926,
          "computed": 463762067,
          "data_original": 4294967295,
          "data_computed": 0
        }
      },
      "children": [],
      "key": null,
      "data": null
    }
  ],
  "key": "hw_version",
  "data": {
    "value": [
      7,
      3995965653
    ],
    "size": 7,
    "crc": 3995965653
  }
}

Obviously the data.value entry is wrong. I checked the nvs_parser.py code to find:

# Deal with variable length entries
if nvs_const.item_type[i_type] in ['string', 'blob_data', 'blob']:
    size = int.from_bytes(data[:2], byteorder='little')
    crc = int.from_bytes(data[4:8], byteorder='little')
    return {'value': [size, crc], 'size': size, 'crc': crc}

... which is exactly what I get, but that is obviously wrong!

[hrushikesh430]

Hi @taliesin, I tried your issue and yes instead of actual value of string, it outputs the CRC of string(as mentioned in nvs_parser.py). In the nvs_parser string, blob_data and blob are spanned over multiple nvs pages, that's why CRC is outputted than it's actual value. But, when you flash your .csv file on device and try to retrieve the string data using nvs apis, the actual value will be retrieved.

[taliesin]

You're perfectly right, strings work in NVS. The intention of nvs_tool.py from my point of view is to retrieve values from an existing partition image. For each other (fixed size) type JSON output contains the actual value, there is no reason to output the size and CRC for a string value, which has a perfect representation in JSON.

My use case was to fetch binary partition data from the device (factory partition), replace some of the values and store it back to the device. This is useful for data that changes like calibration data (re-calibration), repair information etc. while other information hardware version, production date ... remain the same. The partition is read-only for the firmware, the partition is marked as such.

[hrushikesh430]

Hi @taliesin, Is your requirement requires only json format or any format. For text format you can do minimal dump for generating key value pairs.

command -

python /nvs_partition_tool/nvs_tool.py -d minimal -f text factory.bin

[taliesin]

I would have preferred JSON, but you're right the output of -d minimal -f text can be parsed quite easily. In my case using JSON would have generated all information to re-create the modified CSV file, with the text output I'll have to parse both the original CSV file (for the types) and the text output to generate the new CSV file.

I don't want to push it, but I still think the behaviour of -f json for strings is wrong.

[hrushikesh430]

Hi, we are about to fix the output of json format, which will print the required and proper information. Once it is done I will provide the pointer for the code.

[taliesin]

Great news, thanks for the effort.