Make our internal functions to read imro tables consistent with Spikeglx C++ file

[h-mayorquin]

@cwindolf pointed to us the exact mapping from probe type to geometry:

https://github.com/billkarsh/SpikeGLX/blob/master/Src-imro/IMROTbl.cpp#L142

Right now we are doing this with a local ad-hoc solution:

probeinterface/src/probeinterface/io.py

Lines 662 to 894 in 2dfc737

	polygon_description = {
	"default": [
	(0, 10000),
	(0, 0),
	(35, -175),
	(70, 0),
	(70, 10000),
	],
	"nhp90": [
	(0, 10000),
	(0, 0),
	(45, -342),
	(90, 0),
	(90, 10000),
	],
	"nhp125": [
	(0, 10000),
	(0, 0),
	(62.5, -342),
	(125, 0),
	(125, 10000),
	],
	}
	# A map from probe type to geometry_parameters
	npx_probe = {
	# Neuropixels 1.0
	# This probably should be None or something else because NOT ONLY the neuropixels 1.0 have that imDatPrb_type
	0: {
	"probe_name": "Neuropixels 1.0",
	"x_pitch": 32,
	"y_pitch": 20,
	"contact_width": 12,
	"stagger": 16,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["default"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	"ap_hp_filters",
	),
	},
	# Neuropixels 2.0 - Single Shank
	21: {
	"probe_name": "Neuropixels 2.0 - Single Shank",
	"x_pitch": 32,
	"y_pitch": 15,
	"contact_width": 12,
	"stagger": 0.0,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["default"],
	"fields_in_imro_table": ("channel_ids", "banks", "references", "elec_ids"),
	},
	# Neuropixels 2.0 - Four Shank
	24: {
	"probe_name": "Neuropixels 2.0 - Four Shank",
	"x_pitch": 32,
	"y_pitch": 15,
	"contact_width": 12,
	"stagger": 0.0,
	"shank_pitch": 250,
	"shank_number": 4,
	"ncol": 2,
	"polygon": polygon_description["default"],
	"fields_in_imro_table": (
	"channel_ids",
	"shank_id",
	"banks",
	"references",
	"elec_ids",
	),
	},
	# Experimental probes previous to 1.0
	"Phase3a": {
	"probe_name": "Phase3a",
	"x_pitch": 32,
	"y_pitch": 20,
	"contact_width": 12,
	"stagger": 16.0,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["default"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	),
	},
	# Neuropixels 1.0-NHP Short (10mm)
	1015: {
	"probe_name": "Neuropixels 1.0-NHP - short",
	"x_pitch": 32,
	"y_pitch": 20,
	"contact_width": 12,
	"stagger": 0,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["default"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	"ap_hp_filters",
	),
	},
	# Neuropixels 1.0-NHP Medium (25mm)
	1022: {
	"probe_name": "Neuropixels 1.0-NHP - medium",
	"x_pitch": 103,
	"y_pitch": 20,
	"contact_width": 12,
	"stagger": 0.0,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["nhp125"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	"ap_hp_filters",
	),
	},
	# Neuropixels 1.0-NHP 45mm SOI90 - NHP long 90um wide, staggered contacts
	1030: {
	"probe_name": "Neuropixels 1.0-NHP - long SOI90 staggered",
	"x_pitch": 56,
	"y_pitch": 20,
	"stagger": 12,
	"contact_width": 12,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["nhp90"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	"ap_hp_filters",
	),
	},
	# Neuropixels 1.0-NHP 45mm SOI125 - NHP long 125um wide, staggered contacts
	1031: {
	"probe_name": "Neuropixels 1.0-NHP - long SOI125 staggered",
	"x_pitch": 91,
	"y_pitch": 20,
	"contact_width": 12,
	"stagger": 12.0,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["nhp125"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	"ap_hp_filters",
	),
	},
	# 1.0-NHP 45mm SOI115 / 125 linear - NHP long 125um wide, linear contacts
	1032: {
	"probe_name": "Neuropixels 1.0-NHP - long SOI125 linear",
	"x_pitch": 103,
	"y_pitch": 20,
	"contact_width": 12,
	"stagger": 0.0,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 2,
	"polygon": polygon_description["nhp125"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	"ap_hp_filters",
	),
	},
	# Ultra probe
	1100: {
	"probe_name": "Ultra probe",
	"x_pitch": 6,
	"y_pitch": 6,
	"contact_width": 5,
	"stagger": 0.0,
	"shank_pitch": 0,
	"shank_number": 1,
	"ncol": 8,
	"polygon": polygon_description["default"],
	"fields_in_imro_table": (
	"channel_ids",
	"banks",
	"references",
	"ap_gains",
	"lf_gains",
	"ap_hp_filters",
	),
	},
	}
	# Map imDatPrb_pn (probe number) to imDatPrb_type (probe type) when the latter is missing
	probe_number_to_probe_type = {
	"PRB_1_4_0480_1": 0,
	"PRB_1_4_0480_1_C": 0,
	"NP1010": 0,
	"NP1015": 1015,
	"NP1022": 1022,
	"NP1030": 1030,
	"NP1031": 1031,
	"NP1032": 1032,
	None: 0,
	}

We should follow the convention in their code so we can track it more closely and follow changes. I am opening this issue here to keep track for ourselves and I will add more detaill and an actiionable lists of todo later.

[cwindolf]

Relevant for this issue, another SpikeGLX universal reading project from @jenniferColonell which looks great and handles lots of cases! https://github.com/jenniferColonell/SGLXMetaToCoords

[samuelgarcia]

The function done by @jenniferColonnell use snsGeomMap or snsShankMap fields from the meta file.
We chosse the harder way : the imro table.

[h-mayorquin]

You have probably written about this before but would you mind repeating why we have chosen that way, @samuelgarcia ?

It is good for provenance and future decision making.

[jenniferColonell]

@billkarsh and I actually designed the snsGeomMap partly with SpikeInterface in mind, so that all the information to interpret a SpikeGLX file would available in the metadata, without needing to maintain separate tables of probe geometries and other properties.
Just to clarify why I'm using the snsShankMap and snsGeomMap:
(1) They both include only entries for the saved channels in the file. If you choose to use the imro table (which my older version in fact did) you must also read either the saved channel string. Files with just some of the channels will become more common as people tackle analysis of 4 probe data.
(2) They both include the enabled flag, which indicates which sites were disabled during recording. Usually, this is just the reference channels. These are not indicated in the imro table.
(3) The snsGeomMap, in particular, has all the geometry information directly available. If there are new probes (there will be new probes) code that reads the snsGeomMap will be compatible with NO changes.

Indeed, to support older data, you need to handle the shank map as well, and keep a table of geometries that correspond to current probes. But you won't need to expand that table as new probes come along.

[h-mayorquin]

@jenniferColonell @cwindolf

This is very useful information. Thanks a lot.

[billkarsh]

We encourage you to parse ~snsGeomMap as a replacement for parsing the imro table. It already combines information about the saved channel subset and the imro table to report where each readout channel is on the probe surface in microns. You don't need to keep up with new probes or carry tables or databases. Everything you need is automatically generated in the ~snsGeomMap for each dataset.

[alejoe91]

Thanks for the info! We definitely will :)

[samuelgarcia]

Merci beaucoup Jennifer and Bill!

I propose to implement all cases in probeinterface, lets be exhaustive.

Lets use first snsGeomMap when available on recent version.
For older formats lets keep imro and optionaly we can implement parsing snsShankMap (this was the case a while ago)

Reading and writting imro is also good to have because we can generate and debug the imro table directly in probeinterface.