[Feature]: load channel info for Phy data

[vigji]

What would you like to see added to NeuroConv?

Not sure that this is missing - in which case apologies- but I could not get to it after quite some search in the docs and in the code. Also, I'm pretty new to this whole world of .nwb and ECEphys...

I am trying to load some sorted spike data from a Phy folder. This is my code now:

from pathlib import Path
from neuroconv import ConverterPipe
from neuroconv.datainterfaces importPhySortingInterface
from neuroconv.utils.dict import load_dict_from_file, dict_deep_update

subj_path = Path(".../M9/")
folder_path = subj_path / "SpikeData" 
interface_phy = PhySortingInterface(folder_path=folder_path, verbose=False)
converter = ConverterPipe(data_interfaces=[interface_phy], verbose=True)

metadata = converter.get_metadata()
metadata_path = subj_path / "subject_metadata.yml"
metadata_from_yaml = load_dict_from_file(file_path=metadata_path)

metadata = dict_deep_update(metadata, metadata_from_yaml)
conv = converter.run_conversion(nwbfile_path=subj_path / "imported-nwb.nwb", metadata=metadata)

I would like to just keep the sorted data in the file, and not the original recordings as the size is huuuuge. In the Phy folder I have a bunch of channel metadata that could potentially be read, but as far as I can get there is no generate_recording_with_channel_metadata() method in the PhySortingInterface class, so calling add_channel_metadata_to_nwb() is ineffective.

Am I getting this correctly? is this option currently missing for Phy files?

Is your feature request related to a problem?

Loading channel metadata from Phy-sorted data

Do you have any interest in helping implement the feature?

Yes, but I would need guidance.

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you ensured this bug was not already reported?

[CodyCBakerPhD]

In the Phy folder I have a bunch of channel metadata that could potentially be read

Interesting, I don't think we've seen a case of that metadata before...

Would you be willing to share this instance of Phy data with us?

Also, what acquisition system did you use for this? It may ultimately be simpler to add the recording interface to get the channel information, but set the conversion option write_electrical_series=False to prevent writing the bulk data

@alejoe91 Are you familiar with what they're talking about in Phy? Does SI interface with that at all? Or would you recommend they simply generate a probe/dummy recording separately from the sorting extractor to attach that information?

[vigji]

Thank you very much for the super-fast response!

In the Phy folder I have a bunch of channel metadata that could potentially be read

Interesting, I don't think we've seen a case of that metadata before...

Would you be willing to share this instance of Phy data with us?

Sorry, "A bunch of" was a bit rushed as a message, what I see there is just the channel position, which is what I would like to have. I guess that you would want more complete metadata to add the feature.

My goal atm is to just keep track of the position on the probe of the sorted clusters.

I was also wondering if I would then need to manually figure out the positional info for each unit, or there is already code to assign each unit to a channel based on the cluster's footprint template.npy (I guess it is a common problem?). I can't find anything related to position/channel in the nwb.units, although it is info that could be generated from the phy folder automatically

Also, what acquisition system did you use for this? It may ultimately be simpler to add the recording interface to get the channel information, but set the conversion option write_electrical_series=False to prevent writing the bulk data

I missed this option though, this looks the way to go in general to log more info, thanks!

Thank you again for your help and the super useful package.

[CodyCBakerPhD]

I was also wondering if I would then need to manually figure out the positional info for each unit, or there is already code to assign each unit to a channel based on the cluster's footprint template.npy (I guess it is a common problem?)

I have reached out to the SI team about this - I don't personally know off the top of my head

I can't find anything related to position/channel in the nwb.units

It's indirect, but the NWB method of handling this would be to utilize the electrodes field of the UntisTable: https://nwb-schema.readthedocs.io/en/latest/format.html#units

This would link a unit to an electrode, and that electrode could have exact position/location information on its own table

I admit however, we do not currently have this linkage formed automatically in NeuroConv as we haven't seen many cases of it - what you could do for now is manually add the electrodes property on the sorting extractor object with

interface_phy = PhySortingInterface(folder_path=folder_path, verbose=False)

interface_phy.sorting_extractor.add_property(values=[all_your_channel_positions, ...], unit_ids=interface_phy.get_unit_ids())

and that will then automatically add the electrode indices (as integers, not links) which is something at least. My syntax might be a bit off but I'll let @alejoe91 correct me on that

Otherwise, would you be willing to share your example of Phy data that has channel information so that I can build and test a seamless feature for establishing the linkage?

[alejoe91]

Hi @vigji

NeuroConv uses spikeinterface to read from various formats. In SpikeInterface, there are 3 main objects:

• Recording: deals with raw data
• Sorting: deals with sorted data (like phy)
• WaveformExtractor: combines a Recording+Sorting and can extract waveforms/templates and much more.

Now, the way that neuroconv uses spikeinterface is as a reader. The Phy reader will return a sorting object, which only knows about its spikes and clusters. The an estimate of the unit position, since it's inferred from the template, belongs to the WaveformExtractor realm, which is not used here.

@CodyCBakerPhD if the goal here is just to add a unit_location or unit_depth column to the units table, I think that the easies would be to have some simple logic in the PhySortingInterface.__init__() directly. It would look something like this:

folder_path = self.sorting_extractor._kwargs["folder_path"]

# load channel positions
channel_positions = np.load(folder_path / "channel_positions.npy")
# load templates
templates = np.load(folder_path / "templates.npy")
if (folder_path / "template_ind.npy").is_dir(): # in this case templates are sparse (they usually are)
    template_ind = np.load(folder_path / "template_ind.npy")
else:
    template_ind = None

# get unit_depth for each template
unit_depths = np.zeros(len(templates))
for i, template in enumerate(templates):
    if template_ind is None:
        channel_positions_slice = channel_positions
    else:
        channel_positions_slice = channel_positions[template_ind[i]]
    depth = channel_positions_slice[np.argmax(np.ptp(template, axis=1))]
    unit_depths[i] = depth
self.sorting_extractor.set_property("unit_depth", unit_depths)

Otherwise, as Cody suggests, you can "maually" add the depth and have this logic after you instantiate the interface object.

Third option is to make a .tsv file in the phy folder with cluster_id, unit_depth columns named unit_dephts.tsv. This should be automatically loaded as a property (see here).

[alejoe91]

@vigji if I understand correctly you don't have explicit channel information, but you're looking for a way to computed it from channel positions and templates, right?

[vigji]

Thank you very much guys! all your inputs I resorted to reimplementing all my pipeline in spikeinterface, which was immensely needed :) don't know how I could have missed starting using it earlier...

Don't think I will have the need to follow up on this, closing it for now!