mountainsort5 on Neuropixels speed

[hokysung]

Hi,

I'm a new user looking to run MountainSort5 on a Neuropixels dataset.

I was going off of the template code of the Neuropixels quickstart guide following other preprocessing steps (e.g. common average, removing microstimulation artifacts, etc).

I have gotten the setup to a point where I am seeing these messages on my log output:

Detected 19276057 spikes
*** MS5 Elapsed time for detect_spikes: 1735.160 seconds ***
Removing duplicate times
*** MS5 Elapsed time for remove_duplicate_times: 0.114 seconds ***
Extracting 5705070 snippets
*** MS5 Elapsed time for extract_snippets: 156.321 seconds ***
Computing PCA features with npca=768

The problem is that the log has been stuck at this point for more than 5 hours now. Is this expected? What is usually the expected runtime for running ms5 on a 384 channel Neuropixels recording (approx 3 hours)? Please let me know if I might be doing anything wrong, and if so, how to correct it.

Thank you for your guidance!

Best,
Hokyung

[magland]

Hi @hokysung Did you do whitening first? That's important for the proper detection. I'd recommend you raise the detect threshold so there are fewer spikes detected, and you can see how long that takes, and then lower.

Unfortunately I don't have a good estimate for np data at this point.

[hokysung]

Hi,

Thank you so much for the prompt reply!

No, I had not been whitening first. I added a whitening step.

Now, the preprocessing pipeline first applies a highpass filter, phase shift, common average referencing, then whitening. Afterwards, I zero out the phases where electrical stimulation artifacts are present, and then run a motion correction algorithm. This is the recording object I give to MountainSort currently. Does this make sense? I'm unsure if the ordering is appropriate.

I also took your suggestion and ran with detect_threshold 9.0, up from 5.25 previously. However, the overall speed still seems quite slow. I am 12 hours into running MountainSort, and I am at this stage:

Scheme 2 detected 4281086 spikes in chunk 181 of 375
*** MS5 Elapsed time for SCHEME2 detecting spikes: 79.873 seconds ***
Extracting and classifying snippets
*** MS5 Elapsed time for SCHEME2 extracting and classifying snippets: 42.446 seconds ***
Updating events
Removing duplicates
*** MS5 Elapsed time for SCHEME2 updating events: 2.100 seconds ***
Time chunk 182 of 375
Loading traces
*** MS5 Elapsed time for SCHEME2 loading traces: 0.014 seconds ***
Detecting spikes

Does this look reasonable to you? Anything else I might be missing?

Thank you again for your guidance,
Hokyung

[magland]

Whitening should be the very last step before running mountainsort5. The channels need to be normalized to have approximate variance 1.

[hokysung]

I see, running again with whitening at the end has made it extremely fast. Hopefully it works out -- thank you!

[jbmelander]

If trouble persists, I can take a look at your script. Ive had good luck with ms5 on np data in the past. On Aug 20, 2025 9:23 AM, Ho Kyung Sung ***@***.***> wrote: [https://avatars.githubusercontent.com/u/43563226?s=20&v=4]hokysung left a comment (flatironinstitute/mountainsort5#60)<#60 (comment)> I see, running again with whitening at the end has made it extremely fast. Hopefully it works out -- thank you! — Reply to this email directly, view it on GitHub<#60 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHXLMBQ26QMPUMYF3QZYDHT3OSOIZAVCNFSM6AAAAACEIR26HKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTEMBXGA4DIMJVHE>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[hokysung]

Hi,

I think I was able to get it to work.

Two things:

1. The quality of the sorting results seem quite bad. The run gave 159 clusters, but when I ran a stability checking algorithm less than 10 units seem to be stable for more than 10% of the session's trials. I wonder what could be done to improve the quality on this Neuropixels recording?

# Custom MountainSort5 parameters
ms5_custom_params:
  max_num_snippets_per_training_batch: 500
  snippet_mask_radius: 50
  phase1_npca_per_channel: 2
  phase1_npca_per_subdivision: 8
  classifier_npca: 8
  detect_channel_radius: 50
  phase1_detect_channel_radius: 50
  training_recording_sampling_mode: "uniform"
  training_duration_sec: 200
  phase1_detect_threshold: 5.5
  detect_threshold: 5.25
  snippet_T1: 15
  snippet_T2: 40
  detect_sign: 0
  phase1_detect_time_radius_msec: 0.5
  detect_time_radius_msec: 0.5
  classification_chunk_sec: 60
  block_duration_sec: 300

FYI, these are the parameters I used, with scheme 2 (ignore block duration sec).

2. I was looking to create a sorting analyzer again for already sorted results of MountainSort5 using your code (not through Spikeinterface run_sorter).
   To do so, I wanted to first load the sorter object, but read_sorter_folder is giving me an error:

Error loading existing sorting results: This folder /orcd/data/mjaz/001/om4/hokyung/monkey_data/Nielsen/20250821/imec0_sorting_ms5_default/sorter_MS5_imec0_ap does not have spikeinterface_log.json

I went in and checked and all sorter output folders of mountainsort5 do not seem to have spikeinterface_log.json in them. Would there still be a way to recover the sorter object to create a sorting analyzer on it?

Thank you for your guidance,
Hokyung

[magland]

I went in and checked and all sorter output folders of mountainsort5 do not seem to have spikeinterface_log.json in them. Would there still be a way to recover the sorter object to create a sorting analyzer on it?

I think you will need to convert the sorting object to something that will create the appropriate json. I haven't done this before... I think you'll need to consult the SI documentation.