deal with ST gaps

[cparcerisas]

Hello all,

Sometimes ST misses some data (see extract from their manual) @bram-cuyx

Then they write in the XML where those gaps were, and also the total time of "gaps" in a file.
ST gives the option to fill these gaps with 0s when downloading the data, but it is currently an "experimental feature", and not everyone uses it.
Would it be interesting for pbp to implement a check on those gaps to make sure the timekeeping is correct?

Just a question! :)

Thanks,

Clea

[ryjombari]

Good question @cparcerisas.

Here are elements of my experience with this:

• I have always used zero-fill while downloading our SoundTrap data, and it has been consistently reliable – for a total of about 20 deployments. For every 4-5 month deployment, I check temporal coverage in the zero-filled data files, and it has always shown no irregularities in timekeeping, i.e. no recording gaps apparent. I do this outside of PBP, just so that I know the deployment looks good to process with PBP. I think everyone should check the quality of temporal coverage in their deployments, but I don't know if PBP is the right place to do that because... see next point.
• I think the answer to the question of whether or not gaps are an issue depends on the severity of the problem and the nature of the data product and subsequent analyses. For the standard 1-minute hybrid millidecade spectra and subsequent analyses, I don't think it would be a big concern. For example, for commonly used 48 kHz monitoring along the US west coast, file lengths are typically ~4 hours, and timekeeping in PBP gets a regular correction with the start of every file. Thus, there would be no cumulative drift in time reference caused by recording dropouts, only drift introduced by inaccuracy in the hydrophone's clock.

I don't know if others are seeing major problems with dropouts, but if dropouts reached the level of severity that data / product quality and timekeeping were strongly affected, hydrophone replacement would be the solution I think.

[cparcerisas]

Hi @ryjombari, thanks for the answer!

We do face quite some problems with ST gaps. In long deployments we usually want to record 5 minutes every 5 minutes, to have a more manageable size of files (4 hours might be hard to open to manually check for sound events, for example).
We do face some deployments where from the 5 minutes we end un with only one minute of data, but if data are not zero-filled then only by looking at the XML can we know when each gap happened and to which minute each sample falls into.
File name keeps consistent for us as well regarding timekeeping.

The solution then seems to make it compulsory for people to use the zero-filled data option when downloading ST data. We will definitely do this from now on. I will check if this is possible to do for already downloaded data.

[ryjombari]

@cparcerisas I think that if you have the compressed (.sud) files, you should be able to re-extract with zero fill of dropouts.