Write gapped vdif problem

[luojing1211]

I tried to write the gapped data out, however, we meet this error.

import os
from baseband import vdif
import numpy as np
import astropy.units as u
from baseband_tasks.shaping import GetSlice


data_dir = '/drives/CHF/0/20211210T080933Z_CHIME_vdif'
files = os.listdir(data_dir)
files.sort()
data_file = [os.path.join(data_dir, ff) for ff in files[0:-3]]
fh = vdif.open(data_file, 'rs', sample_rate=400 * u.MHz / 1024)
duration = 2048
gap = 1000
out_dir = '/drives/scratch/sandbox'
fw = vdif.open(os.path.join(out_dir, 'gated.vdif'), 'ws',
               sample_rate=fh.sample_rate,
               samples_per_frame=duration,
               time=fh.start_time)

total_time = fh.shape[0] * fh.sample_rate
# set start times
starts = np.arange(0, fh.shape[0], duration + gap)
#start_times = fh.start_time + starts[0:5]
for st in starts:
    part = GetSlice(fh, slice(st, st + duration))
    print(part.shape)
    fw.write(part.read())

fw.close()

ERROR

(2048, 32, 8)
Traceback (most recent call last):
  File "test_data_sclice_write.py", line 30, in <module>
    fw.write(part.read())
  File "/home/jingluo/.local/lib/python3.8/site-packages/baseband/base/base.py", line 1255, in write
    assert data.shape[1:] == self.sample_shape, (
AssertionError: 'data' should have trailing shape SampleShape()

[mhvk]

For the output file, you also need to pass in a sample shape - by default, it assumes just 1 channel.

[luojing1211]

The following script works. However, I will have to checkout the data value.

import astropy.units as u
from baseband_tasks.shaping import GetSlice, Reshape

data_dir = '/drives/CHF/0/20211210T080933Z_CHIME_vdif'
files = os.listdir(data_dir)
files.sort()
data_file = [os.path.join(data_dir, ff) for ff in files[0:-3]]
fh = vdif.open(data_file, 'rs', sample_rate=400 * u.MHz / 1024)
rh = Reshape(fh, (2, 128))
duration = 2048
gap = 1000
out_dir = '/drives/scratch/sandbox/test_vdif_write'
fw = vdif.open(os.path.join(out_dir, 'gated.vdif'), 'ws',
               sample_rate=fh.sample_rate,
               samples_per_frame=duration,
               complex_data=True,
               time=fh.start_time, nthread=2, nchan=128)

total_time = fh.shape[0] * fh.sample_rate
# set start times
starts = np.arange(0, fh.shape[0], duration + gap)
print(fw.sample_shape)
#start_times = fh.start_time + starts[0:5]
for st in starts[0:5]:
    part = GetSlice(rh, slice(st, st + duration))
    dd = part.read()
    print(dd)
    fw.write(dd)

fw.close()

[mhvk]

That looks good!

Obviously, here we have explicitly lost the time information (it now will be really wrong). I'd suggest for now to just write one output file per gate; the reader should be able to work around the gaps if one combines the gated files afterwards.

[mhvk]

I think this was answered, though there still is a general question of whether it should be possible/easier to write gated data to a single file. For that, see #494