VDIF frameset header consistency checker

[cczhu]

Currently only the first header of each VDIF frameset is used by stream readers, though the others are read into memory. It might be useful to check that all headers of the frameset are consistent (ie. only differ by thread_id) and throw a warning if it is not the case.

[mhvk]

Concretely, I guess this might be implemented as a new verify='strict' in VDIFFrameSet (given that we have data that fails the test, I'd prefer not to make it the default...).

A bit more abstract: the frame set should not have to know which parts of an header are meant to be invariable for a given frame_nr, but the different EDV headers should. So, this may need a general header method. In some places we already have a same_stream method; perhaps that could be extended with a VDIF specific same_frameset method?

But fine also to start with just checking the invariant part of the header only in VDIFFrameSet...

[TheMartianLife]

I'd be happy to take a crack at this, but I'd need to know more about what you're wanting to achieve. As per the VDIF spec, several fields should be invariant within a DataThread but everything except the header length is allowed to vary within a DataStream. Even EDV, though why they allowed that I am not sure.

So are you imagining a solution here that would:

1. Check that all VDIFFrames in a VDIFFrameSet match in all but a set of pre-defined fields (thread_id, and time)? Or...

2. Allow a user to specify fields which should match between all VDIFFrames in a VDIFFrameSet (maybe as part of an extension that allows general user-defined checks to be injected as part of verification)? Or...

3. Just check that any VDIFFrames with the same thread_id match as per the spec (frame_nbytes, chan, bps, and station)?

Where 1 and 2 enforce stricter rules than the spec but may be useful for people who expect their header fields to match based on knowing the pipeline that produced the data.

[mhvk]

@TheMartianLife - thanks so much for offering to help! I have not thought about this issue for quite a while, but I think consistency with the specification would be first. I guess most logical would be to give FrameSet a verify() method, but probably one that takes something like a flag to tell how much verification to do (probably string-based for now; I have a branch that uses an Enum instead to signal what verification to do, but that can be tackled later).

Note that in the years since this issue was raised, I've not really had a case where having strict verification would really have made a difference, while I did encounter files where setting strict verification would lead to failure... So, if, e.g., you have particular data files that do not read, or read poorly, definitely go for that instead! (I was intrigued to see from your github page that you work on satellite tracking; if baseband has been useful to you, that's great!!)