Pipeline primer

This document describes the analysis pipeline found in the analysis_IM github repository. In particular it describes the code in the python_km sub directory. This code is almost entirely written in python and makes heavy use of the numpy and scipy libraries. This document is meant to be a broad overview of code providing context and examples. For detailed documentation of the individual functions and classes, please refer to the online documentation that is automatically generated from the module doc-strings. This documentation may be found here. Similar documentation for the helper library kiyopy can be found here.

The analysis pipeline is broadly divided into two halves based on the format that the data takes. These are the time stream, and the map. In the first half of the pipeline, the data is still in the digital time stream format in which it came from the telescope. Example of processing that occurs in this half include calibration and rfi cleaning. There is then a gridding or map making process that compresses the time stream down to a map like object and an associated noise covariance matrix. Once in a map like format, additional processing occurs doing things like foreground subtraction and power spectrum estimation.

The data analysis is meant to be modular, divided into stages. Each stage (whether be in the time_stream section or the map section) reads the data from disk, performs some processing and then writes it back to disk. The stages are glued together into a pipeline and there are special modules to do the gluing.

Time Stream Data Format

Map Data Format

Pipeline and Stages

Setting Up and Getting Started

Unit tests as Examples

If you need an example of how a particular module, function, or class should be used, look no further than the unit tests. Almost every module has a corresponding testing module with the same name except for the prefix 'test_'. You can use this as an example of how a particular class or function should be used. Note that you may want to familiarize yourself with python's built in unittest module.

Contact

Do not hesitate to contact Kiyo if any of this seems unclear or if the information you need is simply missing. I'm doing my best to keep the documentation complete and up to date, but I will no doubt fail (feel free to help and contribute to this primer).