-
Notifications
You must be signed in to change notification settings - Fork 13
Release Notes v0.5.0
... Back to [wiki:LargeSurveyDatabase Large Survey Database Introduction].
The single most important improvement brought by this release are transactions. Transactions ensure the database remains consistent in the face of crashes while updating/importing data.
Performance was further optimized (expect up to ~30% faster execution of queries). {{{lsd-make-object-catalog}}} tool has been made much more intelligent and can now be safely used on a daily basis. A new utility, {{{lsd-check}}}, has been added to check for consistency of object and detection tables.
For developers, it's now possible to profile multiprocess runs (i.e., typical long-running MapReduce jobs).
See below for more details.
- Functionallity
- Implemented atomic, consistent, isolated and durable ([http://en.wikipedia.org/wiki/ACID ACID]) transactions
- {{{lsd-make-object-catalog}}} now knows which detections have already been matched to objects and avoids rematching them again
- {{{lsd-check utility}}}
- Performance
- Indexing of colgroups (ColGroup.getitem) sped up by 2x
- Stopped unnecessarily sorting join output
- Optimized the way join results are materialized
- Switched to BLOSC as default compressor in new tables
- Implemented our own lockfile-like mechanism, and stopped depending on procmail’s lockfile
- For developers
- Profiling of worker processes
- Other
- On-disk database format changed to support transactions/snapshotting
- Renamed all locks/temporary files to begin with ‘.__*, to be easier to find/remove
- Better shutdown behavior on CTRL-C
- Minor bugfixes and additions all over the place
The on-disk format has been changed to support transactions/snapshot facility. When LSD detects a database created with an older version, it will automatically upconvert it to v0.5 format.
Note that this conversion still keeps the table readable by previous versions. However, if new data is added it won't be visible when queried with older versions of LSD.
https://github.com/mjuric/lsd/commit/a6946ed417aaa52cf218e70570272329529f4357
All changes (writes) to LSD tables are now performed within snapshot-based ACID (atomic, consistent, isolated and durable) transactions familiar from other database systems. Until it commits, no modifications made in the transaction are be visible to other concurrent readers. Furthermore, neighbor caches and tablet tree caches (soon to be renamed to 'catalogs') of all affected tables are now automatically (and intelligently) rebuilt on commit, ensuring consistency. In previous versions of LSD these were built haphazardly at various stages, were prone to breaking, and would leave the database inconsistent in case of user error. With the addition of transactions, it’s either all consistent, or it doesn’t commit.
LSD implements transactions using a variant of the [http://en.wikipedia.org/wiki/Snapshot_isolation snapshot isolation technique]. Each LSD table has a {{{snapshots}}} directory whose subdirectories keep snapshot data (one subdirectory per snapshot). Snapshots can be in “opened”, “committed” or “failed” states. A snapshot that is committed contains special file, {{{.committed}}}, as a marker of its committed state. When reading, LSD does not "see" the snapshots that are not committed.
Snapshot directories do not physically store all table data existing at the time a snapshot was taken; they only contain the difference from previous snapshot to the current. Therefore, the logical table consists of a union of contents of all committed snapshots, made from the oldest to the newest, where contents (files) of newer snapshots overwrite eponymous files from older ones (see [wiki:LSDInternals here] for an illustration). This is similar to the way some backup systems handle incremental backups.
When LSD opens a table in the database, it enumerates the available snapshots, picks the newest one, and reconstructs the logical state of the table as described above. In practice, no scanning of directories is involved (would be too slow); the logical table state is precomputed and stored in catalog.pkl file when the snapshot is committed.
Transactions are implemented on top of snapshots. Each transaction creates a new (uncommitted) snapshot, and directs all writes (both data and metadata) to the new snapshot. When a commit is issued, if all post-commit operations (neighbor cache and catalog rebuilds) succeed, it commits the snapshot which then becomes available to other readers in the system.
As a consequence of this implementation:
- A (write-enabled) transaction can be ongoing concurrently with any number of readers. The readers won’t see any changes until the transaction commits and they refresh their database metadata (thus becoming aware of the newly available snapshot).
- As the data is only read from committed snapshots, queries executed within an open transaction can't see any modifications that made within it. For example, if you have a table with 10 rows, then begin a transaction, add or modify some rows and (without committing) query that table again, you will get the original 10 rows as a result. Only after you've called db.commit() will your queries begin returning the new data.
- Upon commit, LSD will do the necessary housekeeping. It will update of table metadata catalogs (catalog.pkl; a lookup table telling LSD where are the files that make up the logical table), and intelligently update the neighbor caches of cells that were modified by the transaction.
- Rolling back to an older snapshot is accomplished by removing all directories containing snapshots newer than the desired one. In principle, the directories don't even have to be removed -- LSD just needs to be told to look for a specific snapshot, and ignore the newer ones -- but this has not been implemented yet.
- To read from a given snapshots, all older snapshots must be present. As snapshots only contains the differences (a “diff”) between the current and previous snapshot, all diffs have to be present to reconstruct the current state.
- If anything goes wrong in a transaction, the snapshot directory it created will not be deleted, but won't have a '.committed' file either. Because of the latter, it will be ignored by LSD. Such transaction is said to have failed and its data can be safely removed, either manually ({{{rm -rf}}}), or using {{{lsd-vacuum}}} (an LSD utility still to be written).
For Python users, transaction API is simple: just use db.transaction() context manager -- anything within its with clause will execute within a transaction. See the wiki for details.
https://github.com/mjuric/lsd/commit/d83aa087cc1fc56b4a0a34ecc468cc0dfab1fea7
lsd-make-object-catalog is able to import only the detections not already matched to objects. Typically, all you will ever need to do is:
lsd-make-object-catalog --auto --fov-radius 2 ps1_obj ps1_det ps1_explsd-make-object-catalog has learned some new command line options:
-a, --auto:: Detect new detections that have appeared in the database since the last time the object table was built. Do it by querying the exposure table for list of all exposures, and querying object+detections table for list of all exposures whose detections have already been matched to objects. Difference the two lists to get a list of new exposures, and match only detections belonging to them to the object table. This is the option to use for everyday work (after new detections have been imported into the database), as it will "do the right thing".
-r, --radius:: Match radius (in arcseconds). Any detection within the match radius of an object, that is not closer to any other object, is assigned to the object. If there is no existing object within the given radius around a detection, one is created and the detection is matched to it.
-f, --fov-radius:: The radius of the field of view of a single exposure. Setting it significantly speeds up the search for yet unmatched detections. IMPORTANT: make sure this radius is always GREATER OR EQUAL to the actual FoV. Otherwise, not all detections will be included in the matching.
-g, --gen-explist:: Generate a list of new exposures in the database and print them out to stdout (this is internally used by the --auto algorithm)
-e, --explist:: Use the list of exposure IDs from the file given, and match only their detections to the objects. The format is one exposure ID per line (same as the one generated by -g). IMPORTANT: the code does not check that the detections from given exposures have not already been matched. You have to make sure they haven't.
https://github.com/mjuric/lsd/commit/fafea7ec5b51a4b29f335e44b993ab1bc9fb4885
Profiling with cProfile is easy for single-process jobs (NWORKERS=1), less so for multiprocess jobs. This version adds profiling support for multiprocess jobs, profiling each process separately and writing out the profile of each process into a file, if PROFILE=1 environment variable is set.
The profile is stored in a series of <process_name>..profile files, one for each worker thread. You can override the directory where these are stored by setting PROFILE_DIR.
You can prevent workers with less than X seconds of runtime to output their profiles by setting PROFILE_MIN_RUNTIME=X. This is useful to avoid having many small profiles generated for short-duration MapReduce jobs (e.g., tablet cache rebuilds).
The profiles can be inspected in detail with the usual pstats utilities. Alternatively, use profile-dump.py for a quick look at the top 10 most time consuming functions.