BitMagic v.7.13.4

1. Improved str_sparse_vector<>::clear_all()  - added parameter to reset remap matrix by default, which is safer (less surprises for the user). Method signature changed, this may cause some minor incompatibility issues.



2. New method bm::sparse_vector<>::get_unsigned_bits() - to extract partial value (only a few bits)

3. Optimizations of bm::bvector<>::inc() (minor)



4. Added bm::bvector<>::swap() - method to swap two bit positions - basis for implementing sort or shuffle algorithms on compressive memory structures

5. Added bm::sparse_vector<>::swap() - to help implement swap of shuffle algorithms on succinct vectors

6. Added new example strsvsample09 to showcase sort algorithm in compressive memory space.

7. Optimization for bit-vector deserialization algorithm to avoid frequent reallocations of top-level memory table. Especially important for large 48/64-bit mode bit-vectors.

8. optimizations of bm::bvector<>::import(), implemented memory reservation (important for large 48/64-bit imports).

9. Fixed bug with collaborative XOR compression, causing debug level asserts and inability to decode compressed vectors.

Various code-cleanups to reduce warnings in different compilers, improves MSVC compatibility.

10. Fixed bug in incorrect binary search in compressive string vector (bm::sparse_vector_scanner<>)

11. Improved build system to better support Apple M2

BitMagic release v7.12.3

Release Notes: BitMagic 7.12.3

Bug fixes:

1. Fixed bugs related to handling of immutable bit-vectors
2. Fixed bug in Rank-Select index assisted bvector<>::count_to()

New features, optinmizations, improvements:

1. Implemented sparse vector read-only deserialization.
   bm::sparse_vector_deserializer<>::set_finalization(bm::finalization::READONLY);
   https://github.com/tlk00/BitMagic/tree/master/samples/svsample02

2. Optimizations: for bvector<>::test( ) membership testing (and changing)
   for GAP compressed blocks. BM implementation uses binary search (ONlogN) for this operation
   with SIMD optimizations (hybrid binary search). The algorithms around this problem were
   reviewed to minimize number of comparisons, better use L1 cache and improve speed.
   Performance gain of 10-20% measured on synthetic tests.

3.Optimizations: bm::aggregator<> SIMD and general purpose optimizations
of aggregate AND-SUB operations. Group AND-SUB operation is at the core of succinct vector
searches, inspired by Bloom filters but uses "data as an index approach"
(works well for bit-sliced succinct vectors).
Latest version adds various optimizations related to SIMD, parallel memory reads,
algorithmically better detection of search reductions.

4. Optimizations: bm::sparse_vector_scanner<>::bfind_eq_str() -
   binary search in compressive memory vectors of strings.
   Implemented a new index for approximating the search using binary index.
   BM allows to setup a binding between bm::sparse_vector_scanner and a sperse
vector itself, defining the fraction of elements to keep decompressed for the fast approximated
search. Maintenance of a index fractionally reduces a memory efficiency, but significantly
improves the search speed.
New example was added to the code repo to illustrate the new API.
https://github.com/tlk00/BitMagic/tree/master/samples/strsvsample08.

Release notes:
http://bitmagic.io/bm-7.12.3.html

BitMagic release v7.11.2

Bug fixes:

1. Fixed bug in rank-select index assisted compute of rank (affecting compressive sparse vector)
2. Fixed bug with potential misaligned reads in SIMD mode
3. Fixed bug in deserialization of empty sparse vector

New features and improvements:

1. Improved read-only mode for bm::bvector<> to allocate one contiguos memory block for the arena to
   better reduce memory fragmentation. This improvement also affects read-only mode memory allocation
   for bit-vector based succinct vectors.

2. succinct vector for strings (bm::str_sparse_vector) remap() method improved to use less memory
   in the process

3. Added bit-vector immutability methods into C-language mappings

Release notes:
http://bitmagic.io/bm-7.11.2.html

BitMagic release v7.10.3

Release Notes: BitMagic 7.10.3

Bug fixes:

1. Fixed bug in serialization/deserialization of empty null-able bm::str_sparse_vector<> (could case crashes and assertion faults)
2. Fixed corner case bug in XOR compression of empty succinct vectors
3. C language interface (libbm) – fixed incorrect handling of CPU SIMD capabilities (AVX2)

New features and improvements:

1. Implemented immutable (read-only) bit-vectors.
   New version of BitMagic library first introduces an API for making vectors immutable.
   Immutable mode has two advantages: better memory footprint and faster analytical operations.

Mutable writable vectors in BitMagic are implemented using block allocation strategy (that is why sparse vectors). To facilitate edits in compressive mode vectors do space reservations to avoid frequent re-allocations which otherwise be a performance killer in multi-threaded scenarios. Such memory reservations is obviously not free and goes against the idea of memory succinct operations.
New version adds APIs to freeze vectors to read-only mode, which assembles all blocks together into one memory arena (heap defragmentation, which can reduce memory footprint for your program) and drops any reservations made for editing (reduce RAM consumption).
An additional extra benefit is that immutable mode now uses contiguous arena of memory, giving CPU a better chance for cache memory reuse and more efficient hardware prefetch. Performance benchmarking shows that this factor can give 1 to 5 percent performance improvements on data-science operations like binary distance functions (Dice). The performance advantage is not super-significant, but it is a nice bonus for applications which can use and benefit from read-only mode.
Special note: current implementation still turns read-write mode back on deserialization of vectors because current serialization format remains without changes for this release.
https://github.com/tlk00/BitMagic/tree/master/samples/bvsample26

2. New method for random access read from the Rank-Select succinct vector bm::rsc_sparse_vector<>::gather() – new method can give you 10% and better performance improvement comparing to naïve random access. Actual performance improvement depends on the access pattern and data pattern.
   https://github.com/tlk00/BitMagic/tree/master/samples/rscsample06

3. Improved performance of Rank-Select index. The most notisable performance improvement is for Arm and configurations without efficient SIMD acceleration. SSE4.2 and AVX2 builds show minor improvement or parity with the previous version

4. Improved “noexcept” declarations in the code (improved WASM target performance)

5. Improved bm::bvector<>insert()/shift_right() to avoid unnecessary deoptimization of compressed blocks

Release notes:
http://bitmagic.io/bm-7.10.3.html

BitMagic release v7.9.3

Release Notes: BitMagic 7.9.3

1. Improved bm::sparse_vector_scanner<>::pipeline::set_search_mask()
   with ability to specify AND mask. Search mask is very useful optimization tool
   for cases when search can be limited/prunned by a prior knoweledge or a prior search on a more selective index.
   SQL example: Field1 = 10 AND Field2 IN ('value1', 'value2')
   One side of an SQL expression Field1 = 10 as a bit-vector can now be fed into a
   pipeline index-free search scanner to potentially make search significantly faster.
   Example: https://github.com/tlk00/BitMagic/tree/master/samples/strsvsample07

2. bm::str_sparse_vector<> - fixed a few bugs related to processing of succinct vectors with NULL values.

3. New API bm::str_sparse_vector<>::compare() - optimized comparison for construction of sort
   method taking two indexes of elements to perform comparison.

4. Optimizations for SSE2, SSE4.2 code for logical set subtraction (AND NOT).

5. Minor optimizations of bm::aggregator<> - collection of algorithms for logical expression search.

6. All succinct vectors: Implemented new API functions for bulk
   set_null() and clear() of vector elements.
   New methods take bm::bvector<> as an input to set/clear marked elements.
   New operations are significantly faster than random access element assignments.
   https://github.com/tlk00/BitMagic/tree/master/samples/strsvsample04

7. All succinct vectors: New method try_get() for conditional access
   to not NULL elements. New method is somewhat faster than separate is_null()/get() calls.
   https://github.com/tlk00/BitMagic/tree/master/samples/rscsample01

8. Integer succinct vectors: optimizations of random element access (10% gain in some cases).

9. New example on how to use algorithms for bit-vector traversal: bm::for_each_bit(),
   bm::for_each_bit_range(), bm::visit_each_bit(), bm::visit_each_bit_range().
   https://github.com/tlk00/BitMagic/tree/master/samples/bvsample25

10. Minor optimizations for Rank-Select index construction and search.

Release notes:
http://bitmagic.io/bm-7.9.3.html

BitMagic release v7.8.0

Release Notes: BitMagic 7.8.0

1. Implemented complete set of index-free search methods on compressive memory vectors with
   bm::sparse_vector_scanner<>: GT, GE, LT, LE, RANGE fast search functions.
   New search logic uses fast bit-vector operations without de-compressing the succinct models
   to solve search logic problems fast and without external indexes overhead
   New functionality is important for both search systems and columnar databases.

2. New example on how to use the new scanner search functions:
   https://github.com/tlk00/BitMagic/tree/master/samples/svsample10

3. SSE2 improvements: added more SSE2 kernels and optimizations.

4. First version of ARM NEON optizations (as SSE2 to NEON translation).
   Tested on R-Pi, it adds up to 25% performance gain in some logic search benchmarks.

Release notes:
http://bitmagic.io/bm-7.8.0.html

BitMagic release v7.7.7

1.Fixed a bug in bm::bvector<>::merge() - destructive OR operation, when arg vectors is empty and uninitialized
function did not implement a correct logical OR (serious issue!)

2.Implemented a new logical idiom bvector<>::bit_or_and()
as
C := C OR (A AND B)
Fused OR+AND is an often used idiom in query systems, implementations of SQL and operation on memory compressed
structures. Fused implementation uses multiple optimizations and does not require a temporary vector, avoiding
allocations and memory copy.
New idiom is 2x times faster in synthetic tests of uncompressed bit-vectors.

3. bm::aggregator::pipeline now implements a fast mode to run multiple AND-SUB queries with an optional
   aggregation of results via an OR function.

Aggregator logical pipeline implements fast idioms used in BitMagic succinct vectors to implement
sparse/dense vector search or query requests.

bm::aggregator::pipeline uses cache memory bandwidth and optimizations to implement
series of AND-SUB as: (bv1 AND bv2 AND bv3…) AND NOT (bvS1 OR bvS2 OR … ) with an optional final
accumulation of multiple search requests using OR logical function.

Aggregator pipeline is used internally in BitMagic to implement succinct bit-sliced vector searches
(bm::scanner<>) 2-3 times faster. The speed achieved in 7.7.7 release demonstrates performance levels
otherwise specific to systems using indexes.
Fast index-free searches can significantly reduce both the systems footprint (RAM and disk) and
complexity of implementation.

4. Algebra of Sets tutorial (bvsetalgebra.cpp) example reworked to illustrate use of new fused OR-AND
   operations and aggregator pipeline (AND-SUB-OR).

https://github.com/tlk00/BitMagic/tree/master/samples/bvsetalgebra

5. bm:scanner::pipeline implements fast multiple string search in memory compressed string vector with an
   optional OR stage (under the hood uses bm::aggregator<>).
   Latest version makes a change in semantics to use compile-time options to configure pipeline,
   new options result in faster code due to compile-time specialization (C++-17 is very useful there).

OR stage helps to implement a SQL idiom: Field1 IN (value1, value2… valueN) for cases where list of
search values is in the tens of thousands or more or Field1 IN (SELECT FieldN FROM…) idiom using
memory compressed index-free search.

New version adds important optimizations of the algorithms to automatically tune-up for a typical
L2 cache size, but also adds a manual override (batch_size) to tune and tweak the system for a specific
HW configuration and data distribution statistics. The auto-tune topic definitely deserves more optimization
in the future.

New usage modes and benchmarks are available at:
https://github.com/tlk00/BitMagic/tree/master/samples/strsvsample07

Release notes:
http://bitmagic.io/bm-7.7.7.html

BitMagic release v7.6.0

• fixed regression bug from 7.5.0 in dynamic matrix allocation algorithms of succinct vectors

• bm::aggregator<> reworked to support new pipeline mode for AND-MINUS ("all this but not that" queries)
  aggregation, using L1/L2 cache and algorithmic optimizations to run thousands of coordinated aggregations times
  faster than it was possible in previous versions

• bm::scanner<> implements string new string search method for bulk search with bm::aggregator<>::pipeline under the hood. New bm::scanner<>::pipeline can run multiple string searches in unordered succinct string vectors times faster.
  It also implements new search methods which can be used to run massive queries and construct results
  (inverted list bit-vectors) or run stat analysis computing results population count without materializing the vectors
  (effectively histogram construction).

-New example: strsvsample07
added to illustrate use of fast scanner and scanner pipeline 3x times, with synthetic benchmarks showing 3x better
performance for bulk search cases.

• Parallel compute engine and thread pools reworked to use Modern C++ approach based on lambdas rather than function pointers. New development lays the internal foundation for BitMagic parallel algorithms to unfold its potential
  on multi-threaded systems.

Release notes:
http://bitmagic.io/bm-7.6.0.html

BitMagic release v7.5.0

Release Notes: BitMagic 7.5.0

• BitMagic now requires C++17 compiler
• bm::sparse_vector<> - succinct vector using principles of bit slicing (or bitwise transposition) reworked to support signed integer types. Previous versions were designed for unsigned types and it was a limitation. Signed types use complement codes, which may not be friendly to bit-slicing approach because negative ints with small absolute values become strings of 1s on the binary level -1 (represented as 11111). New version handles this case and automatically transforms signed values into “sign and magnitude” internal representation to have good memory and disk serialization efficiency (BM vectors are designed to be used as a variable binary encoders for non-sparse cases). New feature is consistently supported in serialization, scanner search and should be completely transparent for the programmer
• bm:: str_sparse_vector<> reworked to transparently support dynamic string length. The initial configuration used as a template parameter is now means the initial value (16 in the example below)
  typedef bm::str_sparse_vector<char, bvector_type, 16> str_sv_type;
  When a string with length above the specified parameter added, the container automatically resize its internal bit-matrix to correctly support it. No need to conservatively pre-allocate values now.
  This is a significant internal change which will allow to implement succinct vectors of arrays based on the same bit-slicing principles.
  -New example: svsample07a added to illustrate use of fast scanner and interval enumerator to study the properties of unsorted integer sequences in compact memory mode.

Release notes:
http://bitmagic.io/bm-7.5.0.html

BitMagic release v7.4.0

1. Continued effort to provide better version for WebAssembly, first WASM SIMD build is available using:
   #define BMWASMSIMDOPT (or define via makefile).

emcc command like may look like:
emcc -std=c++11 -s ALLOW_MEMORY_GROWTH=1 -O2 -msse4.2 -msimd128 -D BMWASMSIMDOPT -s WASM=1 -s DISABLE_EXCEPTION_CATCHING=0 -fno-rtti

Please note that WebAssembly SIMD is implemented as cross-compilation tweak for SSE4.2 (thus needs -msse4.4 flag).

WebAssembly SIMD is still an experimental feature for all browsers but Google Chrome.
Initial performance observations sometimes show faster than native(without SIMD) speed for optimized algorithms.

2. Implemented 3-valued logic operations (INVERT, AND, OR) following the Kleene truth tables.
   https://en.wikipedia.org/wiki/Three-valued_logic

Usage example:
https://github.com/tlk00/BitMagic/tree/master/samples/bv3vlogic

3. Fixed bugs in parallel algorithms (race condition).