Update to htslib 1.10.2 (#184)

* Pull 1.10.2 submodule. Initial pass on i32->i64 type conversion, will eventually fix issue #183

* Fixes issue #109 while I'm at it, long hanging fruit fix

* Fix compression levels

* Going through https://github.com/samtools/htslib/blob/develop/README.large_positions.md to spot 64 bit reference position changes and sam_* function renaming. Only 4 errors left, but need to closely check types afterwards very carefully. Specially for the binary formats where this change does not apply and (sub)fields that do not necessarily require big representations

* Repoint submodule to upstream release tag

* Switch from rust-bio hosted htslib to upstream's samtools htslib.

* Try to amend submodule

* Handle submodules manually on TravisCI, since it is failing now: https://travis-ci.org/rust-bio/rust-htslib/builds/645772614

* Odd balls that would need review marked with XXX. New htslib seems to require libcurl

* Add curl-sys crate for hplugin_curl, aws s3 et al

* Add conditional libcurl compilation flags

* Be nice with cargo fmt [ci skip]

* Try installing libcurl meta-package instead of the openssl dependent one?: https://travis-ci.org/rust-bio/rust-htslib/builds/645790138#L1529

* Does not work with libcurl-dev metapackage: https://travis-ci.org/rust-bio/rust-htslib/jobs/645812050#L166, rollback and try with bionic ubuntu distro. It does compile locally on OSX

* Add matrix compile support for TravisCI, I would like to know why it compiles fine in OSX and fails on Linux now...

* PPA source no longer needed for ubuntu bionic? [ci skip]

* Bionic packages htslib with gnutls instead of openssl: https://packages.ubuntu.com/source/disco/htslib ... and it even comes with pre-shipped libcurl4-gnutls-dev, so no need to install it explicitly?: https://travis-ci.org/rust-bio/rust-htslib/jobs/645816116#L160

* sed is not gsed on OSX... not entirely sure this substitution is 100% needed

* Add libssl-dev to the mix for the openssl-sys: https://travis-ci.org/rust-bio/rust-htslib/jobs/645825022#L1000. Testsuite seems to run partially now: https://travis-ci.org/rust-bio/rust-htslib/jobs/645825022#L623

* Add build-essential in hopes to fix libcurl issue, incorporate #185, fix one testsuite warning

* Add curl on default features for hts-sys

* Point to the right htslib upstream repo, thanks @dlaehnemann [skip ci]

* Appease OSX musl-cross toolchain requirements: https://travis-ci.org/rust-bio/rust-htslib/jobs/645907695#L1222

* TravisCI timesout installing brew deps: https://travis-ci.org/rust-bio/rust-htslib/jobs/647667984#L212 ... Not sure if travis_wait will work with homebrew addons?: https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received... trying

* Try workaround for https://twitter.com/braincode/status/1226103002550849542

* Do not update homebrew (time intensive), switch from travis_wait to verbose to keep the build output being generated, hopefully

* brew install --verbose is not verbose enough for travis, reverting travis_wait and bumping to 40min for brew deps install and musl-cross

* Explicitly install system libcurl on linux and osx

* This .cargo/config makes it test right locally: rust-lang/rust#60149 (comment), only 5 tests failing now and no linking errors from libcurl (on OSX, at least) :)

* Cargo.toml missing quote misshap

* Same linker flags should apply for Linux builds? build-essential not necessary

* tid is i64 now, CRAM vs BAM test l_data is off by 3 and m_data is off too

* There should be no need for travis_wait with -v

* Debugging htslib structs with @jkbonfield guidance...

* Disable osx and musl-cross homebrew on the TravisCI matrix since it times out anyway, start looking into l_extranul and alignment issues for CRAM/BAM. Add curl_sys crate in hts-sys/src/lib.rs.

* Mystery solved for test_read_cram, testing some of the struct fields, explicitly state that l_data should differ according to #184 (comment)

* Extract selective CRAM/BAM field comparison to test helper function

* Formatting

* Temporarily commenting the hfile format category detection makes .bed filetype tests pass but .BAI idx locator fails

* Format detection seems to be broken upstream on htslib, see #184 (comment)

* Revert to libcurl4-openssl-dev instead of gnutls. All TravisCI script targets compiling and passing tests otherwise

* Bump up curl-sys version

* Experimenting musl cross-compilation with rustembedded cross. Ideally this would get rid of the homebrew musl-cross issue (too long compile times). /cc @nlhepler

* Remove assert_ne since cannot guarantee that l_data is different on all bam/crams

* Add minimum deps needed for rustembedded cross docker container

* Run docker in TravisCI, even if it is only supported on linux jobs :/

* Add CFLAGS suggested by @nlhepler

* Separate musl from GNU toolchains on Rust cross

* Bump up bindgen, vendorize openssl, further experimentation with compile flags

* Deprecate the mixed GNU/MUSL container

* Add minimal explanation on how to use the new docker files [ci skip]

* cross 0.2.0 with openssl vendored. Progress but more htslib compile time issues remain still

* Make sure libzma is present in both containers [ci skip]

* Focusing on LLVMv8 GNU cross build first, musl will come later on (if possible w/ htslib)

* Two more usize that should be u64

* Amend and settle BED header debate

* Rename test name accordingly

* Transition to cross instead of cargo for builds and tests

* All GNU/clang/llvm8 builds and tests seem to succeed, transitioning to musl testing now...

* Remove stray (repeated) musl build

* Remove @FiloSottile musl-cross homebrew formula since it takes too long to compile and timeouts CI, using clux/muslrust gets me a bit farther. Remove local rustup musl target... since htslib 1.10, the introduction of openssl complicated local compilation w/ musl greatly, unfortunately we have to resort to docker these days instead :/

* Experiment with toolchain setup in https://gitlab.com/rust_musl_docker/image. Remove vendored openssl since it's shipped in cross containers.

* Remove build.rs flags noise and different tests, containers should handle them transparently depending on the defined environment within, thanks @nlhepler

* Indicate dependencies and build flags required for OSX [ci skip].

* enable musl builds

* add libclang

* minor

* dbg

* revert

* dbg

* dbg

* Revert cross/docker madness, bottled up musl-cross myself over here: https://github.com/brainstorm/homebrew-musl-cross/commit/9304bf9cd6bc2b784083948771977d502030815a, high sierra bottle coming up soon. Cargo build script draft

* Add in openssl to hts-sys

* Make clippy boolean logic happy

* Remove homebrew OSX check since we'll just use rustembedded cross

Co-authored-by: Julian Gehring <[email protected]>
Co-authored-by: Johannes Köster <[email protected]>

Author: 3 people

.cargo/config

@@ -0,0 +1,12 @@
+[target.x86_64-apple-darwin]
+rustflags = [
+  "-C", "link-arg=-undefined",
+  "-C", "link-arg=dynamic_lookup",
+]
+
+[target.x86_64-unknown-linux-musl]
+rustflags = [
+  "-C", "link-arg=-undefined",
+  "-C", "link-arg=dynamic_lookup",
+]
+linker = "x86_64-linux-musl-gcc"

.github/workflows/rust.yml

@@ -39,10 +39,6 @@ jobs:
           override: true
           components: clippy
 
-      - name: Install system dependencies
-        run: |
-            sudo apt-get install --yes zlib1g-dev libbz2-dev musl musl-dev musl-tools
-
       - name: Lint with clippy
         uses: actions-rs/clippy-check@v1
         with:
@@ -67,7 +63,7 @@ jobs:
 
       - name: Install system dependencies
         run: |
-            sudo apt-get install --yes zlib1g-dev libbz2-dev musl musl-dev musl-tools
+            sudo apt-get install --yes zlib1g-dev libbz2-dev musl musl-dev musl-tools clang libc6-dev
 
       - name: Run cargo-tarpaulin
         uses: actions-rs/[email protected]
@@ -79,3 +75,17 @@ jobs:
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           path-to-lcov: ./lcov.info
+
+      - name: Test musl build without default features
+        uses: actions-rs/cargo@v1
+        with:
+          use-cross: true
+          command: build
+          args: --target x86_64-unknown-linux-musl --no-default-features
+
+      - name: Test musl build with all features
+        uses: actions-rs/cargo@v1
+        with:
+          use-cross: true  
+          command: build
+          args: --target x86_64-unknown-linux-musl --all-features

.gitignore

@@ -1,4 +1,5 @@
 .vscode/
+.idea
 *~
 target
 Cargo.lock

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "htslib"]
 	path = hts-sys/htslib
-	url = https://github.com/rust-bio/htslib.git
+	url = https://github.com/samtools/htslib.git

Cargo.toml

@@ -33,9 +33,11 @@ snafu = ">= 0.5.0, <= 0.6.0"
 hts-sys = { version = "^1.9", path = "hts-sys", default-features = false }
 
 [features]
-default = ["bzip2", "lzma"]
+default = ["bzip2", "lzma", "curl"]
 bzip2 = ["hts-sys/bzip2"]
 lzma = ["hts-sys/lzma"]
+curl = ["hts-sys/curl"]
+openssl = ["hts-sys/openssl"]
 serde = ["serde_base", "serde_bytes"]
 
 [dev-dependencies]

Cross.toml

@@ -0,0 +1,4 @@
+[target.x86_64-unknown-linux-musl]
+image = "brainstorm/rust_musl_docker:stable-latest-libcurl"
+[target.x86_64-unknown-linux-gnu]
+image = "brainstorm/cross-x86_64-unknown-linux-gnu:libcurl-openssl"

README.md

@@ -19,7 +19,18 @@ If you only want to use the library, there is no need to clone the repository. G
 
 ## Requirements
 
-To compile this crate you need the development headers of zlib, bzip2 and xz.
+To compile this crate you need the development headers of zlib, bzip2 and xz. For instance, in Debian systems one needs the following dependencies:
+
+```shell
+$ sudo apt-get install zlib1g-dev libbz2-dev liblzma-dev clang
+```
+
+On OSX, this will take a significant amount of time due to musl cross compiling toolchain:
+
+```shell
+$ brew install FiloSottile/musl-cross/musl-cross
+$ brew install bzip2 zlib xz curl-openssl
+```
 
 ## Usage
 

docker/Dockerfile.gnu

@@ -0,0 +1,8 @@
+FROM rustembedded/cross:x86_64-unknown-linux-gnu
+
+ENV LIBCLANG_PATH /usr/lib/x86_64-linux-gnu
+ENV LLVM_CONFIG_PATH /usr/bin
+RUN apt-get update && \
+    apt-get install -y libcurl4-openssl-dev zlib1g-dev libbz2-dev liblzma-dev clang-8 && \
+    ln -sf /usr/bin/llvm-config-8 /usr/bin/llvm-config && \
+    ln -sf /usr/lib/x86_64-linux-gnu/libclang-8.so.1 /usr/lib/x86_64-linux-gnu/libclang.so.1

docker/Dockerfile.musl

@@ -0,0 +1,7 @@
+FROM rustembedded/cross:x86_64-unknown-linux-musl
+
+ENV PKG_CONFIG_ALLOW_CROSS 1
+ENV OPENSSL_LIB_DIR /usr/lib/x86_64-linux-gnu
+ENV OPENSSL_INCLUDE_DIR /usr/include/openssl
+RUN apt-get update && \
+    apt-get install -y libssl-dev libcurl4-openssl-dev zlib1g-dev libbz2-dev liblzma-dev musl musl-dev musl-tools linux-libc-dev linux-headers-4.15.0-20-generic

docker/README.md

@@ -0,0 +1,17 @@
+# cross rustembedded containers
+
+Allows to compile (rust-)htslib in a variety of environments and architectures via [rustembedded cross](https://github.com/rust-embedded/cross).
+
+## Quickstart
+
+```shell
+$ cd docker
+$ docker build -t brainstorm/cross-x86_64-unknown-linux-musl:libcurl-openssl . -f Dockerfile.musl
+$ docker build -t brainstorm/cross-x86_64-unknown-linux-gnu:libcurl-openssl . -f Dockerfile.gnu
+```
+
+Then to build and test rust-htslib with the above containers, proceed as you would with `cargo`, using `cross` instead, i.e:
+
+```shell
+$ cross build --target x86_64-unknown-linux-musl 
+```

hts-sys/Cargo.toml

@@ -8,7 +8,7 @@ description = "This library provides HTSlib bindings."
 readme = "README.md"
 keywords = ["htslib", "bam", "bioinformatics", "pileup", "sequencing"]
 license = "MIT"
-repository = "https://github.com/rust-bio/rust-htslib.git"
+repository = "https://github.com/samtools/htslib.git"
 documentation = "https://docs.rs/rust-htslib"
 edition = "2018"
 
@@ -21,14 +21,18 @@ libc = "0.2"
 libz-sys = "1.0"
 bzip2-sys = { version = "0.1", optional = true }
 lzma-sys = { version = "0.1", optional = true }
+curl-sys = { version = "0.4.26", optional = true }
+openssl-sys = { version = "0.9.54", optional = true }
 
 [features]
-default = ["bzip2", "lzma"]
+default = ["bzip2", "lzma", "curl"]
 bzip2 = ["bzip2-sys"]
 lzma = ["lzma-sys"]
+openssl = ["openssl-sys"]
+curl = ["curl-sys"]
 
 [build-dependencies]
 fs-utils = "1.1"
-bindgen = { version = "0.52.0", default-features = false, features = ["runtime"] }
+bindgen = { version = "0.53.1", default-features = false, features = ["runtime"] }
 cc = "1.0"
 glob = "0.3.0"

hts-sys/build.rs

@@ -13,9 +13,9 @@ use std::fs;
 use std::path::PathBuf;
 use std::process::Command;
 
-fn sed_htslib_makefile(out: &PathBuf, patterns: &Vec<&str>, feature: &str) {
+fn sed_htslib_makefile(out: &PathBuf, patterns: &[&str], feature: &str) {
     for pattern in patterns {
-        if Command::new("sed")
+        if !Command::new("sed")
             .current_dir(out.join("htslib"))
             .arg("-i")
             .arg("-e")
@@ -24,7 +24,6 @@ fn sed_htslib_makefile(out: &PathBuf, patterns: &Vec<&str>, feature: &str) {
             .status()
             .unwrap()
             .success()
-            != true
         {
             panic!("failed to strip {} support", feature);
         }
@@ -63,10 +62,18 @@ fn main() {
         cfg.include(inc);
     }
 
+    let use_curl = env::var("CARGO_FEATURE_CURL").is_ok();
+    if !use_curl {
+        let curl_patterns = vec!["s/ -lcurl//", "/#define HAVE_LIBCURL/d"];
+        sed_htslib_makefile(&out, &curl_patterns, "curl");
+    } else if let Ok(inc) = env::var("DEP_CURL_INCLUDE").map(PathBuf::from) {
+        cfg.include(inc);
+    }
+
     let tool = cfg.get_compiler();
     let (cc_path, cflags_env) = (tool.path(), tool.cflags_env());
     let cc_cflags = cflags_env.to_string_lossy().replace("-O0", "");
-    if Command::new("make")
+    if !Command::new("make")
         .current_dir(out.join("htslib"))
         .arg(format!("CC={}", cc_path.display()))
         .arg(format!("CFLAGS={}", cc_cflags))
@@ -75,7 +82,6 @@ fn main() {
         .status()
         .unwrap()
         .success()
-        != true
     {
         panic!("failed to build htslib");
     }

hts-sys/htslib

@@ -11,6 +11,8 @@ extern crate libz_sys;
 extern crate bzip2_sys;
 #[cfg(feature = "lzma")]
 extern crate lzma_sys;
+#[cfg(feature = "curl")]
+extern crate curl_sys;
 
 // include on-the-fly generated bindings
 include!(concat!(env!("OUT_DIR"), "/bindings.rs"));

hts-sys/src/lib.rs

@@ -86,7 +86,7 @@ impl RecordBuffer {
                 let to_remove = self
                     .inner
                     .iter()
-                    .take_while(|rec| rec.pos() < window_start as i32)
+                    .take_while(|rec| rec.pos() < window_start as i64)
                     .count();
                 for _ in 0..to_remove {
                     self.inner.pop_front();
@@ -111,7 +111,7 @@ impl RecordBuffer {
                     record.cache_cigar();
                 }
 
-                if pos >= end as i32 {
+                if pos >= end as i64 {
                     self.overflow = Some(record);
                     break;
                 } else {