Proposal to evolve to HTSJDK version 3 (DO NOT MERGE)

[magicDGS]

Description

Because the governance model is gonna change soon (#871), I would like to revive the discussion about several issues about the API and proposing to evolve to HTSJDK in a join effort by the community. The related issues are the following:

• Non-backwards compatible API (Appetite for a non-backwards compatible upgrade to core APIs? #520), called in the proposal HTSJDK3. This includes implementing new interfaces as a public API independent of the implementation (see Improved naming and documentation for flag field accessors and mutators. #868 for a discussion about the read interface).
• Evolving API (process for evolving api #378). By moving to HTSJDK-3 we can remove deprecated code and start from scratch. In addition, annotations for beta features could be implemented for the HTSJDK-3 release, marking as beta features that may change.
• Sub-module structure to reduce dependencies in the core functionality (Proposal: separate HTSJDK in different gradle build artifacts #896). This also includes the issue swith the SRA support (e.g., Remove SRA related code and replace with a plugin API #414 and Remove SRA code from htsjdk  #894), which can be moved to a different package.
• Mixing license (Mixed licensing is unfortunate; can we remedy this? #917). By moving to HTSJDK-3, we can ask for porting the code to the new implementation using a common license.

Proposal

The main idea is to create a folder structure as the one in htsjdk3-dev in the root folder of the project and move the current version of htsjdk to a sub-module called htsjdk-2 (not done in the PR for reduce the number of changes). Tasks in gradle to build the current version should be modified to keep the same behavior, and a new task to build the developmental version 3 should be included too.

In addition, I drafted a new annotation for mark the status of the classes with respect to version 3 (@Htsjdk3). As an example, I ported the Locatable and Feature interfaces to the new package. This will allow to see the progress of the new version and keep in sync fixes between versions.

I know that this is a huge proposal, but I expect that most of the code will require just copy-paste and/or repackage (and maybe adding a @Beta annotation for some code that may be changed in the future). I think that the community will benefit from this change to create a more consistent API for NGS data processing. In addition, some dev issues could be solved on the port to HTSJDK-3.

Bringing the attention to the maintainers of the project: @tfenne, @lbergelson, @jacarey

Checklist

• Code compiles correctly
• New tests covering changes and new functionality
• All tests passing
• Extended the README / documentation, if necessary
• Is not backward compatible (breaks binary or source compatibility)

[codecov-io]

Codecov Report

Merging #928 into master will decrease coverage by 0.019%.
The diff coverage is n/a.

@@              Coverage Diff               @@
##              master     #928       +/-   ##
==============================================
- Coverage     65.138%   65.12%   -0.019%     
+ Complexity      7281     7279        -2     
==============================================
  Files            528      528               
  Lines          31975    31975               
  Branches        5469     5469               
==============================================
- Hits           20828    20822        -6     
- Misses          8992     8996        +4     
- Partials        2155     2157        +2

Impacted Files	Coverage Δ	Complexity Δ
src/main/java/htsjdk/tribble/Feature.java	0% <ø> (ø)	0 <0> (ø)	⬇️
...sjdk/samtools/util/Md5CalculatingOutputStream.java	69.231% <0%> (-7.692%)	8% <0%> (-1%)
...samtools/util/AsyncBlockCompressedInputStream.java	72% <0%> (-4%)	12% <0%> (-1%)

[lindenb]

yes please !

The main idea is to create a folder structure as the one in htsjdk3-dev

just a suggestion: project like apache-math have created a package-version since v3.

• org.apache.commons.math3
• org.apache.commons.math4

so we could have

import htsjdk.*
or
import htsjdk3.*

but I don't know what is the best option

[magicDGS]

@lindenb, I was thinking about the new package too. Nevertheless, I opted for proposing this as a temporary solution and then keep the same package for two reasons:

• If there is a new package htsjdk3, what will happen with the previous one? Will be removed from the repo at some point or they will continue living together? It means that htsjdk3will be implemented in a different repo and htsjdk stay here for historical reasons?
• If both htsjdk and htsjdk3artifacts exists, that could bring the problem of pulling down the library twice, which I personally don't like; I rather prefer a complete change of name in case that the library gets that different to need a distinction of packages. Although for development I think that it could be a good idea...

[magicDGS]

Maybe the maintainers can chime in here: @tfenne, @lbergelson, @jacarey? Thank you in advance!

[yfarjoun]

Given that there's a new repo for htsjdk-next-beta, can we close this PR?

[magicDGS]

Closing in favor of the new repo