djot, long-term document conservation/maintenance and version control systems (e.g. git)

[massifrg]

I'm looking at djot in the perspective of long term management of large databases of documents.

In particular, I'm looking for

• a textual, non binary format to reduce data loss in case of errors in the media

• a format that can be converted in current and future formats with a minimal loss of information, especially about its structure

• a format that can work with current diff and merge tools, so that you could work collectively at your documents the same way developers do with code

In the past I chose XHTML, because it's well documented, it's textual, it can be indexed with Lucene-based tools (Solr, Elasticsearch), it's structured, letting you do some searches based on its structure.
You don't have to write a XML grammar that only you know, and customization is still widely possible through classes and attributes.

Unfortunately, it's XML based, so it does not play well with diff and merge tools; there's an explanation here.

At first glance, I'd say djot should play well with version control systems (VCS), because it looks line-oriented like code;
I mean: it should be feasible to diff and merge two versions of a djot document with tools like meld.
Is it really like that?

Pandoc internal document model is simple, yet rich and flexible enough to cover many documents.

It's the best candidate for convertibility, though saving your documents in that format would expose you to future changes in the Pandoc Types.

AFAIK the only two pandoc formats that save all the information of a document are native and json.
But none of them is easily human-readable, indexing requires some work, and you have no tools ready to query their structure.
It would be easy to create an XML version of that format, but still it would not play well with VCS.

Here comes djot. But what do you lose converting from pandoc native to djot?

Perhaps the biggest loss is on tables, where djot can't live up to the richness of pandoc model.

[jgm]

Complex tables, figures, and citations are currently the main things we have in pandoc's AST but not in djot.

[massifrg]

Thanks for clarifying that.

In your reply on complex tables you talk about "list tables".
If they were line-oriented, so to play well with diff and merge tools, it would be great and perhaps it would make "git for documents" possible.

Real time, concurrent editing has some implementations.

I understand its attractiveness because it's indeed a complex coding challenge; but I'd prefer using git for documents as we do for code. It's collaboration on documents in a more reasoned and controlled way.