Road to 1.0 and beyond?

[matklad]

TL;DR: djot doesn't feel very new anymore. More value can be unlocked if we make a stability promise, so that people can start relying on the fact that djot documents written today would be interpreted exactly the same in 2049. Should we promise stability?

---

I've been using djot for almost two years at this point. I've migrated my blog, wrote numerous new articles, did a couple of slide-decks and keep my notes and TODOs in djot. I am more than happy with djot for my "private consumption". While not absolutely perfect, it is a marked improvement over next-best options (AsciiDoc & MDX (though, notably, I haven't tried pandoc markdown)).

However, because there's no formal stability promise, certain use-cases are still impossible. For example, I can't take my private slide-deck generating scripts and publish them as a stand-alone utility for other people to use. If people start using it, and then djot changes its parsing rules, and then 1% of the slide decks created by other people would be rendered incorrectly, I would be feeling bad.

There are many other such "content gravity" use-cases. I might also be designing a new programming language, and be in need of a structural syntax for documentation comments, which would also require near-absolute implementation stability.

In short, saying that "djot documents would always be parsed the same" unlocks qualitatively new use-cases, and makes the world a better place.

While I have a number of my pet features which I would really love djot to implement (#240, #28, #31, jgm/djot.js#74), at this point I think I'd take djot that doesn't do them, but is stable, over djot that implements them, but also continues to change.

There are several paths we can take from here, non exhaustively:

• Do-nothing action, if we think that djot is not ready for stability yet, and that we need some unbounded time for it to mature.
• We could also start moving towards stability and aspirationally aim at having a finalized behavior in a year, and, until that time, work through existing issues and decide whether to fix them or not.
• Finally, I think just saying that djot is perfectly good as it is today, and declare all existing bugs as features, wouldn't be a too terrible solution. Djot is actually quiet good indeed.

And then, there's this tantalizing thought that maybe we can have a cake and eat it to? Perhaps we could do Rust-style "stability without stagnation", where we both provide a rock-stable behavior, but at the same time allow significant evolution. The idea is to treat djot as a family of languages, indexed by years: djot.2024, djot.2025, djot.2026, etc. Yech djot.year is perpetually stable, but there might be slight changes between the years. Then,

$ djot --version 2024 doc.dj

would guarantee exactly the same result irrespective of the version of the djot utility version: djot.js from 2024 and djot.new-lang from 2049 would give exactly the same result if passed --version 2024 argument.

Without an explicit argument

$ djot doc.dj

the semantics would be to use the latest version that the binary knows about. This is of course not absolutely stable, isn't affected by content gravity that much. If you have a pile of djot documents, you probably can guess the right year if you know when the documents were authored. Even if you don't, you can just try different years semi-automatically and see which works.

To ease such investigation/upgrades in practice, there's also

$ djot --migrate-from 2024 --migrate-to 2025 doc.dj

which prints exactly what would be different between two interpretations of the document.

Finally, there's an explicit opt-in for

$ djot --version next doc.dj

which unlocks unstable features. If you use --next, your content might break in non-trivial ways (--version next might be some ill-defined midpoint betwen --version 2026 and --version 2027). We might even gate --next under some build-time flag, so that only developers of djot have access to it, and all the actual users always get a specific djot.year version.

---

I am not sure whether having such "living, year indexed" standard would be a good fit for djot. On the one hand, as a markup language, there's a strong argument that there shouldn't be any flavors or versions, and that it should be just single djot, because content gravity is just too strong.

On the other hand, djot is semi-open language, in a sense that is has a rather wide array of non-primitive constructs (description lists, tables, footnotes, indels), and it is plausible that we'd want to extend that least by at least a couple of more in the coming years.

[Omikhleia]

Very interesting notes, @matklad !

Personally, I'd like to see some sort of roadmap too, perhaps first focused on existing syntax issues and/or minor generalizations:

• Parsing ambiguity: bracketed content followed by footnote? #228 annoys me (parsing ambiguity), as well as URLs with parenthesis break parser djot.js#74 (similar concerns)
• Tight vs. loose list annoys me (I never understood that distinction, but there are several issues on the topic, so it would need clarification, with proper thinking out-of-the-box)
• I wholeheartedly agree with you on Captions #28 and/or Figures #31 (I'm already using a few "generalized" ^-marked captions on other elements than tables in my workflows with a modified Djot parser... so some decision and/or general approach would be neat...). Actually Blockquote attribution / citation #198 probably enters that same category too.
• While I initially was against Proposal: explicit syntax for custom tags #240 (my comments speak for themselves), eh! One is allowed to change their mind. I can see now its potential, if a more "neutral" definition than initially worded was considered (= as semantic markup, not necessarily tied to actual "tags" or "commands" in the output language). So I'd probably agree with it too, after all ;)

Many other proposal are "extensions" (e.g. document transclusion, metadata, etc.), which probably go in another category of changes and might need some more discussions....

But this being said, currently working on a bibliography project, I'd certainly be interested in seeing #32 addressed too, or at least drafted so one can experiment in the right direction.

[faelys]

I mostly share the sentiment, albeit with a different list of pet features (mostly #247 (which for me includes jgm/djot.js#74), and #250, and I wouldn't mind losing the tight/loose distinction), and with a strong dislike of the yearly versioning outside of the source (for a family of languages, I'd rather we go all the way to SemVer with the version number inside the source rather than on the command line; I feel the halfway proposition above gets the worst of both worlds).

On the other hand, the current state of djot feeling unfinished and not changing over the last several months allows me to feel less bad about running my own incompatible faelys-flavored djot fixing all my pet issues (I can lie to myself thinking I'm running future-djot instead of outright dissenting-djot).

[Tamnac]

I'm bullish on the indented sublists issue. It's unintuitive, looks bad and most importantly, people aren't going to follow it! They will end up using the markdown syntax they are familiar with, and we will end up with improper djot everywhere and/or people will modify djot parsers to work with this change and then we end up with a markdown like ecosystem with no universal spec.

There are people who write almost everything through nested lists (look at the popularity of apps like logseq, workflowy, roam research, tana etc.) and this syntax will be an absolute pain.

I think it's worth either

1. Making hard wrappers aware of an edge case, which by the way is already visually intuitive.
2. Breaking Uniformity

To make this happen. Either way I cannot support 1.0 until it is resolved

[jgm]

I've been using djot for almost two years at this point.

Really? It has existed for two years already? Time really flies. I still think of it as a new experimental thing.

I think there are still a number of issues that I'd want to resolve before declaring any kind of stability. I just haven't had bandwidth for this recently. It would be good to draw up a road-map, and I'll try to do that when I have a chance. Off the top of my head, at least these things seem important:

• parsing ambiguities as noted above
• rethink tight/loose lists
• provide a syntax, perhaps list tables, for more complex tables (maybe)
• syntax for citations
• syntax for cross-reference and numbering (maybe)
• figures or captioned things (noted above)
• improved comment syntax
• maybe rethink intraword emphasis and the desideratum of not relying on unicode properties

Perhaps having "releases" (perhaps indexed to year, but there'd be no particular reason to do it that way) would be a way to ensure stability without closing off innovation entirely. It does complicate things. I think the most straightforward way to handle this in JS would be to publish separate packages for parsing each release, and have a small shell executable that loads the appropriate one depending on a --version cli parameter. The alternative would be to add parameters to the compiler that affect parsing depending on the version, but I suspect that will quickly create undesirable complexity. (And note that the types may need to change from version to version.)

[ryanabx]

Just thinking out loud, but if there were different stable versions of djot, perhaps there should be a simple method of identifying the stable version a document is targeting? The two ideas off the top of my head would be either a directive at the top of a document i.e. something like Version: 2024 that identifies the version of the djot document, or having a file extension directive, i.e. my_file.dj24 for a 2024 version.

[bpj]

If any definitely an in-file directive of some kind so that you don't have to change filenames across directories, Makefiles and whatnot.