[FEEDBACK] SHOULD, MAY, and RECOMMENDED are confusing and meaningless

[sffc]

If I'm an implementer looking at building an MF2 implementation, I see these types of words throughout the spec:

• MUST
• MUST NOT
• REQUIRED
• SHOULD
• MAY
• RECOMMENDED
• Proposed (not all caps?)

As far as I am concerned, there should be only the following classes of normative text:

1. Things required for conformance
2. Optional extensions, which, if implemented, must have certain behavior

ECMA-262 has Annex B for the optional extensions, and ECMA-402 has a small number of blocks labeled "normative optional" for this. Both of these systems are clear to me.

Scattering words like SHOULD and MAY and RECOMMENDED throughout the text does not help me figure out what I need to do and what I do not need to do. I would much rather have the non-normative items split out into an appendix. Likewise, things that are "Proposed" should not be in the spec at all; they should be in a separate document.

(tagging this as LDML 47 because this makes me confused about what we're actually approving)

[sffc]

This isn't just about implementation complexity.

I also really want to avoid a situation where someone writes a message using some optional extensions, and the message works in ICU4C, and then when they go use an implementation that doesn't implement the same optional extensions (like ICU4X), their message doesn't work. Avoiding this kind of situation is, in my mind, the only job of a specification.

It would make me slightly happier if implementations with optional extensions required the client code to "opt in" to using them. For example, something like

let mf = new Intl.MessageFormat("de", { extensions: ["calendarOption"] })

And then messages with a :datetime calendar="..." option will work only if the stated extension is enabled. If someone tries building an ICU4X formatter with that extension, we'll know ahead of time in the constructor that the extension is not available, instead of waiting to see the message payloads.

[macchiati]

I think there are 5 categories:

1. You must do X
2. You should do X (but it is not required)
3. You should not do X (but it is not forbidden)
4. You must not do X (it is forbidden)
   OR
5. X is completely optional (none of the above)

For 'optional' options or functions we are really talking about case 1, case 2, or case 5. One possibility to alleviate your concern would be to use a different namespace for items that are not required, eg o:calendar=islamic-civil; that would put users on notice that that option may be ignored in some implementations.

---

X is Proposed is orthogonal. It is giving a head's up to readers that is likely that X will be either case 1 or case 2 in a future version. Having it in the same section of the spec is like having a tech-preview section of the spec; it highlights it to the very users we want feedback from.

[aphillips]

I would point out that the capitalized words are called out in the intro, using boilerplate that is super-common in standards. These are the BCP14 (RFC2119) keywords:

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

The words in question are not "scattered". They are deliberately placed. The specification as a whole is normative, except where it clearly states that it is not normative (cf. examples) and the use of normative language is carefully considered.

The default function set introduces keywords REQUIRED and RECOMMENDED for functions and their options and it defines them using 2119-keyword normative language. The names chosen are consistent with their 2119 meaning (REQUIRED == MUST, RECOMMENDED == SHOULD). One could argue that we should use non-2119 words for these to avoid any confusion.

"Proposed" is limited to the function set and :u namespace and is used to indicate maturity. The items in question have been accepted by the WG but are not yet standardized.

Note that post-tech preview (this morning, actually) the function sets and namespaces were separated from the core specification into separate directories and will be subject to their own management process separate from the specification going forwards. Hopefully that relieves some of the tension.

--

(chair hat)

Please do not feel the need to manage release labels. The chair does that as a service 😉

[sffc]

Okay. I prefer the ECMA-402 or ECMA-262 style that avoids the mixing of normative and normative-optional text, but I understand if this is the style you all are using.

Fewer terms where possible seems like a good improvement.

I would prefer terms that don't appear to take a position on whether an implementation is better off doing something or not. If an implementation SHOULD do something, it is not required, but it empowers users of a library to file bug reports saying "why are you not doing this think you SHOULD be doing?" I prefer language such as MAY OPTIONALLY.

I spun my other comment about making the normative-optional extensions more explicit into a different thread, #978

[aphillips]

The 2119 keyword SHOULD is often misunderstood. It is stronger than MAY--it is, in fact, judgemental in the way you don't care for 🙈

To be specific, it means:

This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

In general, the word means that it doesn't break conformance (and is, thus, "optional") to not do it, but implementers are strongly encouraged to treat it like a soft MUST. This is different from the keyword MAY (which we also use).

MAY means something is optional:

This word, or the adjective "OPTIONAL", mean that an item is
truly optional. One vendor may choose to include the item because a
particular marketplace requires it or because the vendor feels that
it enhances the product while another vendor may omit the same item.
An implementation which does not include a particular option MUST be
prepared to interoperate with another implementation which does
include the option, though perhaps with reduced functionality. In the
same vein an implementation which does include a particular option
MUST be prepared to interoperate with another implementation which
does not include the option (except, of course, for the feature the
option provides.)

You're encouraged to file issues in places where the WG has chosen the stronger SHOULD in which you feel the normative strength ought to be MAY or was intended to be MAY. Frequently in our spec, SHOULD is intended to allow non-implementation but to discourage violation. For example, we say that unpaired surrogates SHOULD NOT be used in messages. I just now reviewed the use of SHOULD and don't actually see any (with one exception) that I would personally change.

(The exception is that we break our own rules and use SHOULD in a note in the :string function document):

Users SHOULD encode messages and their parts in Unicode Normalization Form C (NFC) unless there is a very good reason not to.

(Bug to follow)

[aphillips]

I am marking this resolve-candidate not because I feel we have satisfied @sffc, but because I feel we have addressed the high-level comment. The normative language we use should be clear from the discussion above and there has been no response since then. I would suggest that any confusing normativity in the spec be filed as specific issues (or this issue updated with a specific list).

Note that I am not closing this issue unilaterally to allow @sffc to reply and I'm keeping it in v47.

[sffc]

I accept the WG's choice for the language, even if it's not the choice I would have made.

I still need to do an audit of the spec to make suggestions for changing "should" to "optional".

[aphillips]

@sffc Thanks. I will close this issue then.

I still need to do an audit of the spec to make suggestions for changing "should" to "optional".

Note that the 2119 verb-tense synonym for OPTIONAL is MAY. The normative effect of such a change on conformance is usually (but not always!) minimal.