JSON parser with comment

[redboltz]

The broker part of async_mqtt uses Boost.PropertyTree JSON parser.

async_mqtt/include/async_mqtt/broker/security.hpp

Line 15 in 8d3ef84

#include <boost/property_tree/json_parser.hpp>

It is for Access Control List.

JSON doesn't have a way to write comments. So async_mqtt add comments functionality. It works as preprosessor of Boost.PropertyTree JSON parser:

async_mqtt/include/async_mqtt/broker/security.hpp

Lines 48 to 69 in 8d3ef84

	/** Remove comments from a JSON file (comments start with # and are not inside ' ' or " ") */
	inline std::string json_remove_comments(std::istream& input) {
	bool inside_comment = false;
	bool inside_single_quote = false;
	bool inside_double_quote = false;
	std::ostringstream result;
	while (true) {
	char c;
	if (input.get(c).eof()) break;
	if (!inside_double_quote && !inside_single_quote && c == '#') inside_comment = true;
	if (!inside_comment && !inside_double_quote && c == '\'') inside_single_quote = !inside_single_quote;
	if (!inside_comment && !inside_single_quote && c == '"') inside_double_quote = !inside_double_quote;
	if (c == '\n') inside_comment = false;
	if (!inside_comment) result << c;
	}
	return result.str();
	}

I guess that there is more elegant way.

[redboltz]

This is only used by the broker. It is not the part of library. Just a tool.

[redboltz]

NOTE: broker part is NOT a part of public APIs. It is tool expansion for broker.

[dmazzella]

what do you think about more "commons" way to write comments in json? Like JSON5?

example comments for auth.json

// Single and multi-line comments are allowed
{
    // Grant users to connect the broker
    "authentication": [
        /* 
           For casual use
           On product services, they should be removed
        */
        {
            // Handles all users that login without username / password
            "name": "anonymous",
            "method": "anonymous"
        }
        ,
        {
            // Handles all users that are not authenticated (non-existing user, invalid password)
            "name": "unauthenticated",
            "method": "unauthenticated"
        }
        ,
        // Examples of actual usecases
        {
            /* Authenticates user by password using sha256 hash and specified salt for password sha256(salt + password) */
            "name": "u1",
            "method": "sha256",
            "salt": "salt",
            "digest": "38ea2e5e88fcd692fe177c6cada15e9b2db6e70bee0a0d6678c8d3b2a9aae2ad"
        }
        ,
        {
            // Authenticates user by client certificate
            "name": "u2",
            "method": "client_cert"
        }
        ,
        {
            // Authenticates user by plain password
            "name": "u3",
            "method": "plain_password",
            "password": "insecure_plain_password_for_test"
        }
    ]
    ,
    // Associate users into group
    "group": [
        {
            /* Users can be associated to group, group name starts with @ */
            "name": "@casual",
            "members": ["anonymous", "unauthenticated"]
        }
        ,
        {
            /* Users can be associated to group, group name starts with @ */
            "name": "@g1",
            "members": ["u1", "u2"]
        }
    ]
    ,
    /* Grant users an groups to access topics */
    "authorization": [
        {
            "topic": "#",
            "allow": {
                "sub": ["@casual"],
                "pub": ["@casual"]
            }
        }
        ,
        {
            /* Specified users and groups are denied to publish on this topic */
            "topic": "#",
            "deny": { "pub": ["@g1"] }
        }
        ,
        {
            // Specified users and groups are denied to subscribe on this topic"
            "topic": "#",
            "deny": { "sub": ["@g1"] }
        }
        ,
        {
            // Specified users and groups are allowed to subscribe and publish on this topic"
            "topic": "sub/#",
            "allow": {
                "sub": ["@g1"],
                "pub": ["@g1"]
            }
        }
        ,
        {
            // Specified users and groups are allowed to subscribe and publish on this topic"
            "topic": "sub/u2reject",
            "deny": {
                "sub": ["u2"],
                "pub": ["u2"]
            }
        }
    ]
}

this is the function for remove comments I have wrote, what do you think?

/** Remove comments from a JSON5 file (Single and multi-line comments are allowed) */
inline std::string json_remove_comments(std::istream& input) {
    bool inside_comment = false;
    bool inside_single_quote = false;
    bool inside_double_quote = false;
    bool inside_block_comment = false;

    std::ostringstream result;

    while (true) {
        char c;
        if (input.get(c).eof()) break;

        if (!inside_comment && !inside_block_comment && !inside_double_quote && !inside_single_quote && c == '/') {
            char next;
            if (input.get(next)) {
                if (next == '/') {
                    inside_comment = true;
                } else if (next == '*') {
                    inside_block_comment = true;
                } else {
                    result << c;
                    result << next;
                    continue;
                }
            }
        }

        if (inside_block_comment) {
            if (c == '*' && input.peek() == '/') {
                input.get(c); // consume '/'
                inside_block_comment = false;
            }
            continue;
        }

        if (c == '\n') inside_comment = false;

        if (!inside_comment && !inside_block_comment) {
            if (!inside_double_quote && !inside_single_quote && c == '\'') inside_single_quote = !inside_single_quote;
            if (!inside_comment && !inside_single_quote && c == '"') inside_double_quote = !inside_double_quote;
            result << c;
        }
    }

    return result.str();
}

[redboltz]

@dmazzella it looks good to me.

By the way, is JSON5 the most promising JSON extension that supports comments?

[dmazzella]

Json5 extension support more features that are not necessary, usefull in your project, this is a my point of view.
Relaxed syntax like single line (// ...) and multiple line (/* ... */) C-style comments are already used in rapidjson, nlohmann, jsoncons and others

[redboltz]

Thank you. I agree that the C/C++ style comment is better than the current one. Could you please create a pull request to apply your approach? If you'd prefer, I can also take care of it—just let me know what works best for you.

NOTE:

• implementation
  • https://github.com/redboltz/async_mqtt/blob/main/tool/include/broker/security.hpp#L50-L70
• json files
  • https://github.com/redboltz/async_mqtt/blob/main/tool/auth.json
  • https://github.com/redboltz/async_mqtt/blob/main/docker/conf/auth.json
  • https://github.com/redboltz/async_mqtt/blob/main/test/system/st_auth.json
• coding rules
  • https://github.com/redboltz/async_mqtt/blob/main/doc/modules/ROOT/pages/codingrule.adoc
    Just I granced your code, a line break is requred before else.

[dmazzella]

created PR #394 with your note