Skip to content

Home

Jump to bottom Edit New page
mpvader edited this page Apr 2, 2015 · 19 revisions

CCGX Messaging

To discuss, you are welcome to edit this page, and just add content at the bottom, or add new pages to the wiki.

Existing situation

  • Internally, the CCGX uses D-Bus to share data. See http://www.victronenergy.com/live/open_source:ccgx:start for an overview.
  • To send data to the VRM database, it uses HTTP POST requests to the VRM Database API. A complete snapshot of all running values (voltages/currents/etc) are sent on the user-configured log interval. Configuration values (firmware version/productid) and historic values (BMV-Lowest voltage/BMV-highest voltage/etc) are only sent when changed. And on top of that, certain events cause it to send an extra snapshot of the running values. For example when a Multi changes from Absorption to Invert. This whole mechanism (vrmlogger -> vrm-log-backend) is one way.
  • Lately, some use cases made us implement bidirectional communication:
    • Remotely perform firmware updates: sit behind your PC in Guatemala, and update an MPPT that is connected to a CCGX in the Netherlands.
    • Have live updates in a webbrowser or the VRM App. Since the HTTP POST requests have a minimum interval of one minute, it will never look real time to a user. Pubsub is perfect to have the webbrowser/app subscribe to the right topic(s), and have almost-live updates.
    • Remotely configure devices.
  • Obviously a direct connection between the PC in Guatemala and the CCGX in NL is difficult, so a middle man is necessary. We chose publish/subscribe with a broker in the middle. As pubsub protocol/broker we chose a solution that we were familiar with from an earlier project: Pubnub. Pubnub is a commercial, hosted Publish/subscribe broker running its own protocol. See www.pubnub.com for more information.

Intermezzo - pubsub in a few words

SUBSCRIBE, UNSUBSCRIBE and PUBLISH implement the Publish/Subscribe messaging paradigm where (citing Wikipedia) senders (publishers) are not programmed to send their messages to specific receivers (subscribers). Rather, published messages are characterized into channels, without knowledge of what (if any) subscribers there may be. Subscribers express interest in one or more channels, and only receive messages that are of interest, without knowledge of what (if any) publishers there are. This decoupling of publishers and subscribers can allow for greater scalability and a more dynamic network

From http://redis.io/topics/pubsub

Possible improvements on the existing situation

  • The HTTP post requests can generate quite high data consumption, which is a problem for users in remote areas, using satellite uplinks for example. And, when moving over to HTTPS, data consumption will increase even further.
  • Pubnub doesn't really fit the profile:
    • it is commercial, not free. Does not match our CCGX / open source / community philosophy.
    • it is hosted: cannot be used on networks that are not connected to internet (intranet-use-case). Image for example a boat, with phones/tablets and a CCGX
  • Security handling in our Pubnub implementation has been done in the client. In hindsight a bad choice. It would be much better/cleaner to use the broker for authentication, topic-ACLs, etc.

Optional solutions - protocols

My idea is to:

  • stick with pubsub for the bidirectional communication, but then not Pubnub.

  • include sending data to the database to it, and phase the vrmlogger out.

  • run a broker locally on the ccgx to cater for the intranet-use-case.

  • bridge this local broker it to a public broker, either cloud hosted or hosted ourselves.

  • MQTT protocol, with brokers such as Mosquitto, eMQTT, HyveMQ and many others

  • WAMP protocol, with brokers such as crossbar.io as a possible solution

Topic structure

/[vrmportalid]/[productgroup]/[deviceinstance]/[dbuspath]

For example:

/847e40669664/vebus/0/Ac/ActiveIn/Power
/847e40669664/vebus/0/Ac/ActiveIn/L1/Voltage
/847e40669664/vebus/0/Ac/ActiveIn/L1/Current
/847e40669664/vebus/0/Ac/ActiveIn/L1/Power
/847e40669664/vebus/0/Ac/ActiveIn/L1/Frequency
etc. etc.
/847e40669664/solarcharger/257/Dc/0/Voltage
/847e40669664/solarcharger/257/Dc/0/Current
etc. etc.
/847e40669664/solarcharger/258/Dc/0/Voltage
etc.

the disadvantage will be that every value change creates a single MQTT message. By grouping all together, or in a few groups, we can send the values by JSON, with a settable maximum rate. For example once or twice a second when someone is looking on his phone or webbrowser, and slightly lower when no-one is looking and the only goal of sending it is to store it in the VRM database.

Grouping example 1:

/847e40669664/LiveFeed

Grouping example 2:

/847e40669664/vebus/0/Data
/847e40669664/solarcharger/257/Data
/847e40669664/solarcharger/258/Data

Grouping example 3:

/847e40669664/vebus/0/Ac
/847e40669664/vebus/0/Dc
/847e40669664/solarcharger/257/Dc
/847e40669664/solarcharger/258/Dc

Etc, many options are possible.

Disadvantages/advantages of using one group vs several groups?

QOS

Use QOS = 0 for frequently updated values, and a higher one for state changes and such.

Security

Several topics:

  • Authentication: built in into MQTT, use VRM Portal id (how about intranet-use-case)
  • Authorisation: use access rights as defined in VRM Portal, to set ACLs on topics. MQTT allows auth-plugins for this.
  • Encryption: several options are built into MQTT

Authorisation

Levels in mind are:

  • View : view data only
  • Operator : view and operate (turn on/off etc) <- or is this too fine grained? At first at least leave space for it.
  • Owner : view and do basic control / configuration
  • Dealer : do firmware updates / change configuration etc.
  • Superuser : <- not sure if this is needed, at least leave space.

Configuration is done in the VRM Database. TODO: how to do this on the intranet-use-case?

Interesting reads

Discussion

feel welcome to add your input here.

Add a custom footer
Add a custom sidebar
Clone this wiki locally