Can i limit the amount of Channels that i receive thru the API ?

[devarishi7]

Ok a little bit of background, first of all i somehow don't think so or i wouldn't be asking, but i might have missed something.

The reason i want to do this is as follows. I have a sizeable program that in essence is sending WS2812b signal and the signal is generated based on incoming data, which can be DMX or Art-Net and there is a standalone function included. All great all works without issue, The DMX too thanx to this library. Now in the Program there is also the possibility to record the output on an SD-card and play it back. For Art-Net & standalone this all works like a breeze after some tweaking here and there. The WS2812 signal is sent fully in the background using I2S (Neopixelbus) with 4 universes of output my maximum framerate is close to 50Hz which is easily enough time to write a frame to the SD-card and receive Art-Net , but when i write to the SD-card, the DMX frame gets corrupted, basically i think a break is recognized that isn't there, but it could be something else, so anyway if i disable the dmx-driver during the SD-card write stage all is still good, but then my reception framerate drops by half, since the first brake passes before i switch it on again, and so i loose a whole frame. A DMX frames takes roughly eh.. (1 / (250.000 / 12) ) * 513 + break time, say 25ms , 40Hz that looks fine, 20Hz is not to good. (actually i think may even be losing 2 frames, can't really be sure of that though) Now since usually when dealing with DMX i will not be using all channels, and when recording that situation is even less likely, i figure i could do it the same way when i was Using an AVR & bit-banging. Wait for the break, read the channels i want, start doing what i want to do and if i'm done in time for the next break then all is good. But this system requires the reading of a whole frame, so i was wondering can i just reduce this ?
driver->rx_size = DMX_PACKET_SIZE_MAX;

[someweisguy]

To answer your question: no, there is currently no way to reduce the size of the DMX frame that is received. The DMX driver will automatically receive the entire DMX frame whether it is 1 byte long or 513.

Is it possible to store an array of partial DMX packets in RAM and then, after the recording is finished, disable the DMX driver and write the arrays to flash? It would use a lot of RAM but it might be possible depending on how many DMX slots you are recording. For example if you are recording 32 DMX addresses for 10 seconds at 44Hz, this would use ~14KB. You should be able to heap allocate or statically initialize this memory.

Are you using ESP-IDF or Arduino? If you are using ESP-IDF, you should be able to configure your ESP32 to place the DMX interrupts in IRAM. When interrupts are in IRAM, DMX will continue to be received even when writing to flash. Unfortunately, this option is not available on Arduino due to how Arduino configures the ESP32.

This sounds like a very interesting use-case. I am interested in supporting it if neither of these solutions work. Let me know your thoughts! :)

[devarishi7]

The DMX driver will automatically receive the entire DMX frame whether it is 1 byte long or 513.

But how does it work ? when i call dmx_receive() and it wait for a full packet or until the timeout, (or more correctly if a frame has been received since it was last read since it was enabled i guess) but what is used to end that process, a simple counter ? (to 513) or are there other conditions to be met. If it's just a counter then a simple modification would do the trick.
I can't quite see what is going on, this may have something to do with the running of the tasks on the other core (that is what is going on isn't ?)

It isn't really possible to store received packets in RAM unfortunately, since that would limit the length of the recording, which ever way you look at it. Even SPIFFS is to limited for this, only an SD-card will do the trick. (Seconds or minutes of stream are not the goal, hours is what we want) It is a bit compounded by the fact that i actually record the output and un-compressed. The idea was to just simply record the output stream regardless of whether that is actually changing. As i said, when Art-Net is the source, this works just fine. The time that it takes to store a frame of 2Kb is significantly less than 20ms, even at moderate speed, but once the SPI speed is at 32MHz, which is fine for the SD, there is no problem.

Are you using ESP-IDF or Arduino? If you are using ESP-IDF, you should be able to configure your ESP32 to place the DMX interrupts in IRAM. When interrupts are in IRAM, DMX will continue to be received even when writing to flash. Unfortunately, this option is not available on Arduino due to how Arduino configures the ESP32.

I am using the Arduino core. I thought i could place any ISR in IRAM, is this not true ? could you elaborate on this, The ESP32 is a rather complex MCU with the 2 cores and all those peripherals, it is a little beyond me at times.

So if there is a way to trigger the frame-complete flag at a point where, say only 256 slots have been received (or even less) and processing including writing to the SD, can continue and can be completed so that before the next break arrives, the driver can be re-enabled, that would be the goal.
The issue with DMX is somehow always to receive the break, UART reception can continue automatically after that usually, nothing much will disturb it. The giant FIFO of the ESP32 seems to complicate matters a bit for simpler solutions (as it does on an ES8266) . Still it should not be more complex than to test the state of a GPIO until the minimum break time has elapsed and wait for whatever comes after that, and i have had success with that on an ESP8266 (mind you only on the swap-uart-pins this works for me, where another method works on the default uart0 pins, and for no reason that i can find)
I was hoping that driver->rx_size was the comparing value for the counter, isn't it ?
Anyway, there are more avenues to explore.

[someweisguy]

But how does it work ? when i call dmx_receive() and it wait for a full packet or until the timeout, (or more correctly if a frame has been received since it was last read since it was enabled i guess) but what is used to end that process, a simple counter ?

Yes, it's a simple counter. Essentially, when the DMX break is received, the counter resets to 0. When a slot is received, the counter increments and this is how the DMX driver is able to keep track of which slot is received. By default, the DMX driver expects to receive 513 slots. If fewer than 513 slots are received, some logic will fire when the driver receives the next DMX break telling it to update the number of slots to expect to receive.

I am using the Arduino core. I thought i could place any ISR in IRAM, is this not true ? could you elaborate on this, The ESP32 is a rather complex MCU with the 2 cores and all those peripherals, it is a little beyond me at times.

Yes, you can place any function within IRAM. However, if your IRAM function calls non-IRAM functions your ESP32 can crash. In ESP-IDF v5, the hardware timer API was updated to no longer be in IRAM by default. Because the DMX driver ISR calls hardware timer functions, this means that if your hardware timer is not in IRAM, the DMX functions cannot be placed in IRAM. When you use ESP-IDF, this library will force the hardware timer to be in IRAM. But due to the way that Arduino configures the ESP32, it is impossible to place the hardware timer in IRAM. The next version of Arduino-ESP32 should be released soon. When it is, I'll be doing some testing to see if these functions are in IRAM.

So if there is a way to trigger the frame-complete flag at a point where, say only 256 slots

Yes, I think this should be pretty easy. In fact, I've been working on some features that would play very nicely with this feature. I have a bit of a lengthy to-do list at the moment, but I am very interested in getting this implemented.

[devarishi7]

So anyway i've done some modification, i added to hal.h
bool dmx_driver_enable_rxsize(dmx_port_t dmx_num, uint16_t rx_size);
and then i modified hal.c to accommodate for that.
`
bool dmx_driver_enable(dmx_port_t dmx_num) {
return dmx_driver_enable_rxsize(dmx_num, DMX_PACKET_SIZE_MAX);
}

bool dmx_driver_enable_rxsize(dmx_port_t dmx_num, uint16_t rx_size) {
DMX_CHECK(dmx_num < DMX_NUM_MAX, false, "dmx_num error");
DMX_CHECK(dmx_driver_is_installed(dmx_num), false, "driver is not installed");
DMX_CHECK(!dmx_driver_is_enabled(dmx_num), false,
"driver is already enabled");

dmx_driver_t *const driver = dmx_driver[dmx_num];
dmx_uart_handle_t uart = dmx_context[dmx_num].uart;

driver->rx_size = rx_size;

// Initialize driver flags and reenable interrupts
taskENTER_CRITICAL(DMX_SPINLOCK(dmx_num));
driver->head = -1; // Wait for DMX break before reading data
driver->flags |= (DMX_FLAGS_DRIVER_IS_ENABLED | DMX_FLAGS_DRIVER_IS_IDLE);
driver->flags &= ~(DMX_FLAGS_DRIVER_IS_IN_BREAK | DMX_FLAGS_DRIVER_HAS_DATA);
dmx_uart_rxfifo_reset(uart);
dmx_uart_txfifo_reset(uart);
dmx_uart_enable_interrupt(uart, DMX_INTR_RX_ALL);
dmx_uart_clear_interrupt(uart, DMX_INTR_RX_ALL);
taskEXIT_CRITICAL(DMX_SPINLOCK(dmx_num));

return true;
}
`
and that actually works as far as i can tell. What do you think ?

[someweisguy]

This looks about right. There is some logic in the DMX ISR that detects and updates the rx_size that would need to be updated as well.

[devarishi7]

Ah, you could have pointed it out, but i think i've found it.
if (driver->head > driver->rx_size) { driver->rx_size = driver->head; // Update expected rx_size }
Just something else, it seems that what i have stored locally as your library differs from what is here on github, it doesn't quite make sense.

[someweisguy]

Ah, you could have pointed it out, but i think i've found it.
if (driver->head > driver->rx_size) { driver->rx_size = driver->head; // Update expected rx_size }

You're right, I could have shown you. Sorry about that. You have the correct line!

Just something else, it seems that what i have stored locally as your library differs from what is here on github, it doesn't quite make sense.

Are you looking at the correct tagged version? I have made a few updates since releasing v3.1.0. And I am currently working on more updates in the dev branch. Here is the tagged version.

[devarishi7]

Hmm well whichever way i do it, the only way to get (near) 40Hz reception is by leaving the driver enabled, which causes a conflict with some of the other stuff, also callbacks to from the WiFi webserver (probably the WiFi causing it) although that would still be OK, i can just skip the received frame if i have to. The code is just to complex for me to fiddle with i guess. With other methods i have tried i find that disabling the UART RX interrupt is not as straight forward on an ESP32 as it is on an ESP8266 (switch the pinMode) or an AVR (disable the interrupt) I am now looking at using an extra MAX485 transceiver that is used to find the break(i can mange to find the break just fine) and enable the one connecting to the UART. This is completely outside of your library of course, so unless you have other suggestions on dealing with the issue i am having i shall not bother you with that.

[devarishi7]

Actually come to think of it, i am only getting the 40Hz reception if i do a very sketch, and even then it is blocking in nature.
Can you explain to me why i actually need to call
if (dmx_receive(dmxPort, &packet,DMX_TIMEOUT_TICK)) {
before i can start on a packet. The interrupts are working in the background, so reception should be continuous and so why do i need to wait for the most recent packet to be ready ? and why would i have to wait for that frame to be complete ?
It just doesn't quite make sense to me if you have the interrupts enabled. The packets are coming into the rx-buffer, the transfer from the buffer into a packet should be automated, and all i should need to do is check if a new packet is available, so that it is an asynchronous reception.
Now if i do any sort of processing i end up with 20Hz already.
I am doing something wrong ? I haven't actually checked to see if i get packet errors i guess. I just ping a led every time i successfully receive a packet, and from the blinking i deduce the framerate (i did measure the 40Hz to confirm the blinking speed is what i thought it was)
With enabling and disabling, i get something in the order of 13Hz (i lose 2 frames for every 1 received), and well, that is just simply not good enough.

[someweisguy]

Can you explain to me why i actually need to call if (dmx_receive(dmxPort, &packet,DMX_TIMEOUT_TICK)) { before i can start on a packet

Great question - you actually don't need to call dmx_receive() if you do not wish to. Its main purpose is to block the task until a DMX packet has been fully received. In v3.1, it also handles the RDM response mechanism. In the next release of esp_dmx, I am planning on breaking out the RDM response mechanism into a separate function so that you will not need to call dmx_recieve() to handle RDM.

The simplest way I know to disable the UART interrupts on ESP32 is below.

#include "dmx/hal/uart.h"
#include "driver/uart.h"

uart_disable_intr_mask(dmx_num, DMX_INTR_RX_ALL);  // Disable UART

uart_enable_intr_mask(dmx_num, DMX_INTR_RX_ALL);  // Enable UART

I am not understanding how this fixes your issue, but hopefully it helps.

One last check I recommend is to take a look at the value of DMX_INTR_FLAGS_DEFAULT. This can be done by printing it to the Serial Monitor. Hopefully, it evaluates to 1024. If it evaluates to 0, try going to src/dmx/hal.h and delete lines 56 and 64 through 69. That section looks like this:

#if defined(CONFIG_DMX_ISR_IN_IRAM) || ESP_IDF_VERSION_MAJOR < 5
/** @brief This macro sets certain functions used within DMX interrupt handlers
 * to be placed within IRAM. The current hardware configuration of this device
 * places the DMX driver functions within IRAM. */
#define DMX_ISR_ATTR IRAM_ATTR
/** @brief This macro is used to conditionally compile certain parts of code
 * depending on whether or not the DMX driver is within IRAM.*/
#define DMX_ISR_IN_IRAM
#else
/** @brief This macro sets certain functions used within DMX interrupt handlers
 * to be placed within IRAM. Due to the current hardware configuration of this
 * device, the DMX driver is not currently placed within IRAM. */
#define DMX_ISR_ATTR
#endif

And you want it to look like this:

/** @brief This macro sets certain functions used within DMX interrupt handlers
 * to be placed within IRAM. The current hardware configuration of this device
 * places the DMX driver functions within IRAM. */
#define DMX_ISR_ATTR IRAM_ATTR
/** @brief This macro is used to conditionally compile certain parts of code
 * depending on whether or not the DMX driver is within IRAM.*/
#define DMX_ISR_IN_IRAM

After that, recompile your code and check the value of DMX_INTR_FLAGS_DEFAULT.

This is the section of code that compiles the DMX driver into IRAM. If the value of DMX_INTR_FLAGS_DEFAULT is 1024, that means that the DMX driver is being put into IRAM.

This may be a bit of a goose chase, but it is sort of my last ditch effort to assist with this.

[devarishi7]

Monitor. Hopefully, it evaluates to 1024
Yes it does.

[devarishi7]

The simplest way I know to disable the UART interrupts on ESP32 is below.

#include "dmx/hal/uart.h"
#include "driver/uart.h"

uart_disable_intr_mask(dmx_num, DMX_INTR_RX_ALL); // Disable UART

uart_enable_intr_mask(dmx_num, DMX_INTR_RX_ALL); // Enable UART
I am not understanding how this fixes your issue, but hopefully it helps.

One thing is that i can disable the interrupt rather than disable the whole driver, that might help.

Another idea is that while the processing is waiting for the break,
The rx-fifo & rx buffer won't fill up with data that would have to be removed from it once the break is found. I am not trying to fiddle with your code anymore though, it is just a bit to complicated for me.

Forcing all of the data from the Rx FIFO into the RX buffer is another thing i would like to just be able to do with just a simple line of code as is the direct access to the RX buffer for reading ideally.

Setting the RX-FIFO size to '1' gives me control over it, but that is just a tad inefficient particularly on a 32-bit processor.
With an external device signalling the break to the ESP thru a pin-change interrupt, i have something that works even if it is just by using hardwareSerial.h, but it is gulping up processor time and i can not really implement it into a larger program.

[devarishi7]

Yeah thanks for that, i am of course thinking and pondering,

you actually don't need to call dmx_receive() if you do not wish to. Its main purpose is to block the task until a DMX packet has been fully received

Ok, so does that mean that if i do call it, it always waits for the 'next' complete packet or if a complete packet has arrived already it will just be a dummy call ?

I am actually suspecting that the use of the hardware timers is an (the) issue, The I2S method for the ws2812b signal may use the same timers or prescaler, i am not sure, and this may cause frame errors and so forth, i can't quite tell what is going on, but i am going to explore another avenue, where i will try do a simple reception of Serial, and use an external timer to find the break. Basically i am adding an ATtiny13 to listen to pin state changes and change the state of it's output pin when no change has been observed for at least 80us, Hooking it's output pin to an ESP32 input pin should remove the need for a hardware timer altogether if i am not mistaken ? (except maybe for the reception timeout, but that seems less important to me)

Hardware timer are a resource that is often used by all sorts of things, and if they can not be placed in IRAM that causes issues of course.

I was playing with the ESP32TimerInterrupt.h library and found that even calls to it conflicted with analogWrite(), probably because of the IRAM thing, but i can't be sure, still taking that part out of the equation should have a positive effect.

To implement that in your library is a fair way beyond my skill level. Messing with someone else's code is always more tricky, more so when it involves fairly low-level programming and with fairly extensive features.

I recon i should probably be able to get a basic version working, but of course it would be better if you could support the idea.

[someweisguy]

Ok, so does that mean that if i do call it, it always waits for the 'next' complete packet or if a complete packet has arrived already it will just be a dummy call ?

If you call the function and you have already received a new DMX packet, dmx_receive() will return immediately. There is a flag in the DMX driver that is set in the ISR when when a new packet is received. dmx_receive() looks for this flag, clears it, and returns. If the flag isn't set, it will block until it receives a FreeRTOS task notification, clear the flag, and return.

[devarishi7]

Ok then i must be getting packet errors, otherwise the lower framerate doesn't make sense.

[devarishi7]

So anyway i got the break detection to work on ATtiny and i do have some basic sketch that does receive the data accurately,
The ATtiny sketch is quite straightforward.

#include <avr/io.h>
#include <avr/interrupt.h>

#define MICS 88
#define CHUNKS 88
#define CLICKS (MICS / CHUNKS)


volatile uint8_t count = 0;

ISR (INT0_vect) {
  count = 0;
}

void setup() {
  noInterrupts();
  DDRB |= (1 << PB0);  // PB0 OUTPUT
  DDRB &= ~(1 << PB1); // PB1 INPUT
  GIMSK |= (1 << INT0); // Enable INT0 interrupt
  //  ISCO1 ISCO0
  //    0     0     The LOW level of INT0 calls ISR
  //    0     1     any logical change
  //    1     0     Falling edge
  //    1     1     Rising edge
  MCUCR |= (1 << ISC01);  //   set for falling edge
  MCUCR &= ~(1 << ISC00);
  interrupts();
}

void loop() {
  static bool pinhigh = true;

  while (count < CLICKS) {
    count++;
    delayMicroseconds(CHUNKS);  
  }
  pinhigh = !pinhigh;
  if (pinhigh) {
    PORTB |= (1 << PB0);
  }
  else {
    PORTB &= ~(1 << PB0);
  }
  while (count) {
    delayMicroseconds(1);
  }
}

and if i hook it to an Input pin, pin change interrupts will signal the ESP that the break is found. As said i have a basic sketch working, but i don't really know how to manipulate the UART, it's fifo and it's buffer, I gonna ask on the forum a bit, not having direct access to the RX buffer is rather disturbing, Normally speaking setting the head and the tail to the same value flushes it, which is of course what i want to do, although, actually reading from the FiFo, well i set the FIFO to 1 byte to circumvent these issues, bit that is not very efficient of course. So anyway you are much better at this and i am wondering if you are willing to support this method with external break-detection ?
I mean that does bypass the timer issue doesn't it ?!

[devarishi7]

Anyway did another little test and found that with these settings, the break detection fires for anything over 88us

#define MICS 45
#define CHUNKS 1
#define CLICKS (MICS / CHUNKS)

this allows for the extra cycles of the counter. and the minimum delay makes sure it fires with something like a few us accuracy.
It is anyway chip specific on a MCU that uses an internal resonator, but i guess it doesn't even need to be that accurate.
I was considering to tweak the delay with specific amount of NOPs but it seems a bit of overkill.
Still trying to see if that can be done by setting F_CPU since as far as i know that is used to calculate the amount of NOPs for the delayMicroseconds()