GIGA Shield: Display Libraries

[KurtE]

Not sure if best to open new issues for this or add to others:

As mentioned in the WIKI version 0.3.0 release:

Giga Display shield supported (no wrapper library yet)

Is there example code that demonstrates the usage of the current code?

I see some information up at: https://github.com/zephyrproject-rtos/zephyr/blob/main/boards/shields/arduino_giga_display_shield/doc/index.rst

My quick look at the board data, it appears like maybe it is setup for LVGL?

That index page shows some setup but hoping there is something more Arduino centric.

[mjs513]

Been digging a bit this morning on this and looking at @facchinm giga_display_shield branch (easier to see changes) looks like:

1. Display uses the ST7701 display driver and [#include <zephyr/drivers/display.h>](https://github.com/facchinm/zephyr/blob/a4939539b1f7f7433afca15fd36bb4ffcd7b84a1/include/zephyr/drivers/display.h)

so the wrapper would have to get set up like the camera lib. Guessing here.

EDIT:

1. for me the driver is in: \wsl.localhost\Ubuntu-24.04\home\my_new_zephyr_folder\zephyr\drivers\display
2. display.h is in: \wsl.localhost\Ubuntu-24.04\home\my_new_zephyr_folder\ArduinoCore-zephyr\variants\arduino_giga_r1_stm32h747xx_m7\llext-edk\include\zephyr\include\zephyr\drivers

and would probably have to follow somthing like the sample driver for a Display:
https://github.com/zephyrproject-rtos/zephyr/blob/main/samples/drivers/display/src/main.c

[mjs513]

Right now working on a preliminary wrapper for the display shield. Slow going for me as its a bit different than what I am use to but making progress hopefully by end of day.

[KurtE]

Sounds good, let me know when you have something to play with.

[mjs513]

Not pretty as I don;t really understand how its framebuffer works but have it drawing a rectangle - know its not right yet but at least got some of it working.

Still have to add but running out of steam. Here is the sketch

giga_display-250402a.zip

Am seeing warnings

- Capabilities:
  x_resolution = 480, y_resolution = 800
, supported_pixel_formats = 25
  current_pixel_format = 16, current_orientation = 0
[00:00:05.574,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:05.585,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:05.595,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:05.605,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:05.616,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:05.626,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:05.637,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:05.647,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
this keeps going until image is finished

[mjs513]

@facchinm - @KurtE

Been playing a bit more the driver for the display and am getting slightly confused.

In ST7701.c file I see the api defined as:

static DEVICE_API(display, st7701_api) = {
	.blanking_on = st7701_blanking_on,
	.blanking_off = st7701_blanking_off,
	.set_brightness = st7701_set_brightness,
	.get_capabilities = st7701_get_capabilities,
};

while the display.h has

/**
 * @brief Display driver API
 * API which a display driver should expose
 */
__subsystem struct display_driver_api {
	display_blanking_on_api blanking_on;
	display_blanking_off_api blanking_off;
	display_write_api write;
	display_read_api read;
	display_clear_api clear;
	display_get_framebuffer_api get_framebuffer;
	display_set_brightness_api set_brightness;
	display_set_contrast_api set_contrast;
	display_get_capabilities_api get_capabilities;
	display_set_pixel_format_api set_pixel_format;
	display_set_orientation_api set_orientation;
};

Think missing from the st7701 driver, but that leads me to something else, why is the display even showing anything when I write to it - not sure where its going. Note not sure how init is getting called in the st7701 driver.....

[mjs513]

@facchinm
Need some help need I went ahead and started update display_st7701.c but noticed on rebuild

CMake Warning at /home/my_new_zephyr_folder/zephyr/CMakeLists.txt:1011 (message):
  No SOURCES given to Zephyr library: drivers__display

If I moded the config for the giga to add CONFIG_ST7701=y you can image what happens:

Merged configuration '/home/my_new_zephyr_folder/ArduinoCore-zephyr/loader/../variants/arduino_giga_r1_stm32h747xx_m7/arduino_giga_r1_stm32h747xx_m7.conf'

error: Aborting due to Kconfig warnings

CMake Error at /home/my_new_zephyr_folder/zephyr/cmake/modules/kconfig.cmake:365 (message):
  command failed with return code: 1
Call Stack (most recent call first):
  /home/my_new_zephyr_folder/zephyr/cmake/modules/zephyr_default.cmake:131 (include)
  /home/my_new_zephyr_folder/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:66 (include)
  /home/my_new_zephyr_folder/zephyr/share/zephyr-package/cmake/ZephyrConfig.cmake:97 (include_boilerplate)
  CMakeLists.txt:13 (find_package)


-- Configuring incomplete, errors occurred!
FATAL ERROR: command exited with status 1: /usr/bin/cmake -DWEST_PYTHON=/home/my_new_zephyr_folder/ArduinoCore-zephyr/venv/bin/python3 -B/home/my_new_zephyr_folder/ArduinoCore-zephyr/build/arduino_giga_r1_stm32h747xx_m7 -GNinja -DBOARD=arduino_giga_r1//m7 -S/home/my_new_zephyr_folder/ArduinoCore-zephyr/loader

Looking at the giga overlay further I am not seeing where it is linking back to the display/shield so it knows about the st7701?

So guess goes back to @KurtE's original question - how to tell it to build with the display shield???

UPDATE: Just a quick follow after making some changes to the .c I rebuilt anyway and now even if I revert the changes I am seeing that = zephy_display is not defined anymore - yesterday it was - ARGH::::::

UPDATE2: My fault. Just doing a build.sh for the giga doesn't work have to do a

./extra/build.sh arduino_giga_r1//m7 --shield giga_display_shield

thanks @KurtE for pointing that out.

[mjs513]

@facchinm - @KurtE

Have been playing around with this some more and while I added somethings to the st7701.c driver I am begining to get the feeling that the display config is not using this driver nor display.h:

Hints:

1. Tried to use blanking to turn display on an off:

[00:00:04.510,000] <err> display_stm32_ltdc: There is no panel controller to forward blanking_on call to
[00:00:05.235,000] <err> display_stm32_ltdc: There is no panel controller to forward blanking_on call to
uart:~$

showing using display_stm32_ltdc

2. Tried to change orientation and put some debug printk and even tried log_err's but nothing showed when I ran the sketch - also tried just returning something different than a error message and still got a -134 which should have been what I hard coded it too.

And what I think should be working doesn't.

[KurtE]

@facchinm @mjs513 - Mike, I played around a little with your first sketch/library you posted two days ago.
Wondered if maybe it was more or less working, except that it was not using it's buffer in 16 bit mode, so I modified your sketch slightly

GIGA_Display_first-250404a.zip
I figured that the buffer was like an image data, so I drew it in a 3x5 pattern...

And looks like might be working:

Tomorrow, may play with converting this to work with the Adafruit_gfx library.

@facchinm - I have it setup using your most recent zephyr branch in case it fixed some things

[mjs513]

Thanks @KurtE

You made me get up to play around with your sketch and see what I messed up - still can't rotate screen or do blanking since its using the ltdc driver.

New function added to your sketch - fill screen with a specified color.

#include "Arduino.h"
#include "Arduino_Giga_display.h"

Display display;

uint16_t color_table[] = {
  RGB565_RED, RGB565_MAGENTA, RGB565_YELLOW, RGB565_BLACK, RGB565_GREEN,
  RGB565_WHITE, RGB565_ORANGE, RGB565_GREENYELLOW, RGB565_PINK
};
uint8_t color_index = 0;

uint8_t frameBuffer2[2 * 160 * 160] __attribute__((aligned(32)));
uint32_t sizeof_framebuffer2 = sizeof(frameBuffer2);

void fillScreen(uint16_t color) {
  display.setFrameStatus(false);
  for (int x = 0; x < 480; x += 160) {
    for (int y = 0; y < 800; y += 160) {
      uint16_t *pb = (uint16_t *)frameBuffer2;
      uint32_t count = sizeof_framebuffer2 / 2;
      while(count--) *pb++ = color;
      display.write8(x, y, frameBuffer2);
    }
  }

  display.setFrameStatus(true);
}

void setup() {
  while (!Serial && millis() < 5000)
    ;
  Serial.begin(115200);

  if (!display.begin()) {
    Serial.println("Failed to start display");
  };

  Serial.println("Display configured!!");

  //uint8_t color = 0x11u;
  //memset(frameBuffer2, color, sizeof_framebuffer2);

  //display.setFrameDesc(160, 160, 160, sizeof_framebuffer2);
  display.setFrameStatus(false);

  fillScreen(RGB565_DARKGREY);

  for (int x = 0; x < 320; x += 160) {
    for (int y = 0; y < 480; y += 160) {
      uint16_t *pb = (uint16_t *)frameBuffer2;
      uint32_t count = sizeof_framebuffer2 / 2;
      while(count--) *pb++ = color_table[color_index];
      display.write8(x, y, frameBuffer2);
      color_index++;
      if (color_index == (sizeof(color_table)/sizeof(color_table[0]))) color_index = 0;
    }
  }

  display.setFrameStatus(true);
}

void loop() {
  // put your main code here, to run repeatedly:
}

notice do not have to specify setFrameDesc for some reason.

[mjs513]

@KurtE - @facchinm @pillo79

Been playing with this some more and just added the a getBuffer function to use stm32_ltdc_get_framebuffer to display your checkboard. Still using fillScreen using a stardard buffer approac. And added a fillRect function to just draw a single rectangle after displaying the checkboard.

GIGA_Display_first-250406a.zip

cheers

[KurtE]

@mjs513 @pillo79 @facchinm:

Mike and I have been making some progress with displaying stuff on the GIGA Display shield

So far we have been building sort of libraries into sketch in order to make it easier for now.
Currently we (Mostly MIke) has a wrapper to be able to use Adafruit_GFX. There is lots we can probably do to improve the
performance, but not bad when we tell it to cache up the changes and update the display once...

I also have the DisplayRGB working, which my sketch which I will include below shows...

zephyr_GIGA_shield_touchpaint-250409a.zip

Currently I am playing around with trying to get the touch screen to work...
So I am converting the simple touch paint, that I had earlier converted to run on the MBED version (earlier ones were with many of the Adafruit display libraries),

I see that the PT911 device is defined in the device tree (in the shield/board overlay stuff). I know some stuff is in place as when I touch the screen I see lots of:

[00:00:11.199,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:11.209,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:48.934,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:48.944,000] <wrn> input: Event dropped, queue full, not blocking in syswq.
[00:00:48.955,000] <wrn> input: Event dropped, queue full, not blocking in syswq.

From some of the input examples I have code in the sketch (.cpp file) that does:

static const struct device *const touch_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_touch));
...
static struct {
	size_t x;
	size_t y;
	bool pressed;
} touch_point, touch_point_drawn;

static void touch_event_callback(struct input_event *evt, void *user_data)
{
    printk("touch_event_callback(%p %p): %p %u %u %u %d\n", evt, user_data,
            evt->dev, evt->sync, evt->type, evt->code, evt->value);
}

INPUT_CALLBACK_DEFINE(touch_dev, touch_event_callback, NULL);
,,,
bool Arduino_GigaDisplayTouch::begin() {

	if (!device_is_ready(touch_dev)) {
		Serial.print("Touch Device ");
        Serial.print(touch_dev->name);
        Serial.println(" not found. Aborting.");
		return false;
	}
    return true;
}

I know I need to probably not use these macros to define the callback. Question is how to define an input callback
that we can register at init time? Will also look some more through existing code...

But suggestions would be great

[mjs513]

@KurtE and others
Just to close the loop what has been done so far with the display. Modifing the graphics GFX library and generating a simple wrapper library we tested a few different sketches.

The standard graphics test sketch with the modified libraries embedded in the sketch and buffering when the whole display is finished updated:

Zephyr Giga Display Test!
0
Benchmark                Time (microseconds)
Screen fill              837157
Text                     162244
Lines                    180089
Horiz/Vert Lines         169809
Rectangles (outline)     165466
Rectangles (filled)      607311
Circles (filled)         193723
Circles (outline)        169495
Triangles (outline)      164681
Triangles (filled)       25825
Rounded rects (outline)  166603
Rounded rects (filled)   760426
Done!

Example:

The second complete sketch was modified @KurtE's picture viewer (displays bmps, jpgs, and pngs using @bitbang2 libraries) does work however thing its running out of room since can only the jpegdec and bmps to display. Did have to modify SD library since there is a Freemem function that will cause an error since it is geared to AVR only.
BMP example

JPG example

I am pushing up the graphics test sketches to @KurtE's giga library in sub folder for zephyr giga display
https://github.com/KurtE/Arduino_GIGA-stuff/tree/main/sketches/zephyr_giga_display_examples

[mjs513]

Quick update: See #117 (comment)

[mjs513]

@KurtE - @pillo79 - @facchinm

Started moving our graphics stuff to libraries so easier to test with and hopefully a starting point for when it gets officially incorported.

1. Moved our zephyr changes into the arduino_GigaDisplay_GFX lib: https://github.com/mjs513/Arduino_GigaDisplay_GFX/tree/Arduino_GigaDisplay_GFX_zephyr
2. and the giga display api to Arduino_GigaDisplay - incorporated a new class file: https://github.com/mjs513/Arduino_GigaDisplay/tree/Arduino_GigaDisplay_Zephyr

tested with out graphics test sketch and the touch screen test. As time permits have to test other sketches.

Also was looking at ArduinoGraphics lib but no clue how that gets linked to the display. Going to look more - was late when I started looking at that.

[KurtE]

Thanks @mjs513,
It was also on my list of things I wanted to do! Glad you beat me to it!

@iabdalkader @pillo79 @facchinm

I synced up and then updated our simple touch paint... Ran into a few issues doing it,
like selecting tab going to the delete item, which comes up with a dialog which says something like moving to trashcan... But when it is done tab is still there... Exited IDE restarted and they were gone... Different topic for different ...

Right now I am thinking about creating a PR with our current Touch stuff, basically I added a Callback in the loader fixups.c

#if defined(CONFIG_BOARD_ARDUINO_GIGA_R1) && defined(CONFIG_VIDEO)
#include <zephyr/kernel.h>
#include <zephyr/device.h>
#include <zephyr/drivers/clock_control.h>
#include <zephyr/logging/log.h>

#include <zephyr/input/input.h>

// experiment to try to capture touch screen events
typedef struct {
	int32_t x;
	int32_t y;
	int32_t pressed;
} touch_point_t;

touch_point_t last_touch_point;

static struct k_sem touch_event_sync;

bool getVideoTouchEvent(touch_point_t *tp, k_timeout_t timeout) {
	if (k_sem_take(&touch_event_sync, timeout) != 0) return false;
	// BUGBUG: should probably put stuff in to return only
	// data from whole event, but first see if anything works
	memcpy(tp, &last_touch_point, sizeof(touch_point_t));
	return true;
}


void touch_event_callback(struct input_event *evt, void *user_data)
{
    //printk("touch_event_callback(%p %p): %p %u %u %u %d\n", evt, user_data,
    //        evt->dev, evt->sync, evt->type, evt->code, evt->value);
	if (evt->code == INPUT_ABS_X) {
		last_touch_point.x = evt->value;
	}
	if (evt->code == INPUT_ABS_Y) {
		last_touch_point.y = evt->value;
	}
	if (evt->code == INPUT_BTN_TOUCH) {
		last_touch_point.pressed = evt->value;
	}
	if (evt->sync) {
		k_sem_give(&touch_event_sync);
	}
}
static const struct device *const touch_dev = DEVICE_DT_GET(DT_CHOSEN(zephyr_touch));
INPUT_CALLBACK_DEFINE(touch_dev, touch_event_callback, NULL);


int camera_ext_clock_enable(void)
{
	int ret;
	uint32_t rate;

	// Hack in init semaphore for touch events
	k_sem_init(&touch_event_sync, 0, 1);
...

And an entry in the exports:
#if defined(CONFIG_BOARD_ARDUINO_GIGA_R1) && defined(CONFIG_VIDEO)
FORCE_EXPORT_SYM(getVideoTouchEvent)
#endif

Note; This is probably not a complete solution yet as it only supports one touch point, where as the device I believe
can support up to 5, and when > 1 supports gestures. But none of this is currently enabled within zephyr. Also
I don't know how this might interact with using LVGL with it.

At some point probably need try out LVGL here, but not sure how one would do that without updating the overlay
and config files... But that is true of many other things here, like how much stuff will be included on a GIGA build, especially
if for example if I don't even have a display shield...