Graphics support for SOO.refSO3

ngaranis · June 9, 2020, 7:39am

In this topic I will describe my work to have graphics support in the refSO3 ME. It will be based on my work on SO3.

ngaranis · June 9, 2020, 7:39am

During the last few weeks I’ve been working on SOO, with the goal of being able to read the ME’s framebuffer from the agency. Here are steps I’ve done so far:

Created the so3virt_fb driver replacing the pl111 driver. In this driver I’m allocating memory for the framebuffer manually as there is no more VRAM. I’m also writing the value 0x12345678 to the first pixel.
Created the vfb front-end based on vdummy. In the probe function, I retrieve a grantref from the address of the framebuffer memory and I write this grantref to a VBStore property.
Created the vfb back-end based on vdummy. In the probe function, I watch a VBStore property created beforehand. In the watch callback I retrieve the property’s new value which is the grantref passed from the front-end. I map it to allocated memory and then I read the value of the first pixel which is equal to 0x12345678 (value set in the driver).

This is just a proof of concept and there’s still a lot of work to be done .

ngaranis · June 24, 2020, 2:53pm

I managed to reconfigure the PL111 from the vfb back-end when receiving the grantref from the front-end. Reconfiguring is done in the following way:

retrieving the fb_info structure corresponding to the PL111 (an array of such structure are stored in registered_fb defined in fb.h),
changing values in that structure, namely the virtual (fb_info->screen_base) and physical (fb_info->fix.smem_start) addresses,
calling the framebuffer operation fb_set_par on the fb_info structure so that the framebuffer reconfigures itself with the new values.

Then in the ME I can launch the fillfb demo and the content of the screen will be updated.

For some reason, launching the LVGL demo doesn’t work…

ngaranis · June 29, 2020, 8:47am

Just a note on LittlevGL: since version 7 it has been renamed to LVGL, it could be interesting to use this new version in SO3 before the end of the TB (if that isn’t too much work).

I’m also found out that it should be possible to change the resolution at runtime even in version 6. How-to: https://forum.lvgl.io/t/set-resolution-at-runtime/310/4. This could be interesting for the future.

rossierd · June 30, 2020, 5:20am

Yes, great news. It is definitively an upgrade to do.
Furthermore, I also created a new branch related to this development in order to benefit from the very last upgrades in the master branch (moving towards 5.4 Linux kernel).
The new branch is now 17-soo-graphics. I disabled the push/merge actions on the branch 10.
Thanks to make sure that everything has been corrected aligned.

ngaranis · July 4, 2020, 1:52pm

Multiple MEs can now register a framebuffer with the vfb back-end. Also, each time the focused domain changes (with Ctrl + A), the framebuffer associated with it (ME or agency) is displayed.

The front-end framebuffers are stored in an array in the vfb back-end and can be retrieved using two functions:

struct vfb_fb *vfb_get_fefb(domid_t);
domid_t vfb_current_fefb(void);

The vfb_fb structure contains information about the front-end framebuffer such as the domain id, the physical and virtual addresses, the framebuffer size, etc.

The switch between the framebuffers is done in soo/kernel/console/console.c and uses the following function of the back-end:

void vfb_reconfig(domid_t);

Note that for this to work the Linux framebuffer console (fbcon) needs to be deactivated. The reason is that fbcon will set up a thread that blinks the cursor and if the address of the framebuffer is changed, this thread will crash. It’s possible to prevent this crash by unlinking the console from the framebuffer (using unlink_framebuffer) before changing the address of the framebuffer. However, as we do not need fbcon in the first place this was not implemented.

Also, it’s actually fbcon that enables the framebuffer as well as display the Linux logo during boot. When fbcon is disabled, the framebuffer will remain inactive (although it has already been configured by its driver). This is why we need to enable it manually (this is done once in the vfb back-end in vfb_probe).

ngaranis · July 4, 2020, 2:22pm

Small video to show the switch between framebuffers:

ngaranis · July 6, 2020, 8:59am

I’m now passing input events from the Linux input driver (linux/drivers/input/input.c) to the vinput back-end, and from the back-end to the front-end. The vinput front-end will then pass these events to two new SO3 virtual drivers (so3virt_kbd and so3virt_mse). These drivers function just like the PL050 ones, allowing the userspace to query the keyboard and mouse without further changes.

This means that the mouse and keyboard can be used in the LVGL demo app! The back-end takes care of passing the events to the active ME.

Another change I’ve made is to differentiate between keyboard and mouse regarding device classes, so the demo app has been updated to use /dev/mouse0 and /dev/keyboard0 instead of /dev/inputX. This way it’s just easier to select the proper one.

I will now start deploying SOO on the Raspberry Pi 4.

ngaranis · July 18, 2020, 9:18am

I’ve removed references of the display resolution in the source code. When deploying on vexpress, the resolution comes from the DTS. When deploying on the Raspberry Pi 4, it comes from the config.txt file. This also means that it’s no more required to recompile LVGL when the resolution changes.

In standalone SO3, the resolution still comes from the pl111.c driver.

I’ve described how to change the resolution in this wiki page: https://gitlab.com/smartobject/soo/-/wikis/Changing-the-display-resolution

ngaranis · July 18, 2020, 10:03pm

We now have the same behaviour on vexpress and rpi4 regarding the framebuffer. The rpi4 platform demanded more work because the VideoCore (VC) controller does not permit changing the address of the framebuffer. Or maybe it can, but there is almost no documentation about it.

This means we can not tell the VC to display the content of a particular memory address, like we are doing with the PL111. The solution I implemented was to create a thread which copies the content of the active domain (agency or ME) framebuffer into the VC framebuffer. Surely there is a performance impact, but it still works well. Currently this is done every 50ms.

I also had to allocate some memory to store the content of the VC framebuffer just before switching from the agency to an ME, so that it can be restored just after switching back to the agency.

I also did some experiment with the 3 touchscreens. I can display content on all 3 display but I can’t not get the touch input events. For the two HDMI screens, the events are transmitted via USB but I found out that the Pi 4’s USB ports don’t work with SOO. Regarding the official Raspberry touchscreen, I was not able to get the touch events via DSI.