The second step was to find the correct controller to use (for the VExpress board that we emulated with QEMU). I first discovered the HDLCD controller which looked promising but isn’t actually emulated by QEMU (see hw/arm/vexpress.c). This controller is located on the daugtherboard and is a more performant version of the CLCD controller located on the motherboard. So, the CLCD is the one we’ll have to use for the moment.
ARM documentation:
The third step was to initialise in SO3 the PL111 (CLCD) controller emulated by QEMU. I’ve finally managed to do this. QEMU now stops displaying the “Guest has not initialized the display (yet)” message.
Here are some important code/documentation that helped:
clcdfb_enable, drivers/video/fbdev/amba-clcd.c),pl110_enabled, hw/display/pl110.c)We have talked about the possibility to run Qt applications on SO3. Of course, this will require some adaptations regarding the libc syscalls since we intend to keep the arm-linux-gnueabihf toolchain for simplicity reasons, as far as possible.
Nevertheless, an interesting alterative would be littlevgl which has a very nice and simple GUI very cool for embedded systems.
Looking at how Linux initialises the CLCD controller in SOO, I was able to do the same in the SO3 driver albeit in a more simple way. These are the main steps :
I added some test code to display 3 different colours on the screen. The test code will of course be removed later on.
Concerning the code organisation, I have added an fb folder in so3/devices and inside it is the pl111.c driver code.
I have added the posibility of writing to the framebuffer directly from user space. For this, I have implemented mmap syscall as well as adding support for framebuffer file types in the open syscall. I have added an example of how this can be done in the fillfb app. An example:
/* Get a file descriptor for `fb.0'. */
fd = open("fb.0", 0);
/* Get a virtual address mapped to the fb's memory. */
vram_addr = sys_mmap(0x4b000 * 2, 0, fd, 0);
/* Display a pixel. */
vram_addr[0] = create_px(0xff, 0, 0);
The first step was adding a kind of “wrapper” for frambuffer drivers (devices/fb.c). This allows drivers to register their framebuffer’s file operations. It is then possible to retrieve these fops for a given fb (now we only have PL111 but we could have more).
This is what happens in the do_open function in VFS. If the filename has the format fb.X then it is handled as a framebuffer file type, and the fops associated with the newly created file descriptor will be the fops registered by the driver (currently, PL111 defines only the mmap fop); X determining which fb to use.
I’ve then implemented the mmap syscall (in the VFS). Its implementation is the following:
mmap fop that will to the actual mapping using the above two parameters.The mmap fop is implemented (in the PL111 driver) using the create_mapping and add_page_to_proc functions, very similar to what is done when a new process is created (in the allocate_page function). I was inspired to do this mmap syscall/fop separation after reading Chapter 15 of LDD, which has a section on mmap. It is interesting because this way, the fb’s base address (which is to peculiar to each device) stays within the fb driver.
Two more things could be done:
sys_mmap (in usr/),write fop to the PL111 driver so the construction of the pixel (which depends on the bpp mode) would not be done in user space.This will ease the integration of LittlevGL, which will be my next step.
I welcome any comments on my implementation :).
I integrated LittlevGL into SO3 
. I had to write a Makefile in usr/lvgl/ which compiles LittlevGL as library and then adapt the existing Makefile in usr/ in order to link the apps against the library. I also added a simple app to demonstrate the process of using LittlevGL (lvgl.c).
I believe I’ll need to go back working on the driver in order to add support for the mouse. I was able to display a pointer with lvgl, but of course it would not move.
Congratulations ! This is a very nice result
I’ve spent some time looking at displays for the Pi 4.
All are capacitive touch displays. My opinion is that the official display would be the safest choice but the resolution is not so good. The 52Pi 7" display looks good. For a bigger 10.1", Waveshare is our only option.
I’ve changed the way devices are opened (in vfs.c:do_open) to make it more general. Now, any filename matching the /dev/<dev-class>[dev-id] format will be considered as a device. As such, for framebuffers we should now use /dev/fbX instead of fb.X. This allows us to handle different device classes without having to modify the VFS.
For the moment there will be some code duplication each time we add a driver class, but I have found a better way to implement this and will change it in another commit.
I have implemented a new driver for the PL050 PS/2 Keyboard/Mouse Interface. The code is in devices/input/pl050.c. I have also added a small helper in ps2.c to help with the PS/2 communication protocol.
The driver enables “packet streaming” on the mouse, meaning that the mouse will automatically send packets when it is moved. These packets are then read in the interrupt routine defined in the driver. There is also another way to do this: we could periodically ask the mouse for a single packet.
This could be interesting because in LittlevGL, we must implement a callback which is called periodically by the library. In this callback we could request the mouse position via an ioctl. Then, in the ioctl implementation, we could request a packet from the mouse.
It is currently not implemented in this manner but I will have to try it. For now we have global x, y coordinates which are updated in the ISR. These coordinates can be retrieved in user space via the ioctl GET_STATE command.
Here are some interesting documentations for dealing with PL050 and PS/2:
I added a more complete demo of LittlevGL (usr/src/demo.c). It is based on one of LittlevGL’s own demos. I added support for the mouse and also added an image. The image comes from a C variable (created with this converter) and the next step will be to implement the interface which will allow LittlevGL to read files from the filesystem.
Mouse movements are not perfect but it’s actually pretty good considering the simplicity of the calculation.
Here’s a video of the demo
I finished simplifying the way drivers register their device. This allowed me to delete the fb.c and input.c files as they were doing the same thing. Here is an overview of the system:
On the left side we have drivers such as pl111 and pl050 who want to register their devices in order to be accessed by someone else. For example, the VFS in order to open device files.
A driver can register multiple devices, this could be the case for pl050 which could register both the mouse and keyboard, yet with a different set of fops.
The devices are registered in the device.c file, using a linked list. In the /dev/<dev-class>[dev-id] filename format, the dev-id corresponds to the n-th registered device of the given dev-class. It is not a number decided by the driver.
I added a basic filesystem “driver” for LittlevGL, so it can now read and display images from the SO3 filesystem. However, the implementation is not complete because I could not implement the seek and tell callbacks that LittlevGL requires, and I did not find corresponding syscalls nor functions in vfs.c or fat.c.
Apparently LittlevGL needs these functions when the mouse hovers the images, in order to redraw it.
I also changed the bpp mode for PL111 to 24 (instead of 16).
I refactored the PL050 driver by removing all mouse-related code. The reason is that there are two PL050 controllers on the VExpress board, one for the mouse and another one for the keyboard. I placed all the mouse-related code in the kmi1.c file.
This allowed me to write a new driver for the keyboard (kmi0.c). Parsing PS/2 scan codes coming from the keyboard is trickier than for the mouse, so I’ve only implemented some basic features:
I updated the LittlevGL demo app, so we now have keyboard and mouse support. I also finished writing the “filesystem driver” for LittlevGL (using fseek and ftell). Now, the image is correctly redrawn when the mouse cursor hovers over it.
During the last few weeks I’ve been working on SOO, with the goal of being able to read the ME’s framebuffer from the agency. Here are steps I’ve done so far:
so3virt_fb driver replacing the pl111 driver. In this driver I’m allocating memory for the framebuffer manually as there is no more VRAM. I’m also writing the value 0x12345678 to the first pixel.This is just a proof of concept and there’s still a lot of work to be done 
.
Don’t hesitate to create a topic in the SOO category since it rather belongs to this development.
Fully supported today in the master branch. And moved to CMake for the user applications. I plan to deploy the whole stuff on RPi4 64-bit during this Year (with LVGL on LCD).
Hello all! I hope you’re doing well 
I was very happy to see that the LVGL was upgraded to version 8 and that the founder Gabor apparently came to visit us on this forum!
I noticed that my Bachelor work was not available on http://tb.heig-vd.ch (it’s not the only one that is missing, not sure why…) so I had the idea to convert it to a Sphinx documentation. You can see the result here: nyg.gitlab.io/so3-support-graphique (repository is gitlab.com/nyg/so3-support-graphique).
There may still be some small issues to correct following the conversion from Latex, otherwise it looks good.
As SO3 and SOO are also using Sphinx for documentation, you can definitely integrate my work into SO3 and SOO’s respective documentation ;).
Very nice work Nikolaos, and hope you’re doing well too. I don’t know why the heig-vd site doesn’t work but I can imagine some delays due to the pandemy.
Many thanks for your links. French shouldn’t be a problem thanks to google translate 
Gabor showed up in the forum indeed and using so3 to do testings with LVGL was in his plan, but I haven’t got any news since then. It would be interesting to upgrade LVGL as well, if you are inspired 
I’ll try to ping Gabor in the coming weeks.