Graphics support for SO3

SO3 Operating System
1

I will be implementing graphics support for SO3 as part of my Bachelor project. This topic is here to regroup all related discussions.

1 Like
2

The first step was to have QEMU display a graphical window in which different controllers are emulated (serial0, pl111, etc.). For this, I had to:

  1. recompile QEMU with the --enable-gtk option,
  2. remove the -nographic option from the qemu command in the st script,
  3. (optional, gtk seems to be the default) add the -display gtk option to the qemu command,
  4. (optional) remove the -serial mon:stdio option from the qemu command so that the console output from SO3 is displayed in the graphical window too (but in the serial0 controller — warning: output looks bad).
3

The second step was to find the correct controller to use (for the VExpress board that we emulated with QEMU). I first discovered the HDLCD controller which looked promising but isn’t actually emulated by QEMU (see hw/arm/vexpress.c). This controller is located on the daugtherboard and is a more performant version of the CLCD controller located on the motherboard. So, the CLCD is the one we’ll have to use for the moment.

ARM documentation:

4

The third step was to initialise in SO3 the PL111 (CLCD) controller emulated by QEMU. I’ve finally managed to do this. QEMU now stops displaying the “Guest has not initialized the display (yet)” message.

Here are some important code/documentation that helped:

5

We have talked about the possibility to run Qt applications on SO3. Of course, this will require some adaptations regarding the libc syscalls since we intend to keep the arm-linux-gnueabihf toolchain for simplicity reasons, as far as possible.
Nevertheless, an interesting alterative would be littlevgl which has a very nice and simple GUI very cool for embedded systems.

6

Looking at how Linux initialises the CLCD controller in SOO, I was able to do the same in the SO3 driver albeit in a more simple way. These are the main steps :

  1. Disable interrupts.
  2. Set the timing registers:
    • Timing0 and Timing1 set horizontal and vertical properties such as the resolution,
    • Timing2 mostly concerns TFT vs SPN panels,
    • Timing3 is set to 0.
  3. Set the frame buffer address (here I’ve use the VRAM).
  4. Set the control register which set properties like the bpp mode, TFT or SPN, single or dual panel, RGB vs BGR, endianess, etc. It also allows the device to be enabled and powered on.

I added some test code to display 3 different colours on the screen. The test code will of course be removed later on.

rgb-pl111
rgb-pl111646Ă—537 5.74 KB

Concerning the code organisation, I have added an fb folder in so3/devices and inside it is the pl111.c driver code.

1 Like
7

I have added the posibility of writing to the framebuffer directly from user space. For this, I have implemented mmap syscall as well as adding support for framebuffer file types in the open syscall. I have added an example of how this can be done in the fillfb app. An example:

/* Get a file descriptor for `fb.0'. */
fd = open("fb.0", 0);

/* Get a virtual address mapped to the fb's memory. */
vram_addr = sys_mmap(0x4b000 * 2, 0, fd, 0);

/* Display a pixel. */
vram_addr[0] = create_px(0xff, 0, 0);

The first step was adding a kind of “wrapper” for frambuffer drivers (devices/fb.c). This allows drivers to register their framebuffer’s file operations. It is then possible to retrieve these fops for a given fb (now we only have PL111 but we could have more).

This is what happens in the do_open function in VFS. If the filename has the format fb.X then it is handled as a framebuffer file type, and the fops associated with the newly created file descriptor will be the fops registered by the driver (currently, PL111 defines only the mmap fop); X determining which fb to use.

I’ve then implemented the mmap syscall (in the VFS). Its implementation is the following:

  1. get the fops associated to the given fd,
  2. compute the number of pages that will need to be mapped,
  3. compute the process’ virtual base address from where the pages should start,
  4. call the mmap fop that will to the actual mapping using the above two parameters.

The mmap fop is implemented (in the PL111 driver) using the create_mapping and add_page_to_proc functions, very similar to what is done when a new process is created (in the allocate_page function). I was inspired to do this mmap syscall/fop separation after reading Chapter 15 of LDD, which has a section on mmap. It is interesting because this way, the fb’s base address (which is to peculiar to each device) stays within the fb driver.

Two more things could be done:

  1. adding a more user-friendly wrapper for sys_mmap (in usr/),
  2. adding the write fop to the PL111 driver so the construction of the pixel (which depends on the bpp mode) would not be done in user space.

This will ease the integration of LittlevGL, which will be my next step.

I welcome any comments on my implementation :).

1 Like
8

I integrated LittlevGL into SO3 :slight_smile: . I had to write a Makefile in usr/lvgl/ which compiles LittlevGL as library and then adapt the existing Makefile in usr/ in order to link the apps against the library. I also added a simple app to demonstrate the process of using LittlevGL (lvgl.c).

I believe I’ll need to go back working on the driver in order to add support for the mouse. I was able to display a pointer with lvgl, but of course it would not move.

qemu
qemu646Ă—537 6.92 KB

9

Congratulations ! This is a very nice result :slight_smile:

10

I’ve spent some time looking at displays for the Pi 4.

  1. First of all, there is an official Raspberry Pi 4 display.
  2. Then, there are a couple of 7" touch displays made by 52Pi.com which brand itself as a “high-level open source hardware provider”. All their 7" screens have a 1024 * 600 resolution. They are advertised as “driver free”.
  3. Finally, there is Waveshare.com which makes touch displays of multiple sizes.

All are capacitive touch displays. My opinion is that the official display would be the safest choice but the resolution is not so good. The 52Pi 7" display looks good. For a bigger 10.1", Waveshare is our only option.

11

I’ve changed the way devices are opened (in vfs.c:do_open) to make it more general. Now, any filename matching the /dev/<dev-class>[dev-id] format will be considered as a device. As such, for framebuffers we should now use /dev/fbX instead of fb.X. This allows us to handle different device classes without having to modify the VFS.

For the moment there will be some code duplication each time we add a driver class, but I have found a better way to implement this and will change it in another commit.

1 Like
12

I have implemented a new driver for the PL050 PS/2 Keyboard/Mouse Interface. The code is in devices/input/pl050.c. I have also added a small helper in ps2.c to help with the PS/2 communication protocol.

The driver enables “packet streaming” on the mouse, meaning that the mouse will automatically send packets when it is moved. These packets are then read in the interrupt routine defined in the driver. There is also another way to do this: we could periodically ask the mouse for a single packet.

This could be interesting because in LittlevGL, we must implement a callback which is called periodically by the library. In this callback we could request the mouse position via an ioctl. Then, in the ioctl implementation, we could request a packet from the mouse.

It is currently not implemented in this manner but I will have to try it. For now we have global x, y coordinates which are updated in the ISR. These coordinates can be retrieved in user space via the ioctl GET_STATE command.

Here are some interesting documentations for dealing with PL050 and PS/2:

13

I added a more complete demo of LittlevGL (usr/src/demo.c). It is based on one of LittlevGL’s own demos. I added support for the mouse and also added an image. The image comes from a C variable (created with this converter) and the next step will be to implement the interface which will allow LittlevGL to read files from the filesystem.

Mouse movements are not perfect but it’s actually pretty good considering the simplicity of the calculation.

Here’s a video of the demo :slight_smile:

Also available on YouTube

1 Like
14

I finished simplifying the way drivers register their device. This allowed me to delete the fb.c and input.c files as they were doing the same thing. Here is an overview of the system:

regdev
regdev1966Ă—636 29.9 KB

On the left side we have drivers such as pl111 and pl050 who want to register their devices in order to be accessed by someone else. For example, the VFS in order to open device files.

A driver can register multiple devices, this could be the case for pl050 which could register both the mouse and keyboard, yet with a different set of fops.

The devices are registered in the device.c file, using a linked list. In the /dev/<dev-class>[dev-id] filename format, the dev-id corresponds to the n-th registered device of the given dev-class. It is not a number decided by the driver.

15

I added a basic filesystem “driver” for LittlevGL, so it can now read and display images from the SO3 filesystem. However, the implementation is not complete because I could not implement the seek and tell callbacks that LittlevGL requires, and I did not find corresponding syscalls nor functions in vfs.c or fat.c.

Apparently LittlevGL needs these functions when the mouse hovers the images, in order to redraw it.

I also changed the bpp mode for PL111 to 24 (instead of 16).

16

I refactored the PL050 driver by removing all mouse-related code. The reason is that there are two PL050 controllers on the VExpress board, one for the mouse and another one for the keyboard. I placed all the mouse-related code in the kmi1.c file.

This allowed me to write a new driver for the keyboard (kmi0.c). Parsing PS/2 scan codes coming from the keyboard is trickier than for the mouse, so I’ve only implemented some basic features:

  • lowercase letters
  • uppercase letters (using the shift key)
  • some control keys (arrows, backspace, delete, tab).

I updated the LittlevGL demo app, so we now have keyboard and mouse support. I also finished writing the “filesystem driver” for LittlevGL (using fseek and ftell). Now, the image is correctly redrawn when the mouse cursor hovers over it.

17

During the last few weeks I’ve been working on SOO, with the goal of being able to read the ME’s framebuffer from the agency. Here are steps I’ve done so far:

  • Created the so3virt_fb driver replacing the pl111 driver. In this driver I’m allocating memory for the framebuffer manually as there is no more VRAM. I’m also writing the value 0x12345678 to the first pixel.
  • Created the vfb front-end based on vdummy. In the probe function, I retrieve a grantref from the address of the framebuffer memory and I write this grantref to a VBStore property.
  • Created the vfb back-end based on vdummy. In the probe function, I watch a VBStore property created beforehand. In the watch callback I retrieve the property’s new value which is the grantref passed from the front-end. I map it to allocated memory and then I read the value of the first pixel which is equal to 0x12345678 (value set in the driver).

This is just a proof of concept and there’s still a lot of work to be done :sweat_smile:.

1 Like
18

Don’t hesitate to create a topic in the SOO category since it rather belongs to this development.

19

LVGL-v8 now available. Hope that it will be integrated in SO3 soon.

20

Fully supported today in the master branch. And moved to CMake for the user applications. I plan to deploy the whole stuff on RPi4 64-bit during this Year (with LVGL on LCD).