So, you want to explore writing your own operating system and you want to do it in Rust? Cool, so did I. If you have not read it already, I will point you to OSDev’s list of Beginner Mistakes. OSDev stresses the right things in that article: approach this with some achievable goals (learn about ARM system architecture, learn about Rust, etc.). Approach it with a desire to learn the right way to do things. Do not approach this with the initial goal of writing a complete operating system.
NOTE: I am not an expert in kernel design. I am trying to learn the right ways myself, so it is likely that some of the things I do in the tutorial are not the best way. Treat this tutorial accordingly. Be suspicious. If you find a better way to do something, let me know!
I started by translating Raspberry Pi tutorials written in C to Rust, then expanding on them and cleaning up the code as I learned. That initial project, ROS, quickly went off the rails for a variety of reasons.
When JetBrains made RustRover freely available for non-commercial use, I decided to start over with Propeller. I had two main goals:
main. As such, Propeller’s main branch has cleaner commits and always works.The switch from the CMake + Corrosion + Cargo setup in ROS to just Cargo was not super easy, and I still needed the ARM GNU toolchain for its assembler and debugger. But, I think it worked out nicely enough to put together a coherent tutorial on building a toy kernel with RustRover, the ARM GNU toolchain, Python for some build tooling.
This tutorial’s inclusion of 32-bit ARM is a bit novel from what I have seen of toy kernel tutorials around the Internet. Relatively speaking, 64-bit is easy. In a world where 64-bit systems exist, 32-bit seems weird and hacky when you start learning about how virtual address spaces work.
I decided to walk both paths for the simple pleasure of learning how 32-bit kernels dealt with the address space limitations. It was a good choice that led to some really interesting history lessons about the evolution of how Windows and Linux handled it.
There are a ton of existing tutorials, and I will give them credit in time. I do not want to mention them up front, because I want you to start fresh. I will credit them when I mention specific things I learned from them.
Download the ARM GNU toolchain. You want the arm-none-eabi and aarch64-none-elf variants for your host system. They will be used to compile the assembly sources and provide target-specific builds of GDB for debugging.
Make sure you have a Python distribution on your machine for build tooling. Anything above Python 3.9 should be fine.
Download QEMU for initial testing. QEMU’s default Raspberry Pi profiles work well. We will use the Pi 2 profile for 32-bit and the Pi 3 profile for 64-bit.
If you want to try running your operating system on hardware, buy a cheap Raspberry Pi. A Raspberry Pi 3B is great. It is $35 and it has a Cortex-A53 that can support both AArch64 and ARMv7 kernels. A Raspberry Pi 4 is fine, just do not go crazy on RAM. Remember: a 32-bit operating system can only support up to 3 GiB of physical RAM. 1 GiB is fine and plenty enough to make 32-bit interesting.
I highly recommend buying a JTAG debugger if you want to try running on hardware. Testing in QEMU does not guarantee things will go smoothly on hardware, and you will be left guessing why if the kernel panics early. I have been there, and done that.
I bought the FTDI FT4232H Mini. It took some work to get it set up, but I remembered to thoroughly document the setup and will include that information later. The FTDI module can receive from the UART as well. This means you can see output in a serial terminal while debugging.
My home machine is a MacBook Pro, so I went with CoolTerm which also has Windows and Linux builds. CoolTerm works great, is still actively developed, and is free. But, be cool, leave Roger a donation if you use CoolTerm and like it.
I bought a book on Linux kernel programming and sought out the Windows Research Kernel to use as touchstones.
CAUTION: I am not sure it was legal for HighSchoolSoftwareClub to post the Windows Research Kernel on GitHub. But, it has been there for 8 years and it is not lurking in any shadows. Use at your own risk.
The fun of this project is to try to solve problems first, then look at how Windows and Linux did it. It is really exciting when you think of a way to do something and find out they do it the same way!
The Embedded Rust Book has a lot of good information about writing Rust for bare metal systems.
The Raspberry Pi, aside from being a cheap, fully capable computer, has two nice advantages:
So easy!
Once you get going and feel like you might want to do an x86[_64] port as well, Writing an OS in Rust by Philipp Oppermann can get you started. He actually seems to know what he is doing and even has an x86_64 boot loader you can use.