A journey to create my first operating system: 1 - What is this non-sense?

This operating system (OS) that I made, in year 2017/2018 was called non-volatile memory operating system (NVMOS). The vision of creating this OS is an attempt to unify the two very important data systems in any modern computer systems: main memory (RAM) and storage (disks), with the advancement in memory technology, namely non-volatile memory.

Two data systems? In one computer? Why?

An ideal computer don't need two data systems. If we look at the von Neumann architecture that our computer systems adopt, we can see that there are 3 essential components for a computer system to work: a CPU to do the actual computation, a memory to feed the CPU instructions and data, and to store the computation result, and I/O devices to tell the outside world of the computation result. If we look at the theoratical model of a computer, a Turing Machine, only one very long tape is used, not two. The fundamental role of memory in a computer is to store instructions and data to be processed by the CPU, and store the results from the CPU, as portraited in the Turing Machine model. Fundamentally we do not need a second storage system.

However, functionally we do, because of the limitations of hardware. Modern memory is achieved with DRAM or SRAM. They have very different characteristics and are used in two very different ways, but in short, they both have two huge disadvantages: cost and volatility!

Cost

RAM is expensive in general. As of writing (2024) the price of a piece of 32GB DDR5 RAM is £100, which calculates as £3.125 per GB. In this era of AI and big data, pentabytes level of data is not uncommon among enterprise datacenters. A naive calculation goes to £3 million. Not cheap relative to the hero (disks) that we will discuss.

Volatility

RAM is not durable. SRAM operates as logic gates flip and flop between electricial states and maintain balance to store a state. DRAM uses capacitors, opposite electrical charges that holds themselves together in one place to store a state. Both dependent on the presence of electricity supply and both lose its state when electricity is gone (instantly for SRAM, and eventually for DRAM).

In practicality we cannot make the main memory infinitely large in terms of capacity, and we cannot hold the data within it reliably during a power outage scenario. So we need something else: disks! In short, disks are these magical things that are cheap enough to store a huge amount of data (TB/PB level) practically and durably. We have spinning disks that are magnetic discs that records data in tiny magnets on the discs, and then we have SSDs that are transistor gates that can change it's state semi-permanently through electric potential. (I'm not and expert in storage hardware and these are all made-up.)

Great! Let's replace the memory with SSD! Shall we? No, not yet. SSD is still slower than DRAM/SRAM (by a lot), and it cannot keep up with stuffing the CPU with data! That's why we're not using SSD in the place of DRAM! Also we cannot just plug the SSD onto the memory slot!

Wait, what? We can? And it's almost as fast as DRAM? I don't want to talk about that, for now. We will come back to it in a minute.

Unfortunate hardware limits shapes unnecessarily complicate OS design

So there we have it, two data systems, and that puts a lot of work into designing OSes that work with these hardware. Oh what is operating system? It's just a piece of code that talk to every hardware within your computer, including CPU, memory, I/O controllers, storage, network, display... on behalf of your day-to-day applications, so that those programmers are relieved of the job of reading every single spec document of every hardware and writting software to talk to every single bit and potentially ruinning your computer because all programmers including me are morons (wait what am I doing?) Oh right. So because we need two hardware systems to do the job of a modern computer, we also need two software systems that talk to them and provide abstracted functionality to application programmers.

That's why every self-respecting OS has these two important systems: memory management and file system. One to manage the memory and one to manage your long-term storage that you don't want to lose.

What was that magic that we talked about?

Oh yeah, it's Intel's optane memory, or 3D XPoint is what they called. It came out in roughly 2016 and promised to be a memory hardware that is:

1. Cheap enough to be the role of SSD
2. Fast enough to be the role of RAM
3. Non-volatile

Do you see what it means? After reading all above? That means we can plug that optane memory in the RAM slot, use it as memory AND disks, and then in the OS ditch the memory management / file systems and just have one big thing to manage all the data in a computer!

That is exactly the idea.