How I move to Canada, from planning to moving, in 10 days

I moved from Hong Kong to Canada in 2021. This whole move is orchestrated within 10 days, showing the absolute best of my ability to execution.

It was crazy. How did I do that?

1. Things that I did before these 10 days
2. Getting a list of what needs to be done to get moving as quick as possible
3. Get the todo items nicely scheduled, dependency sorted, action done orderly
4. Keeping myself highly motivated and moving every single day

Why?

Before talking about the hows, let me talk about why I did such a thing.

In 2021, I just came out of the biggest failure in my life, my Masters degree. I was devastated, not able to determine what to do next, uncertain about everything. Also remember I was in Hong Kong and had to witness 2019 and then COVID. And because of my highly unstable emotion I also messed up all my relationships. I was in the deepest valley. I was looking into a new start of my life, or the end of it.

But I was lucky. Canada opened up just in time, and I do have work visa.

How?

Things I did before these 10 days

The 10 days that I mentioned is from the time I totally committed to going to Canada (and started committing all my attention and resources on it), to when I landed in Toronto. I have done a lot of things before these 10 days, but those don't require significant committment, if any.

Again, I did a lot of google search. Absorbed a lot on information on how to go to Canada, how to live in Canada, different places to live and their characteristics, residence, job market, transportation. Basically I gained a lot of random knowledge about Canada.

Then I applied for Open Work Permit in Canada. Don't ask me how. That was just something that's available at that time. This action opened up the option for me to live in Canada. It's always important to keep your options open and know your options.

I also got a rough list of things that I need to do, in general, when I need to move to a new place alone. Like air ticket, residence, transportation, food, language and communitcation, job search,... and made some ready-to-go resource pack for any of my options. The resource packs has proved it worth.

Committing to an option

This step is very simple. I realised that I have to commit to going to Canada. This step is basically digging up my resource pack for Canada, and convert it into action plan.

Todo item scheduling, dependency management and action

I splitted up my resource pack into action items, put them into Jira, and attached deadline to them, and also visualised my tasks on a Gantt chart to emphasise inter-dependency relationships.

Keep myself motivated

This part was easy. I have no other options viable at this point.

---

I am not lazy in writting very short sentences in every steps after the first. Basically every hard part of the moving is done before the 10 days. The moral of this story is that, if you prepare well, everything is easy.

At the end, my move to Canada was a great success. I landed in Toronto, had temporary residence, found a job and semi-permanent residence in two months, sorted out my transportation in one month, and generally stablise and settle in 3 months. This has helped me greatly in having a new environment to recover from everything in the last 2 years.

A journey to create my first operating system: 3 - What it ended up with?

Look at my final report and see it for yourself

Also my code!

https://github.com/SheepiesLab/nvmos

And the demo user-space code!

https://github.com/SheepiesLab/nvmos-demo

Is it a piece of crap? Yes, it's the essence of every proof-of-concept.

Am I proud of it? YES I AM!

A journey to create my first operating system: 2 - Why did I torture myself?

First off, it's not a torture. In fact it is the most fun I have had in my whole life, creating something from scratch, learning a lot of different weird things, architecting a whole system all on my own wish. It was the best time of my computer life!

Challenges

Is writting an OS a challenging deal? I would be disingeneous or lying or a megalomaniac if I say not. There are a list of challenges:

• I have no idea where to start
• Apart from a year 3 course about the theory of operating systems, I have no idea how or where or what to do to even get started writing an OS. I know a bit about Linux and old bootloader but that's it. 
• I do not know assembly
• I had a course on computer architecture where I learned a bit of ASM of I don't remember what architecture! I don't know anything about ASM of x86 or ARM.
• A design of such thing is hard to judge
• I have a pretty good idea of what it should look like: A merge of memory management system and file system, Linux styled, but I don't have a way to judge if a system design is good or bad!
• I have no friends
• Really? Is this even a challenge mentionable? Yes! I don't have friends to talk to, and I don't want to bore my girlfriend to death with all these OS work. So I have to bare this weight on my own. To be honest I did myself disservice by staying with my girlfriend ALL THE TIME and met no one :D
• What are all these special memory addresses and segments?
• x86 is a very old and history-filled architecture. A lot of things rely on convention which is very sparsely documented and very hard to debug. You don't know what hits you until you know.
• Why do I even start from scratch? Can I not just modify Linux?
• My knowledge of Linux at that time was also mediocre and cannot support me changing the kernel structure and implement such thing. I'm forced to make a new OS because of it. I'm still not able to make changes in Linux (but now it's because I'm hopelessly depressed...)
• What should I also implement to make it an OS?
• At what point should I call it a success? It obviously would not host anything or do anything productive, it would just serve as a proof-of-concept. So what is successful is entirely up to my definition. I still think it's an incomplete piece of crap but hey, it has user-space and can load programs!
• How do I plan out the project and split it into small, manageable chunks?
• This isn't something that I know, this isn't something that I can reasonably plan and time for. I still need some ways to plan for it so that I can finish it and not fail!
• Where do I even find these kind of non-volatile memory?
• The hardware just came out. It was non-existent in the market, or it's stupid expensive. There is no way I can get it.
• Uncertainty
• I just don't know what would happen.

In short, this is just not something my 22-year-old brain has seen.

Solutions

By the fact that I'm not a dropout I successfully graduated, I must have made something that can remotely be called an OS, and have stepped through all these difficulties.

• Creating myself a starting point
• Google searches. Lots and lots of them. Reading and accumulating information on the topic of OS development day and night for months. 
• Bootstrapping my development environment
• Gives myself a starting point: OS Dev Bare Bones.
• Sets up a nice little development environment with QEMU emulation, that provides debuggable and rapidly reloadable VM, that also emulates non-volatile memory.
• Sets up the toolchain to cross compile my little OS to a freestanding x86 environment.
• Creates all these things on AWS, an EC2 instance, that is powerful enough to run everything. Oh and I also configured it to startup as quick as possible, and self shutdown 1 hour on no activity so that I don't burn through my money. Because I don't have a decent computer that can do everything.
• Did not intentionally learn assembly at all
• Oh I found out that the need of assembly in my project is quite minimal midway, and I never bothered learning ASM fully. I did struggle choosing Intel or AT&T style assembly. I don't quite remember which way I went as it doesn't even matter in later stages.
• Did not judge the design at all, just work something out and show
• I just made something that makes sense to me and everyone that I asked. As a proof-of-concept I don't have any specific requirement apart from THE DESIGN OF AN UNIFIED DATA MANAGEMENT SYSTEM. Oh what the hell was I thinking not pondering on the design at least a million times!!!
• Anyway I just go with what makes sense. Having something done is better than talking about it everyday and not have something done.
• I still have no friend
• Yeah no one would listen to how I wrote an OS. That's also why I'm writing this blog like 5 years later. I held up, nothing tragic happened. 
• But I know why I'm doing it. My motivation is clear: to explore how things can be done in OSes if such thing as non-volatile memory is real. I'm happy every day working on it, unlike some time later in my life.
• Did I mention that I did a lot of searches?
• Answered a lot of questions when I searched a lot on x86 systems. Learned a lot about TSS, GDT, LDT, x86 flags, CPU memory protection hierarchy, caching, system exclusive instructions, rings, memory-mapped area, low/high memory......
• The year of doing this OS project gave me much more than my 16 year formal education.
• I still don't know Linux
• I just give up on the idea of modifying Linux on day one. It is just too hard. Knowing what I don't know was a blessing. If I was to modify Linux I would not have finished the project nor graduated.
• Bare minimum is what I implemented
• Remember how we define an OS? It's a piece of software that talks to hardware on behalf of applications so that dumb things don't happen. That's exactly what I implemented, nothing more. Not even task switching, scheduling, CPU control, security,.....
• I drew a component map, and I tackled them one by one
• It's just good old divide-and-conquer. Also I don't know why no one is actually planning out a project in real world like this. Are project managers doing their job?
• Emulation, best way to do something you cannot do
• I did try to buy that Optane memory, but it did not happen until 2 years later.
• So for the entire project I just relied on file-backed memory on QEMU emulation.
• If it works it works.
• Just deal with it
• It was so fun I didn't even care about the risk of not being able to graduate!

I learned a lot. Really.

A journey to create my first operating system: 1 - What is this non-sense?

This operating system (OS) that I made, in year 2017/2018 was called non-volatile memory operating system (NVMOS). The vision of creating this OS is an attempt to unify the two very important data systems in any modern computer systems: main memory (RAM) and storage (disks), with the advancement in memory technology, namely non-volatile memory.

Two data systems? In one computer? Why?

An ideal computer don't need two data systems. If we look at the von Neumann architecture that our computer systems adopt, we can see that there are 3 essential components for a computer system to work: a CPU to do the actual computation, a memory to feed the CPU instructions and data, and to store the computation result, and I/O devices to tell the outside world of the computation result. If we look at the theoratical model of a computer, a Turing Machine, only one very long tape is used, not two. The fundamental role of memory in a computer is to store instructions and data to be processed by the CPU, and store the results from the CPU, as portraited in the Turing Machine model. Fundamentally we do not need a second storage system.

However, functionally we do, because of the limitations of hardware. Modern memory is achieved with DRAM or SRAM. They have very different characteristics and are used in two very different ways, but in short, they both have two huge disadvantages: cost and volatility!

Cost

RAM is expensive in general. As of writing (2024) the price of a piece of 32GB DDR5 RAM is £100, which calculates as £3.125 per GB. In this era of AI and big data, pentabytes level of data is not uncommon among enterprise datacenters. A naive calculation goes to £3 million. Not cheap relative to the hero (disks) that we will discuss.

Volatility

RAM is not durable. SRAM operates as logic gates flip and flop between electricial states and maintain balance to store a state. DRAM uses capacitors, opposite electrical charges that holds themselves together in one place to store a state. Both dependent on the presence of electricity supply and both lose its state when electricity is gone (instantly for SRAM, and eventually for DRAM).

In practicality we cannot make the main memory infinitely large in terms of capacity, and we cannot hold the data within it reliably during a power outage scenario. So we need something else: disks! In short, disks are these magical things that are cheap enough to store a huge amount of data (TB/PB level) practically and durably. We have spinning disks that are magnetic discs that records data in tiny magnets on the discs, and then we have SSDs that are transistor gates that can change it's state semi-permanently through electric potential. (I'm not and expert in storage hardware and these are all made-up.)

Great! Let's replace the memory with SSD! Shall we? No, not yet. SSD is still slower than DRAM/SRAM (by a lot), and it cannot keep up with stuffing the CPU with data! That's why we're not using SSD in the place of DRAM! Also we cannot just plug the SSD onto the memory slot!

Wait, what? We can? And it's almost as fast as DRAM? I don't want to talk about that, for now. We will come back to it in a minute.

Unfortunate hardware limits shapes unnecessarily complicate OS design

So there we have it, two data systems, and that puts a lot of work into designing OSes that work with these hardware. Oh what is operating system? It's just a piece of code that talk to every hardware within your computer, including CPU, memory, I/O controllers, storage, network, display... on behalf of your day-to-day applications, so that those programmers are relieved of the job of reading every single spec document of every hardware and writting software to talk to every single bit and potentially ruinning your computer because all programmers including me are morons (wait what am I doing?) Oh right. So because we need two hardware systems to do the job of a modern computer, we also need two software systems that talk to them and provide abstracted functionality to application programmers.

That's why every self-respecting OS has these two important systems: memory management and file system. One to manage the memory and one to manage your long-term storage that you don't want to lose.

What was that magic that we talked about?

Oh yeah, it's Intel's optane memory, or 3D XPoint is what they called. It came out in roughly 2016 and promised to be a memory hardware that is:

1. Cheap enough to be the role of SSD
2. Fast enough to be the role of RAM
3. Non-volatile

Do you see what it means? After reading all above? That means we can plug that optane memory in the RAM slot, use it as memory AND disks, and then in the OS ditch the memory management / file systems and just have one big thing to manage all the data in a computer!

That is exactly the idea.

Hello World!

Hello everyone! I'm Benedict Ng. I specialise in creating software systems from scratch to deployment. Thank you for visiting this blog and viewing my previous work. You're welcomed to contact me via benng@sheepieslab.com for any job enquiry!