All Articles

Notes from Working Through Introduction to OS Development from Scratch: Environment Setup Through Chapter 2

This page has been machine-translated from the original page.

I want to retry Introduction to OS Development from Scratch, which I bought as soon as it was released but left untouched because I couldn’t find the time and got discouraged by the difficulty.

Basically, I plan to proceed mainly by typing in the code, and then write up articles about the points where I get stuck or where my understanding is still shallow.

This time, I’ll cover environment setup through Chapter 2. By the way, I’m skipping Chapter 1 because it’s mostly just doing the exercises.

Table of Contents

Environment setup

I built the development environment with the following setup.

  • Ubuntu 20.04 (built with a GUI on Hyper-V)
  • Windows 10 (host machine)
  • VSCode (host machine)

The plan is to do development by connecting from VSCode to the Ubuntu instance running on Hyper-V via Remote SSH, and only log in to Ubuntu with the GUI when I need to check behavior in Qemu.

Some of the QEMU commands explained in the book require 3D acceleration to be enabled. If (qemu) gtk initialization failed occurs in a virtual desktop environment, check whether 3D acceleration is enabled.

For the environment setup, I’m using the following GitHub repository published by the author as the base.

Reference: Build and run scripts for MikanOS

If your Ubuntu version is different, the steps may not work exactly as written and the amount of trouble increases, so if possible I think it’s best to prepare Ubuntu 20.04 in a virtual machine or WSL.

I’ve summarized how to connect from VSCode to Ubuntu on a virtual machine via Remote SSH here.

Reference: How to edit files directly on a virtual machine running on Hyper-V from VSCode - Yukituna’s Bookshelf

Creating a UEFI application with EDK II

Right from the beginning, it jumped into a technical area I knew absolutely nothing about.

Reference: GitHub - tianocore/edk2: EDK II

The .EFI file created here seems to be a UEFI application that is called from UEFI BIOS. My understanding is that EDK II is something like an SDK for creating UEFI applications.

struct MemoryMap {
  UINTN buffer_size;
  VOID* buffer;
  UINTN map_size;
  UINTN map_key;
  UINTN descriptor_size;
  UINT32 descriptor_version;
};

EFI_STATUS GetMemoryMap(struct MemoryMap* map) {
  if (map->buffer == NULL) {
    return EFI_BUFFER_TOO_SMALL;
  }

  map->map_size = map->buffer_size;
  return gBS->GetMemoryMap(
      &map->map_size,
      (EFI_MEMORY_DESCRIPTOR*)map->buffer,
      &map->map_key,
      &map->descriptor_size,
      &map->descriptor_version);
}

I didn’t really understand what it was doing, so for the time being I followed the code. Here, GetMemoryMap() stores the memory map in a MemoryMap structure named memmap.

GetMemoryMap() uses gBS->GetMemoryMap() to obtain the memory map at the time the function is called.

The second argument, given as (EFI_MEMORY_DESCRIPTOR*)map->buffer, is where the function stores the memory map it retrieves. The stored data is defined as the EFI_MEMORY_DESCRIPTOR structure in edk2/MdePkg/Include/Uefi/UefiSpec.h.

///
/// Memory descriptor version number.
///
#define EFI_MEMORY_DESCRIPTOR_VERSION 1

///
/// Definition of an EFI memory descriptor.
///
typedef struct {
  ///
  /// Type of the memory region.
  /// Type EFI_MEMORY_TYPE is defined in the
  /// AllocatePages() function description.
  ///
  UINT32                Type;
  ///
  /// Physical address of the first byte in the memory region. PhysicalStart must be
  /// aligned on a 4 KiB boundary, and must not be above 0xfffffffffffff000. Type
  /// EFI_PHYSICAL_ADDRESS is defined in the AllocatePages() function description
  ///
  EFI_PHYSICAL_ADDRESS  PhysicalStart;
  ///
  /// Virtual address of the first byte in the memory region.
  /// VirtualStart must be aligned on a 4 KiB boundary,
  /// and must not be above 0xfffffffffffff000.
  ///
  EFI_VIRTUAL_ADDRESS   VirtualStart;
  ///
  /// NumberOfPagesNumber of 4 KiB pages in the memory region.
  /// NumberOfPages must not be 0, and must not be any value
  /// that would represent a memory page with a start address,
  /// either physical or virtual, above 0xfffffffffffff000.
  ///
  UINT64                NumberOfPages;
  ///
  /// Attributes of the memory region that describe the bit mask of capabilities
  /// for that memory region, and not necessarily the current settings for that
  /// memory region.
  ///
  UINT64                Attribute;
} EFI_MEMORY_DESCRIPTOR;

In the end, the memory map obtained here was saved by SaveMemoryMap() as a CSV file named memmmap. Run the following commands directly under edk2/Build/OSLoaderX64/DEBUG_CLANG38/X64/OSLoaderPkg/Loader/DEBUG, where the built image file is created, to inspect the contents of the image file.

mkdir -o mnt
sudo mount -o loop disk.img mnt
cat mnt/memmap

#//

unmount mnt

For now, if I can at least confirm the contents of this memory map, that means I’ve finished the material up through Chapter 2.

Index, Type, Type(name), PhysicalStart, NumberOfPages, Attribute
0, 3, EfiBootServicesCode, 00000000, 1, F
1, 7, EfiConventionalMemory, 00001000, 9F, F
2, 7, EfiConventionalMemory, 00100000, 700, F
3, A, EfiACPIMemoryNVS, 00800000, 8, F
4, 7, EfiConventionalMemory, 00808000, 8, F
5, A, EfiACPIMemoryNVS, 00810000, F0, F
6, 4, EfiBootServicesData, 00900000, B00, F
7, 7, EfiConventionalMemory, 01400000, 3AB36, F
8, 4, EfiBootServicesData, 3BF36000, 20, F
9, 7, EfiConventionalMemory, 3BF56000, 270C, F
10, 1, EfiLoaderCode, 3E662000, 2, F
11, 4, EfiBootServicesData, 3E664000, 219, F
12, 3, EfiBootServicesCode, 3E87D000, B7, F
13, A, EfiACPIMemoryNVS, 3E934000, 12, F
14, 0, EfiReservedMemoryType, 3E946000, 1C, F
15, 3, EfiBootServicesCode, 3E962000, 10A, F
16, 6, EfiRuntimeServicesData, 3EA6C000, 5, F
17, 5, EfiRuntimeServicesCode, 3EA71000, 5, F
18, 6, EfiRuntimeServicesData, 3EA76000, 5, F
19, 5, EfiRuntimeServicesCode, 3EA7B000, 5, F
20, 6, EfiRuntimeServicesData, 3EA80000, 5, F
21, 5, EfiRuntimeServicesCode, 3EA85000, 7, F
22, 6, EfiRuntimeServicesData, 3EA8C000, 8F, F
23, 4, EfiBootServicesData, 3EB1B000, 4DA, F
24, 7, EfiConventionalMemory, 3EFF5000, 4, F
25, 4, EfiBootServicesData, 3EFF9000, 6, F
26, 7, EfiConventionalMemory, 3EFFF000, 1, F
27, 4, EfiBootServicesData, 3F000000, A1B, F
28, 7, EfiConventionalMemory, 3FA1B000, 1, F
29, 3, EfiBootServicesCode, 3FA1C000, 17F, F
30, 5, EfiRuntimeServicesCode, 3FB9B000, 30, F
31, 6, EfiRuntimeServicesData, 3FBCB000, 24, F
32, 0, EfiReservedMemoryType, 3FBEF000, 4, F
33, 9, EfiACPIReclaimMemory, 3FBF3000, 8, F
34, A, EfiACPIMemoryNVS, 3FBFB000, 4, F
35, 4, EfiBootServicesData, 3FBFF000, 201, F
36, 7, EfiConventionalMemory, 3FE00000, 8D, F
37, 4, EfiBootServicesData, 3FE8D000, 20, F
38, 3, EfiBootServicesCode, 3FEAD000, 20, F
39, 4, EfiBootServicesData, 3FECD000, 9, F
40, 3, EfiBootServicesCode, 3FED6000, 1E, F
41, 6, EfiRuntimeServicesData, 3FEF4000, 84, F
42, A, EfiACPIMemoryNVS, 3FF78000, 88, F
43, 6, EfiRuntimeServicesData, FFC00000, 400, 1

About building the image

I didn’t really understand the part where the created EFI file is built and then emulated in Qemu, because there was not much explanation in the book, so I wrote up my understanding here.

In the book, the EFI file is built by using the build command directly under the edk2 directory.

The build target information is written in edk2/Conf/target.txt.

This time, I used a folder called OSLoaderPkg as the workspace for the UEFI application I created myself.

#  PROPERTY              Type       Use         Description
#  ----------------      --------   --------    -----------------------------------------------------------
#  ACTIVE_PLATFORM       Filename   Recommended Specify the WORKSPACE relative Path and Filename
#                                               of the platform description file that will be used for the
#                                               build. This line is required if and only if the current
#                                               working directory does not contain one or more description
#                                               files.
ACTIVE_PLATFORM       = OSLoaderPkg/OSLoaderPkg.dsc

Here, OSLoaderPkg.dsc is defined as follows.

#@range_begin(defines)
[Defines]
  PLATFORM_NAME                  = OSLoaderPkg
  PLATFORM_GUID                  = d3f11f4e-71e9-11e8-a7e1-33fd4f7d5a3e
  PLATFORM_VERSION               = 0.1
  DSC_SPECIFICATION              = 0x00010005
  OUTPUT_DIRECTORY               = Build/OSLoader$(ARCH)
  SUPPORTED_ARCHITECTURES        = X64
  BUILD_TARGETS                  = DEBUG|RELEASE|NOOPT
#@range_end(defines)

#@range_begin(library_classes)
[LibraryClasses]
  UefiApplicationEntryPoint|MdePkg/Library/UefiApplicationEntryPoint/UefiApplicationEntryPoint.inf
  UefiLib|MdePkg/Library/UefiLib/UefiLib.inf
#@range_end(library_classes)

  BaseLib|MdePkg/Library/BaseLib/BaseLib.inf
  BaseMemoryLib|MdePkg/Library/BaseMemoryLib/BaseMemoryLib.inf
  DebugLib|MdePkg/Library/BaseDebugLibNull/BaseDebugLibNull.inf
  DevicePathLib|MdePkg/Library/UefiDevicePathLib/UefiDevicePathLib.inf
  MemoryAllocationLib|MdePkg/Library/UefiMemoryAllocationLib/UefiMemoryAllocationLib.inf
  PcdLib|MdePkg/Library/BasePcdLibNull/BasePcdLibNull.inf
  PrintLib|MdePkg/Library/BasePrintLib/BasePrintLib.inf
  UefiBootServicesTableLib|MdePkg/Library/UefiBootServicesTableLib/UefiBootServicesTableLib.inf
  UefiRuntimeServicesTableLib|MdePkg/Library/UefiRuntimeServicesTableLib/UefiRuntimeServicesTableLib.inf

#@range_begin(components)
[Components]
  OSLoaderPkg/Loader.inf
#@range_end(components)

With this setting, the EFI file built by EDK II is placed in OUTPUT_DIRECTORY = Build/OSLoader$(ARCH), so it ends up being saved under edk2/Build/OSLoaderX64/DEBUG_CLANG38/X64/OSLoaderPkg/Loader/DEBUG/.

Next, by running osbook/devtools/run_qemu.sh in the same directory as Loader.efi—edk2/Build/OSLoaderX64/DEBUG_CLANG38/X64/OSLoaderPkg/Loader/DEBUG/—the built EFI file is emulated in Qemu.

The contents of osbook/devtools/run_qemu.sh looked like this. It calls make_image.sh using the execution directory information passed in as arguments.

#!/bin/sh -ex

if [ $# -lt 1 ]
then
    echo "Usage: $0 <.efi file> [another file]"
    exit 1
fi

DEVENV_DIR=$(dirname "$0")
EFI_FILE=$1
ANOTHER_FILE=$2
DISK_IMG=./disk.img
MOUNT_POINT=./mnt

$DEVENV_DIR/make_image.sh $DISK_IMG $MOUNT_POINT $EFI_FILE $ANOTHER_FILE
$DEVENV_DIR/run_image.sh $DISK_IMG

Let’s look at the contents of make_image.sh.

#!/bin/sh -ex

if [ $# -lt 3 ]
then
    echo "Usage: $0 <image name> <mount point> <.efi file> [another file]"
    exit 1
fi

DEVENV_DIR=$(dirname "$0")
DISK_IMG=$1
MOUNT_POINT=$2
EFI_FILE=$3
ANOTHER_FILE=$4

if [ ! -f $EFI_FILE ]
then
    echo "No such file: $EFI_FILE"
    exit 1
fi

rm -f $DISK_IMG
qemu-img create -f raw $DISK_IMG 200M
mkfs.fat -n 'JISAKU OS' -s 2 -f 2 -R 32 -F 32 $DISK_IMG

$DEVENV_DIR/mount_image.sh $DISK_IMG $MOUNT_POINT
sudo mkdir -p $MOUNT_POINT/EFI/BOOT
sudo cp $EFI_FILE $MOUNT_POINT/EFI/BOOT/BOOTX64.EFI
if [ "$ANOTHER_FILE" != "" ]
then
    sudo cp $ANOTHER_FILE $MOUNT_POINT/
fi
sleep 0.5
sudo umount $MOUNT_POINT

Looking at this script, if disk.img already exists it removes it, then creates a 200 MB image file in raw format with qemu-img create -f raw $DISK_IMG 200M.

Reference: QEMU-img

Next, mkfs.fat formats it as an MS-DOS filesystem, mounts it under the mnt directory, and then copies the created EFI file to EFI/BOOT/BOOTX64.EFI.

Reference: Ubuntu Manpage: mkfs.fat - create an MS-DOS filesystem under Linux

This image file is then used, and run_image.sh emulates it with the qemu-system-x86_64 command.

qemu-system-x86_64 \
    -m 1G \
    -drive if=pflash,format=raw,readonly,file=$DEVENV_DIR/OVMF_CODE.fd \
    -drive if=pflash,format=raw,file=$DEVENV_DIR/OVMF_VARS.fd \
    -drive if=ide,index=0,media=disk,format=raw,file=$DISK_IMG \
    -device nec-usb-xhci,id=xhci \
    -device usb-mouse -device usb-kbd \
    -monitor stdio \
    $QEMU_OPTS

Summary

This is just for my own study, so I’m skipping a detailed explanation.

I’d like to keep going all the way to the end.

Published Sep 25, 2021

Aspiring Reverse Engineer and CTF Player (Team: 0nePadding). Passionate about WinDbg and Anti-Virus internals. OSCP / CISSP. Working at Microsoft Japan, but all views expressed are my own.かしわば(@kash1064) on Twitter