{"componentChunkName":"component---src-templates-post-template-js","path":"/unix-xv6-006-kernel-main-03-en","result":{"data":{"markdownRemark":{"id":"134084b1-1ed1-59b0-b513-144869123820","html":"
\n

This page has been machine-translated from the original page.

\n
\n

Inspired by An Introduction to OS Code Reading: Learning Kernel Internals with UNIX V6, I’m reading xv6 OS.

\n

Because UNIX V6 itself does not run on x86 CPUs, I decided to read the source of kash1064/xv6-public: xv6 OS, a fork of the xv6 OS repository that makes UNIX V6 run on the x86 architecture.

\n

In the previous article, I looked at the behavior of the kvmalloc function and its page table allocation executed from main.

\n

This time, I will trace the behavior of the mpinit function.

\n\n

Table of Contents

\n\n

The mpinit function

\n

The mpinit function is the following function defined in mp.c.

\n

“mp” probably stands for multiprocessor; the function whose role is to detect other processors is mpinit.

\n
void mpinit(void)\n{\n  uchar *p, *e;\n  int ismp;\n  struct mp *mp;\n  struct mpconf *conf;\n  struct mpproc *proc;\n  struct mpioapic *ioapic;\n\n  if((conf = mpconfig(&mp)) == 0) panic(\"Expect to run on an SMP\");\n  ismp = 1;\n  lapic = (uint*)conf->lapicaddr;\n  for(p=(uchar*)(conf+1), e=(uchar*)conf+conf->length; p<e; ){\n    switch(*p){\n    case MPPROC:\n      proc = (struct mpproc*)p;\n      if(ncpu < NCPU) {\n        cpus[ncpu].apicid = proc->apicid;  // apicid may differ from ncpu\n        ncpu++;\n      }\n      p += sizeof(struct mpproc);\n      continue;\n    case MPIOAPIC:\n      ioapic = (struct mpioapic*)p;\n      ioapicid = ioapic->apicno;\n      p += sizeof(struct mpioapic);\n      continue;\n    case MPBUS:\n    case MPIOINTR:\n    case MPLINTR:\n      p += 8;\n      continue;\n    default:\n      ismp = 0;\n      break;\n    }\n  }\n  if(!ismp)\n    panic(\"Didn't find a suitable machine\");\n\n  if(mp->imcrp){\n    // Bochs doesn't support IMCR, so this doesn't run on Bochs.\n    // But it would on real hardware.\n    outb(0x22, 0x70);   // Select IMCR\n    outb(0x23, inb(0x23) | 1);  // Mask external interrupts.\n  }\n}
\n

Let’s read through the source code step by step.

\n

Declaring structure variables

\n

After the function call, several structure variables are declared.

\n
uchar *p, *e;\nint ismp;\n\nstruct mp *mp;\nstruct mpconf *conf;\nstruct mpproc *proc;\nstruct mpioapic *ioapic;
\n

All of these are defined in mp.h.

\n

The structure definitions are as follows. I will skip the details for now and cover them when the source code actually uses them.

\n
// See MultiProcessor Specification Version 1.[14]\n\nstruct mp {             // floating pointer\n  uchar signature[4];           // \"_MP_\"\n  void *physaddr;               // phys addr of MP config table\n  uchar length;                 // 1\n  uchar specrev;                // [14]\n  uchar checksum;               // all bytes must add up to 0\n  uchar type;                   // MP system config type\n  uchar imcrp;\n  uchar reserved[3];\n};\n\nstruct mpconf {         // configuration table header\n  uchar signature[4];           // \"PCMP\"\n  ushort length;                // total table length\n  uchar version;                // [14]\n  uchar checksum;               // all bytes must add up to 0\n  uchar product[20];            // product id\n  uint *oemtable;               // OEM table pointer\n  ushort oemlength;             // OEM table length\n  ushort entry;                 // entry count\n  uint *lapicaddr;              // address of local APIC\n  ushort xlength;               // extended table length\n  uchar xchecksum;              // extended table checksum\n  uchar reserved;\n};\n\nstruct mpproc {         // processor table entry\n  uchar type;                   // entry type (0)\n  uchar apicid;                 // local APIC id\n  uchar version;                // local APIC verison\n  uchar flags;                  // CPU flags\n    #define MPBOOT 0x02           // This proc is the bootstrap processor.\n  uchar signature[4];           // CPU signature\n  uint feature;                 // feature flags from CPUID instruction\n  uchar reserved[8];\n};\n\nstruct mpioapic {       // I/O APIC table entry\n  uchar type;                   // entry type (2)\n  uchar apicno;                 // I/O APIC id\n  uchar version;                // I/O APIC version\n  uchar flags;                  // I/O APIC flags\n  uint *addr;                  // I/O APIC address\n};
\n

About the MP specification

\n

Before reading the source code, let me summarize the MP specification.

\n

The MP table is a mechanism that allows an OS on an x86 CPU to obtain multiprocessor information.

\n

The MP table contains information related to the MP specification of x86 CPUs.

\n

The following diagram, from Intel’s documentation, shows the data structure of the MP specification.

\n

It depicts the FLOATING POINTER STRUCTURE pointing to the FIXED-LENGTH HEADER.

\n

\n \n \n \n \n \n \n \n \n

\n

Reference image: Intel MultiProcessor Specification | ManualsLib

\n

This FLOATING POINTER STRUCTURE is the MP Floating Pointer Structure, defined in xv6OS as the mp structure.

\n

If a system has an MP Floating Pointer Structure, it means the system conforms to the MP specification.

\n

The MP Floating Pointer Structure contains the following information:

\n\n

The MP Configuration Table is defined as the mpconf structure in xv6OS.

\n

The mpconfig function described later defines the processing that obtains the MP Configuration Table after retrieving the MP Floating Pointer Structure.

\n

Now, how does the OS find the MP Floating Pointer Structure? According to the Intel specification, the MP Floating Pointer Structure is defined to exist in one of the following locations, so the OS searches these locations to check whether one is present:

\n\n

In xv6OS, the mpsearch and mpsearch1 functions perform the search in the above regions.

\n

These functions are described below.

\n

Next, regarding the MP Configuration Table — it appears to be an optional component in the default configuration.

\n

If the system uses the default configuration, defining the MP Configuration Table is unnecessary; however, it becomes required when the number of CPUs may vary. (In practice it is effectively required for any general-purpose OS.)

\n

The MP Configuration Table contains configuration information about the APIC, processors, buses, and interrupts.

\n

A new term has appeared: APIC. This is the interrupt control mechanism used in Intel’s multiprocessor CPUs.

\n

I plan to look at APIC in more detail when configuring the interrupt controller in xv6OS.

\n

Reference: APIC - OSDev Wiki

\n

Since I could not find much useful information about the MP specification from web pages or books, reading Intel’s specification document directly is probably the fastest way to learn more.

\n

Reference: Chapter 4 MP Configuration Table; MP Configuration Data Structures - Intel MultiProcessor Specification [Page 37] | ManualsLib

\n

Obtaining the MP Floating Pointer Structure

\n

With that background, let’s look at the following code:

\n
if((conf = mpconfig(&mp)) == 0) panic(\"Expect to run on an SMP\");
\n

The mp structure is passed as an argument to mpconfig, and the return value is stored in conf, a variable of type mpconf.

\n

This line initializes the mpconf variable conf and also checks whether the system is running on an SMP.

\n

SMP stands for Symmetric Multiprocessing (or Shared-Memory Multiprocessing) — essentially a multiprocessor system in which multiple CPUs share memory resources.

\n

Reference: Symmetric multiprocessing - Wikipedia

\n

Reference: Symmetric Multiprocessing - OSDev Wiki

\n

Let’s look at the source code of mpconfig.

\n

mpconfig takes the address of the mp structure object declared in mpinit as its argument, and returns an mpconf structure.

\n

This function searches for the MP table and initializes both the mp structure object passed as the argument and the returned mpconf structure.

\n
// Search for an MP configuration table.  For now,\n// don't accept the default configurations (physaddr == 0).\n// Check for correct signature, calculate the checksum and,\n// if correct, check the version.\n// To do: check extended table checksum.\nstatic struct mpconf* mpconfig(struct mp **pmp)\n{\n  struct mpconf *conf;\n  struct mp *mp;\n\n  if((mp = mpsearch()) == 0 || mp->physaddr == 0) return 0;\n  conf = (struct mpconf*) P2V((uint) mp->physaddr);\n  if(memcmp(conf, \"PCMP\", 4) != 0) return 0;\n  if(conf->version != 1 && conf->version != 4) return 0;\n  if(sum((uchar*)conf, conf->length) != 0) return 0;\n  *pmp = mp;\n  return conf;\n}
\n

The mp structure refers to the MP Floating Pointer Structure described above.

\n

At the point mpconfig is called, the system’s MP Floating Pointer Structure has not yet been obtained, so the search must be performed first.

\n

This is done by calling mpsearch and mpsearch1.

\n
// Look for an MP structure in the len bytes at addr.\nstatic struct mp* mpsearch1(uint a, int len)\n{\n  uchar *e, *p, *addr;\n  addr = P2V(a);\n  e = addr+len;\n  for(p = addr; p < e; p += sizeof(struct mp))\n  {\n    if(memcmp(p, \"_MP_\", 4) == 0 && sum(p, sizeof(struct mp)) == 0) return (struct mp*)p;\n  }\n  return 0;\n}\n\n// Search for the MP Floating Pointer Structure, which according to the\n// spec is in one of the following three locations:\n// 1) in the first KB of the EBDA;\n// 2) in the last KB of system base memory;\n// 3) in the BIOS ROM between 0xE0000 and 0xFFFFF.\nstatic struct mp* mpsearch(void)\n{\n  uchar *bda;\n  uint p;\n  struct mp *mp;\n\n  bda = (uchar *) P2V(0x400);\n  if((p = ((bda[0x0F]<<8)| bda[0x0E]) << 4)){\n    if((mp = mpsearch1(p, 1024))) return mp;\n  } else {\n    p = ((bda[0x14]<<8)|bda[0x13])*1024;\n    if((mp = mpsearch1(p-1024, 1024))) return mp;\n  }\n  return mpsearch1(0xF0000, 0x10000);\n}
\n

As noted above, if the MP Floating Pointer Structure is present, it is located in one of the following:

\n\n

These areas are searched, and if the MP Floating Pointer Structure is found, it is stored in mp.

\n

If the MP Floating Pointer Structure is not found, or if the address of the MP Configuration Table held by the MP Floating Pointer Structure is empty, the kernel terminates.

\n
if((mp = mpsearch()) == 0 || mp->physaddr == 0) return 0;
\n

The MP Floating Pointer Structure has the following layout:

\n
struct mp {             // floating pointer\n  uchar signature[4];           // \"_MP_\"\n  void *physaddr;               // phys addr of MP config table\n  uchar length;                 // 1\n  uchar specrev;                // [14]\n  uchar checksum;               // all bytes must add up to 0\n  uchar type;                   // MP system config type\n  uchar imcrp;\n  uchar reserved[3];\n};
\n

The definition of the mp structure is as above, but the diagram from the Intel specification is easier to visualize, so I am including it here.

\n

\n \n \n \n \n \n \n \n \n

\n

Reference image: Intel MultiProcessor Specification | ManualsLib

\n

The first 4 bytes of SIGNATURE are expected to contain _MP_.

\n

When mpsearch and mpsearch1 perform their search, they look for this SIGNATURE.

\n

physaddr holds the address of the MP Configuration Table; from here we use that information to obtain the MP Configuration Table.

\n

Obtaining the MP Configuration Table

\n

After obtaining the MP Floating Pointer Structure, the virtual address of the MP Configuration Table is retrieved and stored as the pointer variable conf of type mpconf.

\n
struct mpconf *conf;\nconf = (struct mpconf*) P2V((uint) mp->physaddr);\n\nif(memcmp(conf, \"PCMP\", 4) != 0) return 0;\nif(conf->version != 1 && conf->version != 4) return 0;\nif(sum((uchar*)conf, conf->length) != 0) return 0;
\n

The MP Configuration Table has the following structure:

\n
struct mpconf {         // configuration table header\n  uchar signature[4];           // \"PCMP\"\n  ushort length;                // total table length\n  uchar version;                // [14]\n  uchar checksum;               // all bytes must add up to 0\n  uchar product[20];            // product id\n  uint *oemtable;               // OEM table pointer\n  ushort oemlength;             // OEM table length\n  ushort entry;                 // entry count\n  uint *lapicaddr;              // address of local APIC\n  ushort xlength;               // extended table length\n  uchar xchecksum;              // extended table checksum\n  uchar reserved;\n};
\n

The following is a structural diagram quoted from the Intel specification.

\n

\n \n \n \n \n \n \n \n \n

\n

Reference image: Intel MultiProcessor Specification | ManualsLib

\n

The first 4 bytes hold the SIGNATURE, which is expected to be PCMP.

\n

In xv6OS, the memcmp function checks whether the first 4 bytes of the retrieved MP Configuration Table match PCMP.

\n
if(memcmp(conf, \"PCMP\", 4) != 0) return 0;\nif(conf->version != 1 && conf->version != 4) return 0;\nif(sum((uchar*)conf, conf->length) != 0) return 0;
\n

It also checks whether the version information is appropriate and whether the data size matches the actual size.

\n

With this, the following processing in mpinit is complete, and both the MP Floating Pointer Structure and the MP Configuration Table have been obtained.

\n
if((conf = mpconfig(&mp)) == 0) panic(\"Expect to run on an SMP\");
\n

Obtaining the IOAPIC from the MP Configuration Table

\n

Next, the address obtained from lapicaddr in the MP Configuration Table is stored into lapic, and ismp is initialized.

\n
int ismp;\nstruct mpioapic *ioapic;\n\nismp = 1;\nlapic = (uint*)conf->lapicaddr;
\n

The mpioapic structure is as follows:

\n
struct mpioapic {       // I/O APIC table entry\n  uchar type;                   // entry type (2)\n  uchar apicno;                 // I/O APIC id\n  uchar version;                // I/O APIC version\n  uchar flags;                  // I/O APIC flags\n  uint *addr;                  // I/O APIC address\n};
\n

The Intel specification diagram is as follows.

\n

\n \n \n \n \n \n \n \n \n

\n

Reference image: Intel MultiProcessor Specification | ManualsLib

\n

The IOAPIC is a mechanism that distributes external interrupts among multiple CPUs.

\n

I mentioned earlier that APIC is an external interrupt mechanism; there appear to be two types of APIC: the Local APIC and the IOAPIC.

\n

In xv6OS, the Local APIC is implemented in lapic.c, and the IOAPIC is implemented in ioapic.c.

\n

The Local APIC handles interrupts built into the CPU, while the IOAPIC receives interrupts from I/O devices and notifies the CPU based on information in the Redirection Table.

\n

The IOAPIC has an IOAPIC table, and x86 CPUs can define entries in this table through memory-mapped I/O.

\n

In simple terms, memory-mapped I/O is one way to perform input and output between the CPU and I/O devices: a region of the physical address space is reserved for I/O device input and output, and data is exchanged using the CPU’s memory read/write capabilities.

\n

Reference: About I/O APIC - 睡分不足

\n

Reference: Memory-mapped I/O - Wikipedia

\n

In a typical system with PCI I/O devices, the IOAPIC detects changes in PCI interrupt signals and issues interrupt messages to the CPU based on the Redirection Table.

\n

The Local APIC inside the CPU receives this information, calls the interrupt handler to process the interrupt, and then sends an EOI (End of Interrupt) command back to the IOAPIC to notify it that the interrupt has been handled.

\n

I will cover this in more detail when I get to actually implementing interrupt handling. Probably.

\n

Reference: APIC - Wikipedia

\n

Reference: APIC - OSDev Wiki

\n

Reference: IOAPIC - OSDev Wiki

\n

Reference: 82093AA I/O ADVANCED PROGRAMMABLE INTERRUPT CONTROLLER (IOAPIC)

\n

The APIC mechanism was born from the need for an interrupt controller capable of handling interrupts in a multiprocessor environment such as x86 CPUs and their motherboards.

\n

It seems the earlier, simpler interrupt mechanism called the PIC could not handle interrupt processing in multiprocessor configurations.

\n

Because xv6OS assumes a multiprocessor configuration, it ignores interrupts from the PIC and implements interrupt processing using the Local APIC and IOAPIC.

\n

Reference: P45

\n

In the xv6OS code, the address obtained from lapicaddr in the MP Configuration Table is stored into the global variable lapic.

\n

The value stored in lapicaddr is the address of the memory-mapped Local APIC.

\n

The variable lapic is defined as a global variable in defs.h.

\n

It is not used further within this function; it will be used in lapic.c.

\n
extern volatile uint*    lapic;
\n

Obtaining processor information

\n

Let’s look at the loop that comes after obtaining lapicaddr.

\n

conf holds the MP Configuration Table retrieved earlier.

\n

According to Intel’s specification, MP Configuration Table entries follow the MP Configuration Table header (the starting address of the MP Configuration Table) in variable numbers.

\n

In addition to the Processor Entries shown earlier, MP Configuration Table entries include Bus Entry, I/O APIC Entry, I/O Interrupt Entry, and Local Interrupt Entry. (There are also extended entries.)

\n

Each of these has a unique Entry Type defined in its first byte.

\n\n

\n \n \n \n \n \n \n \n \n

\n\n

\n \n \n \n \n \n \n \n \n

\n\n

\n \n