ARM64 Kernel Booting Process
This document describes boot loader requirements to boot Kernel, ARM64 Virtual Memory
Layout, ARM64 IRQ Vectors Setup, FDT mapping and ARM64 Kernel booting process.
1. boot loader requirements to boot Kernel
Boot loader simply to define all software that executes on the CPU(s) before control is passed
to the Linux kernel. This may include secure monitor and hypervisor code, or it may just be a
handful of instructions for preparing a minimal boot environment.
Essentially, the boot loader should provide the following:
(1) Setup and initialize the RAM
(2) Setup the device tree
(3) Decompress the kernel image
(4) Call the kernel image
Setup and initialize the RAM.
Requirement: MANDATORY
The boot loader is expected to find and initialize all RAM that the kernel will use for volatile
data storage in the system. It performs this in a machine dependent manner. (It may use
internal algorithms to automatically locate and size all RAM, or it may use knowledge of
the RAM in the machine, or any other method the boot loader designer sees fit.)
Setup the device tree
Requirement: MANDATORY
The device tree blob (dtb) must be placed on an 8-byte boundary and must not exceed 2
megabytes in size. Since the dtb will be mapped cacheable using blocks of up to 2
megabytes in size, it must not be placed within any 2M region which must be mapped with
any specific attributes.
Decompress the kernel image
Requirement: OPTIONAL
The AArch64 kernel does not currently provide a decompressor and therefore requires
decompression (gzip etc.) to be performed by the boot loader if a compressed Image target
(e.g. Image.gz) is used. For bootloaders that do not implement this requirement, the
uncompressed Image target is available instead.
Call the kernel image
Requirement: MANDATORY
Kernel Image Header
Kernel Image Header: Flags
Example: lf 5.10.y Kernel Image
Other constrains before jumping into kernel
- Quiesce all DMA
- Primary CPU register settings
x0 = dt blob address in RAM
x1-x3 = 0
- MMU off, Instruction cache either on or off
……
Please refer to Documentation/arm64/booting.rst.
2. ARM64 Virtual Memory Layout
AArch64 Linux memory layout with 4KB pages + 4 levels (48-bit):
arch/arm64/kernel/vmlinux.lds.S
Note: kernel image address is started from vmalloc space
#define KIMAGE_VADDR (MODULES_END)
3. ARM64 IRQ Vectors Setup
Vectors objdump:
System IRQ handling Process
el1_irq()->irq_handler()->handle_arch_irq()
NOTE: handle_arch_irq is Top level irq for an ARCH.
For ARM64, usually set by system irqchip driver. e.g. gic
int __init set_handle_irq(void (*handle_irq)(struct pt_regs *))
{
if (handle_arch_irq)
return -EBUSY;
handle_arch_irq = handle_irq;
return 0;
}
drivers/irqchip/irq-gic-v3.c
gic_init_bases() -> set_handle_irq(gic_handle_irq);
gic_handle_irq() -> handle_domain_irq(irq_domain, hwirq, regs) -> irq handler or
route to the next irq_domain handler
4. FDT Mapping
• Page table: init_pg_dir
• VA: FIX_FDT
• PA: dt_phys passed by bootloader
• NOTE: can’t exceed 2M
• #define FIXADDR_TOP (PCI_IO_START - SZ_2M)
• #define __fix_to_virt(x) (FIXADDR_TOP - ((x) << PAGE_SHIFT))
• dt_virt_base = __fix_to_virt(FIX_FDT);
5. ARM64 Kernel booting process
5.1 Prior to start_kernel
First instruction in kernel
b primary_entry // branch to kernel start, magic
Major work prior to start kernel
5.1.1__create_page_tables
(1)Identity mapping for MMU enablement code
Page table: idmap_pg_dir (3 pages pre-allocated in vmlinux.lds.S)
VA: Runtime __pa of section ".idmap.text"
PA: Runtime __pa of section ".idmap.text“
Note: section ".idmap.text" includes MMU on code
(2) Kernel Image Mapping
Page table: init_pg_dir
VA: KIMAGE_VADDR / Compile time __va(text)
PA: Runtime __pa(_text) in DRAM
NOTE: ‘text’ section is in vmalloc address range
5.1.2 __cpu_setup
(1) Invalidate local TLB
(2) Disable PMU/AMU access from EL0
(3) Memory region attributes
(4) 48-bit address range and 4K page table setting
5.1.3 __primary_switch
(1)__enable_mmu
TTBR0: idmap_pg_dir
TTBR1: init_pg_dir
(2) Kernel image address randomization setting (KASLR)
(3) Setup kernel stack, thread_info/init_task
(4) Load VBAR_EL1 with virtual vector table address
(5) Calculate kimage_voffset for supporing __pa(x)/__pa_symbol(x)
(6) Clear BSS
(7) Create FDT mapping (see next slides)
(8) Call into start_kernel()
5.2 Start_kernel
(1) Architecture Setup (e.g. setup_arch())
(2) Memory Subsystem init
Memory zones
Memory buddy system
(3) Schedule init
(4) IRQ init
of_irq_init(__irqchip_of_table)
Driver: IRQCHIP_DECLARE(gic_v3, "arm,gic-v3", gic_of_init);
(5) Timer init
Clocks/clocksource/clockevent/cyclecounter register.
of_clk_init(NULL);
Driver: CLK_OF_DECLARE(imx7ulp_clk_scg1, "fsl,imx7ulp-scg1",
imx7ulp_clk_scg1_init);
TIMER_OF_DECLARE(armv8_arch_timer, "arm,armv8-timer",
arch_timer_of_init);
(6) Console init
(7) Other core functions init
E.g. cgroup_init() / kcsan_init()
(8) Reset Init
5.2.1 Start_kernel -> setup_arch
• early_ioremap_init for early users of early_ioremap(paddr, size)
• setup_machine_fdt
• parse_early_param
- early_param("mem", early_mem);
- early_param("earlycon", param_setup_earlycon);
- early_param("debug", debug_kernel);
• cpu_uninstall_idmap
• arm64_memblock_init
- Reserve memory used by kernel image
- Reserve memory specified in DT and specifical
initialization if any
e.g. RESERVEDMEM_OF_DECLARE(cma, "shared-dma-pool", rmem_cma_setup);
• unflatten_device_tree
paging_init and bootmem_init
5.2.1.1 Start_kernel -> setup_arch -> setup_machine_fdt
• Parse ‘bootargs’ from DT ‘chosen’ node
• Parse Physical Memory base and size, added into memblock subsystem
• Parse Machine model
5.2.1.2 Start_kernel -> setup_arch -> paging_init / bootmem_init
• Remap kernel sections _text, _rodata, _data and etc
with different permissions
• Linear mapping for available physical memory blocks
#define __phys_to_virt(x) ((unsigned long)((x) - PHYS_OFFSET) |
PAGE_OFFSET)
• Switch to swapper_pg_dir
• Build structure pages / vmemmap
− Sparse_init
• Build memory zones
− Usually only one DMA zone for ARM64
Page table dump after paging_init:
5.2.1.3 Start_kernel -> setup_arch -> psci_init
Firmware interface implementing CPU power related operations specified by ARM PSCI
spec
Including CPU_ON/OFF/SUSPNED/MIGRATION and etc.
5.2.2 Start_kernel -> Rest_init
• Populate the first three kernel threads
− PID 0 -> Idle thread per CPU
• cpu_startup_entry -> do_idle -> cpuidle_idle_call ->
cpuidle_enter ->
• cpuidle_enter_state (selected by governor) -> psci
suspend state (cpuidle-psci.c) -> TF-A
− PID 1 -> Init thread
• Keep running kernel_init() for the rest kernel initialization
work
− PID 2 -> kthreadd
• Used for kthread_run(hwrng_fillfn, NULL, "hwrng");
− Userspace dump
5.2.2.1 Start_kernel -> Rest_init -> kernel_init
• SMP init
− Bring up non-boot CPUs
bringup_nonboot_cpus -> cpuhp_up_callbacks ->
boot_secondary -> cpu_psci_cpu_boot -> TF-A
− Difference from Booting CPU
Mainly are same except no __create_page_tables
• do_initcalls
− Usually used for device and driver register
e.g. module_init()
− Call in the order below:
__initcall0_start,
…
__initcall7_start,
− Become invalid when build as module
All default to module_init()
− Platform devices populated in arch level
e.g.arch_initcall_sync(of_platform_default_populate_init);
• Filp_open(/dev/console)
• Mount rootfs in prepare_namespace
• Free init memory between __init_begin and __init_end
• Run the first userpace application in the order
execute_command. E.g. init=/bin/sh
CONFIG_DEFAULT_INIT
/sbin/init, /etc/init, /bin/init, /bin/sh
