Hardware Interrupts

[utterances-bot]

This is a general purpose comment thread for the “Hardware Interrupts” post.

[ZerothLaw]

Hi, I'd be interested in writing the small scancode_set1 crate! You can reach me at trinioler at gmail.com if you want to continue the discussion out of band. :)

[phil-opp]

@ZerothLaw Awesome, I'll write you an email!

[katietheqt-old]

I’m following your tutorials, and (am currently) creating an API for device drivers. Thanks for the amazing work!

[phil-opp]

@rybot666 You're welcome! What are your plans for the API design?

[donald-pinckney]

I'm curious if you have any plans for how to do APIC, as it doesn't look like there is a crate for it.

[katietheqt-old]

@phil-opp I’m planning on using traits to implement it (I’ve got a Driver trait that holds info, and an InterruptHandler trait that allows drivers to register interrupt handlers). You can have a look, but the source code is a bit of a mess. The API for drivers is in master, because I forgot to make a PR. ;)

EDIT: Forgot the link

[thejpster]

I wrote a crate called pc-keyboard which handles raw access to a UK 105 key PS/2 keyboard generating scan code set 2. Happy for patches to make it work with set 1 and other keyboard layouts.

[phil-opp]

@thejpster Awesome! cc @ZerothLaw

[phil-opp]

@rybot666 Ah, so that devices can register and unregister for certain interrupts at runtime?

[phil-opp]

@donald-pinckney Nothing specific yet, just some vague ideas.

[katietheqt-old]

@phil-opp Exactly. Currently (as in right now) trying to get rid of errors about not being able to send Fn(...) types across threads

[phil-opp]

@rybot666 I think you need to add the Send bound to your register function to allow only function types that can be safely sent across threads.

[katietheqt-old]

@phil-opp Yea, I have and it seems to have fixed that. Although my current device isn't very good (an iPad and the Github editor).

[ZerothLaw]

I wrote a crate called pc-keyboard which handles raw access to a UK 105 key PS/2 keyboard generating scan code set 2. Happy for patches to make it work with set 1 and other keyboard layouts.

Hey @thejpster I'll see about sending you a PR with scancode set 1 sometime soon.

[npip99]

"As already mentioned, the 8259 APIC has been superseded by the APIC"

Perhaps "8259 PIC" was intended here?

[kennystrawnmusic]

Another bizarre thing I forgot to mention in the above comment is that the instruction pointer that triggers the fault keeps changing every time I make changes to the code for debugging reasons. Even more of a reason to suspect that the x86_64 crate is attempting to overwrite Index 16 in the error code selector table.

[kennystrawnmusic]

Another weird behavior: the fact that if my general protection fault handler doesn’t halt the CPU, an endless stream of general protection faults appears — even if a call to x86_64::instructions::interrupts::disable() is made from inside the general protection fault handler.

The only difference between my timer Interrupt handler and the timer Interrupt handler in this example is that it calls log::info! instead of the print macro defined in edition 2, but other than that it’s completely identical to this implenentation.

[ethindp]

@kennystrawnmusic This occurs generally for interrupt handlers on x86. If the exception is not handled appropriately, depending on the exception in question, a return from that exceptions associated interrupt handler will just cause a re-occurrence of the exception in question.

[kennystrawnmusic]

I'm well aware of that. What I don't get is the fact that outputting to the screen should be enough to handle the timer interrupt.

Instead, for some weird reason, one of the internals of either the x86_64 crate or the pic8259 crate is causing a #GP on attempt to handle it.

[kennystrawnmusic]

Decided to do a little inspection by running the following command to output the whole thing as an assembly file:

cargo rustc -Zbuild-std=core -Zbuild-std-features=compiler-builtins-mem --target=x86_64-unknown-example_os.json -- --emit=asm -o example_os.S

Opening that assembly file and using Ctrl-F to find the interrupt handler within it revealed something startling:

// snip
.Lfunc_begin88:
	.loc	1 64 0
	.cfi_startproc
	pushq	%r11
	.cfi_def_cfa_offset 16
	pushq	%r10
	.cfi_def_cfa_offset 24
	pushq	%r9
	.cfi_def_cfa_offset 32
	pushq	%r8
	.cfi_def_cfa_offset 40
	pushq	%rdi
	.cfi_def_cfa_offset 48
	pushq	%rsi
	.cfi_def_cfa_offset 56
	pushq	%rdx
	.cfi_def_cfa_offset 64
	pushq	%rcx
	.cfi_def_cfa_offset 72
	pushq	%rax
	.cfi_def_cfa_offset 80
	subq	$208, %rsp
	.cfi_def_cfa_offset 288
	.cfi_offset %rax, -80
	.cfi_offset %rcx, -72
	.cfi_offset %rdx, -64
	.cfi_offset %rsi, -56
	.cfi_offset %rdi, -48
	.cfi_offset %r8, -40
	.cfi_offset %r9, -32
	.cfi_offset %r10, -24
	.cfi_offset %r11, -16
//snip

It appears that the compiler is trying to output a context switch instead of a basic interrupt handler like the one in this example, despite the fact that, again, this example was, with the exception of using log::info! instead of print!, followed to the T.

Since the time of this writing, has the x86_64 crate been modified with some hidden asm! timer handler that automatically puts a context switch there by default without the developer knowing? Or is that in the bootloader crate? Or is something else going on?

[Maweypeyyu]

I just had a similar issue and finally figured out that I set up the log crate in my main.rs and had all other mods included via lib.rs (and then use blog_os::foo; in main.rs - except for my logger.rs). When I added a print! macro to the kernel that re-uses the logger (not the log crate), I noticed that I got kernel panics when trying to use it within interrupts. After some debugging I figured that Rust is doing something to the bin vs lib artifacts and treating statics (even if from the same file) as separate instances. The solution was to ensure that in main.rs everything goes through lib.rs. After removing mod logger; and using use blog_os::logger; from the lib the statics were still the same and no more kernel panics!

Just took 4 hours debugging 😄

[edg-l]

If you want to implement your own PS/2 keyboard controller, here you can find all the scancodes https://www.kbdlayout.info/

[iansjack]

@kennystrawnmusic - I am getting the same GPF as you. From my investigation so far it appears to he happening on the "iretq" instruction of any hardware interrupt. I've tested this by masking off all but the timer interrupt, setting a breakpoint on this handler in gdb and single-stepping the code. This confirms that the GPF occurs immediately upon the "iretq" instruction. This happens even with an empty handler and with the keyboard interrupt if I mask off all other interrupts. So I'm pretty sure the cause is the return instruction. in any hardware interrupt handler. (Probably with exception handlers too, but they never return - I might try creating a non-panicking exception handler to test this.)

But I can't figure out why! The error code (16) is valid - it indicates that the problem is with segment selector 0x10. But the qemu monitor shows this as being valid in all respects, as far as I can see. I suspect that the error is related to the stack, but I'm still trying to figure it out. I'll let you know if I have any success.

[iansjack]

Got it!

You need to add a kernel_data_selector in gdt.rs

lazy_static! {
static ref GDT: (GlobalDescriptorTable, Selectors) = {
let mut gdt = GlobalDescriptorTable::new();
let code_selector = gdt.add_entry(Descriptor::kernel_code_segment());
let data_selector = gdt.add_entry(Descriptor::kernel_data_segment());
let tss_selector = gdt.add_entry(Descriptor::tss_segment(&TSS));
(gdt, Selectors { code_selector, data_selector, tss_selector },)
};
}

struct Selectors {
code_selector: SegmentSelector,
data_selector: SegmentSelector,
tss_selector: SegmentSelector
}

It seems that, otherwise, the tss_segment_selector and data (and hence stack) selectors are the same, which causes the problem.

[kennystrawnmusic]

Initially thought the problem was a deadlock until I ruled that out through some code redistribution. I was able to find a way to store the mapper and frame allocator in statics by wrapping them in their own separate spinlocks in order to then access them in the lazy_static! locked LocalApic and move the code that does the actual memory allocation into the lazy_static! block that LOCAL_APIC is stored in but that made no difference:

//snip
pub static FRAME_ALLOCATOR: Mutex<Option<Falloc>> = Mutex::new(None);
pub static MAPPER: Mutex<Option<OffsetPageTable>> = Mutex::new(None);

pub fn maink(boot_info: &'static mut BootInfo) -> ! {
    //snip
    let offset = boot_info.physical_memory_offset.into_option().unwrap();

    let mapper = unsafe { page_tables::init(VirtAddr::new(offset) };
    let falloc = unsafe { Falloc::init(&boot_info.memory_regions) };

    // needed for frame allocation elsewhere in the kernel
    *MAPPER.lock() = Some(mapper);
    *FRAME_ALLOCATOR.lock() = Some(falloc);
    //snip
}

Still getting the one-IRQ-and-hang despite this, and again also despite calling the EOI function from within my APIC-specific timer handler as already explained in the comment immediately above this one. Does the IOAPIC also need to be enabled in addition to the local APIC in order for the EOI to work or something? @phil-opp?

[kennystrawnmusic]

Turns out the problem was with the way my IDT was configured — was using raw_cpuid::CpuId::new().get_feature_info() to determine which IRQs to map depending on whether the system supports the APIC or not. Unfortunately I made the mistake of using .has_x2apic() there instead of the correct .has_apic() to conduct that check. Once I changed this, the APIC timer began working normally.

In addition, I also had to move all the APIC-building-and-mapping code inside the heap initializer function to prevent deadlocks — and then get down and dirty in response to the resulting lack of a static to work off of:

pub fn raw_apic_eoi() {
    unsafe {
        let offset = xapic_base() + HEAP_START as u64 + 0xb0;
        *(offset as *mut u32) = 0;
    }
}

Granted this is a very crude and un-Rust-like way to go about things, but it was really the only solution I had to this problem.

[kennystrawnmusic]

Getting a completely unrelated #GP on every timer interrupt upon revisiting after a long hiatus that involved some data loss that forced me to start over from scratch, this time with an error code of 0 which rules out any relation to segments. What other causes would there be?

[kennystrawnmusic]

For further info, the stack frame for the GPF I'm getting now has the following values:

• Error code: 0 (see above)
• Instruction pointer — 0x3174D2
• Code segment: 8
• FLAGS: 0x206
• Stack pointer (at a new physical memory offset of 0x800000000000 this time — don't know if this has anything to do with it): 0x800000122F0
• Stack segment: also 16

Going to try changing the physical memory offset back to the value I had before I had to start over and see if that makes a difference, but why would it?

[kennystrawnmusic]

Also, has anyone else gotten a left: 3, right: 0 assertion failure originating from a dependency crate (presumably x86_64)? If so, how did you fix it? I was able to find a similar issue in a user-level that mentioned deleting Cargo.lock and cleaning but doing that here made no difference.

[iansjack]

Here’s the code from my get.rs file, which works for me:

…

--------------------------------------------------------------------------------------------------------------------- pub fn init() { use x86_64::instructions::tables::load_tss; use x86_64::registers::segmentation::{Segment, CS, DS}; GDT.0.load(); unsafe { CS::set_reg(GDT.1.code_selector); DS::set_reg(GDT.1.data_selector); load_tss(GDT.1.tss_selector); } } use x86_64::structures::gdt::{Descriptor, GlobalDescriptorTable, SegmentSelector}; lazy_static! { static ref GDT: (GlobalDescriptorTable, Selectors) = { let mut gdt = GlobalDescriptorTable::new(); let code_selector = gdt.add_entry(Descriptor::kernel_code_segment()); let data_selector = gdt.add_entry(Descriptor::kernel_data_segment()); let tss_selector = gdt.add_entry(Descriptor::tss_segment(&TSS)); let user_data_selector = gdt.add_entry(Descriptor::user_data_segment()); let user_code_selector = gdt.add_entry(Descriptor::user_code_segment()); ( gdt, Selectors { code_selector, data_selector, tss_selector, _user_data_selector: user_data_selector, _user_code_selector: user_code_selector, }, ) }; } struct Selectors { code_selector: SegmentSelector, data_selector: SegmentSelector, tss_selector: SegmentSelector, _user_data_selector: SegmentSelector, _user_code_selector: SegmentSelector, } -----------------------------------------------------------------------------------------------------------

On 30 Mar 2022, at 21:07, Kenny Strawn ***@***.***> wrote: The selector of the not-present error I get this time is 59, just for reference. Might be looking for some non-existent driver of some kind; not entirely sure though. — Reply to this email directly, view it on GitHub <#1003 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMYX4RE2TSY6HIP75J7FULVCSYB3ANCNFSM5DBWJIQA>. You are receiving this because you commented.

[kennystrawnmusic]

Even with those changes, it only works in BIOS mode, not UEFI mode. That’s what those replies dealt with: the new faults I’m getting are UEFI-specific.

[kennystrawnmusic]

Alright, so in summary, here's how I was able to get the APIC timer working properly, for those of you guys who are also looking to do likewise:

• Use the heap allocator initializer function (that is, alloc::init) to initialize and memory-map both the local APIC and the IOAPIC (using the x2apic crate) along with the heap
• Write a custom EOI sender that works around the lack of a global static instance by writing a zero through a raw pointer to the local APIC's memory-mapped address + 0xb0 (the EOI offset) — see my discussion with both @phil-opp and @iansjack above for an example of how to go about this
• Use that aforementioned custom end-of-interrupt sender to notify EOI on the timer, spurious, and APIC error handlers

A lot of work to say the least but it was definitely worth it.

[spanishpear]

Is your code in a repository somewhere? Would be nice to see :)

[AhmedAdel94]

Does anybody has the source code?

[SauravMaheshkar]

I'm fairly certain it's this: https://github.com/kennystrawnmusic/cryptos/blob/master/src/drivers/apic_impl.rs

[JulianFeinauer]

Excellent writeup @phil-opp! I wonder how one would change the settings of the timer? Is this memory mapped somewhere to be accessible?

[bjorn3]

According to https://wiki.osdev.org/Programmable_Interval_Timer you have to configure the PIT using I/O ports.

[JulianFeinauer]

Thanks @bjorn3 i managed to get it work now (via Ports) 👍

[kennystrawnmusic]

Had to restart development from scratch due to failure of the USB flash drive that I had all my code backed up to and now, when trying to get the APICs set up and doing everything I did before (including using the heap allocator function to properly memory-map both the LAPIC and the IOAPIC) I'm getting a panic with this message:

assertion failed (left == right) {
    left: 3
    right: 0
}

Since interrupt handlers are defined in the x86_64 crate and x2apic worked perfectly before, what else could possibly be creating these erroneous values?

[iansjack]

Kenny, I’m glad that solved your immediate problem. I’m afraid that I haven’t looked into the UEFI version as yet, so I can’t really help with that problem. It’s on my “to do” list, but I’m working on a disk driver, and then file system, at the moment. If I get time to look at the UEFI version, and can throw any light on that problem, I’ll let you know. Ian

…

On 7 Mar 2022, at 22:29, Kenny Strawn ***@***.***> wrote: Thanks, that did solve the issue with the legacy (BIOS) version. The UEFI version, however, is still faulting twice: first with a segment-not-present and then with a divide error (presumably the result of attempting to perform a division-type operation on a segment that doesn't exist). What segment is it looking for? I've got the CS, DS, and TSS loaded; what else needs to be present for the UEFI version to work? — Reply to this email directly, view it on GitHub <#1003 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMYX4Q2OLS66MXOVM7BTADU6Z7OFANCNFSM5DBWJIQA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you commented.

[piosystems]

@phil-opp any illustration on the way for APIC support? PIC does not work with UEFI. Or could you point me to an existing illustration of APIC for the bootloader/bootloader_api?

[RoyalFoxy]

I encountered an issue where the port used to listen to keyevents doesn't emit any events for the command key on macs. If anyone has any idea how to fix this then please tell me, Im genuinely confused to what it could be. I thought it could maybe be that qemu doesn't map the command key to the vm but I wouldn't know how to change that either :/

[JensLiu]

Hi everyone :-) I was wondering: would using PICS.lock() in interrupt handlers cause potential deadlock in nested interrupt?

[shabbyboy]

error[E0433]: failed to resolve: could not find instructions in x86_64
--> src/lib.rs:34:17
|
34 | use x86_64::instructions::port::Port;
| ^^^^^^^^^^^^ could not find instructions in x86_64

error[E0433]: failed to resolve: could not find instructions in x86_64
--> src/lib.rs:17:13
|
17 | x86_64::instructions::interrupts::enable();
| ^^^^^^^^^^^^ could not find instructions in x86_64

why could not find instructions in x86_64, Before this, it was fine, but suddenly it was not working

[AtomicGamer9523]

@shabbyboy check if the instructions feature is enabled for the crate

[Souvlaki42]

Many years have passed since this article came out but anyways. Am I the only here who gets the test_println_output function to time out in this form:

#[test_case]
fn test_println_output() {
    use core::fmt::Write;
    use x86_64::instructions::interrupts;

    let s = "Some test string that fits on a single line";
    interrupts::without_interrupts(|| {
        let mut writer = WRITER.lock();
        writeln!(writer, "\n{}", s).expect("writeln failed");
        for (i, c) in s.chars().enumerate() {
            let screen_char = WRITER.lock().buffer.chars[BUFFER_HEIGHT - 2][i].read();
            assert_eq!(char::from(screen_char.ascii_character), c);
        }
    });
}

But it runs fine like that:

#[test_case]
fn test_println_output() {
    let s = "Some test string that fits on a single line";
    println!("{}", s);
    for (i, c) in s.chars().enumerate() {
        let screen_char = WRITER.lock().buffer.chars[BUFFER_HEIGHT - 2][i].read();
        assert_eq!(char::from(screen_char.ascii_character), c);
    }
}

Even after implementing all the other changes made throughout this article?

[ChocolateLoverRaj]

Maybe that's happening because you are trying to lock the writer with WRITER.lock() for a second time, which will block forever because the first lock (let mut writer) doesn't get dropped until after the 2nd lock.

To fix the top example just replace the 2nd WRITER.lock() with writer.

[Souvlaki42]

Oh silly me, thank you very much for that.