How Software Deals with Key Presses - Computerphile
ฝัง
- เผยแพร่เมื่อ 8 ส.ค. 2018
- Once you've pressed a key on the keyboard and the signal gets to the computer, what happens next? Dr Steve Bagley on the software side of a key-press.
Key Presses pt1: • What Happens When I Pr...
/ computerphile
/ computer_phile
This video was filmed and edited by Sean Riley.
Computer Science at the University of Nottingham: bit.ly/nottscomputer
Computerphile is a sister project to Brady Haran's Numberphile. More at www.bradyharan.com
10:38 All keys are equal, but some keys are more equal than others.
All my keys are black
4:05 Not just a security issue: think about languages that require a front-end processor (e.g. Chinese/Japanese/Korean) to translate keystrokes into a large character set: putting this in userland makes it more flexible, replaceable, configurable etc.
At the hardware level on the keyboard, not all keys are treated equal. That's because most modifier keys are often held down while another key is pressed, and if you have too many at once, that could cause more of a path than a single intersection in the matrix. On gaming keyboards, there might be special care on further keys, such as WASD, in order to detect them better.
That was discussed in part 1. It’s why the matrix is scanned, not simply fed with static voltages.
@@lawrencedoliveiro9104 when you have a simple XY (X scanning, Y receiving) matrix of just switches, closing switches of keys (0, 0), (0, 1) and (1, 1) will result in logical 1 on (1, 0) position as well, because Y line 0 will be tied up high to logical 1 voltage from X line 1 to Y 1 line via (1, 1), then from Y 1 line to X 0 line via (0, 1), then from X 0 line to Y 0 line via (0, 0).
Therefore you'll have logical 1 on both lines Y 0 and Y 1, while you expected to have logical 1 on only Y 1, since the (1, 0) button isn't pressed, but this «phantom» keypress will be there. But better not trust me, just go draw it for yourself!
This is why they either add diodes to prevent reverse current flow, or separate them better, or do both.
Probably not gonna write a program like that, but you could.
Me:
Opens up the code editor
Name your project: Tom and Jerry keyboard tool.c
my keypresses go thru so much refinement, and yet my posts are still terrible
This one wasn't.
You probably have a bug - or even a design flaw - in your message-filtering loop.
Make sure not to flip too many bits and suppress the event-bubble, otherwise event-storms might cause stutter ...
I’d chalk that up to user error.
Yeah, the root-cause: PEBKAC
Don't keep this video on "hidden" please
Two things not addressed were: how key bounce is handled and how key release/repeat is handled. A long time ago, I wrote a GUI for DOS that looked a lot like the NetWare interface. The keyboard was relatively easy to handle. The complex part was handling keyboard/mouse interactions.
The IBM model M gave special treatment to the modifier keys. It used a matrix where pressing 3 or more keys would sometimes not register all of them properly. Since the modifier keys are by design intended to be pressed with other keys, they were made immune to that issue with a dedicated line.
as far as i know, many (most?) keyboards do that, not just IBM Model Ms
That's not really "different treatment" as much as "different wiring". A keypress is still just a keypress.
Yeah, would have been nice if they'd covered the 'N-key rollover' issue in the hardware edition. There's still limitations on many keyboards and wiring can be very era-dependent. E.g. once WASD controls for gaming became popular, practically all keyboard manufacturers ensured that you could press those keys, along with e.g. shift, alt, ctrl, space (activation/jump buttons) and a whole bunch more at the same time and have them recognized at the same time. Maybe they can do a short video on that as an addendum, though I'm sure there's tons of videos on the subject already.
The handling of keys pressed by the window server is an iterresting point as this is where window focus stealing can have some side effects. Like typing in a password and whilst some application loads (like a texteditor of word) and it’s main window pops up in the front and your keystrokes get sent to word instead of the password dialogue box. This is a serious issue that still hasn’t been fixed in windows and would make a nice suggestion for a computerphile video.
There is a fair amount of work in some desktop environments to prevent focus stealing.
DEs like KDE and GNOME (and, I assume, LXDE, XFCE and others) expose an input setting called _Focus stealing prevention,_ which allows you to keep focus in a window if you're actively typing into it or have your mouse over it - or even at all times! - and is implemented by the X Window System or Wayland display server.
In Windows Vista or later, if you're typing into the secure desktop implemented by _User Account Control,_ no user program can steal the focus from that *at all.* The reverse problem, which is for _User Account Control_ to receive clicks that were meant for a user program and happen to be right above the Yes button in the prompt, has a mitigation: if you were clicking or typing, the secure desktop doesn't pop up, and the window that requested it merely flashes to request your attention. The secure desktop pops up when you next click its task bar entry.
That doesn't help user programs' password prompts, but it's a start!
The problem that I often experience in windows 7 and 10 is that a splash screen of an application steals focus from an application that is already running. I often start multiple programs in sequence, for instance Excel, Autodesk revit and Autocad. Autocad has a command line and when I'm working in Word I let autocad start in the background. After a while the splash screen appears and it steals focus. It then bring Word back on top but once autocad has completely loaded, Word loses focus and everthing I type gets passed to autocad's command line. This is such a fundamental flaw in windows that they've never fixed and I doubt if they ever will, but it is super annoying. Does the secure desktop only work on enteripse versions of windows or does that also work on the home edition?
dykodesigns2yt: "Does the secure desktop only work on enterprise versions of Windows [...]?"
The secure desktop is the dimmed version of the desktop where only security-sensitive UAC prompts appear and anything else becomes temporarily unresponsive. You can't exactly make use of it in user programs like Word and AutoCAD to work around splash screens' focus stealing, unfortunately.
this is really cool and helpful cause Im studying for my exam atm, thank you!
Here's how keys are handled on Linux:
Low level kernel
- Interrupt handler
Kernel modules/device drivers (for USB)
- uhci_hcd or ohci_hcd or ehci_hcd
- usbcore
- usbhid or usbmouse + usbkbd
Kernel modules/device drivers (for IBM PS/2 mini-DIN and AT connectors)
- atkbd
Kernel subsystem
- ioctls
- evdev
User-space
- libevdev (library for device events)
Window server
- xf86-input-evdev (direct input handler) or
- libinput -> xf86-input-libinput (libinput via X) or
- libinput -> Weston (libinput via Wayland)
For X:
- X window system
- X client (Xlib/XCB or toolkit like Qt)
For Wayland
- Weston
- Wayland client
Other software was involved like udev to detect which device to use and extensions like X keyboard extension (Xkb) to interpret key codes which is shared by X and Wayland.
Operating systems are these ingenious pieces of software that make sure you can type at least half a sentence before realizing that your key presses go to absolutely none of the windows on your screen at the moment.
Re “are all keys treated equally?”: In the USB HID protocol, modifier keys are handled in a special way: their state is sent as a bitmap (8 modifiers in one byte). For all other keys, the keyboard sends a list of the keys that are currently pressed.
10:05 With Linux, was that measured in a GUI window (which GUI?) or in a text console?
Om modern PCs (Since PC/AT) the lights are handled by the computer. On the original IBM PC there were no lights and the interface was unidirectional. I had a clone keyboard that had lights and they were handled by the keyboard. The problem was that if the code for example numlock on (many calculator programs did) the lights got out of sync so I ended up covering the light.
This is brilliant!!
I always test if the computer is hanged if when i press the num lock key if the led responds or not
20 years in IT and that trick hasn't failed me yet.
I recently had that fail: the keyboard controller on my laptop stopped working, so numlock didn't get detected, but the USB system was still working, so I could move the cursor with my trackball. I didn't figure out what was actually wrong, but my guess was that the kernel decided the interrupt was stuck and started ignoring it.
All bets are off on laptops. You never really know how they are connected internally (USB, PS/2, SPI, I2C)?
This is why I have always set my firmware to activate the num lock LED on boot.
I would like to develop a language pack, then add an engine work in the background, which will works like a sentences corrector, all what I found is about how to install a language pack, any idea where to start ? .
I have a spesific question about keyboards. If we want to add a special key which is not found on a standart keyboard such as alpha(latin letter)to our keyboard, how can do it that? If you explain that I would be so grateful.
I suspect a lot of the increased latency is down to font rendering and window redraws. Given that older systems had hardware text screens that you just had to put a character code into a memory location to have a letter show up on screen, that would allow for much faster processing than on modern bitmap displays.
Of course, there's also more latency when you start throwing in things like USB stacks instead of the PS2 connection, and several layers of handlers (device drivers, window systems, etc.) and more complex key code handling (compose keys, etc.).
It would be interesting to compare a normal computer and keyboard with a motherboard built for gaming (with dedicated USB ports) and a gaming grade keyboard, maybe all us gamers have been even more scammed then we know :-)
Keltheran That would be interesting. I do expect most of the latency is in the comms (USB/PS2/etc.) up through the operating system (drivers, GUI, libraries). There is one thing about a lot of gaming keyboards, though. They take special pains to allow more combinations of simulateous keypresses to work properly, and possibly a higher number of simultaneous keypresses overall. I doubt there's much overall improvement in keypress detection latency in the keyoard itself compared to non-gaming keyboards, though.
Latency from when the signal is received is one thing that likely won't change, but the ability to support increased poll rates might make a difference? Just in decreasing the time before the signal even gets "sent".
I personally don't notice with a keyboard, but I do with mice.
I tried setting my mouse's poll rate to the minimum, which generic mice use, and it felt like I had path smoothing turned on.
It was most noticeable because it was a "no acceleration" mouse, which generic mice don't tend to be either (although you can change the registry or equivalent to remove acceleration, the sensors and controllers in the mice still has some.).
I wonder what the equivalent to max poll rate was for PS/2 mice, given that that's interrupt based. There must be a practical limit to that, whether it be CPU speed or microcontroller speed, or sensitivity of the rollers the ball moves in, etc. Hmm.
I find the increased latency thing fascinating (in the horribly fascinating sense) - that it increases even when the processor, bus speeds etc. increase by orders of magnitude. Boot times also seem mind-bogglingly slow, relatively speaking.
Is it possible to get some idea about ghost pressing of keys and the love for mechanical keyboards?
On the PC, keyboard interrupts only occur with PS/2 keyboards. For USB keyboards the kernel has to poll the keyboard for key presses.
Tom & Jerry programmes coming up in 3, 2, 1...
hey how are PS/2 and usb handled differently b/c I have noticed that PS/2 is slightly more reliable and I thought it had to do with interrupts but this vid kind of points otherwise?
ps2 is dedicated for inputs. usb is more flexible. so yeah, pro's and con's
Connor Calarco the Ps/2 can send interrupts and the Usb has to be checked, which is also why you can have a polling rate setting
PS/2 is like calling someone you have a message for on the phone. It will ring on their end and they'll hopefully pick it up. USB is like writing a book with one of the pages containing "whoever reads this please take out an ad in a newspaper containing the instructions to copy the message that follows and to mail it as a letter to the recipient mentioned at the end", leaving the book on the shelf of a library and walking away.
What about IRQs? When we press a key or move the mouse, is it true that the computer does nothing else than processing this input data?
is there still a keyboard interrupt? for PS/2 perhaps but I was under the impression that USB has to be polled constantly for those types of events
Yes and no. The traffic on the cable, that is between SUB controller and USB device, is handled in a polling way. However, the communication between the USB controller and the CPU is not. There the controller sends an interrupt whenever it has data for the cpu. (Or not, there's nothing stopping anyone from making an USB controller that has to be polled, or even using one that can do interrupts with polling software...)
What happens to programs that can accept 'global' key presses when they are not the current window, does the key get sent to the current window first and processed there, and then get polled across any other interested services/programs to see whether those need to process the key too?
Global hotkeys are processed by what Dr. Bagley calls the _window server_ in this video. A user program registers its interest in receiving messages for global hotkeys, and the window server redirects the messages to that program (and possibly to the active window too). That's how your media player grabs the media keys (Play, Pause, Stop, Previous/Next track) regardless of the active window.
So at what level is Ctrl-Alt-Del or local equivalent processed in modern computers? I seem to recall that in older PCs this was handled inside the keyboard, which was just one of the reasons that a PC would refuse to operate without a keyboard attached…
Maybe an interrupt with a higher priority.
In Windows that's the 'secure attention sequence'. It's handled by the operating system, and its supposed to give some security because it can never be handled by an application. Therefore anything that reacts to ctrl-alt-del is presumably a genuine part of windows and you should be able to trust it with typing in your windows password.
On Windows is a special sequence and is handled in the kernel.
It was not handled in hardware on PCs, but it was on Amiga computers (well, control-amiga-amiga was). On PCs it is handled by software, whether firmware (BIOS/EFI) or operating system. Microsoft's official excuse for training people to hit that just to login or unlock a machine is that it's always handled by the OS, which is nearly true (unless a program manages to disconnect the keyboard driver itself), but likely the real reason is that it would make other OSes appear less stable if you happened to hit it on the wrong keyboard.
CTRL + ALT + DEL, ALT + SysRq, etc. are handled at kernel level.
I recall reading that at least on IBM PC the insert key did not have hardware repeat. I think the FN key is handled internally by the keyboard. The keyboard then interprets the combination and sends it to the computer. I for example have a keyboard where FN Down is PgDn. If I press these a keystroke monitor shows that I press just PgDn.
Yes, FN is handled in the keyboard's hardware.
Do applications really need to run code in a loop to pull keyboard events off the queue? I though that when they start up they would just pass the window manager a pointer to a function that will deal with key events, and then when the window manager decides to send them a key event it would wake them up from an idle state and put the execution point at the start of that function.
Short answer: Yes they do.
Longer answer: In some languages like Java they have a built in keyboard manager that runs in a separate thread so you as a programmer don't have to do the looping yourself (it is still there though just hidden behind a library) but in lower level languages like C/C++ you will most likely need to do it yourself (It is often also more speed effective since you can decide what keys you need to check and skip the rest and I can't really see any reason why you would use C/C++ unless efficiency is of the essence (like in games or big servers like facebook and google) )
Keltheran not at all true, it highly depends on your library, Qt also shadows handling those events, but some others might not depending on many factors. Also there's reasons to use C(++) outside of efficiency
This is an operating system architecture question. Your program could be designed as a set of event handlers, and not actually have a running thread unless it is processing an event, but most operating systems dedicate one or more threads to any process, and one of those threads will then have the task of collecting events. It's essentially a pull architecture instead of push.
Barney Laurance hi. Some languages and op sys's may screw up and restrict the user, but in case it's of interest, the very "primitive" Atmel 8 bit chips don't ( nor do many others, I reckon). The user program can be entirely interrupt driven with the "main loop" simply saying, in effect, "sleep until interrupt". On chip peripherals stay awake but only trigger a response to a state change if the user app has asked for it (setup), the cpu ignores them otherwise. This is significant for micro and pico power apps where polling squanders energy.
It really depends on the operating system, but let's look at this for Windows: Yes. At the lowest level there is an event queue for every program and that program has to pull messages from that queue all the time. Note that that's not a keyboard queue, but a queue for all kinds of events. Keyboard, mouse, window redraw, etc.
But virtually nobody codes at that low level anymore. You use some kind of framework to make your job easier, and those all handle that polling and distribute those events to the parts of your code that are interested in them. (And make the difference between the program being 14kB and 14Mb big.)
I would have to look it up (it has been over 20 years afterall), but I think Windows has a "sleep while the queue is empty" system call, so your program isn't actually wasting cpu cycles while nothing is happening.
A very nice continuation of this video will be about processor modes or windows data structures. Oh btw i like his shirt.
Computerphile, can you do more videos on the Linux OS? Or maybe how they are setup, etc. Or do a video on how duel-booting works (like a full in-depth explanation on what happens behind the scenes) Those are just my suggestions
Dual booting really isn't complicated enough to warrant it. It's literally a case of the bootloader (GRUB, Windows Boot Manager, etc.) doing very similar to you finding the right program to run
Thomas Galea oh well. It was worth a shot at suggesting it :)
Thomas key presses could be described in the same simplistic way. What's the difference between GRUB and Windows Boot Manager? How do I stop a new Windows installation from overwriting my GRUB boot loader?
Please do a video like this on processor and how it works please..
what will happen if you press a key and disconnect the keyboard without releasing it?
Key pressed message, followed by keyboard device disconnected message, and then a pain message from your back as you probably pulled something trying to hold down a key and unplug the keyboard at the same time.
With PS/2 keyboards, it would keep "sending" the character (as in, it would continue to appear in the document). I did this when I was very young. It also made lots of beeps come out of the PC beeper, much like holding keys down on boot. (This is because PS/2 wasn't plug and play, so it also wouldn't be able to tell if you plugged it back in without restarting. So it couldn't just say "oh, the keyboard is gone now", like a USB keyboard can.)
Global mass extinction.
This is a little tricky. In layers which observe press/release events only, like many operating systems and programs, the key would appear still held. For a hotpluggable keyboard type like USB, the keyboard removal is a detectable event and should lead to automatic release of all keys that were held there. For a PS/2 keyboard, the key might appear still held but not repeating, as repetition can be done in the keyboard; many operating systems perform repetition in software anyway, and then it might continue to repeat.
deltA Emreald on the lowest level, nothing really, on the higher level just an error after disconnect
USBHID standard uses interrupts? I was under the impression that they do not use hardware interrupts.
USB doesn't use interrupts; this guy is wrong.
USB HID is polled, hence polling rates.
I think the operating system must send events even if just a modifier key is pressed, or at least have the ability to do it sometimes. In Jetbrains IDEs you can open a search box by pressing shift twice.
Yeah, I'm not sure if the event is sent and the program ignores it, or if the event is only sent if the program asks it to be. The latter would probably be a tad more efficient.
On Windows, you have two events, Key Press and Key Release, and your program decides what to do.
Yes, shift keys get press/release events just like other keys. Since they're meant to be held they also frequently have dedicated lines (such as being outside the scanning matrix, or having dedicated bits in a HID report). In the OS level, they'll activate keyboard mapping states such as capitals for decoding other keys, but not decode to printable letters themselves. So a program waiting for text won't notice them, but a program looking at held keys will.
Cool. Thanks for the explanation!
That's unfortunate. The shift key by itself ideally shouldn't do anything. When I'm waiting for the computer to finish doing something, I like to tap out various rhythms on one or both of the shift keys.
Well now we need a video on n-key rollover.
xev shows if modifier keys are pressed too.
I wondered if there'd be a discussion of debouncing. I guess not, but that's kind of a specialized problem, and not really in scope.
At what stage do most keyloggers detect keypresses?
What about ctrl+alt+del?
Mesmerising
- The short answer is yes. While the long answer is no.
While, actually, «yes» is 50% longer than «no» is.
What happens to the program?
"Are all keys treated equally?" - missed opportunity to talk about Ctrl+Alt+Delete.
Magic SysRq key commands are handled at the kernel level
"Interrupt".
"USB".
Pick one.
A fancy label for what is actually continuous polling from the USB host.
You didn't mention the queue (or did i just miss it?)
usually, when your program reads a key from the keyboard it'll take it off the queue.
you can however also do it in such a way that it stays on there, which in some situations can make sense.
however, if the queue is full, all additional keypresses will be ignored until there again is room in the queue.
that used to be a huge problem in earlier times, today it almost never is, because the queue is usually much larger plus computers are much faster.
10:25
- are all keys treated equally?
- the short answer is: yes; the long answer is: no.
"yes" is longer than "no". contradictory statement detected
:)
Shanene Larissa Nerdy
This is wrong on so many levels 🤣🔪
The long answer is "No. ... It depends where you get up to..."
Okay, lemme get this to the next meta level: You _could_ argue that "no" averagely needs more time to be spoken out then "yes", so then the statement would be correct ;)
..okay okay I know where the exit is!!
well, but "short" is actually longer than "long", so... :))
BIOS and polling work too.
Glossing over hardware interrupts... which is really one of the most critical parts of the chain.
It is indeed critical, but not all keyboards will trigger an interrupt, and there are so many ways for the communication to happen. It would have been foolish to focus solely on interrupts when (polled) USB keyboards are so common, and showing more possible communication methods would have taken too long.
I really like, "Short answer: Yes. Long Answer: No". :D
"Is life simple?"
@@Robstafarian
It depends.
@@Robstafarian
It depends.
You left him hanging by not adding a link where he pointed early in the video.
I am inferring things from the last line. You do seem a bit like a +2 owner
What causes the input to be blocked if for example 3 letter keys are pressed simultaneously? Is this Windows or the hardware?
I'd blame hardware before software for the "key overload".
kranser: Multiple inputs on one of the keyboard's "lines" causing ambiguity as to which keys are actually pressed. See the previous video, which discusses the hardware side of keyboards. :)
He describes the typical event loop at 7:00, but on Plan 9 and Inferno it's just
- open your window's equivalent of /dev/tty
- by equivalent to /dev/tty we mean it blocks when there's no new input
- but we're reading it from a dedicated thread, so it doesn't block the whole program
- the thread communicates the keypresses on what Go, as well as Plan 9, calls a channel
- no need to invent silly extra things like non-blocking IO.
10:54 L.E.D. pronounced the french way "lead" is a little ambiguous in English with the verb "to lead" and the 82nd element of the periodic table "Pb" also being a homophone. Obviously context will probably guide a person, but why the unnecessarily ambiguity when we already have lead in our solder for our leds on a PCB to lead electrons to light our way? (Not enough efficiency gain and I'll refer you to Lindybeige on the topic of following the French ;-s )
X is not equivalent to a window manager. You can use many different window managers in X
does anyone else see the cube mac?
4:05 Another issue is the handling of autorepeat: by having this entirely in software, you don’t actually have to insert the repeated keystrokes into any buffers; you simply generate them on demand, as the user program asks for keystrokes. That way, you never have them happening faster than the program can handle.
A key press is a key press, you can't say it's only a half
but "yes" is one char longer than "no".
An A press is an A press, you can't say it's only a half
I think this is rather a vague video. Was that intentional?
People running X Window System in one form or another should take the xev program for a spin.
Round 1.... FIGHT 1:08
A+ Tom and Jerry reference
third
hw > os > inputlib > other input lib > another one > electron/react
dont try this in vr, you will vomit
This guy always looks like he's about to fall asleep.
I THOUGHT THAT THE ONLY WAY A COMPUTER COULD "READ" KEYS WAS THAT EACH KEY WAS A SLIGHTLY DIFFERENT VOLTAGE AND AMPERAGE' SO THAT WHEN YOU PRESSED A KEY ON A KEYBOARD, IT WOULD "READ" THE DIFFERENCES BETWEEN THE KEYS SO THAT IT INTERPRET WHAT KEYS WERE PRESSED.
WHY ARE YOU YELLING
Where can I purchase that wig he is wearing, it is fantastic
All keys are equal, but the keys on a mechanical keyboard are more equal! (I make AI videos)
Not all keys are equal, the any key is actually special, with or without a nice mechanical keyboard ;)
Yeah, some are more equal than others.
Mate, Windows all the way!
This felt like the most awkward video to date on Computerphile. I'm sorry, I love everyone involved in making these, but this particular one was hard to watch and go along with.
I can not hear what you are saying. I'm not from England, I usually have no problems understanding english, but you are speaking unclear.
So damn chatty... Had to yawn for 5 minutes until he got to the point.
I used that time to read the comments while listening "with one ear" to what he said.
This is likely the blandest topic ever covered on this channel.