You're by FAR my favorite Linux teacher! 😊 I think there is a better way to name the keys. Keep the prefix, such as id_rsa and id_ed25519, and add a suffix, like "~/.ssh/id_rsa.acme" (the public key will get the name "id_rsa.acme.pub" automaically). This naming has two benefits. You can see at a glance the key strength/type of all keys, and the filenames sort themselves as id_-prefix which means they all glob at the same position when listing files in the folder, and it also doesn't clash with autocompletion of the other important files in there (config and known_hosts). I also heard some people use subfolders like ~/.ssh/acme/id_rsa, but I saw several people say that it messes up the ability for ssh to automatically find the keys, so basically that you have to both add the IdentityFile to the config AND do ssh-add manually at every startup. Decided to try this theory for myself, so I created a key inside ~/.ssh/foldertest/id_rsa. I then ran "ssh-add -l" to list all keys. The new key wasn't listed. I then did "mv ~/.ssh/foldertest/* ~/.ssh" and ran "ssh-add -l" again, and it immediately listed the new key. So my theory is that ssh-agent automatically scans the ~/.ssh folder but never its subfolders. So yeah, avoid subfolders for your keys! Either way I see no need for subfolders since I use the host name as keyfile suffix instead, which cleanly separates the keys as I described earlier. :) Thanks a lot for refreshing my memory about how the ~/.ssh/config file works! Your tutorials are always lovely!
There is also a fun little fact: By default, SSH tries ALL of your keys when you connect a host. It basically submits every public key until one matches. You can see this process if you type "ssh -v" for verbose mode. In fact, if you have lots of keys (for example 30), then this trial-and-error process can literally lead you to getting a "Too many authentication failures" error from the server, especially for hardened ones that use things like fail2ban. But by instead creating a "~/.ssh/config" file, you can narrow it down to ONE specific file to make things more precise and faster. You can even add the same host with multiple different aliases that each use different usernames and key files. But! It is NOT enough to just add the IdentityFile like you showed in the video. If that key fails, SSH will still try all other keys, and even asks for manual password authentication if all keys failed! You therefore need to also add "IdentitiesOnly yes" to the Host section, which tells it to not try anything else except the exact IdentityFile you provided. However, if you connect to random hosts that are not in your config, it will STILL send all of your keys to them. The way to stop that once and for all, is to put the "IdentitiesOnly yes" at the TOP of your SSH config file, above the rest of the file (which makes it the global default, and you can delete the line from your Host sections). This tells it globally "only authenticate with specified identity files to all hosts". You then have to manually add each host and their identity files to your config. Note that by default, SSH always tries all of your "default filenames" identity files if they exist ("~/.ssh/id_dsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/id_rsa") and this happens even with IdentitiesOnly. So make sure your generated keys don't use those default filenames, otherwise those files will still be tried on all hosts.
2:15 Shinra is from Final Fantasy 7, Skynet is from the Terminator series; I was not expecting references to either of those in this video.. lol. Although, Skynet was a computer network; Cyberdyne Systems was the company that developed the network.
nice rundown on the basics of ssh keys. the question I've got is how to manage keys to 100s of servers. I keep hearing about a certificate server but would love to see a rundown on how to set something like that up
Although there was nothing new for me in this video I enjoy watching each of your videos so much because of your great teaching skills (or would one say didactics?). I even consider ordering your Ubuntu Server book even though Ubuntu isn't my particular distro pick.
Amazing job. I've learned a lot from you. Thank you so much. Please keep posting videos about day to day tasks. I'm looking forward to know the best packages used in everyday chores
I never figured out how to use ssh-agent from the CLI. I did however find it really useful when used with PuTTY from a Windows box to get to the Unix boxes at work. Neat and sneaky little invocation that "eval $(ssh-agent)". I guess it hooks into the file descriptors of the shell session to intercept and feed the passphrase in and out. Linux is full of these really cool and clever mechanisms.
eval $(ssh-agent) runs ssh-agent which dumps out some environment variables and eval then evaluates and inserts those variables into your environment. the existence of these variables tells the ssh command how to talk to the agent. try running ssh-agent without the eval $() and you'll see the environment variables that it outputs.
Hi Jay, perfect timing as this is a subject I have been wanting to learn more about. In your example you simulated contracting with 3 companies and needing to use unique keys for each; this makes complete sense to me. However, what if you had several or lots of servers at each company; would you still just use the 3 keys? Would you use a unique key per server? Or, is there some other decision making like 1 key for internal servers and a different key for DMZ servers, etc.? Another question, if you want to clean up having used the default id_rsa key thus far after setting up a unique key, do you need to remove the old key from the authorized keys file? (I think this is where you would do this clean-up) Thank you for all your content and I missed the config video somehow so going to watch it now.
Great tutorial...I was just wondering if you could make a video on restoring ssh keys. To a new linux installation. Imagine you have to reinstall ubuntu. What keys/files to save and restore in order to be able to logon remote servers again.
Hi Jay, Will you be following up this getting started video with an advanced version at all? I'm thinking about the best ways to add authorised public keys to servers without connecting to each one. The best way I can think of so far is to manage the authorized_keys file via ansible in a git repo. What do you think?
I created an ed ssh file with passphrase on a win 10 machine and copied the .pub file with the command from powershell cat ~/.ssh/intel_ed.pub | ssh user_name@ip_address "cat >> ~/.ssh/authorized_keys" to the server, running linux. Also in the sshd_config file of the server, the PasswordAuthentication is set to no. No matter what, I can ssh from all the machines to that server by only typing the user and pass credentials. What am I missing here?
You never explained what was happening with acme where you never used a username in the cli commands - nor is it in the config file. How does that work?
This video is on the whole good, but I disagree with the advice to create one key pair per remote host. At first consideration it seems to make sense; you wouldn't use the same password to log into multiple servers or web services, after all. But the reason you don't want to reuse passwords is that you have no control how they're stored on the other end. If some web site keeps your password stored in plaintext and a hacker steals it, the hacker can log into any other site where you use the same password. But *an SSH private key never leaves your machine* (except to back it up). All the remote hosts get is a public key, which is useless to hackers. Therefore you gain no advantage by complicating your SSH client setup with multiple key pairs. If someone steals your laptop and somehow decrypts your private key, sure, your remote accounts are in danger. But the same is true if the hacker steals your laptop with three private keys on it. My advice is to create one key pair per client computer (or, if you're sharing a PC with someone else, one key pair per user per computer). If one of your clients is stolen, deauthorize it from logging into remote hosts by removing its public key from them. That's it.
For the moment being, LINODE is a JOKE OF A PROVIDER, rejecting Signup requests for "patterns associated with fraudulent behavior", which besides being highly offensive to potential new users, provides absolutely NO additional information on what is not correct on the user's side, in order to complete the registration process. Well done, Linode ! Not ...
Hands down the best Linux tutorials.
You're by FAR my favorite Linux teacher! 😊
I think there is a better way to name the keys. Keep the prefix, such as id_rsa and id_ed25519, and add a suffix, like "~/.ssh/id_rsa.acme" (the public key will get the name "id_rsa.acme.pub" automaically).
This naming has two benefits. You can see at a glance the key strength/type of all keys, and the filenames sort themselves as id_-prefix which means they all glob at the same position when listing files in the folder, and it also doesn't clash with autocompletion of the other important files in there (config and known_hosts).
I also heard some people use subfolders like ~/.ssh/acme/id_rsa, but I saw several people say that it messes up the ability for ssh to automatically find the keys, so basically that you have to both add the IdentityFile to the config AND do ssh-add manually at every startup. Decided to try this theory for myself, so I created a key inside ~/.ssh/foldertest/id_rsa. I then ran "ssh-add -l" to list all keys. The new key wasn't listed. I then did "mv ~/.ssh/foldertest/* ~/.ssh" and ran "ssh-add -l" again, and it immediately listed the new key.
So my theory is that ssh-agent automatically scans the ~/.ssh folder but never its subfolders. So yeah, avoid subfolders for your keys! Either way I see no need for subfolders since I use the host name as keyfile suffix instead, which cleanly separates the keys as I described earlier. :)
Thanks a lot for refreshing my memory about how the ~/.ssh/config file works! Your tutorials are always lovely!
There is also a fun little fact: By default, SSH tries ALL of your keys when you connect a host. It basically submits every public key until one matches. You can see this process if you type "ssh -v" for verbose mode.
In fact, if you have lots of keys (for example 30), then this trial-and-error process can literally lead you to getting a "Too many authentication failures" error from the server, especially for hardened ones that use things like fail2ban.
But by instead creating a "~/.ssh/config" file, you can narrow it down to ONE specific file to make things more precise and faster.
You can even add the same host with multiple different aliases that each use different usernames and key files.
But! It is NOT enough to just add the IdentityFile like you showed in the video. If that key fails, SSH will still try all other keys, and even asks for manual password authentication if all keys failed!
You therefore need to also add "IdentitiesOnly yes" to the Host section, which tells it to not try anything else except the exact IdentityFile you provided.
However, if you connect to random hosts that are not in your config, it will STILL send all of your keys to them. The way to stop that once and for all, is to put the "IdentitiesOnly yes" at the TOP of your SSH config file, above the rest of the file (which makes it the global default, and you can delete the line from your Host sections). This tells it globally "only authenticate with specified identity files to all hosts". You then have to manually add each host and their identity files to your config.
Note that by default, SSH always tries all of your "default filenames" identity files if they exist ("~/.ssh/id_dsa, ~/.ssh/id_ecdsa, ~/.ssh/id_ed25519 and ~/.ssh/id_rsa") and this happens even with IdentitiesOnly. So make sure your generated keys don't use those default filenames, otherwise those files will still be tried on all hosts.
2:15 Shinra is from Final Fantasy 7, Skynet is from the Terminator series; I was not expecting references to either of those in this video.. lol. Although, Skynet was a computer network; Cyberdyne Systems was the company that developed the network.
nice rundown on the basics of ssh keys. the question I've got is how to manage keys to 100s of servers. I keep hearing about a certificate server but would love to see a rundown on how to set something like that up
Create a bash script or an app that stores the details on a database likely sqlite and creates a config file for you.
Bastion server, pritunl zero, teleport.. Many solutions out there
"Shinra Key"- which number keycard from the Shinra building works with SSH? 22? :)
Thank you so much for making this, the config file is way more powerful than I thought.
Although there was nothing new for me in this video I enjoy watching each of your videos so much because of your great teaching skills (or would one say didactics?).
I even consider ordering your Ubuntu Server book even though Ubuntu isn't my particular distro pick.
Amazing job. I've learned a lot from you. Thank you so much. Please keep posting videos about day to day tasks.
I'm looking forward to know the best packages used in everyday chores
Well that was inspirational video, I learned a lot about SSH keys today and how to use them. Both this and config file video were just great.
I just watched your full ssh video this morning. lucky
I never figured out how to use ssh-agent from the CLI. I did however find it really useful when used with PuTTY from a Windows box to get to the Unix boxes at work.
Neat and sneaky little invocation that "eval $(ssh-agent)". I guess it hooks into the file descriptors of the shell session to intercept and feed the passphrase in and out.
Linux is full of these really cool and clever mechanisms.
eval $(ssh-agent) runs ssh-agent which dumps out some environment variables and eval then evaluates and inserts those variables into your environment. the existence of these variables tells the ssh command how to talk to the agent. try running ssh-agent without the eval $() and you'll see the environment variables that it outputs.
keychain is a nice addition to this setup
Hi Jay, perfect timing as this is a subject I have been wanting to learn more about. In your example you simulated contracting with 3 companies and needing to use unique keys for each; this makes complete sense to me. However, what if you had several or lots of servers at each company; would you still just use the 3 keys? Would you use a unique key per server? Or, is there some other decision making like 1 key for internal servers and a different key for DMZ servers, etc.? Another question, if you want to clean up having used the default id_rsa key thus far after setting up a unique key, do you need to remove the old key from the authorized keys file? (I think this is where you would do this clean-up) Thank you for all your content and I missed the config video somehow so going to watch it now.
Excellent video, my good man. Thanks for helping out us noobs.
Thanks from Brazil!
Thank you so much.
I got distracted from this video by some metal guy blowing up my mako reactor.
Thank you, it's very useful video about ssh keys managment.
Very helpful video it was explained in an excellent way
Wow a lot to learn from this video. Thanks you so much......
Great tutorial...I was just wondering if you could make a video on restoring ssh keys. To a new linux installation. Imagine you have to reinstall ubuntu. What keys/files to save and restore in order to be able to logon remote servers again.
Hi Jay,
Will you be following up this getting started video with an advanced version at all? I'm thinking about the best ways to add authorised public keys to servers without connecting to each one. The best way I can think of so far is to manage the authorized_keys file via ansible in a git repo. What do you think?
17:53 wait how does it know which key?
I created an ed ssh file with passphrase on a win 10 machine and copied the .pub file with the command from powershell cat ~/.ssh/intel_ed.pub | ssh user_name@ip_address "cat >> ~/.ssh/authorized_keys" to the server, running linux.
Also in the sshd_config file of the server, the PasswordAuthentication is set to no. No matter what, I can ssh from all the machines to that server by only typing the user and pass credentials. What am I missing here?
18:57
I am using MS Windows 10 laptop. How to configure this?
Step 1. Switch to Linux..
Very helpful Jay, thanks .-)
You never explained what was happening with acme where you never used a username in the cli commands - nor is it in the config file. How does that work?
Awesome video!
Good stuff!!
What I don't really get is how only one private key would leak, as they are all stored together
Thanks so much!!
Skynet is from the Terminator. Shinra I'm not sure.
Shinra is from Final Fantasy VII.
Nice vid :)
if the passphase was a built-in parameter of ~/.ssh/config , everything would have been much easier...wonder why that isn't the case...
TBF Acme could also be considered an evil company as they only sold junk that always broke upon first use 😆
I would like to have a HAL9000 as a bad Companyname
This video is on the whole good, but I disagree with the advice to create one key pair per remote host. At first consideration it seems to make sense; you wouldn't use the same password to log into multiple servers or web services, after all. But the reason you don't want to reuse passwords is that you have no control how they're stored on the other end. If some web site keeps your password stored in plaintext and a hacker steals it, the hacker can log into any other site where you use the same password.
But *an SSH private key never leaves your machine* (except to back it up). All the remote hosts get is a public key, which is useless to hackers. Therefore you gain no advantage by complicating your SSH client setup with multiple key pairs. If someone steals your laptop and somehow decrypts your private key, sure, your remote accounts are in danger. But the same is true if the hacker steals your laptop with three private keys on it.
My advice is to create one key pair per client computer (or, if you're sharing a PC with someone else, one key pair per user per computer). If one of your clients is stolen, deauthorize it from logging into remote hosts by removing its public key from them. That's it.
For the moment being, LINODE is a JOKE OF A PROVIDER, rejecting Signup requests for "patterns associated with fraudulent behavior", which besides being highly offensive to potential new users, provides absolutely NO additional information on what is not correct on the user's side, in order to complete the registration process.
Well done, Linode ! Not ...