How to Configure SSH Keys on a Server (Complete Beginner Guide)

If you manage Linux servers, VPS hosting, AWS EC2 instances, or cloud servers, securing SSH access should be one of your first priorities. By default, many beginners connect to their server using a username and password. While that works, password authentication is far less secure than SSH key authentication.

SSH keys provide a safer and more professional way to access your server. Instead of typing a password every time, you use a pair of cryptographic keys that verify your identity. This method is faster, more secure, and widely used by Linux server administrators.

In this complete beginner-friendly guide, you will learn exactly how to configure SSH keys on a server step by step, including generating keys, uploading them, disabling password login, and troubleshooting common SSH issues.

What Are SSH Keys?

SSH keys are a pair of encrypted files used for secure authentication between your computer and a remote server.

They consist of:

Private Key → Stored on your local computer and kept secret
Public Key → Uploaded to the server

When you connect via SSH, the server checks your private key against the stored public key. If they match, access is granted.

Think of SSH keys like:

Public key = Lock
Private key = Unique key that opens the lock

Only someone with the correct private key can log in.

Why Use SSH Keys Instead of Passwords?

Using SSH keys offers many advantages:

Better Security

SSH keys are far harder to brute force than passwords.

Faster Login

No need to type passwords every time.

Automation Friendly

Required for scripts, deployments, and DevOps tools.

Industry Standard

Most professional Linux admins and DevOps engineers use SSH keys.

Prerequisites Before You Begin

Make sure you have:

A Linux server or VPS
SSH access to the server
Terminal access on your local machine
Basic Linux command knowledge

Step 1: Generate SSH Key Pair on Your Local Machine

Open your terminal and run:

ssh-keygen -t rsa -b 4096

Explanation:

ssh-keygen = Command to create SSH keys
-t rsa = RSA encryption type
-b 4096 = Strong encryption length

You will see:

Enter file in which to save the key:

Press Enter to use default location.

Step 2: Set a Passphrase (Optional but Recommended)

Next prompt:

Enter passphrase:

You can:

Add a passphrase for extra security
Leave blank for convenience

A passphrase protects your private key if stolen.

Step 3: Verify Key Files

Run:

ls ~/.ssh

You should see:

id_rsa
id_rsa.pub

Where:

id_rsa = Private key
id_rsa.pub = Public key

Step 4: Upload Public Key to Server

Run:

ssh-copy-id user@your-server-ip

Example:

ssh-copy-id [email protected]

This automatically copies your public key to:

~/.ssh/authorized_keys

Step 5: Manual Upload Method (Alternative)

If ssh-copy-id doesn’t work:

Display your public key:

cat ~/.ssh/id_rsa.pub

Copy the output.

ssh user@your-server-ip

Create .ssh folder:

mkdir -p ~/.ssh
chmod 700 ~/.ssh

Edit authorized_keys:

nano ~/.ssh/authorized_keys

Paste public key.

Save file.

Set permissions:

chmod 600 ~/.ssh/authorized_keys

Step 6: Test SSH Key Login

Disconnect and reconnect:

ssh user@your-server-ip

If successful:

No password prompt
SSH key authentication works

Step 7: Disable Password Authentication (Important Security Step)

Edit SSH config:

sudo nano /etc/ssh/sshd_config

Find:

PasswordAuthentication yes

Change to:

PasswordAuthentication no

Also ensure:

PubkeyAuthentication yes

Save file.

Restart SSH:

sudo systemctl restart ssh

Step 8: Test Again Before Closing Terminal

⚠️ Important:

Open a new terminal window before closing the old one.

Test login again:

ssh user@your-server-ip

If it works → safe to close old session.

Common SSH Key Errors and Fixes

Permission Denied (Publickey)

Cause:

Wrong permissions

Fix:

chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Key Not Found

Specify key manually:

ssh -i ~/.ssh/id_rsa user@server

SSH Service Not Restarting

Check syntax:

sudo sshd -t

Best Practices for SSH Security

Change Default SSH Port

Instead of port 22:

Port 2222

Disable Root Login

PermitRootLogin no

Use Firewall

Allow only trusted IPs.

Rotate Keys Periodically

Generate new keys every few months.

Real-World Use Cases

SSH keys are used for:

AWS EC2 login
GitHub deployments
Ansible automation
CI/CD pipelines
Remote Linux administration

FAQs

Are SSH keys safer than passwords?

Yes, much safer.

Can I use SSH keys on Windows?

Yes, using PuTTY or Windows Terminal.

What if I lose my private key?

You must generate a new pair and upload again.

Final Thoughts

SSH key authentication is one of the most important steps in securing a Linux server. Whether you run AWS EC2, a VPS, or dedicated hosting, passwordless login improves both security and convenience.

Once configured correctly, SSH keys make server management safer, faster, and more professional.