SOLVED What to check for when SSH authorized_keys does not work?

[txus.palacios]

I've read and read on Google about this topic, but I can't seem to fix it.

All my VMs are running some kind of UNIX. The fileserver runs OpenIndiana, some run Debian / Ubuntu Server, and everything else runs CentOS 6.3.

All the machines have been installed exactly the same way, using the command line "graphic" installer. All machines have been installed with the same amount of RAM (512MiB), and after installation, the RAM has been increased or decreased (because a print server does not need 512MiB of RAM for sure, and a ZFS file server could use some extra gigs).

All the machines have been targeted by ssh-copy-id from my laptop and from my workstation. All the machines have responded correctly to this command allowing me to ssh without typing a password to the servers, as long as I connect from my laptop or workstation.

All the machines, excluding one. My webserver, does not matter what you try to do, will not accept this. I've tried with ssh-copy-id, copying the keys manually, copying the keys via NFS. Nothing works.

I've tried to fix it checking permissions, deleting the .ssh folder and waiting for ssh-copy-id to create it again, I've checked that sshd_conf is the same for every machine... nothing seems to work. I've even disabled iptables for kicks and giggles. It does not work.

I'm gonna reinstall it because I've done nothing with the webserver. It's still pristine, maybe because SSH is way comfier than virtual machine console, maybe because I've had no time to mess with nginx yet. But, I've kept a snapshot of the VM.

Has someone got any tip to fix this or any explanation for this?

 

[petteyg359]

Connect with sss -vvvv and make sure it's reading the right private key.

 

[txus.palacios]

Connect with sss -vvvv and make sure it's reading the right private key.

txus@laptop ~ % ssh -vvvv root@webserver
OpenSSH_6.1p1, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to webserver [192.168.1.94] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/txus/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /home/txus/.ssh/id_rsa type 1
debug1: identity file /home/txus/.ssh/id_rsa-cert type -1
debug1: identity file /home/txus/.ssh/id_dsa type -1
debug1: identity file /home/txus/.ssh/id_dsa-cert type -1
debug1: identity file /home/txus/.ssh/id_ecdsa type -1
debug1: identity file /home/txus/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.1
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "webserver" from file "/home/txus/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/txus/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],[email protected],ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: [email protected],[email protected],ssh-rsa,[email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: none,[email protected],zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,[email protected]
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,[email protected],hmac-ripemd160,[email protected],hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: none,[email protected]
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 125/256
debug2: bits set: 512/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 2a:8d:2f:d6:78:a0:e8:1a:9d:4e:65:22:36:ca:69:fe
debug3: load_hostkeys: loading entries for host "webserver" from file "/home/txus/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/txus/.ssh/known_hosts:9
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "192.168.1.94" from file "/home/txus/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/txus/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'webserver' is known and matches the RSA host key.
debug1: Found key in /home/txus/.ssh/known_hosts:9
debug2: bits set: 496/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/txus/.ssh/id_rsa (0x22d59f0)
debug2: key: /home/txus/.ssh/id_dsa ((nil))
debug2: key: /home/txus/.ssh/id_ecdsa ((nil))
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic,password
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
[B]debug1: Offering RSA public key: /home/txus/.ssh/id_rsa[/B]
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Trying private key: /home/txus/.ssh/id_dsa
debug3: no such identity: /home/txus/.ssh/id_dsa
debug1: Trying private key: /home/txus/.ssh/id_ecdsa
debug3: no such identity: /home/txus/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password

I take it for granted it is then loading the correct key? I only have one key in my .ssh folder, and that's id_rsa.

 

[petteyg359]

Try restarting sshd with -d on the server, and watching the output while you log in. It ought to have something to say about the key.

 

[txus.palacios]

Strange.

I stopped the sshd service, fired /usr/sbin/sshd -d, and it accepted my key and logged me in without typing the password. Terminated the SSH session, started the sshd service again using service sshd start, tried to log in from the laptop, and it asked for the password. Fired the absolute path again and it worked. Even without the -d flag.

I guess I'll have to look for something service related?

EDIT:

output of service sshd start with $OPTIONS="-d"

root@webserver init.d]# service sshd restart
Stopping sshd:                                             [  OK  ]
Starting sshd: debug1: sshd version OpenSSH_5.3p1
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.1.23 port 38290
debug1: Client protocol version 2.0; client software version OpenSSH_6.1
debug1: match: OpenSSH_6.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: permanently_set_uid: 74/74
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user root service ssh-connection method none
debug1: attempt 0 failures 0
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "laptop.sanjuan46"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: userauth-request for user root service ssh-connection method publickey
debug1: attempt 1 failures 0
debug1: test whether pkalg/pkblob are acceptable
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: restore_uid: 0/0
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys2
debug1: restore_uid: 0/0
Failed publickey for root from 192.168.1.23 port 38290 ssh2

output of /usr/sbin/sshd -d

[root@webserver init.d]# /usr/sbin/sshd -d
debug1: sshd version OpenSSH_5.3p1
debug1: read PEM private key done: type RSA
debug1: private host key: #0 type 1 RSA
debug1: read PEM private key done: type DSA
debug1: private host key: #1 type 2 DSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-d'
Set /proc/self/oom_score_adj from 0 to -1000
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 192.168.1.23 port 38297
debug1: Client protocol version 2.0; client software version OpenSSH_6.1
debug1: match: OpenSSH_6.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug1: permanently_set_uid: 74/74
debug1: list_hostkey_types: ssh-rsa,ssh-dss
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT
debug1: SSH2_MSG_KEX_DH_GEX_REPLY sent
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: KEX done
debug1: userauth-request for user root service ssh-connection method none
debug1: attempt 0 failures 0
debug1: PAM: initializing for "root"
debug1: userauth-request for user root service ssh-connection method publickey
debug1: attempt 1 failures 0
debug1: test whether pkalg/pkblob are acceptable
debug1: PAM: setting PAM_RHOST to "laptop.sanjuan46"
debug1: PAM: setting PAM_TTY to "ssh"
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 3
Found matching RSA key: 40:27:25:55:91:8a:f5:dd:62:71:86:d6:0e:1a:3f:01
debug1: restore_uid: 0/0
Postponed publickey for root from 192.168.1.23 port 38297 ssh2
debug1: userauth-request for user root service ssh-connection method publickey
debug1: attempt 2 failures 0
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: trying public key file /root/.ssh/authorized_keys
debug1: fd 4 clearing O_NONBLOCK
debug1: matching key found: file /root/.ssh/authorized_keys, line 3
Found matching RSA key: 40:27:25:55:91:8a:f5:dd:62:71:86:d6:0e:1a:3f:01
debug1: restore_uid: 0/0
debug1: ssh_rsa_verify: signature correct
debug1: do_pam_account: called
Accepted publickey for root from 192.168.1.23 port 38297 ssh2
debug1: monitor_child_preauth: root has been authenticated by privileged process
debug1: temporarily_use_uid: 0/0 (e=0/0)
debug1: ssh_gssapi_storecreds: Not a GSSAPI mechanism
debug1: restore_uid: 0/0
debug1: SELinux support enabled
debug1: PAM: establishing credentials
debug1: Entering interactive session for SSH2.
debug1: server_init_dispatch_20
debug1: server_input_channel_open: ctype session rchan 0 win 1048576 max 16384
debug1: input_session_request
debug1: channel 0: new [server-session]
debug1: session_new: session 0
debug1: session_open: channel 0
debug1: session_open: session 0: link with channel 0
debug1: server_input_channel_open: confirm session
debug1: server_input_global_request: rtype [email protected] want_reply 0
debug1: server_input_channel_req: channel 0 request pty-req reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req pty-req
debug1: Allocating pty.
debug1: session_pty_req: session 0 alloc /dev/pts/1
debug1: server_input_channel_req: channel 0 request shell reply 1
debug1: session_by_channel: session 0 channel 0
debug1: session_input_channel_req: session 0 req shell
debug1: Setting controlling tty using TIOCSCTTY.

EDIT2: It seems the service cannot access the keys? I'll, for kicks and giggles, set the keys permissions to 777 and report back.
EDIT3: Set /root/.ssh to 777 and /root/.ssh/authorized_keys to 777. Same old story.
EDIT4: Set them to 700 for kicks and giggles. No deal, same output as EDIT3, same output as edit with standard permissions.
EDIT5: Fixed it! Google told me that maybe SELinux was having fun. Ran restorecon -R -v /root/.ssh and that fixed it! Not sure why this has only happened here, and not in any other machine.