ssh strange and bizarre

This shows the ssh problem I had after I upgraded my client to openssh-5.8. The client could not connect anymore to some servers. Those servers happent to be located at the university where I work.

  • The diagnostic on the client is “Connection reset by peer”.
  • The diagnostic on the server is “Connection reset by peer”.

<quote>Être ange
C'est étrange
Dit l'ange
Être âne,
C'est étrane
Dit l'âne


Jacques Prévert </quote>

What triggers the problem as well as workarounds are easily found on internet. The cipher list length is responsible for the problem when client is openssh >= 5.7 and server is openssh < 5.7. A workaround is to specify a short list of ciphers. like this :

schplurtz@imladris:~ (0) $ ssh -l remotelogon -c aes256-cbc ssh-server
ssh-server $ 

Note that the ciphers themselves do not matter, only the length does. It is easily verified with, say :

schplurtz@imladris:~ (0) $ ssh -l remotelogon -c aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr ssh-server
Read from socket failed: Connection reset by peer
schplurtz@imladris:~ (255) $ 

References

server and client in debug mode

Client says connection reset by peer

schplurtz@imladris:~ (255) $ ssh -vv ssh-server
OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010
debug1: Reading configuration data /home/schplurtz/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ssh-server [10.0.1.2] port 22.
debug1: Connection established.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug2: key_type_from_name: unknown key type '-----END'
debug1: identity file /home/schplurtz/.ssh/id_rsa-new type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/schplurtz/.ssh/id_rsa-new-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1 Debian-6
debug1: match: OpenSSH_5.5p1 Debian-6 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
Read from socket failed: Connection reset by peer
schplurtz@imladris:~ (255) $ 

Server says connection reset by peer.

ssh-server:/ (1) # /usr/sbin/sshd -ddde 2>&1 | tee /root/sshd-debug-2
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 624
debug2: parse_server_config: config /etc/ssh/sshd_config len 624
debug3: /etc/ssh/sshd_config:5 setting Port 22
debug3: /etc/ssh/sshd_config:9 setting Protocol 2
debug3: /etc/ssh/sshd_config:11 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: /etc/ssh/sshd_config:14 setting UsePrivilegeSeparation yes
debug3: /etc/ssh/sshd_config:15 setting X11UseLocalhost no
debug3: /etc/ssh/sshd_config:18 setting KeyRegenerationInterval 3600
debug3: /etc/ssh/sshd_config:19 setting ServerKeyBits 768
debug3: /etc/ssh/sshd_config:22 setting SyslogFacility AUTH
debug3: /etc/ssh/sshd_config:23 setting LogLevel INFO
debug3: /etc/ssh/sshd_config:26 setting LoginGraceTime 120
debug3: /etc/ssh/sshd_config:27 setting PermitRootLogin yes
debug3: /etc/ssh/sshd_config:28 setting StrictModes yes
debug3: /etc/ssh/sshd_config:30 setting RSAAuthentication yes
debug3: /etc/ssh/sshd_config:31 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:35 setting IgnoreRhosts yes
debug3: /etc/ssh/sshd_config:37 setting RhostsRSAAuthentication no
debug3: /etc/ssh/sshd_config:39 setting HostbasedAuthentication no
debug3: /etc/ssh/sshd_config:44 setting PermitEmptyPasswords no
debug3: /etc/ssh/sshd_config:48 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:63 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:64 setting X11DisplayOffset 10
debug3: /etc/ssh/sshd_config:65 setting PrintMotd no
debug3: /etc/ssh/sshd_config:66 setting PrintLastLog yes
debug3: /etc/ssh/sshd_config:67 setting TCPKeepAlive yes
debug3: /etc/ssh/sshd_config:74 setting AcceptEnv LANG LC_*
debug3: /etc/ssh/sshd_config:76 setting Subsystem sftp /usr/lib/openssh/sftp-server
debug3: /etc/ssh/sshd_config:78 setting UsePAM yes
debug1: sshd version OpenSSH_5.5p1 Debian-6
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024
debug1: private host key: #0 type 1 RSA
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddde'
debug3: oom_adjust_setup
Set /proc/self/oom_adj from 0 to -17
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
socket: Address family not supported by protocol
debug3: fd 4 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 7 config len 624
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7
debug3: recv_rexec_state: entering fd = 5
debug3: ssh_msg_recv entering
debug3: recv_rexec_state: done
debug2: parse_server_config: config rexec len 624
debug3: rexec:5 setting Port 22
debug3: rexec:9 setting Protocol 2
debug3: rexec:11 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: rexec:14 setting UsePrivilegeSeparation yes
debug3: rexec:15 setting X11UseLocalhost no
debug3: rexec:18 setting KeyRegenerationInterval 3600
debug3: rexec:19 setting ServerKeyBits 768
debug3: rexec:22 setting SyslogFacility AUTH
debug3: rexec:23 setting LogLevel INFO
debug3: rexec:26 setting LoginGraceTime 120
debug3: rexec:27 setting PermitRootLogin yes
debug3: rexec:28 setting StrictModes yes
debug3: rexec:30 setting RSAAuthentication yes
debug3: rexec:31 setting PubkeyAuthentication yes
debug3: rexec:35 setting IgnoreRhosts yes
debug3: rexec:37 setting RhostsRSAAuthentication no
debug3: rexec:39 setting HostbasedAuthentication no
debug3: rexec:44 setting PermitEmptyPasswords no
debug3: rexec:48 setting ChallengeResponseAuthentication no
debug3: rexec:63 setting X11Forwarding yes
debug3: rexec:64 setting X11DisplayOffset 10
debug3: rexec:65 setting PrintMotd no
debug3: rexec:66 setting PrintLastLog yes
debug3: rexec:67 setting TCPKeepAlive yes
debug3: rexec:74 setting AcceptEnv LANG LC_*
debug3: rexec:76 setting Subsystem sftp /usr/lib/openssh/sftp-server
debug3: rexec:78 setting UsePAM yes
debug1: sshd version OpenSSH_5.5p1 Debian-6
debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key.
debug1: read PEM private key done: type RSA
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024
debug1: private host key: #0 type 1 RSA
debug1: inetd sockets after dupping: 3, 3
Connection from 172.16.23.54 port 55000
debug1: Client protocol version 2.0; client software version OpenSSH_5.8p1 Debian-1ubuntu3
debug1: match: OpenSSH_5.8p1 Debian-1ubuntu3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.5p1 Debian-6
debug2: fd 3 setting O_NONBLOCK
debug2: Network child is on pid 17465
debug3: preauth child monitor started
debug3: mm_request_receive entering
debug3: privsep user:group 102:65534
debug1: permanently_set_uid: 102/65534
debug1: list_hostkey_types: ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
Read from socket failed: Connection reset by peer
debug1: do_cleanup
debug3: PAM: sshpam_thread_cleanup entering
debug1: do_cleanup
debug3: PAM: sshpam_thread_cleanup entering
ssh-server:/ (0) # 

This is truely nonsense. Neither the client nor the server says tey want to reset the connection, yet both of them say “connection reset by peer”. Either error messages are false in some way, or a third party network system resets the connection for the client and server. This could well be the university Intrusion Detection Systen (IDS)

Let's see what tcpdump shows

tcpdump

I changed the IP's. that's all.

  • server IP is 10.0.1.2 , hostname ssh-server
  • client IP is 172.16.23.54 , hostname imladris

tcpdump on the client

On the client the last two lines show that two packets with RST ([R]) flag are received from the server.

root@imladris:~ (0) # tcpdump -nr failssh | grep 10.0.1.2
reading from file failssh, link-type EN10MB (Ethernet)
22:04:21.111315 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [S], seq 3508275589, win 14600, options [mss 1460,sackOK,TS val 835942 ecr 0,nop,wscale 6], length 0
22:04:21.179739 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [S.], seq 3162154788, ack 3508275590, win 5792, options [mss 1452,sackOK,TS val 2059539456 ecr 835942,nop,wscale 7], length 0
22:04:21.179830 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [.], ack 1, win 229, options [nop,nop,TS val 835959 ecr 2059539456], length 0
22:04:21.255404 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [P.], seq 1:33, ack 1, win 46, options [nop,nop,TS val 2059539474 ecr 835959], length 32
22:04:21.255511 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [.], ack 33, win 229, options [nop,nop,TS val 835978 ecr 2059539474], length 0
22:04:21.255803 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [P.], seq 1:40, ack 33, win 229, options [nop,nop,TS val 835978 ecr 2059539474], length 39
22:04:21.324524 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [.], ack 40, win 46, options [nop,nop,TS val 2059539492 ecr 835978], length 0
22:04:21.324585 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [P.], seq 40:1184, ack 33, win 229, options [nop,nop,TS val 835995 ecr 2059539492], length 1144
22:04:21.326165 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [P.], seq 33:809, ack 40, win 46, options [nop,nop,TS val 2059539492 ecr 835978], length 776
22:04:21.365353 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [.], ack 809, win 253, options [nop,nop,TS val 836006 ecr 2059539492], length 0
22:04:21.404199 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [R], seq 3162154821, win 0, length 0
22:04:21.431269 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [R], seq 3162155597, win 0, length 0
root@imladris:~ (0) # 

tcpdump on the server

On the server the last two lines show that two packets with RST (R) flag are received from the client.

ssh-server:~ (0) # tcpdump -nr ssh-fail-synchro.dump | grep 172.16.23.54
reading from file ssh-fail-synchro.dump, link-type EN10MB (Ethernet)
22:04:21.035439 IP 172.16.23.54.55000 > 10.0.1.2.22: S 3508275589:3508275589(0) win 14600 <mss 1452,sackOK,timestamp 835942 0,nop,wscale 6>
22:04:21.035493 IP 10.0.1.2.22 > 172.16.23.54.55000: S 3162154788:3162154788(0) ack 3508275590 win 5792 <mss 1460,sackOK,timestamp 2059539456 835942,nop,wscale 7>
22:04:21.101353 IP 172.16.23.54.55000 > 10.0.1.2.22: . ack 1 win 229 <nop,nop,timestamp 835959 2059539456>
22:04:21.110621 IP 10.0.1.2.22 > 172.16.23.54.55000: P 1:33(32) ack 1 win 46 <nop,nop,timestamp 2059539474 835959>
22:04:21.176686 IP 172.16.23.54.55000 > 10.0.1.2.22: . ack 33 win 229 <nop,nop,timestamp 835978 2059539474>
22:04:21.180047 IP 172.16.23.54.55000 > 10.0.1.2.22: P 1:40(39) ack 33 win 229 <nop,nop,timestamp 835978 2059539474>
22:04:21.180080 IP 10.0.1.2.22 > 172.16.23.54.55000: . ack 40 win 46 <nop,nop,timestamp 2059539492 835978>
22:04:21.181391 IP 10.0.1.2.22 > 172.16.23.54.55000: P 33:809(776) ack 40 win 46 <nop,nop,timestamp 2059539492 835978>
22:04:21.257925 IP 172.16.23.54.55000 > 10.0.1.2.22: R 3508275629:3508275629(0) win 0
22:04:21.287167 IP 172.16.23.54.55000 > 10.0.1.2.22: R 3508276773:3508276773(0) win 0
ssh-server:~ (0) # 

Conclusion

  1. error mesages from ssh client and servers are correct
  2. Something between the client and the server sends the fatal RST packet to both of them.

Solution

I contacted the Security person at the university. He said OK I see a line in the log of our IDS. It reads :

Malformed Key exchange init Message - SSH protocol violation

After removal of the rule, ssh works again !