meta données pour cette page
ssh strange and bizarre
This shows the ssh problem I had after I upgraded my client to openssh-5.8. The client could not connect anymore to some servers. Those servers happent to be located at the university where I work.
- The diagnostic on the client is “Connection reset by peer”.
- The diagnostic on the server is “Connection reset by peer”.
<quote>Être ange
C'est étrange
Dit l'ange
Être âne,
C'est étrane
Dit l'âne
…
Jacques Prévert
</quote>
What triggers the problem as well as workarounds are easily found on internet. The cipher list length is responsible for the problem when client is openssh >= 5.7 and server is openssh < 5.7. A workaround is to specify a short list of ciphers. like this :
schplurtz@imladris:~ (0) $ ssh -l remotelogon -c aes256-cbc ssh-server ssh-server $
Note that the ciphers themselves do not matter, only the length does. It is easily verified with, say :
schplurtz@imladris:~ (0) $ ssh -l remotelogon -c aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr,aes256-ctr ssh-server Read from socket failed: Connection reset by peer schplurtz@imladris:~ (255) $
References
-
- some more…
server and client in debug mode
Client says connection reset by peer
schplurtz@imladris:~ (255) $ ssh -vv ssh-server OpenSSH_5.8p1 Debian-1ubuntu3, OpenSSL 0.9.8o 01 Jun 2010 debug1: Reading configuration data /home/schplurtz/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug1: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to ssh-server [10.0.1.2] port 22. debug1: Connection established. debug2: key_type_from_name: unknown key type '-----BEGIN' debug2: key_type_from_name: unknown key type 'Proc-Type:' debug2: key_type_from_name: unknown key type 'DEK-Info:' debug2: key_type_from_name: unknown key type '-----END' debug1: identity file /home/schplurtz/.ssh/id_rsa-new type 1 debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048 debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048 debug1: identity file /home/schplurtz/.ssh/id_rsa-new-cert type -1 debug1: Remote protocol version 2.0, remote software version OpenSSH_5.5p1 Debian-6 debug1: match: OpenSSH_5.5p1 Debian-6 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-1ubuntu3 debug2: fd 3 setting O_NONBLOCK debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found hmac-md5 debug1: kex: server->client aes128-ctr hmac-md5 none debug2: mac_setup: found hmac-md5 debug1: kex: client->server aes128-ctr hmac-md5 none debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP Read from socket failed: Connection reset by peer schplurtz@imladris:~ (255) $
Server says connection reset by peer
.
ssh-server:/ (1) # /usr/sbin/sshd -ddde 2>&1 | tee /root/sshd-debug-2 debug2: load_server_config: filename /etc/ssh/sshd_config debug2: load_server_config: done config len = 624 debug2: parse_server_config: config /etc/ssh/sshd_config len 624 debug3: /etc/ssh/sshd_config:5 setting Port 22 debug3: /etc/ssh/sshd_config:9 setting Protocol 2 debug3: /etc/ssh/sshd_config:11 setting HostKey /etc/ssh/ssh_host_rsa_key debug3: /etc/ssh/sshd_config:14 setting UsePrivilegeSeparation yes debug3: /etc/ssh/sshd_config:15 setting X11UseLocalhost no debug3: /etc/ssh/sshd_config:18 setting KeyRegenerationInterval 3600 debug3: /etc/ssh/sshd_config:19 setting ServerKeyBits 768 debug3: /etc/ssh/sshd_config:22 setting SyslogFacility AUTH debug3: /etc/ssh/sshd_config:23 setting LogLevel INFO debug3: /etc/ssh/sshd_config:26 setting LoginGraceTime 120 debug3: /etc/ssh/sshd_config:27 setting PermitRootLogin yes debug3: /etc/ssh/sshd_config:28 setting StrictModes yes debug3: /etc/ssh/sshd_config:30 setting RSAAuthentication yes debug3: /etc/ssh/sshd_config:31 setting PubkeyAuthentication yes debug3: /etc/ssh/sshd_config:35 setting IgnoreRhosts yes debug3: /etc/ssh/sshd_config:37 setting RhostsRSAAuthentication no debug3: /etc/ssh/sshd_config:39 setting HostbasedAuthentication no debug3: /etc/ssh/sshd_config:44 setting PermitEmptyPasswords no debug3: /etc/ssh/sshd_config:48 setting ChallengeResponseAuthentication no debug3: /etc/ssh/sshd_config:63 setting X11Forwarding yes debug3: /etc/ssh/sshd_config:64 setting X11DisplayOffset 10 debug3: /etc/ssh/sshd_config:65 setting PrintMotd no debug3: /etc/ssh/sshd_config:66 setting PrintLastLog yes debug3: /etc/ssh/sshd_config:67 setting TCPKeepAlive yes debug3: /etc/ssh/sshd_config:74 setting AcceptEnv LANG LC_* debug3: /etc/ssh/sshd_config:76 setting Subsystem sftp /usr/lib/openssh/sftp-server debug3: /etc/ssh/sshd_config:78 setting UsePAM yes debug1: sshd version OpenSSH_5.5p1 Debian-6 debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key. debug1: read PEM private key done: type RSA debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024 debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024 debug1: private host key: #0 type 1 RSA debug1: rexec_argv[0]='/usr/sbin/sshd' debug1: rexec_argv[1]='-ddde' debug3: oom_adjust_setup Set /proc/self/oom_adj from 0 to -17 debug2: fd 3 setting O_NONBLOCK debug1: Bind to port 22 on 0.0.0.0. Server listening on 0.0.0.0 port 22. socket: Address family not supported by protocol debug3: fd 4 is not O_NONBLOCK debug1: Server will not fork when running in debugging mode. debug3: send_rexec_state: entering fd = 7 config len 624 debug3: ssh_msg_send: type 0 debug3: send_rexec_state: done debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7 debug3: recv_rexec_state: entering fd = 5 debug3: ssh_msg_recv entering debug3: recv_rexec_state: done debug2: parse_server_config: config rexec len 624 debug3: rexec:5 setting Port 22 debug3: rexec:9 setting Protocol 2 debug3: rexec:11 setting HostKey /etc/ssh/ssh_host_rsa_key debug3: rexec:14 setting UsePrivilegeSeparation yes debug3: rexec:15 setting X11UseLocalhost no debug3: rexec:18 setting KeyRegenerationInterval 3600 debug3: rexec:19 setting ServerKeyBits 768 debug3: rexec:22 setting SyslogFacility AUTH debug3: rexec:23 setting LogLevel INFO debug3: rexec:26 setting LoginGraceTime 120 debug3: rexec:27 setting PermitRootLogin yes debug3: rexec:28 setting StrictModes yes debug3: rexec:30 setting RSAAuthentication yes debug3: rexec:31 setting PubkeyAuthentication yes debug3: rexec:35 setting IgnoreRhosts yes debug3: rexec:37 setting RhostsRSAAuthentication no debug3: rexec:39 setting HostbasedAuthentication no debug3: rexec:44 setting PermitEmptyPasswords no debug3: rexec:48 setting ChallengeResponseAuthentication no debug3: rexec:63 setting X11Forwarding yes debug3: rexec:64 setting X11DisplayOffset 10 debug3: rexec:65 setting PrintMotd no debug3: rexec:66 setting PrintLastLog yes debug3: rexec:67 setting TCPKeepAlive yes debug3: rexec:74 setting AcceptEnv LANG LC_* debug3: rexec:76 setting Subsystem sftp /usr/lib/openssh/sftp-server debug3: rexec:78 setting UsePAM yes debug1: sshd version OpenSSH_5.5p1 Debian-6 debug3: Not a RSA1 key file /etc/ssh/ssh_host_rsa_key. debug1: read PEM private key done: type RSA debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024 debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024 debug1: private host key: #0 type 1 RSA debug1: inetd sockets after dupping: 3, 3 Connection from 172.16.23.54 port 55000 debug1: Client protocol version 2.0; client software version OpenSSH_5.8p1 Debian-1ubuntu3 debug1: match: OpenSSH_5.8p1 Debian-1ubuntu3 pat OpenSSH* debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_5.5p1 Debian-6 debug2: fd 3 setting O_NONBLOCK debug2: Network child is on pid 17465 debug3: preauth child monitor started debug3: mm_request_receive entering debug3: privsep user:group 102:65534 debug1: permanently_set_uid: 102/65534 debug1: list_hostkey_types: ssh-rsa debug1: SSH2_MSG_KEXINIT sent Read from socket failed: Connection reset by peer debug1: do_cleanup debug3: PAM: sshpam_thread_cleanup entering debug1: do_cleanup debug3: PAM: sshpam_thread_cleanup entering ssh-server:/ (0) #
This is truely nonsense. Neither the client nor the server says tey want to reset the connection, yet both of them say “connection reset by peer”. Either error messages are false in some way, or a third party network system resets the connection for the client and server. This could well be the university Intrusion Detection Systen (IDS)
Let's see what tcpdump shows
tcpdump
I changed the IP's. that's all.
- server IP is 10.0.1.2 , hostname ssh-server
- client IP is 172.16.23.54 , hostname imladris
tcpdump on the client
On the client the last two lines show that two packets with RST ([R]
) flag are received from the server.
root@imladris:~ (0) # tcpdump -nr failssh | grep 10.0.1.2 reading from file failssh, link-type EN10MB (Ethernet) 22:04:21.111315 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [S], seq 3508275589, win 14600, options [mss 1460,sackOK,TS val 835942 ecr 0,nop,wscale 6], length 0 22:04:21.179739 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [S.], seq 3162154788, ack 3508275590, win 5792, options [mss 1452,sackOK,TS val 2059539456 ecr 835942,nop,wscale 7], length 0 22:04:21.179830 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [.], ack 1, win 229, options [nop,nop,TS val 835959 ecr 2059539456], length 0 22:04:21.255404 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [P.], seq 1:33, ack 1, win 46, options [nop,nop,TS val 2059539474 ecr 835959], length 32 22:04:21.255511 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [.], ack 33, win 229, options [nop,nop,TS val 835978 ecr 2059539474], length 0 22:04:21.255803 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [P.], seq 1:40, ack 33, win 229, options [nop,nop,TS val 835978 ecr 2059539474], length 39 22:04:21.324524 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [.], ack 40, win 46, options [nop,nop,TS val 2059539492 ecr 835978], length 0 22:04:21.324585 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [P.], seq 40:1184, ack 33, win 229, options [nop,nop,TS val 835995 ecr 2059539492], length 1144 22:04:21.326165 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [P.], seq 33:809, ack 40, win 46, options [nop,nop,TS val 2059539492 ecr 835978], length 776 22:04:21.365353 IP 172.16.23.54.55000 > 10.0.1.2.22: Flags [.], ack 809, win 253, options [nop,nop,TS val 836006 ecr 2059539492], length 0 22:04:21.404199 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [R], seq 3162154821, win 0, length 0 22:04:21.431269 IP 10.0.1.2.22 > 172.16.23.54.55000: Flags [R], seq 3162155597, win 0, length 0 root@imladris:~ (0) #
tcpdump on the server
On the server the last two lines show that two packets with RST (R
) flag are received from the client.
ssh-server:~ (0) # tcpdump -nr ssh-fail-synchro.dump | grep 172.16.23.54 reading from file ssh-fail-synchro.dump, link-type EN10MB (Ethernet) 22:04:21.035439 IP 172.16.23.54.55000 > 10.0.1.2.22: S 3508275589:3508275589(0) win 14600 <mss 1452,sackOK,timestamp 835942 0,nop,wscale 6> 22:04:21.035493 IP 10.0.1.2.22 > 172.16.23.54.55000: S 3162154788:3162154788(0) ack 3508275590 win 5792 <mss 1460,sackOK,timestamp 2059539456 835942,nop,wscale 7> 22:04:21.101353 IP 172.16.23.54.55000 > 10.0.1.2.22: . ack 1 win 229 <nop,nop,timestamp 835959 2059539456> 22:04:21.110621 IP 10.0.1.2.22 > 172.16.23.54.55000: P 1:33(32) ack 1 win 46 <nop,nop,timestamp 2059539474 835959> 22:04:21.176686 IP 172.16.23.54.55000 > 10.0.1.2.22: . ack 33 win 229 <nop,nop,timestamp 835978 2059539474> 22:04:21.180047 IP 172.16.23.54.55000 > 10.0.1.2.22: P 1:40(39) ack 33 win 229 <nop,nop,timestamp 835978 2059539474> 22:04:21.180080 IP 10.0.1.2.22 > 172.16.23.54.55000: . ack 40 win 46 <nop,nop,timestamp 2059539492 835978> 22:04:21.181391 IP 10.0.1.2.22 > 172.16.23.54.55000: P 33:809(776) ack 40 win 46 <nop,nop,timestamp 2059539492 835978> 22:04:21.257925 IP 172.16.23.54.55000 > 10.0.1.2.22: R 3508275629:3508275629(0) win 0 22:04:21.287167 IP 172.16.23.54.55000 > 10.0.1.2.22: R 3508276773:3508276773(0) win 0 ssh-server:~ (0) #
Conclusion
- error mesages from ssh client and servers are correct
- Something between the client and the server sends the fatal RST packet to both of them.
Solution
I contacted the Security person at the university. He said OK I see a line in the log of our IDS. It reads :
Malformed Key exchange init Message - SSH protocol violation
After removal of the rule, ssh works again !