Hi Peter,
> > I'm happy to help but my responses may be delayed as I'm getting ready
> > to head out for a few days of vacation (leaving in 12 hours)
>
> Oh that sounds amazing! I hope you have a good time.
>
>
> > > > sshd[361421]: debug1: kex: client->server cipher: aes128-ctr MAC:
> > hmac-sha2-256 compression: none [preauth]
> ..
>
> > > What processor does your embedded system have? In particular endianess
> > > and native bit size (32/64?)
>
> > model name : ARMv7 Processor rev 1 (v7l)
> > Hardware : Atmel SAMA5
>
> > /bin/busybox: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux),
> > statically linked, BuildID[sha1]=51f5566abbeca4cee5e53734090d7d37b33deedb,
> > for GNU/Linux 3.2.0, stripped
>
> Thanks, little endian ARM.
>
>
> > > What MAC is negotiated by the dropbear and OpenSSH clients you tested?
> >
> > Dropbear:
> ..
> > May 18 15:42:08 eric-Precision-7520 sshd[444608]: debug1: kex:
> > client->server cipher: aes128-ctr MAC: hmac-sha1 compression: none [preauth]
>
>
> > OpenSSH:
> ..
> > May 19 17:24:01 eric-Precision-7520 sshd[3355904]: debug1: kex:
> > <implicit> compression: none [preauth]
>
> This is interesting and provides some clues. All three clients end up
> using different ciphers/MACs with only libssh2 using hmac-sha2-256.
>
>
> It would be great if you could try OpenSSH with the same cipher+MAC
> as libssh2 and provide the server debug log also for that connection:
>
> ssh -oCiphers=aes128-ctr -oMACs=hmac-sha2-256 server
>
>
> > > Are results identical with a server running an unpatched upstream OpenSSH?
> >
> > Unfortunately, I do not have a server running this version. My sshd
> > server is
> >
> > OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f 31 Mar 2020
>
> Okay. The upstream source code is fairly easy to build but I understand
> that you're leaving literally in hours. If you can try the OpenSSH
> client with specific cipher and mac that log would be very helpful.
# ssh -V
OpenSSH_8.5p1, OpenSSL 1.1.1k 25 Mar 2021
# ssh -v -oCiphers=aes128-ctr -oMACs=hmac-sha2-256 192.168.1.123
OpenSSH_8.5p1, OpenSSL 1.1.1k 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to 192.168.1.123 [192.168.1.123] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa_sk type -1
debug1: identity file /root/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_ed25519_sk type -1
debug1: identity file /root/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.5
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
debug1: compat_banner: match: OpenSSH_8.2p1 Ubuntu-4ubuntu0.2 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 192.168.1.123:22 as 'root'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:NgW5tO4qOstp1cHvompAmFce85z6T+MB3rW92d4FJiM
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '192.168.1.123' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
ssh_dispatch_run_fatal: Connection to 192.168.1.123 port 22: error in libcrypto
Server log:
May 19 20:22:49 eric-Precision-7520 sshd[360940]: debug3: fd 5 is not O_NONBLOCK
May 19 20:22:49 eric-Precision-7520 systemd-journald[409]: Forwarding to syslog missed 310 messages.
May 19 20:22:49 eric-Precision-7520 sshd[360940]: debug1: Forked child 3508801.
May 19 20:22:49 eric-Precision-7520 sshd[360940]: debug3: send_rexec_state: entering fd = 8 config len 330
May 19 20:22:49 eric-Precision-7520 sshd[360940]: debug3: ssh_msg_send: type 0
May 19 20:22:49 eric-Precision-7520 sshd[360940]: debug3: send_rexec_state: done
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: oom_adjust_restore
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: Set /proc/self/oom_score_adj to 0
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: inetd sockets after dupping: 4, 4
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: Connection from 192.168.1.208 port 37310 on 192.168.1.123 port 22 rdomain ""
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.2
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: Remote protocol version 2.0, remote software version OpenSSH_8.5
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: match: OpenSSH_8.5 pat OpenSSH* compat 0x04000000
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: fd 4 setting O_NONBLOCK
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: ssh_sandbox_init: preparing seccomp filter sandbox
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: Network child is on pid 3508802
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: preauth child monitor started
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: privsep user:group 126:65534 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: permanently_set_uid: 126/65534 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: send packet: type 20 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: SSH2_MSG_KEXINIT sent [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: receive packet: type 20 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: SSH2_MSG_KEXINIT received [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: local server KEXINIT proposal [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: KEX algorithms: curve25519-sha256,
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: compression ctos: none,
zlib@openssh.com [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: compression stoc: none,
zlib@openssh.com [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: languages ctos: [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: languages stoc: [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: first_kex_follows 0 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: reserved 0 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: peer client KEXINIT proposal [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: KEX algorithms: curve25519-sha256,
curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: host key algorithms:
ssh-ed25519-cert-v01@openssh.com,
ecdsa-sha2-nistp256-cert-v01@openssh.com,
ecdsa-sha2-nistp384-cert-v01@openssh.com,
ecdsa-sha2-nistp521-cert-v01@openssh.com,
sk-ssh-ed25519-cert-v01@openssh.com,
sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,
rsa-sha2-512-cert-v01@openssh.com,
rsa-sha2-256-cert-v01@openssh.com,
ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,
sk-ssh-ed25519@openssh.com,
sk-ecdsa-sha2-nistp256@openssh.com,rsa-
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: ciphers ctos: aes128-ctr [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: ciphers stoc: aes128-ctr [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: MACs ctos: hmac-sha2-256 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: MACs stoc: hmac-sha2-256 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: compression ctos: none,
zlib@openssh.com,zlib [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: compression stoc: none,
zlib@openssh.com,zlib [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: languages ctos: [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: languages stoc: [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: first_kex_follows 0 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: reserved 0 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: kex: algorithm: curve25519-sha256 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: kex: host key algorithm: ssh-ed25519 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: receive packet: type 30 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_sshkey_sign entering [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_request_send entering: type 6 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_sshkey_sign: waiting for MONITOR_ANS_SIGN [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_request_receive_expect entering: type 7 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_request_receive entering [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_request_receive entering
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: monitor_read: checking request 6
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_answer_sign
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_answer_sign: KEX signature 0x564fca8d1d10(83)
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: mm_request_send entering: type 7
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: monitor_read: 6 used once, disabling now
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: send packet: type 31 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: send packet: type 21 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug2: set_newkeys: mode 1 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: rekey out after 4294967296 blocks [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: Sending SSH2_MSG_EXT_INFO [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug3: send packet: type 7 [preauth]
May 19 20:22:49 eric-Precision-7520 sshd[3508801]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: Connection closed by 192.168.1.208 port 37310 [preauth]
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug1: do_cleanup [preauth]
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug3: PAM: sshpam_thread_cleanup entering [preauth]
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug1: monitor_read_log: child log fd closed
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug3: mm_request_receive entering
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug1: do_cleanup
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug3: PAM: sshpam_thread_cleanup entering
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug1: Killing privsep child 3508802
May 19 20:22:50 eric-Precision-7520 sshd[3508801]: debug1: audit_event: unhandled event 12
May 19 20:22:50 eric-Precision-7520 sshd[360940]: debug1: main_sigchld_handler: Child exited
-Eric