Subject: OpenSSL AES-CTR not working?

OpenSSL AES-CTR not working?

From: Paul Howarth <paul_at_city-fan.org>
Date: Sat, 17 Mar 2012 14:09:17 +0000

I've been trying to build libssh2 (1.4.0) against openssl 1.0.1 in Fedora
Rawhide and it fails to build because global.c calls
_libssh2_init_aes_ctr but that function isn't defined in openssl.c if
HAVE_EVP_AES_128_CTR is defined, resulting in:

/bin/sh ../libtool --tag=CC --mode=link gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wl,-z,relro -o direct_tcpip direct_tcpip.o ../src/libssh2.la
libtool: link: gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wl,-z -Wl,relro -o .libs/direct_tcpip direct_tcpip.o ../src/.libs/libssh2.so -Wl,-rpath -Wl,/usr/lib64
../src/.libs/libssh2.so: undefined reference to `_libssh2_init_aes_ctr'
collect2: error: ld returned 1 exit status

So I tried patching global.c not to call _libssh2_init_aes_ctr if
HAVE_EVP_AES_128_CTR is defined, which fixed the compile problem but
broke the test suite:

Connection from 127.0.0.1 port 49111^M
debug1: Client protocol version 2.0; client software version libssh2_1.4.0^M
debug1: no match: libssh2_1.4.0^M
debug1: Enabling compatibility mode for protocol 2.0^M
debug1: Local version string SSH-2.0-OpenSSH_5.9^M
debug2: fd 3 setting O_NONBLOCK^M
debug1: list_hostkey_types: ssh-rsa^M
debug1: SSH2_MSG_KEXINIT sent^M
debug1: SSH2_MSG_KEXINIT received^M
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1^M
debug2: kex_parse_kexinit: ssh-rsa^M
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc_at_lysator.liu.se^M
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc_at_lysator.liu.se^M
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64_at_openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160_at_openssh.com,hmac-sha1-96,hmac-md5-96^M
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64_at_openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160_at_openssh.com,hmac-sha1-96,hmac-md5-96^M
debug2: kex_parse_kexinit: none,zlib_at_openssh.com^M
debug2: kex_parse_kexinit: none,zlib_at_openssh.com^M
debug2: kex_parse_kexinit: ^M
debug2: kex_parse_kexinit: ^M
debug2: kex_parse_kexinit: first_kex_follows 0 ^M
debug2: kex_parse_kexinit: reserved 0 ^M
debug2: kex_parse_kexinit: diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1^M
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss^M
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes256-cbc,rijndael-cbc_at_lysator.liu.se,aes192-cbc,aes128-cbc,blowfish-cbc,arcfour128,arcfour,cast128-cbc,3des-cbc^M
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,aes256-cbc,rijndael-cbc_at_lysator.liu.se,aes192-cbc,aes128-cbc,blowfish-cbc,arcfour128,arcfour,cast128-cbc,3des-cbc^M
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,hmac-ripemd160_at_openssh.com^M
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,hmac-ripemd160_at_openssh.com^M
debug2: kex_parse_kexinit: none^M
debug2: kex_parse_kexinit: none^M
debug2: kex_parse_kexinit: ^M
debug2: kex_parse_kexinit: ^M
debug2: kex_parse_kexinit: first_kex_follows 0 ^M
debug2: kex_parse_kexinit: reserved 0 ^M
debug2: mac_setup: found hmac-sha1^M
debug1: kex: client->server aes128-ctr hmac-sha1 none^M
debug2: mac_setup: found hmac-sha1^M
debug1: kex: server->client aes128-ctr hmac-sha1 none^M
debug2: dh_gen_key: priv key bits set: 176/320^M
debug2: bits set: 1022/2048^M
debug1: expecting SSH2_MSG_KEXDH_INIT^M
debug2: bits set: 1012/2048^M
debug2: kex_derive_keys^M
debug2: set_newkeys: mode 1^M
debug1: SSH2_MSG_NEWKEYS sent^M
debug1: expecting SSH2_MSG_NEWKEYS^M
debug2: set_newkeys: mode 0^M
debug1: SSH2_MSG_NEWKEYS received^M
debug1: KEX done^M
Corrupted MAC on input.^M
Disconnecting: Packet corrupt^M

Do the openssl EVP functions need some initialization to fix this?

Paul.
_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2012-03-17