Subject: LIBSSH2 random crash in openssl library

LIBSSH2 random crash in openssl library

From: Srikanth Bemineni <bemineni.srikanth_at_gmail.com>
Date: Tue, 17 Feb 2015 00:29:54 -0600

Hi,

We are seeing this random crash in libssh2 while copying files in a multi
threaded application. Our application is threaded application, and each
file transfer happens in its own thread. Each thread creates its own
libssh2 session and a channel to transfer the file. Occasionally we see
this random crash in openssl library. Below is the stack trace of the crash.

               some_application.exe!sha1_block_data_order() + 0x2b5c
bytes
               some_application.exe!SHA1_Update(SHAstate_st * c, const void
* data_, unsigned __int64 len) Line 326 C
               some_application.exe!update(env_md_ctx_st * ctx, const void
* data, unsigned __int64 count) Line 78 + 0x34 bytes C
               some_application.exe!EVP_DigestUpdate(env_md_ctx_st * ctx,
const void * data, unsigned __int64 count) Line 252 C
               some_application.exe!ssleay_rand_bytes(unsigned char * buf,
int num, int pseudo, int lock) Line 499 C
               some_application.exe!ssleay_rand_nopseudo_bytes(unsigned
char * buf, int num) Line 542 C
               some_application.exe!RAND_bytes(unsigned char * buf, int
num) Line 165 + 0x11 bytes C

some_application.exe!_libssh2_transport_send(_LIBSSH2_SESSION * session,
const unsigned char * data, unsigned __int64 data_len, const unsigned char
* data2, unsigned __int64 data2_len) Line 820 C
               some_application.exe!_libssh2_channel_write(_LIBSSH2_CHANNEL
* channel, int stream_id, const unsigned char * buf, unsigned __int64
buflen) Line 2060 + 0x46 bytes C

some_application.exe!libssh2_channel_write_ex(_LIBSSH2_CHANNEL * channel,
int stream_id, const char * buf, unsigned __int64 buflen) Line 2109 + 0x24
bytes C
               some_application.exe!Ssha::scp_write_file(QString *
local_path, _LIBSSH2_CHANNEL * channel_new) Line 761 + 0x1b bytes C++
> some_application.exe!Ssha::PutFile(QString * local_path,
QString * remote_path) Line 854 + 0x4a bytes C++

When I look at the session errmsg Its says "Unable to send channel data"
with errcode LIBSSH2_ERROR_EAGAIN

The locking mechanism is also in place for open ssl. We did check that
openssl global data is locked and released by the mutex. Is there anything
that we are missing from the libssh2 perspective ?

init()
{
    libssh2_init(0);
    CRYPTO_malloc_init();
    CRYPTO_thread_setup()
}

shutDownSequence()
{
    libssh2_exit();
    CRYPTO_thread_cleanup();
}

void locking_function(int mode, int n, const char *file, int line)
{
    if (mode & CRYPTO_LOCK)
    {
        if(mutex_buf[n] != NULL)
        {
            mutex_buf[n]->lock();
        }
    }
    else
    {
        if(mutex_buf[n] != NULL)
        {
            mutex_buf[n]->unlock();
        }
    }
}

unsigned long id_function()
{
    return (unsigned long)QThread::currentThreadId();
}

void Cb_function(CRYPTO_THREADID *id)
{
  CRYPTO_THREADID_set_numeric(id, (unsigned
long)QThread::currentThreadId());
}

bool CRYPTO_thread_setup()
{
  int i;
  int num = CRYPTO_num_locks();
    for (i = 0; i < num; i++)
    {
      lock_count[i] = 0;
      mutex_buf[i] = new QMutex(QMutex::NonRecursive);
    }
    CRYPTO_set_id_callback(id_function);
    CRYPTO_THREADID_set_callback(Cb_function);
    CRYPTO_set_locking_callback(locking_function);
    CRYPTO_set_dynlock_create_callback(dyn_create_function);
    CRYPTO_set_dynlock_lock_callback(dyn_lock_function);
    CRYPTO_set_dynlock_destroy_callback(dyn_destroy_function);
  return true;
}

void CRYPTO_thread_cleanup()
{
  int i;
  int num = CRYPTO_num_locks();
  if(crypto_initialized != 0)
  {
    CRYPTO_set_id_callback(NULL);
    CRYPTO_THREADID_set_callback(NULL);
    CRYPTO_set_locking_callback(NULL);
    CRYPTO_set_dynlock_create_callback(NULL);
    CRYPTO_set_dynlock_lock_callback(NULL);
    CRYPTO_set_dynlock_destroy_callback(NULL);
    for (i = 0; i < num; i++)
    {
      if(mutex_buf[i])
        delete(mutex_buf[i]);
    }
    crypto_initialized = 0;
  }
}

I see a similar issue reported as a bug in
http://trac.libssh2.org/ticket/212 . The resolution says adding
libssh2_init(0); fixed the issue. This has already been taken care in our
code, but we still see the crash.

Srikanth Bemineni

_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2015-02-17