Subject: Re: SFTP Compression support

Re: SFTP Compression support

From: Ezequiel Ruiz <eruiz_at_barcelona04.com>
Date: Wed, 01 Sep 2010 11:57:02 -0300

  Ok, I've been testing the patch you sent me. It seems to solve the
problem when opening files. But, unfortunately I'm getting confused with
some strange behavior in one of my tests:

I've just compiled the "sftp_nonblock.c" example, and tried it with an
80 MB text file in a remote Linux server. I've commented the line where
the output is printed in the stdout to avoid performance penalties. The
problem is that using the compression flag takes more time that without
the using the compression flag! I've tried several times, and all the
tests returned the same results: about 120 seconds with the compression
flag, and about 90 seconds without it. I've tried with buffers of 1K,
16K and 32K.
The compression is enabled at the OpenSSH server side, if I try the
transfer with an SFTP client, like the sftp command, then it takes about
10 or 20 seconds.

Ezequiel

El 01/09/2010 12:23 a.m., Ezequiel Ruiz escribió:
> It seams you find the problem, I'll try your patch and give you some
> feedback.
> Thanks a lot for your research, nice job!
>
> Regards,
>
> Ezequiel
>
> El 31/08/2010 08:03 p.m., TJ Saunders escribió:
>>> [libssh2] 2.676521 Failure Event: -29 - compress/decompression failure
>>> [libssh2] 2.676537 Failure Event: -1 - transport read
>>> [libssh2] 2.676547 Failure Event: -21 - Read part of packet
>>> [libssh2] 2.676558 Failure Event: -1 - Timeout waiting for status message
>>>
>>> And there, I think, is a sign of the issue:
>>>
>>> [libssh2] 2.676521 Failure Event: -29 - compress/decompression failure
>>>
>>> The question is, though: why did the decompression fail? And why does it
>>> fail at this point, when reading file data (as opposed to earlier, when
>>> decompressing the rest of the messages on that subsystem/channel)?
>>>
>>> I'll keep poking at this to see if I can find any additional clues...
>> I patched the src/comp.c file (see first attached patch) to try to get
>> more information; in my particular use case, I saw:
>>
>> [libssh2] 3.316516 Transport: unhandled zlib error -5
>>
>> According to the zlib.h header, a return value of -5 is Z_BUF_ERROR, which
>> indicates (in the case of calling the inflate() zlib function) that there
>> isn't enough space in the output buffer for the decompressor to make any
>> progress.
>>
>> That being the case, I tried increasing the size of the output buffers
>> used for zlib decompression (see second attached patch), and things looked
>> much better. I'm not sure of just what the best increase in output buffer
>> sizes/growth factors should be, though.
>>
>> Cheers,
>> TJ
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>> We are ashamed of everything that is real about us; ashamed of
>> ourselves, of our relatives, of our incomes, of our accents, of
>> our opinions, of our experience, just as we are ashamed of our
>> naked skins.
>>
>> -George Bernard Shaw
>>
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>
>>
>> _______________________________________________
>> libssh2-develhttp://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
>
>
> _______________________________________________
> libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel

_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2010-09-01