Subject: Re: libssh2_sftp_read reads a number of bytes smaller than both the file size and the specified buffer size

Re: libssh2_sftp_read reads a number of bytes smaller than both the file size and the specified buffer size

From: Adam Craig <adamgcraig_at_accessnoexcuse.org>
Date: Tue, 3 Apr 2012 14:41:55 -0700

Mr. Lamaison,

I apologize for not replying sooner. Gmail hid your reply for some
reason, and I did not notice it until just now.

On 3/29/12, Alexander Lamaison <swish_at_lammy.co.uk> wrote:
> On 29 March 2012 01:19, Adam Craig <adamgcraig_at_accessnoexcuse.org> wrote:
>> Hello Libssh2 Dev Mailing List,
>>
>> I am working on a cocoa application that uses libssh2 to SFTP files to
>> and from a server.
>
> ... snip
>
>> I have been testing this method with both small text files of a few kB
>> each and Quicktime .mov files of up to 1MB. I am running the app on
>> 64-bit Intel OS X 10.6.8 on the client side and communicating with a
>> server running the standard linux sftp-server on Redhat Enterprise
>> Linux Server 5.6. The version of libssh2 I am using is 1.4.0-20120229.
>>
>> For all files I have tested, the method downloads and returns the
>> correct data, but it always returns it in chunks no larger than 2K,
>> regardless of how large I make the buffer and regardless of the size
>> of the file. More precisely, if the file is smaller than 2K, the first
>> call downloads the complete file and returns the number of bytes in
>> the file, but, if the file is larger than 2K, libssh2_sftp_read
>> returns a value of 2000 after every call except for the second-to-last
>> call, which returns (file size)%2000, and the last call, which returns
>> 0, indicating we have read all the bytes in the file. At first, I
>> thought it might be a peculiarity of the server's configuration, but I
>> do not see any hard limit on sent packet size in sftp-server's
>> documentation. Is this an inherent limitation of the libssh2_sftp_read
>> method?
>
> So just to summarise, you're not reporting that the transfer is
> incomplete? Just that you wish the transfer used bigger chunks at a
> time? This is a perfectly valid thing to report, I just want to check
> I'm understanding you correctly.
>

That is correct.

> ... snip
>
>> I am testing this with the same client macbook and server and the same
>> version of libssh2. If I upload a file smaller than 30K,
>> libssh2_sftp_write, the program tries to upload it all in one call,
>> succeeds, and returns the correct number of bytes. If I upload a file
>> larger than 30K but smaller than packet_size, it attempts to upload
>> all the data in one call. The complete file appears on the server, but
>> libssh2_sftp_write only returns 30000. If I upload a file larger than
>> the packet size, each full-packet call returns 30000, causing the loop
>> to slide the window forward by 30000. Once it gets to a point where
>> fewer than packet_size bytes remains, it jumps to the last call and
>> tries to upload all the remaining bytes. This call returns up to
>> 30000. When I check the server, it has the complete file.
>> It appears that the return value and the amount of data uploaded do not
>> match.
>
> This sounds like the expected behaviour of libssh2's pipelined SFTP
> architecture. The return value doesn't have to correspond to the
> amount uploaded on _that_ call as long as the overall total is
> correct. The last call that doesn't appear to do anything allows
> libssh2 to do some housekeeping.
What do you mean by the "overall total?"
If I upload a file of 200K all in one call to libssh2_sftp_write and
it returns 30K when it actually uploaded all 200K, what does the 30K
return value mean?
Does the documentation explain this somewhere?
All te information on this I could find was
http://www.libssh2.org/libssh2_sftp_write.html:
"RETURN VALUE

Actual number of bytes written or negative on failure.

If used in non-blocking mode, it returns LIBSSH2_ERROR_EAGAIN when it
would otherwise block. While LIBSSH2_ERROR_EAGAIN is a negative
number, it isn't really a failure per se.

If this function returns 0 (zero) it should not be considered an
error, but simply that there was no error but yet no payload data got
sent to the other end."

>
>> I would like to be able to download larger packets
>
> I'm not sure why yours are limited to 2K but I believe this is
> negotiated with the server (jump in here guys if I'm wrong) so it's
> possible your server is refusing to send bigger packets.

That was my first guess, and it may be the case. After looking at the
documentation for the server-side SFTP tool and not finding any
information about a maximum packet size, I decided to look into
whether it was an issue with Libssh2.

Documentation I read for the SFTP server utility to which the app is connecting:
http://www.openbsd.org/cgi-bin/man.cgi?query=sftp-server&sektion=8
http://www.openbsd.org/cgi-bin/man.cgi?query=sshd_config&sektion=5

Adam

>
> Alex
>
> --
> Swish - Easy SFTP for Windows Explorer (http://www.swish-sftp.org)
> _______________________________________________
> libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
>
_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2012-04-03