On 4/20/12, Daniel Stenberg <daniel_at_haxx.se> wrote:
> On Thu, 19 Apr 2012, Adam Craig wrote:
>
>> I found that, with the remote server, while larger packet sizes yielded
>> some
>> improvement in speed, it was only barely noticeable compared to the
>> fluctuations in speed due to outside factors. On the local network, the
>> improvement in speed is pronounced from 2K to 20K but levels off at higher
>>
>> sizes.
>
> Thanks for doing this!
>
> Unfortunately I still believe there are too many unknowns in this equation
> and
> you've done your tests on a too limited test set to make us draw very many
> conclusions for libssh2 genericly as opposed for libssh2 in your particular
> case.
>
My apologies. I do not have the time or resources right now to do a
more comprehensive study. Hopefully I will when I take care of some of
my other standing commitments. In the mean time, I will study the RFC
and libssh2 source code more carefully.
> For example:
>
> A) you only used OpenSSH servers on Linux, possibly not even using very
> different versions. we must expect different server ends to react
> differently
>
> B) what were the RTT times against these servers? SFTP will perform
> significantly different for varying RTT. One of the major challanges
> with
> SFTP is to make it run fast for the whole range of possible RTT
>
Again, I must apologize. The remote Red Hat server has become
permanently unavailable, so I cannot get an RTT measurement for it.
Ubuntu Server on local network:
PING 10.0.0.16 (10.0.0.16): 56 data bytes
64 bytes from 10.0.0.16: icmp_seq=0 ttl=64 time=3.821 ms
64 bytes from 10.0.0.16: icmp_seq=1 ttl=64 time=2.786 ms
64 bytes from 10.0.0.16: icmp_seq=2 ttl=64 time=3.268 ms
64 bytes from 10.0.0.16: icmp_seq=3 ttl=64 time=2.501 ms
64 bytes from 10.0.0.16: icmp_seq=4 ttl=64 time=2.616 ms
64 bytes from 10.0.0.16: icmp_seq=5 ttl=64 time=4.580 ms
64 bytes from 10.0.0.16: icmp_seq=6 ttl=64 time=3.031 ms
64 bytes from 10.0.0.16: icmp_seq=7 ttl=64 time=5.368 ms
64 bytes from 10.0.0.16: icmp_seq=8 ttl=64 time=3.966 ms
64 bytes from 10.0.0.16: icmp_seq=9 ttl=64 time=3.210 ms
64 bytes from 10.0.0.16: icmp_seq=10 ttl=64 time=4.215 ms
64 bytes from 10.0.0.16: icmp_seq=11 ttl=64 time=6.010 ms
64 bytes from 10.0.0.16: icmp_seq=12 ttl=64 time=2.678 ms
64 bytes from 10.0.0.16: icmp_seq=13 ttl=64 time=2.765 ms
64 bytes from 10.0.0.16: icmp_seq=14 ttl=64 time=2.799 ms
64 bytes from 10.0.0.16: icmp_seq=15 ttl=64 time=2.788 ms
64 bytes from 10.0.0.16: icmp_seq=16 ttl=64 time=2.998 ms
64 bytes from 10.0.0.16: icmp_seq=17 ttl=64 time=3.258 ms
64 bytes from 10.0.0.16: icmp_seq=18 ttl=64 time=3.044 ms
64 bytes from 10.0.0.16: icmp_seq=19 ttl=64 time=2.806 ms
64 bytes from 10.0.0.16: icmp_seq=20 ttl=64 time=2.652 ms
64 bytes from 10.0.0.16: icmp_seq=21 ttl=64 time=2.678 ms
64 bytes from 10.0.0.16: icmp_seq=22 ttl=64 time=3.037 ms
> C) which crypto backend and version was used in libssh2? I've seen people
> report very different results when using gcrypt vs openssl
libssh2 version: 1.4.0-20120229
Since I did not specify gcrypt, it should be using openssl.
OpenSSL version: 0.9.8r 8 Feb 2011, which came with the Macbook. It
seems to have some customizations unique to OS X, like automatically
using the OS X keychain in stead of requiring a trust store from a
file or directory. I have not been able to find a complete list of
differences between the openssl that ships with OS X and the standard
version.
>
> D) how large buffers did you pass in to libssh2_sftp_read() when you ran
> these
> tests and how did did different such sizes affect the test results?
>
I allocated a buffer of size 524288, about twice the size of the
largest file I needed to upload. The stated sizes I passed in to the
libssh2_sftp_read function were the same as the max chunk sizes listed
in the spreadsheet.
>> I noticed the following comment on the line defining the limit on
>> upload chunk size:
>> /*
>> * MAX_SFTP_OUTGOING_SIZE MUST not be larger than 32500 or so. This is the
>> * amount of data sent in each FXP_WRITE packet
>> */
>> #define MAX_SFTP_OUTGOING_SIZE 32500
>>
>> My guess is the "MUST" is due to the following, found in libssh2_priv.h:
>
> It is due to phrasing in the SSH and SFTP protocol specs, yes.
>
>> I also noticed the following in sftp.c:
>> /* This is the maximum packet length to accept, as larger than this
>> indicate
>> some kind of server problem. */
>> #define LIBSSH2_SFTP_PACKET_MAXLEN 80000
>>
>> I do not see why 80K is the cutoff here.
>
> It is no magic "cutoff" really, it is just a way for libssh2 to detect
> problems as if it gets a very large packet size it is a sign that something
> is
> wrong and it should escape rather than trying to allocate memory for it.
>
>> If the spec says the max packet size is 35K, then it would make more sense
>>
>> to make all the hard-coded limits on packet size 35K
>
> Then read the spec. Does it say 35K? And further, does 35K work against all
> the relevant server implementations? We're being slightly conservative in
> the
> name of interoperability.
>
This was a mistake on my part. According to
http://tools.ietf.org/html/rfc4253#section-6.1, 35K is the maximum
total packet size. The uncompressed data payload should only be up to
32768 bytes. Thus, 32768 should work against any spec-compliant server
implementations. Have you encountered many cases where it has needed
to be smaller?
>> and let users limit packet size further through the buffer sizes they pass
>>
>> to the read and write statements.
>
> Then you have grossly missed the finer implementation details of the SFTP
> read
> and write functions libssh2 provides. It does a lot of work to allow
> applications to get high throughput by splitting up the buffer in many
> slices.
> (which is what I alluded to above in my point "D"). Leaving that to the
> application would certainly be possible, but will require a new API to still
> offer high speed transfers.
>
Apparently I have misunderstood something. I will look into it, but it
may be a while, since I have other aspects of my current project that
still need a lot of work.
Thank you again,
Adam Craig
> --
>
> / daniel.haxx.se
> _______________________________________________
> libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
>
_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2012-04-26