Slow SFTP uploads; failure if compression enabled

[This is a resend; it looks as if my message didn't make it to the list the
first time.]

I'm having two types of libssh2 file transfer problems, and I hope someone
can help me.

First, upload performance is slow, especially on Windows.

Secondly, uploads don't work at all if I enable compression.

I am using libssh2-1.2.7 with the compression patch that can be found at:

http://www.libssh2.org/mail/libssh2-devel-archive-2010-09/0000.shtml

I am using libssh2 on two platforms: Windows 2003 R2, and Ubuntu 10.04.

The target servers are Solaris 10 x86 and Linux (standard sshd server).

When I enable compression, uploads fail with error -40 (compression error).

Without compression, uploads succeed, but performance is horrible on Windows
(0.4 MB/sec) and mediocre on Linux (2 MB/sec).

Download performance is good.

The client and server are running on different VMware hosts, with Xeon L5520
CPUs, lots of memory, little competing workload, and a 1 Gbit/sec Ethernet
between the two. So, the hardware is not the limiting factor. At any rate,
other clients on the same VMs get better performance.

Here's a table of performance. In case this doesn't render well in your
email client, I've also uploaded it here:
http://60bits.net/sni/libssh2-perf.htm

Client

Client OS

Server

Comp
Enable

File
Compressible

DL Speed

UL Speed

libssh2

Win

Solaris

No

No

12.98 MB/sec

0.42 MB/sec

libssh2

Win

Solaris

Yes

No

6.59 MB/sec

Fails

libssh2

Linux

Solaris

No

No

9.87 MB/sec

1.85 MB/sec

libssh2

Linux

Solaris

Yes

No

6.58 MB/sec

Fails

libssh2

Win

Linux

No

No

6.63 MB/sec

6.70 MB/sec

libssh2

Win

Linux

Yes

No

Fails: -5

BitVise Tunnelier

Win

Solaris

No

No

13.50 MB/sec

3.95 MB/sec

BitVise Tunnelier

Win

Solaris

Yes

No

8.541 MB/sec

10.2 MB/sec

Ubuntu sftp

Linux

Solaris

?

No

29.6 MB/sec

11.5 MB/sec

psftp

Win

Solaris

No

No

5.06 MB/sec

psftp

Win

Linux

No

No

6.58 MB/sec

I'm attaching a file containing the source (derived from a libssh2 sample)
and a Windows executable.

In case you don't want to look at the attachment, the relevant upload code
is:

      hand = open(settings.localfile.c_str(), O_BINARY | O_RDONLY,
_S_IREAD);

      if (-1 == hand) {

         perror("opening local file");

         exit(2);

      }

      int nwrites = 0;

      // 32500 seems to be the buffer size that actually gets used,

      // so to avoid partial buffers, I'm using exactly that size.

      char mem[32500];

      printf("Using output buffer size of %d bytes\n", sizeof(mem));

      do {

         /* loop until we fail */

         int nbytes = read(hand, mem, sizeof(mem));

         if (nbytes > 0) {

            filesize += nbytes;

            int offset = 0, bytes_to_send = nbytes;

            // Loop to ensure that all of the input buffer is sent.

            do {

               rc = libssh2_sftp_write(sftp_handle, mem + offset,
bytes_to_send);

               if (rc < 0) {

                  printf("libssh2_sftp_write failed with code %d\n", rc);

                  break;

               }

               ++nwrites;

               if (0 == (nwrites % 100)) {

                  printf("libssh2_sftp_write %d sent %d bytes\n", nwrites,
rc);

               }

               offset += rc;

               bytes_to_send -= rc;

            } while (bytes_to_send > 0);

         } else {

            break;

         }

      } while (rc >= 0);

   } else {

      puts("action must be get or put.");

      exit(1);

   }

   close(hand);

   libssh2_sftp_close(sftp_handle);

Thanks for any ideas you might have!

Mark R

_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2010-10-22