Subject: sftp_read degrading performance

sftp_read degrading performance

From: Alexander Lamaison <swish_at_lammy.co.uk>
Date: Tue, 8 May 2012 02:32:11 +0100

It seems all is not quite right with our old friend sftp_read.

It is very unhappy reading from files with a buffer bigger than the
file. In these cases is actually *sends* more than it receives. When
reading a sequence of such files, the reads get successively slower
after each file is opened, read and closed. Using the very scientific
method of stopping the debugger many times at random, it seems to be
spending most of its time in sftp_packetlist_flush.

My guess is that we are sending vastly more read-ahead requests than
we should. The number we send is based on the buffer size but capped
to LIBSSH2_CHANNEL_WINDOW_DEFAULT*4 (1MB). What I don't understand is
why this causes more of a problem the more files we try to read. If
it's a problem, fine, but surely it's a constant problem?

I've attached the script I'm using to reproduce the results. Execute
it as `libssh2_bigread ip_address username password
/path/to/small/file`. The script opens, reads and closes that small
file repeatedly, forever.

The trace output [1] shows an interesting time gap before each
"Closing handle" line which gets bigger and bigger with each iteration
during which nothing appears in the trace. Could it be waiting on the
socket in BLOCK_ADJUST here? The local CPU is at about 80% the entire
time.

Thoughts, suspicions and suggestions most welcome.

Alex

[1] http://dl.dropbox.com/u/6028779/read2.zip

-- 
Swish - Easy SFTP for Windows Explorer (http://www.swish-sftp.org)


_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel

Received on 2012-05-08