Hello Libssh2 Dev Mailing List,
I am working on a cocoa application that uses libssh2 to SFTP files to
and from a server. For the most part, I followed the example
http://www.libssh2.org/examples/sftp.html but made a few changes. Most
of these changes, such as always using public key authentication
instead of password authentication and using the data to initialize an
NSData object instead of writing it to a file once the read operations
have finished, should not affect how much libssh2_sftp_read reads on a
single call.
The only relevant difference is that, instead of reading in data to
one buffer and copying it to another after every call, my version uses
a sliding window approach:
1. Allocate a buffer larger than the largest file the program should
need to read.
2. Create a pointer to the start of the buffer.
3. Call libssh2_sftp_read(sftp_handle, [pointer to start of blank
buffer], [size of blank buffer]);
4. a. If return value is positive,
i. add the number of bytes read to the pointer so that it
points to the start of the bytes that are still blank.
ii. subtract the number of bytes read from the size of the blank buffer.
iii. if buffer is more than half full, allocate a larger
buffer, copy over the data already written, deallocate the old buffer,
point the pointer to the start of the blank spaces in the new buffer,
and set the size of the blank buffer to the number of bytes still
blank in the new buffer.
iv. return to step 3 with the new argument values.
b. If return value is 0, stop and return the data wrapped in an NSData.
c. If return value is negative, report the error and return nil
(similar to null but for Cocoa objects).
This is my version of the read function:
in class header file:
size_t max_file_size;
in initialization method:
max_file_size = 524288;
file_buffer = (char *)malloc(max_file_size);
method implementation:
-(NSData *)readData:(NSString *)filePath error:(NSError **)error
{
NSData *data = nil;
@try {
*error = nil;
//Did we remember to call libssh2_init elsewhere?
if (![SFTPHandler globalsInited]) {
//[...error handling...]
return NO;
}
//We did the authentication and the creation of the
sftp session handle in a previous method call and use this
sftpConnected variable as a boolean to keep track of whether we are
connected or not.
if (!sftpConnected) {
//[...error handling...]
return nil;
}
sftppath = [filePath UTF8String];
sftp_handle = libssh2_sftp_open(sftp_session, sftppath, LIBSSH2_FXF_READ, 0);
if (!sftp_handle) {
//[...error handling...]
return nil;
}
BOOL done = NO;
char *dataEnd;
int dataLength = 0;
int spaceRemaining;
dataEnd = file_buffer;
do {
//If we run out of room, replace the array to which dataStart
points with a bigger array.
spaceRemaining = max_file_size - dataLength;
if (spaceRemaining < data_Length) {//if more than half full,
NSLog(@"ran out of room, making bigger array");
size_t max_file_size_temp = 2*max_file_size;
char *biggerArray = (char *)malloc(max_file_size_temp);
if (!biggerArray) {
//[...error handling...]
done = YES;
break;
}
//Copy over the data we already have.
memcpy(biggerArray, file_buffer, dataLength);
free(file_buffer);
file_buffer = biggerArray;
dataEnd = file_buffer + dataLength;
spaceRemaining = max_file_size_temp - dataLength;
max_file_size = max_file_size_temp;
NSLog(@"max_file_size=%i", max_file_size);
}
//Note the return value is the number of bytes read.
NSLog(@"about to read once: dataEnd=%i, spaceRemaining=%i",
dataEnd, spaceRemaining);
rc = libssh2_sftp_read(sftp_handle, dataEnd, spaceRemaining);
NSLog(@"return value = %i", rc);
if (rc > 0) {
dataEnd += rc;
dataLength += rc;
NSLog(@"updated place markers: dataEnd=%i, dataLength=%i",
dataEnd, dataLength);
}
else {
//Note that we have it set to allow
blocking, so we should not get L_BSSH2_ERROR_EAGAIN
NSLog(@"no data returned this time");
done = YES;
if (rc < 0) {
//[...error handling...]
}
}
} while (!done);
NSLog(@"done reading data");
data = [NSData dataWithBytes:file_buffer length:dataLength];
NSLog(@"created data object for data, length=%i", [data length]);
}
@catch (NSException * e) {
NSLog(@"exception while downloading %@", filePath);
}
@finally {
libssh2_sftp_close(sftp_handle);
NSLog(@"closed file handle");
}
return data;
}
I have been testing this method with both small text files of a few kB
each and Quicktime .mov files of up to 1MB. I am running the app on
64-bit Intel OS X 10.6.8 on the client side and communicating with a
server running the standard linux sftp-server on Redhat Enterprise
Linux Server 5.6. The version of libssh2 I am using is 1.4.0-20120229.
For all files I have tested, the method downloads and returns the
correct data, but it always returns it in chunks no larger than 2K,
regardless of how large I make the buffer and regardless of the size
of the file. More precisely, if the file is smaller than 2K, the first
call downloads the complete file and returns the number of bytes in
the file, but, if the file is larger than 2K, libssh2_sftp_read
returns a value of 2000 after every call except for the second-to-last
call, which returns (file size)%2000, and the last call, which returns
0, indicating we have read all the bytes in the file. At first, I
thought it might be a peculiarity of the server's configuration, but I
do not see any hard limit on sent packet size in sftp-server's
documentation. Is this an inherent limitation of the libssh2_sftp_read
method?
I also wrote a write method along similar lines. It has a maximum
packet size, and, if a file is larger than that size, will use the
sliding window approach to upload one packet's worth at a time until
it has a final piece smaller than the maximum size.
size_t packet_size;
in initialization method:
packet_size = 524288;
file_buffer = (char *)malloc(max_file_size);//see read method
-(BOOL)writeData:(NSData *)data toRemoteFile:(NSString *)filePath
error:(NSError **)error
{
BOOL success;
@try {
*error = nil;
sftppath = [filePath UTF8String];
const char *dataPos = [data bytes];
const char *dataEnd = dataPos + [data length];
if (![SFTPHandler globalsInited]) {
[...error handling...]
return NO;
}
if (!sftpConnected) {
[...error handling...]
return NO;
}
sftp_handle = libssh2_sftp_open(sftp_session, sftppath, write_mode,
permissions);//write_mode=LIBSSH2_FXF_WRITE|LIBSSH2_FXF_CREAT;
permissions=LIBSSH2_SFTP_S_IRWXU|LIBSSH2_SFTP_S_IRGRP; User has full
access. Group has read only access.
if (!sftp_handle) {
[...error handling...]
return NO;
}
//Write a full packet until writing a full packet would take us over
the file size.
while (dataPos + max_file_size < dataEnd) {
//write file
rc = libssh2_sftp_write(sftp_handle, dataPos, max_file_size);
//We allow blocking, so no LIBSSH2_ERROR_EAGAIN.
if (rc <= 0) {
[...error handling...]
success = NO;
break;
}
else {
dataPos += rc;
NSLog(@"complete packet written dataPos=%i, dataEnd=%i, rc=%i",
dataPos, dataEnd, rc);
//dataWritten += rc;
}
}
if (*error == nil) {//When an error occurs, error gets set in the
error handling code I have omitted.
//Check that we still have data to write.
if (dataPos < dataEnd) {
//Write the last partial packet.
NSLog(@"writing last partial packet, size=%i", (dataEnd - dataPos));
rc = libssh2_sftp_write(sftp_handle, dataPos, (dataEnd - dataPos));
NSLog(@"attempted write, return code=%i, packet_size=%i", rc, packet_size);
if (rc < 0) {
[...error handling...]
success = NO;
}
else {
NSLog(@"wrote last partial packet");
success = YES;
}
}
}
else {
success = NO;
}
}
@catch (NSException * e) {
NSLog(@"exception while uploading %@:%@, %@", filePath, [e name], [e reason]);
success = NO;
}
@finally {
libssh2_sftp_close(sftp_handle);
//NSLog(@"closed file");
}
return success;
}
I am testing this with the same client macbook and server and the same
version of libssh2. If I upload a file smaller than 30K,
libssh2_sftp_write, the program tries to upload it all in one call,
succeeds, and returns the correct number of bytes. If I upload a file
larger than 30K but smaller than packet_size, it attempts to upload
all the data in one call. The complete file appears on the server, but
libssh2_sftp_write only returns 30000. If I upload a file larger than
the packet size, each full-packet call returns 30000, causing the loop
to slide the window forward by 30000. Once it gets to a point where
fewer than packet_size bytes remains, it jumps to the last call and
tries to upload all the remaining bytes. This call returns up to
30000. When I check the server, it has the complete file.
It appears that the return value and the amount of data uploaded do not match.
I would like to be able to download larger packets, but I am also
wondering why the functions behave this way. Did I overlook something
in the documentation or in an earlier mailing list post?
Hoping that this is the correct place to send questions about libssh2 usage,
Adam Craig
_______________________________________________
libssh2-devel http://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel
Received on 2012-03-29