From libssh2-devel-bounces@cool.haxx.se Sun Aug 26 00:56:52 2018 Return-Path: Received: from www.haxx.se (mail [127.0.0.1]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTP id w7PMuDhv017863; Sun, 26 Aug 2018 00:56:40 +0200 Received: from mail-yb0-x22a.google.com (mail-yb0-x22a.google.com [IPv6:2607:f8b0:4002:c09:0:0:0:22a]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTPS id w7PMuAV2017809 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sun, 26 Aug 2018 00:56:11 +0200 Received: by mail-yb0-x22a.google.com with SMTP id c4-v6so4779852ybl.6 for ; Sat, 25 Aug 2018 15:56:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=QuZFxlfqsS60p4B9Dx3867TsddU4YKXWg1WITlwUe1k=; b=srvSPvf8cRKr2TbB7PxsTeNw7443WT93QJfwq9SQTEFcOio4TI0Yowr5iNFTGSOOjs py2f4pLg6ypmHvRWFrOdA4PPH+yFrjFirulfs1Zb2ag3p8dweJVzqQ3Jy0ybtFTu99rh 1CkXA93bL7DFuK1TDhDo8ee+VLQzQLTCFbym2ZewY+CUDiBf/ov9rV5w129sarn0Pjxz +aIZGIkF3HwqwWWeUr6YGBWjOZ8I5E/RHC6ESug9XNWxthXztiwJg/gn8LkSr8q5O2QZ nSRZoAgqt5kstjd3a4RgXF/T7Mpp02yH1wNoPXqOMjTj8dIJspnJCASVL/Lvly32qqig +FYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=QuZFxlfqsS60p4B9Dx3867TsddU4YKXWg1WITlwUe1k=; b=Ymr1kcq0VN3I4KPTTXaZb6eqALH/LKFhACkcJ94VpOYS4VYKYzzVemq4qKyMruzKU0 zosJXov1Bxild5/1QjRpowQutDpbEj3rfPMOOodlgthHhkmyyThm83AZfOJQj8cbTOZp eckTI0F8HBdjM49B1f6yp65t60uWdI/ljVUA0nsqatlnp9GLeT6EsyKGODyP009P0TUi C9YxgSkwAcXNHkNa3QKygfflho4tadk1L/uXzW5IuXyZKNTJatVnLEi0nnk+Lni54qVw MIzVk6zcs8V47leRFnEI5SNqHvOeDgZGrihtKXRnS/fJmeDZXQobJ1g9L/iaUAdFvfs/ dZEQ== X-Gm-Message-State: APzg51DPWJmEs4pRm8Ystj66/452az1Z/LV5x0CRS2lPN5Qx8x3lOEsW qrqszHXZvxsSqkZNtBc8QFHBAw0xHj7FSUFo+UvMQrcP X-Google-Smtp-Source: ANB0VdYfu7AK0XSerr4YU2/X5pOvZbeB8pTwI02WjJYKyifdxeQMWSfWGnLtrZYHieiLR5zXqzU/azO/+iHDm9BlfPo= X-Received: by 2002:a25:ae54:: with SMTP id g20-v6mr4087667ybe.83.1535237766824; Sat, 25 Aug 2018 15:56:06 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?Q?Daniel_Jeli=C5=84ski?= Date: Sun, 26 Aug 2018 00:55:55 +0200 Message-ID: Subject: curl / libssh2 sftp write performance (with patch) To: libssh2-devel@cool.haxx.se X-BeenThere: libssh2-devel@cool.haxx.se X-Mailman-Version: 2.1.22 Precedence: list List-Id: libssh2 development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: libssh2 development Content-Type: text/plain; charset="utf-8" Errors-To: libssh2-devel-bounces@cool.haxx.se Sender: "libssh2-devel" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by giant.haxx.se id w7PMuDhv017863 Hello all, I took a look at curl performance when uploading data to sftp servers; curl is internally using a fixed-size buffer, which is only refilled when completely acknowledged by sftp write function. This creates a performance bottleneck on high-latency links, as there's a pause in upload every time curl is waiting for acknowledgements. I propose a change implemented in https://github.com/libssh2/libssh2/pull/264, where sftp write acknowledges data as soon as it is sent to the server, and checks server response when it becomes available, which may happen during sftp close. I realize that this is a breaking change - many code examples do not even check the result of sftp close, which means that a lot of code in the wild would probably break if the patch was accepted in its current form. Would a patch like that be acceptable? What needs to happen to get it accepted? Thanks, Daniel _______________________________________________ libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel From libssh2-devel-bounces@cool.haxx.se Sun Aug 26 10:57:02 2018 Return-Path: Received: from www.haxx.se (mail [127.0.0.1]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTP id w7Q8uSON004829; Sun, 26 Aug 2018 10:56:51 +0200 Received: from giant.haxx.se (mail [127.0.0.1]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTPS id w7Q8uRCP004820 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Sun, 26 Aug 2018 10:56:27 +0200 Received: from localhost (dast@localhost) by giant.haxx.se (8.15.2/8.15.2/Submit) with ESMTP id w7Q8uR4n004816 for ; Sun, 26 Aug 2018 10:56:27 +0200 X-Authentication-Warning: giant.haxx.se: dast owned process doing -bs Date: Sun, 26 Aug 2018 10:56:27 +0200 (CEST) From: Daniel Stenberg X-X-Sender: dast@giant.haxx.se To: libssh2 development Subject: Re: curl / libssh2 sftp write performance (with patch) In-Reply-To: Message-ID: References: User-Agent: Alpine 2.20 (DEB 67 2015-01-07) X-fromdanielhimself: yes MIME-Version: 1.0 Content-Type: multipart/mixed; BOUNDARY="1129329158-221618483-1535273787=:22619" X-BeenThere: libssh2-devel@cool.haxx.se X-Mailman-Version: 2.1.22 Precedence: list List-Id: libssh2 development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: libssh2 development Errors-To: libssh2-devel-bounces@cool.haxx.se Sender: "libssh2-devel" This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --1129329158-221618483-1535273787=:22619 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8BIT On Sun, 26 Aug 2018, Daniel Jeliński wrote: > I propose a change implemented in > https://github.com/libssh2/libssh2/pull/264, where sftp write acknowledges > data as soon as it is sent to the server, and checks server response when it > becomes available, which may happen during sftp close. So will it not care for any ACKs? If you send a 10GB file and the first packet is never acked? Maybe a limit for amount of outstanding un-acked data? > I realize that this is a breaking change - many code examples do not even > check the result of sftp close, which means that a lot of code in the wild > would probably break if the patch was accepted in its current form. Can we make users opt-in to this and if not, do like before? -- / daniel.haxx.se --1129329158-221618483-1535273787=:22619 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KbGlic3NoMi1k ZXZlbCBodHRwczovL2Nvb2wuaGF4eC5zZS9jZ2ktYmluL21haWxtYW4vbGlzdGluZm8vbGlic3No Mi1kZXZlbAo= --1129329158-221618483-1535273787=:22619-- From libssh2-devel-bounces@cool.haxx.se Sun Aug 26 21:25:19 2018 Return-Path: Received: from www.haxx.se (mail [127.0.0.1]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTP id w7QJOicq007894; Sun, 26 Aug 2018 21:25:10 +0200 Received: from mail-yw1-xc30.google.com (mail-yw1-xc30.google.com [IPv6:2607:f8b0:4864:20:0:0:0:c30]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTPS id w7QJOgYn007851 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Sun, 26 Aug 2018 21:24:43 +0200 Received: by mail-yw1-xc30.google.com with SMTP id x83-v6so4852023ywd.4 for ; Sun, 26 Aug 2018 12:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=DTnq0ikQejMTETPp8sLgBzbP2nhLApbtQPJEQDoeoCA=; b=LCv1egV3z6UmEHLjc3t+Vw2/vLRMwlBUE8KWnTQoz2ceYuKw54Ape3Bebrkp8lHPmw nV4VpdvB8QupE5a2CBiwEjbVHj4+e5wupHCakYbP6WEb6Z4DnQezT3+0j96GTLVwcVTf uI3PBBR/7y3OcQ2NArvOfFKhVuA7nqwWtFW9Lr4ysUgUo3OuVBj7v5MuvT3LQNtChUFF R+fCtYGuEyeH6i7uVUzeTCr4eNY+Mcw05H9weCWkr8kpDl9ShbloGA5HAEvoouGGJj5N N+/fFq6nfPfyPLbgS9C6glz/ZDi4dsb1EC2nKFSUvQ3bFPDO01q+IkjhVpBUS47Ru3m8 Owmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=DTnq0ikQejMTETPp8sLgBzbP2nhLApbtQPJEQDoeoCA=; b=V18Lvt509NW/JodGY+cOsuTc677PbT97ULjQ4ZmED5HUgbTdMVT0o+UmMBHvAg5DC/ Egny7nC8VGw5ZdyXUFQvLKLwFraga8vj61kepK1eWFSGWce10oa4gKDFfRQmbwjLefwt t5oEnbp6mN0slWAmYAahlWuopfSc6YSpLGvO1w3YX6d968v7CzMD3CzceK2q9+8pKN2c JuZ7XNL4fpvE1T68x/jHxrU5NeqZBgtgoD9R3bEW2FsO5NHGIsiuh9dre0jfhV/uGu3o ue4Djv1q5wP8F4rcyeWFQaVBdbqAE7aWB/fAq1zVoaxRwEwK9Ly+EVqjxEf7Nh9MUb9A uIvg== X-Gm-Message-State: APzg51Av19tNLqCrjpqfm3B89xIjsbKmIZOXO2nQZL3OYH3637guMudr S8410cbs/JoM0jf2U/N6kIPacWylC4onPxo9X7lzbjPg X-Google-Smtp-Source: ANB0VdaMYjeeBMsx3MQHGhxUCiFS8bFfu1Fvo+k0bHEqkteZP3efIrnz1cOzjDpcRGGXgswb0uh8YGZmCLRufL5DIM4= X-Received: by 2002:a81:9fc2:: with SMTP id w185-v6mr5663014ywg.163.1535311478613; Sun, 26 Aug 2018 12:24:38 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?Q?Daniel_Jeli=C5=84ski?= Date: Sun, 26 Aug 2018 21:24:26 +0200 Message-ID: Subject: Re: curl / libssh2 sftp write performance (with patch) To: libssh2-devel@cool.haxx.se X-MIME-Autoconverted: from quoted-printable to 8bit by giant.haxx.se id w7QJOgYn007851 X-BeenThere: libssh2-devel@cool.haxx.se X-Mailman-Version: 2.1.22 Precedence: list List-Id: libssh2 development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: libssh2 development Content-Type: text/plain; charset="utf-8" Errors-To: libssh2-devel-bounces@cool.haxx.se Sender: "libssh2-devel" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by giant.haxx.se id w7QJOicq007894 niedz., 26 sie 2018 o 10:57 Daniel Stenberg napisał(a): > So will it not care for any ACKs? If you send a 10GB file and the first packet > is never acked? Maybe a limit for amount of outstanding un-acked data? No, not like that; available acks are processed after every successful send. We just don't wait for acks that are not available yet, at least in nonblocking mode. I don't know how the code would behave in blocking mode, I should probably check that. Side note, I had a coding bug that caused just the behavior you describe, and I sent 1GB file before checking for any acks. This resulted in a dramatic slowdown when sftp_close was looking for its ack in the unprocessed list of received packets. Limiting the number of outstanding packets sounds reasonable; it would protect us against evil ssh servers. However, I don't see an easy way to figure out a number that would never limit our transfer rates. Need to think it through. > Can we make users opt-in to this and if not, do like before? How would you suggest to implement that? I don't see any existing mechanism that could be used for this; there's no libssh2_sftp_setopt, no version argument in libssh2_sftp_init, no extra parameter to libssh2_sftp_open that could be used for this. I could implement this as a new set of functions duplicating the existing functionality (like posix defines write and fwrite, I could add libssh2_sftp_fwrite), if you think that's the right direction to take. _______________________________________________ libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel From libssh2-devel-bounces@cool.haxx.se Mon Aug 27 23:54:50 2018 Return-Path: Received: from www.haxx.se (mail [127.0.0.1]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTP id w7RLsEuA025627; Mon, 27 Aug 2018 23:54:38 +0200 Received: from mail.panic.com (mail.panic.com [38.103.165.36]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTPS id w7RLsBD1025570 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Mon, 27 Aug 2018 23:54:13 +0200 Received: from [10.0.0.227] (unknown [10.0.0.227]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.panic.com (Postfix) with ESMTPSA id B6C32256D for ; Mon, 27 Aug 2018 14:54:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=panic.com; s=dkim; t=1535406846; bh=idfMXyIl2gNeECtjeNJbFu4cCPmeKBaI7w/sSdhuiZA=; h=From:Subject:Date:References:To:In-Reply-To; b=TGDac4zFbt1XLtwPSqrfpt9WN0snrYvXwIiPZl/8nYGxHyVqXHqggJykw/aDCmkhk ezCN/bs/kWiYdSKLpjpEApltRDuRlts/9bWOBpbF3FEzW27rCKO12pA9NxiA2iX6fR C5qFprrZlg8cEC6zR9izd5Y+cx7xLvqHS7kl+r40= From: Will Cosgrove Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: curl / libssh2 sftp write performance (with patch) Date: Mon, 27 Aug 2018 14:54:06 -0700 References: To: libssh2 development In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3445.9.1) X-MIME-Autoconverted: from quoted-printable to 8bit by giant.haxx.se id w7RLsBD1025570 X-BeenThere: libssh2-devel@cool.haxx.se X-Mailman-Version: 2.1.22 Precedence: list List-Id: libssh2 development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: libssh2 development Content-Type: text/plain; charset="utf-8" Errors-To: libssh2-devel-bounces@cool.haxx.se Sender: "libssh2-devel" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by giant.haxx.se id w7RLsEuA025627 > >> So will it not care for any ACKs? If you send a 10GB file and the first packet >> is never acked? Maybe a limit for amount of outstanding un-acked data? > > No, not like that; available acks are processed after every successful > send. We just don't wait for acks that are not available yet, at least > in nonblocking mode. I don't know how the code would behave in > blocking mode, I should probably check that. I’ve written custom pipelining code for upload/download using some of the libssh2 primitive functions. I also recommend having a max outstanding packet number, I’ve saturated lesser servers by flooding them with concurrent writes. Currently our value is 64 outstanding writes which, I think, mirrors OpenSSH. The approach I took is slowly increasing the concurrent requests. I send an initial request and wait to see if it fails due of a write error or other transient error and then start sending concurrent requests if the first synchronous write was successful. I then ramp the requests slowly by first sending 2 writes, drain the ready replies, then increase the outstanding requests by one and drain and so on until I reach the max of 64 outstanding requests. This gradual increase allows small files to upload without immediately doing a bunch of concurrent writes to the server, seems to work well in practice. Hope this helps, Will _______________________________________________ libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel From libssh2-devel-bounces@cool.haxx.se Tue Aug 28 12:09:09 2018 Return-Path: Received: from www.haxx.se (mail [127.0.0.1]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTP id w7SA8csm009669; Tue, 28 Aug 2018 12:09:02 +0200 Received: from foo.stuge.se (foo.stuge.se [212.116.89.98]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTPS id w7SA8adu009603 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Tue, 28 Aug 2018 12:08:37 +0200 Received: (qmail 9161 invoked by uid 1000); 28 Aug 2018 10:08:31 -0000 Message-ID: <20180828100831.9160.qmail@stuge.se> Date: Tue, 28 Aug 2018 10:08:31 +0000 From: Peter Stuge To: libssh2-devel@cool.haxx.se Subject: Re: curl / libssh2 sftp write performance (with patch) References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-BeenThere: libssh2-devel@cool.haxx.se X-Mailman-Version: 2.1.22 Precedence: list List-Id: libssh2 development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: libssh2 development Content-Type: text/plain; charset="utf-8" Errors-To: libssh2-devel-bounces@cool.haxx.se Sender: "libssh2-devel" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by giant.haxx.se id w7SA8csm009669 Daniel Jeliński wrote: > Limiting the number of outstanding packets sounds reasonable Can you use channel windowing in the transport protocol here? If not, why not? > > Can we make users opt-in to this and if not, do like before? > > How would you suggest to implement that? I don't see any existing > mechanism that could be used for this Add a new API which would be used to enable the new behavior. The API should probably not be too specific to this behavior, but should also not be too generic. Also consider whether a new API really belongs on SFTP level or perhaps rather on channel level. > I could implement this as a new set of functions duplicating the > existing functionality No, don't do that. //Peter _______________________________________________ libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel From libssh2-devel-bounces@cool.haxx.se Tue Aug 28 13:58:15 2018 Return-Path: Received: from www.haxx.se (mail [127.0.0.1]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTP id w7SBvnvc014799; Tue, 28 Aug 2018 13:58:08 +0200 Received: from mail-yb0-x22c.google.com (mail-yb0-x22c.google.com [IPv6:2607:f8b0:4002:c09:0:0:0:22c]) by giant.haxx.se (8.15.2/8.15.2/Debian-4) with ESMTPS id w7SBvlkw014771 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT) for ; Tue, 28 Aug 2018 13:57:48 +0200 Received: by mail-yb0-x22c.google.com with SMTP id c1-v6so467836ybq.5 for ; Tue, 28 Aug 2018 04:57:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :content-transfer-encoding; bh=4iDhtz9fK9dmBDKBEFULzzLwEOy6bQ+XQ/xXMVcoBFE=; b=GB9Ortx9UTe6wI4zPetJzBTiDyd87Vpaz20Pup24mUBWOJ2v9g8M8tG0CfmDGEUg2u gElKZ3KHt4v0ZT6XJs/QP0U1x8L/AIGdXFbNip5ULknrSsTQv8o2ua0gSKcNh5fpLs0p wgYJeFRVdJgl/2J3xHgzJEuMjJCsxX0HgHM9Ja2GhDNEWINTVzELEs0e+UGMW62VyY8N k9iHEaWsvJq36lbGeW60+aMwCGMltC1RrmVpury5uV52LCnVSmDeOKyrHw0jJ/pLrDPE NDHzlDqKAnwMLWv9BCyYGKqrrgRbeaUqS91bydAn3wi6aAp1V7GZm2X4Cg4AS8ZFukwk P+6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:content-transfer-encoding; bh=4iDhtz9fK9dmBDKBEFULzzLwEOy6bQ+XQ/xXMVcoBFE=; b=MQKx3Bsj/2WPYzA/oqGiVpzEQ7pfe27E7Pg3//LcU3Bpp/Aat89e7FTjlgK1z60ecK VbMbX4sLMvPCukHwMQ3bAu1VMFQsbpnQ46QCjW66alOWVu0eOq8I9HeNyo7pUO+AERqv ObHXHCp1lIZwO98sJyUUZJywQ4CH8u3dH2UuudA4l8xljkl67dWrH3E6Kaxm7md0nXTV PFX5Xjd6VNJjpBkYr9t4eeuVjFze6pmLqQdC/OxjgAmp0+R0ltfJ0O57CNrsfY4XETEZ XYiyVnjUKBfx1F6pYGB5WX3YvxuK2ntycsLrxwQFbEYedoDyFKog5Lo6C1rCUXVADLs1 +DRg== X-Gm-Message-State: APzg51Bf7u4PP+HKub07ZMnRFOtv3hUWwD/KlBmbB2O1JoxuzIducMZr cxQDLF6qjgKIQUD5+J6nSb6/eZq7k4Uj60EC7EMdLwuM X-Google-Smtp-Source: ANB0VdbRTQKR91ozxlyZ5ZBxzYdenjXbxuiZ61MWpAvps+1rmBPumqlARr4BGwjiF4IjbzD/K0HB4aHyJATzGhbFyaY= X-Received: by 2002:a25:2594:: with SMTP id l142-v6mr525432ybl.499.1535457461871; Tue, 28 Aug 2018 04:57:41 -0700 (PDT) MIME-Version: 1.0 References: <20180828100831.9160.qmail@stuge.se> In-Reply-To: <20180828100831.9160.qmail@stuge.se> From: =?UTF-8?Q?Daniel_Jeli=C5=84ski?= Date: Tue, 28 Aug 2018 13:57:28 +0200 Message-ID: Subject: Re: curl / libssh2 sftp write performance (with patch) To: libssh2-devel@cool.haxx.se X-MIME-Autoconverted: from quoted-printable to 8bit by giant.haxx.se id w7SBvlkw014771 X-BeenThere: libssh2-devel@cool.haxx.se X-Mailman-Version: 2.1.22 Precedence: list List-Id: libssh2 development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Reply-To: libssh2 development Content-Type: text/plain; charset="utf-8" Errors-To: libssh2-devel-bounces@cool.haxx.se Sender: "libssh2-devel" Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by giant.haxx.se id w7SBvnvc014799 wt., 28 sie 2018 o 12:10 Peter Stuge napisał(a): > > Can you use channel windowing in the transport protocol here? If not, why not? Channel windowing is used, and under normal circumstances should limit the number of outstanding packets without further intervention. However, an evil server could send window increases without sending status responses, which would bloat our number of outstanding buffers, as we only remove buffers from the list when we receive corresponding status response. Is that what you're referring to? > Add a new API which would be used to enable the new behavior. The API > should probably not be too specific to this behavior, but should also > not be too generic. Something like: typedef enum { SFTPOPT_USESENDBUFFER = 1 } SftpOption; libssh2_sftp_setopt(LIBSSH2_SFTP_HANDLE *handle, SftpOption option, ...); used like: libssh2_sftp_setopt(handle, SFTPOPT_USESENDBUFFER , 1L) to enable the new behavior? > Also consider whether a new API really belongs on SFTP level or perhaps > rather on channel level. Wouldn't we want to enable this behavior on every file handle independently? Daniel _______________________________________________ libssh2-devel https://cool.haxx.se/cgi-bin/mailman/listinfo/libssh2-devel