SftpClient: slow download

[goremykin]

Hi! Thank you for this project, I appreciate your work.

I have a job in js that downloads some files over sftp every night using ssh2-sftp-client.
I decided to rewrite this job in C# using SSH.NET, but it turned out that the same files are now downloaded up to 15 times slower.
For example, a 25 MB file can be downloaded in 5 seconds using ssh2-sftp-client and in over a minute using SSH.NET.

I tested on ubuntu and arch linux in a simple .net 8 project. SSH.NET version - 2023.0.1.

using var sftpClient = new SftpClient(host, port, userName, password);
await using (var localFileStream = new FileStream("/someFile", FileMode.Create, FileAccess.Write))
{
    sftpClient.DownloadFile(file.FullName, localFileStream);
}

In case of reading from a stream it even slower:

await using var remoteStream = await sftpClient.OpenAsync("/someFile", FileMode.Open, FileAccess.Read, CancellationToken.None);
await remoteStream.CopyToAsync(localFileStream);

Do you have any ideas on what I should check or what settings I should play with?

[scott-xu]

Not sure if it relates to compression. Do you know if your SSH service supports and only supports compression?
Well, I don't think it relates to compression.

[Waynezhi]

Same situation, using 2024.0.0, download speed is around 1MB/s only on a sftp server (protocol v3), and same environment with sftp command I got 15MB/s. @WojciechNagorski

[goremykin]

Can confirm the same issue with 2024.0.0

[Waynezhi]

I tried with 2024.0.0 and 2023.0.1, same result @goremykin, with .net 6
#733
#1122

seems this fix does not work for me: #866

[Rob-Hague]

There has been no recent investigation into SFTP performance that I know of. If you make an investigation, please do share your findings. Thanks

[TheGreenAirplane]

Hello, has there been any progress on this? I'm experiencing slow download with the SCP client, this could be related.

[WojciechNagorski]

@TheGreenAirplane I would be happy to review your PR that will solve this problem.

[megapro17]

I would be happy to review your PR that will solve this problem.

You can't solve a problem without finding it's cause first

[scott-xu]

I would be happy to review your PR that will solve this problem.

You can't solve a problem without finding it's cause first

Since this is an open source project, anyone can investigate the issue, find the root cause and propose a fix, including you 😄

[megapro17]

Library reads file more than ten times slower than sftp cli. Tried in release mode, no difference
From profiling it spends most time on WaitOnHandle from RequestRead function

SSH.NET/src/Renci.SshNet/SubsystemSession.cs

Line 241 in 6669309

var result = WaitHandle.WaitAny(waitHandles, millisecondsTimeout);

I've modified a code to use RequestReadAsync and ReadAllBytesAsync but it didn't changed anything

var client = new SftpClient("localhost", 8022, "megapro17", "1");
client.Connect();
//client.BufferSize = 9999999; // No difference
Stopwatch sw = Stopwatch.StartNew();
var file = await client.ReadAllBytesAsync(@"/sdcard/DCIM/Big file.mp4");
sw.Stop();
Console.WriteLine($"{sw.Elapsed.ToString()} size:{file.Length} speed:{file.Length/1024/1024/sw.Elapsed.TotalSeconds} MiB/s");

//00:00:09.8068972 size:114186331 speed:11,012657499866522 MiB/s

[supplemarty]

Seeing terrible performance using 2024.1.0 reading 2.5GB file as stream; reading in 32k chunks using all defaults. On average about .5 MBPS vs 5.5 MBPS using WinSCP.Net for same file/environment streaming. Would love to move to SSH.Net for Azure Functions to be able to run on Linux but cannot with this perf. Is anyone looking into this or any advice on how to get better perf?

[jaredballingham]

I am running into the same issue when dealing with a 25MB file. ReadAllBytes() is taking ~60s. Unsure if it helps you or actually addresses the underlying problem, but I found that you can call DownloadFile() into a MemoryStream. This still doesn't get me the performance that I would like to see (only 3-7x faster) and comes with some overhead, but it was still consistently faster. I did not do exhaustive testing by any means, but this approach was able to meet the urgent need I had.

`
var localStream = new MemoryStream();
sftp.DownloadFile(filePath, localStream);
_fileBytes = localStream.ToArray();

vs

_fileBytes = sftp.ReadAllBytes(filePath);
`
I normally don't post, but thought this work-around could help and maybe assist in diagnosing the root cause of the slowness.

[goremykin]

Upgraded from Ubuntu 24.04 to 24.10 yesterday and now it is 2-3 times faster. Still slow, but much better now. And I have no idea why.

[bugproof]

Tried other lib and it has the same problem https://github.com/tmds/Tmds.Ssh/tree/main

[tmds]

I wrote a small program that downloads a 10MB file from a remote system using different clients. You can find the source here: https://gist.github.com/tmds/119a83fa9772c29934f848b49fe30f9a#file-measure-cs.

It will output something like: https://gist.github.com/tmds/119a83fa9772c29934f848b49fe30f9a#file-example_output-txt.

In my measurement I don't see the factor of magnitude difference that is reported here. Bandwidth vs latency (round-trip-time) can have a large effect on SFTP performance. In your setup you may see a large difference between different clients.

Feel free to open up a ticket in the Tmds.Ssh repo if you can demonstrate a large difference against another client.