如何在 R 中上传多个文件而不会收到 421 Too many connections 错误？

Question

我需要使用RCurl 将超过 65 个文件上传到我的托管 ftp 服务器，但在上传多个文件后我收到错误消息：

 < 421 Too many connections (8) from this IP
* We got a 421 - timeout!
* Closing connection 291

从我在日志中看到的内容来看，它显示为Connection #283 to host ftp.myserver.com left intact，因此在 15 分钟不活动< 220 You will be disconnected after 15 minutes of inactivity. 后连接到期之前，无需进一步授权。

至少这是我的理解，因为出于某种原因，它会在第二个文件之后立即重新发送 USER 和 PASSWORD：

> USER admin@myserver.com
< 331 User admin@myserver.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /

我知道有多个授权请求（每个文件一个），所以我会淹没服务器。

是否可以先打开与 FTP 服务器的连接，然后上传所有文件？就像在 FileZilla 中一样？

请找到我使用的 R 代码：

for(file in list.files(localPath)) {

  ftpUpload(paste0(localPath,file), userpwd = userpwd, to=paste0("ftp://", ftpPath, file), verbose=TRUE)

}

另一个 SO 成员在这里遇到了同样的问题，但看起来我已经使用了与提供给他的回复类似的东西，但它不起作用。 Using R to upload many files

谢谢

---

编辑（添加日志）：

> for(file in list.files(localPath)) {
+   
+   print(paste0("ftp://", ftpPath, file))
+   ftpUpload(paste0(localPath,file), userpwd = userpwd, to=paste0("ftp://", ftpPath, file), verbose=TRUE)
+ 
+ }
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-1.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#310)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 2 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||44516|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 44516
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#310)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-1.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.180 seconds (measured here), 1.15 Kbytes per second
* Connection #310 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-2.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#311)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 3 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||45077|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 45077
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#311)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-2.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.207 seconds (measured here), 1.00 Kbytes per second
* Connection #311 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-3.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#312)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 4 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||40390|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 40390
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#312)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-3.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.189 seconds (measured here), 1.12 Kbytes per second
* Connection #312 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation-4.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#313)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 5 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||46329|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 46329
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#313)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation-4.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.187 seconds (measured here), 1.11 Kbytes per second
* Connection #313 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-1.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#314)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 6 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||46556|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 46556
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#314)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-1.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.186 seconds (measured here), 2.75 Kbytes per second
* Connection #314 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-2.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#315)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 7 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||43730|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 43730
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#315)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-2.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.190 seconds (measured here), 2.69 Kbytes per second
* Connection #315 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-3.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#316)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 8 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||43399|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 43399
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#316)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-3.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
< 226-File successfully transferred
< 226 0.196 seconds (measured here), 2.61 Kbytes per second
* Connection #316 to host ftp.mywebsite.com left intact
[1] "ftp://ftp.mywebsite.com/data/csv/data_private_allocation_protected-4.csv"
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#317)
< 220---------- Welcome to Pure-FTPd [privsep] [TLS] ----------
< 220-You are user number 9 of 1000 allowed.
< 220-Local time is now 08:50. Server port: 21.
< 220-This is a private system - No anonymous login
< 220-IPv6 connections are also welcome on this server.
< 220 You will be disconnected after 15 minutes of inactivity.
> USER cadmin@mywebsite.com
< 331 User cadmin@mywebsite.com OK. Password required
> PASS mypassword
< 230 OK. Current restricted directory is /
> PWD
< 257 "/" is your current location
* Entry path is '/'
> CWD portfolios
* ftp_perform ends with SECONDARY: 0
< 250 OK. Current directory is /portfolios
> CWD csv
< 250 OK. Current directory is /data/csv
> EPSV
* Connect data stream passively
< 229 Extended Passive mode OK (|||48536|)
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connecting to 11.22.33.44 (11.22.33.44) port 48536
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#317)
> TYPE I
< 200 TYPE is now 8-bit binary
> STOR data_private_allocation_protected-4.csv
< 150 Accepted data connection
* Remembering we are in dir "data/csv/"
[1] "ftp://ftp.mywebsite.com/data/csv/data_coins_average-1.csv"
< 226-File successfully transferred
< 226 0.184 seconds (measured here), 2.77 Kbytes per second
* Connection #317 to host ftp.mywebsite.com left intact
* Hostname was NOT found in DNS cache
*   Trying 11.22.33.44...
* Connected to ftp.mywebsite.com (11.22.33.44) port 21 (#318)
< 421 Too many connections (8) from this IP
* We got a 421 - timeout!
* Closing connection 318
Error in function (type, msg, asError = TRUE)  : 
  Uploading to a URL without a file name!

Answer 1

这是一种解决方法，而不是我的问题的解决方案，但它就像一个魅力。我在里面创建了一个带有 ftp 命令的文本文件，然后使用 R system() 启动 Ubuntu ftp。

ftpcommands.txt：

open ftp.mywebsite.com
user admin@mywebsite.com mypassword
lcd ~/R/data/csv
cd data/csv
prompt
mput *.csv
bye

然后是R 命令：

system("ftp -n < ~/R/data/ftpcommands.txt")

Answer 2

看起来您正在为每个文件打开一个新连接，这不是 文件传输协议 的本意。一些不同的解决方案：

1. 在您的 R 代码中打开较少的连接。
2. 限制 curl 使用的最大并发连接数。
3. 在 FTP 服务器 的配置中增加允许的并发连接数限制。 （只有当你真的需要时才这样做，正如我所提到的，不要为每个文件打开一个连接。）

检查this detailed answer 以更好地了解问题。不要误会我的意思，同时有一个以上的连接对性能有好处，但不是每个文件，这绝对是太多了。大多数 FTP 客户端使用 2。