Skip to content

Commit ad23a01

Browse files
committed
adding rclone support
1 parent c5b1ba8 commit ad23a01

13 files changed

Lines changed: 1216 additions & 121 deletions

File tree

Documentation/Classes/FileTransfer_rclone.md

Lines changed: 506 additions & 0 deletions
Large diffs are not rendered by default.

Documentation/Readme_rclone.MD

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# rclone class
2+
3+
## Introduction
4+
[rclone](https://rclone.org) is a kind of swiss army knife for file transfer to cloud services.
5+
It supports almost all services, such as FTP, SFTP, FTPS, WebDAV (which allows to support many systems such as OwnCloud, NextCloud, etc), Amazon S3, Alibaba Cloud, Box, Dropbox, DigitalOcean Spaces, Cloudfare R2, Google Drive, HiDrive, HTTP, Rackspace, Seafile, Zoho WorkDrive, just to name a few. See web site for full list.
6+
7+
The class follows the concept of the curl/dropbox/gdrive class to provide a similar interface to access these services, allowing to use a common code base.
8+
9+
rclone is usually used with a config file. Do a one time configuration using Terminal/Console to setup the remote system, dealing with OAuth and similar. Then use the same commands, whatever cloud you need to talk to.
10+
11+
Some cloud systems allow direct providing of credentials (such as FTP), some optional (such as Gdrive with Service Account support), some require OAUTH2 and need pre-setup.
12+
See for each cloud service the "Config" button beside the list on [rclone](https://rclone.org).
13+
I recommend to start with the configuration process as recommended on rclone web site, at least for first testing, even if a service allows direct usage.
14+
15+
Some cloud systems allow, some request additional parameters. Use .setPrefix() to add those.
16+
17+
# Usage example for a service configured as "mydrive"
18+
19+
```4D
20+
var $ftp : cs.FileTransfer_rclone
21+
$ftp:=cs.FileTransfer_rclone.new("mydrive")
22+
$source:="/product/4D.dmg"
23+
$target:=Convert path system to POSIX(System folder(Desktop))
24+
$result:=$ftp.download($source; $target)
25+
If ($result.success)
26+
...
27+
End if
28+
```
29+
30+
# Usage example without configuration, passing all data
31+
32+
```4D
33+
var $ftp : cs.FileTransfer_rclone
34+
$ftp:=cs.FileTransfer_rclone.new(":ftp") // use :ftp or :http or :ftps or :S3, etc
35+
$source:="/product/4D.dmg"
36+
$target:=Convert path system to POSIX(System folder(Desktop))
37+
$pass:=$ftp.obscure($credentials.pass)
38+
$ftp.setPrefix("--ftp-host ftp.test.com --ftp-port 3421 --ftp-user 'myself' --ftp-pass "+$pass)
39+
$result:=$ftp.download($source; $target)
40+
If ($result.success)
41+
...
42+
End if
43+
```
44+
45+
Note: you need to obscure the password. You might want to store that instead of the clear password.
46+
47+
For more examples see the method "test_rclone".
48+
Check the download part of the method to see how to use progress bar with Cancel Button.
49+
50+
# Feature overview
51+
- Upload
52+
- Download
53+
- Directory Listing
54+
- Rename and Delete (file and directory)
55+
- Sync
56+
57+
# Installation
58+
59+
rclone is a command line tool supporting all standard file transfer operations:
60+
[rclone](https://rclone.org)
61+
62+
The tool is written in the language 'go', which allows a compilation of an executable without any dependencies. This means you an easily deploy it as part of your application (in Resource folder or similar), it uses the MIT License,
63+
64+
To update, just download their latest release and replace the binary.
65+
66+
You can compile it yourself (see instruction in link above) or download a precompiled version.
67+
68+
For Windows the downloaded version could be placed in Resources folder and so easily deployed.
69+
70+
For Mac, the availalbe version is not signed, making deployment more difficult.
71+
For testing or internal usage, just download and double click once with control-key pressed (to open Apple's Gatekeeper warning and confirm execution.) Future execution will work without confirm, so also by command language.
72+
For deployment, you can either use my (downloaded from their web site and signed, based) or compile and sign yourself, to allow shipping it as notarized merged application inside the Resource folder.
73+
74+
Copy class methods:
75+
FileTansfer_rclone.4dm
76+
SytemWorkerProperties.4dm
77+
Project Method ErrorHandler.4dm
78+
79+
If you already have a silent ErrorHandler method (no output, just to prevent an error message displayed on screen, error handled by code), just use your existing one.
80+
81+
Method ProgressCallback.4dm can be used as example how to create a progress bar - if needed.
82+
83+
Copy this 2-4 files into your project, done.
84+
85+
# Configuration
86+
87+
As explained in introduction, rclone authentication is based on a configuration file.
88+
To create it, use a terminal window and (drag and drop your rconfig file to avoid entering the full path) enter:
89+
rconfig config
90+
This will ask interactive a list of questins, based on the service you want to use.
91+
Read rconfig documentationf for details:
92+
[config](https://rclone.org/docs/)
93+
94+
For some services you can pass credentials directly, see example in introduction or method rclone_test.
95+
96+
# Class Documentation
97+
98+
See Class documentation (in 4D use Explorer/Documentation)
99+
[Documentation online](Classes/FileTransfer_rclone.md)
100+
101+

Project/Sources/Classes/FileTransfer_Dropbox.4dm

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ Function setTimeout($timeout : Integer)
111111
Function setAsyncMode($async : Boolean)
112112
This:C1470._async:=$async
113113

114-
Function enableStopButton($enable : Object)
114+
Function enableStopButton($enable : Object) // needs to be a shared object
115115
This:C1470._enableStopButton:=$enable
116116

117117
Function stop()

Project/Sources/Classes/FileTransfer_curl.4dm

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -76,7 +76,7 @@ Function setPath($path : Text)
7676
Function enableProgressData($enable : Boolean)
7777
This:C1470._noProgress:=Not:C34($enable)
7878

79-
Function enableStopButton($enable : Boolean)
79+
Function enableStopButton($enable : Object) // this is a shared object!
8080
This:C1470._enableStopButton:=$enable
8181

8282
Function useCallback($callback : 4D:C1709.Function; $ID : Text)

Project/Sources/Classes/FileTransfer_rclone.4dm

Lines changed: 85 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Function getDirectoryListing($targetpath : Text)->$success : Object
1818
$targetpath:="/"
1919
End if
2020
// add +This.config+":/"
21-
$url:="lsjson "+This:C1470.config+":"+$targetpath
21+
$url:="lsjson "+This:C1470._wrapRemote($targetpath)
2222
$success:=This:C1470._runWorker($url)
2323
If ($success.success)
2424
If ($success.data="[@")
@@ -35,48 +35,67 @@ Function upload($sourcepath : Text; $targetpath : Text)->$success : Object
3535
// targetpath is full remote path (starting with /, ending with file name
3636
ASSERT:C1129($sourcepath#""; "source path must not be empty")
3737
ASSERT:C1129($targetpath#""; "target path must not be empty")
38-
$url:="copyto "+$sourcepath+" "+This:C1470.config+":"+$targetpath
38+
$url:="copyto "+This:C1470._wrapLocal($sourcepath)+" "+This:C1470._wrapRemote($targetpath)
3939
$success:=This:C1470._runWorker($url)
40-
If ($success.data#"")
40+
If (($success.data#"") & ($success.data#"Transferred@"))
4141
$success.success:=False:C215
4242
$success.error:=$success.data
4343
End if
4444

4545
Function download($sourcepath : Text; $targetpath : Text)->$success : Object
46-
$success:=This:C1470.upload($sourcepath; $targetpath)
4746
ASSERT:C1129($sourcepath#""; "source path must not be empty")
4847
ASSERT:C1129($targetpath#""; "target path must not be empty")
49-
$url:="copyto "+This:C1470.config+":"+$sourcepath+" "+$targetpath
48+
$url:="copyto "+This:C1470._wrapRemote($targetpath)+" "+This:C1470._wrapLocal($targetpath)
5049
$success:=This:C1470._runWorker($url)
51-
If ($success.data#"")
50+
If (($success.data#"") & ($success.data#"Transferred@"))
51+
$success.success:=False:C215
52+
$success.error:=$success.data
53+
End if
54+
55+
Function syncUp($sourcepath : Text; $targetpath : Text)->$success : Object
56+
ASSERT:C1129($sourcepath#""; "source path must not be empty")
57+
ASSERT:C1129($targetpath#""; "target path must not be empty")
58+
$url:="sync "+This:C1470._wrapLocal($sourcepath)+" "+This:C1470._wrapRemote($targetpath)
59+
$success:=This:C1470._runWorker($url)
60+
If (($success.data#"") & ($success.data#"Transferred@"))
61+
$success.success:=False:C215
62+
$success.error:=$success.data
63+
End if
64+
65+
Function syncDown($sourcepath : Text; $targetpath : Text)->$success : Object
66+
ASSERT:C1129($sourcepath#""; "source path must not be empty")
67+
ASSERT:C1129($targetpath#""; "target path must not be empty")
68+
$url:="sync "+This:C1470._wrapRemote($sourcepath)+" "+This:C1470._wrapLocal($targetpath)
69+
$success:=This:C1470._runWorker($url)
70+
If (($success.data#"") & ($success.data#"Transferred@"))
5271
$success.success:=False:C215
5372
$success.error:=$success.data
5473
End if
5574

5675
Function createDirectory($targetpath : Text)->$success : Object
5776
ASSERT:C1129($targetpath#""; "target path must not be empty")
58-
$url:="mkdir "+$targetpath
77+
$url:="mkdir "+This:C1470._wrapRemote($targetpath)
5978
$success:=This:C1470._runWorker($url)
6079

6180
Function deleteDirectory($targetpath : Text; $force : Boolean)->$success : Object
6281
ASSERT:C1129($targetpath#""; "target path must not be empty")
6382
If ($force)
64-
$url:="rm -f "+$targetpath
83+
$url:="purge -f "+This:C1470._wrapRemote($targetpath)
6584
Else
66-
$url:="rm "+$targetpath
85+
$url:="rmdir "+This:C1470._wrapRemote($targetpath)
6786
End if
6887
$success:=This:C1470._runWorker($url)
6988

7089
Function deleteFile($targetpath : Text)->$success : Object
7190
// same as deleteDirectory
7291
ASSERT:C1129($targetpath#""; "target path must not be empty")
73-
$url:="rm "+$targetpath
92+
$url:="delete "+This:C1470._wrapRemote($targetpath)
7493
$success:=This:C1470._runWorker($url)
7594

7695
Function renameFile($sourcepath : Text; $targetpath : Text)->$success : Object
7796
ASSERT:C1129($sourcepath#""; "source path must not be empty")
7897
ASSERT:C1129($targetpath#""; "target path must not be empty")
79-
$url:="mv "+$sourcepath+" "+$targetpath
98+
$url:="moveto "+This:C1470._wrapRemote($sourcepath)+" "+This:C1470._wrapRemote($targetpath)
8099
$success:=This:C1470._runWorker($url)
81100

82101
Function moveFile($sourcepath : Text; $targetpath : Text)->$success : Object
@@ -85,13 +104,24 @@ Function moveFile($sourcepath : Text; $targetpath : Text)->$success : Object
85104
Function copyFile($sourcepath : Text; $targetpath : Text)->$success : Object
86105
ASSERT:C1129($sourcepath#""; "source path must not be empty")
87106
ASSERT:C1129($targetpath#""; "target path must not be empty")
88-
$url:="cp "+$sourcepath+" "+$targetpath
107+
$url:="copyto "+This:C1470._wrapRemote($sourcepath)+" "+This:C1470._wrapRemote($targetpath)
89108
$success:=This:C1470._runWorker($url)
109+
If (($success.data#"") & ($success.data#"Transferred@"))
110+
$success.success:=False:C215
111+
$success.error:=$success.data
112+
End if
90113

91114
Function executeCommand($command : Text)->$success : Object
92115
ASSERT:C1129($command#""; "command must not be empty")
93116
$success:=This:C1470._runWorker($command)
94117

118+
Function obscure($password : Text)->$obscured : Text
119+
$command:="obscure '"+$password+"'"
120+
$success:=This:C1470._runWorker($command)
121+
If ($success.success)
122+
$obscured:=This:C1470._trim($success.data)
123+
End if
124+
95125
//MARK: Settings
96126
Function validate()->$success : Object
97127
$success:=This:C1470._runWorker("about "+This:C1470.config+":/")
@@ -102,6 +132,14 @@ Function version()->$data : Object
102132
Function setPath($path : Text)
103133
This:C1470._Path:=$path
104134

135+
Function setMaxTime($seconds : Real)
136+
// sets --max-duration duration
137+
This:C1470._maxTime:=$seconds
138+
139+
Function setPrefix($prefix : Text)
140+
// allows to set any parameters directly after curl
141+
This:C1470._prefix:=$prefix
142+
105143
Function useCallback($callback : 4D:C1709.Function; $ID : Text)
106144
ASSERT:C1129(Value type:C1509($callback)=Is object:K8:27; "Callback must be of type function")
107145
ASSERT:C1129(OB Instance of:C1731($callback; 4D:C1709.Function); "Callback must be of type function")
@@ -135,10 +173,18 @@ Function status()->$status : Object
135173
Function wait($max : Integer)
136174
This:C1470._worker.wait($max)
137175

176+
177+
138178
// MARK: Internal helper calls
139179
Function _runWorker($para : Text)->$result : Object
180+
$postfix:=""
140181
If (This:C1470._Callback#Null:C1517)
141182
$workerpara:=cs:C1710.SystemWorkerProperties.new("rclone"; This:C1470.onData; This:C1470._Callback; This:C1470._CallbackID; This:C1470._enableStopButton)
183+
If ((This:C1470._noProgress#Null:C1517) && (This:C1470._noProgress))
184+
// nothing, opposite!
185+
Else
186+
$postfix+="--progress --stats-one-line"
187+
End if
142188
Else
143189
$workerpara:=cs:C1710.SystemWorkerProperties.new("rclone"; This:C1470.onData)
144190
End if
@@ -149,30 +195,20 @@ Function _runWorker($para : Text)->$result : Object
149195
$path:="rclone"
150196
End if
151197

152-
/*
153-
154-
If ((This._noProgress#Null) && (This._noProgress))
155-
$path+=" --no-progress-meter"
156-
End if
157-
If (This._connectTimeout#Null)
158-
$path+=" --connect-timeout "+String(This._connectTimeout)
159-
End if
160-
If (This._maxTime#Null)
161-
$path+=" --max-time "+String(This._maxTime)
162-
End if
163-
If (This._range#Null)
164-
$path+=" --range "+This._range
165-
End if
166-
If ((This._ActiveMode#Null) && (This._ActiveMode)) // default passive
167-
$path+=" --ftp-port "+This.ActiveModeIP
168-
End if
169-
If (This._prefix#Null)
170-
$path+=(" "+This._prefix)
171-
End if
172-
173-
*/
198+
199+
If (This:C1470._maxTime#Null:C1517)
200+
$path+=(" --max-duration "+String:C10(This:C1470._maxTime))
201+
End if
202+
203+
If (This:C1470._prefix#Null:C1517)
204+
$path+=(" "+This:C1470._prefix)
205+
End if
206+
174207

175208
$command:=$path+" "+$para
209+
If ($postfix#"")
210+
$command:=$command+" "+$postfix
211+
End if
176212
$old:=Method called on error:C704
177213
ON ERR CALL:C155(Formula:C1597(ErrorHandler).source)
178214
This:C1470._worker:=4D:C1709.SystemWorker.new($command; $workerpara)
@@ -202,19 +238,31 @@ End if
202238
End if
203239

204240
If (This:C1470._Callback#Null:C1517)
205-
This:C1470._worker.onTerminate(New object:C1471; New object:C1471)
241+
//This._worker.onTerminate(New object; New object) // does not exists?
242+
$workerpara.onTerminate(New object:C1471; New object:C1471)
206243
End if
207244
Else
208245
$result:=New object:C1471("success"; False:C215; "responseError"; "rclone execution error")
209246
End if
247+
If ($result.data=Null:C1517)
248+
$result.data:=""
249+
End if
210250
ON ERR CALL:C155($old)
211251

212252
Function _trim($text : Text)->$result : Text
213253
$result:=$text
214-
While (Substring:C12($result; 1; 1)=" ")
254+
While (Character code:C91(Substring:C12($result; 1; 1))<=32)
215255
$result:=Substring:C12($result; 2)
216256
End while
217-
While (Substring:C12($result; Length:C16($result); 1)=" ")
257+
While (Character code:C91(Substring:C12($result; Length:C16($result); 1))<=32)
218258
$result:=Substring:C12($result; 1; Length:C16($result)-1)
219259
End while
260+
261+
Function _wrapLocal($text : Text)->$result : Text
262+
$result:=Char:C90(Double quote:K15:41)+$text+Char:C90(Double quote:K15:41)
263+
264+
Function _wrapRemote($text : Text)->$result : Text
265+
$result:=Char:C90(Double quote:K15:41)+This:C1470.config+":"+$text+Char:C90(Double quote:K15:41)
266+
267+
220268

0 commit comments

Comments
 (0)