Merge branch 'master' into improve-config

Author: FabianKramm

docs/docs/advanced/architecture.md

@@ -22,17 +22,6 @@ The DevSpace CLI will deploy the Helm chart specified under [chart/](/docs/confi
 
 **Note:** If you do not have a [Helm chart](/docs/configuration/chart.html) yet, the DevSpace CLI will automatically create one for you.
 
-## 2-Way Code Sync
-The DevSpace CLI will establish a bi-directional code synchronization between your local source code directory and the main dev pod inside your DevSpace. The synchonization has the following characteristics:
-
-- No server-side component is required to use the sync mechanism.
-- The sync algorithm keeps track of all (remote and local) files within an in-built, local database.
-- The code sync works with any container filesystem (i.e. ephemeral storage and mounted volumes).
-- The code sync is lightning fast and very reliable. It detects remote changes as well as local changes quickly and synchronizes them respectively.
-- The sync algorithm only requires the following stardard POSIX commands to be available inside your development container: sleep, mkdir, rm, mv, cat, ls, printf, echo, kill, tar, sh
-
-**Note:** You can configure the sync path mappings via [.devspace/config.yaml](/docs/configuration/config.yaml.html).
-
 ## Terminal Proxy
 The terminal proxy opens a terminal within your dev container and connects your local command-line to it. This way, you can directly run commands inside your DevSpace and stream the response directly to your local computer.
 

docs/docs/advanced/sync.md

@@ -0,0 +1,38 @@
+---
+title: Source Code Synchronization
+---
+
+# Code Synchronization
+The DevSpace CLI synchronization mechanism is written from scratch and made for kubernetes development with hot reloading. While current other kubernetes development solutions (like draft, skaffold and telepresence) help the developer with the problem how to test and deploy cloud-native applications in kubernetes clusters, actual iterative development, testing and hot reloading of cloud-native applications within kubernetes is still an annoying problem. Especially for developers of programming languages that support hot reload development environments, such as nodejs, this is a hassle. The DevSpace CLI has a syncing mechanism that is able to sync local file changes to remote containers directly without the need of restarting the container, a deployment pipeline or a docker build. This greatly accelerates development, debugging and testing directly in remote containers.
+
+Our main requirements to the sync mechanism were:
+- Easy to integrate, no cluster dependencies, client-only implementation
+- Should work in every kubernetes cluster
+- Should work with most containers, without installing additional binaries or changing the Dockerfile
+- Should work with every container filesystem (i.e. ephemeral storage and mounted volumes)
+- File watchers and hot reload tools like nodemon should recognize sync changes like on local filesystem
+- Fast and reliable
+
+ If synchronization is configured (check with `devspace list sync`), the DevSpace CLI will establish a bi-directional code synchronization between the specified local folders and the remote container folders. It automatically recognizes any changes within the specified folders during the session and will update the corresponding files locally and remotely in the background. You can check the latest sync activity by running the command `devspace status sync` or take a look at the `sync.log` in `.devspace/logs`.
+
+## Sync Requirements
+No server-side component for code synchronization is required, the sync is client-only. The synchronization mechanism works with any container filesystem and no special binaries have to be installed into the containers. File watchers running within the containers like nodemon will also recognize changes made by the synchronization mechanism.
+
+Some basic POSIX binaries have to be present in the container (which usually exist in most containers): `sh, tar, cd, sleep, find, stat, mkdir, rm, cat, printf, echo, kill`
+
+## Excluding Files and Folders
+You are able to fully or partly exclude certain files and folders from synchronization. Take a look at [.devspace/config.yaml](/docs/configuration/config.yaml.html) for more information where to specify the ignore rules in the configuration. The exclude path syntax is the [.gitignore](https://git-scm.com/docs/gitignore) syntax. 
+
+There are 3 different options for each sync path to exclude files and folders:
+1. excludePaths: Matched paths are completely ignored during synchronization
+2. downloadExcludePaths: Local changes are uploaded, but remote changes are not downloaded 
+3. uploadExcludePaths: Local changes are not uploaded, but remote changes are downloaded
+
+## Initial Sync
+If synchronization is started, the sync initially compares the remote folder and the local folder and merges the contents with the following rules:
+- If a file or folder exists remote, but not locally, then download file / folder
+- If a file or folder exists locally, but not remote, then upload file / folder
+- If a file is newer locally than remote then upload the file (The opposite case is not true, older local files are not overriden by newer remote files)
+
+## Performance Notes
+The sync mechanism is normally very reliable and fast. Syncing several thousand files is usually not a problem. Changes are packed together and compressed before synchronization, which improves performance especially for transferring text files. Transferring large compressed binary files is possible, however can affect performance negatively. Rename operations are currently recognized as a separate remove and create operation, which in normal workflows has at most a minor performance impact, however renaming huge folders with tens of thousands of files can impact performance negatively and should be avoided. Remote changes can sometimes have a delay of 1-2 seconds till they are downloaded, depending on how big the synchronized folder is. It should be generally avoided to sync the complete container filesystem.

docs/docs/getting-started/installation.md

@@ -19,15 +19,18 @@ del "%Temp%\install-devspace.bat"
 **Note:** After running the install script, you should reopen the terminal window to refresh the environment variables.
 
 ### For Linux
-1. Run this install script with **root privileges**:
+1. Run the command:
 ```bash
-tmpdir=$(dirname $(mktemp -u))
-curl -s "https://raw.githubusercontent.com/covexo/devspace/master/scripts/installer-linux.sh" >"$tmpdir/install-devspace.sh"
-"$tmpdir/install-devspace.sh" "/usr/bin/devspace"
-rm -r "$tmpdir"
+curl --silent "https://api.github.com/repos/covexo/devspace/releases/latest" | sed -nr 's!.*"(https://github.com[^"]*devspace-linux-amd64)".*!\1!p' | xargs -n 1 curl -L -o devspace && chmod +x devspace && sudo mv devspace /usr/local/bin
+```
+
+### For Mac
+1. Run the command:
+```bash
+curl --silent "https://api.github.com/repos/covexo/devspace/releases/latest" | sed -nr 's!.*"(https://github.com[^"]*devspace-darwin-amd64)".*!\1!p' | xargs -n 1 curl -L -o devspace && chmod +x devspace && sudo mv devspace /usr/local/bin
 ```
 
 ## Binary Download
 An alternative to the install scripts is to:
 1. download the latest release from the [GitHub releases page](https://github.com/covexo/devspace/releases)
-2. add the binary folder path to the PATH environment variable **OR** run `PATH_TO_YOUR_RELEASE_FILE install` with admin/root priveleges
+2. add the binary folder path to your PATH environment variable **OR** run `PATH_TO_YOUR_RELEASE_FILE install` with admin/root priveleges

pkg/devspace/builder/docker/auth.go

@@ -42,16 +42,16 @@ func getDefaultAuthConfig(client client.CommonAPIClient, checkCredStore bool, se
 	var authconfig types.AuthConfig
 	var err error
 
-	configfile, _ := loadDockerConfig()
-
 	if !isDefaultRegistry {
 		serverAddress = registry.ConvertToHostname(serverAddress)
 	}
 
 	if checkCredStore {
-		authconfig, err = configfile.GetAuthConfig(serverAddress)
-	} else {
-		authconfig = types.AuthConfig{}
+		configfile, err := loadDockerConfig()
+
+		if configfile != nil && err == nil {
+			authconfig, err = configfile.GetAuthConfig(serverAddress)
+		}
 	}
 
 	authconfig.ServerAddress = serverAddress

pkg/devspace/sync/downstream.go

@@ -216,7 +216,7 @@ func (d *downstream) downloadFiles(files []*fileInformation) (string, error) {
 
 	// Each file is represented in one line
 	for _, element := range files {
-		if lenFiles <= 3 {
+		if lenFiles <= 3 || d.config.verbose {
 			d.config.Logf("[Downstream] Download file %s, size: %d", element.Name, element.Size)
 		}
 
@@ -335,7 +335,7 @@ func (d *downstream) removeFilesAndFolders(removeFiles map[string]*fileInformati
 		absFilepath := filepath.Join(d.config.WatchPath, key)
 
 		if shouldRemoveLocal(absFilepath, value, d.config) {
-			if numRemoveFiles <= 3 {
+			if numRemoveFiles <= 3 || d.config.verbose {
 				d.config.Logf("[Downstream] Remove %s", key)
 			}
 
@@ -369,7 +369,7 @@ func (d *downstream) createFolders(createFolders []*fileInformation) {
 
 	for _, element := range createFolders {
 		if element.IsDirectory {
-			if numCreateFolders <= 3 {
+			if numCreateFolders <= 3 || d.config.verbose {
 				d.config.Logln("[Downstream] Create folder: " + element.Name)
 			}
 

pkg/devspace/sync/evaluater.go

@@ -49,7 +49,7 @@ func shouldUpload(relativePath string, stat os.FileInfo, s *SyncConfig, isInitia
 
 	// Exclude changes on the upload exclude list
 	// if s.uploadIgnoreMatcher != nil {
-	// 	if s.uploadIgnoreMatcher.MatchesPath(relativePath) {
+	//	if s.uploadIgnoreMatcher.MatchesPath(relativePath) {
 	//		return false
 	//	}
 	// }

pkg/devspace/sync/file_information.go

@@ -56,7 +56,7 @@ func (p parsingError) Error() string {
 }
 
 func getFindCommand(destPath string) string {
-	return "mkdir -p '" + destPath + "' && find '" + destPath + "' -exec stat -c \"%n///%s,%Y,%f,%a,%u,%g\" {} + 2>/dev/null && echo -n \"" + EndAck + "\" || echo \"" + ErrorAck + "\"\n"
+	return "mkdir -p '" + destPath + "' && find -L '" + destPath + "' -exec stat -c \"%n///%s,%Y,%f,%a,%u,%g\" {} + 2>/dev/null && echo -n \"" + EndAck + "\" || echo \"" + ErrorAck + "\"\n"
 }
 
 func parseFileInformation(fileline, destPath string) (*fileInformation, error) {

pkg/devspace/sync/sync_config.go

@@ -50,13 +50,16 @@ type SyncConfig struct {
 	upstream   *upstream
 	downstream *downstream
 
-	silent   bool
+	silent  bool
+	verbose bool
+
 	stopOnce sync.Once
 	stopped  bool
 
 	// Used for testing
 	testing   bool
 	errorChan chan error
+	readyChan chan bool
 }
 
 // Logf prints the given information to the synclog with context data
@@ -76,7 +79,11 @@ func (s *SyncConfig) Logln(line interface{}) {
 		if s.Pod != nil {
 			syncLog.WithKey("pod", s.Pod.Name).WithKey("local", s.WatchPath).WithKey("container", s.DestPath).Info(line)
 		} else {
-			syncLog.WithKey("local", s.WatchPath).WithKey("container", s.DestPath).Info(line)
+			syncLog.
+				WithKey("local",
+					s.WatchPath).
+				WithKey("container", s.DestPath).
+				Info(line)
 		}
 	}
 }
@@ -224,6 +231,10 @@ func (s *SyncConfig) startUpstream() {
 
 	defer notify.Stop(s.upstream.events)
 
+	if s.readyChan != nil {
+		s.readyChan <- true
+	}
+
 	err = s.upstream.mainLoop()
 	if err != nil {
 		s.Error(err)
@@ -293,10 +304,10 @@ func (s *SyncConfig) diffServerClient(filepath string, sendChanges *[]*fileInfor
 
 	delete(downloadChanges, relativePath)
 
-	s.fileIndex.fileMapMutex.Lock()
 	// Exclude changes on the upload exclude list
 	if s.uploadIgnoreMatcher != nil {
 		if s.uploadIgnoreMatcher.MatchesPath(relativePath) {
+			s.fileIndex.fileMapMutex.Lock()
 			// Add to file map and prevent download if local file is newer than the remote one
 			if s.fileIndex.fileMap[relativePath] != nil && s.fileIndex.fileMap[relativePath].Mtime < ceilMtime(stat.ModTime()) {
 				// Add it to the fileMap
@@ -307,11 +318,13 @@ func (s *SyncConfig) diffServerClient(filepath string, sendChanges *[]*fileInfor
 					IsDirectory: stat.IsDir(),
 				}
 			}
+			s.fileIndex.fileMapMutex.Unlock()
 
 			return nil
 		}
 	}
 
+	s.fileIndex.fileMapMutex.Lock()
 	shouldUpload := shouldUpload(relativePath, stat, s, true)
 	s.fileIndex.fileMapMutex.Unlock()
 
@@ -343,7 +356,7 @@ func (s *SyncConfig) diffDir(filepath string, stat os.FileInfo, sendChanges *[]*
 		return nil
 	}
 
-	if len(files) == 0 {
+	if len(files) == 0 && relativePath != "" {
 		*sendChanges = append(*sendChanges, &fileInformation{
 			Name:        relativePath,
 			Mtime:       ceilMtime(stat.ModTime()),