Skip to content

fix: make config sync directories configurable via config.yaml#17

Closed
LLQWQ wants to merge 27 commits into
Go1c:mainfrom
LLQWQ:feat/setting-sync-v2
Closed

fix: make config sync directories configurable via config.yaml#17
LLQWQ wants to merge 27 commits into
Go1c:mainfrom
LLQWQ:feat/setting-sync-v2

Conversation

@LLQWQ
Copy link
Copy Markdown
Contributor

@LLQWQ LLQWQ commented Apr 27, 2026

Problem

CLI's FileSync and SettingSync have inconsistent handling of dot-prefixed directories like .agents:

  • FileSync._collect_local_files() skips ALL dot-prefixed directories
  • SettingSync uses _is_config_path() which treats ALL dot-prefixed dirs as config
  • But SyncEngine._is_config() also treats ALL dot-prefixed dirs as config

This causes problems when syncing custom config directories:

  1. If .agents/ exists on Obsidian but not on CLI vault, CLI won't send it to server
  2. Server may then tell other devices to delete .agents/ files
  3. Custom config directories get lost across devices

Impact

  • Custom config directories (.agents, .my-config, etc.) are not properly synchronized
  • Data loss when using CLI alongside Obsidian plugin
  • Inconsistent behavior between FileSync and SettingSync

Root Cause

In file_sync.py:

first = rel.split("/")[0]
if first.startswith("."):
    continue  # Skips .agents/ files

In sync_engine.py:

def _is_config(self, rel_path: str) -> bool:
    first = rel_path.split("/")[0]
    return first.startswith(".")  # Too broad

Proposed Fix

Make config directory handling explicit:

  1. Always treat .obsidian and .agents as config directories
  2. Let other dot-prefixed dirs follow sync_config setting
  3. FileSync should skip dot-prefixed dirs only when sync_config=True
    (so they go through SettingSync)

Files Changed

  • fns_cli/config.py: Add config_sync_dirs to config schema
  • fns_cli/file_sync.py: Skip dot-prefixed dirs based on config_sync_dirs
  • fns_cli/setting_sync.py: Use config_sync_dirs for config path check
  • fns_cli/sync_engine.py: Use explicit config directory list

Testing

  1. Set up Obsidian plugin with .agents in configSyncOtherDirs
  2. Add files to .agents/ directory in Obsidian
  3. Run CLI sync on another device (without .agents/ directory)
  4. Verify .agents/ files are synced to CLI
  5. Run Obsidian sync again — .agents/ files should remain

Environment

  • CLI version: latest (feat/setting-sync branch)
  • OS: Linux/Any

Summary by Sourcery

Make config directories explicitly configurable and adjust sync behavior so dot-prefixed folders are treated as config only when configured, aligning FileSync, SettingSync, FolderSync, and SyncEngine.

New Features:

  • Add configurable config_sync_dirs option in config.yaml to control which dot-prefixed directories are treated as config.

Enhancements:

  • Align FileSync, SettingSync, FolderSync, and SyncEngine logic to consistently skip or handle dot-prefixed config directories based on config_sync_dirs and sync_config settings.
  • Update initial sync and full file push logic to avoid uploading config directories through FileSync, delegating them to SettingSync instead.

Go1c and others added 27 commits March 21, 2026 17:58
Three bugs prevented attachments from syncing:

1. FileSyncUpdate handler silently skipped files without inline content.
   Attachments are binary and never include inline content — the client
   must send a FileChunkDownload request to initiate chunked transfer.

2. Binary message prefix mismatch: server sends "00" (VaultFileMsgType)
   but client checked for "BC", so all download chunks were ignored.

3. file_content_hash_binary used SHA-256 instead of the djb2 byte-hash
   the server/plugin expect, causing perpetual hash mismatches.

Also: send local file hashes in FileSync request so the server can
properly diff, wait for in-flight downloads before declaring sync
complete, and increase file sync timeout to 300s.

Closes #1
fix: attachment sync not downloading from server
Five bugs fixed:

1. FileSync: _on_sync_end set _sync_complete immediately, but the server
   streams FileSyncDelete/Update messages *after* FileSyncEnd. pull/sync-once
   closed the connection before those messages arrived, so remote deletions
   were never applied locally. Now mirrors NoteSync: tracks expected vs
   received counts and only marks complete when all messages are processed.

2. FileSync: _on_sync_update returned early on unexpected content type
   without incrementing _received_modify, causing _wait_file_sync to
   stall until the 300s timeout.

3. Watcher: on_moved did not check is_ignored, so server-triggered renames
   (which temporarily ignore both paths) could fire on_local_rename after
   the ignore window closed, pushing a redundant delete+upload back to the
   server.

4. Daemon (fns run): after a WebSocket reconnect _initial_sync was never
   retried, so any changes made on other clients during the outage
   (including deletions) were permanently missed. Added on_reconnect hook
   to WSClient; SyncEngine registers a callback that pauses the watcher
   and re-runs _initial_sync on every reconnect.

5. sync_once: called _initial_sync (incremental NoteSync) then immediately
   called request_full_sync, sending two NoteSync requests per session.
   sync_once now calls request_full_sync + file_sync.request_sync directly.
Cover the bugs fixed in the previous commit:

- FileSyncEnd before FileSyncDelete: sync must stay incomplete until all
  expected delete messages are received
- FileSyncDelete before FileSyncEnd: completes correctly on End arrival
- Multiple deletes: only completes after the last one
- Delete of nonexistent file: still counted so sync doesn't stall
- Mixed modify + delete: both counters must drain before complete
- request_sync counter reset: second sync not polluted by first
- Unexpected content type: no longer stalls for 300s
- Watcher on_moved: ignored src/dest paths are skipped
_try_remove_empty_parent previously only checked one level up,
leaving ancestor directories (e.g. diagrams/, notes/) as empty
shells after all their contents were deleted via pull sync.

Now walks up the tree until vault_path, removing each empty
directory in turn.
heartbeat_interval was defined in config (default 30s) but never
used — ping_interval/ping_timeout were both None, so silently dead
connections were never detected and the receive loop hung indefinitely.

Now passes heartbeat_interval to websockets ping_interval and
ping_timeout. A missed pong raises ConnectionClosed, exits _listen(),
and triggers the reconnect loop which re-syncs any missed events.
When the CLI receives a sync event from the server and writes/deletes
a file locally, the watchdog inotify event is queued by the kernel and
delivered asynchronously. By the time the observer thread processes it,
unignore_file had already been called, causing the watcher to treat the
server-written file as a local change and push it back — creating an
echo loop confirmed in logs (NoteSyncModify received → NoteModify sent
back → server pushes again).

Fix: add asyncio.sleep(0.6) before unignore_file in all server→client
file operation handlers (write, delete, rename, chunked download) in
both NoteSync and FileSync. This keeps the file ignored until the
watchdog debounce window (0.5s) has expired.
… timeout (#8)

PR #6 used ping_interval=30 but the server does not respond to WebSocket
ping frames — confirmed by real-environment test: connection dropped every
30s with "keepalive ping timeout". Reverted to ping_interval=None.

Instead, an asyncio background task (_inactivity_watchdog) tracks the
last received message time and closes the connection if no data arrives
for 2 × heartbeat_interval (60s default), triggering the reconnect loop.
_last_received_at is updated on every text or binary frame received.

Also extended the NoteSync timeout in _initial_sync from 60s to 300s.
The server sends NoteSyncEnd immediately but may delay actual note
delivery by over a minute on full syncs — the 60s window was too short
and caused NoteSync to abort before any notes were written.

Verified with real-server pull: 24 notes (including Test-456/123.md,
456.md, 789.md) fully synced, no ping disconnects, no echo push-backs.
Core changes across the sync stack:

* client.py — re-enable websockets lib's own ping with generous timeout
  (ping_interval=45, ping_timeout=90). Auto-pongs to the server's 25s
  pings keep its 60s deadline refreshed; the long timeout tolerates Pong
  delay behind large binary chunk writes that was tripping keepalive at
  30s and 60s during initial sync.

* note_sync.py / file_sync.py — replace the 0.6s/2s time-based ignore
  with a content-hash echo cache. The sleep-in-handler serialized the
  whole WS receive loop: 178 modifies × 2s ate the 300s sync window.
  _echo_hashes[path] now tracks the last known synced state and is
  updated on BOTH inbound (server write) and outbound (push) so a
  legitimate revert to a previous value still propagates. _DELETED
  sentinel for deletes, with the same outbound update so recreate +
  re-delete works. push_modify accepts force=True and NoteSyncNeedPush
  uses it to bypass the echo check.

* watcher.py — directory rename now enumerates the new tree and
  schedules per-file transitions. _schedule_move_transition handles the
  excluded-path edge cases: move into excluded → emit delete of old,
  move out of excluded → emit modify of new.

* doc/server-protocol.md — new reference distilled from the Go source:
  heartbeat protocol, broadcast mechanics, sync flow, and the specific
  client pitfalls we hit.

* tests — new test_echo_cache.py covers the revert, tombstone-reuse,
  NeedPush-force, and failed-write-no-poison cases; test_file_sync adds
  the excluded-path rename transitions.

23 unit tests pass. Bidirectional integration tests (not committed)
confirm notes, attachments, and folders sync correctly in both
directions for create / modify / delete / rename.
Fix realtime sync startup and directory deletes
…n, .agents, etc.)

Adds full support for synchronizing Obsidian config directories
(e.g. .obsidian, .agents/skills) via the SettingSync WebSocket protocol.

- protocol.py: add SettingSync action constants
- state.py: persist last_setting_sync_time
- setting_sync.py: new module handling SettingSyncModify/Delete/Rename/Mtime/End
- sync_engine.py: wire setting_sync into pull/push/run and watcher callbacks
- file_sync.py: exclude all dot-prefixed dirs from FileSync to avoid conflicts

Resolves the issue where config files uploaded by the Obsidian plugin
never reached the CLI because the CLI only implemented NoteSync/FileSync.
feat: implement SettingSync protocol for config directories (.obsidian, .agents, etc.)
Config files (.obsidian/*, .agents/*, etc.) should only be synced via
SettingSync protocol, not FileSync. This fixes three issues:

1. _initial_sync(): only call file_sync.request_sync() when sync_files
   is true, not when sync_config is true
2. _push_all_files(): skip config files entirely, let _push_all_settings
   handle them via SettingSync
3. file_sync._collect_local_files(): always skip dot-prefixed directories,
   regardless of sync_config setting

Previously, config files were uploaded to both file DB and setting DB,
causing conflicts and plugin loss during sync.
Config directories (.obsidian, .agents, etc.) are managed by SettingSync
protocol, not FolderSync. This prevents accidental deletion of the entire
.obsidian directory when the server sends FolderSyncDelete after config
records are removed from the file database.

The fix adds checks in all three FolderSync handlers:
- _on_sync_modify: skip creating dot-prefixed dirs
- _on_sync_delete: skip deleting dot-prefixed dirs (CRITICAL)
- _on_sync_rename: skip renaming dot-prefixed dirs
Previously, FileSync and SettingSync had inconsistent hardcoded handling
of dot-prefixed directories. This caused custom config directories like
.agents to not be properly synchronized.

Changes:
- config.py: add config_sync_dirs field to SyncConfig (default: [.obsidian, .agents])
- sync_engine.py: use config_sync_dirs from config instead of hardcoded list
- setting_sync.py: pass config_sync_dirs to _is_config_path()
- file_sync.py: skip dot-prefixed dirs only when sync_config is enabled
- config.yaml: add config_sync_dirs example configuration

Users can now customize which dot-prefixed directories are treated as config
by editing config.yaml:

  config_sync_dirs:
    - .obsidian
    - .agents
    - .my-custom-config

Fixes: custom config directories (.agents) not being synced by CLI
@sourcery-ai
Copy link
Copy Markdown

sourcery-ai Bot commented Apr 27, 2026

Reviewer's Guide

Makes handling of dot-prefixed config directories explicit and configurable so that only selected directories (e.g. .obsidian, .agents) are treated as config and are synced via SettingSync, while FolderSync/FileSync ignore them appropriately based on sync_config/sync_files and config_sync_dirs in config.yaml.

Sequence diagram for deciding config vs file sync for dot-prefixed directories

sequenceDiagram
    actor User
    participant SyncEngine
    participant FileSync
    participant SettingSync
    participant FolderSync

    Note over User: User triggers a sync operation

    User->>SyncEngine: start_sync()
    SyncEngine->>SyncEngine: _initial_sync()

    Note over SyncEngine: Config: sync_files, sync_config, config_sync_dirs

    SyncEngine->>NoteSync: request_sync()
    NoteSync-->>SyncEngine: notes_synced

    alt sync_files is true
        SyncEngine->>FileSync: request_sync()
        FileSync-->>SyncEngine: file_sync_started
    end

    alt sync_config is true
        SyncEngine->>SettingSync: request_sync()
        SettingSync-->>SyncEngine: setting_sync_started
    end

    loop For each local file
        SyncEngine->>SyncEngine: _is_config(rel_path)
        Note over SyncEngine: first = rel_path.split("/")[0]
        alt first not dot-prefixed
            SyncEngine-->>SyncEngine: returns False
            alt sync_files is true
                SyncEngine->>FileSync: push_upload(rel_path)
            else sync_files is false
                SyncEngine-->>SyncEngine: skip file
            end
        else first dot-prefixed
            alt first in config_sync_dirs
                SyncEngine-->>SyncEngine: returns True
                alt sync_config is true
                    SyncEngine->>SettingSync: push_upload(rel_path)
                else sync_config is false
                    SyncEngine-->>SyncEngine: skip file
                end
            else first not in config_sync_dirs
                alt sync_config is true
                    SyncEngine-->>SyncEngine: returns True
                    SyncEngine->>SettingSync: push_upload(rel_path)
                else sync_config is false
                    SyncEngine-->>SyncEngine: returns False
                    alt sync_files is true
                        SyncEngine->>FileSync: push_upload(rel_path)
                    else sync_files is false
                        SyncEngine-->>SyncEngine: skip file
                    end
                end
            end
        end
    end

    Note over FolderSync: Handles folder events from server

    Server->>FolderSync: _on_sync_modify(msg)
    FolderSync->>FolderSync: first = rel_path.split("/")[0]
    alt first startswith dot
        FolderSync-->>FolderSync: log ignore and return
    else not dot-prefixed
        FolderSync->>FolderSync: mkdir for folder
    end

    Server->>FolderSync: _on_sync_delete(msg)
    FolderSync->>FolderSync: first = rel_path.split("/")[0]
    alt first startswith dot
        FolderSync-->>FolderSync: log ignore and return
    else not dot-prefixed
        FolderSync->>FolderSync: delete folder if exists
    end

    Server->>FolderSync: _on_sync_rename(msg)
    FolderSync->>FolderSync: check first segments of old and new
    alt either startswith dot
        FolderSync-->>FolderSync: log ignore and return
    else neither dot-prefixed
        FolderSync->>FolderSync: rename folder
    end
Loading

Class diagram for configurable config_sync_dirs and sync routing

classDiagram
    class SyncConfig {
        +list~str~ exclude_patterns
        +int file_chunk_size
        +list~str~ config_sync_dirs
    }

    class AppConfig {
        +SyncConfig sync
        +ClientConfig client
        +ServerConfig server
    }

    class SyncEngine {
        +AppConfig config
        +Path vault_path
        +FileSync file_sync
        +SettingSync setting_sync
        +NoteSync note_sync
        +FolderSync folder_sync
        +bool _is_note(rel_path)
        +bool _is_config(rel_path)
        +bool _should_sync_file(rel_path)
        +async _initial_sync()
        +async _push_all_files()
    }

    class FileSync {
        +AppConfig config
        +SyncEngine engine
        +list~dict~ _collect_local_files()
    }

    class SettingSync {
        +AppConfig config
        +SyncEngine engine
        +async request_sync()
        +async push_upload(rel_path)
    }

    class SettingSyncHelpers {
        +bool _is_config_path(rel, config_sync_dirs)
        +dict _extract_inner(msg_data)
    }

    class FolderSync {
        +Path vault_path
        +SyncEngine engine
        +async _on_sync_modify(msg)
        +async _on_sync_delete(msg)
        +async _on_sync_rename(msg)
    }

    class WSMessage {
        +dict data
    }

    class ConfigLoader {
        +AppConfig load_config(path)
    }

    SyncEngine --> SyncConfig : uses
    SyncEngine --> FileSync : coordinates
    SyncEngine --> SettingSync : coordinates
    SyncEngine --> FolderSync : coordinates

    AppConfig --> SyncConfig : has

    FileSync --> SyncEngine : engine

    SettingSync --> SyncEngine : engine

    FolderSync --> SyncEngine : engine

    ConfigLoader --> AppConfig : constructs

    SettingSyncHelpers --> SettingSync : utility
    SettingSyncHelpers ..> SyncConfig : uses config_sync_dirs

    SyncEngine ..> SyncConfig : uses config_sync_dirs
    FileSync ..> SyncConfig : uses config_sync_dirs and sync_config
    FolderSync ..> SyncConfig : indirectly uses via engine
Loading

File-Level Changes

Change Details Files
Introduce configurable list of config sync directories and wire it into sync logic
  • Add SyncConfig.config_sync_dirs with default ['.obsidian', '.agents']
  • Load config_sync_dirs from config.yaml into AppConfig
  • Use config_sync_dirs when determining whether a path is treated as config in SyncEngine and SettingSync
fns_cli/config.py
fns_cli/sync_engine.py
fns_cli/setting_sync.py
Align FileSync, FolderSync, and SyncEngine behavior for dot-prefixed directories
  • FileSync now skips dot-prefixed directories only when sync_config is enabled, treating them as config handled by SettingSync
  • SyncEngine._is_config now checks first path segment against config_sync_dirs and falls back to sync_config for other dot-dirs
  • SyncEngine initial sync and _push_all_files logic updated so file sync runs based on sync_files and never uploads config files
  • FolderSync now ignores create/delete/rename events for dot-prefixed directories, logging them as config dirs handled elsewhere
fns_cli/file_sync.py
fns_cli/sync_engine.py
fns_cli/folder_sync.py
Update SettingSync to use configurable config directories while remaining backward compatible
  • _is_config_path now accepts a config_sync_dirs parameter and checks first segment against it
  • If a path is dot-prefixed but not listed, it is still treated as config by default to preserve prior behavior
fns_cli/setting_sync.py

Possibly linked issues

  • #(unknown): They describe the same dot-prefixed config directory sync bug; the PR implements the issue’s proposed configuration-based fix.

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

Copy link
Copy Markdown

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • FolderSync currently hardcodes all dot-prefixed directories as config (skipping them unconditionally), which diverges from the new config_sync_dirs + sync_config logic used elsewhere; consider reusing the same _is_config/config logic here so FolderSync behavior stays consistent with FileSync/SettingSync.
  • The new config_sync_dirs setting is only partially wired through (e.g., _is_config uses it but _is_config_path’s default behavior remains "all dot dirs are config" unless it is explicitly passed); it would be cleaner to ensure all config-path checks consistently receive and honor config_sync_dirs so the effective rules don’t differ between components.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- FolderSync currently hardcodes all dot-prefixed directories as config (skipping them unconditionally), which diverges from the new `config_sync_dirs` + `sync_config` logic used elsewhere; consider reusing the same `_is_config`/config logic here so FolderSync behavior stays consistent with FileSync/SettingSync.
- The new `config_sync_dirs` setting is only partially wired through (e.g., `_is_config` uses it but `_is_config_path`’s default behavior remains "all dot dirs are config" unless it is explicitly passed); it would be cleaner to ensure all config-path checks consistently receive and honor `config_sync_dirs` so the effective rules don’t differ between components.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

@LLQWQ LLQWQ closed this Apr 27, 2026
@LLQWQ LLQWQ force-pushed the feat/setting-sync-v2 branch from 09f4e40 to 57024ca Compare April 27, 2026 06:57
@LLQWQ LLQWQ deleted the feat/setting-sync-v2 branch April 27, 2026 06:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants