linux - what options to use with rsync to sync online new files to remote ntfs drive?

Question

i run a mixed windows and linux network with different desktops, notebooks and raspberry pis. i am trying to establish an off-site backup between a local raspberry pi and an remote raspberry pi. both run on dietpi/raspbian and have an external hdd with ntfs to store the backup data. as the data to be backuped is around 800GB i already initially mirrored the data on the external hdd, so that only the new files have to be sent to the remote drive via rsync.

i now tried various combinations of options including --ignore-existing --size-only -u -c und of course combinations of other options like -avz etc.

the problem is: nothing of the above really changes anything, the system tries to upload all the files (although they are remotely existing) or at least a good number of them.

could you give me hint how to solve this?

question from:https://stackoverflow.com/questions/65937939/what-options-to-use-with-rsync-to-sync-online-new-files-to-remote-ntfs-drive

Answer 1

I do this exact thing. Here is my solution to this task.

rsync -re "ssh -p 1234” -K -L --copy-links --append --size-only --delete [email protected]:/home/pi/source-directory/* /home/pi/target-directory/

The options I am using are:

-r - recursive

-e - specifies to utilize ssh / scp as a method to transmit data, a note, my ssh command uses a non-standard port 1234 as is specified by -p in the -e flag

-K - keep directory links

-L - copy links

--copy-links - a duplicate flag it would seem...

--append - this will append data onto smaller files in case of a partial copy

--size-only - this skips files that match in size

--delete - CAREFUL - this will delete local files that are not present on the remote device..

This solution will run on a schedule and will "sync" the files in the target directories with the files from the source directory. to test it out, you can always run the command with --dry-run which will not make any changes at all and only show you what will be transferred and/or deleted...

all of this info and additional details can be found in the rsync man page man rsync

**NOTE: I use ssh keys to allow connection/transfer without having to respond to a password prompt between these devices.