repository - Is git worth for managing many files bigger than 500MB

Question

I would put under version control a big amount of data, i.e. a directory structure (with depth<=5) with hundreds files with size about 500Mb).

The things I need is a system that help me: - to detect if an files has been changed - to detect if files were added/removed - to clone the entire repository in another location - to store a "checkpoint" and restore it later

I don't need sha1 for change detect, something faster is acceptable.

Is git worth for this? There is a better alternative?

Answer 1

As I mentioned in "What are the Git limits", Git is not made to manage big files (or big binary files for that matter).

Git would be needed if you needed to:

• know what has actually changed within a file. But for the directory-level, the other answers are better (Unison or rsynch)
• keep a close proximity (i.e. "same referential") between your development data, and those large resources. Having only one referential would help, but then you would need a fork of Git, like git-bigfiles to efficiently manage them.

---

Note: still using Git, you can try this approach

Unfortunately, rsync isn't really perfect for our purposes either.

• First of all, it isn't really a version control system. If you want to store multiple revisions of the file, you have to make multiple copies, which is wasteful, or xdelta them, which is tedious (and potentially slow to reassemble, and makes it hard to prune intermediate versions), or check them into git, which will still melt down because your files are too big.
• Plus rsync really can't handle file renames properly - at all.

Okay, what about another idea: let's split the file into chunks, and check each of those blocks into git separately.
Then git's delta compression won't have too much to chew on at a time, and we only have to send modified blocks...

Based on gzip --rsyncable, with a POC available in this Git repo.