我正在寻找如何处理我的源代码(web应用程序)依赖的大型二进制文件的意见。我们目前正在讨论几种替代方案:

Copy the binary files by hand. Pro: Not sure. Contra: I am strongly against this, as it increases the likelihood of errors when setting up a new site/migrating the old one. Builds up another hurdle to take. Manage them all with Git. Pro: Removes the possibility to 'forget' to copy a important file Contra: Bloats the repository and decreases flexibility to manage the code-base and checkouts, clones, etc. will take quite a while. Separate repositories. Pro: Checking out/cloning the source code is fast as ever, and the images are properly archived in their own repository. Contra: Removes the simpleness of having the one and only Git repository on the project. It surely introduces some other things I haven't thought about.

你对此有什么经验/想法?

还有:有人有在一个项目中使用多个Git存储库并管理它们的经验吗?

这些文件是用于生成包含这些文件的pdf文件的程序的图像。这些文件不会经常更改(例如几年),但它们与程序非常相关。没有这些文件,程序将无法工作。


当前回答

SVN似乎比Git更有效地处理二进制增量。

我必须决定文档的版本控制系统(JPEG文件、PDF文件和.odt文件)。我刚刚测试了添加一个JPEG文件并将其旋转90度4次(以检查二进制增量的有效性)。Git的存储库增长了400%。SVN的存储库仅增长了11%。

因此,看起来SVN使用二进制文件更有效率。

所以我选择Git作为源代码,SVN作为文档之类的二进制文件。

其他回答

在我看来,如果您可能经常修改这些大文件,或者您打算进行大量的git克隆或git签出,那么您应该认真考虑使用另一个git存储库(或者可能是访问这些文件的另一种方法)。

但是如果您像我们一样工作,并且您的二进制文件不经常修改,那么第一次克隆/签出将会很长,但是在那之后它应该和您想要的一样快(考虑到您的用户一直使用他们拥有的第一个克隆存储库)。

我正在寻找如何处理我的源代码(web应用程序)依赖的大型二进制文件的意见。你对此有什么经验/想法?

当我的web应用程序二进制数据超过3gb时,我个人在我的一些云主机上就遇到过Git同步失败的情况。我当时考虑过BFT回购清洁,但感觉像一个黑客。从那时起,我开始将文件置于Git的权限之外,而是利用专门构建的工具(如Amazon S3)来管理文件、版本控制和备份。

有人有在一个项目中使用多个Git存储库并管理它们的经验吗?

是的。雨果主题主要是这样管理的。这有点滑稽,但它能完成任务。


我的建议是选择适合这项工作的工具。如果它是为一个公司,你在GitHub上管理你的代码线,付钱并使用Git-LFS。否则,您可以探索更有创意的选项,例如使用区块链进行分散加密文件存储。

需要考虑的其他选项包括Minio和s3cmd。

git克隆——过滤从git 2.19 +浅克隆

这个新选项可能最终会成为二进制文件问题的最终解决方案,如果Git和GitHub开发并使其足够友好(他们可以说仍然没有实现子模块例如)。

它实际上只允许为服务器获取您想要的文件和目录,并与远程协议扩展一起引入。

有了这个,我们可以先做一个浅克隆,然后自动使用构建系统为每种类型的构建获取哪些blobs。

甚至已经有一个——filter=blob:limit<size>,它允许限制读取的最大blob大小。

我提供了一个关于该特性的最小详细示例:如何克隆Git存储库的子目录?

看看camlistore。它不是真正基于git的,但我发现它更适合您必须做的事情。

我想提出的解决方案是基于孤立分支和对标记机制的轻微滥用,因此称为*孤立标记二进制存储(OTABS)

如果你可以使用github的LFS或其他第三方,无论如何你应该。如果你不能,那么继续读下去。请注意,这个解决方案是一个黑客,应该被这样对待。

OTABS的理想属性

it is a pure git and git only solution -- it gets the job done without any 3rd party software (like git-annex) or 3rd party infrastructure (like github's LFS). it stores the binary files efficiently, i.e. it doesn't bloat the history of your repository. git pull and git fetch, including git fetch --all are still bandwidth efficient, i.e. not all large binaries are pulled from the remote by default. it works on Windows. it stores everything in a single git repository. it allows for deletion of outdated binaries (unlike bup).

OTABS的不良属性

it makes git clone potentially inefficient (but not necessarily, depending on your usage). If you deploy this solution you might have to advice your colleagues to use git clone -b master --single-branch <url> instead of git clone. This is because git clone by default literally clones entire repository, including things you wouldn't normally want to waste your bandwidth on, like unreferenced commits. Taken from SO 4811434. it makes git fetch <remote> --tags bandwidth inefficient, but not necessarily storage inefficient. You can can always advise your colleagues not to use it. you'll have to periodically use a git gc trick to clean your repository from any files you don't want any more. it is not as efficient as bup or git-bigfiles. But it's respectively more suitable for what you're trying to do and more off-the-shelf. You are likely to run into trouble with hundreds of thousands of small files or with files in range of gigabytes, but read on for workarounds.

添加二进制文件

在开始之前,请确保您已经提交了所有的更改,您的工作树是最新的,并且您的索引不包含任何未提交的更改。这可能是一个好主意,把你所有的本地分支推到你的远程(github等)以防任何灾难发生。

Create a new orphan branch. git checkout --orphan binaryStuff will do the trick. This produces a branch that is entirely disconnected from any other branch, and the first commit you'll make in this branch will have no parent, which will make it a root commit. Clean your index using git rm --cached * .gitignore. Take a deep breath and delete entire working tree using rm -fr * .gitignore. Internal .git directory will stay untouched, because the * wildcard doesn't match it. Copy in your VeryBigBinary.exe, or your VeryHeavyDirectory/. Add it && commit it. Now it becomes tricky -- if you push it into the remote as a branch all your developers will download it the next time they invoke git fetch clogging their connection. You can avoid this by pushing a tag instead of a branch. This can still impact your colleague's bandwidth and filesystem storage if they have a habit of typing git fetch <remote> --tags, but read on for a workaround. Go ahead and git tag 1.0.0bin Push your orphan tag git push <remote> 1.0.0bin. Just so you never push your binary branch by accident, you can delete it git branch -D binaryStuff. Your commit will not be marked for garbage collection, because an orphan tag pointing on it 1.0.0bin is enough to keep it alive.

签出二进制文件

How do I (or my colleagues) get the VeryBigBinary.exe checked out into the current working tree? If your current working branch is for example master you can simply git checkout 1.0.0bin -- VeryBigBinary.exe. This will fail if you don't have the orphan tag 1.0.0bin downloaded, in which case you'll have to git fetch <remote> 1.0.0bin beforehand. You can add the VeryBigBinary.exe into your master's .gitignore, so that no-one on your team will pollute the main history of the project with the binary by accident.

完全删除二进制文件

如果你决定完全清除VeryBigBinary.exe从你的本地存储库,你的远程存储库和你的同事的存储库,你可以:

Delete the orphan tag on the remote git push <remote> :refs/tags/1.0.0bin Delete the orphan tag locally (deletes all other unreferenced tags) git tag -l | xargs git tag -d && git fetch --tags. Taken from SO 1841341 with slight modification. Use a git gc trick to delete your now unreferenced commit locally. git -c gc.reflogExpire=0 -c gc.reflogExpireUnreachable=0 -c gc.rerereresolved=0 -c gc.rerereunresolved=0 -c gc.pruneExpire=now gc "$@". It will also delete all other unreferenced commits. Taken from SO 1904860 If possible, repeat the git gc trick on the remote. It is possible if you're self-hosting your repository and might not be possible with some git providers, like github or in some corporate environments. If you're hosting with a provider that doesn't give you ssh access to the remote just let it be. It is possible that your provider's infrastructure will clean your unreferenced commit in their own sweet time. If you're in a corporate environment you can advice your IT to run a cron job garbage collecting your remote once per week or so. Whether they do or don't will not have any impact on your team in terms of bandwidth and storage, as long as you advise your colleagues to always git clone -b master --single-branch <url> instead of git clone. All your colleagues who want to get rid of outdated orphan tags need only to apply steps 2-3. You can then repeat the steps 1-8 of Adding the Binary Files to create a new orphan tag 2.0.0bin. If you're worried about your colleagues typing git fetch <remote> --tags you can actually name it again 1.0.0bin. This will make sure that the next time they fetch all the tags the old 1.0.0bin will be unreferenced and marked for subsequent garbage collection (using step 3). When you try to overwrite a tag on the remote you have to use -f like this: git push -f <remote> <tagname>

后记

OTABS doesn't touch your master or any other source code/development branches. The commit hashes, all of the history, and small size of these branches is unaffected. If you've already bloated your source code history with binary files you'll have to clean it up as a separate piece of work. This script might be useful. Confirmed to work on Windows with git-bash. It is a good idea to apply a set of standard trics to make storage of binary files more efficient. Frequent running of git gc (without any additional arguments) makes git optimise underlying storage of your files by using binary deltas. However, if your files are unlikely to stay similar from commit to commit you can switch off binary deltas altogether. Additionally, because it makes no sense to compress already compressed or encrypted files, like .zip, .jpg or .crypt, git allows you to switch off compression of the underlying storage. Unfortunately it's an all-or-nothing setting affecting your source code as well. You might want to script up parts of OTABS to allow for quicker usage. In particular, scripting steps 2-3 from Completely Deleting Binary Files into an update git hook could give a compelling but perhaps dangerous semantics to git fetch ("fetch and delete everything that is out of date"). You might want to skip the step 4 of Completely Deleting Binary Files to keep a full history of all binary changes on the remote at the cost of the central repository bloat. Local repositories will stay lean over time. In Java world it is possible to combine this solution with maven --offline to create a reproducible offline build stored entirely in your version control (it's easier with maven than with gradle). In Golang world it is feasible to build on this solution to manage your GOPATH instead of go get. In python world it is possible to combine this with virtualenv to produce a self-contained development environment without relying on PyPi servers for every build from scratch. If your binary files change very often, like build artifacts, it might be a good idea to script a solution which stores 5 most recent versions of the artifacts in the orphan tags monday_bin, tuesday_bin, ..., friday_bin, and also an orphan tag for each release 1.7.8bin 2.0.0bin, etc. You can rotate the weekday_bin and delete old binaries daily. This way you get the best of two worlds: you keep the entire history of your source code but only the relevant history of your binary dependencies. It is also very easy to get the binary files for a given tag without getting entire source code with all its history: git init && git remote add <name> <url> && git fetch <name> <tag> should do it for you.