使用版本控制下的IPython / Jupyter笔记本

将IPython笔记本保持在版本控制下的好策略是什么?

笔记本格式非常适合版本控制:如果想对笔记本和输出进行版本控制，那么这种方法非常有效。当人们只想对输入进行版本控制，而不包括单元格输出时，就会出现烦恼。“构建产品”)，可以是大的二进制blob，特别是电影和情节。特别是，我试图找到一个好的工作流程:

allows me to choose between including or excluding output, prevents me from accidentally committing output if I do not want it, allows me to keep output in my local version, allows me to see when I have changes in the inputs using my version control system (i.e. if I only version control the inputs but my local file has outputs, then I would like to be able to see if the inputs have changed (requiring a commit). Using the version control status command will always register a difference since the local file has outputs.) allows me to update my working notebook (which contains the output) from an updated clean notebook. (update)

如前所述，如果我选择包含输出(例如，在使用nbviewer时，这是可取的)，那么一切都没问题。问题是当我不想对输出进行版本控制时。有一些工具和脚本可以剥离笔记本的输出，但我经常遇到以下问题:

I accidentally commit a version with the the output, thereby polluting my repository. I clear output to use version control, but would really rather keep the output in my local copy (sometimes it takes a while to reproduce for example). Some of the scripts that strip output change the format slightly compared to the Cell/All Output/Clear menu option, thereby creating unwanted noise in the diffs. This is resolved by some of the answers. When pulling changes to a clean version of the file, I need to find some way of incorporating those changes in my working notebook without having to rerun everything. (update)

我已经考虑了下面将要讨论的几个选项，但是还没有找到一个好的全面的解决方案。完整的解决方案可能需要对IPython进行一些更改，或者可能依赖于一些简单的外部脚本。我目前使用mercurial，但希望有一个解决方案也能与git一起工作:一个理想的解决方案是版本控制不可知的。

这个问题已经讨论过很多次了，但是从用户的角度来看，还没有明确的解决方案。这个问题的答案应该能提供明确的策略。如果它需要IPython的最新(甚至是开发版)版本或易于安装的扩展，那是没问题的。

更新:我一直在玩我修改过的笔记本版本，它可以选择保存一个.clean版本，每次保存都使用Gregory Crosswhite的建议。这满足了我的大部分约束条件，但留下了以下问题:

This is not yet a standard solution (requires a modification of the ipython source. Is there a way of achieving this behaviour with a simple extension? Needs some sort of on-save hook. A problem I have with the current workflow is pulling changes. These will come in to the .clean file, and then need to be integrated somehow into my working version. (Of course, I can always re-execute the notebook, but this can be a pain, especially if some of the results depend on long calculations, parallel computations, etc.) I do not have a good idea about how to resolve this yet. Perhaps a workflow involving an extension like ipycache might work, but that seems a little too complicated.

笔记

移除(剥离)输出

When the notebook is running, one can use the Cell/All Output/Clear menu option for removing the output. There are some scripts for removing output, such as the script nbstripout.py which remove the output, but does not produce the same output as using the notebook interface. This was eventually included in the ipython/nbconvert repo, but this has been closed stating that the changes are now included in ipython/ipython,but the corresponding functionality seems not to have been included yet. (update) That being said, Gregory Crosswhite's solution shows that this is pretty easy to do, even without invoking ipython/nbconvert, so this approach is probably workable if it can be properly hooked in. (Attaching it to each version control system, however, does not seem like a good idea — this should somehow hook in to the notebook mechanism.)

新闻组

关于版本控制的笔记本格式的思考。

问题

977:笔记本功能请求(打开)。 1280:清除-all保存选项(打开)。(从下面的讨论。) 3295:自动导出的笔记本:只导出显式标记的单元格(关闭)。扩展解决11添加写和执行魔法(合并)。

把请求

1621: clear In[] prompt numbers on "Clear All Output" (Merged). (See also 2519 (Merged).) 1563: clear_output improvements (Merged). 3065: diff-ability of notebooks (Closed). 3291: Add the option to skip output cells when saving. (Closed). This seems extremely relevant, however was closed with the suggestion to use a "clean/smudge" filter. A relevant question what can you use if you want to strip off output before running git diff? seems not to have been answered. 3312: WIP: Notebook save hooks (Closed). 3747: ipynb -> ipynb transformer (Closed). This is rebased in 4175. 4175: nbconvert: Jinjaless exporter base (Merged). 142: Use STDIN in nbstripout if no input is given (Open).

当前回答

我还将添加到其他人建议的https://nbdev.fast.ai/，这是一个最先进的“文学编程环境，正如Donald Knuth在1983年所设想的那样!”

它也有一些git钩子，可以帮助https://nbdev.fast.ai/#Avoiding-and-handling-git-conflicts和其他命令，如:

nbdev_read_nbs nbdev_clean_nbs nbdev_diff_nbs nbdev_test_nbs

所以你也可以创建你的文档，就像在写一个库，例如:

https://dev.fast.ai/ https://ohmeow.github.io/blurr/ https://rbracco.github.io/fastai2_audio/

除了第一个链接，您还可以在这里看到一个视频nbdev教程。

2020-06-19 23:56:40

其他回答

更新:现在你可以编辑Jupyter笔记本文件直接在Visual Studio Code。您可以选择编辑笔记本或转换后的python文件。

我终于找到了一个高效而简单的方法，可以让Jupyter和Git很好地一起工作。我还在第一步，但我已经认为它比所有其他复杂的解决方案要好得多。

Visual Studio Code是微软的一个很酷的开源代码编辑器。它有一个优秀的Python扩展，现在允许你导入一个Jupyter笔记本作为Python代码。现在你也可以直接编辑Jupyter笔记本。

将笔记本导入python文件后，所有代码和标记都将放在一个普通的python文件中，注释中有特殊标记。如下图所示:

你的python文件只有笔记本输入单元格的内容。输出将在拆分窗口中生成。你的笔记本里有纯代码，你只是执行它，它不会改变。没有混合输出与您的代码。没有奇怪的JSON难以理解的格式来分析你的差异。

只是纯粹的python代码，你可以很容易地识别每一个差异。

我甚至不再需要为我的.ipynb文件设置版本了。我可以写上*。Ipynb线在。gitignore。

需要生成一个笔记本来发布或与他人共享?没问题，只需在交互式python窗口中单击导出按钮

如果你直接编辑笔记本，现在有一个图标Convert and save to a python script。

这是Visual Studio Code中一个笔记本的截图:

我只使用了一天，但我终于可以愉快地使用Jupyter和Git了。

注:VSCode的代码完成比Jupyter好得多。

2018-11-21 00:48:50

你可以使用这个jupyter扩展。它可以让你直接上传你的ipython笔记本到github。

https://github.com/sat28/githubcommit

我还制作了一个视频来演示这些步骤 youtube链接

2018-01-17 11:02:31

不幸的是，我对Mercurial了解不多，但我可以为您提供一个与Git一起工作的可能解决方案，希望您能够将我的Git命令转换为对应的Mercurial命令。

对于后台，在Git中，add命令将对文件所做的更改存储到暂存区域。完成此操作后，Git将忽略对文件的任何后续更改，除非您告诉它也执行这些更改。因此，下面的脚本，对于每个给定的文件，剥离所有的输出和prompt_number部分，运行剥离的文件，然后恢复原始文件:

注意:如果运行此命令会得到类似ImportError的错误消息:没有名为IPython的模块。Nbformat，然后使用ipython而不是python来运行脚本。

from IPython.nbformat import current
import io
from os import remove, rename
from shutil import copyfile
from subprocess import Popen
from sys import argv

for filename in argv[1:]:
    # Backup the current file
    backup_filename = filename + ".backup"
    copyfile(filename,backup_filename)

    try:
        # Read in the notebook
        with io.open(filename,'r',encoding='utf-8') as f:
            notebook = current.reads(f.read(),format="ipynb")

        # Strip out all of the output and prompt_number sections
        for worksheet in notebook["worksheets"]:
            for cell in worksheet["cells"]:
               cell.outputs = []
               if "prompt_number" in cell:
                    del cell["prompt_number"]

        # Write the stripped file
        with io.open(filename, 'w', encoding='utf-8') as f:
            current.write(notebook,f,format='ipynb')

        # Run git add to stage the non-output changes
        print("git add",filename)
        Popen(["git","add",filename]).wait()

    finally:
        # Restore the original file;  remove is needed in case
        # we are running in windows.
        remove(filename)
        rename(backup_filename,filename)

一旦脚本在您想要提交的文件上运行，只需运行git commit。

2013-11-04 04:27:58

下面是Cyrille rosant为IPython 3.0提供的一个新解决方案，它坚持标记文件，而不是基于json的ipymd文件:

https://github.com/rossant/ipymd

2015-02-21 22:09:36

(2017-02)

策略

on_commit (): 去掉输出>的名称。Ipynb (nbstripout) 去掉输出> name.clean.ipynb (nbstripout，) 总是nbconvert to python: name.ipynb.py (nbconvert) 始终转换为markdown: name.ipynb.md (nbconvert, ipymd) vcs.configure (): Git difftool, mergetool: nbdiff和nbmerge from nbdime

工具

nbstripout: strip the outputs from a notebook src: https://gist.github.com/minrk/6176788 src: https://github.com/kynan/nbstripout pip install nbstripout; nbstripout install ipynb_output_filter: strip the outputs from a notebook src: https://github.com/toobaz/ipynb_output_filter/blob/master/ipynb_output_filter.py ipymd: convert between {Jupyter, Markdown, O'Reilly Atlas Markdown, OpenDocument, .py} src: https://github.com/rossant/ipymd nbdime: "Tools for diffing and merging of Jupyter notebooks." (2015) src: https://github.com/jupyter/nbdime docs: http://nbdime.readthedocs.io/ nbdiff: compare notebooks in a terminal-friendly way nbdime nbdiff works as a git diff tool: https://nbdime.readthedocs.io/en/latest/#git-integration-quickstart nbmerge: three-way merge of notebooks with automatic conflict resolution nbdime nbmerge works as a git merge tool nbdiff-web: shows you a rich rendered diff of notebooks nbmerge-web: gives you a web-based three-way merge tool for notebooks nbshow: present a single notebook in a terminal-friendly way

2017-02-09 04:40:37

使用版本控制下的IPython / Jupyter笔记本

推荐文章

最新文章

标签