我目前通过一堆不好记的AWK、sed、Bash和一小部分Perl来做我的文本文件操作。
我已经看到提到的一些地方,python很适合做这种事情。我如何使用Python来代替shell脚本,AWK, sed和朋友?
我目前通过一堆不好记的AWK、sed、Bash和一小部分Perl来做我的文本文件操作。
我已经看到提到的一些地方,python很适合做这种事情。我如何使用Python来代替shell脚本,AWK, sed和朋友?
当前回答
任何shell都有几组特性。
The Essential Linux/Unix commands. All of these are available through the subprocess library. This isn't always the best first choice for doing all external commands. Look also at shutil for some commands that are separate Linux commands, but you could probably implement directly in your Python scripts. Another huge batch of Linux commands are in the os library; you can do these more simply in Python. And -- bonus! -- more quickly. Each separate Linux command in the shell (with a few exceptions) forks a subprocess. By using Python shutil and os modules, you don't fork a subprocess. The shell environment features. This includes stuff that sets a command's environment (current directory and environment variables and what-not). You can easily manage this from Python directly. The shell programming features. This is all the process status code checking, the various logic commands (if, while, for, etc.) the test command and all of it's relatives. The function definition stuff. This is all much, much easier in Python. This is one of the huge victories in getting rid of bash and doing it in Python. Interaction features. This includes command history and what-not. You don't need this for writing shell scripts. This is only for human interaction, and not for script-writing. The shell file management features. This includes redirection and pipelines. This is trickier. Much of this can be done with subprocess. But some things that are easy in the shell are unpleasant in Python. Specifically stuff like (a | b; c ) | something >result. This runs two processes in parallel (with output of a as input to b), followed by a third process. The output from that sequence is run in parallel with something and the output is collected into a file named result. That's just complex to express in any other language.
特定的程序(awk、sed、grep等)通常可以被重写为Python模块。不要走极端。替换您需要的内容并改进“grep”模块。不要一开始就编写一个替换“grep”的Python模块。
最好的事情是你可以一步一步来做。
用Python替换AWK和PERL。不要管其他的事情。 看看用Python替换GREP。这可能有点复杂,但是您的GREP版本可以根据您的处理需求进行定制。 看看用使用os.walk的Python循环替换FIND。这是一个巨大的胜利,因为您不需要生成那么多的进程。 看看用Python脚本替换常见的shell逻辑(循环、决策等)。
其他回答
我建议你看一本很棒的在线书籍Dive Into Python。这就是我最初学习这门语言的方式。
除了教你语言的基本结构和大量有用的数据结构之外,它还有一个关于文件处理的很好的章节,以及关于正则表达式等的后续章节。
任何shell都有几组特性。
The Essential Linux/Unix commands. All of these are available through the subprocess library. This isn't always the best first choice for doing all external commands. Look also at shutil for some commands that are separate Linux commands, but you could probably implement directly in your Python scripts. Another huge batch of Linux commands are in the os library; you can do these more simply in Python. And -- bonus! -- more quickly. Each separate Linux command in the shell (with a few exceptions) forks a subprocess. By using Python shutil and os modules, you don't fork a subprocess. The shell environment features. This includes stuff that sets a command's environment (current directory and environment variables and what-not). You can easily manage this from Python directly. The shell programming features. This is all the process status code checking, the various logic commands (if, while, for, etc.) the test command and all of it's relatives. The function definition stuff. This is all much, much easier in Python. This is one of the huge victories in getting rid of bash and doing it in Python. Interaction features. This includes command history and what-not. You don't need this for writing shell scripts. This is only for human interaction, and not for script-writing. The shell file management features. This includes redirection and pipelines. This is trickier. Much of this can be done with subprocess. But some things that are easy in the shell are unpleasant in Python. Specifically stuff like (a | b; c ) | something >result. This runs two processes in parallel (with output of a as input to b), followed by a third process. The output from that sequence is run in parallel with something and the output is collected into a file named result. That's just complex to express in any other language.
特定的程序(awk、sed、grep等)通常可以被重写为Python模块。不要走极端。替换您需要的内容并改进“grep”模块。不要一开始就编写一个替换“grep”的Python模块。
最好的事情是你可以一步一步来做。
用Python替换AWK和PERL。不要管其他的事情。 看看用Python替换GREP。这可能有点复杂,但是您的GREP版本可以根据您的处理需求进行定制。 看看用使用os.walk的Python循环替换FIND。这是一个巨大的胜利,因为您不需要生成那么多的进程。 看看用Python脚本替换常见的shell逻辑(循环、决策等)。
在ShellPy库中可以使用python而不是bash。
下面是一个从Github下载Python用户头像的例子:
import json
import os
import tempfile
# get the api answer with curl
answer = `curl https://api.github.com/users/python
# syntactic sugar for checking returncode of executed process for zero
if answer:
answer_json = json.loads(answer.stdout)
avatar_url = answer_json['avatar_url']
destination = os.path.join(tempfile.gettempdir(), 'python.png')
# execute curl once again, this time to get the image
result = `curl {avatar_url} > {destination}
if result:
# if there were no problems show the file
p`ls -l {destination}
else:
print('Failed to download avatar')
print('Avatar downloaded')
else:
print('Failed to access github api')
如您所见,所有在grave重音(')符号内的表达式都在shell中执行。在Python代码中,您可以捕获此执行的结果并对其执行操作。例如:
log = `git log --pretty=oneline --grep='Create'
这一行首先在shell中执行git log——pretty=oneline——grep='Create',然后将结果赋值给log变量。结果具有以下属性:
从已执行进程的Stdout中Stdout整个文本
Stderr从已执行进程的Stderr得到的整个文本
Returncode执行的返回码
这是该库的总体概述,更详细的描述和示例可以在这里找到。
我喜欢Python的一个原因是它比POSIX工具更加标准化。我必须反复检查每个比特是否与其他操作系统兼容。在Linux系统上编写的程序在OSX的BSD系统上可能无法正常工作。对于Python,我只需要检查目标系统是否有足够现代的Python版本。
更棒的是,用标准Python编写的程序甚至可以在Windows上运行!
我刚刚发现了如何结合bash和ipython的最佳部分。到目前为止,这似乎比使用subprocess等更舒服。你可以很容易地复制现有bash脚本的大部分内容,例如以python的方式添加错误处理:) 这是我的结果:
#!/usr/bin/env ipython3
# *** How to have the most comfort scripting experience of your life ***
# ######################################################################
#
# … by using ipython for scripting combined with subcommands from bash!
#
# 1. echo "#!/usr/bin/env ipython3" > scriptname.ipy # creates new ipy-file
#
# 2. chmod +x scriptname.ipy # make in executable
#
# 3. starting with line 2, write normal python or do some of
# the ! magic of ipython, so that you can use unix commands
# within python and even assign their output to a variable via
# var = !cmd1 | cmd2 | cmd3 # enjoy ;)
#
# 4. run via ./scriptname.ipy - if it fails with recognizing % and !
# but parses raw python fine, please check again for the .ipy suffix
# ugly example, please go and find more in the wild
files = !ls *.* | grep "y"
for file in files:
!echo $file | grep "p"
# sorry for this nonsense example ;)
请参阅IPython文档,了解系统shell命令并将其用作系统shell。