我收到了一些编码的文本,但我不知道使用的是什么字符集。是否有一种方法可以使用Python确定文本文件的编码?如何检测文本文件的编码/代码页处理c#。
当前回答
你可以使用' python-magic package,它不会将整个文件加载到内存中:
import magic
def detect(
file_path,
):
return magic.Magic(
mime_encoding=True,
).from_file(file_path)
输出是编码名称,例如:
iso - 8859 - 1 us - ascii utf - 8
其他回答
# Function: OpenRead(file)
# A text file can be encoded using:
# (1) The default operating system code page, Or
# (2) utf8 with a BOM header
#
# If a text file is encoded with utf8, and does not have a BOM header,
# the user can manually add a BOM header to the text file
# using a text editor such as notepad++, and rerun the python script,
# otherwise the file is read as a codepage file with the
# invalid codepage characters removed
import sys
if int(sys.version[0]) != 3:
print('Aborted: Python 3.x required')
sys.exit(1)
def bomType(file):
"""
returns file encoding string for open() function
EXAMPLE:
bom = bomtype(file)
open(file, encoding=bom, errors='ignore')
"""
f = open(file, 'rb')
b = f.read(4)
f.close()
if (b[0:3] == b'\xef\xbb\xbf'):
return "utf8"
# Python automatically detects endianess if utf-16 bom is present
# write endianess generally determined by endianess of CPU
if ((b[0:2] == b'\xfe\xff') or (b[0:2] == b'\xff\xfe')):
return "utf16"
if ((b[0:5] == b'\xfe\xff\x00\x00')
or (b[0:5] == b'\x00\x00\xff\xfe')):
return "utf32"
# If BOM is not provided, then assume its the codepage
# used by your operating system
return "cp1252"
# For the United States its: cp1252
def OpenRead(file):
bom = bomType(file)
return open(file, 'r', encoding=bom, errors='ignore')
#######################
# Testing it
#######################
fout = open("myfile1.txt", "w", encoding="cp1252")
fout.write("* hi there (cp1252)")
fout.close()
fout = open("myfile2.txt", "w", encoding="utf8")
fout.write("\u2022 hi there (utf8)")
fout.close()
# this case is still treated like codepage cp1252
# (User responsible for making sure that all utf8 files
# have a BOM header)
fout = open("badboy.txt", "wb")
fout.write(b"hi there. barf(\x81\x8D\x90\x9D)")
fout.close()
# Read Example file with Bom Detection
fin = OpenRead("myfile1.txt")
L = fin.readline()
print(L)
fin.close()
# Read Example file with Bom Detection
fin = OpenRead("myfile2.txt")
L =fin.readline()
print(L) #requires QtConsole to view, Cmd.exe is cp1252
fin.close()
# Read CP1252 with a few undefined chars without barfing
fin = OpenRead("badboy.txt")
L =fin.readline()
print(L)
fin.close()
# Check that bad characters are still in badboy codepage file
fin = open("badboy.txt", "rb")
fin.read(20)
fin.close()
一些文本文件知道它们的编码,大多数则不是。意识到:
具有BOM的文本文件 XML文件以UTF-8编码或其编码在序言中给出 JSON文件总是用UTF-8编码
没有意识到:
CSV文件 任意文本文件
有些编码是通用的,即它们可以解码任何字节序列,有些则不是。US-ASCII不是万能的,因为任何大于127的字节都不能映射到任何字符。UTF-8不是万能的,因为任何字节序列都是无效的。
相反,Latin-1, Windows-1252等是通用的(即使一些字节没有正式映射到一个字符):
>>> [b.to_bytes(1, 'big').decode("latin-1") for b in range(256)]
['\x00', ..., 'ÿ']
给定一个以字节序列编码的随机文本文件,除非该文件知道其编码,否则无法确定其编码,因为有些编码是通用的。但有时可以排除非通用编码。所有通用编码仍然是可能的。chardet模块使用字节的频率来猜测哪种编码最适合已编码的文本。
如果你不想使用这个模块或类似的模块,这里有一个简单的方法:
检查文件是否知道其编码(BOM) 检查非通用编码并接受第一个可以解码字节的编码(ASCII在UTF-8之前,因为它更严格) 选择一个回退编码。
如果您只检查一个示例,那么第二步有点风险,因为文件其余部分中的某些字节可能是无效的。
代码:
def guess_encoding(data: bytes, fallback: str = "iso8859_15") -> str:
"""
A basic encoding detector.
"""
for bom, encoding in [
(codecs.BOM_UTF32_BE, "utf_32_be"),
(codecs.BOM_UTF32_LE, "utf_32_le"),
(codecs.BOM_UTF16_BE, "utf_16_be"),
(codecs.BOM_UTF16_LE, "utf_16_le"),
(codecs.BOM_UTF8, "utf_8_sig"),
]:
if data.startswith(bom):
return encoding
if all(b < 128 for b in data):
return "ascii" # you may want to use the fallback here if data is only a sample.
decoder = codecs.getincrementaldecoder("utf_8")()
try:
decoder.decode(data, final=False)
except UnicodeDecodeError:
return fallback
else:
return "utf_8" # not certain if data is only a sample
记住,非通用编码可能会失败。decode方法的errors参数可以设置为'ignore', 'replace'或'backslashreplace'以避免异常。
如果你知道文件的一些内容,你可以尝试用几种编码来解码它,看看哪个丢失了。一般来说没有办法,因为文本文件就是文本文件,这些都是愚蠢的;)
你可以使用chardet模块
import chardet
with open (filepath , "rb") as f:
data= f.read()
encode=chardet.UniversalDetector()
encode.close()
print(encode.result)
或者你可以在linux中使用chardet3命令,但这需要一些时间:
chardet3 fileName
例子:
chardet3 donnee/dir/donnee.csv
donnee/dir/donnee.csv: ISO-8859-1 with confidence 0.73
这可能会有帮助
from bs4 import UnicodeDammit
with open('automate_data/billboard.csv', 'rb') as file:
content = file.read()
suggestion = UnicodeDammit(content)
suggestion.original_encoding
#'iso-8859-1'