我正在运行一个程序,它正在处理3万个类似的文件。随机数量的它们停止并产生此错误…

  File "C:\Importer\src\dfman\importer.py", line 26, in import_chr
    data = pd.read_csv(filepath, names=fields)
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 400, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 205, in _read
    return parser.read()
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 608, in read
    ret = self._engine.read(nrows)
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 1028, in read
    data = self._reader.read(nrows)
  File "parser.pyx", line 706, in pandas.parser.TextReader.read (pandas\parser.c:6745)
  File "parser.pyx", line 728, in pandas.parser.TextReader._read_low_memory (pandas\parser.c:6964)
  File "parser.pyx", line 804, in pandas.parser.TextReader._read_rows (pandas\parser.c:7780)
  File "parser.pyx", line 890, in pandas.parser.TextReader._convert_column_data (pandas\parser.c:8793)
  File "parser.pyx", line 950, in pandas.parser.TextReader._convert_tokens (pandas\parser.c:9484)
  File "parser.pyx", line 1026, in pandas.parser.TextReader._convert_with_dtype (pandas\parser.c:10642)
  File "parser.pyx", line 1046, in pandas.parser.TextReader._string_convert (pandas\parser.c:10853)
  File "parser.pyx", line 1278, in pandas.parser._string_box_utf8 (pandas\parser.c:15657)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 6: invalid    continuation byte

这些文件的来源/创建都来自同一个地方。纠正这个问题以继续导入的最佳方法是什么?


当前回答

尝试改变编码。 在我的例子中,encoding = "utf-16"起作用了。

df = pd.read_csv(“file.csv”,encoding='utf-16')

其他回答

我无法打开从网上银行下载的简体中文CSV文件。 我试过latin1,我试过iso-8859-1,我试过cp1252,都没有用。

但pd。Read_csv ("",encoding ='gbk')只是做这个工作。

在我的例子中,一个文件具有USC-2 LE BOM编码,根据notepad++。 对于python,它是encoding="utf_16_le"。

希望,这能帮助别人更快地找到答案。

Pandas允许指定编码,但不允许忽略错误,不允许自动替换违规字节。因此,没有一种适合所有情况的方法,而是根据实际用例使用不同的方法。

You know the encoding, and there is no encoding error in the file. Great: you have just to specify the encoding: file_encoding = 'cp1252' # set file_encoding to the file encoding (utf8, latin1, etc.) pd.read_csv(input_file_and_path, ..., encoding=file_encoding) You do not want to be bothered with encoding questions, and only want that damn file to load, no matter if some text fields contain garbage. Ok, you only have to use Latin1 encoding because it accept any possible byte as input (and convert it to the unicode character of same code): pd.read_csv(input_file_and_path, ..., encoding='latin1') You know that most of the file is written with a specific encoding, but it also contains encoding errors. A real world example is an UTF8 file that has been edited with a non utf8 editor and which contains some lines with a different encoding. Pandas has no provision for a special error processing, but Python open function has (assuming Python3), and read_csv accepts a file like object. Typical errors parameter to use here are 'ignore' which just suppresses the offending bytes or (IMHO better) 'backslashreplace' which replaces the offending bytes by their Python’s backslashed escape sequence: file_encoding = 'utf8' # set file_encoding to the file encoding (utf8, latin1, etc.) input_fd = open(input_file_and_path, encoding=file_encoding, errors = 'backslashreplace') pd.read_csv(input_fd, ...)

这个问题困扰了我一段时间,我想我应该发布这个问题,因为它是第一个搜索结果。将encoding="iso-8859-1"标签添加到pandas read_csv中不起作用,任何其他编码也不起作用,一直给出UnicodeDecodeError。

如果将文件句柄传递给pd.read_csv(),则需要将encoding属性放在打开的文件上,而不是在read_csv中。事后看来很明显,但这是一个需要追查的微妙错误。

在我的案例中,我无法使用之前提供的任何方法来克服这个问题。将编码器类型更改为utf-8、utf-16、iso-8859-1或任何其他类型都不工作。

但是不用pd。read_csv(文件名,分隔符=';'),我用的是;

pd。Read_csv (open(filename, 'r'), delimiter=';')

一切似乎都很顺利。