在Pandas with Python中读取CSV文件时出现UnicodeDecodeError

我正在运行一个程序，它正在处理3万个类似的文件。随机数量的它们停止并产生此错误…

  File "C:\Importer\src\dfman\importer.py", line 26, in import_chr
    data = pd.read_csv(filepath, names=fields)
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 400, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 205, in _read
    return parser.read()
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 608, in read
    ret = self._engine.read(nrows)
  File "C:\Python33\lib\site-packages\pandas\io\parsers.py", line 1028, in read
    data = self._reader.read(nrows)
  File "parser.pyx", line 706, in pandas.parser.TextReader.read (pandas\parser.c:6745)
  File "parser.pyx", line 728, in pandas.parser.TextReader._read_low_memory (pandas\parser.c:6964)
  File "parser.pyx", line 804, in pandas.parser.TextReader._read_rows (pandas\parser.c:7780)
  File "parser.pyx", line 890, in pandas.parser.TextReader._convert_column_data (pandas\parser.c:8793)
  File "parser.pyx", line 950, in pandas.parser.TextReader._convert_tokens (pandas\parser.c:9484)
  File "parser.pyx", line 1026, in pandas.parser.TextReader._convert_with_dtype (pandas\parser.c:10642)
  File "parser.pyx", line 1046, in pandas.parser.TextReader._string_convert (pandas\parser.c:10853)
  File "parser.pyx", line 1278, in pandas.parser._string_box_utf8 (pandas\parser.c:15657)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xda in position 6: invalid    continuation byte

这些文件的来源/创建都来自同一个地方。纠正这个问题以继续导入的最佳方法是什么?

当前回答

with open('filename.csv') as f:
   print(f)

执行这段代码后，你会发现'filename.csv'的编码，然后执行如下代码

data=pd.read_csv('filename.csv', encoding="encoding as you found earlier"

好了

2018-12-15 13:49:32

其他回答

我无法打开从网上银行下载的简体中文CSV文件。我试过latin1，我试过iso-8859-1，我试过cp1252，都没有用。

但pd。Read_csv (""，encoding ='gbk')只是做这个工作。

2019-11-03 06:01:07

你可以试试:

Df = pd.read_csv('./file_name.csv'， encoding='gbk')

2020-07-07 09:44:37

你总是可以尝试先检测文件的编码，使用chardet或cchardet或charset-normalizer:

from pathlib import Path
import chardet

filename = "file_name.csv"
detected = chardet.detect(Path(filename).read_bytes())
# detected is something like {'encoding': 'utf-8', 'confidence': 0.99, 'language': ''}

encoding = detected.get("encoding")
assert encoding, "Unable to detect encoding, is it a binary file?"

df = pd.read_csv(filename, encoding=encoding)

2022-03-02 06:50:38

在传递给熊猫之前检查编码。它会让你慢下来，但是……

with open(path, 'r') as f:
    encoding = f.encoding 

df = pd.read_csv(path,sep=sep, encoding=encoding)

在python 3.7中

2019-10-11 14:12:16

with open('filename.csv') as f:
   print(f)

执行这段代码后，你会发现'filename.csv'的编码，然后执行如下代码

data=pd.read_csv('filename.csv', encoding="encoding as you found earlier"

好了

2018-12-15 13:49:32

在Pandas with Python中读取CSV文件时出现UnicodeDecodeError

推荐文章

最新文章

标签