我需要在c++中加载和使用CSV文件数据。在这一点上,它实际上只是一个以逗号分隔的解析器(即不用担心转义新行和逗号)。主要需要的是逐行解析器,它将在每次调用方法时为下一行返回一个向量。
我发现这篇文章看起来很有前途: http://www.boost.org/doc/libs/1_35_0/libs/spirit/example/fundamental/list_parser.cpp
我从未使用过Boost's Spirit,但我愿意尝试一下。但前提是我忽略了一个更直接的解决方案。
我需要在c++中加载和使用CSV文件数据。在这一点上,它实际上只是一个以逗号分隔的解析器(即不用担心转义新行和逗号)。主要需要的是逐行解析器,它将在每次调用方法时为下一行返回一个向量。
我发现这篇文章看起来很有前途: http://www.boost.org/doc/libs/1_35_0/libs/spirit/example/fundamental/list_parser.cpp
我从未使用过Boost's Spirit,但我愿意尝试一下。但前提是我忽略了一个更直接的解决方案。
当前回答
另一个类似于Loki Astari的答案的解决方案,在c++ 11中。这里的行是给定类型的std::元组。代码扫描一行,然后扫描到每个分隔符,然后将值直接转换并转储到元组中(使用一些模板代码)。
for (auto row : csv<std::string, int, float>(file, ',')) {
std::cout << "first col: " << std::get<0>(row) << std::endl;
}
优势:
非常干净,使用简单,只有c++ 11。 自动类型转换为std::tuple<t1,…>通过算子>>。
缺少什么:
转义和引用 没有错误处理的情况下畸形的CSV。
主要代码:
#include <iterator>
#include <sstream>
#include <string>
namespace csvtools {
/// Read the last element of the tuple without calling recursively
template <std::size_t idx, class... fields>
typename std::enable_if<idx >= std::tuple_size<std::tuple<fields...>>::value - 1>::type
read_tuple(std::istream &in, std::tuple<fields...> &out, const char delimiter) {
std::string cell;
std::getline(in, cell, delimiter);
std::stringstream cell_stream(cell);
cell_stream >> std::get<idx>(out);
}
/// Read the @p idx-th element of the tuple and then calls itself with @p idx + 1 to
/// read the next element of the tuple. Automatically falls in the previous case when
/// reaches the last element of the tuple thanks to enable_if
template <std::size_t idx, class... fields>
typename std::enable_if<idx < std::tuple_size<std::tuple<fields...>>::value - 1>::type
read_tuple(std::istream &in, std::tuple<fields...> &out, const char delimiter) {
std::string cell;
std::getline(in, cell, delimiter);
std::stringstream cell_stream(cell);
cell_stream >> std::get<idx>(out);
read_tuple<idx + 1, fields...>(in, out, delimiter);
}
}
/// Iterable csv wrapper around a stream. @p fields the list of types that form up a row.
template <class... fields>
class csv {
std::istream &_in;
const char _delim;
public:
typedef std::tuple<fields...> value_type;
class iterator;
/// Construct from a stream.
inline csv(std::istream &in, const char delim) : _in(in), _delim(delim) {}
/// Status of the underlying stream
/// @{
inline bool good() const {
return _in.good();
}
inline const std::istream &underlying_stream() const {
return _in;
}
/// @}
inline iterator begin();
inline iterator end();
private:
/// Reads a line into a stringstream, and then reads the line into a tuple, that is returned
inline value_type read_row() {
std::string line;
std::getline(_in, line);
std::stringstream line_stream(line);
std::tuple<fields...> retval;
csvtools::read_tuple<0, fields...>(line_stream, retval, _delim);
return retval;
}
};
/// Iterator; just calls recursively @ref csv::read_row and stores the result.
template <class... fields>
class csv<fields...>::iterator {
csv::value_type _row;
csv *_parent;
public:
typedef std::input_iterator_tag iterator_category;
typedef csv::value_type value_type;
typedef std::size_t difference_type;
typedef csv::value_type * pointer;
typedef csv::value_type & reference;
/// Construct an empty/end iterator
inline iterator() : _parent(nullptr) {}
/// Construct an iterator at the beginning of the @p parent csv object.
inline iterator(csv &parent) : _parent(parent.good() ? &parent : nullptr) {
++(*this);
}
/// Read one row, if possible. Set to end if parent is not good anymore.
inline iterator &operator++() {
if (_parent != nullptr) {
_row = _parent->read_row();
if (!_parent->good()) {
_parent = nullptr;
}
}
return *this;
}
inline iterator operator++(int) {
iterator copy = *this;
++(*this);
return copy;
}
inline csv::value_type const &operator*() const {
return _row;
}
inline csv::value_type const *operator->() const {
return &_row;
}
bool operator==(iterator const &other) {
return (this == &other) or (_parent == nullptr and other._parent == nullptr);
}
bool operator!=(iterator const &other) {
return not (*this == other);
}
};
template <class... fields>
typename csv<fields...>::iterator csv<fields...>::begin() {
return iterator(*this);
}
template <class... fields>
typename csv<fields...>::iterator csv<fields...>::end() {
return iterator();
}
我在GitHub上放了一个小的工作示例;我一直用它来解析一些数值数据,它达到了它的目的。
其他回答
当对CSV文件使用Boost Tokenizer escaped_list_separator时,应该注意以下几点:
它需要一个转义字符(默认的反斜杠- \) 它需要一个分割符/分隔符-字符(默认逗号-,) 它需要一个引号字符(默认的引号- ")
wiki指定的CSV格式规定数据字段可以包含引号分隔符(支持):
1997年,福特E350,“超级豪华卡车”
由wiki指定的CSV格式规定单引号应该用双引号处理(escaped_list_separator将剥离所有引号字符):
1997年,福特E350,“超级”“豪华”“卡车”
CSV格式没有指定应该删除任何反斜杠字符(escaped_list_separator将删除所有转义字符)。
修复boost escaped_list_separator的默认行为的一个可能的变通方法:
首先将所有反斜杠字符(\)替换为两个反斜杠字符(\\),这样它们就不会被剥离。 其次,将所有双引号("")替换为一个反斜杠字符和一个引号(\")
这种变通方法有一个副作用,即由双引号表示的空数据字段将被转换为单引号标记。在遍历令牌时,必须检查令牌是否是单引号,并将其视为空字符串。
不漂亮,但它工作,只要在引号中没有换行。
如果您所需要的只是加载一个双精度数据文件(没有整数,没有文本),那么这里有一个随时可用的函数。
#include <sstream>
#include <fstream>
#include <iterator>
#include <string>
#include <vector>
#include <algorithm>
using namespace std;
/**
* Parse a CSV data file and fill the 2d STL vector "data".
* Limits: only "pure datas" of doubles, not encapsulated by " and without \n inside.
* Further no formatting in the data (e.g. scientific notation)
* It however handles both dots and commas as decimal separators and removes thousand separator.
*
* returnCodes[0]: file access 0-> ok 1-> not able to read; 2-> decimal separator equal to comma separator
* returnCodes[1]: number of records
* returnCodes[2]: number of fields. -1 If rows have different field size
*
*/
vector<int>
readCsvData (vector <vector <double>>& data, const string& filename, const string& delimiter, const string& decseparator){
int vv[3] = { 0,0,0 };
vector<int> returnCodes(&vv[0], &vv[0]+3);
string rowstring, stringtoken;
double doubletoken;
int rowcount=0;
int fieldcount=0;
data.clear();
ifstream iFile(filename, ios_base::in);
if (!iFile.is_open()){
returnCodes[0] = 1;
return returnCodes;
}
while (getline(iFile, rowstring)) {
if (rowstring=="") continue; // empty line
rowcount ++; //let's start with 1
if(delimiter == decseparator){
returnCodes[0] = 2;
return returnCodes;
}
if(decseparator != "."){
// remove dots (used as thousand separators)
string::iterator end_pos = remove(rowstring.begin(), rowstring.end(), '.');
rowstring.erase(end_pos, rowstring.end());
// replace decimal separator with dots.
replace(rowstring.begin(), rowstring.end(),decseparator.c_str()[0], '.');
} else {
// remove commas (used as thousand separators)
string::iterator end_pos = remove(rowstring.begin(), rowstring.end(), ',');
rowstring.erase(end_pos, rowstring.end());
}
// tokenize..
vector<double> tokens;
// Skip delimiters at beginning.
string::size_type lastPos = rowstring.find_first_not_of(delimiter, 0);
// Find first "non-delimiter".
string::size_type pos = rowstring.find_first_of(delimiter, lastPos);
while (string::npos != pos || string::npos != lastPos){
// Found a token, convert it to double add it to the vector.
stringtoken = rowstring.substr(lastPos, pos - lastPos);
if (stringtoken == "") {
tokens.push_back(0.0);
} else {
istringstream totalSString(stringtoken);
totalSString >> doubletoken;
tokens.push_back(doubletoken);
}
// Skip delimiters. Note the "not_of"
lastPos = rowstring.find_first_not_of(delimiter, pos);
// Find next "non-delimiter"
pos = rowstring.find_first_of(delimiter, lastPos);
}
if(rowcount == 1){
fieldcount = tokens.size();
returnCodes[2] = tokens.size();
} else {
if ( tokens.size() != fieldcount){
returnCodes[2] = -1;
}
}
data.push_back(tokens);
}
iFile.close();
returnCodes[1] = rowcount;
return returnCodes;
}
如果可以的话,这是我简单快速的贡献。 没有提高。
接受分隔符和分隔符中的分隔符,只要成对或远离分隔符即可。
#include <iostream>
#include <vector>
#include <fstream>
std::vector<std::string> SplitCSV(const std::string &data, char separator, char delimiter)
{
std::vector<std::string> Values;
std::string Val = "";
bool VDel = false; // Is within delimiter?
size_t CDel = 0; // Delimiters counter within delimiters.
const char *C = data.c_str();
size_t P = 0;
do
{
if ((Val.length() == 0) && (C[P] == delimiter))
{
VDel = !VDel;
CDel = 0;
P++;
continue;
}
if (VDel)
{
if (C[P] == delimiter)
{
if (((CDel % 2) == 0) && ( (C[P+1] == separator) || (C[P+1] == 0) || (C[P+1] == '\n') || (C[P+1] == '\r') ))
{
VDel = false;
CDel = 0;
P++;
continue;
}
else
CDel++;
}
}
else
{
if (C[P] == separator)
{
Values.push_back(Val);
Val = "";
P++;
continue;
}
if ((C[P] == 0) || (C[P] == '\n') || (C[P] == '\r'))
break;
}
Val += C[P];
P++;
} while(P < data.length());
Values.push_back(Val);
return Values;
}
bool ReadCsv(const std::string &fname, std::vector<std::vector<std::string>> &data,
char separator = ',', char delimiter = '\"')
{
bool Ret = false;
std::ifstream FCsv(fname);
if (FCsv)
{
FCsv.seekg(0, FCsv.end);
size_t Siz = FCsv.tellg();
if (Siz > 0)
{
FCsv.seekg(0);
data.clear();
std::string Line;
while (getline(FCsv, Line, '\n'))
data.push_back(SplitCSV(Line, separator, delimiter));
Ret = true;
}
FCsv.close();
}
return Ret;
}
int main(int argc, char *argv[])
{
std::vector<std::vector<std::string>> Data;
ReadCsv("fsample.csv", Data);
return 0;
}
可以使用std::regex。
根据文件大小和可用内存,可以逐行读取,也可以完全在std::string中读取。
读取文件可以使用:
std::ifstream t("file.txt");
std::string sin((std::istreambuf_iterator<char>(t)),
std::istreambuf_iterator<char>());
然后你可以和这个相匹配,它实际上是根据你的需要定制的。
std::regex word_regex(",\\s]+");
auto what =
std::sregex_iterator(sin.begin(), sin.end(), word_regex);
auto wend = std::sregex_iterator();
std::vector<std::string> v;
for (;what!=wend ; wend) {
std::smatch match = *what;
v.push_back(match.str());
}
另一种快速简单的方法是使用Boost。I / O:融合
#include <iostream>
#include <sstream>
#include <boost/fusion/adapted/boost_tuple.hpp>
#include <boost/fusion/sequence/io.hpp>
namespace fusion = boost::fusion;
struct CsvString
{
std::string value;
// Stop reading a string once a CSV delimeter is encountered.
friend std::istream& operator>>(std::istream& s, CsvString& v) {
v.value.clear();
for(;;) {
auto c = s.peek();
if(std::istream::traits_type::eof() == c || ',' == c || '\n' == c)
break;
v.value.push_back(c);
s.get();
}
return s;
}
friend std::ostream& operator<<(std::ostream& s, CsvString const& v) {
return s << v.value;
}
};
int main() {
std::stringstream input("abc,123,true,3.14\n"
"def,456,false,2.718\n");
typedef boost::tuple<CsvString, int, bool, double> CsvRow;
using fusion::operator<<;
std::cout << std::boolalpha;
using fusion::operator>>;
input >> std::boolalpha;
input >> fusion::tuple_open("") >> fusion::tuple_close("\n") >> fusion::tuple_delimiter(',');
for(CsvRow row; input >> row;)
std::cout << row << '\n';
}
输出:
(abc 123 true 3.14)
(def 456 false 2.718)