我的背景——在Hadoop世界工作了4周。使用Cloudera的Hadoop VM对Hive, Pig和Hadoop进行了一些尝试。已阅读谷歌关于Map-Reduce和GFS的论文(PDF链接)。
我明白——
猪的语言猪的拉丁语是一种转变
来自(适合程序员的思维方式)
SQL喜欢声明式的
编程和Hive的查询语言密切相关
类似于SQL。
Pig位于Hadoop之上
原则也可以凌驾于之上
德律阿得斯。我可能错了,但蜂巢错了
与Hadoop紧密耦合。
都是Pig Latin和Hive命令
编译映射和减少作业。
我的问题是——当一个(比如猪)可以达到目的时,拥有两者的目标是什么?难道只是因为雅虎宣传了Pig !和Facebook的Hive ?
Pig-latin is data flow style, is more suitable for software engineer. While sql is more suitable for analytics person who are get used to sql. For complex task, for hive you have to manually to create temporary table to store intermediate data, but it is not necessary for pig.
Pig-latin is suitable for complicated data structure( like small graph). There's a data structure in pig called DataBag which is a collection of Tuple. Sometimes you need to calculate metrics which involve multiple tuples ( there's a hidden link between tuples, in this case I would call it graph). In this case, it is very easy to write a UDF to calculate the metrics which involve multiple tuples. Of course it could be done in hive, but it is not so convenient as it is in pig.
Writing UDF in pig much is easier than in Hive in my opinion.
Pig has no metadata support, (or it is optional, in future it may integrate hcatalog). Hive has tables' metadata stored in database.
You can debug pig script in local environment, but it would be hard for hive to do that. The reason is point 3. You need to set up hive metadata in your local environment, very time consuming.
Pig-latin is data flow style, is more suitable for software engineer. While sql is more suitable for analytics person who are get used to sql. For complex task, for hive you have to manually to create temporary table to store intermediate data, but it is not necessary for pig.
Pig-latin is suitable for complicated data structure( like small graph). There's a data structure in pig called DataBag which is a collection of Tuple. Sometimes you need to calculate metrics which involve multiple tuples ( there's a hidden link between tuples, in this case I would call it graph). In this case, it is very easy to write a UDF to calculate the metrics which involve multiple tuples. Of course it could be done in hive, but it is not so convenient as it is in pig.
Writing UDF in pig much is easier than in Hive in my opinion.
Pig has no metadata support, (or it is optional, in future it may integrate hcatalog). Hive has tables' metadata stored in database.
You can debug pig script in local environment, but it would be hard for hive to do that. The reason is point 3. You need to set up hive metadata in your local environment, very time consuming.
Pig允许在管道中的任何位置加载数据和用户代码。如果数据是流数据,例如来自卫星或仪器的数据,这一点可能特别重要。
Hive是基于RDBMS的,它需要首先导入(或加载)数据,然后才能对其进行处理。因此,如果您在流数据上使用Hive,您将不得不不断填充桶(或文件),并在每个填充桶上使用Hive,同时使用其他桶来继续存储新到达的数据。
Pig也使用惰性求值。它使编程变得更加容易,人们可以用它来以不同的方式分析数据,比在像Hive这样的SQL类语言中更自由。因此,如果你真的想分析一些你拥有的非结构化数据中的矩阵或模式,并想对它们进行有趣的计算,使用Pig你可以走得很远,而使用Hive,你需要其他东西来处理结果。
Pig在数据导入方面更快,但在实际执行方面比像Hive这样的RDBMS友好语言要慢。
Pig非常适合并行化,因此它可能在数据集庞大的系统中具有优势,即在您更关心结果吞吐量而不是延迟(获得任何特定结果数据的时间)的系统中。
看看“dezyre”文章中关于猪和蜂巢的坚果壳比较
Hive在分区、服务器、Web接口和JDBC/ODBC支持方面优于PIG。
一些差异:
Hive is best for structured Data & PIG is best for semi structured data
Hive is used for reporting & PIG for programming
Hive is used as a declarative SQL & PIG as a procedural language
Hive supports partitions & PIG does not
Hive can start an optional thrift based server & PIG cannot
Hive defines tables beforehand (schema) + stores schema information in a database & PIG doesn't have a dedicated metadata of database
Hive does not support Avro but PIG does. EDIT: Hive supports Avro, specify the serde as org.apache.hadoop.hive.serde2.avro
Pig also supports additional COGROUP feature for performing outer joins but hive does not. But both Hive & PIG can join, order & sort dynamically.