根据我对上面提供的代码的理解，它加载文件并进行映射操作并保存回来。没有需要shuffle的操作。此外，没有任何操作需要将数据传输到驱动程序，因此调优与shuffle或驱动程序相关的任何内容都不会产生影响。当任务太多时，驱动程序确实会有问题，但这只是在spark 2.0.2版本之前。可能会有两件事出错。

There are only one or a few executors. Increase the number of executors so that they can be allocated to different slaves. If you are using yarn need to change num-executors config or if you are using spark standalone then need to tune num cores per executor and spark max cores conf. In standalone num executors = max cores / cores per executor . The number of partitions are very few or maybe only one. So if this is low even if we have multi-cores,multi executors it will not be of much help as parallelization is dependent on the number of partitions. So increase the partitions by doing imageBundleRDD.repartition(11)

广义上讲，spark Executor JVM内存可以分为两部分。Spark内存和User内存。这是由spark.memory.fraction属性控制的——值在0到1之间。 在spark应用程序中处理图像或执行内存密集型处理时，请考虑降低spark.memory.fraction。这将为应用程序工作提供更多内存。Spark可能溢出，所以它仍然可以在较少的内存共享下工作。

The second part of the problem is division of work. If possible, partition your data into smaller chunks. Smaller data possibly needs less memory. But if that is not possible, you are sacrifice compute for memory. Typically a single executor will be running multiple cores. Total memory of executors must be enough to handle memory requirements of all concurrent tasks. If increasing executor memory is not a option, you can decrease the cores per executor so that each task gets more memory to work with. Test with 1 core executors which have largest possible memory you can give and then keep increasing cores until you find the best core count.

设置内存堆大小的位置(至少在spark-1.0.0中)在conf/spark-env中。 相关变量为SPARK_EXECUTOR_MEMORY和SPARK_DRIVER_MEMORY。 部署指南中有更多的文档

此外，不要忘记将配置文件复制到所有从节点。

你应该配置offHeap内存设置如下所示:

val spark = SparkSession
     .builder()
     .master("local[*]")
     .config("spark.executor.memory", "70g")
     .config("spark.driver.memory", "50g")
     .config("spark.memory.offHeap.enabled",true)
     .config("spark.memory.offHeap.size","16g")   
     .appName("sampleCodeForReference")
     .getOrCreate()

根据您机器的RAM可用性提供驱动程序内存和执行程序内存。如果仍然面临OutofMemory问题，可以增加offHeap大小。

在使用动态资源分配时，我经常遇到这个问题。我原以为它会利用我的集群资源来最适合这个应用程序。

但事实上，动态资源分配并没有设置驱动程序内存，而是将其保持为默认值，即1G。

我通过将spark.driver.memory设置为适合我的驱动器内存的数字来解决这个问题(对于32GB ram，我将其设置为18G)。

可以使用spark submit命令进行设置，方法如下:

spark-submit --conf spark.driver.memory=18g

非常重要的一点是，如果你从代码中设置这个属性，将不会被考虑，根据Spark文档-动态加载Spark属性:

Spark properties mainly can be divided into two kinds: one is related to deploy, like “spark.driver.memory”, “spark.executor.instances”, this kind of properties may not be affected when setting programmatically through SparkConf in runtime, or the behavior is depending on which cluster manager and deploy mode you choose, so it would be suggested to set through configuration file or spark-submit command line options; another is mainly related to Spark runtime control, like “spark.task.maxFailures”, this kind of properties can be set in either way.

为了添加一个通常不被讨论的用例，我将在本地模式下通过Spark -submit提交Spark应用程序时提出一个解决方案。

根据Jacek Laskowski的giitbook Mastering Apache Spark:

您可以在本地模式下运行Spark。在这种非分布式单JVM部署模式下，Spark在同一个JVM中生成所有执行组件——驱动程序、执行程序、后端和主机。这是驱动程序用于执行的唯一模式。

因此，如果您在堆中遇到OOM错误，调整驱动程序内存而不是执行程序内存就足够了。

这里有一个例子:

spark-1.6.1/bin/spark-submit
  --class "MyClass"
  --driver-memory 12g
  --master local[*] 
  target/scala-2.10/simple-project_2.10-1.0.jar