- 我的理解——节点是实际的物理机器。一个节点可以包含主驱动程序，而其他节点将包含工作程序。
(This is Correct as a starting Point)
Q - 一个节点可以有多个驱动程序（如果我有多个应用程序）？
Yes Because driver is just a process that gets created based on the program that you might have written. And you can have multiple process running on the same node.
- 我的理解 - 工作人员是节点内的一个进程。尽管不推荐，但每个节点内可以有多个工作器。
your understanding here seems wrong because worker is actually a node or machine. Either you say it worker or worker node both are same
- 我的理解 - 执行者是工作进程中的子进程（？）。每个工人可以有多个执行者。
An executor is a process inside the worker node and a single worker node can have multiple executors
问。什么指标决定了每个工作人员的执行者数量？
configuration(Number of cores and memory) of your worker node decides what is the max executors it can run on any specific worker node.
问。 JVM 的想法是与执行程序进程相关联还是在更高的“工作者”级别？
It is associated with the executor process. Spark executor is a single JVM instance on a node that serves a single spark application
问。 core和executor是什么关系？
Core property controls the number of concurrent tasks an executor can run. For example if you request 2 executor each with 2 cores then you can run 4 concurrent tasks at the same time during your job execution.
问 - RAM 和 HD 可以在执行程序级别分配吗？
例如，如果我有一个具有 100GB RAM 和 5 TB HD 的工作节点，我可以为每个执行程序分配 20 GB RAM 和 1 TB HDD 吗？
Generally spark perform all its computation in memory. RAM is allocated at the executor level and HD would be allocated at the Worker node level only. Spark would just spill the data to the disk only when it does not fit in memory
我的理解 - 分区是实际数据的一部分。这种拆分可以使用散列、循环或范围进行。
Q - 是什么决定了这些数据分区的位置？
These partitions could be anywhere and might not be equally distributed in most of the cases.It could happen some of the executors does not have a single partition and other executors have more than 2 partitions.
In order to have colocated partitions or partitions that have same keys you would have to repartition data based on the specific column in your dataframe and then it would partition your data based on the values of that column and make sure that same column values are there in the same partition
When you repartition the data to 2 partitions then it would shuffle the data between all the executors and then break the dat into 2 partitions and then that data could be on any of the executors and other executors would be empty or idle in that case.
- 假设 - 任务是执行实际询问的最低工作单元。任务的数量取决于分区的数量。因此，如果有 20 个分区，我将在每个阶段有 20 个任务。
you would have 20 tasks for that specific stage and it wont remain same for all the stages as stage gets created when there is data shuffle that needs to happen. If there is no shuffle happening based on the code that you might have written it would just create a single stage with 20 tasks for sure.
问 - 这些任务是由个人执行者执行的吗？ Yes
Q - 如果我的执行程序（例如 10 个）少于分区（例如 20 个），这是否意味着在任何时候只有 10 个任务会并行执行？并行度是否受执行者数量的限制？ Yes