Handling Namespace Conflicts When Running Spark SQL Views in Parallel

Learn the differences between Spark SQL Temporary Views and Global Temporary Views, and how to avoid namespace conflicts during parallel execution.

2 min read

In Apache Spark’s Spark SQL, you can create “Views” as temporary tables. There are two types of views depending on their scope.

Types of Views in Spark SQL

  1. Temporary View:

    • Associated only with the SparkSession that created it.
    • The view’s namespace is limited to the internal scope of the SparkSession that created it.
    • When the SparkSession terminates, the temporary view is automatically destroyed.
    • Views created by default are temporary views.

  2. Global Temporary View:

    • Associated not with a single SparkSession, but with the SparkSession instance shared across the entire Spark application.
    • The view’s namespace is shared across all SparkSession instances within the Spark application.
    • It is not destroyed when a SparkSession terminates, and persists until the Spark application ends.
    • Created using the CREATE GLOBAL TEMPORARY VIEW syntax.

Namespace Conflicts During Parallel Execution

When executing Spark SQL queries in parallel, whether namespace conflicts become a problem depends on the type of view used.

  • Temporary Views: Temporary views are associated with their own independent SparkSession. In Spark’s parallel processing, if each task or thread has its own SparkSession, the temporary views created within each SparkSession have a namespace independent from views in other SparkSession instances. Therefore, namespace conflicts do not occur when running temporary views in parallel.

  • Global Temporary Views: Since global temporary views are shared across the entire Spark application, if multiple SparkSession instances try to create a global temporary view with the same name, a namespace conflict will occur.

Conclusion

In Spark SQL, unless you explicitly use the GLOBAL keyword, views created by default are temporary views. Since temporary views are scoped to each SparkSession, namespace conflicts during parallel execution generally do not occur.

If you need to share views across multiple SparkSession instances, you would use global temporary views, but in that case, you need to consider naming conventions and management strategies to avoid name conflicts.

関連記事