This article contains the release notes for the Snowflake Connector for Spark, including the following when applicable:
- Behavior changes
- New features
- Customer-facing bug fixes
Note: Beginning with the January 2022 release, all release note information for this connector is published on this page. For release note information for previous versions, see the
Server Release Notes.
Note: As of September 2023, Snowpark Connector for Spark release notes are published on the Snowflake documentation site and will no longer appear on this page. Please update any links you might have saved.
Version 2.12.0 (May 23, 2023)
Note: Starting with this version (2.12.0), the Snowflake Connector for Spark no longer supports Spark 3.1, but continues to support versions 3.2, 3.3, and 3.4. Previous versions of the connector continue to support Spark 3.1.
New Features
- Added support for Spark 3.4.
Bug Fixes
Version 2.11.3 (April 21, 2023)
New Features
- Updated the mechanism for writing DataFrames to accounts on GCP. After December 2023, previous versions of the Spark Connector will no longer be able to write DataFrames, due to changes in GCP.
- Added the option to disable preactions and postactions validation for session sharing.
To disable validation, set the option FORCE_SKIP_PRE_POST_ACTION_CHECK_FOR_SHARED_SESSION to true. The default is false.
Important: Before setting this option, make sure that the queries in preactions and postactions don't affect the session settings. Otherwise, you might encounter issues with results.
Bug Fixes
- Fixed an issue when performing a join or union across different schemas when the two DataFrames are accessing tables with different sfSchema and the same name table in sfSchema is in the left DataFrame.
Version 2.11.2 (March 21, 2023)
New Features
- Added support for sharing a JDBC connection.
The Snowflake Connector for Spark can now use the same JDBC connection for different jobs and actions when the client uses the same connection options to access Snowflake. Previously, the Spark Connector created a new JDBC connection for each job or action.
The Spark Connector supports the following options and API methods for enabling and disabling this feature:
- To specify that the connector should not use the same JDBC connection, set the support_share_connection connector option to false. (The default value is true, which means that the feature is enabled.)
- To enable or disable the feature programmatically, call one of the following global static functions: SparkConnectorContext.disableSharedConnection() and SparkConnectorContext.enableSharingJDBCConnection().
Note: In the following special cases, the Spark Connector will not use the shared connection:
- If preactions or postactions are set, and those preactions or postactions are not CREATE TABLE, DROP TABLE, or MERGE INTO, the Spark Connector will not use the shared connection.
- Utility functions in Utils, such as Utils.runQuery() and Utils.getJDBCConnection(), will not use the shared connection.
- Updated the connector to use the Snowflake JDBC driver 3.13.29.
Bug Fixes
Version 2.11.1 (December 13, 2022)
New Features
- Added support for AWS VPCE deployments by adding the S3_STAGE_VPCE_DNS_NAME configuration parameter to specifying the VPCE DNS name at the session level.
- Added a new configuration option treat_decimal_as_long to enable the Spark Connector to return Long values instead of BigDecimal values, if the query returns Decimal(<any_precision>, 0). WARNING: If the value is greater than the maximum value of Long, an error will be raised.
- Added a new option proxy_protocol for specifying the proxy protocol (http or https) with AWS deployments. (The option has no effect on Azure and GCP deployments.).
- Added support for counting rows in a table where the row count is greater than the maximum value of Integer.
- Updated the connector to use the Snowflake JDBC driver 3.13.24.
Bug Fixes
- Updated the connector to close JDBC connections to avoid connection leakage.
- Fixed a NullPointerException issue when sending telemetry messages.
Version 2.11.0 (September 2, 2022)
Compatible JDBC Driver version: 3.13.22
Added support for Spark 3.3 and fixed some bugs:
- Upgraded the version of the PostgreSQL JDBC Driver that tests depend on to avoid the security vulnerability CVE-2022-31197.
- Updated the connector to use the Snowflake JDBC driver 3.13.22 and the Snowflake Ingest SDK 0.10.8.
NOTE:
Version 2.10.1 (August 15, 2022)
Compatible JDBC Driver version: 3.13.14
Bug Fixes
- Removed unnecessary dependencies on libraries to avoid the security vulnerabilities CVE-2020-8908 and CVE-2018-10237.
- Added support for using the JDBC data type TIMESTAMP_WITH_TIMEZONE when reading data from Snowflake.
- Changed the logic for checking for the existence of a table before saving a DataFrame to Snowflake:
- The connector now reuses the existing connection (rather than creating a new connection) to avoid potential problems with token expiration.
- If the table name is not fully qualified (i.e. does not include the schema name), the connector now checks for the table under the schema specified by
sfSchema, rather than the schema that is currently in use in the session.
Note: If you need to save a DataFrame to a table in a schema other than sfSchema, specify the schema as part of the fully qualified name of the table, rather than executing USE SCHEMA to change the current schema.
- Improved performance by avoiding unnecessary
parse_json() calls in the COPY INTO TABLE command when writing a DataFrame with ArrayType, MapType, or StructType columns to Snowflake. - Added the
getLastSelectQueryId and getLastCopyLoadQueryId methods to the Utils class. These methods return the query ID of the last query that read data from Snowflake and the last COPY INTO TABLE statement that was executed (respectively).
Version 2.10.0 (February 17, 2022)
Compatible JDBC Driver version: 3.13.14
Behavior Change
- Added support for Spark, version 3.2. Beginning with this release, the Snowflake Connector for Spark only supports Spark 3.0, 3.1 and 3.2. Spark version 2.4 is no longer supported.
Bug Fix
- Fixed an issue where string "null" is regarded as type NULL.
