Skip to content

Commit 87e3861

Browse files
sarutakpan3793
authored andcommitted
[SPARK-56353][BUILD][TESTS][FOLLOWUP] Fix resource loading to work when resources are inside JARs
### What changes were proposed in this pull request? This PR fixes `FileSystemNotFoundException` in `TestSpark21101Jar`, `TestUDTFJar`, and `TestHiveUdfsJar` when running tests with Maven. These files used `Paths.get(url.toURI)` to read Java source files from classpath resources. This works with SBT where resources are extracted to `target/test-classes/` as regular files, but fails with Maven where resources may be inside JAR files, producing a `jar:file:...!/path` URI that `Paths.get()` cannot handle without a ZipFileSystem. The fix replaces `Files.readAllBytes(Paths.get(url.toURI))` with `url.openStream().readAllBytes()`, which works regardless of whether the resource is a regular file or inside a JAR. ### Why are the changes needed? CI runs with Maven and hits `FileSystemNotFoundException` when initializing `TestUDTFJar`, causing `CliSuite` and `HiveThriftBinaryServerSuite` to fail: ``` Cause: java.lang.ExceptionInInitializerError: Exception java.nio.file.FileSystemNotFoundException at jdk.zipfs/jdk.nio.zipfs.ZipFileSystemProvider.getFileSystem(ZipFileSystemProvider.java:156) at java.base/java.nio.file.Paths.get(Paths.java:98) at org.apache.spark.sql.hive.test.TestUDTFJar$.loadResource(TestUDTFJar.scala:37) ``` Reported in #55272 (comment) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Verified with Maven (the environment where the failure occurs): - CliSuite: 44/44 passed - HiveThriftBinaryServerSuite: 30/30 passed ### Was this patch authored or co-authored using generative AI tooling? Kiro CLI / Opus 4.6 Closes #55299 from sarutak/fix-resource-loading. Authored-by: Kousuke Saruta <sarutak@amazon.co.jp> Signed-off-by: Cheng Pan <chengpan@apache.org>
1 parent d4b59dd commit 87e3861

3 files changed

Lines changed: 9 additions & 6 deletions

File tree

sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestHiveUdfsJar.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ package org.apache.spark.sql.hive.test
2020
import java.io.File
2121
import java.lang.management.ManagementFactory
2222
import java.nio.charset.StandardCharsets
23-
import java.nio.file.{Files, Paths}
2423

2524
import org.apache.spark.util.{SparkTestUtils, Utils}
2625

@@ -64,6 +63,8 @@ object TestHiveUdfsJar {
6463
private def readResource(name: String): String = {
6564
val url = Thread.currentThread().getContextClassLoader.getResource(name)
6665
assert(url != null, s"Resource not found: $name")
67-
new String(Files.readAllBytes(Paths.get(url.toURI)), StandardCharsets.UTF_8)
66+
Utils.tryWithResource(url.openStream()) { is =>
67+
new String(is.readAllBytes(), StandardCharsets.UTF_8)
68+
}
6869
}
6970
}

sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestSpark21101Jar.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ package org.apache.spark.sql.hive.test
2020
import java.io.File
2121
import java.lang.management.ManagementFactory
2222
import java.nio.charset.StandardCharsets
23-
import java.nio.file.{Files, Paths}
2423

2524
import org.apache.spark.util.{SparkTestUtils, Utils}
2625

@@ -52,6 +51,8 @@ object TestSpark21101Jar {
5251
private def readResource(name: String): String = {
5352
val url = Thread.currentThread().getContextClassLoader.getResource(name)
5453
assert(url != null, s"Resource not found: $name")
55-
new String(Files.readAllBytes(Paths.get(url.toURI)), StandardCharsets.UTF_8)
54+
Utils.tryWithResource(url.openStream()) { is =>
55+
new String(is.readAllBytes(), StandardCharsets.UTF_8)
56+
}
5657
}
5758
}

sql/hive/src/test/scala/org/apache/spark/sql/hive/test/TestUDTFJar.scala

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ package org.apache.spark.sql.hive.test
2020
import java.io.File
2121
import java.lang.management.ManagementFactory
2222
import java.nio.charset.StandardCharsets
23-
import java.nio.file.{Files, Paths}
2423

2524
import org.apache.spark.TestUtils
2625
import org.apache.spark.util.Utils
@@ -34,7 +33,9 @@ object TestUDTFJar {
3433
private def loadResource(name: String): String = {
3534
val url = Thread.currentThread().getContextClassLoader.getResource(name)
3635
assert(url != null, s"Resource not found: $name")
37-
new String(Files.readAllBytes(Paths.get(url.toURI)), StandardCharsets.UTF_8)
36+
Utils.tryWithResource(url.openStream()) { is =>
37+
new String(is.readAllBytes(), StandardCharsets.UTF_8)
38+
}
3839
}
3940

4041
private val source = Map(

0 commit comments

Comments
 (0)