【问题标题】:unresolved dependency org.apache.spark#spark-network-common_2.10 while building spark 1.2.0构建 spark 1.2.0 时未解决的依赖 org.apache.spark#spark-network-common_2.10
【发布时间】:2015-04-15 05:31:59
【问题描述】:

用sbt版本0.13.6

rof@railsonfire_1ad6bd90-94b5-0132-e2d9-5692d74b909b_7f496a9f21bf:~/bin/spark$ sbt/sbt --version
Using /usr/lib/jvm/java-7-openjdk-amd64/ as default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
sbt launcher version 0.13.6

Spark 1.2.0 失败了

rof@railsonfire_1ad6bd90-94b5-0132-e2d9-5692d74b909b_7f496a9f21bf:~/bin/spark$ ~/bin/sbt/bin/sbt compile
[info] Loading project definition from /home/rof/bin/spark/project/project
[info] Loading project definition from /home/rof/.sbt/0.13/staging/ad8e8574a5bcb2d22d23/sbt-pom-reader/project
[warn] Multiple resolvers having different access mechanism configured with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * com.typesafe.sbt:sbt-git:0.6.1 -> 0.6.2
[warn]  * com.typesafe.sbt:sbt-site:0.7.0 -> 0.7.1
[warn] Run 'evicted' to see detailed eviction warnings
[info] Loading project definition from /home/rof/bin/spark/project
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * org.apache.maven.wagon:wagon-provider-api:1.0-beta-6 -> 2.2
[warn] Run 'evicted' to see detailed eviction warnings
[info] Set current project to spark-parent (in build file:/home/rof/bin/spark/)
[info] Updating {file:/home/rof/bin/spark/}network-shuffle...
[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * com.google.guava:guava:10.0.1 -> 14.0.1
[warn] Run 'evicted' to see detailed eviction warnings
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.spark#spark-network-common_2.10;1.2.0: configuration not public in org.apache.spark#spark-network-common_2.10;1.2.0: 'test'. It was required from org.apache.spark#spark-network-shuffle_2.10;1.2.0 test
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn]  Note: Unresolved dependencies path:
[warn]      org.apache.spark:spark-network-common_2.10:1.2.0 ((com.typesafe.sbt.pom.MavenHelper) MavenHelper.scala#L76)
[warn]        +- org.apache.spark:spark-network-shuffle_2.10:1.2.0
sbt.ResolveException: unresolved dependency: org.apache.spark#spark-network-common_2.10;1.2.0: configuration not public in org.apache.spark#spark-network-common_2.10;1.2.0: 'test'. It was required from org.apache.spark#spark-network-shuffle_2.10;1.2.0 test
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:243)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:158)
    at sbt.IvyActions$$anonfun$updateEither$1.apply(IvyActions.scala:156)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:147)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:124)
    at sbt.IvySbt.sbt$IvySbt$$action$1(Ivy.scala:56)
    at sbt.IvySbt$$anon$3.call(Ivy.scala:64)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:93)
    at xsbt.boot.Locks$GlobalLock.xsbt$boot$Locks$GlobalLock$$withChannelRetries$1(Locks.scala:78)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:97)
    at xsbt.boot.Using$.withResource(Using.scala:10)
    at xsbt.boot.Using$.apply(Using.scala:9)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:58)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:48)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.IvySbt.withDefaultLogger(Ivy.scala:64)
    at sbt.IvySbt.withIvy(Ivy.scala:119)
    at sbt.IvySbt.withIvy(Ivy.scala:116)
    at sbt.IvySbt$Module.withModule(Ivy.scala:147)
    at sbt.IvyActions$.updateEither(IvyActions.scala:156)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1282)
    at sbt.Classpaths$$anonfun$sbt$Classpaths$$work$1$1.apply(Defaults.scala:1279)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1309)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$84.apply(Defaults.scala:1307)
    at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1312)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:1306)
    at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
    at sbt.Classpaths$.cachedUpdate(Defaults.scala:1324)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1264)
    at sbt.Classpaths$$anonfun$updateTask$1.apply(Defaults.scala:1242)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:40)
    at sbt.std.Transform$$anon$4.work(System.scala:63)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:226)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:17)
    at sbt.Execute.work(Execute.scala:235)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:226)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:159)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:28)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
[error] (network-shuffle/*:update) sbt.ResolveException: unresolved dependency: org.apache.spark#spark-network-common_2.10;1.2.0: configuration not public in org.apache.spark#spark-network-common_2.10;1.2.0: 'test'. It was required from org.apache.spark#spark-network-shuffle_2.10;1.2.0 test
[error] Total time: 1 s, completed Feb 13, 2015 10:10:34 PM

这究竟是什么原因,之前的1.1.0版本没有问题。

【问题讨论】:

    标签: dependencies sbt apache-spark sbt-assembly


    【解决方案1】:

    我在尝试运行 sbt/sbt eclipse 时收到了类似的消息,删除了 ivy 缓存 (~/.ivy2) 并重复了对我有用的操作。 FWIW,在随后的迭代中,我意识到我已经省略了我在原始构建中使用的 hadoop 选项,例如,运行 sbt/sbt -Phadoop-2.4 -Dhadoop.version=2.4.0 eclipse 没有工作必须删除缓存。

    【讨论】:

      【解决方案2】:

      我遇到了同样的问题。 当我尝试使用清晰的 sbt env(缓存中没有任何 jars)时,

      build/sbt -Pyarn -Phadoop-2.4 -Dhadoop.version=2.6.0 assembly

      工作,也许你可以尝试使用maven
      you can see here

      【讨论】:

        【解决方案3】:

        我尝试了sbt/sbt assembly -Phadoop-2.4 -Dhadoop.version=2.4.0,但也失败了。我的 spark 安装在 /usr/bin 中,所以我尝试使用 root 权限重新运行,这似乎可以解决它。我想知道是否存在显示为完全不同的错误的失败写入。

        sudo sbt/sbt assembly -Phadoop-2.4 -Dhadoop.version=2.4.0

        【讨论】:

          【解决方案4】:

          这已在 sbt 0.13.10-RC1 中修复 https://github.com/sbt/sbt/pull/2345

          正如 PR 解释的那样,这是因为 spark 使用了 sbt-pomreader,并且

          在 Maven 上,测试配置被认为是私有的,因此在某些情况下项目依赖于 test->test 配置可能不会:条件是当有人解析 x:y:1.0 并且您有一个名为和版本化的子项目时,和其他一些子项目试图依赖它。 (182374c) 当一个项目发布到 Maven,然后有人从 Github 获取代码时,确实会发生这种情况,twitter-util 就是这种情况。我自己在 sbt 模块化过程中遇到了这个问题,显然它也发生在 spark 开发中。

          【讨论】:

            猜你喜欢
            • 1970-01-01
            • 2018-02-02
            • 2017-10-29
            • 1970-01-01
            • 2019-12-09
            • 1970-01-01
            • 1970-01-01
            • 1970-01-01
            • 2017-03-15
            相关资源
            最近更新 更多