【发布时间】:2016-05-16 10:03:17
【问题描述】:
我是游戏框架的新手,我想定期向亚马逊询问一些产品,以便将它们插入到 kafka 主题中,当我尝试编译代码时发生错误。
这是KafkaProducer的代码:
文件 example.model.AmazonProducerExample
//根据用户的建议更新方法,谢谢大家!
package example.utils
import jodd.lagarto.dom.{NodeSelector, LagartoDOMBuilder}
import example.model.AmazonProduct
import scala.collection.JavaConversions._
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Future
import play.api.libs.json._
import example.utils._
import example.producer._
object AmazonPageParser {
private val topicName = "amazonRatingsTopic"
private val producer = Producer[String](topicName)
def parse(productId: String): Future[AmazonProduct] = {
val url = s"http://www.amazon.com/dp/$productId"
HttpClient.fetchUrl(url) map {
httpResponse =>
if (httpResponse.getStatusCode == 200) {
val body = httpResponse.getResponseBody
val domBuilder = new LagartoDOMBuilder()
val doc = domBuilder.parse(body)
val responseUrl = httpResponse.getUri.toString
val nodeSelector = new NodeSelector(doc)
val title = nodeSelector.select("span#productTitle").head.getTextContent
val img = nodeSelector.select("div#main-image-container img").head.getAttribute("src")
val description = nodeSelector.select("div#feature-bullets").headOption.map(_.getHtml).mkString
val amazonProduct = AmazonProduct(productId, title, responseUrl, img, description)
println("amazonProduct is " + amazonProduct.toString)
amazonProduct
} else {
println("An error happened!")
throw new RuntimeException(s"Invalid url $url")
}
}//map
}//parse method
def main(args: Array[String]): Unit = {
//Scala Puzzlers...
AmazonPageParser.parse("0981531679").onSuccess { case amazonProduct =>
implicit val amazonFormat = Json.format[AmazonProduct]
producer.send(Json.toJson(amazonProduct).toString)
println("amazon product sent to kafka cluster..." + amazonProduct.toString)
}
}
}
文件 example.model.Models
package example.model
import play.api.libs.json.Json
import reactivemongo.bson.Macros
case class AmazonProduct(itemId: String, title: String, url: String, img: String, description: String)
case class AmazonRating(userId: String, productId: String, rating: Double)
case class AmazonProductAndRating(product: AmazonProduct, rating: AmazonRating)
// For MongoDB
object AmazonRating {
implicit val amazonRatingHandler = Macros.handler[AmazonRating]
implicit val amazonRatingFormat = Json.format[AmazonRating]
}
文件 example.utils.AmazonPageParser
编译器返回这个错误:
[error] /Users/aironman/my-recommendation-spark-engine/src/main/scala/example/producer/AmazonProducerExample.scala:25: No Json serializer found for type scala.concurrent.Future[example.model.AmazonProduct]. Try to implement an implicit Writes or Format for this type.
[error] producer.send(Json.toJson(amazonProduct).toString)
[error] ^
我已阅读此post with most votes,但它对我不起作用。
有人可以帮我吗?
【问题讨论】:
-
对于
Future[T],您最好使用Writes[T],并将写入操作称为“内部”Future -
您好 chantep,感谢您的回答。据我了解,您指的是对 kafka 主题的写入操作将在 AmazonPageParser.parse 方法中,不是吗?但我想以一种解耦的方式拥有这个功能......
-
您应该在未来完成后进行写入,因此使用
map、flatMap或对值进行适当操作的for-comprehension。更多关于我理解Future的一般性问题。
标签: json scala playframework