本文整理汇总了Scala中org.apache.parquet.hadoop.api.WriteSupport.WriteContext类的典型用法代码示例。如果您正苦于以下问题:Scala WriteContext类的具体用法?Scala WriteContext怎么用?Scala WriteContext使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
在下文中一共展示了WriteContext类的1个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Scala代码示例。
示例1: ScalaPBWriteSupport
//设置package包名称以及导入依赖的类
package com.trueaccord.scalapb.parquet
import java.util
import com.google.protobuf.Descriptors.Descriptor
import com.trueaccord.scalapb.{GeneratedMessage, Message}
import org.apache.hadoop.conf.Configuration
import org.apache.parquet.hadoop.BadConfigurationException
import org.apache.parquet.hadoop.api.WriteSupport
import org.apache.parquet.hadoop.api.WriteSupport.WriteContext
import org.apache.parquet.io.api.RecordConsumer
import org.apache.parquet.schema.MessageType
class ScalaPBWriteSupport[T <: GeneratedMessage with Message[T]] extends WriteSupport[T] {
var pbClass: Class[T] = null
var recordConsumer: RecordConsumer = null
override def init(configuration: Configuration): WriteContext = {
if (pbClass == null) {
pbClass = configuration.getClass(ScalaPBWriteSupport.SCALAPB_CLASS_WRITE, null, classOf[GeneratedMessage]).asInstanceOf[Class[T]]
if (pbClass == null) {
throw new BadConfigurationException("ScalaPB class not specified. Please use ScalaPBOutputFormat.setMessageClass.")
}
}
val descriptor: Descriptor = pbClass.getMethod("descriptor").invoke(null).asInstanceOf[Descriptor]
val rootSchema: MessageType = SchemaConverter.convert(descriptor)
val extraMetaDtata = new util.HashMap[String, String]
extraMetaDtata.put(ScalaPBReadSupport.PB_CLASS, pbClass.getName)
new WriteContext(rootSchema, extraMetaDtata)
}
override def write(record: T): Unit = {
MessageWriter.writeTopLevelMessage(recordConsumer, record)
}
override def prepareForWrite(recordConsumer: RecordConsumer): Unit = {
this.recordConsumer = recordConsumer
}
}
object ScalaPBWriteSupport {
val SCALAPB_CLASS_WRITE = "parquet.scalapb.writeClass"
def setSchema[T <: GeneratedMessage](config: Configuration, protoClass: Class[T]) = {
config.setClass(SCALAPB_CLASS_WRITE, protoClass, classOf[GeneratedMessage])
}
}
开发者ID:scalapb,项目名称:sparksql-scalapb,代码行数:48,代码来源:ScalaPBWriteSupport.scala
注:本文中的org.apache.parquet.hadoop.api.WriteSupport.WriteContext类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论