本文整理汇总了Java中org.apache.orc.Writer类的典型用法代码示例。如果您正苦于以下问题:Java Writer类的具体用法?Java Writer怎么用?Java Writer使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
Writer类属于org.apache.orc包,在下文中一共展示了Writer类的4个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: flush
import org.apache.orc.Writer; //导入依赖的package包/类
private boolean flush(BufferSegment segment, String path, TypeDescription schema)
{
Configuration conf = new Configuration();
try {
Writer writer = OrcFile.createWriter(new Path(path),
OrcFile.writerOptions(conf)
.setSchema(schema)
.stripeSize(orcFileStripeSize)
.bufferSize(orcFileBufferSize)
.blockSize(orcFileBlockSize)
.compress(CompressionKind.ZLIB)
.version(OrcFile.Version.V_0_12));
VectorizedRowBatch batch = schema.createRowBatch();
while (segment.hasNext()) {
String[] contents = segment.getNext();
int rowCount = batch.size++;
// System.out.println("contents : message.getValues() : " + Arrays.toString(contents));
System.out.println("contents.length : " + contents.length);
for (int i = 0; i < contents.length; i++) {
((BytesColumnVector) batch.cols[i]).setVal(rowCount, contents[i].getBytes());
//batch full
if (batch.size == batch.getMaxSize()) {
writer.addRowBatch(batch);
batch.reset();
}
}
if (batch.size != 0) {
writer.addRowBatch(batch);
batch.reset();
}
writer.close();
segment.setFilePath(path);
System.out.println("path : " + path);
}
return true;
}
catch (IOException e) {
e.printStackTrace();
return false;
}
}
开发者ID:dbiir,项目名称:paraflow,代码行数:42,代码来源:OrcFlushThread.java
示例2: AWriterImpl
import org.apache.orc.Writer; //导入依赖的package包/类
public AWriterImpl(FileSystem fs, Path path, OrcFile.WriterOptions opts) throws IOException {
this.fs = fs;
this.path = path;
this.conf = opts.getConfiguration();
this.callback = opts.getCallback();
this.schema = opts.getSchema();
if (callback != null) {
callbackContext = new OrcFile.WriterContext() {
@Override
public Writer getWriter() {
return AWriterImpl.this;
}
};
} else {
callbackContext = null;
}
this.adjustedStripeSize = opts.getStripeSize();
this.defaultStripeSize = opts.getStripeSize();
this.version = opts.getVersion();
this.encodingStrategy = opts.getEncodingStrategy();
this.compressionStrategy = opts.getCompressionStrategy();
this.addBlockPadding = opts.getBlockPadding();
this.blockSize = opts.getBlockSize();
this.paddingTolerance = opts.getPaddingTolerance();
this.compress = opts.getCompress();
this.rowIndexStride = opts.getRowIndexStride();
this.memoryManager = opts.getMemoryManager();
buildIndex = rowIndexStride > 0;
codec = createCodec(compress);
int numColumns = schema.getMaximumId() + 1;
if (opts.isEnforceBufferSize()) {
this.bufferSize = opts.getBufferSize();
} else {
this.bufferSize = getEstimatedBufferSize(defaultStripeSize, numColumns, opts.getBufferSize());
}
if (version == OrcFile.Version.V_0_11) {
/* do not write bloom filters for ORC v11 */
this.bloomFilterColumns = new boolean[schema.getMaximumId() + 1];
} else {
this.bloomFilterColumns = OrcUtils.includeColumns(opts.getBloomFilterColumns(), schema);
}
this.bloomFilterFpp = opts.getBloomFilterFpp();
treeWriter = createTreeWriter(schema, streamFactory, false);
if (buildIndex && rowIndexStride < MIN_ROW_INDEX_STRIDE) {
throw new IllegalArgumentException("Row stride must be at least " + MIN_ROW_INDEX_STRIDE);
}
// ensure that we are able to handle callbacks before we register ourselves
memoryManager.addWriter(path, opts.getStripeSize(), this);
// LOG.info("ORC writer created for path: {} with stripeSize: {} blockSize: {}" +
// " compression: {} bufferSize: {}", path, defaultStripeSize, blockSize,
// compress, bufferSize);
}
开发者ID:ampool,项目名称:monarch,代码行数:55,代码来源:AWriterImpl.java
示例3: OrcEntityProcessor
import org.apache.orc.Writer; //导入依赖的package包/类
OrcEntityProcessor(Writer writer, VectorizedRowBatch batch) {
this.writer = writer;
this.batch = batch;
}
开发者ID:mojodna,项目名称:osm2orc,代码行数:5,代码来源:OrcWriter.java
示例4: OrcMetaDataWriter
import org.apache.orc.Writer; //导入依赖的package包/类
public OrcMetaDataWriter( Writer writer ) {
this.writer = writer;
}
开发者ID:pentaho,项目名称:pentaho-hadoop-shims,代码行数:4,代码来源:OrcMetaDataWriter.java
注:本文中的org.apache.orc.Writer类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论