本文整理汇总了Java中org.apache.flume.formatter.output.BucketPath类的典型用法代码示例。如果您正苦于以下问题：Java BucketPath类的具体用法？Java BucketPath怎么用？Java BucketPath使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
BucketPath类属于org.apache.flume.formatter.output包，在下文中一共展示了BucketPath类的12个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: makeEndPoint
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
private HiveEndPoint makeEndPoint(String metaStoreUri, String database, String table,
List<String> partVals, Map<String, String> headers,
TimeZone timeZone, boolean needRounding,
int roundUnit, Integer roundValue,
boolean useLocalTime) {
if (partVals == null) {
return new HiveEndPoint(metaStoreUri, database, table, null);
}
ArrayList<String> realPartVals = Lists.newArrayList();
for (String partVal : partVals) {
realPartVals.add(BucketPath.escapeString(partVal, headers, timeZone,
needRounding, roundUnit, roundValue, useLocalTime));
}
return new HiveEndPoint(metaStoreUri, database, table, realPartVals);
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:17，代码来源:HiveSink.java
示例2: buildPartitionSpec
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
private Map<String, String> buildPartitionSpec(List<String> partitionValues, Map<String, String> headers, TimeZone timeZone,
boolean needRounding, int roundUnit, Integer roundValue, boolean useLocalTime)
throws OdpsException {
Map<String, String> partitionMap = Maps.newHashMap();
if (partitionValues == null || partitionValues.size() == 0) {
return partitionMap;
}
if (configure.getMaxcomputePartitionCols().size() != configure.getMaxcomputePartitionVals().size()) {
throw new RuntimeException("MaxCompute partition fields number not equals input partition values number");
}
for (int i = 0; i < partitionValues.size(); i++) {
String realPartVal = BucketPath
.escapeString(partitionValues.get(i), headers, timeZone, needRounding,
roundUnit, roundValue, useLocalTime);
partitionMap.put(configure.getMaxcomputePartitionCols().get(i), realPartVal);
}
return partitionMap;
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:19，代码来源:DatahubSink.java
示例3: buildPartitionSpec
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
private String buildPartitionSpec(List<String> partitionValues, Map<String, String> headers, TimeZone timeZone,
boolean needRounding, int roundUnit, Integer roundValue, boolean useLocalTime)
throws OdpsException {
if (partitionValues == null || partitionValues.size() == 0) {
return StringUtils.EMPTY;
}
if (partitionValues.size() != partitionFields.size()) {
throw new RuntimeException("Odps partition fields number not equals input partition values number");
}
StringBuilder sb = new StringBuilder();
String sep = "";
for (int i = 0; i < partitionValues.size(); i++) {
String realPartVal = BucketPath.escapeString(partitionValues.get(i), headers, timeZone, needRounding,
roundUnit, roundValue, useLocalTime);
sb.append(sep).append(partitionFields.get(i)).append("='").append(realPartVal).append("'");
sep = ",";
}
String partitionSpec = sb.toString();
if (autoCreatePartition && !partitionMap.containsKey(partitionSpec)) {
odpsTable.createPartition(new PartitionSpec(partitionSpec), true);
partitionMap.put(partitionSpec, true);
}
return partitionSpec;
}
开发者ID:aliyun，项目名称:aliyun-maxcompute-data-collectors，代码行数:25，代码来源:OdpsSink.java
示例4: getIndexName
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
/**
* Gets the name of the index to use for an index request
* @param event
* Event for which the name of index has to be prepared
* @return index name of the form 'indexPrefix-formattedTimestamp'
*/
@Override
public String getIndexName(Event event) {
TimestampedEvent timestampedEvent = new TimestampedEvent(event);
long timestamp = timestampedEvent.getTimestamp();
String realIndexPrefix = BucketPath.escapeString(indexPrefix, event.getHeaders());
return new StringBuilder(realIndexPrefix).append('-')
.append(fastDateFormat.format(timestamp)).toString();
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:15，代码来源:TimeBasedIndexNameBuilder.java
示例5: createIndexRequest
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
/**
* Creates and prepares an {@link IndexRequestBuilder} from the supplied
* {@link Client} via delegation to the subclass-hook template methods
* {@link #getIndexName(String, long)} and
* {@link #prepareIndexRequest(IndexRequestBuilder, String, String, Event)}
*/
@Override
public IndexRequestBuilder createIndexRequest(Client client,
String indexPrefix, String indexType, Event event) throws IOException {
IndexRequestBuilder request = prepareIndex(client);
String realIndexPrefix = BucketPath.escapeString(indexPrefix, event.getHeaders());
String realIndexType = BucketPath.escapeString(indexType, event.getHeaders());
TimestampedEvent timestampedEvent = new TimestampedEvent(event);
long timestamp = timestampedEvent.getTimestamp();
String indexName = getIndexName(realIndexPrefix, timestamp);
prepareIndexRequest(request, indexName, realIndexType, timestampedEvent);
return request;
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:21，代码来源:AbstractElasticSearchIndexRequestBuilderFactory.java
示例6: createIndexRequest
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
/**
* Creates and prepares an {@link IndexRequestBuilder} from the supplied
* {@link Client} via delegation to the subclass-hook template methods
* {@link #getIndexName(String, long)} and
* {@link #prepareIndexRequest(IndexRequestBuilder, String, String, Event)}
*/
@Override
public IndexRequestBuilder createIndexRequest(Client client,
String indexPrefix, String indexType, Event event) throws IOException {
IndexRequestBuilder request = prepareIndex(client);
String realIndexPrefix = BucketPath.escapeString(indexPrefix, event.getHeaders());
String realIndexType = BucketPath.escapeString(indexType, event.getHeaders());
TimestampedEvent timestampedEvent = new TimestampedEvent(event);
long timestamp = timestampedEvent.getTimestamp();
String indexName = getIndexName(realIndexPrefix, timestamp);
prepareIndexRequest(request, indexName, realIndexType, timestampedEvent);
return request;
}
开发者ID:Redliver，项目名称:flume-ng-elasticsearch5-sink，代码行数:21，代码来源:AbstractElasticSearchIndexRequestBuilderFactory.java
示例7: setBucketClock
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
@VisibleForTesting
void setBucketClock(Clock clock) {
BucketPath.setClock(clock);
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:5，代码来源:HDFSEventSink.java
示例8: getIndexPrefix
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
@Override
public String getIndexPrefix(Event event) {
return BucketPath.escapeString(indexPrefix, event.getHeaders());
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:5，代码来源:TimeBasedIndexNameBuilder.java
示例9: getIndexName
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
@Override
public String getIndexName(Event event) {
return BucketPath.escapeString(indexName, event.getHeaders());
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:5，代码来源:SimpleIndexNameBuilder.java
示例10: getIndexPrefix
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
@Override
public String getIndexPrefix(Event event) {
return BucketPath.escapeString(indexName, event.getHeaders());
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:5，代码来源:SimpleIndexNameBuilder.java
示例11: process
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
@Override
public Status process() throws EventDeliveryException {
logger.debug("processing...");
Status status = Status.READY;
Channel channel = getChannel();
Transaction txn = channel.getTransaction();
try {
txn.begin();
int count;
for (count = 0; count < batchSize; ++count) {
Event event = channel.take();
if (event == null) {
break;
}
String realIndexType = BucketPath.escapeString(indexType, event.getHeaders());
client.addEvent(event, indexNameBuilder, realIndexType, ttlMs);
}
if (count <= 0) {
sinkCounter.incrementBatchEmptyCount();
counterGroup.incrementAndGet("channel.underflow");
status = Status.BACKOFF;
} else {
if (count < batchSize) {
sinkCounter.incrementBatchUnderflowCount();
status = Status.BACKOFF;
} else {
sinkCounter.incrementBatchCompleteCount();
}
sinkCounter.addToEventDrainAttemptCount(count);
client.execute();
}
txn.commit();
sinkCounter.addToEventDrainSuccessCount(count);
counterGroup.incrementAndGet("transaction.success");
} catch (Throwable ex) {
try {
txn.rollback();
counterGroup.incrementAndGet("transaction.rollback");
} catch (Exception ex2) {
logger.error(
"Exception in rollback. Rollback might not have been successful.",
ex2);
}
if (ex instanceof Error || ex instanceof RuntimeException) {
logger.error("Failed to commit transaction. Transaction rolled back.",
ex);
Throwables.propagate(ex);
} else {
logger.error("Failed to commit transaction. Transaction rolled back.",
ex);
throw new EventDeliveryException(
"Failed to commit transaction. Transaction rolled back.", ex);
}
} finally {
txn.close();
}
return status;
}
开发者ID:moueimei，项目名称:flume-release-1.7.0，代码行数:63，代码来源:ElasticSearchSink.java
示例12: process
import org.apache.flume.formatter.output.BucketPath; //导入依赖的package包/类
@Override
public Status process() throws EventDeliveryException {
logger.debug("processing...");
Status status = Status.READY;
Channel channel = getChannel();
Transaction txn = channel.getTransaction();
try {
txn.begin();
int count;
for (count = 0; count < batchSize; ++count) {
Event event = channel.take();
if (event == null) {
break;
}
String realIndexType = BucketPath.escapeString(indexType, event.getHeaders());
client.addEvent(event, indexNameBuilder, realIndexType, ttlMs);
}
if (count <= 0) {
sinkCounter.incrementBatchEmptyCount();
counterGroup.incrementAndGet("channel.underflow");
status = Status.BACKOFF;
} else {
if (count < batchSize) {
sinkCounter.incrementBatchUnderflowCount();
status = Status.BACKOFF;
} else {
sinkCounter.incrementBatchCompleteCount();
}
sinkCounter.addToEventDrainAttemptCount(count);
client.execute();
}
txn.commit();
sinkCounter.addToEventDrainSuccessCount(count);
counterGroup.incrementAndGet("transaction.success");
} catch (Throwable ex) {
try {
txn.rollback();
counterGroup.incrementAndGet("transaction.rollback");
} catch (Exception ex2) {
logger.error(
"Exception in rollback. Rollback might not have been successful.",
ex2);
}
if (ex instanceof Error || ex instanceof RuntimeException) {
logger.error(FAILED_TO_COMMIT_TRANSACTION_TRANSACTION_ROLLED_BACK,
ex);
Throwables.propagate(ex);
} else {
logger.error(FAILED_TO_COMMIT_TRANSACTION_TRANSACTION_ROLLED_BACK,
ex);
throw new EventDeliveryException(
FAILED_TO_COMMIT_TRANSACTION_TRANSACTION_ROLLED_BACK, ex);
}
} finally {
txn.close();
}
return status;
}
开发者ID:lucidfrontier45，项目名称:ElasticsearchSink2，代码行数:63，代码来源:ElasticSearchSink.java
注：本文中的org.apache.flume.formatter.output.BucketPath类示例整理自Github/MSDocs等源码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。 |
客服电话
APP下载
官方微信
请发表评论