本文整理汇总了Java中org.apache.tinkerpop.gremlin.process.computer.MapReduce类的典型用法代码示例。如果您正苦于以下问题:Java MapReduce类的具体用法?Java MapReduce怎么用?Java MapReduce使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
MapReduce类属于org.apache.tinkerpop.gremlin.process.computer包,在下文中一共展示了MapReduce类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: executeMapReduce
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public void executeMapReduce(final Consumer<MapReduce> worker) throws InterruptedException {
for (int i = 0; i < this.numberOfWorkers; i++) {
this.completionService.submit(() -> {
final MapReduce mr = this.mapReducePool.take();
worker.accept(mr);
this.mapReducePool.offer(mr);
return null;
});
}
for (int i = 0; i < this.numberOfWorkers; i++) {
try {
this.completionService.take().get();
} catch (InterruptedException ie) {
throw ie;
} catch (final Exception e) {
throw new IllegalStateException(e.getMessage(), e);
}
}
}
开发者ID:ShiftLeftSecurity,项目名称:tinkergraph-gremlin,代码行数:20,代码来源:TinkerWorkerPool.java
示例2: execute
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Number> execute() {
LOGGER.info("SumMapReduce is called");
long startTime = System.currentTimeMillis();
initSubGraph();
getAllSubTypes();
AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);
ComputerResult result = getGraphComputer().compute(
new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
new SumMapReduce(statisticsResourceLabelIds, dataType,
DegreeVertexProgram.DEGREE),
allSubLabelIds);
Map<Serializable, Number> sum = result.memory().get(SumMapReduce.class.getName());
Number finalResult = sum.get(MapReduce.NullObject.instance());
LOGGER.info("Sum = " + finalResult);
LOGGER.info("SumMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
return Optional.of(finalResult);
}
开发者ID:graknlabs,项目名称:grakn,代码行数:27,代码来源:SumQueryImpl.java
示例3: execute
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Number> execute() {
LOGGER.info("MinMapReduce is called");
long startTime = System.currentTimeMillis();
initSubGraph();
getAllSubTypes();
AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);
ComputerResult result = getGraphComputer().compute(
new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
new MinMapReduce(statisticsResourceLabelIds, dataType,
DegreeVertexProgram.DEGREE),
allSubLabelIds);
Map<Serializable, Number> min = result.memory().get(MinMapReduce.class.getName());
LOGGER.debug("Min = " + min.get(MapReduce.NullObject.instance()));
LOGGER.info("MinMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
return Optional.of(min.get(MapReduce.NullObject.instance()));
}
开发者ID:graknlabs,项目名称:grakn,代码行数:25,代码来源:MinQueryImpl.java
示例4: execute
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Number> execute() {
LOGGER.info("MaxMapReduce is called");
long startTime = System.currentTimeMillis();
initSubGraph();
getAllSubTypes();
AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);
ComputerResult result = getGraphComputer().compute(
new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
new MaxMapReduce(statisticsResourceLabelIds, dataType,
DegreeVertexProgram.DEGREE),
allSubLabelIds);
Map<Serializable, Number> max = result.memory().get(MaxMapReduce.class.getName());
LOGGER.debug("Max = " + max.get(MapReduce.NullObject.instance()));
LOGGER.info("MaxMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
return Optional.of(max.get(MapReduce.NullObject.instance()));
}
开发者ID:graknlabs,项目名称:grakn,代码行数:25,代码来源:MaxQueryImpl.java
示例5: compute
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public ComputerResult compute(@Nullable VertexProgram program, @Nullable MapReduce mapReduce,
@Nullable Set<LabelId> types, Boolean includesRolePlayerEdges) {
try {
graphComputer = getGraphComputer();
if (program != null) {
graphComputer.program(program);
} else {
filterAllEdges = true;
}
if (mapReduce != null) graphComputer.mapReduce(mapReduce);
applyFilters(types, includesRolePlayerEdges);
return graphComputer.submit().get();
} catch (InterruptedException | ExecutionException e) {
throw asRuntimeException(e.getCause());
}
}
开发者ID:graknlabs,项目名称:grakn,代码行数:18,代码来源:GraknComputerImpl.java
示例6: GraknSparkMemory
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public GraknSparkMemory(final VertexProgram<?> vertexProgram,
final Set<MapReduce> mapReducers,
final JavaSparkContext sparkContext) {
if (null != vertexProgram) {
for (final MemoryComputeKey key : vertexProgram.getMemoryComputeKeys()) {
this.memoryComputeKeys.put(key.getKey(), key);
}
}
for (final MapReduce mapReduce : mapReducers) {
this.memoryComputeKeys.put(
mapReduce.getMemoryKey(),
MemoryComputeKey.of(mapReduce.getMemoryKey(), Operator.assign, false, false));
}
for (final MemoryComputeKey memoryComputeKey : this.memoryComputeKeys.values()) {
this.sparkMemory.put(
memoryComputeKey.getKey(),
sparkContext.accumulator(ObjectWritable.empty(), memoryComputeKey.getKey(),
new MemoryAccumulator<>(memoryComputeKey)));
}
this.broadcast = sparkContext.broadcast(Collections.emptyMap());
}
开发者ID:graknlabs,项目名称:grakn,代码行数:22,代码来源:GraknSparkMemory.java
示例7: SparkMemory
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public SparkMemory(final VertexProgram<?> vertexProgram, final Set<MapReduce> mapReducers, final JavaSparkContext sparkContext) {
if (null != vertexProgram) {
for (final MemoryComputeKey key : vertexProgram.getMemoryComputeKeys()) {
this.memoryComputeKeys.put(key.getKey(), key);
}
}
for (final MapReduce mapReduce : mapReducers) {
this.memoryComputeKeys.put(mapReduce.getMemoryKey(), MemoryComputeKey.of(mapReduce.getMemoryKey(), Operator.assign, false, false));
}
for (final MemoryComputeKey memoryComputeKey : this.memoryComputeKeys.values()) {
this.sparkMemory.put(
memoryComputeKey.getKey(),
sparkContext.accumulator(ObjectWritable.empty(), memoryComputeKey.getKey(), new MemoryAccumulator<>(memoryComputeKey)));
}
this.broadcast = sparkContext.broadcast(Collections.emptyMap());
}
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:17,代码来源:SparkMemory.java
示例8: shouldEndlessBulkResultsWithNullObject
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Test
public void shouldEndlessBulkResultsWithNullObject() {
long total = 0;
final List<Tuple2<MapReduce.NullObject, Long>> numbers = new ArrayList<>();
for (long i = 0; i < 5000000; i++) {
total = total + i;
numbers.add(new Tuple2<>(MapReduce.NullObject.instance(), i));
}
final CombineIterator<MapReduce.NullObject, Long, MapReduce.NullObject, Long> combineIterator = new CombineIterator<>(new MapReduceB(), numbers.iterator());
assertTrue(combineIterator.hasNext());
final Tuple2<MapReduce.NullObject, Long> tuple = combineIterator.next();
assertFalse(combineIterator.hasNext());
assertEquals(MapReduce.NullObject.instance(), tuple._1());
assertEquals(total, tuple._2().longValue());
}
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:17,代码来源:CombineIteratorTest.java
示例9: process
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public void process(TitanVertex vertex, ScanMetrics metrics) {
PreloadedVertex v = (PreloadedVertex) vertex;
if (vertexMemory != null) {
VertexMemoryHandler vh = new VertexMemoryHandler(vertexMemory, v);
v.setPropertyMixing(vh);
}
v.setAccessCheck(MAPREDUCE_CHECK);
if (idManager.isPartitionedVertex(v.longId()) && !idManager.isCanonicalVertexId(v.longId())) {
return; //Only consider the canonical partition vertex representative
} else {
for (Map.Entry<MapReduce, FulgoraMapEmitter> mapJob : mapJobs.entrySet()) {
MapReduce job = mapJob.getKey();
try {
job.map(v, mapJob.getValue());
metrics.incrementCustom(MAP_JOB_SUCCESS);
} catch (Throwable ex) {
log.error("Encountered exception executing map job [" + job + "] on vertex [" + vertex + "]:", ex);
metrics.incrementCustom(MAP_JOB_FAILURE);
}
}
}
}
开发者ID:graben1437,项目名称:titan1withtp3.1,代码行数:24,代码来源:VertexMapJob.java
示例10: executeMapReduce
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public void executeMapReduce(Consumer<MapReduce<?, ?, ?, ?, R>> worker) {
for (int i = 0; i < numberOfWorkers; i++) {
completionService.submit(() -> {
@SuppressWarnings("unchecked")
MapReduce<?, ?, ?, ?, R> mr = mapReducePool.take();
worker.accept(mr);
mapReducePool.offer(mr);
return null;
});
}
for (int i = 0; i < numberOfWorkers; i++) {
try {
completionService.take().get();
} catch (Exception e) {
throw new IllegalStateException(e.getMessage(), e);
}
}
}
开发者ID:PureSolTechnologies,项目名称:DuctileDB,代码行数:19,代码来源:DuctileWorkerPool.java
示例11: complete
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
protected void complete(final MapReduce<?, ?, OK, OV, ?> mapReduce) {
if (mapReduce.getReduceKeySort().isPresent()) {
final Comparator<OK> comparator = mapReduce.getReduceKeySort().get();
final List<KeyValue<OK, OV>> list = new ArrayList<>(this.reduceQueue);
Collections.sort(list, Comparator.comparing(KeyValue::getKey, comparator));
this.reduceQueue.clear();
this.reduceQueue.addAll(list);
}
}
开发者ID:ShiftLeftSecurity,项目名称:tinkergraph-gremlin,代码行数:10,代码来源:TinkerReduceEmitter.java
示例12: TinkerMemory
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public TinkerMemory(final VertexProgram<?> vertexProgram, final Set<MapReduce> mapReducers) {
this.currentMap = new ConcurrentHashMap<>();
this.previousMap = new ConcurrentHashMap<>();
if (null != vertexProgram) {
for (final MemoryComputeKey memoryComputeKey : vertexProgram.getMemoryComputeKeys()) {
this.memoryKeys.put(memoryComputeKey.getKey(), memoryComputeKey);
}
}
for (final MapReduce mapReduce : mapReducers) {
this.memoryKeys.put(mapReduce.getMemoryKey(), MemoryComputeKey.of(mapReduce.getMemoryKey(), Operator.assign, false, false));
}
}
开发者ID:ShiftLeftSecurity,项目名称:tinkergraph-gremlin,代码行数:13,代码来源:TinkerMemory.java
示例13: execute
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Double> execute() {
LOGGER.info("StdMapReduce is called");
long startTime = System.currentTimeMillis();
initSubGraph();
getAllSubTypes();
AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);
ComputerResult result = getGraphComputer().compute(
new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
new StdMapReduce(statisticsResourceLabelIds, dataType,
DegreeVertexProgram.DEGREE),
allSubLabelIds);
Map<Serializable, Map<String, Double>> std = result.memory().get(StdMapReduce.class.getName());
Map<String, Double> stdTuple = std.get(MapReduce.NullObject.instance());
double squareSum = stdTuple.get(StdMapReduce.SQUARE_SUM);
double sum = stdTuple.get(StdMapReduce.SUM);
double count = stdTuple.get(StdMapReduce.COUNT);
double finalResult = Math.sqrt(squareSum / count - (sum / count) * (sum / count));
LOGGER.debug("Std = " + finalResult);
LOGGER.info("StdMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
return Optional.of(finalResult);
}
开发者ID:graknlabs,项目名称:grakn,代码行数:32,代码来源:StdQueryImpl.java
示例14: execute
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Double> execute() {
LOGGER.info("MeanMapReduce is called");
long startTime = System.currentTimeMillis();
initSubGraph();
getAllSubTypes();
AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);
ComputerResult result = getGraphComputer().compute(
new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
new MeanMapReduce(statisticsResourceLabelIds, dataType,
DegreeVertexProgram.DEGREE),
allSubLabelIds);
Map<Serializable, Map<String, Double>> mean = result.memory().get(MeanMapReduce.class.getName());
Map<String, Double> meanPair = mean.get(MapReduce.NullObject.instance());
double finalResult = meanPair.get(MeanMapReduce.SUM) / meanPair.get(MeanMapReduce.COUNT);
LOGGER.debug("Mean = " + finalResult);
LOGGER.info("MeanMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
return Optional.of(finalResult);
}
开发者ID:graknlabs,项目名称:grakn,代码行数:28,代码来源:MeanQueryImpl.java
示例15: clone
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@SuppressWarnings("unchecked")
@Override
public MapReduce<Serializable, T, Serializable, T, Map<Serializable, T>> clone() {
try {
return (GraknMapReduce) super.clone();
} catch (final CloneNotSupportedException e) {
throw CommonUtil.unreachableStatement(e);
}
}
开发者ID:graknlabs,项目名称:grakn,代码行数:10,代码来源:GraknMapReduce.java
示例16: executeMap
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public static <K, V> JavaPairRDD<K, V> executeMap(
final JavaPairRDD<Object, VertexWritable> graphRDD, final MapReduce<K, V, ?, ?, ?> mapReduce,
final Configuration graphComputerConfiguration) {
JavaPairRDD<K, V> mapRDD = graphRDD.mapPartitionsToPair(partitionIterator -> {
KryoShimServiceLoader.applyConfiguration(graphComputerConfiguration);
return () -> new MapIterator<>(MapReduce.<MapReduce<K, V, ?, ?, ?>>createMapReduce(HadoopGraph.open(graphComputerConfiguration), graphComputerConfiguration), partitionIterator);
});
if (mapReduce.getMapKeySort().isPresent()){
mapRDD = mapRDD.sortByKey(mapReduce.getMapKeySort().get(), true, 1);}
return mapRDD;
}
开发者ID:graknlabs,项目名称:grakn,代码行数:12,代码来源:GraknSparkExecutor.java
示例17: executeCombine
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public static <K, V, OK, OV> JavaPairRDD<OK, OV> executeCombine(final JavaPairRDD<K, V> mapRDD,
final Configuration graphComputerConfiguration) {
return mapRDD.mapPartitionsToPair(partitionIterator -> {
KryoShimServiceLoader.applyConfiguration(graphComputerConfiguration);
return () -> new CombineIterator<>(MapReduce.<MapReduce<K, V, OK, OV, ?>>createMapReduce(HadoopGraph.open(graphComputerConfiguration), graphComputerConfiguration), partitionIterator);
});
}
开发者ID:graknlabs,项目名称:grakn,代码行数:8,代码来源:GraknSparkExecutor.java
示例18: executeReduce
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public static <K, V, OK, OV> JavaPairRDD<OK, OV> executeReduce(
final JavaPairRDD<K, V> mapOrCombineRDD, final MapReduce<K, V, OK, OV, ?> mapReduce,
final Configuration graphComputerConfiguration) {
JavaPairRDD<OK, OV> reduceRDD = mapOrCombineRDD.groupByKey().mapPartitionsToPair(partitionIterator -> {
KryoShimServiceLoader.applyConfiguration(graphComputerConfiguration);
return () -> new ReduceIterator<>(MapReduce.<MapReduce<K, V, OK, OV, ?>>createMapReduce(HadoopGraph.open(graphComputerConfiguration), graphComputerConfiguration), partitionIterator);
});
if (mapReduce.getReduceKeySort().isPresent()){
reduceRDD = reduceRDD.sortByKey(mapReduce.getReduceKeySort().get(), true, 1);}
return reduceRDD;
}
开发者ID:graknlabs,项目名称:grakn,代码行数:12,代码来源:GraknSparkExecutor.java
示例19: setup
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public void setup(final Reducer<ObjectWritable, ObjectWritable, ObjectWritable, ObjectWritable>.Context context) {
final Configuration apacheConfiguration = ConfUtil.makeApacheConfiguration(context.getConfiguration());
KryoShimServiceLoader.applyConfiguration(apacheConfiguration);
this.mapReduce = MapReduce.createMapReduce(HadoopGraph.open(apacheConfiguration), apacheConfiguration);
this.mapReduce.workerStart(MapReduce.Stage.REDUCE);
}
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:8,代码来源:HadoopReduce.java
示例20: setup
import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public void setup(final Reducer<ObjectWritable, ObjectWritable, ObjectWritable, ObjectWritable>.Context context) {
final Configuration apacheConfiguration = ConfUtil.makeApacheConfiguration(context.getConfiguration());
KryoShimServiceLoader.applyConfiguration(apacheConfiguration);
this.mapReduce = MapReduce.createMapReduce(HadoopGraph.open(apacheConfiguration), apacheConfiguration);
this.mapReduce.workerStart(MapReduce.Stage.COMBINE);
}
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:8,代码来源:HadoopCombine.java
注:本文中的org.apache.tinkerpop.gremlin.process.computer.MapReduce类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。 |
请发表评论