• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    公众号

Java MapReduce类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中org.apache.tinkerpop.gremlin.process.computer.MapReduce的典型用法代码示例。如果您正苦于以下问题:Java MapReduce类的具体用法?Java MapReduce怎么用?Java MapReduce使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



MapReduce类属于org.apache.tinkerpop.gremlin.process.computer包,在下文中一共展示了MapReduce类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: executeMapReduce

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public void executeMapReduce(final Consumer<MapReduce> worker) throws InterruptedException {
    for (int i = 0; i < this.numberOfWorkers; i++) {
        this.completionService.submit(() -> {
            final MapReduce mr = this.mapReducePool.take();
            worker.accept(mr);
            this.mapReducePool.offer(mr);
            return null;
        });
    }
    for (int i = 0; i < this.numberOfWorkers; i++) {
        try {
            this.completionService.take().get();
        } catch (InterruptedException ie) {
            throw ie;
        } catch (final Exception e) {
            throw new IllegalStateException(e.getMessage(), e);
        }
    }
}
 
开发者ID:ShiftLeftSecurity,项目名称:tinkergraph-gremlin,代码行数:20,代码来源:TinkerWorkerPool.java


示例2: execute

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Number> execute() {
    LOGGER.info("SumMapReduce is called");
    long startTime = System.currentTimeMillis();

    initSubGraph();
    getAllSubTypes();

    AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
    if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
    Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
    Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);

    ComputerResult result = getGraphComputer().compute(
            new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
            new SumMapReduce(statisticsResourceLabelIds, dataType,
                    DegreeVertexProgram.DEGREE),
            allSubLabelIds);
    Map<Serializable, Number> sum = result.memory().get(SumMapReduce.class.getName());

    Number finalResult = sum.get(MapReduce.NullObject.instance());
    LOGGER.info("Sum = " + finalResult);

    LOGGER.info("SumMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
    return Optional.of(finalResult);
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:27,代码来源:SumQueryImpl.java


示例3: execute

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Number> execute() {
    LOGGER.info("MinMapReduce is called");
    long startTime = System.currentTimeMillis();

    initSubGraph();
    getAllSubTypes();

    AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
    if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
    Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
    Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);

    ComputerResult result = getGraphComputer().compute(
            new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
            new MinMapReduce(statisticsResourceLabelIds, dataType,
                    DegreeVertexProgram.DEGREE),
            allSubLabelIds);
    Map<Serializable, Number> min = result.memory().get(MinMapReduce.class.getName());

    LOGGER.debug("Min = " + min.get(MapReduce.NullObject.instance()));
    LOGGER.info("MinMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
    return Optional.of(min.get(MapReduce.NullObject.instance()));
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:25,代码来源:MinQueryImpl.java


示例4: execute

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Number> execute() {
    LOGGER.info("MaxMapReduce is called");
    long startTime = System.currentTimeMillis();

    initSubGraph();
    getAllSubTypes();

    AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
    if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
    Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
    Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);

    ComputerResult result = getGraphComputer().compute(
            new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
            new MaxMapReduce(statisticsResourceLabelIds, dataType,
                    DegreeVertexProgram.DEGREE),
            allSubLabelIds);
    Map<Serializable, Number> max = result.memory().get(MaxMapReduce.class.getName());

    LOGGER.debug("Max = " + max.get(MapReduce.NullObject.instance()));
    LOGGER.info("MaxMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
    return Optional.of(max.get(MapReduce.NullObject.instance()));
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:25,代码来源:MaxQueryImpl.java


示例5: compute

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public ComputerResult compute(@Nullable VertexProgram program, @Nullable MapReduce mapReduce,
                              @Nullable Set<LabelId> types, Boolean includesRolePlayerEdges) {
    try {
        graphComputer = getGraphComputer();
        if (program != null) {
            graphComputer.program(program);
        } else {
            filterAllEdges = true;
        }
        if (mapReduce != null) graphComputer.mapReduce(mapReduce);
        applyFilters(types, includesRolePlayerEdges);
        return graphComputer.submit().get();
    } catch (InterruptedException | ExecutionException e) {
        throw asRuntimeException(e.getCause());
    }
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:18,代码来源:GraknComputerImpl.java


示例6: GraknSparkMemory

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public GraknSparkMemory(final VertexProgram<?> vertexProgram,
                        final Set<MapReduce> mapReducers,
                        final JavaSparkContext sparkContext) {
    if (null != vertexProgram) {
        for (final MemoryComputeKey key : vertexProgram.getMemoryComputeKeys()) {
            this.memoryComputeKeys.put(key.getKey(), key);
        }
    }
    for (final MapReduce mapReduce : mapReducers) {
        this.memoryComputeKeys.put(
                mapReduce.getMemoryKey(),
                MemoryComputeKey.of(mapReduce.getMemoryKey(), Operator.assign, false, false));
    }
    for (final MemoryComputeKey memoryComputeKey : this.memoryComputeKeys.values()) {
        this.sparkMemory.put(
                memoryComputeKey.getKey(),
                sparkContext.accumulator(ObjectWritable.empty(), memoryComputeKey.getKey(),
                        new MemoryAccumulator<>(memoryComputeKey)));
    }
    this.broadcast = sparkContext.broadcast(Collections.emptyMap());
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:22,代码来源:GraknSparkMemory.java


示例7: SparkMemory

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public SparkMemory(final VertexProgram<?> vertexProgram, final Set<MapReduce> mapReducers, final JavaSparkContext sparkContext) {
    if (null != vertexProgram) {
        for (final MemoryComputeKey key : vertexProgram.getMemoryComputeKeys()) {
            this.memoryComputeKeys.put(key.getKey(), key);
        }
    }
    for (final MapReduce mapReduce : mapReducers) {
        this.memoryComputeKeys.put(mapReduce.getMemoryKey(), MemoryComputeKey.of(mapReduce.getMemoryKey(), Operator.assign, false, false));
    }
    for (final MemoryComputeKey memoryComputeKey : this.memoryComputeKeys.values()) {
        this.sparkMemory.put(
                memoryComputeKey.getKey(),
                sparkContext.accumulator(ObjectWritable.empty(), memoryComputeKey.getKey(), new MemoryAccumulator<>(memoryComputeKey)));
    }
    this.broadcast = sparkContext.broadcast(Collections.emptyMap());
}
 
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:17,代码来源:SparkMemory.java


示例8: shouldEndlessBulkResultsWithNullObject

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Test
public void shouldEndlessBulkResultsWithNullObject() {
    long total = 0;
    final List<Tuple2<MapReduce.NullObject, Long>> numbers = new ArrayList<>();
    for (long i = 0; i < 5000000; i++) {
        total = total + i;
        numbers.add(new Tuple2<>(MapReduce.NullObject.instance(), i));
    }

    final CombineIterator<MapReduce.NullObject, Long, MapReduce.NullObject, Long> combineIterator = new CombineIterator<>(new MapReduceB(), numbers.iterator());
    assertTrue(combineIterator.hasNext());
    final Tuple2<MapReduce.NullObject, Long> tuple = combineIterator.next();
    assertFalse(combineIterator.hasNext());
    assertEquals(MapReduce.NullObject.instance(), tuple._1());
    assertEquals(total, tuple._2().longValue());
}
 
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:17,代码来源:CombineIteratorTest.java


示例9: process

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public void process(TitanVertex vertex, ScanMetrics metrics) {
    PreloadedVertex v = (PreloadedVertex) vertex;
    if (vertexMemory != null) {
        VertexMemoryHandler vh = new VertexMemoryHandler(vertexMemory, v);
        v.setPropertyMixing(vh);
    }
    v.setAccessCheck(MAPREDUCE_CHECK);
    if (idManager.isPartitionedVertex(v.longId()) && !idManager.isCanonicalVertexId(v.longId())) {
        return; //Only consider the canonical partition vertex representative
    } else {
        for (Map.Entry<MapReduce, FulgoraMapEmitter> mapJob : mapJobs.entrySet()) {
            MapReduce job = mapJob.getKey();
            try {
                job.map(v, mapJob.getValue());
                metrics.incrementCustom(MAP_JOB_SUCCESS);
            } catch (Throwable ex) {
                log.error("Encountered exception executing map job [" + job + "] on vertex [" + vertex + "]:", ex);
                metrics.incrementCustom(MAP_JOB_FAILURE);
            }
        }
    }
}
 
开发者ID:graben1437,项目名称:titan1withtp3.1,代码行数:24,代码来源:VertexMapJob.java


示例10: executeMapReduce

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public void executeMapReduce(Consumer<MapReduce<?, ?, ?, ?, R>> worker) {
for (int i = 0; i < numberOfWorkers; i++) {
    completionService.submit(() -> {
	@SuppressWarnings("unchecked")
	MapReduce<?, ?, ?, ?, R> mr = mapReducePool.take();
	worker.accept(mr);
	mapReducePool.offer(mr);
	return null;
    });
}
for (int i = 0; i < numberOfWorkers; i++) {
    try {
	completionService.take().get();
    } catch (Exception e) {
	throw new IllegalStateException(e.getMessage(), e);
    }
}
   }
 
开发者ID:PureSolTechnologies,项目名称:DuctileDB,代码行数:19,代码来源:DuctileWorkerPool.java


示例11: complete

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
protected void complete(final MapReduce<?, ?, OK, OV, ?> mapReduce) {
    if (mapReduce.getReduceKeySort().isPresent()) {
        final Comparator<OK> comparator = mapReduce.getReduceKeySort().get();
        final List<KeyValue<OK, OV>> list = new ArrayList<>(this.reduceQueue);
        Collections.sort(list, Comparator.comparing(KeyValue::getKey, comparator));
        this.reduceQueue.clear();
        this.reduceQueue.addAll(list);
    }
}
 
开发者ID:ShiftLeftSecurity,项目名称:tinkergraph-gremlin,代码行数:10,代码来源:TinkerReduceEmitter.java


示例12: TinkerMemory

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public TinkerMemory(final VertexProgram<?> vertexProgram, final Set<MapReduce> mapReducers) {
    this.currentMap = new ConcurrentHashMap<>();
    this.previousMap = new ConcurrentHashMap<>();
    if (null != vertexProgram) {
        for (final MemoryComputeKey memoryComputeKey : vertexProgram.getMemoryComputeKeys()) {
            this.memoryKeys.put(memoryComputeKey.getKey(), memoryComputeKey);
        }
    }
    for (final MapReduce mapReduce : mapReducers) {
        this.memoryKeys.put(mapReduce.getMemoryKey(), MemoryComputeKey.of(mapReduce.getMemoryKey(), Operator.assign, false, false));
    }
}
 
开发者ID:ShiftLeftSecurity,项目名称:tinkergraph-gremlin,代码行数:13,代码来源:TinkerMemory.java


示例13: execute

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Double> execute() {
    LOGGER.info("StdMapReduce is called");
    long startTime = System.currentTimeMillis();

    initSubGraph();
    getAllSubTypes();

    AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
    if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();

    Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
    Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);

    ComputerResult result = getGraphComputer().compute(
            new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
            new StdMapReduce(statisticsResourceLabelIds, dataType,
                    DegreeVertexProgram.DEGREE),
            allSubLabelIds);
    Map<Serializable, Map<String, Double>> std = result.memory().get(StdMapReduce.class.getName());
    Map<String, Double> stdTuple = std.get(MapReduce.NullObject.instance());
    double squareSum = stdTuple.get(StdMapReduce.SQUARE_SUM);
    double sum = stdTuple.get(StdMapReduce.SUM);
    double count = stdTuple.get(StdMapReduce.COUNT);

    double finalResult = Math.sqrt(squareSum / count - (sum / count) * (sum / count));
    LOGGER.debug("Std = " + finalResult);

    LOGGER.info("StdMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
    return Optional.of(finalResult);
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:32,代码来源:StdQueryImpl.java


示例14: execute

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public Optional<Double> execute() {
    LOGGER.info("MeanMapReduce is called");
    long startTime = System.currentTimeMillis();

    initSubGraph();
    getAllSubTypes();

    AttributeType.DataType dataType = getDataTypeOfSelectedResourceTypes();
    if (!selectedResourceTypesHaveInstance(statisticsResourceLabels)) return Optional.empty();
    Set<LabelId> allSubLabelIds = convertLabelsToIds(getCombinedSubTypes());
    Set<LabelId> statisticsResourceLabelIds = convertLabelsToIds(statisticsResourceLabels);

    ComputerResult result = getGraphComputer().compute(
            new DegreeStatisticsVertexProgram(statisticsResourceLabelIds),
            new MeanMapReduce(statisticsResourceLabelIds, dataType,
                    DegreeVertexProgram.DEGREE),
            allSubLabelIds);
    Map<Serializable, Map<String, Double>> mean = result.memory().get(MeanMapReduce.class.getName());
    Map<String, Double> meanPair = mean.get(MapReduce.NullObject.instance());

    double finalResult = meanPair.get(MeanMapReduce.SUM) / meanPair.get(MeanMapReduce.COUNT);
    LOGGER.debug("Mean = " + finalResult);

    LOGGER.info("MeanMapReduce is done in " + (System.currentTimeMillis() - startTime) + " ms");
    return Optional.of(finalResult);
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:28,代码来源:MeanQueryImpl.java


示例15: clone

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@SuppressWarnings("unchecked")
@Override
public MapReduce<Serializable, T, Serializable, T, Map<Serializable, T>> clone() {
    try {
        return (GraknMapReduce) super.clone();
    } catch (final CloneNotSupportedException e) {
        throw CommonUtil.unreachableStatement(e);
    }
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:10,代码来源:GraknMapReduce.java


示例16: executeMap

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public static <K, V> JavaPairRDD<K, V> executeMap(
        final JavaPairRDD<Object, VertexWritable> graphRDD, final MapReduce<K, V, ?, ?, ?> mapReduce,
        final Configuration graphComputerConfiguration) {
    JavaPairRDD<K, V> mapRDD = graphRDD.mapPartitionsToPair(partitionIterator -> {
        KryoShimServiceLoader.applyConfiguration(graphComputerConfiguration);
        return () -> new MapIterator<>(MapReduce.<MapReduce<K, V, ?, ?, ?>>createMapReduce(HadoopGraph.open(graphComputerConfiguration), graphComputerConfiguration), partitionIterator);
    });
    if (mapReduce.getMapKeySort().isPresent()){
        mapRDD = mapRDD.sortByKey(mapReduce.getMapKeySort().get(), true, 1);}
    return mapRDD;
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:12,代码来源:GraknSparkExecutor.java


示例17: executeCombine

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public static <K, V, OK, OV> JavaPairRDD<OK, OV> executeCombine(final JavaPairRDD<K, V> mapRDD,
                                                                final Configuration graphComputerConfiguration) {
    return mapRDD.mapPartitionsToPair(partitionIterator -> {
        KryoShimServiceLoader.applyConfiguration(graphComputerConfiguration);
        return () -> new CombineIterator<>(MapReduce.<MapReduce<K, V, OK, OV, ?>>createMapReduce(HadoopGraph.open(graphComputerConfiguration), graphComputerConfiguration), partitionIterator);
    });
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:8,代码来源:GraknSparkExecutor.java


示例18: executeReduce

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
public static <K, V, OK, OV> JavaPairRDD<OK, OV> executeReduce(
        final JavaPairRDD<K, V> mapOrCombineRDD, final MapReduce<K, V, OK, OV, ?> mapReduce,
        final Configuration graphComputerConfiguration) {
    JavaPairRDD<OK, OV> reduceRDD = mapOrCombineRDD.groupByKey().mapPartitionsToPair(partitionIterator -> {
        KryoShimServiceLoader.applyConfiguration(graphComputerConfiguration);
        return () -> new ReduceIterator<>(MapReduce.<MapReduce<K, V, OK, OV, ?>>createMapReduce(HadoopGraph.open(graphComputerConfiguration), graphComputerConfiguration), partitionIterator);
    });
    if (mapReduce.getReduceKeySort().isPresent()){
        reduceRDD = reduceRDD.sortByKey(mapReduce.getReduceKeySort().get(), true, 1);}
    return reduceRDD;
}
 
开发者ID:graknlabs,项目名称:grakn,代码行数:12,代码来源:GraknSparkExecutor.java


示例19: setup

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public void setup(final Reducer<ObjectWritable, ObjectWritable, ObjectWritable, ObjectWritable>.Context context) {
    final Configuration apacheConfiguration = ConfUtil.makeApacheConfiguration(context.getConfiguration());
    KryoShimServiceLoader.applyConfiguration(apacheConfiguration);
    this.mapReduce = MapReduce.createMapReduce(HadoopGraph.open(apacheConfiguration), apacheConfiguration);
    this.mapReduce.workerStart(MapReduce.Stage.REDUCE);
}
 
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:8,代码来源:HadoopReduce.java


示例20: setup

import org.apache.tinkerpop.gremlin.process.computer.MapReduce; //导入依赖的package包/类
@Override
public void setup(final Reducer<ObjectWritable, ObjectWritable, ObjectWritable, ObjectWritable>.Context context) {
    final Configuration apacheConfiguration = ConfUtil.makeApacheConfiguration(context.getConfiguration());
    KryoShimServiceLoader.applyConfiguration(apacheConfiguration);
    this.mapReduce = MapReduce.createMapReduce(HadoopGraph.open(apacheConfiguration), apacheConfiguration);
    this.mapReduce.workerStart(MapReduce.Stage.COMBINE);
}
 
开发者ID:PKUSilvester,项目名称:LiteGraph,代码行数:8,代码来源:HadoopCombine.java



注:本文中的org.apache.tinkerpop.gremlin.process.computer.MapReduce类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java SnippetType类代码示例发布时间:2022-05-23
下一篇:
Java ShaderProvider类代码示例发布时间:2022-05-23
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap