• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    公众号
登陆 注册

Java EventType类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中org.apache.hadoop.mapreduce.jobhistory.EventType类的典型用法代码示例。如果您正苦于以下问题：Java EventType类的具体用法？Java EventType怎么用？Java EventType使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



EventType类属于org.apache.hadoop.mapreduce.jobhistory包，在下文中一共展示了EventType类的14个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: createStubbedJob

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
private static StubbedJob createStubbedJob(Configuration conf,
    Dispatcher dispatcher, int numSplits, AppContext appContext) {
  JobID jobID = JobID.forName("job_1234567890000_0001");
  JobId jobId = TypeConverter.toYarn(jobID);
  if (appContext == null) {
    appContext = mock(AppContext.class);
    when(appContext.hasSuccessfullyUnregistered()).thenReturn(true);
  }
  StubbedJob job = new StubbedJob(jobId,
      ApplicationAttemptId.newInstance(ApplicationId.newInstance(0, 0), 0),
      conf,dispatcher.getEventHandler(), true, "somebody", numSplits, appContext);
  dispatcher.register(JobEventType.class, job);
  EventHandler mockHandler = mock(EventHandler.class);
  dispatcher.register(TaskEventType.class, mockHandler);
  dispatcher.register(org.apache.hadoop.mapreduce.jobhistory.EventType.class,
      mockHandler);
  dispatcher.register(JobFinishEvent.Type.class, mockHandler);
  return job;
}
开发者ID:naver，项目名称:hadoop，代码行数:20，代码来源:TestJobImpl.java


示例2: handleEvent

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Override
public void handleEvent(HistoryEvent event)  { 
  EventType type = event.getEventType();

  switch (type) {
  case MAP_ATTEMPT_FINISHED:
    handleMapAttemptFinishedEvent((MapAttemptFinishedEvent) event);
super.handleEvent(event);
    break;
  case REDUCE_ATTEMPT_FINISHED:
    handleReduceAttemptFinishedEvent((ReduceAttemptFinishedEvent) event);
super.handleEvent(event);
    break;
  default:
  	super.handleEvent(event);
    break;
  }
}
开发者ID:Impetus，项目名称:jumbune，代码行数:19，代码来源:DecoratedJobHistoryParser.java


示例3: testJobNoTasks

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Test
public void testJobNoTasks() {
  Configuration conf = new Configuration();
  conf.setInt(MRJobConfig.NUM_REDUCES, 0);
  conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
  conf.set(MRJobConfig.WORKFLOW_ID, "testId");
  conf.set(MRJobConfig.WORKFLOW_NAME, "testName");
  conf.set(MRJobConfig.WORKFLOW_NODE_NAME, "testNodeName");
  conf.set(MRJobConfig.WORKFLOW_ADJACENCY_PREFIX_STRING + "key1", "value1");
  conf.set(MRJobConfig.WORKFLOW_ADJACENCY_PREFIX_STRING + "key2", "value2");
  conf.set(MRJobConfig.WORKFLOW_TAGS, "tag1,tag2");
  
 
  AsyncDispatcher dispatcher = new AsyncDispatcher();
  dispatcher.init(conf);
  dispatcher.start();
  OutputCommitter committer = mock(OutputCommitter.class);
  CommitterEventHandler commitHandler =
      createCommitterEventHandler(dispatcher, committer);
  commitHandler.init(conf);
  commitHandler.start();

  JobSubmittedEventHandler jseHandler = new JobSubmittedEventHandler("testId",
      "testName", "testNodeName", "\"key2\"=\"value2\" \"key1\"=\"value1\" ",
      "tag1,tag2");
  dispatcher.register(EventType.class, jseHandler);
  JobImpl job = createStubbedJob(conf, dispatcher, 0, null);
  job.handle(new JobEvent(job.getID(), JobEventType.JOB_INIT));
  assertJobState(job, JobStateInternal.INITED);
  job.handle(new JobStartEvent(job.getID()));
  assertJobState(job, JobStateInternal.SUCCEEDED);
  dispatcher.stop();
  commitHandler.stop();
  try {
    Assert.assertTrue(jseHandler.getAssertValue());
  } catch (InterruptedException e) {
    Assert.fail("Workflow related attributes are not tested properly");
  }
}
开发者ID:naver，项目名称:hadoop，代码行数:40，代码来源:TestJobImpl.java


示例4: handle

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Override
public void handle(JobHistoryEvent jhEvent) {
  if (jhEvent.getType() != EventType.JOB_SUBMITTED) {
    return;
  }
  JobSubmittedEvent jsEvent = (JobSubmittedEvent) jhEvent.getHistoryEvent();
  if (!workflowId.equals(jsEvent.getWorkflowId())) {
    setAssertValue(false);
    return;
  }
  if (!workflowName.equals(jsEvent.getWorkflowName())) {
    setAssertValue(false);
    return;
  }
  if (!workflowNodeName.equals(jsEvent.getWorkflowNodeName())) {
    setAssertValue(false);
    return;
  }
 
  String[] wrkflowAdj = workflowAdjacencies.split(" ");
  String[] jswrkflowAdj = jsEvent.getWorkflowAdjacencies().split(" ");
  Arrays.sort(wrkflowAdj);
  Arrays.sort(jswrkflowAdj);
  if (!Arrays.equals(wrkflowAdj, jswrkflowAdj)) {
    setAssertValue(false);
    return;
  }
  if (!workflowTags.equals(jsEvent.getWorkflowTags())) {
    setAssertValue(false);
    return;
  }
  setAssertValue(true);
}
开发者ID:naver，项目名称:hadoop，代码行数:34，代码来源:TestJobImpl.java


示例5: createJobHistoryHandler

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Override
protected EventHandler<JobHistoryEvent> createJobHistoryHandler(
    AppContext context) {
  return new JobHistoryEventHandler(context, getStartCount()) {
    @Override
    public void handle(JobHistoryEvent event) {
      if (event.getHistoryEvent().getEventType() == EventType.JOB_INITED) {
        JobInitedEvent jie = (JobInitedEvent) event.getHistoryEvent();
        jobLaunchTime.set(jie.getLaunchTime());
      }
      super.handle(event);
    }
  };
}
开发者ID:aliyun-beta，项目名称:aliyun-oss-hadoop-fs，代码行数:15，代码来源:TestMRAppMaster.java


示例6: testJobNoTasks

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Test
public void testJobNoTasks() {
  Configuration conf = new Configuration();
  conf.setInt(MRJobConfig.NUM_REDUCES, 0);
  conf.set(MRJobConfig.MR_AM_STAGING_DIR, stagingDir);
  conf.set(MRJobConfig.WORKFLOW_ID, "testId");
  conf.set(MRJobConfig.WORKFLOW_NAME, "testName");
  conf.set(MRJobConfig.WORKFLOW_NODE_NAME, "testNodeName");
  conf.set(MRJobConfig.WORKFLOW_ADJACENCY_PREFIX_STRING + "key1", "value1");
  conf.set(MRJobConfig.WORKFLOW_ADJACENCY_PREFIX_STRING + "key2", "value2");
  conf.set(MRJobConfig.WORKFLOW_TAGS, "tag1,tag2");
  
 
  AsyncDispatcher dispatcher = new AsyncDispatcher();
  dispatcher.init(conf);
  dispatcher.start();
  OutputCommitter committer = mock(OutputCommitter.class);
  CommitterEventHandler commitHandler =
      createCommitterEventHandler(dispatcher, committer);
  commitHandler.init(conf);
  commitHandler.start();

  JobSubmittedEventHandler jseHandler = new JobSubmittedEventHandler("testId",
      "testName", "testNodeName", "\"key2\"=\"value2\" \"key1\"=\"value1\" ",
      "tag1,tag2");
  dispatcher.register(EventType.class, jseHandler);
  JobImpl job = createStubbedJob(conf, dispatcher, 0);
  job.handle(new JobEvent(job.getID(), JobEventType.JOB_INIT));
  assertJobState(job, JobStateInternal.INITED);
  job.handle(new JobStartEvent(job.getID()));
  assertJobState(job, JobStateInternal.SUCCEEDED);
  dispatcher.stop();
  commitHandler.stop();
  try {
    Assert.assertTrue(jseHandler.getAssertValue());
  } catch (InterruptedException e) {
    Assert.fail("Workflow related attributes are not tested properly");
  }
}
开发者ID:ict-carch，项目名称:hadoop-plus，代码行数:40，代码来源:TestJobImpl.java


示例7: createStubbedJob

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
private static StubbedJob createStubbedJob(Configuration conf,
    Dispatcher dispatcher, int numSplits) {
  JobID jobID = JobID.forName("job_1234567890000_0001");
  JobId jobId = TypeConverter.toYarn(jobID);
  StubbedJob job = new StubbedJob(jobId,
      ApplicationAttemptId.newInstance(ApplicationId.newInstance(0, 0), 0),
      conf,dispatcher.getEventHandler(), true, "somebody", numSplits);
  dispatcher.register(JobEventType.class, job);
  EventHandler mockHandler = mock(EventHandler.class);
  dispatcher.register(TaskEventType.class, mockHandler);
  dispatcher.register(org.apache.hadoop.mapreduce.jobhistory.EventType.class,
      mockHandler);
  dispatcher.register(JobFinishEvent.Type.class, mockHandler);
  return job;
}
开发者ID:ict-carch，项目名称:hadoop-plus，代码行数:16，代码来源:TestJobImpl.java


示例8: readJustAMInfos

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
private List<AMInfo> readJustAMInfos() {
  List<AMInfo> amInfos = new ArrayList<AMInfo>();
  FSDataInputStream inputStream = null;
  try {
    inputStream = getPreviousJobHistoryStream(getConfig(), appAttemptID);
    EventReader jobHistoryEventReader = new EventReader(inputStream);

    // All AMInfos are contiguous. Track when the first AMStartedEvent
    // appears.
    boolean amStartedEventsBegan = false;

    HistoryEvent event;
    while ((event = jobHistoryEventReader.getNextEvent()) != null) {
      if (event.getEventType() == EventType.AM_STARTED) {
        if (!amStartedEventsBegan) {
          // First AMStartedEvent.
          amStartedEventsBegan = true;
        }
        AMStartedEvent amStartedEvent = (AMStartedEvent) event;
        amInfos.add(MRBuilderUtils.newAMInfo(
          amStartedEvent.getAppAttemptId(), amStartedEvent.getStartTime(),
          amStartedEvent.getContainerId(),
          StringInterner.weakIntern(amStartedEvent.getNodeManagerHost()),
          amStartedEvent.getNodeManagerPort(),
          amStartedEvent.getNodeManagerHttpPort()));
      } else if (amStartedEventsBegan) {
        // This means AMStartedEvents began and this event is a
        // non-AMStarted event.
        // No need to continue reading all the other events.
        break;
      }
    }
  } catch (IOException e) {
    LOG.warn("Could not parse the old history file. "
        + "Will not have old AMinfos ", e);
  } finally {
    if (inputStream != null) {
      IOUtils.closeQuietly(inputStream);
    }
  }
  return amInfos;
}
开发者ID:naver，项目名称:hadoop，代码行数:43，代码来源:MRAppMaster.java


示例9: testRecoverySuccessAttempt

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Test
public void testRecoverySuccessAttempt() {
  LOG.info("--- START: testRecoverySuccessAttempt ---");

  long clusterTimestamp = System.currentTimeMillis();
  EventHandler mockEventHandler = mock(EventHandler.class);
  MapTaskImpl recoverMapTask = getMockMapTask(clusterTimestamp,
      mockEventHandler);

  TaskId taskId = recoverMapTask.getID();
  JobID jobID = new JobID(Long.toString(clusterTimestamp), 1);
  TaskID taskID = new TaskID(jobID,
      org.apache.hadoop.mapreduce.TaskType.MAP, taskId.getId());

  //Mock up the TaskAttempts
  Map<TaskAttemptID, TaskAttemptInfo> mockTaskAttempts =
      new HashMap<TaskAttemptID, TaskAttemptInfo>();

  TaskAttemptID taId1 = new TaskAttemptID(taskID, 2);
  TaskAttemptInfo mockTAinfo1 = getMockTaskAttemptInfo(taId1,
      TaskAttemptState.SUCCEEDED);
  mockTaskAttempts.put(taId1, mockTAinfo1);

  TaskAttemptID taId2 = new TaskAttemptID(taskID, 1);
  TaskAttemptInfo mockTAinfo2 = getMockTaskAttemptInfo(taId2,
      TaskAttemptState.FAILED);
  mockTaskAttempts.put(taId2, mockTAinfo2);

  OutputCommitter mockCommitter = mock (OutputCommitter.class);
  TaskInfo mockTaskInfo = mock(TaskInfo.class);
  when(mockTaskInfo.getTaskStatus()).thenReturn("SUCCEEDED");
  when(mockTaskInfo.getTaskId()).thenReturn(taskID);
  when(mockTaskInfo.getAllTaskAttempts()).thenReturn(mockTaskAttempts);

  recoverMapTask.handle(
      new TaskRecoverEvent(taskId, mockTaskInfo,mockCommitter, true));

  ArgumentCaptor<Event> arg = ArgumentCaptor.forClass(Event.class);
  verify(mockEventHandler,atLeast(1)).handle(
      (org.apache.hadoop.yarn.event.Event) arg.capture());

  Map<TaskAttemptID, TaskAttemptState> finalAttemptStates =
      new HashMap<TaskAttemptID, TaskAttemptState>();
  finalAttemptStates.put(taId1, TaskAttemptState.SUCCEEDED);
  finalAttemptStates.put(taId2, TaskAttemptState.FAILED);

  List<EventType> jobHistoryEvents = new ArrayList<EventType>();
  jobHistoryEvents.add(EventType.TASK_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FINISHED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FAILED);
  jobHistoryEvents.add(EventType.TASK_FINISHED);
  recoveryChecker(recoverMapTask, TaskState.SUCCEEDED, finalAttemptStates,
      arg, jobHistoryEvents, 2L, 1L);
}
开发者ID:naver，项目名称:hadoop，代码行数:57，代码来源:TestRecovery.java


示例10: testRecoveryAllFailAttempts

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Test
public void testRecoveryAllFailAttempts() {
  LOG.info("--- START: testRecoveryAllFailAttempts ---");

  long clusterTimestamp = System.currentTimeMillis();
  EventHandler mockEventHandler = mock(EventHandler.class);
  MapTaskImpl recoverMapTask = getMockMapTask(clusterTimestamp,
      mockEventHandler);

  TaskId taskId = recoverMapTask.getID();
  JobID jobID = new JobID(Long.toString(clusterTimestamp), 1);
  TaskID taskID = new TaskID(jobID,
      org.apache.hadoop.mapreduce.TaskType.MAP, taskId.getId());

  //Mock up the TaskAttempts
  Map<TaskAttemptID, TaskAttemptInfo> mockTaskAttempts =
      new HashMap<TaskAttemptID, TaskAttemptInfo>();

  TaskAttemptID taId1 = new TaskAttemptID(taskID, 2);
  TaskAttemptInfo mockTAinfo1 = getMockTaskAttemptInfo(taId1,
      TaskAttemptState.FAILED);
  mockTaskAttempts.put(taId1, mockTAinfo1);

  TaskAttemptID taId2 = new TaskAttemptID(taskID, 1);
  TaskAttemptInfo mockTAinfo2 = getMockTaskAttemptInfo(taId2,
      TaskAttemptState.FAILED);
  mockTaskAttempts.put(taId2, mockTAinfo2);

  OutputCommitter mockCommitter = mock (OutputCommitter.class);

  TaskInfo mockTaskInfo = mock(TaskInfo.class);
  when(mockTaskInfo.getTaskStatus()).thenReturn("FAILED");
  when(mockTaskInfo.getTaskId()).thenReturn(taskID);
  when(mockTaskInfo.getAllTaskAttempts()).thenReturn(mockTaskAttempts);

  recoverMapTask.handle(
      new TaskRecoverEvent(taskId, mockTaskInfo, mockCommitter, true));

  ArgumentCaptor<Event> arg = ArgumentCaptor.forClass(Event.class);
  verify(mockEventHandler,atLeast(1)).handle(
      (org.apache.hadoop.yarn.event.Event) arg.capture());

  Map<TaskAttemptID, TaskAttemptState> finalAttemptStates =
      new HashMap<TaskAttemptID, TaskAttemptState>();
  finalAttemptStates.put(taId1, TaskAttemptState.FAILED);
  finalAttemptStates.put(taId2, TaskAttemptState.FAILED);

  List<EventType> jobHistoryEvents = new ArrayList<EventType>();
  jobHistoryEvents.add(EventType.TASK_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FAILED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FAILED);
  jobHistoryEvents.add(EventType.TASK_FAILED);
  recoveryChecker(recoverMapTask, TaskState.FAILED, finalAttemptStates,
      arg, jobHistoryEvents, 2L, 2L);
}
开发者ID:naver，项目名称:hadoop，代码行数:58，代码来源:TestRecovery.java


示例11: testRecoveryTaskSuccessAllAttemptsFail

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Test
public void testRecoveryTaskSuccessAllAttemptsFail() {
  LOG.info("--- START:  testRecoveryTaskSuccessAllAttemptsFail ---");

  long clusterTimestamp = System.currentTimeMillis();
  EventHandler mockEventHandler = mock(EventHandler.class);
  MapTaskImpl recoverMapTask = getMockMapTask(clusterTimestamp,
      mockEventHandler);

  TaskId taskId = recoverMapTask.getID();
  JobID jobID = new JobID(Long.toString(clusterTimestamp), 1);
  TaskID taskID = new TaskID(jobID,
      org.apache.hadoop.mapreduce.TaskType.MAP, taskId.getId());

  //Mock up the TaskAttempts
  Map<TaskAttemptID, TaskAttemptInfo> mockTaskAttempts =
      new HashMap<TaskAttemptID, TaskAttemptInfo>();

  TaskAttemptID taId1 = new TaskAttemptID(taskID, 2);
  TaskAttemptInfo mockTAinfo1 = getMockTaskAttemptInfo(taId1,
      TaskAttemptState.FAILED);
  mockTaskAttempts.put(taId1, mockTAinfo1);

  TaskAttemptID taId2 = new TaskAttemptID(taskID, 1);
  TaskAttemptInfo mockTAinfo2 = getMockTaskAttemptInfo(taId2,
      TaskAttemptState.FAILED);
  mockTaskAttempts.put(taId2, mockTAinfo2);

  OutputCommitter mockCommitter = mock (OutputCommitter.class);
  TaskInfo mockTaskInfo = mock(TaskInfo.class);
  when(mockTaskInfo.getTaskStatus()).thenReturn("SUCCEEDED");
  when(mockTaskInfo.getTaskId()).thenReturn(taskID);
  when(mockTaskInfo.getAllTaskAttempts()).thenReturn(mockTaskAttempts);

  recoverMapTask.handle(
      new TaskRecoverEvent(taskId, mockTaskInfo, mockCommitter, true));

  ArgumentCaptor<Event> arg = ArgumentCaptor.forClass(Event.class);
  verify(mockEventHandler,atLeast(1)).handle(
      (org.apache.hadoop.yarn.event.Event) arg.capture());

  Map<TaskAttemptID, TaskAttemptState> finalAttemptStates =
      new HashMap<TaskAttemptID, TaskAttemptState>();
  finalAttemptStates.put(taId1, TaskAttemptState.FAILED);
  finalAttemptStates.put(taId2, TaskAttemptState.FAILED);
  // check for one new attempt launched since successful attempt not found
  TaskAttemptID taId3 = new TaskAttemptID(taskID, 2000);
  finalAttemptStates.put(taId3, TaskAttemptState.NEW);

  List<EventType> jobHistoryEvents = new ArrayList<EventType>();
  jobHistoryEvents.add(EventType.TASK_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FAILED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FAILED);
  recoveryChecker(recoverMapTask, TaskState.RUNNING, finalAttemptStates,
      arg, jobHistoryEvents, 2L, 2L);
}
开发者ID:naver，项目名称:hadoop，代码行数:59，代码来源:TestRecovery.java


示例12: testRecoveryTaskSuccessAllAttemptsSucceed

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Test
public void testRecoveryTaskSuccessAllAttemptsSucceed() {
  LOG.info("--- START:  testRecoveryTaskSuccessAllAttemptsFail ---");

  long clusterTimestamp = System.currentTimeMillis();
  EventHandler mockEventHandler = mock(EventHandler.class);
  MapTaskImpl recoverMapTask = getMockMapTask(clusterTimestamp,
      mockEventHandler);

  TaskId taskId = recoverMapTask.getID();
  JobID jobID = new JobID(Long.toString(clusterTimestamp), 1);
  TaskID taskID = new TaskID(jobID,
      org.apache.hadoop.mapreduce.TaskType.MAP, taskId.getId());

  //Mock up the TaskAttempts
  Map<TaskAttemptID, TaskAttemptInfo> mockTaskAttempts =
      new HashMap<TaskAttemptID, TaskAttemptInfo>();

  TaskAttemptID taId1 = new TaskAttemptID(taskID, 2);
  TaskAttemptInfo mockTAinfo1 = getMockTaskAttemptInfo(taId1,
      TaskAttemptState.SUCCEEDED);
  mockTaskAttempts.put(taId1, mockTAinfo1);

  TaskAttemptID taId2 = new TaskAttemptID(taskID, 1);
  TaskAttemptInfo mockTAinfo2 = getMockTaskAttemptInfo(taId2,
      TaskAttemptState.SUCCEEDED);
  mockTaskAttempts.put(taId2, mockTAinfo2);

  OutputCommitter mockCommitter = mock (OutputCommitter.class);
  TaskInfo mockTaskInfo = mock(TaskInfo.class);
  when(mockTaskInfo.getTaskStatus()).thenReturn("SUCCEEDED");
  when(mockTaskInfo.getTaskId()).thenReturn(taskID);
  when(mockTaskInfo.getAllTaskAttempts()).thenReturn(mockTaskAttempts);

  recoverMapTask.handle(
      new TaskRecoverEvent(taskId, mockTaskInfo, mockCommitter, true));

  ArgumentCaptor<Event> arg = ArgumentCaptor.forClass(Event.class);
  verify(mockEventHandler,atLeast(1)).handle(
      (org.apache.hadoop.yarn.event.Event) arg.capture());

  Map<TaskAttemptID, TaskAttemptState> finalAttemptStates =
      new HashMap<TaskAttemptID, TaskAttemptState>();
  finalAttemptStates.put(taId1, TaskAttemptState.SUCCEEDED);
  finalAttemptStates.put(taId2, TaskAttemptState.SUCCEEDED);

  List<EventType> jobHistoryEvents = new ArrayList<EventType>();
  jobHistoryEvents.add(EventType.TASK_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FINISHED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_FINISHED);
  jobHistoryEvents.add(EventType.TASK_FINISHED);
  recoveryChecker(recoverMapTask, TaskState.SUCCEEDED, finalAttemptStates,
      arg, jobHistoryEvents, 2L, 0L);
}
开发者ID:naver，项目名称:hadoop，代码行数:57，代码来源:TestRecovery.java


示例13: testRecoveryAllAttemptsKilled

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
@Test
public void testRecoveryAllAttemptsKilled() {
  LOG.info("--- START:  testRecoveryAllAttemptsKilled ---");

  long clusterTimestamp = System.currentTimeMillis();
  EventHandler mockEventHandler = mock(EventHandler.class);
  MapTaskImpl recoverMapTask = getMockMapTask(clusterTimestamp,
      mockEventHandler);

  TaskId taskId = recoverMapTask.getID();
  JobID jobID = new JobID(Long.toString(clusterTimestamp), 1);
  TaskID taskID = new TaskID(jobID,
      org.apache.hadoop.mapreduce.TaskType.MAP, taskId.getId());

  //Mock up the TaskAttempts
  Map<TaskAttemptID, TaskAttemptInfo> mockTaskAttempts =
      new HashMap<TaskAttemptID, TaskAttemptInfo>();
  TaskAttemptID taId1 = new TaskAttemptID(taskID, 2);
  TaskAttemptInfo mockTAinfo1 = getMockTaskAttemptInfo(taId1,
      TaskAttemptState.KILLED);
  mockTaskAttempts.put(taId1, mockTAinfo1);

  TaskAttemptID taId2 = new TaskAttemptID(taskID, 1);
  TaskAttemptInfo mockTAinfo2 = getMockTaskAttemptInfo(taId2,
      TaskAttemptState.KILLED);
  mockTaskAttempts.put(taId2, mockTAinfo2);

  OutputCommitter mockCommitter = mock (OutputCommitter.class);
  TaskInfo mockTaskInfo = mock(TaskInfo.class);
  when(mockTaskInfo.getTaskStatus()).thenReturn("KILLED");
  when(mockTaskInfo.getTaskId()).thenReturn(taskID);
  when(mockTaskInfo.getAllTaskAttempts()).thenReturn(mockTaskAttempts);

  recoverMapTask.handle(
      new TaskRecoverEvent(taskId, mockTaskInfo, mockCommitter, true));

  ArgumentCaptor<Event> arg = ArgumentCaptor.forClass(Event.class);
  verify(mockEventHandler,atLeast(1)).handle(
      (org.apache.hadoop.yarn.event.Event) arg.capture());

  Map<TaskAttemptID, TaskAttemptState> finalAttemptStates =
      new HashMap<TaskAttemptID, TaskAttemptState>();
  finalAttemptStates.put(taId1, TaskAttemptState.KILLED);
  finalAttemptStates.put(taId2, TaskAttemptState.KILLED);

  List<EventType> jobHistoryEvents = new ArrayList<EventType>();
  jobHistoryEvents.add(EventType.TASK_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_KILLED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_STARTED);
  jobHistoryEvents.add(EventType.MAP_ATTEMPT_KILLED);
  jobHistoryEvents.add(EventType.TASK_FAILED);
  recoveryChecker(recoverMapTask, TaskState.KILLED, finalAttemptStates,
      arg, jobHistoryEvents, 2L, 0L);
}
开发者ID:naver，项目名称:hadoop，代码行数:56，代码来源:TestRecovery.java


示例14: recoveryChecker

import org.apache.hadoop.mapreduce.jobhistory.EventType; //导入依赖的package包/类
private void recoveryChecker(MapTaskImpl checkTask, TaskState finalState,
    Map<TaskAttemptID, TaskAttemptState> finalAttemptStates,
    ArgumentCaptor<Event> arg, List<EventType> expectedJobHistoryEvents,
    long expectedMapLaunches, long expectedFailedMaps) {

  assertEquals("Final State of Task", finalState, checkTask.getState());

  Map<TaskAttemptId, TaskAttempt> recoveredAttempts =
      checkTask.getAttempts();
  assertEquals("Expected Number of Task Attempts",
      finalAttemptStates.size(), recoveredAttempts.size());
  for (TaskAttemptID taID : finalAttemptStates.keySet()) {
    assertEquals("Expected Task Attempt State",
        finalAttemptStates.get(taID),
        recoveredAttempts.get(TypeConverter.toYarn(taID)).getState());
  }

  Iterator<Event> ie = arg.getAllValues().iterator();
  int eventNum = 0;
  long totalLaunchedMaps = 0;
  long totalFailedMaps = 0;
  boolean jobTaskEventReceived = false;

  while (ie.hasNext()) {
    Object current = ie.next();
    ++eventNum;
    LOG.info(eventNum + " " + current.getClass().getName());
    if (current instanceof JobHistoryEvent) {
      JobHistoryEvent jhe = (JobHistoryEvent) current;
      LOG.info(expectedJobHistoryEvents.get(0).toString() + " " +
          jhe.getHistoryEvent().getEventType().toString() + " " +
          jhe.getJobID());
      assertEquals(expectedJobHistoryEvents.get(0),
          jhe.getHistoryEvent().getEventType());
      expectedJobHistoryEvents.remove(0);
    }  else if (current instanceof JobCounterUpdateEvent) {
      JobCounterUpdateEvent jcue = (JobCounterUpdateEvent) current;

      LOG.info("JobCounterUpdateEvent "
          + jcue.getCounterUpdates().get(0).getCounterKey()
          + " " + jcue.getCounterUpdates().get(0).getIncrementValue());
      if (jcue.getCounterUpdates().get(0).getCounterKey() ==
          JobCounter.NUM_FAILED_MAPS) {
        totalFailedMaps += jcue.getCounterUpdates().get(0)
            .getIncrementValue();
      } else if (jcue.getCounterUpdates().get(0).getCounterKey() ==
          JobCounter.TOTAL_LAUNCHED_MAPS) {
        totalLaunchedMaps += jcue.getCounterUpdates().get(0)
            .getIncrementValue();
      }
    } else if (current instanceof JobTaskEvent) {
      JobTaskEvent jte = (JobTaskEvent) current;
      assertEquals(jte.getState(), finalState);
      jobTaskEventReceived = true;
    }
  }
  assertTrue(jobTaskEventReceived || (finalState == TaskState.RUNNING));
  assertEquals("Did not process all expected JobHistoryEvents",
      0, expectedJobHistoryEvents.size());
  assertEquals("Expected Map Launches",
      expectedMapLaunches, totalLaunchedMaps);
  assertEquals("Expected Failed Maps",
      expectedFailedMaps, totalFailedMaps);
}
开发者ID:naver，项目名称:hadoop，代码行数:65，代码来源:TestRecovery.java



注：本文中的org.apache.hadoop.mapreduce.jobhistory.EventType类示例整理自Github/MSDocs等源码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。

鲜花
握手
雷人
路过
鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
More+
上一篇：
Java JavaScriptFileType类代码示例发布时间：2022-05-22
下一篇：
Java RegistrationPolicy类代码示例发布时间：2022-05-22
热门推荐
More+
热门话题
More+
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服（服务时间 9:00～18:00）

在线QQ客服
地址：深圳市南山区西丽大学城创智工业园
电邮：jeky_zhao#qq.com
移动电话：139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap