• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    公众号

Java JobReference类代码示例

原作者: [db:作者] 来自: [db:来源] 收藏 邀请

本文整理汇总了Java中com.google.api.services.bigquery.model.JobReference的典型用法代码示例。如果您正苦于以下问题:Java JobReference类的具体用法?Java JobReference怎么用?Java JobReference使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。



JobReference类属于com.google.api.services.bigquery.model包,在下文中一共展示了JobReference类的20个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: cleanup

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
@Override
void cleanup(PassThroughThenCleanup.ContextContainer c) throws Exception {
  PipelineOptions options = c.getPipelineOptions();
  BigQueryOptions bqOptions = options.as(BigQueryOptions.class);
  String jobUuid = c.getJobId();
  final String extractDestinationDir =
      resolveTempLocation(bqOptions.getTempLocation(), "BigQueryExtractTemp", jobUuid);
  final String executingProject = bqOptions.getProject();
  JobReference jobRef =
      new JobReference()
          .setProjectId(executingProject)
          .setJobId(getExtractJobId(createJobIdToken(bqOptions.getJobName(), jobUuid)));

  Job extractJob = getBigQueryServices().getJobService(bqOptions).getJob(jobRef);

  if (extractJob != null) {
    List<ResourceId> extractFiles =
        getExtractFilePaths(extractDestinationDir, extractJob);
    if (extractFiles != null && !extractFiles.isEmpty()) {
      FileSystems.delete(
          extractFiles, MoveOptions.StandardMoveOptions.IGNORE_MISSING_FILES);
    }
  }
}
 
开发者ID:apache,项目名称:beam,代码行数:25,代码来源:BigQueryIO.java


示例2: executeQuery

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
private void executeQuery(
    String executingProject,
    String jobId,
    TableReference destinationTable,
    JobService jobService) throws IOException, InterruptedException {
  JobReference jobRef = new JobReference()
      .setProjectId(executingProject)
      .setJobId(jobId);

  JobConfigurationQuery queryConfig = createBasicQueryConfig()
      .setAllowLargeResults(true)
      .setCreateDisposition("CREATE_IF_NEEDED")
      .setDestinationTable(destinationTable)
      .setPriority("BATCH")
      .setWriteDisposition("WRITE_EMPTY");

  jobService.startQueryJob(jobRef, queryConfig);
  Job job = jobService.pollJob(jobRef, JOB_POLL_MAX_RETRIES);
  if (BigQueryHelpers.parseStatus(job) != Status.SUCCEEDED) {
    throw new IOException(String.format(
        "Query job %s failed, status: %s.", jobId,
        BigQueryHelpers.statusToPrettyString(job.getStatus())));
  }
}
 
开发者ID:apache,项目名称:beam,代码行数:25,代码来源:BigQueryQuerySource.java


示例3: testStartLoadJobSucceeds

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * Tests that {@link BigQueryServicesImpl.JobServiceImpl#startLoadJob} succeeds.
 */
@Test
public void testStartLoadJobSucceeds() throws IOException, InterruptedException {
  Job testJob = new Job();
  JobReference jobRef = new JobReference();
  jobRef.setJobId("jobId");
  jobRef.setProjectId("projectId");
  testJob.setJobReference(jobRef);

  when(response.getContentType()).thenReturn(Json.MEDIA_TYPE);
  when(response.getStatusCode()).thenReturn(200);
  when(response.getContent()).thenReturn(toStream(testJob));

  Sleeper sleeper = new FastNanoClockAndSleeper();
  JobServiceImpl.startJob(
      testJob, new ApiErrorExtractor(), bigquery, sleeper,
      BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff()));

  verify(response, times(1)).getStatusCode();
  verify(response, times(1)).getContent();
  verify(response, times(1)).getContentType();
  expectedLogs.verifyInfo(String.format("Started BigQuery job: %s", jobRef));
}
 
开发者ID:apache,项目名称:beam,代码行数:26,代码来源:BigQueryServicesImplTest.java


示例4: testStartLoadJobSucceedsAlreadyExists

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * Tests that {@link BigQueryServicesImpl.JobServiceImpl#startLoadJob} succeeds
 * with an already exist job.
 */
@Test
public void testStartLoadJobSucceedsAlreadyExists() throws IOException, InterruptedException {
  Job testJob = new Job();
  JobReference jobRef = new JobReference();
  jobRef.setJobId("jobId");
  jobRef.setProjectId("projectId");
  testJob.setJobReference(jobRef);

  when(response.getStatusCode()).thenReturn(409); // 409 means already exists

  Sleeper sleeper = new FastNanoClockAndSleeper();
  JobServiceImpl.startJob(
      testJob, new ApiErrorExtractor(), bigquery, sleeper,
      BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff()));

  verify(response, times(1)).getStatusCode();
  verify(response, times(1)).getContent();
  verify(response, times(1)).getContentType();
  expectedLogs.verifyNotLogged("Started BigQuery job");
}
 
开发者ID:apache,项目名称:beam,代码行数:25,代码来源:BigQueryServicesImplTest.java


示例5: testStartLoadJobRetry

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * Tests that {@link BigQueryServicesImpl.JobServiceImpl#startLoadJob} succeeds with a retry.
 */
@Test
public void testStartLoadJobRetry() throws IOException, InterruptedException {
  Job testJob = new Job();
  JobReference jobRef = new JobReference();
  jobRef.setJobId("jobId");
  jobRef.setProjectId("projectId");
  testJob.setJobReference(jobRef);

  // First response is 403 rate limited, second response has valid payload.
  when(response.getContentType()).thenReturn(Json.MEDIA_TYPE);
  when(response.getStatusCode()).thenReturn(403).thenReturn(200);
  when(response.getContent())
      .thenReturn(toStream(errorWithReasonAndStatus("rateLimitExceeded", 403)))
      .thenReturn(toStream(testJob));

  Sleeper sleeper = new FastNanoClockAndSleeper();
  JobServiceImpl.startJob(
      testJob, new ApiErrorExtractor(), bigquery, sleeper,
      BackOffAdapter.toGcpBackOff(FluentBackoff.DEFAULT.backoff()));

  verify(response, times(2)).getStatusCode();
  verify(response, times(2)).getContent();
  verify(response, times(2)).getContentType();
}
 
开发者ID:apache,项目名称:beam,代码行数:28,代码来源:BigQueryServicesImplTest.java


示例6: testPollJobSucceeds

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * Tests that {@link BigQueryServicesImpl.JobServiceImpl#pollJob} succeeds.
 */
@Test
public void testPollJobSucceeds() throws IOException, InterruptedException {
  Job testJob = new Job();
  testJob.setStatus(new JobStatus().setState("DONE"));

  when(response.getContentType()).thenReturn(Json.MEDIA_TYPE);
  when(response.getStatusCode()).thenReturn(200);
  when(response.getContent()).thenReturn(toStream(testJob));

  BigQueryServicesImpl.JobServiceImpl jobService =
      new BigQueryServicesImpl.JobServiceImpl(bigquery);
  JobReference jobRef = new JobReference()
      .setProjectId("projectId")
      .setJobId("jobId");
  Job job = jobService.pollJob(jobRef, Sleeper.DEFAULT, BackOff.ZERO_BACKOFF);

  assertEquals(testJob, job);
  verify(response, times(1)).getStatusCode();
  verify(response, times(1)).getContent();
  verify(response, times(1)).getContentType();
}
 
开发者ID:apache,项目名称:beam,代码行数:25,代码来源:BigQueryServicesImplTest.java


示例7: testPollJobFailed

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * Tests that {@link BigQueryServicesImpl.JobServiceImpl#pollJob} fails.
 */
@Test
public void testPollJobFailed() throws IOException, InterruptedException {
  Job testJob = new Job();
  testJob.setStatus(new JobStatus().setState("DONE").setErrorResult(new ErrorProto()));

  when(response.getContentType()).thenReturn(Json.MEDIA_TYPE);
  when(response.getStatusCode()).thenReturn(200);
  when(response.getContent()).thenReturn(toStream(testJob));

  BigQueryServicesImpl.JobServiceImpl jobService =
      new BigQueryServicesImpl.JobServiceImpl(bigquery);
  JobReference jobRef = new JobReference()
      .setProjectId("projectId")
      .setJobId("jobId");
  Job job = jobService.pollJob(jobRef, Sleeper.DEFAULT, BackOff.ZERO_BACKOFF);

  assertEquals(testJob, job);
  verify(response, times(1)).getStatusCode();
  verify(response, times(1)).getContent();
  verify(response, times(1)).getContentType();
}
 
开发者ID:apache,项目名称:beam,代码行数:25,代码来源:BigQueryServicesImplTest.java


示例8: testPollJobUnknown

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * Tests that {@link BigQueryServicesImpl.JobServiceImpl#pollJob} returns UNKNOWN.
 */
@Test
public void testPollJobUnknown() throws IOException, InterruptedException {
  Job testJob = new Job();
  testJob.setStatus(new JobStatus());

  when(response.getContentType()).thenReturn(Json.MEDIA_TYPE);
  when(response.getStatusCode()).thenReturn(200);
  when(response.getContent()).thenReturn(toStream(testJob));

  BigQueryServicesImpl.JobServiceImpl jobService =
      new BigQueryServicesImpl.JobServiceImpl(bigquery);
  JobReference jobRef = new JobReference()
      .setProjectId("projectId")
      .setJobId("jobId");
  Job job = jobService.pollJob(jobRef, Sleeper.DEFAULT, BackOff.STOP_BACKOFF);

  assertEquals(null, job);
  verify(response, times(1)).getStatusCode();
  verify(response, times(1)).getContent();
  verify(response, times(1)).getContentType();
}
 
开发者ID:apache,项目名称:beam,代码行数:25,代码来源:BigQueryServicesImplTest.java


示例9: testGetJobSucceeds

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
@Test
public void testGetJobSucceeds() throws Exception {
  Job testJob = new Job();
  testJob.setStatus(new JobStatus());

  when(response.getContentType()).thenReturn(Json.MEDIA_TYPE);
  when(response.getStatusCode()).thenReturn(200);
  when(response.getContent()).thenReturn(toStream(testJob));

  BigQueryServicesImpl.JobServiceImpl jobService =
      new BigQueryServicesImpl.JobServiceImpl(bigquery);
  JobReference jobRef = new JobReference()
      .setProjectId("projectId")
      .setJobId("jobId");
  Job job = jobService.getJob(jobRef, Sleeper.DEFAULT, BackOff.ZERO_BACKOFF);

  assertEquals(testJob, job);
  verify(response, times(1)).getStatusCode();
  verify(response, times(1)).getContent();
  verify(response, times(1)).getContentType();
}
 
开发者ID:apache,项目名称:beam,代码行数:22,代码来源:BigQueryServicesImplTest.java


示例10: testGetJobNotFound

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
@Test
public void testGetJobNotFound() throws Exception {
  when(response.getContentType()).thenReturn(Json.MEDIA_TYPE);
  when(response.getStatusCode()).thenReturn(404);

  BigQueryServicesImpl.JobServiceImpl jobService =
      new BigQueryServicesImpl.JobServiceImpl(bigquery);
  JobReference jobRef = new JobReference()
      .setProjectId("projectId")
      .setJobId("jobId");
  Job job = jobService.getJob(jobRef, Sleeper.DEFAULT, BackOff.ZERO_BACKOFF);

  assertEquals(null, job);
  verify(response, times(1)).getStatusCode();
  verify(response, times(1)).getContent();
  verify(response, times(1)).getContentType();
}
 
开发者ID:apache,项目名称:beam,代码行数:18,代码来源:BigQueryServicesImplTest.java


示例11: startExtractJob

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
@Override
public void startExtractJob(JobReference jobRef, JobConfigurationExtract extractConfig)
    throws InterruptedException, IOException {
  checkArgument(extractConfig.getDestinationFormat().equals("AVRO"),
      "Only extract to AVRO is supported");
  synchronized (allJobs) {
    verifyUniqueJobId(jobRef.getJobId());
    ++numExtractJobCalls;

    Job job = new Job();
    job.setJobReference(jobRef);
    job.setConfiguration(new JobConfiguration().setExtract(extractConfig));
    job.setKind(" bigquery#job");
    job.setStatus(new JobStatus().setState("PENDING"));
    allJobs.put(jobRef.getProjectId(), jobRef.getJobId(), new JobInfo(job));
  }
}
 
开发者ID:apache,项目名称:beam,代码行数:18,代码来源:FakeJobService.java


示例12: checkJobOutcome

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * Returns true if the provided job succeeded, false if it failed, and throws an exception if it
 * is still pending.
 */
private boolean checkJobOutcome() {
  Job job = null;
  String jobRefString =
      toJobReferenceString(new JobReference().setProjectId(projectId).setJobId(jobId));

  try {
    job = bigquery.jobs().get(projectId, jobId).execute();
  } catch (IOException e) {
    // We will throw a new exception because done==false, but first log this exception.
    logger.warning(e, e.getMessage());
  }
  // If job is not yet done, then throw an exception so that we'll return a failing HTTP status
  // code and the task will be retried.
  if (job == null || !job.getStatus().getState().equals("DONE")) {
    throw new NotModifiedException(jobRefString);
  }

  // Check if the job ended with an error.
  if (job.getStatus().getErrorResult() != null) {
    logger.severefmt("Bigquery job failed - %s - %s", jobRefString, job);
    return false;
  }
  logger.infofmt("Bigquery job succeeded - %s", jobRefString);
  return true;
}
 
开发者ID:google,项目名称:nomulus,代码行数:30,代码来源:BigqueryPollJobAction.java


示例13: makeLoadJob

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
private Job makeLoadJob(JobReference jobRef, String sourceUri, String tableId) {
  TableReference tableReference = new TableReference()
      .setProjectId(jobRef.getProjectId())
      .setDatasetId(SNAPSHOTS_DATASET)
      .setTableId(tableId);
  return new Job()
      .setJobReference(jobRef)
      .setConfiguration(new JobConfiguration()
          .setLoad(new JobConfigurationLoad()
              .setWriteDisposition(WriteDisposition.WRITE_EMPTY.toString())
              .setSourceFormat(SourceFormat.DATASTORE_BACKUP.toString())
              .setSourceUris(ImmutableList.of(sourceUri))
              .setDestinationTable(tableReference)));
}
 
开发者ID:google,项目名称:nomulus,代码行数:15,代码来源:LoadSnapshotAction.java


示例14: testSuccess_enqueuePollTask_withChainedTask

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
@Test
public void testSuccess_enqueuePollTask_withChainedTask() throws Exception {
  TaskOptions chainedTask = TaskOptions.Builder
      .withUrl("/_dr/something")
      .method(Method.POST)
      .header("X-Testing", "foo")
      .param("testing", "bar");
  new BigqueryPollJobEnqueuer(ENQUEUER).enqueuePollTask(
      new JobReference().setProjectId(PROJECT_ID).setJobId(JOB_ID),
      chainedTask,
      getQueue(CHAINED_QUEUE_NAME));
  assertTasksEnqueued(BigqueryPollJobAction.QUEUE, newPollJobTaskMatcher("POST"));
  TaskStateInfo taskInfo = getOnlyElement(
      TaskQueueHelper.getQueueInfo(BigqueryPollJobAction.QUEUE).getTaskInfo());
  ByteArrayInputStream taskBodyBytes = new ByteArrayInputStream(taskInfo.getBodyAsBytes());
  TaskOptions taskOptions = (TaskOptions) new ObjectInputStream(taskBodyBytes).readObject();
  assertThat(taskOptions).isEqualTo(chainedTask);
}
 
开发者ID:google,项目名称:nomulus,代码行数:19,代码来源:BigqueryPollJobActionTest.java


示例15: startLoadJob

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * {@inheritDoc}
 *
 * <p>Tries executing the RPC for at most {@code MAX_RPC_RETRIES} times until it succeeds.
 *
 * @throws IOException if it exceeds {@code MAX_RPC_RETRIES} attempts.
 */
@Override
public void startLoadJob(
    JobReference jobRef,
    JobConfigurationLoad loadConfig) throws InterruptedException, IOException {
  Job job = new Job()
      .setJobReference(jobRef)
      .setConfiguration(new JobConfiguration().setLoad(loadConfig));

  startJob(job, errorExtractor, client);
}
 
开发者ID:apache,项目名称:beam,代码行数:18,代码来源:BigQueryServicesImpl.java


示例16: startExtractJob

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * {@inheritDoc}
 *
 * <p>Tries executing the RPC for at most {@code MAX_RPC_RETRIES} times until it succeeds.
 *
 * @throws IOException if it exceeds {@code MAX_RPC_RETRIES} attempts.
 */
@Override
public void startExtractJob(JobReference jobRef, JobConfigurationExtract extractConfig)
    throws InterruptedException, IOException {
  Job job = new Job()
      .setJobReference(jobRef)
      .setConfiguration(
          new JobConfiguration().setExtract(extractConfig));

  startJob(job, errorExtractor, client);
}
 
开发者ID:apache,项目名称:beam,代码行数:18,代码来源:BigQueryServicesImpl.java


示例17: startQueryJob

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * {@inheritDoc}
 *
 * <p>Tries executing the RPC for at most {@code MAX_RPC_RETRIES} times until it succeeds.
 *
 * @throws IOException if it exceeds {@code MAX_RPC_RETRIES} attempts.
 */
@Override
public void startQueryJob(JobReference jobRef, JobConfigurationQuery queryConfig)
    throws IOException, InterruptedException {
  Job job = new Job()
      .setJobReference(jobRef)
      .setConfiguration(
          new JobConfiguration().setQuery(queryConfig));

  startJob(job, errorExtractor, client);
}
 
开发者ID:apache,项目名称:beam,代码行数:18,代码来源:BigQueryServicesImpl.java


示例18: startCopyJob

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
/**
 * {@inheritDoc}
 *
 * <p>Tries executing the RPC for at most {@code MAX_RPC_RETRIES} times until it succeeds.
 *
 * @throws IOException if it exceeds {@code MAX_RPC_RETRIES} attempts.
 */
@Override
public void startCopyJob(JobReference jobRef, JobConfigurationTableCopy copyConfig)
    throws IOException, InterruptedException {
  Job job = new Job()
      .setJobReference(jobRef)
      .setConfiguration(
          new JobConfiguration().setCopy(copyConfig));

  startJob(job, errorExtractor, client);
}
 
开发者ID:apache,项目名称:beam,代码行数:18,代码来源:BigQueryServicesImpl.java


示例19: pollJob

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
@Override
public Job pollJob(JobReference jobRef, int maxAttempts) throws InterruptedException {
  BackOff backoff =
      BackOffAdapter.toGcpBackOff(
          FluentBackoff.DEFAULT
              .withMaxRetries(maxAttempts)
              .withInitialBackoff(INITIAL_JOB_STATUS_POLL_BACKOFF)
              .withMaxBackoff(Duration.standardMinutes(1))
              .backoff());
  return pollJob(jobRef, Sleeper.DEFAULT, backoff);
}
 
开发者ID:apache,项目名称:beam,代码行数:12,代码来源:BigQueryServicesImpl.java


示例20: executeExtract

import com.google.api.services.bigquery.model.JobReference; //导入依赖的package包/类
private List<ResourceId> executeExtract(
    String jobId, TableReference table, JobService jobService, String executingProject,
    String extractDestinationDir)
        throws InterruptedException, IOException {
  JobReference jobRef = new JobReference()
      .setProjectId(executingProject)
      .setJobId(jobId);

  String destinationUri = BigQueryIO.getExtractDestinationUri(extractDestinationDir);
  JobConfigurationExtract extract = new JobConfigurationExtract()
      .setSourceTable(table)
      .setDestinationFormat("AVRO")
      .setDestinationUris(ImmutableList.of(destinationUri));

  LOG.info("Starting BigQuery extract job: {}", jobId);
  jobService.startExtractJob(jobRef, extract);
  Job extractJob =
      jobService.pollJob(jobRef, JOB_POLL_MAX_RETRIES);
  if (BigQueryHelpers.parseStatus(extractJob) != Status.SUCCEEDED) {
    throw new IOException(String.format(
        "Extract job %s failed, status: %s.",
        extractJob.getJobReference().getJobId(),
        BigQueryHelpers.statusToPrettyString(extractJob.getStatus())));
  }

  LOG.info("BigQuery extract job completed: {}", jobId);

  return BigQueryIO.getExtractFilePaths(extractDestinationDir, extractJob);
}
 
开发者ID:apache,项目名称:beam,代码行数:30,代码来源:BigQuerySourceBase.java



注:本文中的com.google.api.services.bigquery.model.JobReference类示例整理自Github/MSDocs等源码及文档管理平台,相关代码片段筛选自各路编程大神贡献的开源项目,源码版权归原作者所有,传播和使用请参考对应项目的License;未经允许,请勿转载。


鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
上一篇:
Java GetContainersResponsePBImpl类代码示例发布时间:2022-05-22
下一篇:
Java PictureMarkerSymbol类代码示例发布时间:2022-05-22
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap