Java EncodedDataBlock类代码示例

OStack程序员社区-中国程序员成长平台 › 门户 › 编程› Java›Java编程经验

原作者: [db:作者] 来自: [db:来源] 收藏邀请

本文整理汇总了Java中org.apache.hadoop.hbase.io.encoding.EncodedDataBlock类的典型用法代码示例。如果您正苦于以下问题：Java EncodedDataBlock类的具体用法？Java EncodedDataBlock怎么用？Java EncodedDataBlock使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。

EncodedDataBlock类属于org.apache.hadoop.hbase.io.encoding包，在下文中一共展示了EncodedDataBlock类的14个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: verifyCodecs

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Verify if all data block encoders are working properly.
 *
 * @param scanner Of file which was compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @return true if all data block encoders compressed/decompressed correctly.
 * @throws IOException thrown if scanner is invalid
 */
public boolean verifyCodecs(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  KeyValue currentKv;

  scanner.seek(KeyValue.LOWESTKEY);
  List<Iterator<Cell>> codecIterators =
      new ArrayList<Iterator<Cell>>();
  for(EncodedDataBlock codec : codecs) {
    codecIterators.add(codec.getIterator(HFileBlock.headerSize(useHBaseChecksum)));
  }

  int j = 0;
  while ((currentKv = KeyValueUtil.ensureKeyValue(scanner.next())) != null && j < kvLimit) {
    // Iterates through key/value pairs
    ++j;
    for (Iterator<Cell> it : codecIterators) {
      Cell c = it.next();
      KeyValue codecKv = KeyValueUtil.ensureKeyValue(c);
      if (codecKv == null || 0 != Bytes.compareTo(
          codecKv.getBuffer(), codecKv.getOffset(), codecKv.getLength(),
          currentKv.getBuffer(), currentKv.getOffset(),
          currentKv.getLength())) {
        if (codecKv == null) {
          LOG.error("There is a bug in codec " + it +
              " it returned null KeyValue,");
        } else {
          int prefix = 0;
          int limitLength = 2 * Bytes.SIZEOF_INT +
              Math.min(codecKv.getLength(), currentKv.getLength());
          while (prefix < limitLength &&
              codecKv.getBuffer()[prefix + codecKv.getOffset()] ==
              currentKv.getBuffer()[prefix + currentKv.getOffset()]) {
            prefix++;
          }

          LOG.error("There is bug in codec " + it.toString() +
              "\n on element " + j +
              "\n codecKv.getKeyLength() " + codecKv.getKeyLength() +
              "\n codecKv.getValueLength() " + codecKv.getValueLength() +
              "\n codecKv.getLength() " + codecKv.getLength() +
              "\n currentKv.getKeyLength() " + currentKv.getKeyLength() +
              "\n currentKv.getValueLength() " + currentKv.getValueLength() +
              "\n codecKv.getLength() " + currentKv.getLength() +
              "\n currentKV rowLength " + currentKv.getRowLength() +
              " familyName " + currentKv.getFamilyLength() +
              " qualifier " + currentKv.getQualifierLength() +
              "\n prefix " + prefix +
              "\n codecKv   '" + Bytes.toStringBinary(codecKv.getBuffer(),
                  codecKv.getOffset(), prefix) + "' diff '" +
                  Bytes.toStringBinary(codecKv.getBuffer(),
                      codecKv.getOffset() + prefix, codecKv.getLength() -
                      prefix) + "'" +
              "\n currentKv '" + Bytes.toStringBinary(
                 currentKv.getBuffer(),
                 currentKv.getOffset(), prefix) + "' diff '" +
                 Bytes.toStringBinary(currentKv.getBuffer(),
                     currentKv.getOffset() + prefix, currentKv.getLength() -
                     prefix) + "'"
              );
        }
        return false;
      }
    }
  }

  LOG.info("Verification was successful!");

  return true;
}

开发者ID:fengchen8086，项目名称:ditb，代码行数:78，代码来源:DataBlockEncodingTool.java

示例2: benchmarkCodecs

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Benchmark codec's speed.
 */
public void benchmarkCodecs() throws IOException {
  LOG.info("Starting a throughput benchmark for data block encoding codecs");
  int prevTotalSize = -1;
  for (EncodedDataBlock codec : codecs) {
    prevTotalSize = benchmarkEncoder(prevTotalSize, codec);
  }

  benchmarkDefaultCompression(prevTotalSize, rawKVs);
}

开发者ID:fengchen8086，项目名称:ditb，代码行数:13，代码来源:DataBlockEncodingTool.java

示例3: displayStatistics

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Display statistics of different compression algorithms.
 * @throws IOException
 */
public void displayStatistics() throws IOException {
  final String comprAlgo = compressionAlgorithmName.toUpperCase();
  long rawBytes = totalKeyLength + totalPrefixLength + totalValueLength;

  System.out.println("Raw data size:");
  outputTuple("Raw bytes", INT_FORMAT, rawBytes);
  outputTuplePct("Key bytes", totalKeyLength);
  outputTuplePct("Value bytes", totalValueLength);
  outputTuplePct("KV infrastructure", totalPrefixLength);
  outputTuplePct("CF overhead", totalCFLength);
  outputTuplePct("Total key redundancy", totalKeyRedundancyLength);

  int compressedSize = EncodedDataBlock.getCompressedSize(
      compressionAlgorithm, compressor, rawKVs, 0, rawKVs.length);
  outputTuple(comprAlgo + " only size", INT_FORMAT,
      compressedSize);
  outputSavings(comprAlgo + " only", compressedSize, rawBytes);
  System.out.println();

  for (EncodedDataBlock codec : codecs) {
    System.out.println(codec.toString());
    long encodedBytes = codec.getSize();
    outputTuple("Encoded bytes", INT_FORMAT, encodedBytes);
    outputSavings("Key encoding", encodedBytes - totalValueLength,
        rawBytes - totalValueLength);
    outputSavings("Total encoding", encodedBytes, rawBytes);

    int encodedCompressedSize = codec.getEncodedCompressedSize(
        compressionAlgorithm, compressor);
    outputTuple("Encoding + " + comprAlgo + " size", INT_FORMAT,
        encodedCompressedSize);
    outputSavings("Encoding + " + comprAlgo, encodedCompressedSize, rawBytes);
    outputSavings("Encoding with " + comprAlgo, encodedCompressedSize,
        compressedSize);

    System.out.println();
  }
}

开发者ID:fengchen8086，项目名称:ditb，代码行数:43，代码来源:DataBlockEncodingTool.java

示例4: benchmarkCodecs

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Benchmark codec's speed.
 */
public void benchmarkCodecs() {
  int prevTotalSize = -1;
  for (EncodedDataBlock codec : codecs) {
    prevTotalSize = benchmarkEncoder(prevTotalSize, codec);
  }

  byte[] buffer = codecs.get(0).getRawKeyValues();

  benchmarkDefaultCompression(prevTotalSize, buffer);
}

开发者ID:fengchen8086，项目名称:LCIndex-HBase-0.94.16，代码行数:14，代码来源:DataBlockEncodingTool.java

示例5: displayStatistics

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Display statistics of different compression algorithms.
 * @throws IOException
 */
public void displayStatistics() throws IOException {
  final String comprAlgo = compressionAlgorithmName.toUpperCase(Locale.ROOT);
  long rawBytes = totalKeyLength + totalPrefixLength + totalValueLength;

  System.out.println("Raw data size:");
  outputTuple("Raw bytes", INT_FORMAT, rawBytes);
  outputTuplePct("Key bytes", totalKeyLength);
  outputTuplePct("Value bytes", totalValueLength);
  outputTuplePct("KV infrastructure", totalPrefixLength);
  outputTuplePct("CF overhead", totalCFLength);
  outputTuplePct("Total key redundancy", totalKeyRedundancyLength);

  int compressedSize = EncodedDataBlock.getCompressedSize(
      compressionAlgorithm, compressor, rawKVs, 0, rawKVs.length);
  outputTuple(comprAlgo + " only size", INT_FORMAT,
      compressedSize);
  outputSavings(comprAlgo + " only", compressedSize, rawBytes);
  System.out.println();

  for (EncodedDataBlock codec : codecs) {
    System.out.println(codec.toString());
    long encodedBytes = codec.getSize();
    outputTuple("Encoded bytes", INT_FORMAT, encodedBytes);
    outputSavings("Key encoding", encodedBytes - totalValueLength,
        rawBytes - totalValueLength);
    outputSavings("Total encoding", encodedBytes, rawBytes);

    int encodedCompressedSize = codec.getEncodedCompressedSize(
        compressionAlgorithm, compressor);
    outputTuple("Encoding + " + comprAlgo + " size", INT_FORMAT,
        encodedCompressedSize);
    outputSavings("Encoding + " + comprAlgo, encodedCompressedSize, rawBytes);
    outputSavings("Encoding with " + comprAlgo, encodedCompressedSize,
        compressedSize);

    System.out.println();
  }
}

开发者ID:apache，项目名称:hbase，代码行数:43，代码来源:DataBlockEncodingTool.java

示例6: checkStatistics

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Check statistics for given HFile for different data block encoders.
 * @param scanner Of file which will be compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @throws IOException thrown if scanner is invalid
 */
public void checkStatistics(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  scanner.seek(KeyValue.LOWESTKEY);

  KeyValue currentKV;

  byte[] previousKey = null;
  byte[] currentKey;

  DataBlockEncoding[] encodings = DataBlockEncoding.values();

  ByteArrayOutputStream uncompressedOutputStream =
      new ByteArrayOutputStream();

  int j = 0;
  while ((currentKV = KeyValueUtil.ensureKeyValue(scanner.next())) != null && j < kvLimit) {
    // Iterates through key/value pairs
    j++;
    currentKey = currentKV.getKey();
    if (previousKey != null) {
      for (int i = 0; i < previousKey.length && i < currentKey.length &&
          previousKey[i] == currentKey[i]; ++i) {
        totalKeyRedundancyLength++;
      }
    }

    uncompressedOutputStream.write(currentKV.getBuffer(),
        currentKV.getOffset(), currentKV.getLength());

    previousKey = currentKey;

    int kLen = currentKV.getKeyLength();
    int vLen = currentKV.getValueLength();
    int cfLen = currentKV.getFamilyLength(currentKV.getFamilyOffset());
    int restLen = currentKV.getLength() - kLen - vLen;

    totalKeyLength += kLen;
    totalValueLength += vLen;
    totalPrefixLength += restLen;
    totalCFLength += cfLen;
  }

  rawKVs = uncompressedOutputStream.toByteArray();
  boolean useTag = (currentKV.getTagsLength() > 0);
  for (DataBlockEncoding encoding : encodings) {
    if (encoding == DataBlockEncoding.NONE) {
      continue;
    }
    DataBlockEncoder d = encoding.getEncoder();
    HFileContext meta = new HFileContextBuilder()
                        .withCompression(Compression.Algorithm.NONE)
                        .withIncludesMvcc(includesMemstoreTS)
                        .withIncludesTags(useTag).build();
    codecs.add(new EncodedDataBlock(d, encoding, rawKVs, meta ));
  }
}

开发者ID:fengchen8086，项目名称:ditb，代码行数:63，代码来源:DataBlockEncodingTool.java

示例7: checkStatistics

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Check statistics for given HFile for different data block encoders.
 * @param scanner Of file which will be compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @throws IOException thrown if scanner is invalid
 */
public void checkStatistics(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  scanner.seek(KeyValue.LOWESTKEY);

  KeyValue currentKv;

  byte[] previousKey = null;
  byte[] currentKey;

  List<DataBlockEncoder> dataBlockEncoders =
      DataBlockEncoding.getAllEncoders();

  for (DataBlockEncoder d : dataBlockEncoders) {
    codecs.add(new EncodedDataBlock(d, includesMemstoreTS));
  }

  int j = 0;
  while ((currentKv = scanner.next()) != null && j < kvLimit) {
    // Iterates through key/value pairs
    j++;
    currentKey = currentKv.getKey();
    if (previousKey != null) {
      for (int i = 0; i < previousKey.length && i < currentKey.length &&
          previousKey[i] == currentKey[i]; ++i) {
        totalKeyRedundancyLength++;
      }
    }

    for (EncodedDataBlock codec : codecs) {
      codec.addKv(currentKv);
    }

    previousKey = currentKey;

    totalPrefixLength += currentKv.getLength() - currentKv.getKeyLength() -
        currentKv.getValueLength();
    totalKeyLength += currentKv.getKeyLength();
    totalValueLength += currentKv.getValueLength();
  }
}

开发者ID:fengchen8086，项目名称:LCIndex-HBase-0.94.16，代码行数:47，代码来源:DataBlockEncodingTool.java

示例8: verifyCodecs

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Verify if all data block encoders are working properly.
 * 
 * @param scanner Of file which was compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @return true if all data block encoders compressed/decompressed correctly.
 * @throws IOException thrown if scanner is invalid
 */
public boolean verifyCodecs(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  KeyValue currentKv;

  scanner.seek(KeyValue.LOWESTKEY);
  List<Iterator<KeyValue>> codecIterators =
      new ArrayList<Iterator<KeyValue>>();
  for(EncodedDataBlock codec : codecs) {
    codecIterators.add(codec.getIterator());
  }

  int j = 0;
  while ((currentKv = scanner.next()) != null && j < kvLimit) {
    // Iterates through key/value pairs
    ++j;
    for (Iterator<KeyValue> it : codecIterators) {
      KeyValue codecKv = it.next();
      if (codecKv == null || 0 != Bytes.compareTo(
          codecKv.getBuffer(), codecKv.getOffset(), codecKv.getLength(),
          currentKv.getBuffer(), currentKv.getOffset(),
          currentKv.getLength())) {
        if (codecKv == null) {
          LOG.error("There is a bug in codec " + it +
              " it returned null KeyValue,");
        } else {
          int prefix = 0;
          int limitLength = 2 * Bytes.SIZEOF_INT +
              Math.min(codecKv.getLength(), currentKv.getLength());
          while (prefix < limitLength &&
              codecKv.getBuffer()[prefix + codecKv.getOffset()] ==
              currentKv.getBuffer()[prefix + currentKv.getOffset()]) {
            prefix++;
          }

          LOG.error("There is bug in codec " + it.toString() +
              "\n on element " + j +
              "\n codecKv.getKeyLength() " + codecKv.getKeyLength() +
              "\n codecKv.getValueLength() " + codecKv.getValueLength() +
              "\n codecKv.getLength() " + codecKv.getLength() +
              "\n currentKv.getKeyLength() " + currentKv.getKeyLength() +
              "\n currentKv.getValueLength() " + currentKv.getValueLength() +
              "\n codecKv.getLength() " + currentKv.getLength() +
              "\n currentKV rowLength " + currentKv.getRowLength() +
              " familyName " + currentKv.getFamilyLength() +
              " qualifier " + currentKv.getQualifierLength() +
              "\n prefix " + prefix +
              "\n codecKv   '" + Bytes.toStringBinary(codecKv.getBuffer(),
                  codecKv.getOffset(), prefix) + "' diff '" +
                  Bytes.toStringBinary(codecKv.getBuffer(),
                      codecKv.getOffset() + prefix, codecKv.getLength() -
                      prefix) + "'" +
              "\n currentKv '" + Bytes.toStringBinary(
                 currentKv.getBuffer(),
                 currentKv.getOffset(), prefix) + "' diff '" +
                 Bytes.toStringBinary(currentKv.getBuffer(),
                     currentKv.getOffset() + prefix, currentKv.getLength() -
                     prefix) + "'"
              );
        }
        return false;
      }
    }
  }

  LOG.info("Verification was successful!");

  return true;
}

开发者ID:fengchen8086，项目名称:LCIndex-HBase-0.94.16，代码行数:77，代码来源:DataBlockEncodingTool.java

示例9: displayStatistics

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Display statistics of different compression algorithms.
 */
public void displayStatistics() {
  int totalLength = totalPrefixLength + totalKeyLength + totalValueLength;
  if (compressor != null) {  // might be null e.g. for pure-Java GZIP 
    compressor.reset();
  }

  for(EncodedDataBlock codec : codecs) {
    System.out.println(codec.toString());
    int saved = totalKeyLength + totalPrefixLength + totalValueLength
        - codec.getSize();
    System.out.println(
        String.format("  Saved bytes:                 %8d", saved));
    double keyRatio = (saved * 100.0) / (totalPrefixLength + totalKeyLength);
    double allRatio = (saved * 100.0) / totalLength;
    System.out.println(
        String.format("  Key compression ratio:        %.2f %%", keyRatio));
    System.out.println(
        String.format("  All compression ratio:        %.2f %%", allRatio));

    String compressedSizeCaption =
        String.format("  %s compressed size:         ",
            compressionAlgorithmName.toUpperCase());
    String compressOnlyRatioCaption =
        String.format("  %s compression ratio:        ",
            compressionAlgorithmName.toUpperCase());

    if (compressor != null) {
      int compressedSize = codec.checkCompressedSize(compressor);
      System.out.println(compressedSizeCaption +
          String.format("%8d", compressedSize));
      double compressOnlyRatio =
          100.0 * (1.0 - compressedSize / (0.0 + totalLength));
      System.out.println(compressOnlyRatioCaption
          + String.format("%.2f %%", compressOnlyRatio));
    } else {
      System.out.println(compressedSizeCaption + "N/A");
      System.out.println(compressOnlyRatioCaption + "N/A");
    }
  }

  System.out.println(
      String.format("Total KV prefix length:   %8d", totalPrefixLength));
  System.out.println(
      String.format("Total key length:         %8d", totalKeyLength));
  System.out.println(
      String.format("Total key redundancy:     %8d",
          totalKeyRedundancyLength));
  System.out.println(
      String.format("Total value length:       %8d", totalValueLength));
}

开发者ID:fengchen8086，项目名称:LCIndex-HBase-0.94.16，代码行数:54，代码来源:DataBlockEncodingTool.java

示例10: checkStatistics

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Check statistics for given HFile for different data block encoders.
 * @param scanner Of file which will be compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @throws IOException thrown if scanner is invalid
 */
public void checkStatistics(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  scanner.seek(KeyValue.LOWESTKEY);

  KeyValue currentKV;

  byte[] previousKey = null;
  byte[] currentKey;

  DataBlockEncoding[] encodings = DataBlockEncoding.values();

  ByteArrayOutputStream uncompressedOutputStream =
      new ByteArrayOutputStream();

  int j = 0;
  while ((currentKV = scanner.next()) != null && j < kvLimit) {
    // Iterates through key/value pairs
    j++;
    currentKey = currentKV.getKey();
    if (previousKey != null) {
      for (int i = 0; i < previousKey.length && i < currentKey.length &&
          previousKey[i] == currentKey[i]; ++i) {
        totalKeyRedundancyLength++;
      }
    }

    uncompressedOutputStream.write(currentKV.getBuffer(),
        currentKV.getOffset(), currentKV.getLength());

    previousKey = currentKey;

    int kLen = currentKV.getKeyLength();
    int vLen = currentKV.getValueLength();
    int cfLen = currentKV.getFamilyLength(currentKV.getFamilyOffset());
    int restLen = currentKV.getLength() - kLen - vLen;

    totalKeyLength += kLen;
    totalValueLength += vLen;
    totalPrefixLength += restLen;
    totalCFLength += cfLen;
  }

  rawKVs = uncompressedOutputStream.toByteArray();
  boolean useTag = (currentKV.getTagsLength() > 0);
  for (DataBlockEncoding encoding : encodings) {
    if (encoding == DataBlockEncoding.NONE) {
      continue;
    }
    DataBlockEncoder d = encoding.getEncoder();
    HFileContext meta = new HFileContextBuilder()
                        .withCompression(Compression.Algorithm.NONE)
                        .withIncludesMvcc(includesMemstoreTS)
                        .withIncludesTags(useTag).build();
    codecs.add(new EncodedDataBlock(d, encoding, rawKVs, meta ));
  }
}

开发者ID:tenggyut，项目名称:HIndex，代码行数:63，代码来源:DataBlockEncodingTool.java

示例11: verifyCodecs

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Verify if all data block encoders are working properly.
 *
 * @param scanner Of file which was compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @return true if all data block encoders compressed/decompressed correctly.
 * @throws IOException thrown if scanner is invalid
 */
public boolean verifyCodecs(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  KeyValue currentKv;

  scanner.seek(KeyValue.LOWESTKEY);
  List<Iterator<Cell>> codecIterators =
      new ArrayList<Iterator<Cell>>();
  for(EncodedDataBlock codec : codecs) {
    codecIterators.add(codec.getIterator(HFileBlock.headerSize(useHBaseChecksum)));
  }

  int j = 0;
  while ((currentKv = scanner.next()) != null && j < kvLimit) {
    // Iterates through key/value pairs
    ++j;
    for (Iterator<Cell> it : codecIterators) {
      Cell c = it.next();
      KeyValue codecKv = KeyValueUtil.ensureKeyValue(c);
      if (codecKv == null || 0 != Bytes.compareTo(
          codecKv.getBuffer(), codecKv.getOffset(), codecKv.getLength(),
          currentKv.getBuffer(), currentKv.getOffset(),
          currentKv.getLength())) {
        if (codecKv == null) {
          LOG.error("There is a bug in codec " + it +
              " it returned null KeyValue,");
        } else {
          int prefix = 0;
          int limitLength = 2 * Bytes.SIZEOF_INT +
              Math.min(codecKv.getLength(), currentKv.getLength());
          while (prefix < limitLength &&
              codecKv.getBuffer()[prefix + codecKv.getOffset()] ==
              currentKv.getBuffer()[prefix + currentKv.getOffset()]) {
            prefix++;
          }

          LOG.error("There is bug in codec " + it.toString() +
              "\n on element " + j +
              "\n codecKv.getKeyLength() " + codecKv.getKeyLength() +
              "\n codecKv.getValueLength() " + codecKv.getValueLength() +
              "\n codecKv.getLength() " + codecKv.getLength() +
              "\n currentKv.getKeyLength() " + currentKv.getKeyLength() +
              "\n currentKv.getValueLength() " + currentKv.getValueLength() +
              "\n codecKv.getLength() " + currentKv.getLength() +
              "\n currentKV rowLength " + currentKv.getRowLength() +
              " familyName " + currentKv.getFamilyLength() +
              " qualifier " + currentKv.getQualifierLength() +
              "\n prefix " + prefix +
              "\n codecKv   '" + Bytes.toStringBinary(codecKv.getBuffer(),
                  codecKv.getOffset(), prefix) + "' diff '" +
                  Bytes.toStringBinary(codecKv.getBuffer(),
                      codecKv.getOffset() + prefix, codecKv.getLength() -
                      prefix) + "'" +
              "\n currentKv '" + Bytes.toStringBinary(
                 currentKv.getBuffer(),
                 currentKv.getOffset(), prefix) + "' diff '" +
                 Bytes.toStringBinary(currentKv.getBuffer(),
                     currentKv.getOffset() + prefix, currentKv.getLength() -
                     prefix) + "'"
              );
        }
        return false;
      }
    }
  }

  LOG.info("Verification was successful!");

  return true;
}

开发者ID:tenggyut，项目名称:HIndex，代码行数:78，代码来源:DataBlockEncodingTool.java

示例12: verifyCodecs

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Verify if all data block encoders are working properly.
 *
 * @param scanner Of file which was compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @return true if all data block encoders compressed/decompressed correctly.
 * @throws IOException thrown if scanner is invalid
 */
public boolean verifyCodecs(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  KeyValue currentKv;

  scanner.seek(KeyValue.LOWESTKEY);
  List<Iterator<Cell>> codecIterators = new ArrayList<>();
  for(EncodedDataBlock codec : codecs) {
    codecIterators.add(codec.getIterator(HFileBlock.headerSize(useHBaseChecksum)));
  }

  int j = 0;
  while ((currentKv = KeyValueUtil.ensureKeyValue(scanner.next())) != null && j < kvLimit) {
    // Iterates through key/value pairs
    ++j;
    for (Iterator<Cell> it : codecIterators) {
      Cell c = it.next();
      KeyValue codecKv = KeyValueUtil.ensureKeyValue(c);
      if (codecKv == null || 0 != Bytes.compareTo(
          codecKv.getBuffer(), codecKv.getOffset(), codecKv.getLength(),
          currentKv.getBuffer(), currentKv.getOffset(),
          currentKv.getLength())) {
        if (codecKv == null) {
          LOG.error("There is a bug in codec " + it +
              " it returned null KeyValue,");
        } else {
          int prefix = 0;
          int limitLength = 2 * Bytes.SIZEOF_INT +
              Math.min(codecKv.getLength(), currentKv.getLength());
          while (prefix < limitLength &&
              codecKv.getBuffer()[prefix + codecKv.getOffset()] ==
              currentKv.getBuffer()[prefix + currentKv.getOffset()]) {
            prefix++;
          }

          LOG.error("There is bug in codec " + it.toString() +
              "\n on element " + j +
              "\n codecKv.getKeyLength() " + codecKv.getKeyLength() +
              "\n codecKv.getValueLength() " + codecKv.getValueLength() +
              "\n codecKv.getLength() " + codecKv.getLength() +
              "\n currentKv.getKeyLength() " + currentKv.getKeyLength() +
              "\n currentKv.getValueLength() " + currentKv.getValueLength() +
              "\n codecKv.getLength() " + currentKv.getLength() +
              "\n currentKV rowLength " + currentKv.getRowLength() +
              " familyName " + currentKv.getFamilyLength() +
              " qualifier " + currentKv.getQualifierLength() +
              "\n prefix " + prefix +
              "\n codecKv   '" + Bytes.toStringBinary(codecKv.getBuffer(),
                  codecKv.getOffset(), prefix) + "' diff '" +
                  Bytes.toStringBinary(codecKv.getBuffer(),
                      codecKv.getOffset() + prefix, codecKv.getLength() -
                      prefix) + "'" +
              "\n currentKv '" + Bytes.toStringBinary(
                 currentKv.getBuffer(),
                 currentKv.getOffset(), prefix) + "' diff '" +
                 Bytes.toStringBinary(currentKv.getBuffer(),
                     currentKv.getOffset() + prefix, currentKv.getLength() -
                     prefix) + "'"
              );
        }
        return false;
      }
    }
  }

  LOG.info("Verification was successful!");

  return true;
}

开发者ID:apache，项目名称:hbase，代码行数:77，代码来源:DataBlockEncodingTool.java

示例13: checkStatistics

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Check statistics for given HFile for different data block encoders.
 * @param scanner Of file which will be compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @throws IOException thrown if scanner is invalid
 */
public void checkStatistics(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  scanner.seek(KeyValue.LOWESTKEY);

  KeyValue currentKV;

  byte[] previousKey = null;
  byte[] currentKey;

  DataBlockEncoding[] encodings = DataBlockEncoding.values();

  ByteArrayOutputStream uncompressedOutputStream =
      new ByteArrayOutputStream();

  int j = 0;
  while ((currentKV = scanner.next()) != null && j < kvLimit) {
    // Iterates through key/value pairs
    j++;
    currentKey = currentKV.getKey();
    if (previousKey != null) {
      for (int i = 0; i < previousKey.length && i < currentKey.length &&
          previousKey[i] == currentKey[i]; ++i) {
        totalKeyRedundancyLength++;
      }
    }

    uncompressedOutputStream.write(currentKV.getBuffer(),
        currentKV.getOffset(), currentKV.getLength());

    previousKey = currentKey;

    int kLen = currentKV.getKeyLength();
    int vLen = currentKV.getValueLength();
    int cfLen = currentKV.getFamilyLength(currentKV.getFamilyOffset());
    int restLen = currentKV.getLength() - kLen - vLen;

    totalKeyLength += kLen;
    totalValueLength += vLen;
    totalPrefixLength += restLen;
    totalCFLength += cfLen;
  }

  rawKVs = uncompressedOutputStream.toByteArray();

  for (DataBlockEncoding encoding : encodings) {
    if (encoding == DataBlockEncoding.NONE) {
      continue;
    }
    DataBlockEncoder d = encoding.getEncoder();
    codecs.add(new EncodedDataBlock(d, includesMemstoreTS, encoding, rawKVs));
  }
}

开发者ID:cloud-software-foundation，项目名称:c5，代码行数:59，代码来源:DataBlockEncodingTool.java

示例14: verifyCodecs

import org.apache.hadoop.hbase.io.encoding.EncodedDataBlock; //导入依赖的package包/类
/**
 * Verify if all data block encoders are working properly.
 *
 * @param scanner Of file which was compressed.
 * @param kvLimit Maximal count of KeyValue which will be processed.
 * @return true if all data block encoders compressed/decompressed correctly.
 * @throws IOException thrown if scanner is invalid
 */
public boolean verifyCodecs(final KeyValueScanner scanner, final int kvLimit)
    throws IOException {
  KeyValue currentKv;

  scanner.seek(KeyValue.LOWESTKEY);
  List<Iterator<Cell>> codecIterators =
      new ArrayList<Iterator<Cell>>();
  for(EncodedDataBlock codec : codecs) {
    codecIterators.add(codec.getIterator(HFileBlock.headerSize(minorVersion)));
  }

  int j = 0;
  while ((currentKv = scanner.next()) != null && j < kvLimit) {
    // Iterates through key/value pairs
    ++j;
    for (Iterator<Cell> it : codecIterators) {
      Cell c = it.next();
      KeyValue codecKv = KeyValueUtil.ensureKeyValue(c);
      if (codecKv == null || 0 != Bytes.compareTo(
          codecKv.getBuffer(), codecKv.getOffset(), codecKv.getLength(),
          currentKv.getBuffer(), currentKv.getOffset(),
          currentKv.getLength())) {
        if (codecKv == null) {
          LOG.error("There is a bug in codec " + it +
              " it returned null KeyValue,");
        } else {
          int prefix = 0;
          int limitLength = 2 * Bytes.SIZEOF_INT +
              Math.min(codecKv.getLength(), currentKv.getLength());
          while (prefix < limitLength &&
              codecKv.getBuffer()[prefix + codecKv.getOffset()] ==
              currentKv.getBuffer()[prefix + currentKv.getOffset()]) {
            prefix++;
          }

          LOG.error("There is bug in codec " + it.toString() +
              "\n on element " + j +
              "\n codecKv.getKeyLength() " + codecKv.getKeyLength() +
              "\n codecKv.getValueLength() " + codecKv.getValueLength() +
              "\n codecKv.getLength() " + codecKv.getLength() +
              "\n currentKv.getKeyLength() " + currentKv.getKeyLength() +
              "\n currentKv.getValueLength() " + currentKv.getValueLength() +
              "\n codecKv.getLength() " + currentKv.getLength() +
              "\n currentKV rowLength " + currentKv.getRowLength() +
              " familyName " + currentKv.getFamilyLength() +
              " qualifier " + currentKv.getQualifierLength() +
              "\n prefix " + prefix +
              "\n codecKv   '" + Bytes.toStringBinary(codecKv.getBuffer(),
                  codecKv.getOffset(), prefix) + "' diff '" +
                  Bytes.toStringBinary(codecKv.getBuffer(),
                      codecKv.getOffset() + prefix, codecKv.getLength() -
                      prefix) + "'" +
              "\n currentKv '" + Bytes.toStringBinary(
                 currentKv.getBuffer(),
                 currentKv.getOffset(), prefix) + "' diff '" +
                 Bytes.toStringBinary(currentKv.getBuffer(),
                     currentKv.getOffset() + prefix, currentKv.getLength() -
                     prefix) + "'"
              );
        }
        return false;
      }
    }
  }

  LOG.info("Verification was successful!");

  return true;
}

开发者ID:cloud-software-foundation，项目名称:c5，代码行数:78，代码来源:DataBlockEncodingTool.java

注：本文中的org.apache.hadoop.hbase.io.encoding.EncodedDataBlock类示例整理自Github/MSDocs等源码及文档管理平台，相关代码片段筛选自各路编程大神贡献的开源项目，源码版权归原作者所有，传播和使用请参考对应项目的License；未经允许，请勿转载。

鲜花

握手

雷人

路过

鸡蛋

该文章已有0人参与评论

请发表评论

全部评论

专题导读

More+

10-27 六六分期app的软件客服如何联系？(六六分期

11-06 可心卡盟:win10系统火狐flash插件崩溃怎么

11-06 亲亲特价:怎么删除回收站图标

11-06 济南大学虚拟社区:鲁大师节能降温的具体办

11-06 xlueops.exe:无线网络安装向导

11-06 女斗合众国:win7系统cf与主机连接不稳定怎

11-06 0xc000022-[cf烟雾头]cf怎么调烟雾头

11-06 qizideyouhuo:应用程序无法正常启动0xc0000

11-06 ipz-185:win7系统vcf文件怎么打开

11-06 傻哥蹦迪:win10系统s4怎么打开usb调试

11-06 八神浩树gtaste:回收站清空了怎么恢复

11-06 妖尾之黑色守护:win10系统电脑没有1440x900

11-06 校园至尊魔王小说:win7系统浏览网页时字体

11-06 女斗合众国:win10系统访问共享文件夹提示请

11-06 tokyo hot n0654:恢复win7系统默认字体一招

11-06 雨酷仙境:设置win7系统转移临时文件夹腾出

11-06 阿穆纳伊之杖:win7系统开始菜单在右边还原

11-06 tunespotting:win10系统火狐flash插件总是

11-06 甘尔葛分析师：计谋网站seo关键词暴涨有什

11-06 蔡贵霖: 计谋网站seo关键词暴涨有什么秘密

11-06 博益网首页:ao3网页版进入不了解决方法

11-06 漏斗子专栏: 网站数据分析小白易懂精华篇

11-06 见证双虹怎么做:win7系统开启telnet命令的

11-06 颾狐蝶蜋:系统资源不足无法完成请求的服务

11-06 国光中学校歌:提交网站到alexa查询详细步骤

11-06 西安有情天:静态网页和动态网页的区别

11-06 红木雅尚斋:外部链接构造对网站的好处

11-06 前官礼遇：防止域名劫持–增强域安全性的10

11-06 密传二转答案: 中文分词算法有哪些

11-06 金泉家园邮编:百度快照劫持的表现及应对方

Java ForkJoinWorkerThreadFactory类代码示例发布时间：2022-05-21

Java GSSNameSpi类代码示例发布时间：2022-05-21

剪的笔顺,诠释剪的笔画,认识剪的部首

1 六六分期app的软件客服如何联系？(六六分期

六六分期app的软件客服如何联系？不知道吗？加qq群【895510560】即可！标题：六六分期

阅读：19148|2023-10-27

2 可心卡盟:win10系统火狐flash插件崩溃怎么

今天小编告诉大家如何处理win10系统火狐flash插件总是崩溃的问题，可能很多用户都不知

阅读：9973|2022-11-06

3 亲亲特价:怎么删除回收站图标

今天小编告诉大家如何对win10系统删除桌面回收站图标进行设置，可能很多用户都不知道

阅读：8317|2022-11-06

4 济南大学虚拟社区:鲁大师节能降温的具体办

今天小编告诉大家如何对win10系统电脑设置节能降温的设置方法，想必大家都遇到过需要

阅读：8686|2022-11-06

5 xlueops.exe:无线网络安装向导

我们在使用xp系统的过程中,经常需要对xp系统无线网络安装向导设置进行设置，可能很多

阅读：8628|2022-11-06

6 女斗合众国:win7系统cf与主机连接不稳定怎

今天小编告诉大家如何处理win7系统玩cf老是与主机连接不稳定的问题，可能很多用户都不

阅读：9643|2022-11-06

7 0xc000022-[cf烟雾头]cf怎么调烟雾头

电脑对日常生活的重要性小编就不多说了，可是一旦碰到win7系统设置cf烟雾头的问题，很

阅读：8611|2022-11-06

8 qizideyouhuo:应用程序无法正常启动0xc0000

我们在日常使用电脑的时候，有的小伙伴们可能在打开应用的时候会遇见提示应用程序无法

阅读：7991|2022-11-06

9 ipz-185:win7系统vcf文件怎么打开

今天小编告诉大家如何对win7系统打开vcf文件进行设置，可能很多用户都不知道怎么对win

阅读：8642|2022-11-06

10 傻哥蹦迪:win10系统s4怎么打开usb调试

今天小编告诉大家如何对win10系统s4开启USB调试模式进行设置，可能很多用户都不知道怎

阅读：7527|2022-11-06

客服电话

电子邮件

Java EncodedDataBlock类代码示例

示例1: verifyCodecs

示例2: benchmarkCodecs

示例3: displayStatistics

示例4: benchmarkCodecs

示例5: displayStatistics

示例6: checkStatistics

示例7: checkStatistics

示例8: verifyCodecs

示例9: displayStatistics

示例10: checkStatistics

示例11: verifyCodecs

示例12: verifyCodecs

示例13: checkStatistics

示例14: verifyCodecs

请发表评论

全部评论

上一篇：

下一篇：

remysaissy/mongodb-macosx-prefspane: Mac

sussillo/hfopt-matlab: A parallel, cpu-b

Matlab利用plot作图线形颜色等选择

emersion/go-ostatus: An OStatus library

CVE-2022-28674

剪的笔顺,诠释剪的笔画,认识剪的部首

六六分期app的软件客服如何联系？(六六分期

florent37/ViewAnimator: A fluent Android

florent37/Shrine-MaterialDesign2: implem

CVE-2020-36276

SimpleSoftwareIO/simple-sms: Send and re

关于我们

产品与服务

解决方案

139-2527-9053