android - MediaMuxer video file size reducing (re-compress, decrease resolution)

Question

I'm looking for efficient way to reduce some video weight (as a File, for upload) and obvious answer for that is: lets reduce resolution! (fullHD or 4K not needed, simple HD is sufficient for me) I've tried lot of ways which should work through lot of APIs (needed 10) and best way was using android-ffmpeg-java, BUT... on my pretty fast almost-current flagship device whole process lasts about length_of_video*4 seconds and also this lib weight is 9 Mb, this amount increases my app size... No wai! (12 Mb to 1 Mb is nice result, but still too many flaws)

So I've decided to use native Android ways to do this, MediaMuxer and MediaCodec - they are available from API18 and API16 respectivelly (older devices users: sorry; but they also often have "lower-res" camera). Below method almost works - MediaMuxer do NOT respect MediaFormat.KEY_WIDTH and MediaFormat.KEY_HEIGHT - extracted File is "re-compressed", weight is a bit smaller, but resolution is the same as in original video File...

So, question: How to compress and re-scale/change resolution of video using MediaMuxer and other accompanying classes and methods?

public File getCompressedFile(String videoPath) throws IOException{
    MediaExtractor extractor = new MediaExtractor();
    extractor.setDataSource(videoPath);
    int trackCount = extractor.getTrackCount();

    String filePath = videoPath.substring(0, videoPath.lastIndexOf(File.separator));
    String[] splitByDot = videoPath.split("\.");
    String ext="";
    if(splitByDot!=null && splitByDot.length>1)
        ext = splitByDot[splitByDot.length-1];
    String fileName = videoPath.substring(videoPath.lastIndexOf(File.separator)+1,
                    videoPath.length());
    if(ext.length()>0)
        fileName=fileName.replace("."+ext, "_out."+ext);
    else
        fileName=fileName.concat("_out");

    final File outFile = new File(filePath, fileName);
    if(!outFile.exists())
        outFile.createNewFile();

    MediaMuxer muxer = new MediaMuxer(outFile.getAbsolutePath(),
            MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
    HashMap<Integer, Integer> indexMap = new HashMap<Integer, Integer>(trackCount);
    for (int i = 0; i < trackCount; i++) {
        extractor.selectTrack(i);
        MediaFormat format = extractor.getTrackFormat(i);
        String mime = format.getString(MediaFormat.KEY_MIME);
        if(mime!=null && mime.startsWith("video")){
            int currWidth = format.getInteger(MediaFormat.KEY_WIDTH);
            int currHeight = format.getInteger(MediaFormat.KEY_HEIGHT);
            format.setInteger(MediaFormat.KEY_WIDTH, currWidth>currHeight ? 960 : 540);
            format.setInteger(MediaFormat.KEY_HEIGHT, currWidth>currHeight ? 540 : 960);
            //API19 MediaFormat.KEY_MAX_WIDTH and KEY_MAX_HEIGHT
            format.setInteger("max-width", format.getInteger(MediaFormat.KEY_WIDTH));
            format.setInteger("max-height", format.getInteger(MediaFormat.KEY_HEIGHT));
        }
        int dstIndex = muxer.addTrack(format);
        indexMap.put(i, dstIndex);
    }

    boolean sawEOS = false;
    int bufferSize = 256 * 1024;
    int offset = 100;
    ByteBuffer dstBuf = ByteBuffer.allocate(bufferSize);
    MediaCodec.BufferInfo bufferInfo = new MediaCodec.BufferInfo();
    muxer.start();
    while (!sawEOS) {
        bufferInfo.offset = offset;
        bufferInfo.size = extractor.readSampleData(dstBuf, offset);
        if (bufferInfo.size < 0) {
            sawEOS = true;
            bufferInfo.size = 0;
        } else {
            bufferInfo.presentationTimeUs = extractor.getSampleTime();
            bufferInfo.flags = extractor.getSampleFlags();
            int trackIndex = extractor.getSampleTrackIndex();
            muxer.writeSampleData(indexMap.get(trackIndex), dstBuf,
                    bufferInfo);
            extractor.advance();
        }
    }

    muxer.stop();
    muxer.release();

    return outFile;
}

PS. lot of usefull stuff about muxer here, above code bases on MediaMuxerTest.java, method cloneMediaUsingMuxer

Answer 1

Basing on bigflake.com/mediacodec/ (awesome source of knowledge about Media-classes) I've tried few ways and finally ExtractDecodeEditEncodeMuxTest turned out very helpfull. This test wasn't described in article on bigflake site, but it can be found HERE next to other classes mentioned in text.

So, I've copied most of code from above mentioned ExtractDecodeEditEncodeMuxTest class and there it is: VideoResolutionChanger. It gives me 2Mb HD video from 16 Mb fullHD. Nice! And fast! On my device whole process is a bit longer than input video duration, e.g. 10 secs video input -> 11-12 secs of processing. With ffmpeg-java it would be smth about 40 secs or more (and 9 Mb more for app).

Here we go:

VideoResolutionChanger:

@TargetApi(18)
public class VideoResolutionChanger {

    private static final int TIMEOUT_USEC = 10000;

    private static final String OUTPUT_VIDEO_MIME_TYPE = "video/avc";
    private static final int OUTPUT_VIDEO_BIT_RATE = 2048 * 1024;
    private static final int OUTPUT_VIDEO_FRAME_RATE = 30;
    private static final int OUTPUT_VIDEO_IFRAME_INTERVAL = 10;
    private static final int OUTPUT_VIDEO_COLOR_FORMAT =
            MediaCodecInfo.CodecCapabilities.COLOR_FormatSurface;

    private static final String OUTPUT_AUDIO_MIME_TYPE = "audio/mp4a-latm";
    private static final int OUTPUT_AUDIO_CHANNEL_COUNT = 2;
    private static final int OUTPUT_AUDIO_BIT_RATE = 128 * 1024;
    private static final int OUTPUT_AUDIO_AAC_PROFILE =
            MediaCodecInfo.CodecProfileLevel.AACObjectHE;
    private static final int OUTPUT_AUDIO_SAMPLE_RATE_HZ = 44100;

    private int mWidth = 1280;
    private int mHeight = 720;
    private String mOutputFile, mInputFile;

    public String changeResolution(File f)
            throws Throwable {
        mInputFile=f.getAbsolutePath();

        String filePath = mInputFile.substring(0, mInputFile.lastIndexOf(File.separator));
        String[] splitByDot = mInputFile.split("\.");
        String ext="";
        if(splitByDot!=null && splitByDot.length>1)
            ext = splitByDot[splitByDot.length-1];
        String fileName = mInputFile.substring(mInputFile.lastIndexOf(File.separator)+1,
                mInputFile.length());
        if(ext.length()>0)
            fileName=fileName.replace("."+ext, "_out.mp4");
        else
            fileName=fileName.concat("_out.mp4");

        final File outFile = new File(Environment.getExternalStorageDirectory(), fileName);
        if(!outFile.exists())
            outFile.createNewFile();

        mOutputFile=outFile.getAbsolutePath();

        ChangerWrapper.changeResolutionInSeparatedThread(this);

        return mOutputFile;
    }

    private static class ChangerWrapper implements Runnable {

        private Throwable mThrowable;
        private VideoResolutionChanger mChanger;

        private ChangerWrapper(VideoResolutionChanger changer) {
            mChanger = changer;
        }

        @Override
        public void run() {
            try {
                mChanger.prepareAndChangeResolution();
            } catch (Throwable th) {
                mThrowable = th;
            }
        }

        public static void changeResolutionInSeparatedThread(VideoResolutionChanger changer)
                throws Throwable {
            ChangerWrapper wrapper = new ChangerWrapper(changer);
            Thread th = new Thread(wrapper, ChangerWrapper.class.getSimpleName());
            th.start();
            th.join();
            if (wrapper.mThrowable != null)
                throw wrapper.mThrowable;
        }
    }

    private void prepareAndChangeResolution() throws Exception {
        Exception exception = null;

        MediaCodecInfo videoCodecInfo = selectCodec(OUTPUT_VIDEO_MIME_TYPE);
        if (videoCodecInfo == null)
            return;
        MediaCodecInfo audioCodecInfo = selectCodec(OUTPUT_AUDIO_MIME_TYPE);
        if (audioCodecInfo == null)
            return;

        MediaExtractor videoExtractor = null;
        MediaExtractor audioExtractor = null;
        OutputSurface outputSurface = null;
        MediaCodec videoDecoder = null;
        MediaCodec audioDecoder = null;
        MediaCodec videoEncoder = null;
        MediaCodec audioEncoder = null;
        MediaMuxer muxer = null;
        InputSurface inputSurface = null;
        try {
            videoExtractor = createExtractor();
            int videoInputTrack = getAndSelectVideoTrackIndex(videoExtractor);
            MediaFormat inputFormat = videoExtractor.getTrackFormat(videoInputTrack);

            MediaMetadataRetriever m = new MediaMetadataRetriever();
            m.setDataSource(mInputFile);
            int inputWidth, inputHeight;
            try {
                inputWidth = Integer.parseInt(m.extractMetadata(MediaMetadataRetriever.METADATA_KEY_VIDEO_WIDTH));
                inputHeight = Integer.parseInt(m.extractMetadata(MediaMetadataRetriever.METADATA_KEY_VIDEO_HEIGHT));
            } catch (Exception e) {
                Bitmap thumbnail = m.getFrameAtTime();
                inputWidth = thumbnail.getWidth();
                inputHeight = thumbnail.getHeight();
                thumbnail.recycle();
            }

            if(inputWidth>inputHeight){
                if(mWidth<mHeight){
                    int w = mWidth;
                    mWidth=mHeight;
                    mHeight=w;
                }
            }
            else{
                if(mWidth>mHeight){
                    int w = mWidth;
                    mWidth=mHeight;
                    mHeight=w;
                }
            }

            MediaFormat outputVideoFormat =
                    MediaFormat.createVideoFormat(OUTPUT_VIDEO_MIME_TYPE, mWidth, mHeight);
            outputVideoFormat.setInteger(
                    MediaFormat.KEY_COLOR_FORMAT, OUTPUT_VIDEO_COLOR_FORMAT);
            outputVideoFormat.setInteger(MediaFormat.KEY_BIT_RATE, OUTPUT_VIDEO_BIT_RATE);
            outputVideoFormat.setInteger(MediaFormat.KEY_FRAME_RATE, OUTPUT_VIDEO_FRAME_RATE);
            outputVideoFormat.setInteger(
                    MediaFormat.KEY_I_FRAME_INTERVAL, OUTPUT_VIDEO_IFRAME_INTERVAL);

            AtomicReference<Surface> inputSurfaceReference = new AtomicReference<Surface>();
            videoEncoder = createVideoEncoder(
                    videoCodecInfo, outputVideoFormat, inputSurfaceReference);
            inputSurface = new InputSurface(inputSurfaceReference.get());
            inputSurface.makeCurrent();

            outputSurface = new OutputSurface();
            videoDecoder = createVideoDecoder(inputFormat, outputSurface.getSurface());

            audioExtractor = createExtractor();
            int audioInputTrack = getAndSelectAudioTrackIndex(audioExtractor);
            MediaFormat inputAudioFormat = audioExtractor.getTrackFormat(audioInputTrack);
            MediaFormat outputAudioFormat =
                MediaFormat.createAudioFormat(inputAudioFormat.getString(MediaFormat.KEY_MIME),
                        inputAudioFormat.getInteger(MediaFormat.KEY_SAMPLE_RATE),
                        inputAudioFormat.getInteger(MediaFormat.KEY_CHANNEL_COUNT));
            outputAudioFormat.setInteger(MediaFormat.KEY_BIT_RATE, OUTPUT_AUDIO_BIT_RATE);
            outputAudioFormat.setInteger(MediaFormat.KEY_AAC_PROFILE, OUTPUT_AUDIO_AAC_PROFILE);

            audioEncoder = createAudioEncoder(audioCodecInfo, outputAudioFormat);
            audioDecoder = createAudioDecoder(inputAudioFormat);

            muxer = new MediaMuxer(mOutputFile, MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);

            changeResolution(videoExtractor, audioExtractor,
                    videoDecoder, videoEncoder,
                    audioDecoder, audioEncoder,
                    muxer, inputSurface, outputSurface);
        } finally {
            try {
                if (videoExtractor != null)
                    videoExtractor.release();
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (audioExtractor != null)
                    audioExtractor.release();
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (videoDecoder != null) {
                    videoDecoder.stop();
                    videoDecoder.release();
                }
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (outputSurface != null) {
                    outputSurface.release();
                }
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (videoEncoder != null) {
                    videoEncoder.stop();
                    videoEncoder.release();
                }
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (audioDecoder != null) {
                    audioDecoder.stop();
                    audioDecoder.release();
                }
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (audioEncoder != null) {
                    audioEncoder.stop();
                    audioEncoder.release();
                }
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (muxer != null) {
                    muxer.stop();
                    muxer.release();
                }
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
            try {
                if (inputSurface != null)
                    inputSurface.release();
            } catch(Exception e) {
                if (exception == null)
                    exception = e;
            }
        }
        if (exception != null)
            throw exception;
    }

    private MediaExtractor createExtractor() throws IOException {
        MediaExtractor extractor;
        extractor = new MediaExtractor();
        extractor.setDataSource(mInputFile);
        return extractor;
    }

    private MediaCodec createVideoDecoder(MediaFormat inputFormat, Surface surface) throws IOException {
        MediaCodec decoder = MediaCodec.createDecoderByType(getMimeTypeFor(inputFormat));
        decoder.configure(inputFormat, surface, null, 0);
        decoder.start();
        return decoder;
    }

    private MediaCodec createVideoEncoder(MediaCodecInfo codecInfo, MediaFormat format,
            AtomicReference<Surface> surfaceReference) throws IOException {
        MediaCodec encoder = MediaCodec.createByCodecName(codecInfo.getName());
        encoder.configure(format, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
        su