audio - Playing multiple byte arrays simultaneously in Java

Question

How can you play multiple (audio) byte arrays simultaneously? This "byte array" is recorded by TargetDataLine, transferred using a server.

What I've tried so far

Using SourceDataLine:

There is no way to play mulitple streams using SourceDataLine, because the write method blocks until the buffer is written. This problem cannot be fixed using Threads, because only one SourceDataLine can write concurrently.

Using the AudioPlayer Class:

ByteInputStream stream2 = new ByteInputStream(data, 0, data.length);
AudioInputStream stream = new AudioInputStream(stream2, VoiceChat.format, data.length);
AudioPlayer.player.start(stream);

This just plays noise on the clients.

EDIT I don't receive the voice packets at the same time, it's not simultaneously, more "overlapping".

Answer 1

Apparently Java's Mixer interface was not designed for this.

http://docs.oracle.com/javase/7/docs/api/javax/sound/sampled/Mixer.html:

A mixer is an audio device with one or more lines. It need not be designed for mixing audio signals.

And indeed, when I try to open multiple lines on the same mixer this fails with a LineUnavailableException. However if all your audio recordings have the same audio format it's quite easy to manually mix them together. For example if you have 2 inputs:

1. Convert both to the appropriate data type (for example byte[] for 8 bit audio, short[] for 16 bit, float[] for 32 bit floating point etc)
2. Sum them in another array. Make sure summed values do not exceed the range of the datatype.
3. Convert output back to bytes and write that to the SourceDataLine

See also How is audio represented with numbers?

Here's a sample mixing down 2 recordings and outputting as 1 signal, all in 16bit 48Khz stereo.

    // print all devices (both input and output)
    int i = 0;
    Mixer.Info[] infos = AudioSystem.getMixerInfo();
    for (Mixer.Info info : infos)
        System.out.println(i++ + ": " + info.getName());

    // select 2 inputs and 1 output
    System.out.println("Select input 1: ");
    int in1Index = Integer.parseInt(System.console().readLine());
    System.out.println("Select input 2: ");
    int in2Index = Integer.parseInt(System.console().readLine());
    System.out.println("Select output: ");
    int outIndex = Integer.parseInt(System.console().readLine());

    // ugly java sound api stuff
    try (Mixer in1Mixer = AudioSystem.getMixer(infos[in1Index]);
            Mixer in2Mixer = AudioSystem.getMixer(infos[in2Index]);
            Mixer outMixer = AudioSystem.getMixer(infos[outIndex])) {
        in1Mixer.open();
        in2Mixer.open();
        outMixer.open();
        try (TargetDataLine in1Line = (TargetDataLine) in1Mixer.getLine(in1Mixer.getTargetLineInfo()[0]);
                TargetDataLine in2Line = (TargetDataLine) in2Mixer.getLine(in2Mixer.getTargetLineInfo()[0]);
                SourceDataLine outLine = (SourceDataLine) outMixer.getLine(outMixer.getSourceLineInfo()[0])) {

            // audio format 48khz 16 bit stereo (signed litte endian)
            AudioFormat format = new AudioFormat(48000.0f, 16, 2, true, false);

            // 4 bytes per frame (16 bit samples stereo)
            int frameSize = 4;
            int bufferSize = 4800;
            int bufferBytes = frameSize * bufferSize;

            // buffers for java audio
            byte[] in1Bytes = new byte[bufferBytes];
            byte[] in2Bytes = new byte[bufferBytes];
            byte[] outBytes = new byte[bufferBytes];

            // buffers for mixing
            short[] in1Samples = new short[bufferBytes / 2];
            short[] in2Samples = new short[bufferBytes / 2];
            short[] outSamples = new short[bufferBytes / 2];

            // how long to record & play
            int framesProcessed = 0;
            int durationSeconds = 10;
            int durationFrames = (int) (durationSeconds * format.getSampleRate());

            // open devices
            in1Line.open(format, bufferBytes);
            in2Line.open(format, bufferBytes);
            outLine.open(format, bufferBytes);
            in1Line.start();
            in2Line.start();
            outLine.start();

            // start audio loop
            while (framesProcessed < durationFrames) {

                // record audio
                in1Line.read(in1Bytes, 0, bufferBytes);
                in2Line.read(in2Bytes, 0, bufferBytes);

                // convert input bytes to samples
                ByteBuffer.wrap(in1Bytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(in1Samples);
                ByteBuffer.wrap(in2Bytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(in2Samples);

                // mix samples - lower volume by 50% since we're mixing 2 streams
                for (int s = 0; s < bufferBytes / 2; s++)
                    outSamples[s] = (short) ((in1Samples[s] + in2Samples[s]) * 0.5);

                // convert output samples to bytes
                ByteBuffer.wrap(outBytes).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().put(outSamples);

                // play audio
                outLine.write(outBytes, 0, bufferBytes);

                framesProcessed += bufferBytes / frameSize;
            }

            in1Line.stop();
            in2Line.stop();
            outLine.stop();
        }
    }