openwrt - How to get a smaller toolchain from scratch?

Question

I have built a custom toolchain from scratch for a MIPS 24kc (dragino) target platform, using gcc-6.3.0, musl-1.1.16 and binutils-2.27. That toolchain is completely functional.

However, the size of my-custom toolchain is five times bigger than size of equivalent OpenWRT generated toolchain (557M vs 113M). Toolchain binaries generated by OpenWRT (mips-openwrt-linux-musl-*) are dynamically linked against libstdc++ and libgcc_s libraries. However, my toolchain binaries are linked with these libs statically:

$ ldd mips-openwrt-linux-musl-gcc-5.3.0
    linux-vdso.so.1 =>  (0x00007ffc4d534000)
    libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f878936f000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8789159000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8788d8f000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8788a86000)
    /lib64/ld-linux-x86-64.so.2 (0x000055df3ef8b000)

$ ldd mips-linux-musl-gcc-6.3.0
    linux-vdso.so.1 =>  (0x00007ffd40940000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f2de4b8f000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2de47c6000)
    /lib64/ld-linux-x86-64.so.2 (0x0000560bba4b6000)

The same linking issue happens with cc1 and cc1plus. Sizes of these two binaries are huge (cc1 is 124MB, cc1plus is 134MB).

My question is: what is the proper way to setup the cross-toolchain generation to get a cross-platform toolchain as small as the OpenWRT generated toolchain?

Bellow you can find the steps I'm doing to build my-custom toolchain for reference:

Download sources:

wget http://ftpmirror.gnu.org/binutils/binutils-2.27.tar.bz2                
wget http://ftpmirror.gnu.org/gcc/gcc-6.3.0/gcc-6.3.0.tar.bz2               
wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.12.6.tar.bz2      
wget https://www.musl-libc.org/releases/musl-1.1.16.tar.gz                  
wget http://ftpmirror.gnu.org/mpfr/mpfr-3.1.5.tar.bz2                       
wget http://ftpmirror.gnu.org/gmp/gmp-6.1.2.tar.bz2                         
wget http://ftpmirror.gnu.org/mpc/mpc-1.0.2.tar.gz       

Setup build environment:

export ROOTDIR="${HOME}/custom-toolchains/MIPS"                             
export NATIVE_PREFIX="${ROOTDIR}/opt/native"                                
export CROSS_PREFIX="${ROOTDIR}/opt/cross"                                  
export TARGET_MACHINE=mips                                                  
export CPU=mips                                                             
export ARCH=24kc                                                            
export CLIB=musl                                                            
export TARGET_TRIPLET=${CPU}-linux-${CLIB}  

Build native binutils:

cd ${ROOTDIR}/src                                                           
mkdir build-binutils                                                        
cd build-binutils                                                           
../binutils-2.27/configure --prefix="${NATIVE_PREFIX}" --disable-nls --disable-werror --disable-multilib
make                                                                        
make install  

Build native gcc:

cd ${ROOTDIR}/src/gcc-6.3.0                                                 
ln -s ../mpfr-3.1.5 mpfr                                                    
ln -s ../gmp-6.1.2 gmp                                                      
ln -s ../mpc-1.0.3 mpc                                                      
cd ..                                                                       
mkdir build-gcc                                                             
cd build-gcc                                                                
../gcc-6.3.0/configure --prefix=${NATIVE_PREFIX} --disable-nls --enable-languages=c --disable-multilib
make                                                                        
make install   

Build cross-binutils:

cd ${ROOTDIR}/src                                                           
mkdir build-${CPU}-binutils                                                 
cd build-${CPU}-binutils                                                    
../binutils-2.27/configure --target=${TARGET_TRIPLET} --prefix=${CROSS_PREFIX} --with-sysroot --disable-nls --disable-werror --disable-multilib
make                                                                        
make install  

Install kernel headers:

cd ${ROOTDIR}/src                                                           
cd linux-3.12.6                                                             
make ARCH=${TARGET_MACHINE} INSTALL_HDR_PATH=${CROSS_PREFIX}/${TARGET_TRIPLET} headers_install

Build cross-gcc (stage 1)

cd ${ROOTDIR}/src                                                           
mkdir build-bootstrap-${CPU}-gcc                                            
cd build-bootstrap-${CPU}-gcc                                               
../gcc-6.3.0/configure --target=${TARGET_TRIPLET} --prefix=${CROSS_PREFIX} --disable-nls --enable-languages=c --disable-multilib --disable-threads --disable-shared --with-float=soft --with-arch=${ARCH}
make all-gcc install-gcc                                                    
make all-target-libgcc install-target-libgcc   

Build cross-musl

cd ${ROOTDIR}/src                                                           
mkdir build-${CLIB}                                                         
cd build-${CLIB}                                                            
CC=${TARGET_TRIPLET}-gcc CFLAGS=-Wa,-msoft-float ../musl-1.1.16/configure --prefix=${CROSS_PREFIX}/${TARGET_TRIPLET}/ --enable-optimize CROSS_COMPILE=${TARGET_TRIPLET}-
make                                                                        
make install     

Build cross-gcc (stage 2)

cd ${ROOTDIR}/src                                                           
mkdir build-${CPU}-gcc                                                      
cd build-${CPU}-gcc                                                         
../gcc-6.3.0/configure --target=${TARGET_TRIPLET} --prefix=${CROSS_PREFIX} --disable-nls --enable-languages=c,c++ --disable-multilib --enable-threads --enable-shared --with-float=soft --with-arch=${ARCH} --enable-target-optspace --disable-libgomp --disable-libmudflap --without-isl --without-cloog --disable-decimal-float --disable-libssp --disable-libsanitizer --enable-lto --with-host-libstdcxx=-lstdc++
make                                                                        
make install  

Thank you for all your comments in advance!

Answer 1

Fixing this issue has been simpler than expected... All toolchain binaries are installed non-stripped. So this can be fixed calling

$ make install-strip

instead of

$ make install

when you install cross-gcc (stage 2).

Size of generated toolchain is 122MB, what is nice compared with 557M of the original toolchain. So this issue is fixed! Hope this information will be useful for somebody else in the future. Thank you!