How to cross-compile under Linux

Recently I’ve been asked by a few people how to crosscompile programs/packages under Linux. This happened especially after the article on customizing the DGN2200v3 (you can find here) but also for work reasons. In this article I’ll try to give some insights on this operation that is often feared, but is generally not a very hard task (yes I said generally: try for example crosscompiling something like the openjdk and you’ll understand what I mean!). This article will try to be generic but with some more examples on the DGN2200v3 crosscompilation and some ARM references.

Get and install the right tools

The first thing to do if of course to install on the system you are working (which is called host system) the tools that run on the host and will permit you to compile something for the target system you want to compile the programs for.
This set of programming tools is usually refeered as a toolchain and contains various things: usually one or a few compilers, one or a few linkers, an assembler, various header files, little or a lot of libraries and other simillar tools. If you are lucky you may get also a debugger packed. All this tools of course are compiled to run on your host machine but produce binaries that run on the target. Depending on how the toolchain was created you may also find supplied with it some set of tools that are ready to be run on the target to help your crosscompilation adventure.
This operation may be very easy if you somehow received the crosscompiler package ready (as for example the DGN2200v3 you just download it from the Netgear site or you collegue may have passed it to you) as you just have to unpack it somewhere and then refeer to it.

The second option is that you have to find a crosscompiler for yourself. In this case before starting to query your favourite search engine you should generally try to understand from your target two main things: the architecture (this is the processor/system type, such as MIPS or ARM or m68k to name a few; and then some more details such as if talking about ARM it’s an v4 core v5 or v6 an so on) and the library set used on the target (this is often uclibc for smaller systems or glibc for more powerfull systems, then again which version of the library and in some cases also the type of library interface such as EABI or old-ABI).
To get the architecture information you can try to read some files in the /proc and /sys trees (/proc/cpuinfo is a very good start, then again many files under /sys/devices/system/cpu) or use the uname (-m) tool even better you can use, if installed, the lscpu (from the util-linux package) program from command line. Also you could try using the file tool if present on a preexisting binary to get some informations about it. In some cases you may need to do a crosscheck using the CPU name of your product with the Linux kernel sources or with the CPU datasheet at worse.
On the other side to find the used libraries, given that you don’t want to compile just statically linked binaries in which case this information may be irrelevant, you may have to examine the contents of the /lib and /usr/lib directories and try to figure out the type of library used (check for example if you see and libuClibc or which version the libc is).
Now with this two informations you should find a crosscompiler that has the same architecture target and the same (sometimes a very close version may work aswell but may be risky in some cases) library set.
A very good site where you can find many interesting free crosscompilers (you can then purchase an advanced IDE or support aswell) is for example CodeSourcery but of course it may depend on your needs.

The third situation one may come to is that there is no available toolchain ready to use and one has to be created from scratch. This operation is of course quite more advanced and I’ll not dig much into this possibility in this article. Let me just point out a great project to create toolchains that I used often that may be of great use if you are in this situation: crosstool-ng. This tool gives you the possibility to create in a quite “easy” way your personal optimized toolchain.

Now that you obtained the toolchain usually you would unpack it someplace (for example in /opt or so). Usually it would look like a “root filesystem” with a /bin directory (containing the binaries such as the compiler), the /lib for the libraries and so on. Give a look and explore it.
For ease of use you should put the binaries directory into your path so in the next step when compiling something you can easily find all the tools needed by your compilation procedure:

fede@sphynx:~$ export PATH=$PATH:/opt/toolchains/uclibc-crosstools-gcc-4.4.2-1/usr/bin/
fede@sphynx:~$ mips-linux-gcc -v
Using built-in specs.
Target: mips-linux-uclibc
Configured with: /shared/myviews/toolchain/buildroot-4.4.2-1/output/toolchain/gcc-4.4.2/configure --prefix=/usr --build=i386-pc-linux-gnu --host=i386-pc-linux-gnu 
--target=mips-linux-uclibc --enable-languages=c,c++ --with-sysroot=/opt/toolchains/uclibc-crosstools-gcc-4.4.2-1 --with-build-time-tools=/opt/toolchains/uclibc-
crosstools-gcc-4.4.2-1/usr/mips-linux-uclibc/bin --disable-__cxa_atexit --enable-target-optspace --with-gnu-ld --disable-libssp --disable-tls --enable-shared 
--with-gmp=/shared/myviews/toolchain/buildroot-4.4.2-1/output/toolchain/gmp --with-mpfr=/shared/myviews/toolchain/buildroot-4.4.2-1/output/toolchain/mpfr 
--disable-nls --enable-threads --disable-multilib --disable-decimal-float --with-float=soft --with-abi=32 --with-tune=mips32 --with-arch=mips32 --with-
pkgversion='Buildroot 2010.02-git' --with-bugurl=http://bugs.buildroot.net/
Thread model: posix
gcc version 4.4.2 (Buildroot 2010.02-git)

fede@sphynx:~$ export PATH=$PATH:/opt/toolchains/arm-crunch-linux-gnueabi/bin/
fede@sphynx:~$ arm-crunch-linux-gnueabi-gcc -v
Using built-in specs.
Target: arm-crunch-linux-gnueabi
Configured with: /opt/toolchains/crosstool-ng-1.4.2/targets/src/gcc-4.2.4/configure --build=i486-build_pc-linux-gnu --host=i486-build_pc-linux-gnu --target=arm-
crunch-linux-gnueabi --prefix=/opt/toolchains/arm-crunch-linux-gnueabi --with-sysroot=/opt/toolchains/arm-crunch-linux-gnueabi/arm-crunch-linux-gnueabi//sys-root 
--enable-languages=c --disable-multilib --with-cpu=ep9312 --with-fpu=maverick --with-float=soft --with-gmp=/opt/toolchains/arm-crunch-linux-gnueabi --with-
mpfr=/opt/toolchains/arm-crunch-linux-gnueabi --disable-sjlj-exceptions --enable-__cxa_atexit --with-local-prefix=/opt/toolchains/arm-crunch-linux-gnueabi
/arm-crunch-linux-gnueabi//sys-root --disable-nls --enable-threads=posix --enable-symvers=gnu --enable-c99 --enable-long-long --enable-target-optspace
Thread model: posix
gcc version 4.2.4

As you see we can now call the gcc compiler (with the appropriate prefix depending on the toolchain compilation) and see it’s version. The first example is the DGN2200v3 compiler shipped by Netgear while the second one is a custom build ARM9 Cirrus Logic EP9302 compiler created using the crosstool-ng package. Now we can then proceed to the next step.

Compiling

Now that you have the tools you can start compiling what you want to run on the target machine. Of course the compilation step may very different depending if you want to compile a single .c file with no libraries used, if you want to compile a program with a makefile, a program that uses an AutoConf autoconfiguration script, the Linux Kernel or something else.
Let’s start from the most basic single file hello.c program such as:

#include <stdio.h>
#include <stdlib.h>

int main(void) {
        printf("Hello world! (%s %s)\n",__DATE__,__TIME__);
}

In this case just the command line invocation should do it:

mips-linux-gcc hello.c -o hello

Will build an advanced hello world program for the DGN2200N router. Be aware that you may need to pass some more flags to the compiler in some cases. A typical example is when using an ARM compiler that supports more architectures (for example ARMv4 and ARMv5, or both thumb and not-thumb instruction set) and not forcing a specific one may result in the compilation working perfectly but the resulting binary acting in a very weird way! (in this specific case the important option to use is -march for example -march=armv4t)
To see if the file was really crosscompiled correctly you can use again the file tool on the generated file, for example:

fede@sphynx:~/dgn2200$ file hello
hello: ELF 32-bit MSB executable, MIPS, MIPS32 version 1 (SYSV), dynamically linked (uses shared libs),
 with unknown capability 0x41000000 = 0xf676e75, with unknown capability 0x10000 = 0x70403, not stripped

or another example:

fede@sphynx:~/arm/new_fs/usr/bin$ file inadyn 
inadyn: ELF 32-bit LSB executable, ARM, version 1 (SYSV), dynamically linked (uses shared libs),
 for GNU/Linux 2.6.14, not stripped

The second case you may walk upon is a project with a Makefile that builds the project using various files in the source tree. In this case you should replace all the references to the compiler, linker and other tools with the references to your crosscompiler. If the Makefile was done in a clear way this should be defined at it’s beginning with variabiles such as CC, LD, CFLAGS, LDFLAGS and so on. In some cases you may even just have a CROSS-COMPILE variable that gets prepended automatically to all the tools. Of course everything depends on the specific Makefile (and mostly on its author!) so you have to examine it.

Another common scenario you may find yourself in front of is a package that uses the AutoConf scripting system to automatically detect what is installed on your system and create accordingly correct Makefiles to proceed with the compilation. When compiling for your local system you would most probably just do a:

./configure
make

and the trick would be done: the first command would search your system for required files (headers, libraries and so on) and the second would compile the package. Of course the first step may result in missing dependencies and so on.
When crosscompiling the main difference is that you should tell to configure that you are doing so using the built in options (./configure –help):

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]
  --target=TARGET   configure for building compilers for TARGET [HOST]

So adding –target=, for example –target=mips-linux should do the trick, the configure will understand it’s a crosscompilation and create the makefiles accordigly. If you need to pass some flags to the compiler or linker setting the CFLAGS and LDFLAGS on the command line to configure is suggested.
So briefly for example:

CFLAGS="-cflags -morecflags" LDFLAGS="-linkerflags" ./configure --host=arch-linux
make

Could be a generic recipe.
Of course this may not be all: if the package (and it’s configure) requires some extra libraries you may need to first crosscompile and place them someplace (inside the toolchain tree may be a good option to have them shared in the future) using a simillar approach (and eventually adding the paths to where you installed the libraries to configure). Of course on the other side as when compiling for local usage you may decide to disable some option that is giving you compilation headaches and you don’t really need it.
Another thing may happen that could give you headaches: sometimes the configure script is created in a way (ignoring maybe a bit the cross-compilation needs) that it will need some binaries from the package to be executed (for example to create a temporary table with data or so). In this case of course you’ll need to cheat the build system creating it first with your local host compiler and then doing the crosscompilation step. One very useful option of configure is the –cache option which gives you the possibility to pass a text file that contains certain answers that you prepare by hand to avoid configure troubles. The format of the text file is quite simple, here is an example to predefine some glib variables since they are usually generated by a glib support tool:

glib_cv_long_long_format=ll
glib_cv_stack_grows=no
ac_cv_func_posix_getpwuid_r=yes
glib_cv_uscore=yes
ac_cv_have_abstract_sockets=yes

Remember also that the compilation tests (often found as make test rule) cannot be run when crosscompiling for obious reasons.

Last (but not least) a few words about the Linux Kernel crosscompilation. In this case in general you have to set the ARCH variable to the architecture name (ie. ARM, MIPS) and the CROSS_COMPILE variable to the prefix of your toolchain tools (ie. mips-linux- or arm-linux-gnueabi- ; notice the final dash that is not required when we invoke configure!) and then you can proceed as usual with your favourite configuration step (make configure, menuconfig, xconfig and so on) and then build step.

export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabi-
make menuconfig
...
make

Be aware that on embedded systems since a particullar boot loader is used the build step may be different from what you’re used on your PC (make zImage or bzImage or so) since the final format has to be understood by the boot loader. For example when preparing an image for the popular U-Boot boot loader you should invoke make uImage to create an appropriate image (this will require also the mkimage tool to be installed on your system).

Other hints:

Be aware that libtool while generating libtool libraries files (.la extension) inserts the full path. So be carefull that you’re not mixing host/target paths accidently if you crosscompilation is requiring libraries creation.
Many packages nowadays use pkg-config tool to query the system where some libraries are installed and which command line options are required to link and use them. Setting the environment variable PKG_CONFIG_PATH to the directory containing the .pc files for your target (that could often be found inside the toolchain directory tree) is therefore a very good idea not to have to tinker manually huge library flags every time.

Last but not least: if you have a complicated project to crosscompile and you’re lucky enough to have some mainstream developement board available for that architecture (for example if you’re working with ARM something like the OpenRD or Raspberry PI are good choices, or an emulated ARM machine using QEmu is also a good but longer shot) don’t insist banging your head on the crosscompiler, it’s fine if you do everything on the target platform, nothing to be ashamed of 😉

Comments #

• on 2013-05-04 08:02:41 +0100

Awesome post.