Discussion:
Updating kernel.org cross compilers?
(too old to reply)
Andre Przywara
2017-04-26 14:14:16 UTC
Permalink
Hi!

(Tony: I've seen you redirecting to "Chris" in an older email, but the
web archive doesn't have his email address)

It seems that many people (even outside the Linux kernel community) use
the cross compilers provided at kernel.org/pub/tools/crosstool.
The latest compiler I find there is 4.9.0, which celebrated its third
birthday at the weekend, also has been superseded by 4.9.4 meanwhile.

So I took Segher's buildall scripts from [1] and threw binutils 2.28 and
GCC 6.3.0 at them. After removing --enable-sjlj-exceptions from
build-gcc and adding --disable-multilib (for building x86-64 on a x86-64
box without 32-bit libs) I was able to build (bare-metal) toolchains for
all architectures except arc, m68k, tilegx and tilepro.
Now my understanding is that a baremetal/stage 1 compiler should be
build with "all-gcc" instead of the implicit "all" make target, and
"install-gcc" instead of "install". And indeed adding this to build-gcc
makes all toolchains build now - except arc, which needs an explicit
"--with-cpu=arc700" on the GCC configure command line.

So after those changes, the sequence:
$ ./buildall --toolchain
$ PATH=$PATH:/opt/cross/bin
$ ./buildall --kernel
worked reasonably well for me (binutils 2.28, GCC 6.3.0, Linux 4.11-rc8
on an Intel host with Ubuntu 14.04.5). Just tilegx/pro is still broken
due to toolchain issues.

So I wonder what the process is to get those compilers updated?

And what is a good build setup, so that the binaries run on as many
systems as possible?

Also the web page (https://www.kernel.org/pub/tools/crosstool) seems to
only link outdated builds (4.6.3, mostly), is that on purpose?

Cheers,
Andre.

[1] http://git.infradead.org/users/segher/buildall.git/
Guenter Roeck
2017-04-30 03:37:49 UTC
Permalink
Post by Andre Przywara
Hi!
(Tony: I've seen you redirecting to "Chris" in an older email, but the
web archive doesn't have his email address)
It seems that many people (even outside the Linux kernel community) use
the cross compilers provided at kernel.org/pub/tools/crosstool.
The latest compiler I find there is 4.9.0, which celebrated its third
birthday at the weekend, also has been superseded by 4.9.4 meanwhile.
So I took Segher's buildall scripts from [1] and threw binutils 2.28 and
GCC 6.3.0 at them. After removing --enable-sjlj-exceptions from
build-gcc and adding --disable-multilib (for building x86-64 on a x86-64
box without 32-bit libs) I was able to build (bare-metal) toolchains for
all architectures except arc, m68k, tilegx and tilepro.
Now my understanding is that a baremetal/stage 1 compiler should be
build with "all-gcc" instead of the implicit "all" make target, and
"install-gcc" instead of "install". And indeed adding this to build-gcc
makes all toolchains build now - except arc, which needs an explicit
"--with-cpu=arc700" on the GCC configure command line.
$ ./buildall --toolchain
$ PATH=$PATH:/opt/cross/bin
$ ./buildall --kernel
worked reasonably well for me (binutils 2.28, GCC 6.3.0, Linux 4.11-rc8
on an Intel host with Ubuntu 14.04.5). Just tilegx/pro is still broken
due to toolchain issues.
Those need libgcc, and to build libgcc you'll have to install the kernel
header files. Same is true for cris (and possibly others).

Also, crisv32 builds as well.
Post by Andre Przywara
So I wonder what the process is to get those compilers updated?
If you find out let me know.

Guenter
Post by Andre Przywara
And what is a good build setup, so that the binaries run on as many
systems as possible?
Also the web page (https://www.kernel.org/pub/tools/crosstool) seems to
only link outdated builds (4.6.3, mostly), is that on purpose?
Cheers,
Andre.
[1] http://git.infradead.org/users/segher/buildall.git/
Segher Boessenkool
2017-04-30 05:29:57 UTC
Permalink
Hi,
Post by Andre Przywara
It seems that many people (even outside the Linux kernel community) use
the cross compilers provided at kernel.org/pub/tools/crosstool.
The latest compiler I find there is 4.9.0, which celebrated its third
birthday at the weekend, also has been superseded by 4.9.4 meanwhile.
So I took Segher's buildall scripts from [1] and threw binutils 2.28 and
GCC 6.3.0 at them.
Happy to see people are still using these!
Post by Andre Przywara
After removing --enable-sjlj-exceptions from build-gcc
This was needed to build some targets. It does prevent aarch64 from
building without patch.
Post by Andre Przywara
and adding --disable-multilib (for building x86-64 on a x86-64
box without 32-bit libs)
Why is this needed? What error are you seeing.
Post by Andre Przywara
I was able to build (bare-metal) toolchains for
all architectures except arc, m68k, tilegx and tilepro.
arc needs a more recent GCC; the other probably as well. GCC 7 should
be out very soon, you probably want to wait for that :-)
Post by Andre Przywara
$ ./buildall --toolchain
$ PATH=$PATH:/opt/cross/bin
$ ./buildall --kernel
You should have the target dir in your PATH before doing anything else.
Is this not documented? Hrm I guess not, let me fix that.
Post by Andre Przywara
And what is a good build setup, so that the binaries run on as many
systems as possible?
Run contrib/download_prerequisites in the gcc source dir: this will
make GMP, MPFR, MPC statically linked, and use a version of each that
is known to work (and work correctly).
Post by Andre Przywara
[1] http://git.infradead.org/users/segher/buildall.git/
I'll push some changes that make aarch64 work out-of-the-box in a
minute. It does some other things as well (e.g., make ia64 work again
after *that* change); more changes are still needed to make sh4 work
without patches.


Segher
Andre Przywara
2017-05-09 14:59:27 UTC
Permalink
On 30/04/17 06:29, Segher Boessenkool wrote:

Hi,
Post by Segher Boessenkool
Post by Andre Przywara
It seems that many people (even outside the Linux kernel community) use
the cross compilers provided at kernel.org/pub/tools/crosstool.
The latest compiler I find there is 4.9.0, which celebrated its third
birthday at the weekend, also has been superseded by 4.9.4 meanwhile.
So I took Segher's buildall scripts from [1] and threw binutils 2.28 and
GCC 6.3.0 at them.
Happy to see people are still using these!
Post by Andre Przywara
After removing --enable-sjlj-exceptions from build-gcc
This was needed to build some targets. It does prevent aarch64 from
building without patch.
I saw you fixing that in your repo, thanks for that!
Post by Segher Boessenkool
Post by Andre Przywara
and adding --disable-multilib (for building x86-64 on a x86-64
box without 32-bit libs)
Why is this needed? What error are you seeing.
It was something along the lines of not finding 32-bit compat libraries
(which I don't have on that build machine):
------------------------
configure: error: I suspect your system does not have 32-bit development
libraries (libc and headers). If you have them, rerun configure with
--enable-multilib. If you do not have them, and want to build a
64-bit-only compiler, rerun configure with --disable-multilib.
------------------------

I don't think we need multilib for a kernel build, also we have an i386
compiler for 32-bit kernels. So adding this flags allows x86_64 to be
build on pure 64-bit systems.
Post by Segher Boessenkool
Post by Andre Przywara
I was able to build (bare-metal) toolchains for
all architectures except arc, m68k, tilegx and tilepro.
arc needs a more recent GCC; the other probably as well. GCC 7 should
be out very soon, you probably want to wait for that :-)
Well, GCC 7 indeed builds better, but then again is a very new compiler.
For instance in the moment it spits a lot of warnings when compiling the
kernel (mostly due to some *printf analysis). It's not hard to fix, but
this will take a while to trickle in and it's questionable whether this
will be backported everywhere.
So in addition to GCC 7.1 I'd like to have at least GCC 6.3 around,
which builds kernels without warnings today.

For GCC 6.3 (and probably before) arc was breaking because missing a
(default) CPU type. Adding "--with-cpu=arc700" to EXTRA_GCC_CONF fixed
that, but GCC 7 indeed builds fine, even without it.
I was wondering if we could have that flags added to the new
TARGET_GCC_CONF variable for arc-elf, to cover pre GCC 7 compilers.
Post by Segher Boessenkool
Post by Andre Przywara
$ ./buildall --toolchain
$ PATH=$PATH:/opt/cross/bin
$ ./buildall --kernel
You should have the target dir in your PATH before doing anything else.
Is this not documented? Hrm I guess not, let me fix that.
Post by Andre Przywara
And what is a good build setup, so that the binaries run on as many
systems as possible?
Run contrib/download_prerequisites in the gcc source dir: this will
make GMP, MPFR, MPC statically linked, and use a version of each that
is known to work (and work correctly).
Ah, that's a good hint! Thanks, that solves a lot of problems.

Also I removed documentation (share/ directory, which won't be in
MANPATH mostly anyway) from the tarball and stripped the (host) binaries
to get the tarball size down (to about 16 MB per arch)

I built both toolchains and kernels for almost all 31 supported
architectures. Some kernel builds fails (sparc, sparc64, arc), but not
due to toolchain issues, as it seems. Only sh4 complains:
sh4-linux-gcc: error: command line option '-m4-nofpu' is not supported
by this configuration.

So while I have now 31 GCC 7.1.0 tarballs, pushing those binaries to
some public webspace raises some legal eyebrows here, so I was wondering
if someone (probably with less affiliation to a hardware (arch) vendor)
could build, package and upload them? I am happy to assist with that
process.

Cheers,
Andre.
Post by Segher Boessenkool
Post by Andre Przywara
[1] http://git.infradead.org/users/segher/buildall.git/
I'll push some changes that make aarch64 work out-of-the-box in a
minute. It does some other things as well (e.g., make ia64 work again
after *that* change); more changes are still needed to make sh4 work
without patches.
Segher
Guenter Roeck
2017-05-09 16:26:55 UTC
Permalink
Post by Andre Przywara
Hi,
Post by Segher Boessenkool
Post by Andre Przywara
It seems that many people (even outside the Linux kernel community) use
the cross compilers provided at kernel.org/pub/tools/crosstool.
The latest compiler I find there is 4.9.0, which celebrated its third
birthday at the weekend, also has been superseded by 4.9.4 meanwhile.
So I took Segher's buildall scripts from [1] and threw binutils 2.28 and
GCC 6.3.0 at them.
Happy to see people are still using these!
Post by Andre Przywara
After removing --enable-sjlj-exceptions from build-gcc
This was needed to build some targets. It does prevent aarch64 from
building without patch.
I saw you fixing that in your repo, thanks for that!
Post by Segher Boessenkool
Post by Andre Przywara
and adding --disable-multilib (for building x86-64 on a x86-64
box without 32-bit libs)
Why is this needed? What error are you seeing.
It was something along the lines of not finding 32-bit compat libraries
------------------------
configure: error: I suspect your system does not have 32-bit development
libraries (libc and headers). If you have them, rerun configure with
--enable-multilib. If you do not have them, and want to build a
64-bit-only compiler, rerun configure with --disable-multilib.
------------------------
I don't think we need multilib for a kernel build, also we have an i386
compiler for 32-bit kernels. So adding this flags allows x86_64 to be
build on pure 64-bit systems.
Post by Segher Boessenkool
Post by Andre Przywara
I was able to build (bare-metal) toolchains for
all architectures except arc, m68k, tilegx and tilepro.
arc needs a more recent GCC; the other probably as well. GCC 7 should
be out very soon, you probably want to wait for that :-)
Well, GCC 7 indeed builds better, but then again is a very new compiler.
For instance in the moment it spits a lot of warnings when compiling the
kernel (mostly due to some *printf analysis). It's not hard to fix, but
this will take a while to trickle in and it's questionable whether this
will be backported everywhere.
So in addition to GCC 7.1 I'd like to have at least GCC 6.3 around,
which builds kernels without warnings today.
For GCC 6.3 (and probably before) arc was breaking because missing a
(default) CPU type. Adding "--with-cpu=arc700" to EXTRA_GCC_CONF fixed
that, but GCC 7 indeed builds fine, even without it.
I was wondering if we could have that flags added to the new
TARGET_GCC_CONF variable for arc-elf, to cover pre GCC 7 compilers.
I think arcv2 requires gcc 7.
Post by Andre Przywara
Post by Segher Boessenkool
Post by Andre Przywara
$ ./buildall --toolchain
$ PATH=$PATH:/opt/cross/bin
$ ./buildall --kernel
You should have the target dir in your PATH before doing anything else.
Is this not documented? Hrm I guess not, let me fix that.
Post by Andre Przywara
And what is a good build setup, so that the binaries run on as many
systems as possible?
Run contrib/download_prerequisites in the gcc source dir: this will
make GMP, MPFR, MPC statically linked, and use a version of each that
is known to work (and work correctly).
Ah, that's a good hint! Thanks, that solves a lot of problems.
Also I removed documentation (share/ directory, which won't be in
MANPATH mostly anyway) from the tarball and stripped the (host) binaries
to get the tarball size down (to about 16 MB per arch)
I built both toolchains and kernels for almost all 31 supported
architectures. Some kernel builds fails (sparc, sparc64, arc), but not
Are those missing libgcc ? If so, I think it is a toolchain issue.
I solved it by adding all-target-libgcc / install-target-libgcc
to the build (and installing the kernel headers where needed, ie for
m68k and tilegx/tilepro).
Post by Andre Przywara
So while I have now 31 GCC 7.1.0 tarballs, pushing those binaries to
sh4-linux-gcc: error: command line option '-m4-nofpu' is not supported
What configuration are you trying to build, and how does your make command
line look like (wondering, because I don't see the problem) ?
Post by Andre Przywara
by this configuration.
some public webspace raises some legal eyebrows here, so I was wondering
if someone (probably with less affiliation to a hardware (arch) vendor)
could build, package and upload them? I am happy to assist with that
process.
How do you create the per-target tarballs ? The script in the
repository pushes eherything into a single target directory.

Thanks,
Guenter
Segher Boessenkool
2017-05-09 22:18:43 UTC
Permalink
Hello again,
Post by Andre Przywara
Post by Segher Boessenkool
Post by Andre Przywara
and adding --disable-multilib (for building x86-64 on a x86-64
box without 32-bit libs)
Why is this needed? What error are you seeing.
It was something along the lines of not finding 32-bit compat libraries
------------------------
configure: error: I suspect your system does not have 32-bit development
libraries (libc and headers). If you have them, rerun configure with
--enable-multilib. If you do not have them, and want to build a
64-bit-only compiler, rerun configure with --disable-multilib.
------------------------
I don't think we need multilib for a kernel build, also we have an i386
compiler for 32-bit kernels. So adding this flags allows x86_64 to be
build on pure 64-bit systems.
Oh hrm, so it is building a native compiler? I thought I got rid of
that everywhere, always build crosses instead. I'll investigate.
Post by Andre Przywara
Post by Segher Boessenkool
Post by Andre Przywara
I was able to build (bare-metal) toolchains for
all architectures except arc, m68k, tilegx and tilepro.
arc needs a more recent GCC; the other probably as well. GCC 7 should
be out very soon, you probably want to wait for that :-)
Well, GCC 7 indeed builds better, but then again is a very new compiler.
For instance in the moment it spits a lot of warnings when compiling the
kernel (mostly due to some *printf analysis). It's not hard to fix, but
this will take a while to trickle in and it's questionable whether this
will be backported everywhere.
So in addition to GCC 7.1 I'd like to have at least GCC 6.3 around,
which builds kernels without warnings today.
If you don't want warnings, turn off the warnings or just don't look at
them... or fix the problems? Many of the new warnings point out actual
problems.

Many of those sprintf problems in the kernel have already been fixed.
Post by Andre Przywara
For GCC 6.3 (and probably before) arc was breaking because missing a
(default) CPU type. Adding "--with-cpu=arc700" to EXTRA_GCC_CONF fixed
that, but GCC 7 indeed builds fine, even without it.
There were other problems, also in binutils.
Post by Andre Przywara
Also I removed documentation (share/ directory, which won't be in
MANPATH mostly anyway) from the tarball and stripped the (host) binaries
to get the tarball size down (to about 16 MB per arch)
I used to make tarballs for everything together, which is about 200MB
using lzma (which compresses this *much* better than e.g. bzip2). But
that is a while ago, the compiler grows in size really fast.
Post by Andre Przywara
I built both toolchains and kernels for almost all 31 supported
architectures. Some kernel builds fails (sparc, sparc64, arc), but not
sh4-linux-gcc: error: command line option '-m4-nofpu' is not supported
by this configuration.
Everything builds for me. I do have a few local patches, also one for
that SuperH problem, yeah. And my kernel trees are a few weeks old,
who knows what snuck in :-)


Segher
Arnd Bergmann
2017-05-10 07:58:13 UTC
Permalink
On Wed, May 10, 2017 at 12:18 AM, Segher Boessenkool
Post by Segher Boessenkool
Post by Andre Przywara
Post by Segher Boessenkool
Post by Andre Przywara
I was able to build (bare-metal) toolchains for
all architectures except arc, m68k, tilegx and tilepro.
arc needs a more recent GCC; the other probably as well. GCC 7 should
be out very soon, you probably want to wait for that :-)
Well, GCC 7 indeed builds better, but then again is a very new compiler.
For instance in the moment it spits a lot of warnings when compiling the
kernel (mostly due to some *printf analysis). It's not hard to fix, but
this will take a while to trickle in and it's questionable whether this
will be backported everywhere.
So in addition to GCC 7.1 I'd like to have at least GCC 6.3 around,
which builds kernels without warnings today.
If you don't want warnings, turn off the warnings or just don't look at
them... or fix the problems? Many of the new warnings point out actual
problems.
Many of those sprintf problems in the kernel have already been fixed.
I've been using gcc-7.0 for a long time and fixed a lot of bugs it found,
along with more harmless warnings, but I had disabled a couple of
warning options when I first installed gcc-7 and ended up ignoring
those.

The exact set of additional options I used is:

-Wimplicit-fallthrough=0 -Wno-duplicate-decl-specifier
-Wno-int-in-bool-context -Wno-bool-operation -Wno-format-truncation
-Wno-format-overflow

there were a couple of others that I sent kernel fixes for instead.
I should probably revisit that list and for each of them either
only enable it with "make W=1" or fix all known warnings.
In the long run, I'd actually hope to fix all W=1 warnings too
and enable them by default.

Arnd
Segher Boessenkool
2017-05-10 13:40:22 UTC
Permalink
Hi Arnd, long time no see,
Post by Arnd Bergmann
Post by Segher Boessenkool
Post by Andre Przywara
So in addition to GCC 7.1 I'd like to have at least GCC 6.3 around,
which builds kernels without warnings today.
If you don't want warnings, turn off the warnings or just don't look at
them... or fix the problems? Many of the new warnings point out actual
problems.
Many of those sprintf problems in the kernel have already been fixed.
I've been using gcc-7.0 for a long time and fixed a lot of bugs it found,
along with more harmless warnings, but I had disabled a couple of
warning options when I first installed gcc-7 and ended up ignoring
those.
-Wimplicit-fallthrough=0 -Wno-duplicate-decl-specifier
-Wno-int-in-bool-context -Wno-bool-operation -Wno-format-truncation
-Wno-format-overflow
there were a couple of others that I sent kernel fixes for instead.
I should probably revisit that list and for each of them either
only enable it with "make W=1" or fix all known warnings.
In the long run, I'd actually hope to fix all W=1 warnings too
and enable them by default.
Most of those usually point out actual problems (at least code that
isn't as clear as it should be). I do hate that first one though.


Segher
Arnd Bergmann
2017-05-10 19:32:36 UTC
Permalink
On Wed, May 10, 2017 at 3:40 PM, Segher Boessenkool
Post by Segher Boessenkool
Hi Arnd, long time no see,
Post by Arnd Bergmann
Post by Segher Boessenkool
Post by Andre Przywara
So in addition to GCC 7.1 I'd like to have at least GCC 6.3 around,
which builds kernels without warnings today.
If you don't want warnings, turn off the warnings or just don't look at
them... or fix the problems? Many of the new warnings point out actual
problems.
Many of those sprintf problems in the kernel have already been fixed.
I've been using gcc-7.0 for a long time and fixed a lot of bugs it found,
along with more harmless warnings, but I had disabled a couple of
warning options when I first installed gcc-7 and ended up ignoring
those.
-Wimplicit-fallthrough=0 -Wno-duplicate-decl-specifier
-Wno-int-in-bool-context -Wno-bool-operation -Wno-format-truncation
-Wno-format-overflow
there were a couple of others that I sent kernel fixes for instead.
I should probably revisit that list and for each of them either
only enable it with "make W=1" or fix all known warnings.
In the long run, I'd actually hope to fix all W=1 warnings too
and enable them by default.
Most of those usually point out actual problems (at least code that
isn't as clear as it should be). I do hate that first one though.
My point is that we have others in W=1 some of which are equally useful:
warning-1 := -Wextra -Wunused -Wno-unused-parameter
warning-1 += -Wmissing-declarations
warning-1 += -Wmissing-format-attribute
warning-1 += $(call cc-option, -Wmissing-prototypes)
warning-1 += -Wold-style-definition
warning-1 += $(call cc-option, -Wmissing-include-dirs)
warning-1 += $(call cc-option, -Wunused-but-set-variable)
warning-1 += $(call cc-option, -Wunused-const-variable)
warning-1 += $(call cc-disable-warning, missing-field-initializers)
warning-1 += $(call cc-disable-warning, sign-compare)

I've looked through arm and x86 gcc-7 allmodconfig builds (without
my longish fixup series) again and found these added warnings
compared to gcc-6.3.1 overall:

2 -Werror=bool-operation
4 -Werror=maybe-uninitialized
1 -Werror=parentheses
2 -Werror=stringop-overflow=
2 -Werror=tautological-compare

I probably submitted patches for those in the past, will have
another look to see if I need to resubmit them, or if some
of them might be regressions.

148 -Werror=duplicate-decl-specifier

Only a few files are affected, I can take care of fixing them all:

| 1 arch/arm/mach-at91/pm.c
| 1 arch/arm/mach-bcm/bcm_kona_smc.c
| 1 arch/arm/mach-cns3xxx/core.c
| 1 arch/arm/mach-omap2/prm_common.c
| 1 arch/arm/mach-omap2/vc.c
| 1 arch/arm/mach-spear/time.c
| 2 drivers/input/keyboard/cros_ec_keyb.c
| 4 sound/soc/codecs/rt5514.c
| 136 sound/soc/codecs/rt5665.c

89 -Werror=int-in-bool-context

This showed up in 26 files in allmodconfig alone. I had started on some
of them but given up at some point. I can certainly submit the ones
I did already, but probably won't have the patience to address all of them
myself.

54 -Werror=format-overflow=

Same here.

494 -Werror=format-truncation=

These are all over the place, in 187 files.

The last one in particular seems less useful than -Wformat-security
which we already disable (for all levels), and I'd rather have them both in
"make W=1". For -Wint-in-bool-context and -Wformat-overflow=
it's less obvious whether we should try to get them all fixed
quickly, using >100 patches or put them into W=1 along with
-Wformat-truncation.

Arnd
Chris Metcalf
2017-05-23 18:15:20 UTC
Permalink
Post by Andre Przywara
It seems that many people (even outside the Linux kernel community) use
the cross compilers provided at kernel.org/pub/tools/crosstool.
The latest compiler I find there is 4.9.0, which celebrated its third
birthday at the weekend, also has been superseded by 4.9.4 meanwhile.
So I took Segher's buildall scripts from [1] and threw binutils 2.28 and
GCC 6.3.0 at them.
I am belatedly catching up on this thread. It sounds like the
tilegx/tilepro issues were sorted out -- as someone noted, you need
to have the kernel headers available to build glibc. However, if
there are any outstanding tile issues, please feel free to loop me in!
--
Chris Metcalf, Mellanox Technologies
http://www.mellanox.com
Continue reading on narkive:
Loading...