Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bcc tools fail to run on arm64 #1202

Closed
maranp opened this issue May 25, 2017 · 18 comments
Closed

bcc tools fail to run on arm64 #1202

maranp opened this issue May 25, 2017 · 18 comments

Comments

@maranp
Copy link

maranp commented May 25, 2017

Find at the end, the error generated on trying to execute bcc scripts on arm64.
My configuration is Linux kernel 4.11.1 + clang 3.8. and upstream branch of ebpf.
I went through the discussion in this thread https://lists.iovisor.org/pipermail/iovisor-dev/2016-December/000557.html
and also added
cflags->push_back("-D__ASM_SYSREG_H");
to
src/cc/frontends/clang/kbuild_helper.c

However, could not find a way to fix this. Can bcc scripts be run on arm64?

./examples/hello_world.py
bpf: Invalid argument

Traceback (most recent call last):
File "./examples/hello_world.py", line 11, in
BPF(text='int kprobe__sys_clone(void *ctx) { bpf_trace_printk("Hello, World!\n"); return 0; }').trace_print()
File "/usr/lib/python2.7/dist-packages/bcc/init.py", line 301, in init
self.trace_autoload()
File "/usr/lib/python2.7/dist-packages/bcc/init.py", line 904, in trace_autoload
fn = self.load_func(func_name, BPF.KPROBE)
File "/usr/lib/python2.7/dist-packages/bcc/init.py", line 345, in load_func
(func_name, errstr))
Exception: Failed to load BPF program kprobe__sys_clone: Invalid argument
mpakkirisamy@crb9:~/ebpf/bcc$ sudo ./examples/tracing/hello_perf_output.py
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:9:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/list.h:8:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/kernel.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/printk.h:8:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/cache.h:5:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cache.h:19:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cachetype.h:19:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:112:9: warning: implicit declaration of
function 'read_sysreg_s' is invalid in C99 [-Wimplicit-function-declaration]
return read_cpuid(MIDR_EL1);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:103:27: note: expanded from macro
'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS
## reg)
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:112:9: error: use of undeclared identifier
'SYS_MIDR_EL1'
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:103:41: note: expanded from macro
'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS
## reg)
^
:55:1: note: expanded from here
SYS_MIDR_EL1
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:9:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/list.h:8:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/kernel.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/printk.h:8:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/cache.h:5:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cache.h:19:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cachetype.h:19:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:117:9: error: use of undeclared identifier
'SYS_MPIDR_EL1'
return read_cpuid(MPIDR_EL1);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:103:41: note: expanded from macro
'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
:55:1: note: expanded from here
SYS_MPIDR_EL1
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:9:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/list.h:8:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/kernel.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/printk.h:8:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/cache.h:5:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cache.h:19:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cachetype.h:19:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:132:9: error: use of undeclared identifier
'SYS_CTR_EL0'
return read_cpuid(CTR_EL0);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:103:41: note: expanded from macro
'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
:55:1: note: expanded from here
SYS_CTR_EL0
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rcupdate.h:38:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/spinlock.h:87:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/spinlock.h:21:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/processor.h:34:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/hw_breakpoint.h:20:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cpufeature.h:207:53: error: use of undeclared
identifier 'ID_AA64MMFR0_BIGENDEL_SHIFT'
return cpuid_feature_extract_unsigned_field(mmfr0, ID_AA64MMFR0_BIGENDEL_SHIFT) == 0x1 ||
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cpufeature.h:208:47: error: use of undeclared
identifier 'ID_AA64MMFR0_BIGENDEL0_SHIFT'
cpuid_feature_extract_unsigned_field(mmfr0, ID_AA64MMFR0_BIGENDEL0_SHIFT) == 0x1;
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cpufeature.h:213:55: error: use of undeclared
identifier 'ID_AA64PFR0_EL0_SHIFT'
u32 val = cpuid_feature_extract_unsigned_field(pfr0, ID_AA64PFR0_EL0_SHIFT);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cpufeature.h:215:16: error: use of undeclared
identifier 'ID_AA64PFR0_EL0_32BIT_64BIT'
return val == ID_AA64PFR0_EL0_32BIT_64BIT;
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cpufeature.h:233:39: error: use of undeclared
identifier 'SYS_ID_AA64MMFR0_EL1'
return id_aa64mmfr0_mixed_endian_el0(read_cpuid(ID_AA64MMFR0_EL1));
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cputype.h:103:41: note: expanded from macro
'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
:159:1: note: expanded from here
SYS_ID_AA64MMFR0_EL1
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rcupdate.h:38:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/spinlock.h:87:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/spinlock.h:21:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/processor.h:34:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/hw_breakpoint.h:20:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/cpufeature.h:243:55: error: use of undeclared
identifier 'SYS_ID_AA64MMFR0_EL1'
return id_aa64mmfr0_mixed_endian_el0(read_system_reg(SYS_ID_AA64MMFR0_EL1));
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rcupdate.h:38:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/spinlock.h:87:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/spinlock.h:21:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/processor.h:34:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/hw_breakpoint.h:22:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/virt.h:81:9: warning: implicit declaration of
function 'read_sysreg' is invalid in C99 [-Wimplicit-function-declaration]
return read_sysreg(CurrentEL) == CurrentEL_EL2;
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/virt.h:81:21: error: use of undeclared identifier
'CurrentEL'
return read_sysreg(CurrentEL) == CurrentEL_EL2;
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rcupdate.h:38:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/spinlock.h:87:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/spinlock.h:21:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/processor.h:34:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/hw_breakpoint.h:152:29: error: use of undeclared
identifier 'SYS_ID_AA64DFR0_EL1'
u64 dfr0 = read_system_reg(SYS_ID_AA64DFR0_EL1);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/hw_breakpoint.h:155:7: error: use of undeclared
identifier 'ID_AA64DFR0_BRPS_SHIFT'
ID_AA64DFR0_BRPS_SHIFT);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/hw_breakpoint.h:161:29: error: use of undeclared
identifier 'SYS_ID_AA64DFR0_EL1'
u64 dfr0 = read_system_reg(SYS_ID_AA64DFR0_EL1);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/hw_breakpoint.h:164:7: error: use of undeclared
identifier 'ID_AA64DFR0_WRPS_SHIFT'
ID_AA64DFR0_WRPS_SHIFT);
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/sched.h:13:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/pid.h:4:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/rcupdate.h:46:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/ktime.h:25:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/jiffies.h:8:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/include/linux/timex.h:65:
In file included from /lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/timex.h:19:
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/arch_timer.h:71:4: warning: implicit declaration of
function 'write_sysreg' is invalid in C99 [-Wimplicit-function-declaration]
write_sysreg(val, cntp_ctl_el0);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/arch_timer.h:71:22: error: use of undeclared
identifier 'cntp_ctl_el0'
write_sysreg(val, cntp_ctl_el0);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/arch_timer.h:74:22: error: use of undeclared
identifier 'cntp_tval_el0'
write_sysreg(val, cntp_tval_el0);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/arch_timer.h:80:22: error: use of undeclared
identifier 'cntv_ctl_el0'
write_sysreg(val, cntv_ctl_el0);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/arch_timer.h:83:22: error: use of undeclared
identifier 'cntv_tval_el0'
write_sysreg(val, cntv_tval_el0);
^
/lib/modules/4.11.1-4478c-bpfsamp/build/arch/arm64/include/asm/arch_timer.h:97:23: error: use of undeclared
identifier 'cntp_ctl_el0'
return read_sysreg(cntp_ctl_el0);
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
3 warnings and 20 errors generated.
Traceback (most recent call last):
File "./examples/tracing/hello_perf_output.py", line 34, in
b = BPF(text=prog)
File "/usr/lib/python2.7/dist-packages/bcc/init.py", line 294, in init
raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

@joelagnel
Copy link
Contributor

Hi Maran, So I hit the same thing, although when just building the samples from the kernel sources: https://lkml.org/lkml/2017/8/5/143 . Ping me if you're interested in working together on this problem ([email protected]) as well

@yonghong-song
Copy link
Collaborator

Hi, @maranp @joelagnel , I have just added inline asm support in llvm trunk. I checked the offending asm codes in linux:arch/arm64/include/asm/sysreg.h. It contains a bunch of directives which the newly added bpf inline asm functionality should support, although it ultimately ignores them since there is no use.

Could you build llvm from latest source and to see whether the issue still exists in bcc and linux:samples/bpf? Please let me know if you still have issues and I would like to see whether the issue can be resolved from llvm/clang part.

@joelagnel
Copy link
Contributor

I can try this soon, thanks!

@abhijitayarekar
Copy link

Hi @yonghong-song ,
I tried the latest llvm build and saw similar errors.
Following are llc and clang versions
LLVM (http://llvm.org/):
LLVM version 6.0.0svn
Optimized build.
Default target: x86_64-unknown-linux-gnu
Host CPU: ivybridge

Registered Targets:
bpf - BPF (host endian)
bpfeb - BPF (big endian)
bpfel - BPF (little endian)

clang version 6.0.0 (http://llvm.org/git/clang.git 29487927c0f5d8cd6b23978a0216b17041161cc5) (http://llvm.org/git/llvm.git df6ca162269f9d756f8742bf4b658dcf690e3eb5)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/abhijit/xdp/llvm/build/bin

Following is excerpt from the build failure.
/home/abhijit/xdp/llvm/build/bin/clang -nostdinc -isystem /usr/lib/gcc-cross/aarch64-linux-gnu/5/include -I./arch/arm64/include -I./arch/arm64/include/generated -I./include -I./arch/arm64/include/uapi -I./arch/arm64/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -Isamples/bpf
-I./tools/testing/selftests/bpf/
-D__KERNEL__ -Wno-unused-value -Wno-pointer-sign
-D__TARGET_ARCH_arm64 -Wno-compare-distinct-pointer-types
-Wno-gnu-variable-sized-type-not-at-end
-Wno-address-of-packed-member -Wno-tautological-compare
-Wno-unknown-warning-option
-O2 -emit-llvm -c samples/bpf/tracex5_kern.c -o -| /home/abhijit/xdp/llvm/build/bin/llc -march=bpf -filetype=obj -o samples/bpf/tracex5_kern.o
In file included from samples/bpf/tracex5_kern.c:7:
In file included from ./include/linux/ptrace.h:5:
In file included from ./include/linux/sched.h:13:
In file included from ./include/linux/pid.h:4:
In file included from ./include/linux/rculist.h:10:
In file included from ./include/linux/rcupdate.h:40:
In file included from ./include/linux/preempt.h:80:
In file included from ./arch/arm64/include/generated/asm/preempt.h:1:
In file included from ./include/asm-generic/preempt.h:4:
In file included from ./include/linux/thread_info.h:37:
In file included from ./arch/arm64/include/asm/thread_info.h:31:
./arch/arm64/include/asm/stack_pointer.h:7:51: error: register 'sp' unsuitable for global register variables on this target
register unsigned long current_stack_pointer asm ("sp");
^
In file included from samples/bpf/tracex5_kern.c:7:
In file included from ./include/linux/ptrace.h:5:
In file included from ./include/linux/sched.h:14:
In file included from ./include/linux/sem.h:8:
In file included from ./include/uapi/linux/sem.h:4:
In file included from ./include/linux/ipc.h:4:
In file included from ./include/linux/spinlock.h:87:
./arch/arm64/include/asm/spinlock.h:67:28: error: value '65536' out of range for constraint 'I'
: "Q" (lock->owner), "I" (1 << TICKET_SHIFT)
^~~~~~~~~~~~~~~~~
./arch/arm64/include/asm/spinlock.h:96:9: error: value '65536' out of range for constraint 'I'
: "I" (1 << TICKET_SHIFT)

@yonghong-song
Copy link
Collaborator

Thanks for trying. Now I see. The issue is not just asm codes outside the function body. The function with assembly code inside is used somehow. Need to check this instance more.

@yonghong-song
Copy link
Collaborator

HI, @abhijitayarekar, I do not have arm64 build environment. Could you generate a preprocessed file (-E) and attached here? This will make it easy for me to investigate. Thanks!

@yonghong-song
Copy link
Collaborator

Looked at this error a little more closely. Still not sure about the second error.
./arch/arm64/include/asm/spinlock.h:67:28: error: value '65536' out of range for constraint 'I'
Looks like the inline function is used somewhere in the code ...

For the first error, the fundamental error is due to we are actually compiling for x86.
/home/abhijit/xdp/llvm/build/bin/clang -nostdinc -isystem /usr/lib/gcc-cross/aarch64-linux-gnu/5/include -I./arch/arm64/include -I./arch/arm64/include/generated -I./include -I./arch/arm64/include/uapi -I./arch/arm64/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/kconfig.h -Isamples/bpf
-I./tools/testing/selftests/bpf/
-D__KERNEL__ -Wno-unused-value -Wno-pointer-sign
-D__TARGET_ARCH_arm64 -Wno-compare-distinct-pointer-types
-Wno-gnu-variable-sized-type-not-at-end
-Wno-address-of-packed-member -Wno-tautological-compare
-Wno-unknown-warning-option
-O2 -emit-llvm -c samples/bpf/tracex5_kern.c -o -

From the above command line options, there is no indication that this is compiling for bpf.
We cannot infer target just based on header include path!
So the error is correct since "sp" register name is not supported for x64 architecture.

However, adding "-target bpf" may not help either since "sp" is not a legal register name in bpf as well.

@abhijitayarekar
Copy link

Hi @yonghong-song,
Please see attached clang -E output
clang-e.txt

@yonghong-song
Copy link
Collaborator

@abhijitayarekar I can reproduce your failure now. As mentioned in my previous comments, the error message you saw is because the clang/llc compiler is on x86_64 host and default target
is x86_64 and it won't recognize the arm inline assembly syntax.

You can use the following way to resolve your issue.
(1). Add "-target aarch64" to clang compilation flag.
The following patch illustrates the change. Obviously it is a hack.
[[email protected] ~/work/net-next/samples/bpf] git diff Makefile
diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile
index ebc2ad6..3452384 100644
--- a/samples/bpf/Makefile
+++ b/samples/bpf/Makefile
@@ -233,5 +233,5 @@ $(obj)/%.o: $(src)/%.c
-D__TARGET_ARCH_$(ARCH) -Wno-compare-distinct-pointer-types
-Wno-gnu-variable-sized-type-not-at-end
-Wno-address-of-packed-member -Wno-tautological-compare \

  •           -Wno-unknown-warning-option \
    
  •           -Wno-unknown-warning-option -target aarch64 \
              -O2 -emit-llvm -c $< -o -| $(LLC) -march=bpf -filetype=obj -o $@
    

[[email protected] ~/work/net-next/samples/bpf]

clang compiler is able to successfully compile the program since arm64 headers
will be correctly recognized by the aarch64 target.

(2). later on, "llc" command with bpf target will succeed as well since the arm64 inline
assembly has been removed as they are not used anywhere.

Please give a try and let me know whether this mechanism works for you or not.
If possible, please try all programs in samples/bpf directory.

@abhijitayarekar
Copy link

abhijitayarekar commented Oct 3, 2017 via email

@yonghong-song
Copy link
Collaborator

Could you describe what did you do for "Although i still have to disable assembly definitions in arch/arm64/include/asm/sysreg.h"? I am curious why you need to do this since "-target aarch64" should take care of this?

@joelagnel
Copy link
Contributor

joelagnel commented Oct 4, 2017 via email

@abhijitayarekar
Copy link

abhijitayarekar commented Oct 4, 2017 via email

@maranp
Copy link
Author

maranp commented Oct 26, 2017

Thanks for looking into the issue. I couldn't get back to it sooner.

I still face the same issue.

My kernel has this patch
https://lkml.org/lkml/2017/10/12/878

And I use upstream llvm (6.0) and made sure this patch is included
llvm-mirror/llvm@2865ab6

Both the linux kernel and llvm+clang are built natively on arm64 board.
I did not face any issues in building the kernel or bcc.

However, I face the same issue as I posted in the initial report.
I had to add
cflags->push_back("-D__ASM_SYSREG_H");
to
src/cc/frontends/clang/kbuild_helper.c without which I end up with this error
LLVM ERROR: Inline asm not supported by this streamer because we don't have an asm parser for this target

I am pasting one compilation command line, if it helps to identify issues with the way the build is going on.

cd /home/mpakkirisamy/ebpf/bcc/build/src/cc/frontends/clang && /usr/bin/c++ -I/usr/local/include/../tools/clang/include -I/home/mpakkirisamy/ebpf/bcc/src -I/home/mpakkirisamy/ebpf/bcc/build/src -I/home/mpakkirisamy/ebpf/bcc/build/src/cc -I/home/mpakkirisamy/ebpf/bcc/src/cc -I/home/mpakkirisamy/ebpf/bcc/build/src/cc/frontends/b -I/home/mpakkirisamy/ebpf/bcc/src/cc/frontends/b -I/home/mpakkirisamy/ebpf/bcc/src/cc/frontends/clang -I/usr/local/include -I/home/mpakkirisamy/ebpf/bcc/src/cc/compat -Wall -isystem /usr/local/include -fno-rtti -fPIC -DBCC_PROG_TAG_DIR='"/var/tmp/bcc"' -DLLVM_MAJOR_VERSION=6 -DKERNEL_MODULES_DIR='"/lib/modules"' -O3 -DNDEBUG -D_GNU_SOURCE -D_DEBUG -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -std=gnu++11 -o CMakeFiles/clang_frontend.dir/kbuild_helper.cc.o -c /home/mpakkirisamy/ebpf/bcc/src/cc/frontends/clang/kbuild_helper.cc

Am I missing something trivial?

Here is the error log when attempting to execute a bcc tool
/usr/share/bcc/tools$ sudo ./cpudist
In file included from /virtual/main.c:1:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/uapi/linux/ptrace.h:100:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/ptrace.h:76:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/bug.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/bug.h:66:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/asm-generic/bug.h:15:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/kernel.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/printk.h:8:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/cache.h:5:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cache.h:19:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:116:9: warning: implicit declaration of function 'read_sysreg_s' is invalid in C99 [-Wimplicit-function-declaration]
return read_cpuid(MIDR_EL1);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:27: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:116:9: error: use of undeclared identifier 'SYS_MIDR_EL1'
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:41: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
:64:1: note: expanded from here
SYS_MIDR_EL1
^
In file included from /virtual/main.c:1:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/uapi/linux/ptrace.h:100:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/ptrace.h:76:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/bug.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/bug.h:66:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/asm-generic/bug.h:15:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/kernel.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/printk.h:8:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/cache.h:5:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cache.h:19:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:121:9: warning: implicit declaration of function 'read_sysreg_s' is invalid in C99 [-Wimplicit-function-declaration]
return read_cpuid(MPIDR_EL1);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:27: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:121:9: error: use of undeclared identifier 'SYS_MPIDR_EL1'
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:41: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
:65:1: note: expanded from here
SYS_MPIDR_EL1
^
In file included from /virtual/main.c:1:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/uapi/linux/ptrace.h:100:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/ptrace.h:76:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/bug.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/bug.h:66:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/asm-generic/bug.h:15:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/kernel.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/printk.h:8:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/cache.h:5:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cache.h:19:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:136:9: warning: implicit declaration of function 'read_sysreg_s' is invalid in C99 [-Wimplicit-function-declaration]
return read_cpuid(CTR_EL0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:27: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:136:9: error: use of undeclared identifier 'SYS_CTR_EL0'
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:41: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
:66:1: note: expanded from here
SYS_CTR_EL0
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/sched.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/pid.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rcupdate.h:43:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/processor.h:34:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:20:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cpufeature.h:215:53: error: use of undeclared identifier 'ID_AA64MMFR0_BIGENDEL_SHIFT'
return cpuid_feature_extract_unsigned_field(mmfr0, ID_AA64MMFR0_BIGENDEL_SHIFT) == 0x1 ||
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cpufeature.h:216:47: error: use of undeclared identifier 'ID_AA64MMFR0_BIGENDEL0_SHIFT'
cpuid_feature_extract_unsigned_field(mmfr0, ID_AA64MMFR0_BIGENDEL0_SHIFT) == 0x1;
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cpufeature.h:221:55: error: use of undeclared identifier 'ID_AA64PFR0_EL0_SHIFT'
u32 val = cpuid_feature_extract_unsigned_field(pfr0, ID_AA64PFR0_EL0_SHIFT);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cpufeature.h:223:16: error: use of undeclared identifier 'ID_AA64PFR0_EL0_32BIT_64BIT'
return val == ID_AA64PFR0_EL0_32BIT_64BIT;
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cpufeature.h:241:39: warning: implicit declaration of function 'read_sysreg_s' is invalid in C99 [-Wimplicit-function-declaration]
return id_aa64mmfr0_mixed_endian_el0(read_cpuid(ID_AA64MMFR0_EL1));
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:27: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/sched.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/pid.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rcupdate.h:43:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/processor.h:34:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:20:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cpufeature.h:241:39: error: use of undeclared identifier 'SYS_ID_AA64MMFR0_EL1'
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cputype.h:107:41: note: expanded from macro 'read_cpuid'
#define read_cpuid(reg) read_sysreg_s(SYS_ ## reg)
^
:130:1: note: expanded from here
SYS_ID_AA64MMFR0_EL1
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/sched.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/pid.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rcupdate.h:43:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/processor.h:34:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:20:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/cpufeature.h:251:62: error: use of undeclared identifier 'SYS_ID_AA64MMFR0_EL1'
return id_aa64mmfr0_mixed_endian_el0(read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1));
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/sched.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/pid.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rcupdate.h:43:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/processor.h:34:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:22:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/virt.h:94:9: warning: implicit declaration of function 'read_sysreg' is invalid in C99 [-Wimplicit-function-declaration]
return read_sysreg(CurrentEL) == CurrentEL_EL2;
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/virt.h:94:21: error: use of undeclared identifier 'CurrentEL'
return read_sysreg(CurrentEL) == CurrentEL_EL2;
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/sched.h:13:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/pid.h:4:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rculist.h:10:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/rcupdate.h:43:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/processor.h:34:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:152:36: error: use of undeclared identifier 'SYS_ID_AA64DFR0_EL1'
u64 dfr0 = read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:155:7: error: use of undeclared identifier 'ID_AA64DFR0_BRPS_SHIFT'
ID_AA64DFR0_BRPS_SHIFT);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:161:36: error: use of undeclared identifier 'SYS_ID_AA64DFR0_EL1'
u64 dfr0 = read_sanitised_ftr_reg(SYS_ID_AA64DFR0_EL1);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/hw_breakpoint.h:164:7: error: use of undeclared identifier 'ID_AA64DFR0_WRPS_SHIFT'
ID_AA64DFR0_WRPS_SHIFT);
^
In file included from /virtual/main.c:2:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/sched.h:19:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/hrtimer.h:19:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/ktime.h:25:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/jiffies.h:9:
In file included from /lib/modules/4.13.0-bcc-1+/build/include/linux/timex.h:65:
In file included from /lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/timex.h:19:
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:92:4: warning: implicit declaration of function 'write_sysreg' is invalid in C99 [-Wimplicit-function-declaration]
write_sysreg(val, cntp_ctl_el0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:92:22: error: use of undeclared identifier 'cntp_ctl_el0'
write_sysreg(val, cntp_ctl_el0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:95:22: error: use of undeclared identifier 'cntp_tval_el0'
write_sysreg(val, cntp_tval_el0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:101:4: warning: implicit declaration of function 'write_sysreg' is invalid in C99 [-Wimplicit-function-declaration]
write_sysreg(val, cntv_ctl_el0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:101:22: error: use of undeclared identifier 'cntv_ctl_el0'
write_sysreg(val, cntv_ctl_el0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:104:22: error: use of undeclared identifier 'cntv_tval_el0'
write_sysreg(val, cntv_tval_el0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:118:11: warning: implicit declaration of function 'read_sysreg' is invalid in C99 [-Wimplicit-function-declaration]
return read_sysreg(cntp_ctl_el0);
^
/lib/modules/4.13.0-bcc-1+/build/arch/arm64/include/asm/arch_timer.h:118:23: error: use of undeclared identifier 'cntp_ctl_el0'
return read_sysreg(cntp_ctl_el0);
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
8 warnings and 20 errors generated.
Traceback (most recent call last):
File "./cpudist", line 155, in
b = BPF(text=bpf_text)
File "/usr/lib/python2.7/dist-packages/bcc/init.py", line 296, in init
raise Exception("Failed to compile BPF module %s" % src_file)
Exception: Failed to compile BPF module

@yonghong-song
Copy link
Collaborator

I just fixed an issue related to the following error message:

LLVM ERROR: Inline asm not supported by this streamer because we don't have an asm parser for this target

Could you pull the latest bcc code and give a try?

@maranp
Copy link
Author

maranp commented Oct 31, 2017

With the latest bcc code, I could successfully execute the bcc scripts on aarch64 machine. Thanks for fixing this. I am closing the issue.

@maranp maranp closed this as completed Oct 31, 2017
@wuxiuchao
Copy link

@yonghong-song ,

I assumed that the errors related with sysreg was fixed by the commit 7abe63a.

Just wonder how "bpfasmparser" in the following line solved this issue.
"list(APPEND llvm_raw_libs bpfasmparser)"

Is it transformed to some flags used by clang? Or is the parser called by bcc to do some preprocessing?

I'm now facing the same problem, and look forward to the internals since we are compiling eBPF code without bcc.

Thanks in advance.

@retpolanne
Copy link

@yonghong-song is bpfasmparser passed as a flag to clang? I'm trying to port this tool to ARM (kinvolk/traceloop#28) and we call clang from a Makefile.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants