thread 'main' panicked at 'Couldn't find intel_rapl modules.'

[frntn]

Bug description

Unable to start scaphandre

To Reproduce

docker run -v /sys/class/powercap:/sys/class/powercap -v /proc:/proc -e RUST_BACKTRACE=full -ti hubblo/scaphandre stdout -t 15

Expected behavior

The container starts and write data on stdout

Screenshots

thread 'main' panicked at 'Couldn't find intel_rapl modules.', src/sensors/powercap_rapl.rs:72:13
stack backtrace:
   0:     0x5604d03f429a - std::backtrace_rs::backtrace::libunwind::trace::h72c2fb8038f1bbee
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/../../backtrace/src/backtrace/libunwind.rs:96
   1:     0x5604d03f429a - std::backtrace_rs::backtrace::trace_unsynchronized::h1e3b084883f1e78c
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/../../backtrace/src/backtrace/mod.rs:66
   2:     0x5604d03f429a - std::sys_common::backtrace::_print_fmt::h3bf6a7ebf7f0394a
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:79
   3:     0x5604d03f429a - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h2e8cb764b7fe02e7
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:58
   4:     0x5604d02559cc - core::fmt::write::h7a1184eaee6a8644
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/fmt/mod.rs:1080
   5:     0x5604d03f3951 - std::io::Write::write_fmt::haeeb374d93a67eac
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/io/mod.rs:1516
   6:     0x5604d03f324b - std::sys_common::backtrace::_print::h1d14a7f6ad632dc8
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:61
   7:     0x5604d03f324b - std::sys_common::backtrace::print::h301abac8bb2e3e81
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:48
   8:     0x5604d03f324b - std::panicking::default_hook::{{closure}}::hde0cb80358a6920a
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:208
   9:     0x5604d03f324b - std::panicking::default_hook::h9b1a691049a0ec8f
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:227
  10:     0x5604d03f324b - std::panicking::rust_panic_with_hook::h2bdec87b60580584
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:577
  11:     0x5604d03d7a73 - std::panicking::begin_panic::{{closure}}::h1c5ca4054bfcced4
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:506
  12:     0x5604d03d7a4c - std::sys_common::backtrace::__rust_end_short_backtrace::hdf2e24f90f64c859
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:153
  13:     0x5604d038ef1a - std::panicking::begin_panic::h911c93aa12195ab2
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/panicking.rs:505
  14:     0x5604d03cb8c2 - <scaphandre::sensors::powercap_rapl::PowercapRAPLSensor as scaphandre::sensors::Sensor>::generate_topology::hc322b9f765cb7099
                               at /app/src/sensors/powercap_rapl.rs:72
  15:     0x5604d03cc2a7 - <scaphandre::sensors::powercap_rapl::PowercapRAPLSensor as scaphandre::sensors::Sensor>::get_topology::h7ceb1ed47b7fd14f
                               at /app/src/sensors/powercap_rapl.rs:112
  16:     0x5604d018ff42 - scaphandre::exporters::stdout::StdoutExporter::new::he1ab57086641bd51
                               at /app/src/exporters/stdout.rs:51
  17:     0x5604d018ff42 - scaphandre::run::hec5d090badbd7be0
                               at /app/src/lib.rs:60
  18:     0x5604d018ff42 - scaphandre::main::h9d3de27ff1861588
                               at /app/src/main.rs:90
  19:     0x5604d03f0ac3 - core::ops::function::FnOnce::call_once::h72eaaae983a9915d
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/core/src/ops/function.rs:227
  20:     0x5604d03f0ac3 - std::sys_common::backtrace::__rust_begin_short_backtrace::h8b70d5c3e456bdd5
                               at /rustc/7eac88abb2e57e752f3302f02be5f3ce3d7adfb4/library/std/src/sys_common/backtrace.rs:137
  21:     0x5604d0190f05 - main
  22:     0x7f34e5688cb2 - __libc_start_main
  23:     0x5604d018b09a - _start
  24:                0x0 - <unknown>

Environment

• Linux distribution version

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.1 LTS
Release:        20.04
Codename:       focal

• Kernel version

$ uname -r
5.4.0-62-generic

Additional context

Proc reference : https://ark.intel.com/content/www/fr/fr/ark/products/87038/intel-xeon-processor-d-1520-6m-cache-2-20-ghz.html

[bpetit]

Hi,

Thanks for reporting. What is the ouput of lsmod | grep intel_rapl ?

[frntn]

Hello,

No output for this command

$ lsmod | grep intel_rapl
$

If I try to load the module I get an error

$ modprobe intel_rapl
modprobe: FATAL: Module intel_rapl not found in directory /lib/modules/5.4.0-62-generic

But some related files do exist

$ locate rapl
/usr/src/linux-headers-5.4.0-62/include/linux/intel_rapl.h
/usr/src/linux-headers-5.4.0-62-generic/include/config/intel/rapl
/usr/src/linux-headers-5.4.0-62-generic/include/config/intel/rapl.h
/usr/src/linux-headers-5.4.0-62-generic/include/config/intel/rapl/core.h
/usr/src/linux-headers-5.4.0-62-generic/include/config/perf/events/intel/rapl.h
/usr/src/linux-headers-5.4.0-62-generic/include/config/proc/thermal/mmio/rapl.h

[bpetit]

Could you try a modprobe intel_rapl_common and modprobe rapl ?

[frntn]

$ modprobe rapl
modprobe: FATAL: Module rapl not found in directory /lib/modules/5.4.0-62-generic

$ modprobe intel_rapl_common
modprobe: FATAL: Module intel_rapl_common not found in directory /lib/modules/5.4.0-62-generic

Additionally :

$ grep -rni rapl /lib/modules/$(uname -r)/
/lib/modules/5.4.0-62-generic/modules.order:2:kernel/arch/x86/events/rapl.ko
/lib/modules/5.4.0-62-generic/modules.order:3782:kernel/drivers/powercap/intel_rapl_common.ko
/lib/modules/5.4.0-62-generic/modules.order:3783:kernel/drivers/powercap/intel_rapl_msr.ko

$ find /lib/modules/$(uname -r) -type f -name '*.ko' | grep rapl
$

"Is this real life ?" 😅

[bpetit]

I'm on ubuntu 20.04.1 too but I guess there is something different in your setup as I dont have the modules.order:XX-like folders you have:

find /lib/modules/$(uname -r)/ -name *rapl*
/lib/modules/5.8.0-38-generic/kernel/arch/x86/events/rapl.ko
/lib/modules/5.8.0-38-generic/kernel/drivers/powercap/intel_rapl_common.ko
/lib/modules/5.8.0-38-generic/kernel/drivers/powercap/intel_rapl_msr.ko

I don't know where it could come from 🤔

I'm honestly on try and fail mode here, what happens if you manually insmod one of those .ko files available ?

# insmod /lib/modules/5.4.0-62-generic/modules.order:3782:kernel/drivers/powercap/intel_rapl_common.ko

Certainly not a good solution, but I try to understand what's happening and it may give us some lead.

Another question: are you using a kernel you compiled yourself or is it the default one that is managed by your distribution ?

[bpetit]

@uggla just made me realize you may have to install the linux-modules-extra-$(uname-r) package !

I'm very curious about how it is going 🤓 ?

[frntn]

Spotted ! 🥳

$ apt install linux-modules-extra-$(uname -r)

$ modprobe intel_rapl_common 

$ docker run -v /sys/class/powercap:/sys/class/powercap -v /proc:/proc -e RUST_BACKTRACE=full -ti hubblo/scaphandre stdout -t 15
Measurement step is: 2s
Host:   0 W     Core            Uncore          DRAM
Socket0 ? W     0 W
Top 5 consumers:
Power   PID     Exe
------------------------------------------------------------

Host:   11.838703 W     Core            Uncore          DRAM
Socket0 11.88223 W      8.319255 W
Top 5 consumers:
Power   PID     Exe
5.7872815 W     377006  ""
1.218375 W      848     ""
0.30459374 W    558     ""
0.30459374 W    905     ""
0 W     1       ""
------------------------------------------------------------

Host:   13.552203 W     Core            Uncore          DRAM
Socket0 13.544806 W     10.927634 W
Top 5 consumers:
Power   PID     Exe
9.757587 W      377006  ""
0.7744116 W     1891    ""
0.61952925 W    1620    ""
0.15488231 W    1       ""
0.15488231 W    11      ""
------------------------------------------------------------

Thank you @uggla & @bpetit 💪

[bpetit]

Great !

It's still strange that you have "traces" of the kernel modules, if you never installed the package before 🤔

If someone has a clear understanding on why it may happen I'm interested ! :)

[DarylSaucier]

Hi everyone,

I had the same problem, but mine goes further 😞
I'm working on a Raspberry Pi 3b+ on which I installed the latest version of RaspiOS (former Raspbian). When I try to install the extra packages using apt, this is what I get :
"Impossible to find the package linux-module-extra-5.10.17-v8". I looked up for answers in several forums but no one seems to provide a way for the installer to find these packages. I'm not very used to install manually Linux packages, so I wonder if anyone has an idea to do so, maybe by adding a repository in the etc/apt/source.plist file ?

Thanks :)

[uggla]

Hi @DarylSaucier, unless if i'm wrong because I have currently no Rasberry Pi to verify. But there is not "rapl" sensor for the Rasberry Pi. The rapl technology is provided only on x86 (Intel or Amd) hardware. So scaphandre cannot run on a rpi.
Looking at the package, I think that it does not make sens to have a module-extra package on the rpi because as the hardware is the same, there are no need to provide various hardware configuration options/extensions.

[DarylSaucier]

Thank you very much @uggla, we'll thus find another machine