Question: Help with Local Configuration

[norsemangrey]

I am trying to set up ipmi_exporter locally (without running in Docker) on an Ubuntu Server machine to collect metrics by a Prometheus instance on the same host. However, I am struggling to to get ipmi_exporter to scrape the metrics I want. I have probably missed something, but I cannot seem to figure out what. I am hoping someone can look this over and maybe point out what I need to do differently.

I have installed FreeIPMI and can run the various tools using sudo.
I have set up ipmi_exporter to run as a service with its own system user/group "exporters" and the following config:

[Unit]
Description=Prometheus IPMI Exporter
Documentation=https://github.com/prometheus-community/ipmi_exporter

[Service]
Type=simple
User=exporters
Group=exporters
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/ipmi_exporter \
    --config.file=/usr/local/bin/ipmi-exporter-config.yml \
    --web.listen-address=:9200

[Install]
WantedBy=multi-user.target

Since the FreeIPMI tool are run as sudo I have added the following to the sudoers file to allow the exporters user to run the tools without password.

exporters ALL = NOPASSWD: /usr/sbin/ipmimonitoring,\
                          /usr/sbin/ipmi-sensors,\
                          /usr/sbin/ipmi-dcmi,\
                          /usr/sbin/ipmi-raw,\
                          /usr/sbin/bmc-info,\
                          /usr/sbin/ipmi-chassis,\
                          /usr/sbin/ipmi-sel

I added it after this section in the sudoers file:

# User privilege specification
root    ALL=(ALL:ALL) ALL

The ipmi_exporter configuration file looks like this:

modules:
        default:
                # Available collectors are bmc, ipmi, chassis, dcmi, sel, and sm-lan-mode
                collectors:
                - bmc
                - ipmi
                - dcmi
                - chassis
                - sel
                # Got any sensors you don't care about? Add them here.
                exclude_sensor_ids:
        advanced:
                collectors:
                - bmc
                - ipmi
                - dcmi
                - chassis
                - sel
                collector_cmd:
                        bmc: sudo
                        ipmi: sudo
                        dcmi: sudo
                        chassis: sudo
                        sel: sudo
                custom_args:
                        bmc:
                        - "bmc-info"
                        ipmi:
                        - "ipmimonitoring"
                        dcmi:
                        - "ipmi-dcmi"
                        chassis:
                        - "ipmi-chassis"
                        sel:
                        - "ipmi-sel"

After starting the node_exporter service and use curl to check the endpoint (curl http://localhost:9200/metrics) I get a response, but it does not seem to contain any actual scraped metrics.

As you can see from the service status below I am getting a permissioned denied error for all the collectors.

● ipmi_exporter.service - Prometheus IPMI Exporter
     Loaded: loaded (/etc/systemd/system/ipmi_exporter.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-11-01 20:22:55 CET; 39s ago
       Docs: https://github.com/prometheus-community/ipmi_exporter
   Main PID: 1534642 (ipmi_exporter)
      Tasks: 12 (limit: 38232)
     Memory: 2.3M
     CGroup: /system.slice/ipmi_exporter.service
             └─1534642 /usr/local/bin/ipmi_exporter --config.file=/usr/local/bin/ipmi-exporter-config.yml --web.listen-address=:9200

 systemd[1]: Started Prometheus IPMI Exporter.
ipmi_exporter[1534642]: time="2021-11-01T20:22:55+01:00" level=info msg="Starting ipmi_exporter" source="main.go:83"
ipmi_exporter[1534642]: time="2021-11-01T20:22:55+01:00" level=info msg="Loaded config file /usr/local/bin/ipmi-exporter-config.yml" source="config.go:234"
ipmi_exporter[1534642]: time="2021-11-01T20:22:55+01:00" level=info msg="Listening on :9200" source="main.go:147"
ipmi_exporter[1534642]: time="2021-11-01T20:23:25+01:00" level=error msg="Failed to collect BMC data from [local]: error running bmc-info: exit status 1: bmc-info: permission denied\n" source="collector_bmc.go:40"
ipmi_exporter[1534642]: time="2021-11-01T20:23:25+01:00" level=error msg="Failed to collect sensor data from [local]: error running ipmimonitoring: exit status 1: /usr/sbin/ipmi-sensors: permission denied\n" source="colle>
ipmi_exporter[1534642]: time="2021-11-01T20:23:25+01:00" level=error msg="Failed to collect DCMI data from [local]: error running ipmi-dcmi: exit status 1: ipmi-dcmi: permission denied\n" source="collector_dcmi.go:40"
ipmi_exporter[1534642]: time="2021-11-01T20:23:25+01:00" level=error msg="Failed to collect chassis data from [local]: error running ipmi-chassis: exit status 1: ipmi-chassis: permission denied\n" source="collector_chassi>
ipmi_exporter[1534642]: time="2021-11-01T20:23:25+01:00" level=error msg="Failed to collect SEL data from [local]: error running ipmi-sel: exit status 1: ipmi-sel: permission denied\n" source="collector_sel.go:47"

[aminvakil]

I also have this problem on debian 10 buster.

[bitfehler]

Hi, sorry for the delay, just getting back into things here. Since you are using the custom_args for the process execution I think you may have to use the full path to the executable there (as that is what you have in the sudo config):

                custom_args:
                        bmc:
                        - "/usr/sbin/bmc-info"
                        ipmi:
                        - "/usr/sbin/ipmimonitoring"
                        dcmi:
                        - "/usr/sbin/ipmi-dcmi"
                        chassis:
                        - "/usr/sbin/ipmi-chassis"
                        sel:
                        - "/usr/sbin/ipmi-sel"

Can you let me know if that helps?

[yunzhihui]

Hi, sorry for the delay, just getting back into things here. Since you are using the custom_args for the process execution I think you may have to use the full path to the executable there (as that is what you have in the sudo config):

                custom_args:
                        bmc:
                        - "/usr/sbin/bmc-info"
                        ipmi:
                        - "/usr/sbin/ipmimonitoring"
                        dcmi:
                        - "/usr/sbin/ipmi-dcmi"
                        chassis:
                        - "/usr/sbin/ipmi-chassis"
                        sel:
                        - "/usr/sbin/ipmi-sel"

Can you let me know if that helps?

NO， This is no help for me

[yunzhihui]

buster

can you slove it?

[bitfehler]

@yunzhihui please provide more information. What does your exporter config look like? Are you also trying to use sudo? If so, please post your sudo config, too.

[Heimspiel1]

Hi,

Is there already a solution? I have the same problem on Ubuntu 18.04.5 LTS. I tried the ipmi_exporter version 1.61.1 and 1.5.1. I get the same error on both version.

If not, here is my configuration:

sudo config:

prometheus      ALL = NOPASSWD: /usr/sbin/ipmimonitoring, \
                                /usr/sbin/ipmi-sensors, \
                                /usr/sbin/ipmi-dcmi, \
                                /usr/sbin/ipmi-raw, \
                                /usr/sbin/bmc-info, \
                                /usr/sbin/ipmi-chassis, \ 
                                /usr/sbin/ipmi-sel

the sudo config works fine. I can execute all commands successfully from the command line.

For test purposes, the exporter config is very small:

modules:
        default:
                # Available collectors are bmc, ipmi, chassis, dcmi, sel, and sm-lan-mode
                collectors:
                - ipmi
        advanced:
                collectors:
                - ipmi
                collector_cmd:
                  ipmi: sudo
                custom_args:
                  ipmi:
                  - "/usr/sbin/ipmimonitoring"

Finally i turned on the debug level and start the service

 ./ipmi_exporter --log.level=debug --config.file=ipmi.conf
ts=2022-09-27T06:38:37.592Z caller=main.go:107 level=info msg="Starting ipmi_exporter" version="(version=1.6.1, branch=HEAD, revision=344b8b4a565a9ced936aad4d4ac9a29892515cba)"
ts=2022-09-27T06:38:37.592Z caller=config.go:243 level=info msg="Loaded config file" path=ipmi.conf
ts=2022-09-27T06:38:37.593Z caller=main.go:172 level=info msg="Listening on" address=:9290
ts=2022-09-27T06:38:37.593Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
ts=2022-09-27T06:38:45.902Z caller=collector.go:100 level=debug msg="Running collector" target= collector=ipmi
ts=2022-09-27T06:38:45.902Z caller=freeipmi.go:143 level=debug msg=Executing command=ipmimonitoring args="[-Q --ignore-unrecognized-events --comma-separated-output --no-header-output --sdr-cache-recreate --output-event-bitmask --output-sensor-state --config-file /tmp/ipmi_exporter-3cb3e2fd8b5555c654d24ce66083d719]"
ts=2022-09-27T06:38:45.904Z caller=collector_ipmi.go:151 level=error msg="Failed to collect sensor data" target=[local] error="error running ipmimonitoring: exit status 1: /usr/sbin/ipmi-sensors: permission denied\n"
ts=2022-09-27T06:38:45.904Z caller=collector.go:84 level=debug msg="Scrape duration" target=[local] duration=0.001767177

I got the permission denied error.

The curl command looks like: curl http://127.0.0.1:9290/metrics

[bitfehler]

Hey all. Thank you @Heimspiel1 for providing the debug output, it finally gave me the right direction. As you can see in the debug output, the command is not actually being executed with sudo at all. I now also realized why. You are defining multiple modules in your config, but you all seem to be using local mode. Local mode will always use the default module. Having multiple modules only makes sense for remote mode. If you all just get rid of the default module in your config and rename the advanced one to default then it should work fine.

[bitfehler]

Ok, I also see now where you all get the name advanced from 😅 I guess this needs to be made more clear in the examples.

[Heimspiel1]

@bitfehler thank you for the quick response. I changed the config file as you suggested and now it works.

Here is the working config:

modules:
        default:
               collectors:
               - ipmi
               collector_cmd:
                 ipmi: sudo
               custom_args:
                 ipmi:
                 - "ipmimonitoring"

and sudo is in the debug log:

./ipmi_exporter --log.level=debug --config.file=ipmi.conf
ts=2022-09-27T15:30:09.705Z caller=main.go:107 level=info msg="Starting ipmi_exporter" version="(version=1.6.1, branch=HEAD, revision=344b8b4a565a9ced936aad4d4ac9a29892515cba)"
ts=2022-09-27T15:30:09.705Z caller=config.go:243 level=info msg="Loaded config file" path=ipmi.conf
ts=2022-09-27T15:30:09.705Z caller=main.go:172 level=info msg="Listening on" address=:9290
ts=2022-09-27T15:30:09.705Z caller=tls_config.go:195 level=info msg="TLS is disabled." http2=false
ts=2022-09-27T15:30:24.205Z caller=collector.go:100 level=debug msg="Running collector" target= collector=ipmi
ts=2022-09-27T15:30:24.205Z caller=freeipmi.go:143 level=debug msg=Executing command=sudo args="[ipmimonitoring -Q --ignore-unrecognized-events --comma-separated-output --no-header-output --sdr-cache-recreate --output-event-bitmask --output-sensor-state --config-file /tmp/ipmi_exporter-92c0d83882ff913115a54cca6ae68cdb]"
....