Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wazuh service does not run together with Caddy #4321

Open
3 tasks done
ArjenR opened this issue Oct 23, 2024 · 5 comments
Open
3 tasks done

Wazuh service does not run together with Caddy #4321

ArjenR opened this issue Oct 23, 2024 · 5 comments

Comments

@ArjenR
Copy link

ArjenR commented Oct 23, 2024

Important notices
Before you add a new report, we ask you kindly to acknowledge the following:

Describe the bug
Wazuh agent will not start completely due to wazuh-modulesd quitting after start with an "Illegal Instruction"
Clean install of version 1.0 and later version 1.1 of the plugin

To Reproduce
Steps to reproduce the behavior:

  1. Install Caddy plugin
  2. Configure and start Caddy service with a simple reverse proxy.
  3. Install Wazuh plugin
  4. Configure IP address of wazuh server.
  5. Service fails to start.
  6. Stop Caddy service
  7. Start Wazuh service
  8. Wazuh runs

Expected behavior
Wazuh service should run.

Screenshots
*** With caddy running ***

# /var/ossec/bin/wazuh-modulesd -fdd
2024/10/23 15:28:25 wazuh-modulesd[13670] debug_op.c:305 at os_logging_config(): DEBUG: (1228): Element 'log_format' without any option.
2024/10/23 15:28:25 wazuh-modulesd[13670] debug_op.c:116 at _log_function(): DEBUG: Logging module auto-initialized
2024/10/23 15:28:25 wazuh-modulesd[13670] main.c:77 at main(): DEBUG: Wazuh home directory: /var/ossec
2024/10/23 15:28:25 wazuh-modulesd[13670] main.c:87 at main(): INFO: Started (pid: 13670).
2024/10/23 15:28:25 wazuh-modulesd[13670] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:28:25 wazuh-modulesd[13670] main.c:95 at main(): DEBUG: Created new thread for the 'agent-upgrade' module.
2024/10/23 15:28:25 wazuh-modulesd[13670] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:28:25 wazuh-modulesd:agent-upgrade[13670] wm_agent_upgrade_agent.c:96 at wm_agent_upgrade_start_agent_module(): INFO: (8153): Module Agent Upgrade started.
2024/10/23 15:28:25 wazuh-modulesd[13670] main.c:95 at main(): DEBUG: Created new thread for the 'syscollector' module.
2024/10/23 15:28:25 wazuh-modulesd[13670] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:28:25 wazuh-modulesd[13670] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:28:25 wazuh-modulesd[13670] mq_op.c:52 at StartMQWithSpecificOwnerAndPerms(): DEBUG: Connected succesfully to 'queue/sockets/queue' after 0 attempts
2024/10/23 15:28:25 wazuh-modulesd[13670] main.c:95 at main(): DEBUG: Created new thread for the 'control' module.
2024/10/23 15:28:25 wazuh-modulesd[13670] mq_op.c:53 at StartMQWithSpecificOwnerAndPerms(): DEBUG: (unix_domain) Maximum send buffer set to: '65792'.
2024/10/23 15:28:25 wazuh-modulesd[13670] mq_op.c:52 at StartMQWithSpecificOwnerAndPerms(): DEBUG: Connected succesfully to 'queue/sockets/queue' after 0 attempts
2024/10/23 15:28:25 wazuh-modulesd[13670] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:28:25 wazuh-modulesd:control[13670] wm_control.c:132 at wm_control_main(): INFO: Starting control thread.
2024/10/23 15:28:25 wazuh-modulesd[13670] mq_op.c:53 at StartMQWithSpecificOwnerAndPerms(): DEBUG: (unix_domain) Maximum send buffer set to: '65792'.
2024/10/23 15:28:25 wazuh-modulesd[13670] wmcom.c:136 at wmcom_main(): DEBUG: Local requests thread ready
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] wm_syscollector.c:158 at wm_sys_main(): DEBUG: Starting Syscollector.
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] wm_syscollector.c:100 at wm_sys_log_config(): DEBUG: {"syscollector":{"disabled":"no","scan-on-start":"yes","interval":3600,"network":"yes","os":"yes","hardware":"yes","packages":"yes","ports":"yes","ports_all":"no","processes":"yes","sync_max_eps":10}}
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:31 at taggedLogFunction(): INFO: Module started.
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:31 at taggedLogFunction(): INFO: Starting evaluation.
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting hardware scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending hardware scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting os scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending os scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting network scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending network scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting packages scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending packages scan
2024/10/23 15:28:25 wazuh-modulesd:syscollector[13670] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting ports scan
Illegal instruction

*** With caddy stopped ***

# /var/ossec/bin/wazuh-modulesd -fdd
2024/10/23 15:36:14 wazuh-modulesd[62105] debug_op.c:305 at os_logging_config(): DEBUG: (1228): Element 'log_format' without any option.
2024/10/23 15:36:14 wazuh-modulesd[62105] debug_op.c:116 at _log_function(): DEBUG: Logging module auto-initialized
2024/10/23 15:36:14 wazuh-modulesd[62105] main.c:77 at main(): DEBUG: Wazuh home directory: /var/ossec
2024/10/23 15:36:14 wazuh-modulesd[62105] main.c:87 at main(): INFO: Started (pid: 62105).
2024/10/23 15:36:14 wazuh-modulesd[62105] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:36:14 wazuh-modulesd[62105] main.c:95 at main(): DEBUG: Created new thread for the 'agent-upgrade' module.
2024/10/23 15:36:14 wazuh-modulesd[62105] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:36:14 wazuh-modulesd:agent-upgrade[62105] wm_agent_upgrade_agent.c:96 at wm_agent_upgrade_start_agent_module(): INFO: (8153): Module Agent Upgrade started.
2024/10/23 15:36:14 wazuh-modulesd[62105] main.c:95 at main(): DEBUG: Created new thread for the 'syscollector' module.
2024/10/23 15:36:14 wazuh-modulesd[62105] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:36:14 wazuh-modulesd[62105] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:36:14 wazuh-modulesd[62105] main.c:95 at main(): DEBUG: Created new thread for the 'control' module.
2024/10/23 15:36:14 wazuh-modulesd[62105] mq_op.c:52 at StartMQWithSpecificOwnerAndPerms(): DEBUG: Connected succesfully to 'queue/sockets/queue' after 0 attempts
2024/10/23 15:36:14 wazuh-modulesd[62105] mq_op.c:52 at StartMQWithSpecificOwnerAndPerms(): DEBUG: Connected succesfully to 'queue/sockets/queue' after 0 attempts
2024/10/23 15:36:14 wazuh-modulesd:control[62105] wm_control.c:132 at wm_control_main(): INFO: Starting control thread.
2024/10/23 15:36:14 wazuh-modulesd[62105] mq_op.c:53 at StartMQWithSpecificOwnerAndPerms(): DEBUG: (unix_domain) Maximum send buffer set to: '65792'.
2024/10/23 15:36:14 wazuh-modulesd[62105] pthreads_op.c:45 at CreateThreadJoinable(): DEBUG: Thread stack size set to: 8192 KiB
2024/10/23 15:36:14 wazuh-modulesd[62105] mq_op.c:53 at StartMQWithSpecificOwnerAndPerms(): DEBUG: (unix_domain) Maximum send buffer set to: '65792'.
2024/10/23 15:36:14 wazuh-modulesd:syscollector[62105] wm_syscollector.c:158 at wm_sys_main(): DEBUG: Starting Syscollector.
2024/10/23 15:36:14 wazuh-modulesd[62105] wmcom.c:136 at wmcom_main(): DEBUG: Local requests thread ready
2024/10/23 15:36:14 wazuh-modulesd:syscollector[62105] wm_syscollector.c:100 at wm_sys_log_config(): DEBUG: {"syscollector":{"disabled":"no","scan-on-start":"yes","interval":3600,"network":"yes","os":"yes","hardware":"yes","packages":"yes","ports":"yes","ports_all":"no","processes":"yes","sync_max_eps":10}}
2024/10/23 15:36:14 wazuh-modulesd:syscollector[62105] logging_helper.c:31 at taggedLogFunction(): INFO: Module started.
2024/10/23 15:36:14 wazuh-modulesd:syscollector[62105] logging_helper.c:31 at taggedLogFunction(): INFO: Starting evaluation.
2024/10/23 15:36:14 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting hardware scan
2024/10/23 15:36:14 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending hardware scan
2024/10/23 15:36:14 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting os scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending os scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting network scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending network scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting packages scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending packages scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting ports scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending ports scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Starting processes scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:40 at taggedLogFunction(): DEBUG: Ending processes scan
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:31 at taggedLogFunction(): INFO: Evaluation finished.
2024/10/23 15:36:15 wazuh-modulesd:syscollector[62105] logging_helper.c:37 at taggedLogFunction(): DEBUG: Starting syscollector sync
2024/10/23 15:36:15 wazuh-modulesd[62105] wait_op.c:117 at os_wait_primitive(): DEBUG: Process locked due to agent is offline. Waiting for connection..

Relevant log files
dmesg reports:
pid 13670 (wazuh-modulesd), jid 0, uid 0: exited on signal 4 (no core dump - bad address)

Additional context
Add any other context about the problem here.

Environment
OPNsense 24.7.7-amd64
os-caddy (installed) 1.7.3
os-wazuh-agent (installed) 1.1
@Monviech
Copy link
Member

My first hunch is overlapping ports.

Do sockstat -l | grep -i waz to find out what wazuh uses. If its either of 80 or 443 then it can not run at the same time as caddy without changing either its ports, or the wazuh ports.

@fichtner
Copy link
Member

Looks more like it fails scanning the port and crashes itself by mere fact of being suddenly connected to caddy...

@ArjenR
Copy link
Author

ArjenR commented Oct 24, 2024

My first hunch is overlapping ports.

The plugin is a Wazuh agent. Not the server which does indeed require ports 80 and 443 amongst others.

@Monviech
Copy link
Member

Installing the wazuh agent the first time reveals some more issues under the hood:

Template does not render:

[meta sequenceId="4"] error generating template OPNsense/WazuhAgent : Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/template.py", line 262, in _generate     content = j2_page.render(cnf_data)               ^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 1304, in render     self.environment.handle_exception()   File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 939, in handle_exception     raise rewrite_traceback_stack(source=source)   File "/usr/local/opnsense/service/templates/OPNsense/WazuhAgent/ossec.conf", line 4, in top-level template code     <address>{{OPNsense.WazuhAgent.general.server_address}}</address>     ^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/lib/python3.11/site-packages/jinja2/environment.py", line 487, in getattr     return getattr(obj, attribute)            ^^^^^^^^^^^^^^^^^^^^^^^ jinja2.exceptions.UndefinedError: 'collections.OrderedDict object' has no attribute 'WazuhAgent'  During handling of the above exception, another exception occurred:  Traceback (most recent call last):   File "/usr/local/opnsense/service/modules/template.py", line 328, in generate     for filename in self._generate(template_name, create_directory):                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^   File "/usr/local/opnsense/service/modules/template.py", line 265, in _generate     raise Exception("%s %s %s" % (module_name, template_filename, render_exception)) Exception: OPNsense/WazuhAgent OPNsense/WazuhAgent/ossec.conf 'collections.OrderedDict object' has no attribute 'WazuhAgent'

And some validation thats too strict on first install so the migration fails:

<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="3"] [OPNsense\WazuhAgent\WazuhAgent:general.server_address] A value is required.{}
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="4"] [OPNsense\WazuhAgent\WazuhAgent:logcollector.syslog_programs] Specify valid source applications.{filterlog,openvpn,unbound,audit,sshd}
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="5"] Model OPNsense\WazuhAgent\WazuhAgent can't be saved, skip ( OPNsense\Base\ValidationException: [OPNsense\WazuhAgent\WazuhAgent:general.server_address] A value is required.{}
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="6"] [OPNsense\WazuhAgent\WazuhAgent:logcollector.syslog_programs] Specify valid source applications.{filterlog,openvpn,unbound,audit,sshd}
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="7"]  in /usr/local/opnsense/mvc/app/models/OPNsense/Base/BaseModel.php:666
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="8"] Stack trace:
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="9"] #0 /usr/local/opnsense/mvc/app/models/OPNsense/Base/BaseModel.php(791): OPNsense\Base\BaseModel->serializeToConfig()
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="10"] #1 /usr/local/opnsense/mvc/script/run_migrations.php(54): OPNsense\Base\BaseModel->runMigrations()
<147>1 2024-10-24T16:00:36+00:00 opn-ce-01.ad.pischem.com config 18188 - [meta sequenceId="11"] #2 {main} )

Probably unrelated but I just wanted to add it here. Seems like a small cleanup of that plugin could be needed sometime.

@Monviech
Copy link
Member

I won't be able to troubleshoot this in combination with caddy since I do not have a wazuh collection endpoint and the service will not start when I do not specify a valid one. So I can't be of much use. If it is indeed related to caddy though, please give me another ping so I can ask around.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@fichtner @ArjenR @Monviech