ossec-dbd not reconnecting after database failure

[ChristianBeer]

I was babysitting a new server and ran into memory trouble twice. During this I noticed an odd behaviour of ossec-dbd. The mysql server threads where killed at 03:57:53 (out of memory) but in the same second restarted. Ossec looses connectivity for only this query but can't reconnect after this. I have to manually restart ossec in order to reconnect ossec-dbd back to the database.

Can someone else please confirm this behaviour? Do I have to change something on the mysql server side?

2014/11/22 03:57:58 ossec-dbd(5203): ERROR: Error executing query 'SELECT id FROM location WHERE name = 'hopper->/var/log/messages' AND server_id = '1' LIMIT 1'. Error: 'MySQL server has gone away'.
2014/11/22 03:57:58 ossec-dbd(5209): INFO: Closing connection to database.
2014/11/22 03:57:58 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2014/11/22 03:57:58 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server host 'localhost' (2).
2014/11/22 03:58:00 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2014/11/22 03:58:00 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server host 'localhost' (2).
2014/11/22 03:58:04 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2014/11/22 03:58:04 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server host 'localhost' (2).
2014/11/22 03:58:12 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2014/11/22 03:58:12 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server host 'localhost' (2).
2014/11/22 03:58:28 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2014/11/22 03:58:28 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server host 'localhost' (2).
2014/11/22 03:59:00 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2014/11/22 03:59:00 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server host 'localhost' (2).
2014/11/22 04:00:04 ossec-dbd(5210): INFO: Attempting to reconnect to database.
2014/11/22 04:00:04 ossec-dbd(5202): ERROR: Error connecting to database 'localhost'(ossec): ERROR: Unknown MySQL server host 'localhost' (2).

[jrossi]

@atomicturtle Have you seen anything like this you do a fair amount of DB and OSSEC :)

[atomicturtle]

Yes, there is an internal_options setting to increase the attempts:

dbd.reconnect_attempts=10

The error itself is correct, its attempting a TCP connection (it does not use the socket because of the chroot) over 3306 and mysql is not responding. It could happen if mysqld wasnt running, exceeded max connections, firewall rule was blocking the port, etc

[drknite72]

I Just had the same issue and in my case the data table was corrupt
logging in to MySQL and running
OPTIMIZE TABLE alert;
OPTIMIZE TABLE data;

show my data table to be corrupt
used

REPAIR TABLE data;
to repair it and restarted ossec and everything worked again.

hope this helps you, or points you in the right direction.

[ChristianBeer]

As far as I can tell I had no data corruption. I wondered why ossec couldn't reconnect to the DB even after it was back up. I always had to restart ossec manually to get it reconnected.

[atomicturtle]

It probably exceeded dbd.reconnect_attempts=10