Server crash after the implementation of flashcache

[kurtihnet]

Hi Guys,

We have implemented flashcache in our many servers. But some of the servers intermittenly crashed due to fc. We are using 250GB SSD for flashcache in most of the servers. We have partitioned the SSD drive for 2 servers(125GB each). Those are OpenVZ VE's mounted on LVM. We have not got any errors from logs. The only seeing is pasted below.

kernel: [ 10.913549] device-mapper: flashcache: Allocate 1311596KB (22B per) mem for 61048832-entry cache(capacity:238472MB, associativity:512, block size:8 sectors(4KB))

Aug 17 22:25:04 kernel: [ 14.070413] device-mapper: flashcache: Allocate 1311596KB (22B per) mem for 61048832-entry cache(capacity:238472MB, associativity:512, block size:8 sectors(4KB))

The above logs seems to be related with flashcache allocation.

Is there any way to findout the reason?

Note : Those servers were stable during the past time. But after flashcache implementation, server is going down everyday. Server is running with latest OpenVZ kernel "2.6.32-042stab108.8" with flashcache module loaded into it.

[root@~]# lsmod | grep flashcache
flashcache 97759 2

dm_mod 95654 13 flashcache,dm_mirror,dm_log

Any help from your end is much appreciated.

[mohans]

Hi
I need the crash dump from the console from at least one of the servers to be able to tell what is going on here.
I don't have any experience running OpenVZ with flashcache at all. So I don't know if there are any bugs that might be triggered by that workload. And I am not able to use OpenVZ to try and reproduce at my end.
mohan
From: kurtihnet [email protected]
To: facebook/flashcache [email protected]
Sent: Saturday, August 22, 2015 12:45 AM
Subject: [flashcache] Server crash after the implementation of flashcache (#208)

Hi Guys,We have implemented flashcache in our many servers. But some of the servers intermittenly crashed due to fc. We are using 250GB SSD for flashcache in most of the servers. We have partitioned the SSD drive for 2 servers(125GB each). Those are OpenVZ VE's mounted on LVM. We have not got any errors from logs. The only seeing is pasted below.

kernel: [ 10.913549] device-mapper: flashcache: Allocate 1311596KB (22B per) mem for 61048832-entry cache(capacity:238472MB, associativity:512, block size:8 sectors(4KB))
Aug 17 22:25:04 kernel: [ 14.070413] device-mapper: flashcache: Allocate 1311596KB (22B per) mem for 61048832-entry cache(capacity:238472MB, associativity:512, block size:8 sectors(4KB))
The above logs seems to be related with flashcache allocation. Is there any way to findout the reason?Note : Those servers were stable during the past time. But after flashcache implementation, server is going down everyday. Server is running with latest OpenVZ kernel "2.6.32-042stab108.8" with flashcache module loaded into it.

[root@~]# lsmod | grep flashcache
flashcache 97759 2
dm_mod 95654 13 flashcache,dm_mirror,dm_log
Any help from your end is much appreciated.—
Reply to this email directly or view it on GitHub.

[kurtihnet]

Hi Mohan,

The latest crash screen is attached. PFA.

Some of our other host nodes are not having issues with flashcache. But few nodes were crashed periodically.

[mohans]

Hi
I don't see any flashcache functions in the stacktrace - also can't tell from this snippet where the node might have crashed. It is possible that the useful sections of the stacktrace just scrolled off. 
We really need the entire dump (incl stacktrace) to debug this.
mohan
From: kurtihnet [email protected]
To: facebook/flashcache [email protected]
Cc: Mohan Srinivasan [email protected]
Sent: Saturday, August 22, 2015 10:34 PM
Subject: Re: [flashcache] Server crash after the implementation of flashcache (#208)

Hi Mohan,The latest crash screen is attached. PFA.Some of our other host nodes are not having issues with flashcache. But few nodes were crashed periodically.

—
Reply to this email directly or view it on GitHub.

[kurtihnet]

Hi Mohan,

I will provide the full stackstrace soon.

[kurtihnet]

Hi Mohan ,

We have setup a test server with OpenVZ kernel and also it is having the same configurations of the server having issue.
Could you please let us know your Email address so that we can share you with the test server login details for you to test if there thers is any bug while using fc with OpenVZ kernel.

[mohans]

mohan_srinivasan at yahoo dot com
What I really need is the entire stack dump from the crash (from the console). 
From: kurtihnet [email protected]
To: facebook/flashcache [email protected]
Cc: Mohan Srinivasan [email protected]
Sent: Wednesday, September 2, 2015 12:18 AM
Subject: Re: [flashcache] Server crash after the implementation of flashcache (#208)

Hi Mohan , We have setup a test server with OpenVZ kernel and also it is having the same configurations of the server having issue.
Could you please let us know your Email address so that we can share you with the test server login details for you to test if there thers is any bug while using fc with OpenVZ kernel.—
Reply to this email directly or view it on GitHub.