When logging dmesg output, let time be readable

[berland]

When the forward model runner detects an out-of-memory kill, it logs some lines from dmesg which might be interesting to look at.

These are printed with kernel times which are relative to boot (?) and not very useful when logged.

Try using dmest -T to make this useful.

[berland]

In an example dmesg trace, the following was found:

  1 [57539.949044] Out of memory: Killed process 328391 (fluvial_facies) total-vm:167572220kB, anon-rss:163272252kB, file-rss:0kB, shmem-rss:0kB, UID:322617 pgtables:327568kB oom_score_adj:0
  2 [625328.156297] Out of memory: Killed process 1597132 (eclipse.exe) total-vm:8069004kB, anon-rss:7239092kB, file-rss:0kB, shmem-rss:0kB, UID:30521 pgtables:15480kB oom_score_adj:0
  3 [625351.710167] Out of memory: Killed process 1595933 (rms) total-vm:14655464kB, anon-rss:8908940kB, file-rss:0kB, shmem-rss:0kB, UID:368287 pgtables:19908kB oom_score_adj:0
  4 [625378.557848] Out of memory: Killed process 1596142 (rms) total-vm:15583780kB, anon-rss:9804376kB, file-rss:0kB, shmem-rss:0kB, UID:368287 pgtables:21656kB oom_score_adj:0
  5 [625412.489837] Out of memory: Killed process 1596008 (rms) total-vm:15702684kB, anon-rss:9901112kB, file-rss:0kB, shmem-rss:0kB, UID:368287 pgtables:21788kB oom_score_adj:0
  6 [626949.779329] Out of memory: Killed process 1596188 (rms) total-vm:18780540kB, anon-rss:13037440kB, file-rss:0kB, shmem-rss:0kB, UID:368287 pgtables:27984kB oom_score_adj:0
  7 [627287.170137] Out of memory: Killed process 1595868 (rms) total-vm:21948292kB, anon-rss:16095384kB, file-rss:0kB, shmem-rss:0kB, UID:368287 pgtables:34148kB oom_score_adj:0
  8 [627464.720212] Out of memory: Killed process 1596185 (rms) total-vm:24510156kB, anon-rss:18633648kB, file-rss:0kB, shmem-rss:0kB, UID:368287 pgtables:39144kB oom_score_adj:0
  9 [965090.271790] Out of memory: Killed process 4039164 (rms) total-vm:17557976kB, anon-rss:10074960kB, file-rss:0kB, shmem-rss:0kB, UID:22347 pgtables:22588kB oom_score_adj:0
 10 [965699.964882] Out of memory: Killed process 4039270 (rms) total-vm:18716860kB, anon-rss:11232916kB, file-rss:0kB, shmem-rss:0kB, UID:22347 pgtables:24876kB oom_score_adj:0
 11 [966701.950283] Out of memory: Killed process 4039412 (rms) total-vm:20317476kB, anon-rss:12804304kB, file-rss:0kB, shmem-rss:0kB, UID:22347 pgtables:28000kB oom_score_adj:0
 12 [968684.050556] Out of memory: Killed process 4039389 (rms) total-vm:25184236kB, anon-rss:17148872kB, file-rss:0kB, shmem-rss:0kB, UID:22347 pgtables:36532kB oom_score_adj:0
 13 [1166174.591596] Out of memory: Killed process 1243815 (res2csv) total-vm:22056024kB, anon-rss:21181208kB, file-rss:0kB, shmem-rss:0kB, UID:10984 pgtables:42628kB oom_score_adj:0
 14 [1253075.238778] Out of memory: Killed process 1856051 (res2csv) total-vm:86803776kB, anon-rss:85507788kB, file-rss:0kB, shmem-rss:0kB, UID:236801 pgtables:169316kB oom_score_adj:0
 15 [1256914.897521] Out of memory: Killed process 1882935 (res2csv) total-vm:89559408kB, anon-rss:87969804kB, file-rss:4kB, shmem-rss:0kB, UID:236801 pgtables:174700kB oom_score_adj:0
 16 [1303665.065444] Out of memory: Killed process 2213351 (res2csv) total-vm:61655428kB, anon-rss:60481840kB, file-rss:0kB, shmem-rss:0kB, UID:230011 pgtables:120096kB oom_score_adj:0

after manual formatting. Maybe only the last line is relevant, and the other ones are historic and should maybe not be logged. Filter on time?