FIX: steno multiple thread collision

[dougburks]

Originally posted by @jtgreen-cse in #7372

I'd like to use more than one thread with steno. My storage performs better if multiple processes are writing to it. I've added:

steno:
  threads: 8

to global.sls. Things work well for a while until eventually I get this in stenographer.log:

2022-03-01T17:15:55.954502Z T:442717 [stenotype.cc:567] Starting, page size is 4096
2022-03-01T17:15:55.954645Z T:442717 [stenotype.cc:594] Setting up AF_PACKET sockets for packet reading
panic: runtime error: slice bounds out of range [-822726196:]

goroutine 20 [running]:
github.com/golang/leveldb/table.block.seek.func1(0xc45bd06, 0xc000a46000)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:60 +0x20a
sort.Search(0x188b7a0d, 0xc00007d930, 0xc0001db530)
        /usr/lib/golang/src/sort/search.go:66 +0x58
github.com/golang/leveldb/table.block.seek(0xc000a46000, 0xd25ec, 0xd25f1, 0xbd6d20, 0x16fc060, 0xc0008c8012, 0x1, 0x1, 0xc00007da00, 0x40fb38, ...)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:59 +0x33a
github.com/golang/leveldb/table.(*tableIter).nextBlock(0xc0002be180, 0xc0008c8012, 0x1, 0x1, 0x0, 0xc0008c8012)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:197 +0x1d4
github.com/golang/leveldb/table.(*Reader).find(0xc000488820, 0xc0008c8012, 0x1, 0x1, 0x0, 0x0, 0x16fc060, 0xc00007db00)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:377 +0x1ea
github.com/golang/leveldb/table.(*Reader).Get(0xc000488820, 0xc0008c8012, 0x1, 0x1, 0x0, 0x1, 0x100000000000003, 0x0, 0x30, 0xc0004866f0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/golang/leveldb/table/reader.go:337 +0x8d
github.com/google/stenographer/indexfile.NewIndexFile(0xc0004866f0, 0x30, 0xc00021fb90, 0x3, 0xaff380, 0x3)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/indexfile/indexfile.go:67 +0x23d
github.com/google/stenographer/blockfile.NewBlockFile(0xc000486660, 0x30, 0xc00021fb90, 0xc000486660, 0x30, 0x1)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/blockfile/blockfile.go:63 +0x14b
github.com/google/stenographer/thread.(*Thread).trackNewFile(0xc000276aa0, 0xc0000c0020, 0x10, 0x10, 0x16fe000)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/thread/thread.go:172 +0xc9
github.com/google/stenographer/thread.(*Thread).syncFilesWithDisk(0xc000276aa0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/thread/thread.go:139 +0x2af
github.com/google/stenographer/thread.(*Thread).SyncFiles(0xc000276aa0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/thread/thread.go:363 +0x45
github.com/google/stenographer/env.(*Env).syncFiles(0xc0001761c0)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/env/env.go:267 +0x50
github.com/google/stenographer/env.(*Env).callEvery(0xc0001761c0, 0xc0002566a0, 0x37e11d600)
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/env/env.go:180 +0x83
created by github.com/google/stenographer/env.New
        /home/mreeves/rpmbuild/BUILD/stenographer-e8db1ee8f6b62790ebc8081e464dcedd7c961766/src/github.com/google/stenographer/env/env.go:131 +0x3f8

I'm going to guess that:

#5911 (review)

is still a thing? May I suggest salt create separate folders for each thread so that they don't clobber each others files. I didn't create an issue since steno threads probably aren't ready for prime time. Just letting you know that there's at least one person waiting for it :)

[jtgreen-cse]

Thanks Doug. I've actually implemented this locally but my nsm filled up because of the issue below (I am aware this is an unsupported config):

google/stenographer#214

I had patched steno's config and init.sls to make the multiple folders and use them. It worked fine until stenotype tried to cleanup at 10% free disk space on /nsm .. because it's working on one disk, the threads don't clean up the same as if each thread had a separate disk. The above issue describes this.

[jtgreen-cse]

Not sure how you're building the rpm referenced in so-steno's Dockerfile, but maybe you can add a patch from rskjetlein/stenographer@6f6d2cb to the source you use.

[jtgreen-cse]

Fork that includes what I did to get so-steno to write to multiple folders.

https://github.com/jtgreen-cse/securityonion/tree/jtgreen-cse-patch-1

[jtgreen-cse]

I've compiled stenotype and stenographer with the random thread picker commit as seen in the previous comments. I inserted the new binaries in one of my so-steno containers and manually applied the capabilities and correct perms. Everything now deletes correctly, no more filling up \nsm. I'm looking forward to seeing this is prod as I don't want to maintain this duct tape :)

[dougburks]

Thanks for the updates @jtgreen-cse !

We'll take a look as time allows.

[TOoSmOotH]

2.4 has the option to run multiple steno threads pointing to different locations.