-
Notifications
You must be signed in to change notification settings - Fork 48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Added log processing #75
base: master
Are you sure you want to change the base?
Changes from 3 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,7 +9,11 @@ import qualified Codec.Archive.Tar as Tar | |
import qualified Codec.Compression.GZip as GZip | ||
import Control.Monad (filterM, liftM, unless) | ||
import Control.Monad.IO.Class | ||
import Data.Bits | ||
import qualified Data.ByteString.Lazy as BS | ||
import qualified Data.ByteString as BSS | ||
import Data.Conduit (ConduitT, yield) | ||
import qualified Data.Conduit.Binary as CB | ||
import Data.Monoid ((<>)) | ||
import qualified Data.Text as T | ||
import qualified Data.Text.IO as TIO | ||
|
@@ -129,3 +133,16 @@ exclusionCheck f ps = any id (map (\(ExclusionPattern p) -> f ~~ T.unpack p) ps) | |
inclusionCheck :: FilePath -> [InclusionPattern] -> Bool | ||
inclusionCheck f ps = any id (map (\(InclusionPattern p) -> f ~~ T.unpack p) ps) | ||
|
||
|
||
processLog :: Monad m => ConduitT BSS.ByteString BSS.ByteString m () | ||
processLog = do | ||
-- metadata (is the next string stdout or stderr) | ||
_ <- CB.take 4 | ||
len' <- CB.take 4 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why do we only take 4 bytes at a time? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You mean, why don't we read the first 8 bytes in one go? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure reading it in one go and partitioning into two parts would be terribly more efficient. |
||
let len = BS.foldl (\i w -> shiftL i 8 .|. fromIntegral w) 0 len' | ||
case len of | ||
0 -> return () | ||
n -> do | ||
chunk <- CB.take n | ||
yield . BS.toStrict $ chunk | ||
processLog |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do I understand correctly that the first 4 bytes in the STREAM just denote that the data marked as collected from stdout/stderr?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As I explained in the issue, I couldn't find any documentation regarding the format. All I could find was this repo: https://github.com/mafintosh/docker-raw-stream
In that repo, the second 4 bytes represent the length of the subsequent string, and the first bit denotes whether the string is from stdout or stderr.
I assume that the first 4 bytes are reserved for metadata.
I have had the opportunity to test this branch and it works as expected.