Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added log processing #75

Open
wants to merge 4 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions src/Docker/Client/Api.hs
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ import Control.Monad.Reader (ask, lift)
import Data.Aeson (FromJSON, eitherDecode')
import qualified Data.ByteString as BS
import qualified Data.ByteString.Lazy as BSL
import Data.Conduit (Sink)
import Data.Conduit (Sink, (.|))
import qualified Data.Conduit.Binary as Conduit
import qualified Data.Text as T
import qualified Data.Text as Text
Expand Down Expand Up @@ -172,7 +172,7 @@ inspectContainer cid = requestHelper GET (InspectContainerEndpoint cid) >>= pars
-- __NOTE__: It's recommended to use one of the other 'LogDriverType's available (like
-- syslog) for creating your containers.
getContainerLogs :: forall m. (MonadIO m, MonadMask m) => LogOpts -> ContainerID -> DockerT m (Either DockerError BSL.ByteString)
getContainerLogs logopts cid = requestHelper GET (ContainerLogsEndpoint logopts False cid)
getContainerLogs logopts cid = getContainerLogsStream logopts cid Conduit.sinkLbs

{-| Continuously gets the container's logs as a stream. Uses conduit.

Expand All @@ -189,7 +189,8 @@ __Example__:

-}
getContainerLogsStream :: forall m b . (MonadIO m, MonadMask m) => LogOpts -> ContainerID -> Sink BS.ByteString m b -> DockerT m (Either DockerError b)
getContainerLogsStream logopts cid = requestHelper' GET (ContainerLogsEndpoint logopts True cid)
getContainerLogsStream logopts cid sink =
requestHelper' GET (ContainerLogsEndpoint logopts True cid) (processLog .| sink)
-- JP: Should the second (follow) argument be True? XXX

-- | Build an Image from a Dockerfile
Expand Down Expand Up @@ -224,4 +225,3 @@ createNetwork opts = requestHelper POST (CreateNetworkEndpoint opts) >>= parseR
-- | Removes a network
removeNetwork :: forall m. (MonadIO m, MonadMask m) => NetworkID -> DockerT m (Either DockerError ())
removeNetwork nid = requestUnit DELETE $ RemoveNetworkEndpoint nid

1 change: 0 additions & 1 deletion src/Docker/Client/Internal.hs
Original file line number Diff line number Diff line change
Expand Up @@ -105,4 +105,3 @@ getEndpointRequestBody (RemoveNetworkEndpoint _) = Nothing
getEndpointContentType :: Endpoint -> BSC.ByteString
getEndpointContentType (BuildImageEndpoint _ _) = BSC.pack "application/tar"
getEndpointContentType _ = BSC.pack "application/json; charset=utf-8"

17 changes: 17 additions & 0 deletions src/Docker/Client/Utils.hs
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,11 @@ import qualified Codec.Archive.Tar as Tar
import qualified Codec.Compression.GZip as GZip
import Control.Monad (filterM, liftM, unless)
import Control.Monad.IO.Class
import Data.Bits
import qualified Data.ByteString.Lazy as BS
import qualified Data.ByteString as BSS
import Data.Conduit (ConduitT, yield)
import qualified Data.Conduit.Binary as CB
import Data.Monoid ((<>))
import qualified Data.Text as T
import qualified Data.Text.IO as TIO
Expand Down Expand Up @@ -129,3 +133,16 @@ exclusionCheck f ps = any id (map (\(ExclusionPattern p) -> f ~~ T.unpack p) ps)
inclusionCheck :: FilePath -> [InclusionPattern] -> Bool
inclusionCheck f ps = any id (map (\(InclusionPattern p) -> f ~~ T.unpack p) ps)


processLog :: Monad m => ConduitT BSS.ByteString BSS.ByteString m ()
processLog = do
-- metadata (is the next string stdout or stderr)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do I understand correctly that the first 4 bytes in the STREAM just denote that the data marked as collected from stdout/stderr?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As I explained in the issue, I couldn't find any documentation regarding the format. All I could find was this repo: https://github.com/mafintosh/docker-raw-stream

In that repo, the second 4 bytes represent the length of the subsequent string, and the first bit denotes whether the string is from stdout or stderr.

I assume that the first 4 bytes are reserved for metadata.

I have had the opportunity to test this branch and it works as expected.

_ <- CB.take 4
len' <- CB.take 4
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we only take 4 bytes at a time?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean, why don't we read the first 8 bytes in one go?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure reading it in one go and partitioning into two parts would be terribly more efficient.

let len = BS.foldl (\i w -> shiftL i 8 .|. fromIntegral w) 0 len'
case len of
0 -> return ()
n -> do
chunk <- CB.take n
yield . BS.toStrict $ chunk
processLog