Best way to replace a worker container with individual containers for jobs

[DrewVartanian]

Right now, I have a web container (A Load Balanced Web Service), a worker container (A Backend Service), and a couple scheduled jobs. As offline jobs (jobs that are offloaded to the worker container) come in, the web container puts the job info on a Redis queue, and a running supervisor process on the worker container pulls that info from the queue, before running the job. This architecture was setup when we were running on EC2 instances (no docker or containers), and I'm thinking it might be better to programmatically run a container for each offline job. Kind of like a scheduled job, but triggered from our codebase rather than on a schedule. What would be the recommended way to set this up with or along side copilot? I already have the image that would be used in ECR, as the scheduled jobs all share an image. My initial thought would be to setup a step function that I can trigger to execute container with the jobs image, making sure to hit the same tag as my web image, and providing the entrypoint to run the appropriate code for the offline job. That said, I'm not sure if there is a simpler/more direct way to start and run a container from another container with copilot, or a better way to set this up with infrastructure as code, as opposed to setting up a step function manually in the AWS Console.

Thanks!

[efekarakus]

Hi @DrewVartanian !

To make sure that I'm understanding correctly, are you trying to simplify your architecture by getting rid of the intermediary Backend Service and have the Load Balanced Web Service run a Scheduled Job directly?

Executing the state machine sounds great, but I'm not sure if this is possible: "and providing the entrypoint to run the appropriate code for the offline job." My understanding is that you can't override the command of the container while starting the execution of a state machine.

Have you explored calling just RunTask instead from the web service? You can run-task using the scheduled job task definition and override the command and periodically DescribeTasks to make sure the task stopped successfully.

However, these options create a tight coupling between the scheduled jobs and the web service. On the other hand, putting a queue in between and letting a backend service consume the work asynchronously from a queue would decouple the web service and increase its availability.

I might be misunderstanding your architecture completely 😅 please let me know if my answer isn't relevant.