watch Option for SAM Build command

[jfuss]

Describe your idea/feature/enhancement

When using sam build to build functions that will later be invoked through sam local [invoke|start-lambda|start-api], you need to run sam build before each invoke. The reason for this requirement is because the build command places artifacts (code + dependencies + updated template) into a build folder. The sam local [invoke|start-lambda|start-api] use these built templates (this has updated locations to the built function code) by default but there is no way to update the built code without running sam build again.

Proposal

For the initial implementation of sam build, the --watch option was out of scope. The proposal is to add support for this option to all customers to enable building when source code changes.

An alternative could be to make the sam local suite of commands build the function during invoke. The main concern with this is the speed at which you can now invoke (other build enhancements might be a requirement for this, speed, incremental builds, etc).

Workaround

All of the sam local command allow 'reloading' of the function code. This means for a given template, if you update the contents of a functions Code/CodeUri, SAM CLI will mount this updated content. So the current option for making interactions easier is to have two terminal windows open, one that builds and one that you local invoke with.

[alxx]

To me this only happens using Ruby once I've added a gem into the equation, with my functions having Gemfiles in their respective directories. Then I'm forced to build using --use-container parameter, which takes a lot of time; simply running sam build doesn't work because the app doesn't find the gems.

If I don't use gems at all, then I have live reloading; but once I need gems, everything only works by rebuilding inside a container. And since that takes about a minute, it makes SAM pretty much unusable for me. I can't build every time I make a small change, it just takes too long.

[AmillerSC]

Is there any progress on this? Having to build every time for the most minor of code changes is not ideal.

[sliedig]

I'd like to see a slightly different take on this proposal. Rather than having a continual build function through --watch option, only build functions that have changed since the last build. I often find myself having to rebuild functions that haven't been modified, or just making changes to the template file, making the feedback loop unnecessarily long.

[tamvm]

To me this only happens using Ruby once I've added a gem into the equation, with my functions having Gemfiles in their respective directories. Then I'm forced to build using --use-container parameter, which takes a lot of time; simply running sam build doesn't work because the app doesn't find the gems.

If I don't use gems at all, then I have live reloading; but once I need gems, everything only works by rebuilding inside a container. And since that takes about a minute, it makes SAM pretty much unusable for me. I can't build every time I make a small change, it just takes too long.

Hi @alxx,

So what is your workaround for that? Or can you share your experience on that?

Thank you,

[mims92]

I don't like to compare but Chalice as this feature if I'm not mistaken ;)

[justengland]

Assuming this is hard to build the watch option since this has been open for the better part of a year. It seems an easier option would be to use a docker volume to share your code rather than requiring a build, this way you do not have to have an actual watch.

[dgomesbr]

For a workaround, you're able to simply delete the .aws-sam folder at the root of your project and get back invoking your function.

ref: rm -rf .aws-sam/*at root.

[dz902]

Lacking of a watch mechanism is eating much of my time (and laptop battery). Would love to have this feature. Especially we are so fully-heartedly dedicate ourselves to serverless thus creating lots of APIs and functions that could really use some watching.

[serverhorror]

For a workaround, you're able to simply delete the .aws-sam folder at the root of your project and get back invoking your function.

ref: rm -rf .aws-sam/*at root.

That doesn't work for me. Running in python3.8 now and it doesn't pick up the packages in requirements.txt.

[codecounselor]

If you're using a Node runtime you may want to consider using Webpack to build bundles for each function instead of sam build. Then you can use webpack -w to have it watch for changes which sam local picks up.

I followed this very excellent write-up: https://dev.to/elthrasher/managing-multiple-functions-with-aws-sam-and-webpack-1581
I found https://www.npmjs.com/package/aws-sam-webpack-plugin also but preferred the simplicity of the former setup.

The relevant config:

webpack.config.js

module.exports = {
	entry: {
		functionA: path.join(__dirname, 'src/functionA.js'),
		functionB: path.join(__dirname, 'src/functionB.js'),
	},
	output: {
		libraryTarget: 'commonjs2',
		path: `${__dirname}/build`,
		filename: '[name].js',
	},
	mode: process.env.NODE_ENV === 'dev' ? 'development' : 'production',
	target: "node", // Let webpack know to generate a Node.js bundle
}

template.yaml

  PingFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: build/
      Handler: functionA.lambdaHandler
      Runtime: nodejs12.x

1. npx webpack -w
2. sam local start-api

[kamrankhanwastaken]

I just used nodemon for this. Going to put this here in case it would make anyone else's life easier:

npm i -g nodemon
nodemon --exec sam build

Run this in the root folder in one terminal and run sam local start-api in another. nodemon should build the application automatically on any code changes.

[eamarce]

Hey guys,

So, we created a small npm package based in nodemon (samwatch):

https://www.npmjs.com/package/samwatch

https://github.com/mxitgo/samwatch

The way it works is, it copies your js/json files as you save them from the source folder to the corresponding .aws-sam/build folder (as long as the names are the same for your lambda and the code uri in template.yaml).

That is useful to see your changes reflected on the fly if you are running sam local start-api, so you don't have to run sam build after every change.

On the other hand, if the corresponding file copy is not within the .aws-sam/build folder already, the package will trigger a sam build (unless you use n parameter).

The package has helped us get a hot reload feeling for our local lambda function development with sam.

I hope anyone can find it useful.

Thanks!

[butsjoh]

My usage is also with ruby code so i am using the live reloading as follows currently without actually using sam build. Not sure though if this is recommend or bad practice so would like to get feedback on it.

Let say i have the following folder structure with 2 functions:

lambda_example/
|--hello
||--app.rb
||--Gemfile
|--hello_other
||--app.rb
||--Gemfile
|--template.yaml

instead of using sam build i just do: (i do this from the root directory lambda_example)

# To install new gems for hello (only need to run this when you update the Gemfile or when you don't have a Gemfile.lock yet)
bundle install --gemfile=hello_world/Gemfile

# Now the following installs the gems into the vendor bundle inside the function directory
# similar what sam build does actually if you would look in the .aws-sam folder
bundle install --gemfile=hello_world/Gemfile --deployment --path vendor/bundle

Same thing then for the other function

bundle install --gemfile=hello_other/Gemfile
bundle install --gemfile=hello_other/Gemfile --deployment --path vendor/bundle

Now if you then issue sam local start-api and you make changes to the code then the live reloading stuff should be working without actually needing to build each time. Ofcourse this only probably works if you don't need to build native extensions for gems but esentially you only need to call the bundle install commands above if you change gem dependencies which should not happen that often.

The samwatch thing is cool but does not work for ruby cause it only monitors js or json files it seems. So maybe add an extra option in your cli @eamarce to specify which files to watch?

[paulmowat]

Now that warm containers has been delivered under #2383 . Is there anyway this can be prioritized? It would help make that dev experience so much better.

[anubhavmalik]

Its such a painful process running sam build even for a small change such as a console.log(). Any good workaround? The solutions above (webpack and samwatch) force a folder structure which I am not willing to do.

[mims92]

@anubhavmalik only thing is to modify the code in the build folder.

[dror-g]

such a pain... maybe instead of a "watch" option, which will still need to build on every change and slow down development, we could mount the source folder in the container?

[ppejovic]

In the case of lambda container images it looks like a sam build is always required as the build output is an image rather than code being pushed to the .aws-sam directory. I have a working docker compose setup that spins up two services; one serving the API with sam local start-lambda and the other that rebuilds the images with sam build on code changes using nodemon. The feedback loop could be quicker but it works fairly well.

[9oelM]

In the case of lambda container images it looks like a sam build is always required as the build output is an image rather than code being pushed to the .aws-sam directory. I have a working docker compose setup that spins up two services; one serving the API with sam local start-lambda and the other that rebuilds the images with sam build on code changes using nodemon. The feedback loop could be quicker but it works fairly well.

I've been actually doing this and I am curious if this will produce many local images as it builds again. Is it going to do that? Seems like that's what's happening to mine.

[jhechtf]

I remember using a previous version of the SAM cli and being able to just run the sam local start-api and so long as I didn't need to add a new endpoint to the template.yml file the code was reflected automatically, no need for sam build every single time. I even have a repository setup with a nodemon script set explicitly to watch only for changes in the template.yml file so that I wouldn't forget to restart it if I modified the template file.

I'm wondering why this behavior was changed? Now I'm honestly considering not using the SAM cli because this workflow is rather annoying.

[mims92]

Maybe a simple solution to that: make a symbolic-link (ln -s) from the source code to the build folder?

Something like:

sam build ...
ln -s src/* .aws-sam/build/FunctionName

[jfuss]

@jhechtf That behavior still holds true the difference is when you use build, SAM CLI places the artifacts into another directory with an updated template. When running sam local start-api, it will read the "built" template. So you can get the same behavior through sam build in a different terminal or not using sam build at all and therefore the "source" template is always read.

As a note, we do prioritize off of reactions. So more reactions do help us in deciding priority for outstanding feature requests from the community.

[jhechtf]

@jfuss I've seen that mentioned before, about simply not using "sam build", but when I remove the .aws-sam directory and attempt to run sam local start-api I receive errors stating that it can't find any node modules, e.g. I use the @aws-sdk/client-dynamodb and without running sam build I get the error of Error: Cannot find module '@aws-sdk/client-dynamodb'

[jfuss]

@jhechtf Is the node modules within the CodeUri of the Function? If not, you need to install your dependencies into the CodeUri location. SAM CLI mounts the CodeUri into the container, which means all the code needs to be under that directory. This is what sam build does for you, it just does not mutate the source and instead places everything under .aws-sam/

[jhechtf]

@jfuss the template that I am using is the nodejs14.x generated by the sam cli, which does not include any CodeUri properties. I'll try to add that in, but considering the code gets generated in the <project root>/src/* and the node_modules are at <project root> I'm not entirely sure that's going to work out for me.

sam --version
SAM CLI, version 1.27.2

[jfuss]

@jhechtf Happy to help but let's spin this off into a different issue. This will distract from the original intent of this issue/request.

[memeyoo]

Check it.

• template.yaml 예시
  • 사용하는 라이브러리를 로컬경로의 Layer로 설정

CommonLayer:
    Type: AWS::Serverless::LayerVersion
    Properties:
        LayerName: common-dependencies
        Description: Common dependencies
        ContentUri: layers/common/
        CompatibleRuntimes:
            - python3.8
        RetentionPolicy: Retain

HelloWorldFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
        CodeUri: functions/consult/
        Handler: app.lambda_handler
        Runtime: python3.8
        Layers:
          - !Ref CommonLayer

• 로컬경로에 requirements.txt에 명시된 라이브러리 설치
  • 경로 예시: layers/common/python 아래에 설치 (최상위 폴더이름은 "python" 필수)

$ cd layers/commons/
$ pip install -r ../상대경로/requirements.txt -t ./python

• 로컬 실행 시 템플릿 파일 직접 명시 (프로젝트 내 build 폴더는 없어야 함)

$ sam local start-api -t template.yaml --skip-pull-image

[metaskills]

Summary

I am 100% NOT a fan of this watch idea. Editing code and hitting refresh would still take a long time. Perhaps the solution to the problem is rethinking what is needed. Perhaps... can we have a start-api option that does a simple API Gateway => Lambda simulation similar to Zip which would do the following so we can have code reloading for containers?

• Does not need sam build to run.
• Assumes the host has everything needed like platform, packages, system deps, etc.
• Avoids using a "runtime" container with docker mounts. Because above.

I do not know if this is a fair ask of the SAM team. Maybe I need the work y'all have done inside of SAM to be a smaller components to assemble. Maybe we have this already and I have not found a good way to do just that... for my needs?

Context & Hack Attempts

For both Ruby, Node, etc, we use the SAM build image as a Docker in Docker development image. A good example of this in practice for Ruby/Rails is here. https://lamby.custominktech.com/docs/quick_start. When doing this, we can assume the local "host" (docker dev image) has everything we need, system dependencies, dev code which can be edited, etc. All we need is a simple API Gateway => Lambda emulator.

For Rails, we completely side step this issue by using the built-in ./bin/rails server and the need for API Gateway => Lambda handler integration because that is tested in abstract at another layer. See here: https://github.com/customink/lamby

Today I was coming back to an old Node zip function I wrote a few years ago and wanted to move it to our latest patterns which includes SAM dev containers, production package type image, etc and was super frustrated that SAM does not have a way to support start-api with container images. I tried a few hacks. First, I tried a little bit of @mims92 recommendation by leaning into the need to do sam build (which technically I do not need because I an assume the host has everything) to trick start-api into thinking it had what it needed. Along with forcing the host because Docker in Docker with compose, etc. Again, see Lamby Rails cookiecutter. Still, I could not edit my code and hit refresh in the browser.

if [ ! -d "./.aws-sam" ]; then
  sam build \
    --parameter-overrides "StageEnv=development"
  mkdir  ./.aws-sam/build/Lambda
  pushd ./.aws-sam/build/Lambda
  ln -s ../../../src src
  popd
fi

sam local start-api \
  --host '0.0.0.0' \
  --port 3031 \
  --container-host 'host.docker.internal'

I then had the crazy idea maybe I could use the RIE (https://docs.aws.amazon.com/lambda/latest/dg/images-test.html) built into my image. So I tried adding this to my Dockerfile-build (dev & build image) along with a change to my local ./bin/server script. But I quickly learned this is too LOW LEVEL and not going to work for me. I need the API Gateway integration.

RUN curl -Lo /usr/local/bin/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie \
    && chmod +x /usr/local/bin/aws-lambda-rie

docker-compose run \
  --entrypoint "/usr/local/bin/aws-lambda-rie --log-level debug" \
  -p 9001:3031 \
  node-container-test

For this hack I tried to see if I could have a development only version of my (template-dev.yaml) to use just with start-api to see if I could get some simple Zip behavior in my dev container. The idea being this would only be used for the local development server, not used during package & deploy. The start-api command would pass this template.

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
  DevLambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: .
      Events:
        HttpApiProxy:
          Type: HttpApi
          Properties:
            ApiId: !Ref DevHttpApi
      Runtime: nodejs14.x
      Handler: src/index.handler
      MemorySize: 512
      Timeout: 30
  DevHttpApi:
    Type: AWS::Serverless::HttpApi
    Properties:
      StageName: development

sam local start-api \
  --template 'template-dev.yaml' \
  --container-host 'host.docker.internal' \
  --docker-volume-basedir $DIDPWD \
  --host '0.0.0.0' \
  --port 3031

Where $DIDPWD would be $PWD passed down from the host. This used to be needed during a really old SAM version but I can NOT confirm it works (it should) because...

💥💣💥 SAM HAS ANOTHER BUG THAT PREVENTS THIS. SEE HERE: #2837

But even if that did work, it would not be a good workaround for a container function that might have done a system dependency via yum install on both the production and development Docker files.

Wrapping Up

I GET IT! I get why this is hard. The sam build historic ways of working make it difficult to solve everything. But I do think we have been pained into a corner and exposing an interface like a watcher here is adding to the list. It would be prone to being slow. Maybe there is a better way forward?

Is there a way to use SAM's API Gateway => Lambda integration in some way to make local development work?

[metaskills]

Using @sbhvt's comment here #921 (comment) I was able to change my last hack above the 💥💣💥 in the following ways. First in my docker-compose.yml file. Then in my script used to start the server in the SAM dev container.

volumes:
  - ${PWD}:${PWD}
working_dir: $PWD

sam local start-api \
  --template 'template-dev.yaml' \
  --container-host 'host.docker.internal' \
  --host '0.0.0.0' \
  --port 3031

But this only works on Mac! I can not get it to work with GitHub Codespaces yet. I tried following this advice (https://dev.to/natterstefan/docker-tip-how-to-get-host-s-ip-address-inside-a-docker-container-5anh) and used 172.17.0.1 but that did not help.

So I'm blocked even on the hacks of hacks :(

[metaskills]

GOT IT TO WORK ON CODESPACES. Just needed to clear out Docker images and start from scratch. I'll be using COMPOSE_FILE to add another compose layer for Codespaces that adds this.

extra_hosts:
  - host.docker.internal:172.17.0.1

Which allows the bin/server wrapper to do the following, the key here is was adding --container-host-interface '0.0.0.0' which is needed for Linux and does not hurt the Mac side of things. So final bin/server is something like this.

sam local start-api \
  --template 'template-dev.yaml' \
  --container-host 'host.docker.internal' \
  --container-host-interface '0.0.0.0' \
  --host '0.0.0.0' \
  --port 3031

AGAIN! This is hacks on top of hacks. IMO we do not need a watch command but better tooling to expose a simple API Gateway => Lambda server option that works with both Docker in Docker and Lambda Containers where it can be assumed that the Docker dev env has everything setup (built) where sam build is not needed.

[lgabrielvleon]

Hey guys,

So, we created a small npm package based in nodemon (samwatch):

https://www.npmjs.com/package/samwatch

https://github.com/mxitgo/samwatch

The way it works is, it copies your js/json files as you save them from the source folder to the corresponding .aws-sam/build folder (as long as the names are the same for your lambda and the code uri in template.yaml).

That is useful to see your changes reflected on the fly if you are running sam local start-api, so you don't have to run sam build after every change.

On the other hand, if the corresponding file copy is not within the .aws-sam/build folder already, the package will trigger a sam build (unless you use n parameter).

The package has helped us get a hot reload feeling for our local lambda function development with sam.

I hope anyone can find it useful.

Thanks!

Hi!

Works with PackageType: Image?

Thks!

[yuyokk]

Workaround
All of the sam local command allow 'reloading' of the function code. This means for a given template, if you update the contents of a functions Code/CodeUri, SAM CLI will mount this updated content. So the current option for making interactions easier is to have two terminal windows open, one that builds and one that you local invoke with.

Is there a workaround for projects created with aws-cdk. Do I need to run cdk watch or npm run build (ts => js) in the second terminal?

[S33G]

Just having a sensible way to have a mounted volume inside the docker container would be awesome for my use case

[vavdoshka]

AWS, What does it take to solve that significant UX problem?

Thanks to everyone who contributed with their solution; I'll re-iterate on what worked for me for Python project

# Running Locally

1. Start lambda build loop (will watch for changes and rebuild)

nodemon --exec "sam build --template template.yaml --use-container" -e py
* install nodemon globally if you don't have it already `npm install -g nodemon`

2. Start local api server on port 5000

sam local start-api -p 5000

[S33G]

AWS, What does it take to solve that significant UX problem?

Thanks to everyone who contributed with their solution; I'll re-iterate on what worked for me for Python project

# Running Locally

1. Start lambda build loop (will watch for changes and rebuild)

nodemon --exec "sam build --template template.yaml --use-container" -e py
* install nodemon globally if you don't have it already `npm install -g nodemon`

2. Start local api server on port 5000

sam local start-api -p 5000

This appears to be a JavaScript example

But Python would need to be handled also

[vavdoshka]

AWS, What does it take to solve that significant UX problem?
Thanks to everyone who contributed with their solution; I'll re-iterate on what worked for me for Python project

# Running Locally

1. Start lambda build loop (will watch for changes and rebuild)

nodemon --exec "sam build --template template.yaml --use-container" -e py
* install nodemon globally if you don't have it already `npm install -g nodemon`

2. Start local api server on port 5000

sam local start-api -p 5000

This appears to be a JavaScript example

But Python would need to be handled also

Nope, this works in python, it watches the files with the .py extension - -e py

After playing with it for some time, I've added a delay to avoid re-building too quickly.

nodemon --exec "sam build --template template.yaml --use-container" -e py --delay 2.5

[cesar-presence]

almost 4 years of this issue. Any news?

[DanielKucal]

Here's the command I ended up with for now:

sam build && concurrently "nodemon --on-change-only --ext ts --exec sam build" "sam local start-api"

Requires npm i -g nodemon concurrently

It runs nodemon and sam local start-api in the background simultaneously. The trick is to have the build available before creating docker image, so it's done beforehand and nodemon does it only after changes (using --on-change-only flag).

[paul-uz]

+1 for this feature

[wyatt-shein]

I've only been experiencing this same issue for a bit so im only vaguely familiar but id like to a propose another solution that the sam team at AWS may be able to implement more easily than creating hot reload for the .aws-sam built folder:

Context:
Im using node but you can do this with others like python in a similar way.
If i have a lambda layer that manages my dependencies and some lambda functions that utilize the dependencies in that layer, I can set my folders up in the following way that allows me to use my functions successfully with hot reload and without having to run sam build.

Article: https://aws.amazon.com/blogs/compute/working-with-aws-lambda-and-lambda-layers-in-aws-sam/
$ tree sam-app
sam-app
├── README.md
├── dependencies #layer example
│ └── nodejs
│ └── package.json
├── hello-world #function example
│ ├── app.js
│ └── tests
│ └── unit
└── template.yaml

When using the nodejs folder as described in the article and the docs, as long as you have generated the node modules manually using npm i, it should work just fine to run your code locally and have hot reloading without using sam build.

The issue with this method is that when you need to move beyond local environments to testing or production, most pipelines will utilize sam build. When sam build is run, it will generate a nodejs folder inside your nodejs folder and not generate the dependencies correctly as it would if you didnt have the nodejs folder created (discussed in previous comments).

So what im saying is maybe the sam developers would have an easier time fixing the current issue of duplicate nodejs folders which causes the dependency generation to not work, rather than trying to fix the issue of hot reload not working for the 'sam build' built .aws-sam folder.