RFC to add Cloud Native Buildpacks to CF Deployment

[Gerg]

For easier viewing: https://github.com/cloudfoundry/community/blob/rfc_cnbs/toc/rfc/rfc-draft-add-cnbs.md

cc @ryanmoran, @robdimsdale, @dsabeti, @aramprice, @dsboulder

[emalm]

Thanks! Very excited to see this proposal come forward. I suggested some corrections and had a few initial questions.

[emalm]

Do we have an idea of how the Paketo CNBs would produce and provide SBOM data when running in Cloud Foundry and how the SBOM data would be made visible to app teams and platform teams? This innovation doesn't seem to be addressed explicitly in the proposal below.

[robdimsdale]

Buildpacks produce SBOMs at known locations on the filesystem of the OCI image. It would probably be the responsibility of the shim layer to ensure these SBOM files are processed. I assume they wouldn't be packaged into the droplet, as that's just the executable, so they would have to provided to some Cloud Foundry handler via CF-specific metadata.

Maybe you knew all that already and your question was just around how CF will define, consume, and render this information. In which case I don't have any extra information :)

[emalm]

Thanks, @robdimsdale ! Actually, I've been unclear on the specifics of how the SBOM data gets propagated even in a K8s context, especially so it can integrate with system such as Grype that can process the SBOM data. Does the OCI image metadata contain the SBOM, or do those processing systems have to unpack the filesystem layers in the image to get it from a canonical location? If the former, does kpack then propagate the SBOM data from the filesystem into the image metadata?

Anyway, I definitely think we can work out all the details of exposing CNB-generated SBOM data to the authorized set of CF users in due time, but it'd be great to refer to it at a high level as intended work in this part of the proposal.

[robdimsdale]

The SBOM is defined files on the filesystem of the resultant application image, not OCI image metadata. I suspect that's because there's too much content for the metadata - a typical application could easily run into 10k-100k lines of SBOM content. Paketo buildpacks initially provided some SBOM data via the OCI image, but that is now deprecated in favor of the CNB-defined filesystem-based SBOM format.

The platform (i.e. pack, kpack) is responsible for providing the SBOM information to end users. For example, pack provides a pack sbom download command which downloads the sbom files from the resultant application image onto the local filesystem. Commercial vendors choose to process the SBOM data in other ways (via a UI, integration with third-party tools, etc).

The V3 platform is responsible for orchestrating the V3 lifecycle, so one way I think of the V2<->V3 shim (plus CAPI, diego, etc) is that it is effectively implementing parts of the V3 lifecycle specification. So, to expose SBOM data in CF we would want to modify the shim to extract the SBOM from the resultant OCI image, and then propagate that back to CAPI in some CF-specific format.

If you want dig in further (and would like some light bed-time reading) you can take a look at the V3 buildpacks spec. You'd probably also need to read a bit of the platform spec too.

[beyhan]

I also agree that having SBOM will be a great improvement but it will require additional effort to make it available to CF users. That is why, I agree with @emalm that we should describe it in this RFC in "high level" so that we can have an agreement on it.

[Gerg]

@robdimsdale Can you expand more on why you think the SBOM shouldn't be in the droplet? There are some potential advantages to bundling the SBOM with the executable. For example:

• Benefiting people who are scanning droplets directly from the blobstore (though maybe this won't be as common if we have built-in SBOM support).
• Preserving the SBOM when downloading/uploading droplets.

Either way, 10k-100k lines per SBOM is probably too large to put in the CCDB, so we will likely want to store the SBOMs in the blobstore. We could take an iterative approach where the consolidated SBOM is initially only available via the droplet, and then we can build more complex workflows on top of that.

[robdimsdale]

Maybe my mental model is wrong, but I was thinking the droplet was intended only to be the executable and its supporting filesystem. If there's already precedent for storing metadata in the droplet (or we're willing to create that precedent) then that would be fine. Storing the SBOM data in the droplet is closer to how the V3 lifecycle handles the SBOM data.

[robdimsdale]

And yes, I don't think you'd want to parse the SBOM data to store it in a structured format in CCDB. I was assuming there was already some way of storing large amounts of (potentially compressed) metadata already in CF/CAPI.

[AP-Hunt]

TOC discussion summary:

• out of scope to have them available for lifecycle
• can iterate on SBOMs stuff after first pass
• don't know what tooling we want
• resolve to note in the RFC that the SBOM data will reside in the droplet

[Gerg]

5a7ebf4

[emalm]

Which v2 and v3 buildpacks process service-binding data? I understand that the CF Java buildpack and the Paketo Java CNB both do, but are there others that do as well?

[robdimsdale]

In general service bindings aren't used by most of the v3 buildpacks. They are predominantly used in the APM buildpacks (e.g. to provide credentials to third-part systems). Most of the top-level Paketo buildpacks (e.g. python, nodejs) don't ship with APM buildpack integration, but they could be modified to include those APM buildpacks if needed. In order to provide a similar surface area to the v2 buildpacks, they probably would have to do so.

[emalm]

What will the experience for application developers be like as this new line of buildpacks are introduced? How would we recommend that platform teams help them adopt the CNBs in order eventually to retire the CF v2 buildpacks?

[stephanme]

How would the buildpack detection work when v2 and v3 buildpacks are installed both?

[robdimsdale]

How would the buildpack detection work when v2 and v3 buildpacks are installed both?

I'd expect them to be installed under different names, so you would specific them specifically with cf push -b v3_python_buildpack, or rely on the buildpack ordering for detect. However, given that they will generally detect on the same things (e.g. presence of language-toolchain-specific files) I would expect whichever is first in the detection order would likely win. For backwards-compatibility we would probably want to suggest operators put the v3 buildpacks lower in the order and ask users to explicitly opt-in via cf push -b.

[emalm]

Great, that makes sense to me about the distinct buildpack names and v2/v3 ordering for appropriate autodetection. We'll need to choose those names carefully so that we're all happy enough with them for the long term, since presumably any app manifests or cf push commands that select buildpacks by name will use those new names and it's generally a pain for app teams to change them later without a good reason.

[Gerg]

I agree with the above. Also, if the v3 buildpacks are using the Paketo stacks, then the app's stack will determine which set of buildpacks it uses.

[AP-Hunt]

TOC discussion:

• resolve to provide recommendations for naming, so people can knowingly opt in

[Gerg]

ba6fa1a

[emalm]

It looks like the identifiers for types of services may already vary between the v2 and v3 buildpack lines: for example, the Java buildpack support for Google Stackdriver Profiler looks for a service binding of name/type/label google-stackdriver-profiler, but the Paketo Google Stackdriver CNB looks for a binding of type GoogleStackdriverProfiler. How will app teams use the new buildpack integrations against existing service brokers that return the old identifiers? Will the translation of VCAP_SERVICES data into K8s service-binding file hierarchies also include a translation of known service identifiers to the corresponding Paketo-recognized type, or do we have some other proposal in mind?

Also, the K8s service-binding spec has only 3 canonical places for metadata: the required type and optional provider fields in the binding payload, and the name of the binding directory under the service-binding root directory. Have we given any consideration to how the VCAP_SERVICES metadata should map to these identifiers, and whether and how any additional service-binding metadata that CF buildpacks might inspect should map to conventional fields (presumably other files in the binding-specific directory)?

[emalm]

BTW, there's some prior art here with Korifi as it translates its user-provided service instances into the K8s-service-binding file hierarchy presentation. I believe the only thing it's doing automatically is mounting each binding in VCAP_SERVICES under ${SERVICE_BINDING_ROOT}/<instance-guid>/, and the end user has to set type and provider themselves in the UPSI credentials, but @tcdowney would be able to verify. And I don't think those decisions are set in stone, either.

[tcdowney]

Yeah that's correct. If the user does nothing then they will get credentials mounted at ${SERVICE_BINDING_ROOT}/<instance-guid>/ and the type value will be set to user-provided (I don't believe we default provider to anything). The user has the opportunity to override these by specifying --binding-name during binding creation (this will mount it at ${SERVICE_BINDING_ROOT}/<binding-name>/) and by including the type/provider fields in the credential payload like you said.

For what it's worth, I don't think the ${SERVICE_BINDING_ROOT}/<instance-guid>/ structure is very useful and perhaps not even idiomatic. I think Cloud Controller has sufficient validations in place to ensure that binding name is unique at the app level (and that an instance can only be bound once) so it should mount them all at ${SERVICE_BINDING_ROOT}/<instance-name>/ / ${SERVICE_BINDING_ROOT}/<binding-name>/ (you could just reference name in VCAP_SERVICES for this) by default. The GUID is too dynamic for an app to meaningfully make use of it here.

[loewenstein]

Maybe buildpacks/libcnb#227 and buildpacks/libcnb#228 are making a change here?

@emalm I did no real thorough search yet, but I think it is mainly apm buildpacks that need service bindings, I know for sure for the dynatrace-buildpack.

[Gerg]

Adding @c0d1ngm0nk3y @dmikusa @samj1912, in case y'all are interested in this RFC.

[dmikusa]

This RFC seems awesome!

We're working out something in buildpacks/libcnb#228, but hopefully, that is resolved soon and we can cut a release. Then buildpacks based on libcnb can get access to service binding info on CF in a transparent way. There are a couple of caveats though. 1. That's only for libcnb-based buildpacks and 2.) you can't really represent all of the VCAP_SERVICES data with K8s service bindings so you lose labels, for example.

APM buildpacks are a big user of service bindings, but they are used elsewhere also. Maven/Gradle use them for build secrets, like accessing a private repo. They are also used for injecting trusted CA certificates, and for dependency mappings also (a mechanism to redirect where the buildpack fetches a dependency from).

[loewenstein]

1. That's only for libcnb-based buildpacks and 2.) you can't really represent all of the VCAP_SERVICES data with K8s service bindings so you lose labels, for example.

Good points @dmikusa. We should probably check for packit based Paketo buildpacks that need service binding data. I supposed this can be addressed on Paketo side by either pushing packit to use libcnb (I understood that's rather a priority thing than anyone vetoing that) or provide similar logic directly in packit.
Regarding the second point, we should probably search the Paketo buildpacks for usage of service binding data - it's likely not needing anything that can't be translated from VCAP_SERVICES or did you have a case in mind where this direction of information loss could pose a problem?

[dmikusa]

(I understood that's rather a priority thing than anyone vetoing that) or provide similar logic directly in packit.

It is something we want to do, but I don't know the priority. We've been working for a libcnb v2 for a while now and when that releases, the changes in it will enable packit to base on top of libcnb v2. I don't know how long that will take though. Just getting libcnb v2 shipped has taken some time, but I hope we're getting close. It needs one more thing, support for buildpack extensions.

did you have a case in mind where this direction of information loss could pose a problem?

I was mostly thinking that it could be a change for application developers. Right now, as an app dev using CF and say the java-cfenv, you can search by things like the label or tags for a class of service. That wouldn't be possible, at least not with the current implementation. If an application were doing that, it would need to change to look by vendor or search by name. The binding type is set to the vendor name, and the service name is set to the binding name.

[dmikusa]

Yes, that sounds right. Although, I'm not 100% sure I follow the first option.

You mentioned, Maybe we should contain this logic in a buildpack instead? but you can't have a buildpack that translates VCAP_SERVICES to bindings because the buildpack detection process runs in parallel so one buildpack cannot intercept and translate them before the others run. Many of the buildpacks check for service bindings at detect time to determine if they will execute their builds, so that is important.

Whatever performs the translation step needs to do it before the buildpacks run. The libcnb approach works because the buildpacks use that library to fetch bindings, so it can translate just in time. The other option would be to translate them before you run the buildpack lifecycle, but then you'd have the translation logic in the platform, i.e. CF, and it seems like that's not desirable.

[Gerg]

You mentioned, Maybe we should contain this logic in a buildpack instead? but you can't have a buildpack that translates VCAP_SERVICES to bindings because the buildpack detection process runs in parallel so one buildpack cannot intercept and translate them before the others run. Many of the buildpacks check for service bindings at detect time to determine if they will execute their builds, so that is important.

Ah, that's good to know.

The other option would be to translate them before you run the buildpack lifecycle, but then you'd have the translation logic in the platform, i.e. CF, and it seems like that's not desirable.

I think it would be desirable, if we had a way to automatically convert all CF bindings into their equivalent k8s bindings. For example, using @emalm's example above, CF binding name/type/label google-stackdriver-profiler becomes k8s binding type GoogleStackdriverProfiler. If we could say "Always replace hyphens with intercapitalization" the mapping would be forwards-compatible. If we have to do a manual mapping between CF and k8s bindings, then I'm worried that a CNB adding support for a new binding type will also require a corresponding update to the mapping logic in Cloud Foundry.

This is outside of my core area of knowledge, so let me know if what I'm saying doesn't make sense or is missing something.

[dmikusa]

I definitely understand your concern there. I think the problem with an automatic mapping like that will be that they don't match up to what the Paketo buildpacks currently expect. For example, we look for StackdriverProfiler and StackdriverDebugger.

I think that on either side of the service binding interface, you have a problem that the implementor can pick arbitrary values. For example, whoever implements the service broker on CF can pick the values they use for the provider and credentials. Similarly, whoever implements the buildpacks can pick different types to look for in the service bindings.

Since we're talking about CF and Paketo and we're both part of the CFF, it seems reasonable that we agree upon either a standard way of mapping them or that we agree on a well-known set that should be used. We can't stop using the current types in Paketo buildpacks, but we can certainly add new types to look for that are compatible with well-known CF services.

FYI, there are some well-known secret entries in the service binding spec, and we should definitely try to use these. Spring Cloud Bindings also has a number of types that it knows about and reacts to, so we'll want to take that into consideration as well since we have a lot of Spring users.

[AP-Hunt]

TOC discussion:

• what buildpacks need VCAP_SERVICES at build time? Is it just those that are libcnb based?
  • some buidpacks (e.g. datadog) looking for env variables and injecting an agent at build time
• having buildpacks be aware of VCAP_SERVICES is a viable path
• resolve to update RFC to say buildpacks require access to service bindings at build time, but leave the implementation as an open question to be answered with another RFC

[Gerg]

894bfb3

[beyhan]

I also agree that having SBOM will be a great improvement but it will require additional effort to make it available to CF users. That is why, I agree with @emalm that we should describe it in this RFC in "high level" so that we can have an agreement on it.

[beyhan]

Should we explicitly address the change required in Diego in this RFC? This is a prerequisite to have a Kubernetes Service Binding support. Or is this classified as nice to have?

[loewenstein]

As commented elsewhere, I am not sure we need to do anything. Once the change to libcnb is backported to libcnb/v1 and consumed by Paketo buildpacks, those buildpacks should be able to read service binding credentials directly from VCAP_SERVICES.

If this seems to be an agreeable approach, I suppose the service binding chapter in this RFC should reflect that of course.

[beyhan]

This mean that there are no changes planned for the current V2 buildpack lifecycle. I think that we should mention it explicitly to avoid misunderstanding here.

[Gerg]

I was intentionally non-prescriptive here about whether it would be appropriate to extend the current Buildpack App Lifecycle to support v3 buildpacks, or whether it made sense to create an entirely separate lifecycle (CNB App Lifecycle?).

If desired, we can update the RFC to be more opinionated on that matter.

[beyhan]

I read it as decided and wanted to make it clear but it is even better that you think for different options here. I agree that it makes sense to decide on this when we have more input from phase 1. Regarding extending the current buildpack lifecycle or having a CNB app lifecycle I think we should go with the approach which makes possible next phases/steps like building OCI images and integration with Container registries easier. At the moment I'm in favour of a separate CNB app lifecycle.

[loewenstein]

I am not sure if you meant this that way @Gerg, but I agree with @beyhan that we should keep the longer-term in mind and sight when discussing and working on a shorter-term workaround.

I see the benefit of munching Paketo buildpacks and their execution environment (aka CNB lifecycle) together and wrap it up in a v2 compatible shim. Any user can try cf push -b v3_java_buildpack on any foundation and give feedback on if and how their app still works.

When starting efforts inside the Cloud Foundry codebase, we should take into account how this can end up in running CNBs natively. We anyway need to run the CNB lifecycle in a Diego scheduled container. We should make sure this stays usable as is and could be pushed into a registry, even if we add logic that extracts a droplet from the result.

For what it's worth, we have written a Cloud Native Buildpack platform with capabilities similar to pack build that can run fully inside a container. The cnbBuild step is a bit buried in a larger codebase for a whole CI/CD system / library, but we could certainly help to extract (and maintain) it in a different place or try to push some of the platform logic directly into the lifecycle. I've never tried to run in in a Diego Task, but it sure doesn't need a Docker Daemon.

[AP-Hunt]

TOC discussion:

• follow on RFC required once we've learned lessons (phase 2)

[Gerg]

6d6bd81

[beyhan]

Are there no changes required in CF to support this goal (Easier buildpack customization/composition)? In the RFC I couldn't find anything in this direction. Or this is covered by the multiple-buildpacks support already available in CF?

[Gerg]

I don't think there are any necessary changes to CF to get this capability. The main Paketo buildpacks are themselves composites of multiple other buildpacks, which makes it easier to add/remove sub-buildpacks to meet your custom needs.

That said, this could open the door for more advanced operator controls for buildpacks in the future. For instance, if platform operators could customize buildpacks via platform configuration instead of modifying the buildpacks themselves. Definitely outside the scope of this RFC though.

[loewenstein]

I like incremental approaches in principle. One problem I see with this specific one is that we are heading ~180 degree in the other direction - if we wanted to follow up with pushing towards OCI images and container registries.
At least I don't see us reusing anything of the two phases laid out.

[Gerg]

From the system architecture perspective, this proposal is definitely divergent from building/running OCI images.

However, I think this proposal is convergent from the app developer perspective. My hypothesis is that app developers don't care that much about how their apps are packaged/run. With this proposal, app developers can get the benefits of the Paketo buildpacks, without requiring significant changes to the CF Deployment platform engineer/operator experience or system architecture.

[loewenstein]

I agree with the app dev's point of view, but I also do see the platform engineer/operator view. I don't think that we should increase the system architecture's complexity with a shim in the mid- to long-term.

FWIW we need a CNB lifecycle run in a Diego scheduled container anyway, right? So the main addition to the system architecture would be a registry, right? I would assume that when configured to use a blobstore like S3 this registry implementation will be mostly if not fully stateless.

[stephanme]

I think there is more needed than 'just a registry' to switch from droplets to OCI images. Droplets don't contain the stack and app developers are used to get stack updates automatically for buildpack apps without having to redeploy/restage their apps.
If the CNB lifecycle would produce OCI images, a stack rebase needs to be added to the runtime evacuation process if keep this user experience.

[loewenstein]

Good point and agreed, updating the stack should lead to a rebase operation (which is supported by CNBs) and a rollout of that new image.

[beyhan]

This RFC will be discussed during the TOC meeting next week on 30th of May. If you are interested you are welcome to join the TOC zoom meeting.

[Gerg]

I've made a round of edits, based on feedback from the last TOC meeting. I propose we move this RFC into the final comment period.

[AP-Hunt]

Final comment period to end on 27th June

[krismarc]

@Gerg any chance for the community update on this RFC implementation? Is there any additional place where we could read/keep track on it?

I'm excited by this one. I was thinking that paketo buildpacks will be only available in Korifi. This brings back a hope for the main buildpack used by us. Actually, IBM gave up on maintaining the old Liberty buildpack and switched over all their efforts on a new one.

https://github.com/cloudfoundry/ibm-websphere-liberty-buildpack
https://github.com/paketo-buildpacks/liberty

I'm just wondering how I could anyhow inflence on making this buildpack to participate in this initiative. Is there any chance we could loop it into this project? :)

[loewenstein]

@krismarc There has been a "successor" RFC with https://github.com/cloudfoundry/community/blob/main/toc/rfc/rfc-0028-cnb-lifecycle.md and all implementation pull requests are open already and ready for review.

There is also an additional RFC to include Cloud Native Buildpacks as system buildpacks with #830.

[krismarc]

Sorry to necro this once again. Not sure in which repository I should raise my questions about this feature.

Our CF deployment is completely air-gapped and it's unable to fetch internet dependencies.

I'm trying to run nodejs sample app using cnb described here:
https://docs.cloudfoundry.org/buildpacks/cnb/index.html

..however, I get this:

   Writing application metadata: /home/vcap/layers/paketo-buildpacks_ca-certificates/launch.toml <= {Labels:[] Processes:[] Slices:[] BOM:[{Name:helper Metadata:map[layer:helper names:[ca-certificates-helper] version:3.9.0] Launch:true Build:false}]}
   Paketo Buildpack for Node Engine 5.1.4
   Resolving Node Engine version
   Candidate version sources (in priority order):
   <unknown> -> ""
   Selected Node Engine version (using <unknown>): 20.18.1
   Checking if layer /home/vcap/layers/paketo-buildpacks_node-engine/node can be reused
   Checksum of dependency: sha256:c6fa75c841cbffac851678a472f2a5bd612fff8308ef39236190e1f8dbb0e567
   Checksum of layer:
   Build requirements match: false
   Launch requirements match: false
   Executing build process
   Installing Node Engine 20.18.1
   failed to fetch dependency: failed to make request: Get "https://nodejs.org/dist/v20.18.1/node-v20.18.1-linux-x64.tar.xz": dial tcp 104.20.23.46:443: i/o timeout
   ERROR: failed 'build' phase, error: exit status 1
   Error: building failed
   Exit status 234
   Cell 00b37c58-f656-4f88-8cfe-dc709198043d stopping instance 5d5d2e31-bbfd-41ee-9d85-ba9bfbcd695f
   Cell 00b37c58-f656-4f88-8cfe-dc709198043d destroying container for instance 5d5d2e31-bbfd-41ee-9d85-ba9bfbcd695f
CNBBuildFailed - cnb: building failed
FAILED

I tried to use binding as described here:
https://paketo.io/docs/howto/configuration/#dependency-mappings

# tree bindings/
bindings/
└── node-engine
    └── node
        ├── c6fa75c841cbffac851678a472f2a5bd612fff8308ef39236190e1f8dbb0e567
        └── type

unfortunately, with no luck.

My question is, if it's anyhow supported? How can I use our mirrors? or bind individual dependencies?

[krismarc]

I managed it to work with BP_DEPENDENCY_MIRROR. https://paketo.io/docs/howto/configuration/#dependency-mirrors

Yet not clear to me how and if possible to use bindings directly. Paketo instruction doesn't really provide any real examples.

[loewenstein-sap]

My question is, if it's anyhow supported? How can I use our mirrors? or bind individual dependencies?

I have never tried this, but if it works at all - you will have to provide the binding in VCAP_SERVICES form I believe. The documentation in Paketo reads like the name wouldn't matter, the type should be dependency-mapping and the content a mapping from sha256 to URI.

[krismarc]

@loewenstein-sap , could you come up with an example so I could try myself? Not sure if I fully understood properly. Do you mean as 'user-provided' service which will became part of VCAP_SERVICES?

@beyhan I believe this is something to be generally considered, supported and documented as there might be many companies behind sort of proxies and own mirrors which do not let direct downloads from the internet.

[krismarc]

My question is, if it's anyhow supported? How can I use our mirrors? or bind individual dependencies?

I have never tried this, but if it works at all - you will have to provide the binding in VCAP_SERVICES form I believe. The documentation in Paketo reads like the name wouldn't matter, the type should be dependency-mapping and the content a mapping from sha256 to URI.

This part of RFC seems to explain what you have said:
https://github.com/cloudfoundry/community/blob/8c7298337a8515d7dfae058b3bd1f88ad0eeaf95/toc/rfc/rfc-0030-add-support-for-file-based-service-binding.md#app-runtime-interfaces-wg

[loewenstein-sap]

I managed it to work with BP_DEPENDENCY_MIRROR. https://paketo.io/docs/howto/configuration/#dependency-mirrors

Yet not clear to me how and if possible to use bindings directly. Paketo instruction doesn't really provide any real examples.

I have no example of individual dependency-mappings in Cloud Foundry available, but it should be about cf cups and using the right type. The buildpacks themselves are prepared to read from file system at SERVICE_BINDING_ROOT or VCAP_SERVICES.

However, why would you not stick to BP_DEPENDENCY_MIRRORS over of individual bindings?

[beyhan]

could you come up with an example so I could try myself? Not sure if I fully understood properly. Do you mean as 'user-provided' service which will became part of VCAP_SERVICES?

Currently, CF supports only the VCAP_SERVICES way of defining service bindings. I think what @loewenstein-sap is saying that you have to define the bindings in that format because CF doesn't understand other formats. The RFC you found is about introducing another service binding option based on servicebindings.io but it is still in implementation phase.

I believe this is something to be generally considered, supported and documented as there might be many companies behind sort of proxies and own mirrors which do not let direct downloads from the internet.

That functionality is something provided by the CNBs and is not a CF feature. If you manage to get the bindings approach also working then we could think where it belongs. There is another RFC in progress which will introduce system buildpacks support for CNB. Providing offline buildpacks as system buildpacks in CF is an approach which works well for air gapped environments.

[krismarc]

I managed it to work with BP_DEPENDENCY_MIRROR. https://paketo.io/docs/howto/configuration/#dependency-mirrors
Yet not clear to me how and if possible to use bindings directly. Paketo instruction doesn't really provide any real examples.

I have no example of individual dependency-mappings in Cloud Foundry available, but it should be about cf cups and using the right type. The buildpacks themselves are prepared to read from file system at SERVICE_BINDING_ROOT or VCAP_SERVICES.

However, why would you not stick to BP_DEPENDENCY_MIRRORS over of individual bindings?

@loewenstein-sap We don't have mirrors of all possible sources and the path differs in some cases. I was thinking about building some automation which would dynamically craft service bindings depending on our own sources locations. Yet a lot of considerations ahead of me. I've just started playing with CNBs in CF 😄 Another option would be using this: https://paketo.io/docs/howto/configuration/#setting-hostname-mirrors

That functionality is something provided by the CNBs and is not a CF feature. If you manage to get the bindings approach also working then we could think where it belongs.

@beyhan, sure, thx! I'll give it a try.

There is another #965 which will introduce system buildpacks support for CNB. Providing offline buildpacks as system buildpacks in CF is an approach which works well for air gapped environments.

That's what we are currently using. We do provide our own dependencies repos, or we relay on default dependencies shipped with the buildpack, eg. with java buildpack.

[loewenstein-sap]

We don't have mirrors of all possible sources and the path differs in some cases. I was thinking about building some automation which would dynamically craft service bindings depending on our own sources locations. Yet a lot of considerations ahead of me. I've just started playing with CNBs in CF 😄 Another option would be using this: https://paketo.io/docs/howto/configuration/#setting-hostname-mirrors

Got it. https://github.com/dmikusa/binding-tool might be worth a look, although it is focusing on the pack build use case and hence creating a file system layout.

[krismarc]

We don't have mirrors of all possible sources and the path differs in some cases. I was thinking about building some automation which would dynamically craft service bindings depending on our own sources locations. Yet a lot of considerations ahead of me. I've just started playing with CNBs in CF 😄 Another option would be using this: https://paketo.io/docs/howto/configuration/#setting-hostname-mirrors

Got it. https://github.com/dmikusa/binding-tool might be worth a look, although it is focusing on the pack build use case and hence creating a file system layout.

Thanks for the tool.

I've contacted our 'mirrors' guys and they could create any mirror for me. I'd just need to provide them with the list of all sources locations. In this case, I'd need to build some sort of scrapper and probably buildpacks tomls are going to be the best source as @dmikusa does in his tool here. Eventually he could add such a function which would just list dependencies locations instead of downloading them. Unfortunately, I don't know Rust yet to contribute.

I had no time yet. However, I'll test individual service bindings via VCAP_SERVICES anyway. You never know when it might be needed with some edge cases.