Migrate tooling off of gs://kubernetes-jenkins

[michelle192837]

Follow-up to the control plane migration (#33350).

Tools like Kettle, TestGrid, and others read from gs://kubernetes-jenkins, but should switch to using the new bucket (gs://kubernetes-ci-logs) or something else if needed.

• Ensure results from kubernetes-jenkins are synced to kubernetes-ci-logs (believe this is already done as part of the control plane migration?)
• Migrate Kettle
• Migrate TestGrid
• Migrate other miscellaneous use
• K8s Infra Prow writes to kubernetes-ci-logs instead of kubernetes-jenkins
• Remove access permissions from kubernetes-jenkins
• Update references to kubernetes-jenkins across repos
  • kubernetes-sigs / prow
  • (any others?)
• Ensure TTL for gs://kubernetes-jenkins
  • NOTE: Leave gs://kubernetes-jenkins (and its project) around to reserve the bucket name.

/assign
/sig testing

[jbpratt]

/assign

[michelle192837]

Other updates (copying from Slack):

• Purge messages on k8s-infra-kettle subscription
• Stop the kettle deployment in kubernetes-public temporarily
• Update k8s.io deployment.yaml
  • Auto-deploy should handle it once submitted?
  • Update deployment.yaml in test-infra as well, but nothing should care about that file

Kettle is updated, looks like it is now pulling from the new subscription successfully according to logs. (I'm not sure if it's correctly acking finished.json files since I don't know if we have an example of that yet 🤔 So still confirming that)

Okay good, it's acknowledging finished.json.
I think we're good to go on the Prow change then? I'm going to let the bucket sync run one more time then stop it temporarily, then we can submit the prow change and hopefully it all works smoothly?
Run has complete and the sync job is disabled!

[michelle192837]

Kettle has had recent issues (see Slack threads in testing-ops and sig-k8s-infra)

Ref: #33628

kubernetes/k8s.io#7454 should hopefully fix this. (And I'll make the change in test-infra to match just in case, though it shouldn't change anything).

[BenTheElder]

https://cs.k8s.io/?q=kubernetes-jenkins&i=nope&files=&excludeFiles=&repos=

there's a lot more of these than I hoped, but most of them are just links in docs. a few are pulling data. a few more don't need changing and are discussing the migration.

[BenTheElder]

I manually set a matching TTL on the new GCS bucket to match the old one (90d+ Age => delete)

We still need to set this in the terraform but I wanted to make sure we started aging out those old logs now as it's already like 200 TB.

[BenTheElder]

@upodroid plans to migrate the TTL to terraform (thanks!)

[michelle192837]

Existing lifecycle on gs://kubernetes-jenkins is also 90 days:

gcloud storage buckets describe gs://kubernetes-jenkins --format="default(lifecycle_config)"
lifecycle_config:
  rule:
  - action:
      type: Delete
    condition:
      age: 90

Migration happened August 21-ish, but log bucket migration happened later (largely early October) so things haven't hit TTL yet. We'll probably need to wait until ~January.

I don't have permissions to update gs://kubernetes-jenkins (just to view), so if we want to restrict permissions on it someone else will need to do it.