Skip to content

Commit 0b18179

Browse files
stevsmitSteven Smith
stevsmit
and
Steven Smith
authored
Adds auto-pruning docs to 3.10 (#817)
Co-authored-by: Steven Smith <[email protected]>
1 parent 90230ee commit 0b18179

9 files changed

+408
-8
lines changed

‎architecture/master.adoc

+6
Original file line numberDiff line numberDiff line change
@@ -65,3 +65,9 @@ include::modules/sizing-intro.adoc[leveloffset=+1]
6565
include::modules/sizing-sample.adoc[leveloffset=+2]
6666
include::modules/subscription-intro.adoc[leveloffset=+2]
6767
include::modules/quay-internal-registry-intro.adoc[leveloffset=+2]
68+
69+
// Quota management
70+
include::modules/quota-management-arch.adoc[leveloffset=+1]
71+
72+
//Namespace auto-pruning
73+
include::modules/namespace-auto-pruning-arch.adoc[leveloffset=+1]

‎config_quay/master.adoc

+2-1
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ include::modules/attributes.adoc[]
77

88
include::modules/config-intro.adoc[leveloffset=+1]
99
include::modules/config-disclaimer.adoc[leveloffset=+1]
10-
include::modules/config-updates-39.adoc[leveloffset=+2]
10+
include::modules/config-updates-310.adoc[leveloffset=+2]
11+
//include::modules/config-updates-39.adoc[leveloffset=+2]
1112
//include::modules/config-updates-38.adoc[leveloffset=+2]
1213
//include::modules/config-updates-37.adoc[leveloffset=+2]
1314
//include::modules/config-updates-36.adoc[leveloffset=+2]

‎images/auto-prune-policies-page.png

38 KB
Loading

‎manage_quay/master.adoc

+6-3
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ include::modules/metrics-authentication.adoc[leveloffset=+3]
105105
//include::modules/proc_manage-quay-geo-replication.adoc[leveloffset=+1]
106106

107107
include::modules/quota-management-and-enforcement.adoc[leveloffset=+1]
108-
include::modules/quota-management-arch.adoc[leveloffset=+2]
108+
//include::modules/quota-management-arch.adoc[leveloffset=+2]
109109
include::modules/quota-management-limitations.adoc[leveloffset=+2]
110110
include::modules/red-hat-quay-quota-management-configure-39.adoc[leveloffset=+2]
111111

@@ -116,16 +116,19 @@ include::modules/quota-establishment-api.adoc[leveloffset=+2]
116116
include::modules/quota-management-query-39.adoc[leveloffset=+2]
117117
include::modules/deleting-tag-permanently.adoc[leveloffset=+2]
118118

119+
//namespace auto-pruning
119120

121+
include::modules/red-hat-quay-namespace-auto-pruning-overview.adoc[leveloffset=+1]
122+
include::modules/managing-namespace-auto-pruning-policies.adoc[leveloffset=+2]
120123

121124
include::modules/georepl-intro.adoc[leveloffset=+1]
122125
include::modules/arch-georpl-features.adoc[leveloffset=+2]
123126
include::modules/georepl-prereqs.adoc[leveloffset=+2]
124-
include::modules/georepl-arch-standalone.adoc[leveloffset=+2]
127+
//include::modules/georepl-arch-standalone.adoc[leveloffset=+2]
125128
include::modules/config-ui-storage-georepl.adoc[leveloffset=+3]
126129
include::modules/georepl-deploy-standalone.adoc[leveloffset=+3]
127130
include::modules/standalone-georepl-site-removal.adoc[leveloffset=+3]
128-
include::modules/georepl-arch-operator.adoc[leveloffset=+2]
131+
//include::modules/georepl-arch-operator.adoc[leveloffset=+2]
129132
include::modules/georepl-deploy-operator.adoc[leveloffset=+3]
130133
include::modules/operator-georepl-site-removal.adoc[leveloffset=+3]
131134
include::modules/georepl-mixed-storage.adoc[leveloffset=+2]

‎modules/config-updates-310.adoc

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
:_content-type: REFERENCE
2+
[id="config-updates-310"]
3+
= Configuration updates for {productname} 3.10
4+
5+
The following sections detail new configuration fields added in {productname} 3.10.
6+
7+
[id="auto-pruner-namespace"]
8+
== Namespace auto-pruning configuration fields
9+
10+
With {productname} 3.10, deployments can be configured to automatically prune old image tags by a specified, allotted amount, or by the time in which they were created.
11+
12+
.Namespace auto-pruning configuration field
13+
|===
14+
|Field | Type |Description
15+
| **FEATURE_AUTO_PRUNE** | Boolean | When set to `True`, enables functionality related to the auto-pruning of tags.
16+
+
17+
*Default:* `False`
18+
|===

‎modules/managing-namespace-auto-pruning-policies.adoc

+243
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,243 @@
1+
:_content-type: PROCEDURE
2+
[id="managing-namespace-auto-pruning-policies"]
3+
= Managing namespace auto-pruning policies using the {productname} UI
4+
5+
Namespace auto-pruning policies are created using the {productname} UI v2. This can be done after you have configured your {productname} `config.yaml` file to enable the auto-pruning feature.
6+
7+
[NOTE]
8+
====
9+
This feature is not available when using the {productname} legacy UI.
10+
====
11+
12+
[id="configuring-namespace-auto-prune-feature"]
13+
== Configuring the {productname} namespace auto-pruning feature
14+
15+
Use the following procedure to configure your {productname} `config.yaml` file to enable the namespace auto-pruning feature.
16+
17+
.Procedure
18+
19+
* In your {productname} `config.yaml` file, add, and set, the `FEATURE_AUTO_PRUNE` environment variable to `True`. For example:
20+
+
21+
[source,yaml]
22+
----
23+
# ...
24+
FEATURE_AUTO_PRUNE: True
25+
# ...
26+
----
27+
28+
[id="creating-policy-v2-ui"]
29+
== Creating an auto-prune policy using the {productname} v2 UI
30+
31+
Use the following procedure to create an auto-prune policy using the {productname} UI v2.
32+
33+
.Prerequisites
34+
35+
* You have enabled the `FEATURE_AUTO_PRUNE` feature.
36+
37+
.Procedure
38+
39+
. Tag three sample images, for example, `busybox`, that will be pushed to the repository with auto-pruning enabled. For example:
40+
+
41+
[source,terminal]
42+
----
43+
$ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test
44+
----
45+
+
46+
[source,terminal]
47+
----
48+
$ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test2
49+
----
50+
+
51+
[source,terminal]
52+
----
53+
$ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test3
54+
----
55+
+
56+
[source,terminal]
57+
----
58+
$ podman tag docker.io/library/busybox <quay-server.example.com>/<quayadmin>/busybox:test4
59+
----
60+
61+
. Push the three sample images, for example, `busybox`, to the repository with auto-pruning enabled by entering the following commands:
62+
+
63+
[source,terminal]
64+
----
65+
$ podman push <quay-server.example.com>/quayadmin/busybox:test
66+
----
67+
+
68+
[source,terminal]
69+
----
70+
$ podman push <quay-server.example.com>/<quayadmin>/busybox:test2
71+
----
72+
+
73+
[source,terminal]
74+
----
75+
$ podman push <quay-server.example.com>/<quayadmin>/busybox:test3
76+
----
77+
+
78+
[source,terminal]
79+
----
80+
sudo podman push --tls-verify=false <quay-server.example.com>/<quayadmin>/busybox:test4
81+
----
82+
83+
. Check that there are four tags in your repository.
84+
85+
. On the {productname} UI v2, click *Organizations* in the navigation pane.
86+
87+
. Select the name of an organization that you will apply the auto-pruning feature to, for example, `test_organization`.
88+
89+
. Click *Settings*.
90+
91+
. Click *Auto-Prune Policies*. For example:
92+
+
93+
image:auto-prune-policies-page.png[Auto-Prune Policies page]
94+
95+
. Click the drop down menu and select the desired policy, for example, *By number of tags*.
96+
97+
. Select the desired number of tags to keep. By default, this is set at *20* tags. For this example, the number of tags to keep is set at *3*.
98+
99+
Click *Save*. The following alert is received: *Successfully updated auto-prune policy*.
100+
101+
.Verification
102+
103+
* Navigate to the *Tags* page of your repository. After a few minutes, the auto-pruner worker removes tags that no longer fit within the established criteria. In this example, it removes the `busybox:test` tag, and keeps the `busybox:test2`, `busybox:test3`, and `busybox:test4` tag.
104+
+
105+
After tags are automatically pruned, they go into the {productname} time machine, or the amount of time, after a tag is deleted, that the tag is accessible before being garbage collected. The expiration time of an image tag is dependent on your organization's settings. For more information, see link:https://access.redhat.com/documentation/en-us/red_hat_quay/3/html-single/manage_red_hat_quay/index#garbage-collection[{productname} garbage collection].
106+
107+
[id="creating-policy-api"]
108+
== Creating an auto-prune policy using the {productname} API
109+
110+
You can use {productname} API endpoints to manage auto-pruning policies for an organization.
111+
112+
.Prerequisites
113+
114+
* You have set `BROWSER_API_CALLS_XHR_ONLY: false` in your `config.yaml` file.
115+
* You have created an OAuth access token.
116+
* You have logged into {productname}.
117+
118+
.Procedure
119+
120+
. Enter the following `POST` command create a new policy that limits the number of tags allowed in an organization:
121+
+
122+
[source,terminal]
123+
----
124+
$ curl -X POST -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" -H "Content-Type: application/json" -d '{
125+
"method": "number_of_tags",
126+
"value": 10
127+
}' http://<quay-server.example.com>/api/v1/organization/<quayadmin>/autoprunepolicy/
128+
----
129+
+
130+
Alternatively, you can can set tags to expire for a specified time after their creation date:
131+
+
132+
[source,terminal]
133+
----
134+
$ curl -X POST -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" -H "Content-Type: application/json" -d '{
135+
"method": "creation_date",
136+
"value": "7d"
137+
}' http://<quay-server.example.com>/api/v1/organization/<quayadmin>/autoprunepolicy/
138+
----
139+
+
140+
.Example output
141+
[source,terminal]
142+
----
143+
{"uuid": "73d64f05-d587-42d9-af6d-e726a4a80d6e"}
144+
----
145+
+
146+
Attempting to create multiple policies returns the following error:
147+
+
148+
[source,terminal]
149+
----
150+
{"detail": "Policy for this namespace already exists, delete existing to create new policy", "error_message": "Policy for this namespace already exists, delete existing to create new policy", "error_type": "invalid_request", "title": "invalid_request", "type": "http://<quay-server.example.com>/api/v1/error/invalid_request", "status": 400}
151+
----
152+
153+
. Check your auto-prune policy by entering the following command:
154+
+
155+
[source,terminal]
156+
----
157+
$ curl -X GET -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" http://<quay-server.example.com>/api/v1/organization/<quayadmin>/autoprunepolicy/
158+
----
159+
+
160+
.Example output
161+
+
162+
[source,terminal]
163+
----
164+
{"policies": [{"uuid": "73d64f05-d587-42d9-af6d-e726a4a80d6e", "method": "creation_date", "value": "7d"}]}
165+
----
166+
167+
. You can delete the auto-prune policy by entering the following command:
168+
+
169+
[source,terminal]
170+
----
171+
$ curl -X DELETE -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" http://<quay-server.example.com>/api/v1/organization/<quayadmin>/autoprunepolicy/
172+
----
173+
174+
[id="creating-policy-api-current-user"]
175+
== Creating an auto-prune policy for the current user using the API
176+
177+
You can use {productname} API endpoints to manage auto-pruning policies for your account.
178+
179+
[NOTE]
180+
====
181+
The use of `/user/` in the following commands represents the user that is currently logged into {productname}.
182+
====
183+
184+
.Prerequisites
185+
186+
* You have set `BROWSER_API_CALLS_XHR_ONLY: false` in your `config.yaml` file.
187+
* You have created an OAuth access token.
188+
* You have logged into {productname}.
189+
190+
.Procedure
191+
192+
. Enter the following `POST` command create a new policy that limits the number of tags for the current user:
193+
+
194+
[source,terminal]
195+
----
196+
$ curl -X POST -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" -H "Content-Type: application/json" -d '{
197+
"method": "number_of_tags",
198+
"value": 10
199+
}' http://<quay-server.example.com>/api/v1/user/autoprunepolicy/
200+
----
201+
+
202+
.Example output
203+
+
204+
[source,terminal]
205+
----
206+
{"uuid": "8c03f995-ca6f-4928-b98d-d75ed8c14859"}
207+
----
208+
209+
. Check your auto-prune policy by entering the following command:
210+
+
211+
[source,terminal]
212+
----
213+
$ curl -X GET -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" http://<quay-server.example.com>/api/v1/user/autoprunepolicy/8c03f995-ca6f-4928-b98d-d75ed8c14859
214+
----
215+
+
216+
Alternatively, you can include the UUID:
217+
+
218+
[source,terminal]
219+
----
220+
$ curl -X GET -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" http://<quay-server.example.com>/api/v1/user/autoprunepolicy/8c03f995-ca6f-4928-b98d-d75ed8c14859
221+
{"uuid": "8c03f995-ca6f-4928-b98d-d75ed8c14859", "method": "number_of_tags", "value": 10}
222+
----
223+
+
224+
.Example output
225+
+
226+
[source,terminal]
227+
----
228+
{"policies": [{"uuid": "8c03f995-ca6f-4928-b98d-d75ed8c14859", "method": "number_of_tags", "value": 10}]}
229+
----
230+
231+
. You can delete the auto-prune policy by entering the following command. Note that deleting the policy require the UUID.
232+
+
233+
[source,terminal]
234+
----
235+
$ curl -X DELETE -H "Authorization: Bearer j8mvpgf0rP5wpbS7LCfDaAxE0NZvCmOC123456" http://<quay-server.example.com>/api/v1/user/autoprunepolicy/8c03f995-ca6f-4928-b98d-d75ed8c14859
236+
----
237+
+
238+
.Example output
239+
+
240+
[source,terminal]
241+
----
242+
{"uuid": "8c03f995-ca6f-4928-b98d-d75ed8c14859"}
243+
----

‎modules/namespace-auto-pruning-arch.adoc

+76
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
:_content-type: CONCEPT
2+
[id="namespace-auto-pruning-arch"]
3+
= Namespace auto-pruning architecture
4+
5+
For the namespace auto-pruning feature, two distinct database tables within a database schema were created: one for `namespaceautoprunepolicy` and another for `autoprunetaskstatus`. An auto-prune worker carries out the configured policies.
6+
7+
[discrete]
8+
[id="namespaceautoprunepolicy-database-table"]
9+
== Namespace auto prune policy database table
10+
11+
The `namespaceautoprunepolicy` database table holds the policy configuration for a single namespace. There is only one entry per namespace, but there is support for multiple rows per `namespace_id`. The `policy` field holds the policy details, such as `{method: "creation_date", olderThan: "2w"}` or `{method: "number_of_tags", numTags: 100}`.
12+
13+
.`namespaceautoprunepolicy` database table
14+
[cols="1a,1a,1a,1a",options="header"]
15+
|===
16+
| Field | Type |Attributes | Description
17+
18+
| `uuid` | character varying (225) | Unique, indexed | Unique identifier for this policy
19+
20+
| `namespace_id` | Integer | Foreign Key |Namespace that the policy falls under
21+
22+
| `policy` | text | JSON | Policy configuration
23+
|===
24+
25+
[discrete]
26+
[id="autoprunetaskstatus-database-table"]
27+
== Auto-prune task status database table
28+
29+
The `autoprunetaskstatus` table registers tasks to be executed by the auto-prune worker. Tasks are executed within the context of a single namespace. Only one task per namespace exists.
30+
31+
.`autoprunetaskstatus` database table
32+
[cols="1a,1a,1a,1a",options="header"]
33+
34+
|===
35+
| Field | Type |Attributes | Description
36+
| `namespace_id` | Integer | Foreign Key | Namespace that this task belongs to
37+
38+
| `last_ran_ms` | Big Integer (bigint) | Nullable, indexed | Last time that the worker executed the policies for this namespace
39+
40+
| `status` | text | Nullable | Details from the last execution task
41+
|===
42+
43+
[id="auto-prune-worker"]
44+
== Auto-prune worker
45+
46+
The following sections detail information about the auto-prune worker.
47+
48+
[id="auto-prune-task-creation"]
49+
=== Auto-prune-task-creation
50+
51+
When a new policy is created in the `namespaceautoprunepolicy` database table, a row is also created in the `autoprunetask` table. This is done in the same transaction. The auto-prune worker uses the entry in the `autoprunetask` table to identify which namespace it should execute policies for.
52+
53+
[id="auto-prune-worker-execution"]
54+
=== Auto-prune worker execution
55+
56+
The auto-pruning worker is an asynchronous job that executes configured policies. Its workflow is based on values in the `autoprunetask` table. When a task begins, the following occurs:
57+
58+
* The auto-prune worker starts on a set interval, which defaults at 30 seconds.
59+
* The auto-prune worker selects a row from `autoprunetask` with the least, or null, `last_ran_ms` and `FOR UPDATE SKIP LOCKED`.
60+
** A null `last_ran_ms` indicates that the task was never ran.
61+
** A task that hasn't been ran in he longest amount of time, or has never been run at all, is prioritized.
62+
63+
* The auto-prune worker obtains the policy configuration from the `namespaceautoprunepolicy` table.
64+
** If no policy configuration exists, the entry from `autoprunetask` is deleted for this namespace and the procedure stops immediately.
65+
66+
* The auto-prune worker begins a paginated loop of all repositories under the organization.
67+
** The auto-prune worker determines much pruning method to use based on `policy.method`.
68+
* The auto-prune worker executes the pruning method with the policy configuration retrieved earlier.
69+
** For pruning by the number of tags: the auto-pruner worker gets the number of currently active tags sorted by creation date, and deletes the older tags to the configured number.
70+
** For pruning by date: the auto-pruner worker gets the active tags older than the specified time span and any tags returned are deleted.
71+
72+
* The auto-prune worker adds audit logs of the tags deleted.
73+
74+
* The `last_ran_ms` gets updated after a row from `autoprunetask` is selected.
75+
76+
* The auto-prune worker ends.

0 commit comments

Comments
 (0)