Why is the wrong file being fetched for `packagesite.yaml` #4

dlangille · 2025-02-01T16:56:34Z

This is now a debugging issue.

Looking at the logs, I found this fetch in these logs:
[16:24 dvl-ingress01 dvl /var/log/freshports] % sudo grep -l 'https://pkg.freebsd.org/FreeBSD:15:amd64/latest/meta.conf' freshports.log.*
freshports.log.10
freshports.log.13
freshports.log.8
Jan 23 06:02:53 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[88001]: fetch --quiet https://pkg.freebsd.org/FreeBSD:15:amd64/latest/meta.conf
Jan 23 06:02:53 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[88019]: rm -f packagesite.txz packagesite.tar
Jan 23 06:02:53 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[88024]: fetch --quiet https://pkg.freebsd.org/FreeBSD:15:amd64/latest/packagesite.txz
Jan 23 06:02:54 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[88080]: tar -xf packagesite.txz
This would be right after the 2025-01-22 build was available on the server.

NOTE, we do not see the 'WARNING: unable to fetch ... meta.conf(from https://github.com/FreshPorts/packages-import/blob/main/fetch-extract-parse-import-one-abi.sh#L57). Which means it is processingmeta.conf`.

Originally posted by @dlangille in #615

The text was updated successfully, but these errors were encountered:

dlangille · 2025-02-01T16:59:34Z

I did a test on dvl-ingress01, looks good: it is grabbing packagesite.tzst as we want it to.

Feb  1 16:51:42 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[17238]: starting ./fetch-extract-parse-import-one-abi.sh
Feb  1 16:51:42 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[17242]: creating /var/db/freshports/packagesite/FreeBSD:15:amd64/latest
Feb  1 16:51:42 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[17254]: cd /var/db/freshports/packagesite/FreeBSD:15:amd64/latest/
Feb  1 16:51:42 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[17264]: fetch --quiet https://pkg.freebsd.org/FreeBSD:15:amd64/latest/meta.conf
Feb  1 16:51:43 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[17313]: parsing the fetch meta.conf file for pkg archive details.
Feb  1 16:51:43 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[17356]: rm -f packagesite.tzst packagesite.tar
Feb  1 16:51:44 dvl-ingress01 fetch-extract-parse-import-one-abi.sh[17368]: fetch --quiet https://pkg.freebsd.org/FreeBSD:15:amd64/latest/packagesite.tzst

dlangille · 2025-02-01T17:10:10Z

I think I know the cause: get_packagesite.txz_date

fetch-extract-parse-import-one-abi.sh is meta.conf-ware, but get_packagesite.txz_date is not.

background:

Every hour, check_repos_for_new_stuff.py is launched and checks all the repos for new stuff. It invokes get_packagesite.txz_date to get the Last-modified date.

re:

dlangille · 2025-02-01T17:49:00Z

get_packagesite.txz_date is now meta.conf aware.

Look at what is now queued at home for import.

Feb  1 17:47:45 dvl-ingress01 check_repos_for_new_stuff.py[1634]: Number of repos ready for import: 12. They are: [
'FreeBSD:13:amd64/latest : Thu, 30 Jan 2025 22:08:56 GMT', 
'FreeBSD:14:aarch64/kmod : Sat, 01 Feb 2025 11:07:42 GMT', 
'FreeBSD:14:amd64/latest : Tue, 28 Jan 2025 14:45:50 GMT', 
'FreeBSD:14:amd64/kmod : Fri, 31 Jan 2025 23:32:30 GMT', 
'FreeBSD:14:armv7/kmod : Sat, 01 Feb 2025 11:00:42 GMT', 
'FreeBSD:14:i386/kmod : Fri, 31 Jan 2025 10:12:16 GMT', 
'FreeBSD:14:powerpc/kmod : Sat, 01 Feb 2025 11:21:10 GMT', 
'FreeBSD:14:powerpc64/kmod : Sat, 01 Feb 2025 11:28:10 GMT', 
'FreeBSD:14:powerpc64le/kmod : Sat, 01 Feb 2025 11:35:07 GMT', 
'FreeBSD:15:aarch64/latest : Tue, 28 Jan 2025 19:25:54 GMT', 
'FreeBSD:15:amd64/latest : Fri, 31 Jan 2025 18:58:16 GMT', 
'FreeBSD:15:armv7/latest : Thu, 30 Jan 2025 18:20:04 GMT']

dlangille · 2025-02-01T18:04:53Z

Looking in the files downloaded, I see packagesite.tzst, as expected based on https://pkg.freebsd[.]org/FreeBSD:15:aarch64/latest/

[18:02 dvl-ingress01 dvl ~freshports/packagesite/FreeBSD:15:aarch64/latest] % ls -lt
total 36420
-rw-r--r--  1 freshports freshports  2417456 2025.02.01 17:51 packagesite.tsv
-rw-r--r--  1 freshports freshports 10023484 2025.01.28 19:25 packagesite.tzst
-rw-r--r--  1 freshports freshports 53816698 2025.01.28 19:25 packagesite.yaml
-rw-r--r--  1 freshports freshports      179 2025.01.22 22:26 meta.conf
-rw-r--r--  1 freshports freshports  7395088 2025.01.21 06:10 packagesite.txz
-rw-r--r--  1 freshports freshports      451 1970.01.01 00:00 packagesite.yaml.pub
-rw-r--r--  1 freshports freshports      256 1970.01.01 00:00 packagesite.yaml.sig

dlangille · 2025-02-01T18:13:20Z

Here, at a private URL, we have pkg-2.0.5 showing for FreeBSD:15:latest on amd64.

Packages are still being processed.

dlangille · 2025-02-01T18:34:02Z

@grahamperrin & @EHWV - I think I've fixed it. Are there any issues here?

dlangille · 2025-02-01T20:06:47Z

After fixing a kmod issue, we have these being imported. kmod is different from latest and quarterly in the directory is not the name given. It's kmods_latest_2.

Which means special-casing code:

# kmod is special on the url
if [ $package_set == "kmod" ]
then
  url_package_set="kmods_latest_2"
else
  url_package_set=$package_set
fi

This just came into the dvl website:

Feb  1 20:01:07 dvl-ingress01 check_repos_for_new_stuff.py[96509]: Number of repos ready for import: 7. They are: ['FreeBSD:14:aarch64/kmod : ', 
'FreeBSD:14:amd64/kmod : ', 
'FreeBSD:14:armv7/kmod : ', 
'FreeBSD:14:i386/kmod : ', 
'FreeBSD:14:powerpc/kmod : ', '
FreeBSD:14:powerpc64/kmod : ', 
'FreeBSD:14:powerpc64le/kmod : ']

EHWV · 2025-02-01T21:51:08Z

@grahamperrin & @EHWV - I think I've fixed it. Are there any issues here?

Thanks for the quickly implemented solution.

After fixing a kmod issue, [...]

Yes, I was was about to make a remark about that (you beat me to it); something along the line of 'if it would be worth implementing', because it's a temporary and sliding window. Indeed, you're table has be expanded for the (small) number of 'kmods' specials. Thank you for taking the additional effort.

Which means special-casing code:

# kmod is special on the url
if [ $package_set == "kmod" ]
then
  url_package_set="kmods_latest_2"
else
  url_package_set=$package_set
fi

I realise that you cannot use the FreeBSD environment (you are targeting packages beyond on what host Freshports actually happens to run of course), but (instead of the hardcoded '2' in "kmods_latest_2") perhaps consider the 'special-casing' code with some (python I presume) variable, that would be in the spririt of (copy from my forum post):

T-Aoki said:
If you're tracking quarterly, replace kmods_latest_2 to kmods_quarterly_2.

Just to be clear: the output of pkg -vv is parsed.The appropriate (=future proof) way to specify it
in a .conf file (e.g. /usr/local/etc/pkg/repos/FreeBSD.conf like shown in message #34) is:

url: pkg+https://pkg.freebsd.org/${ABI}/kmods_latest_${VERSION_MINOR}, for the latest package repositories
url: pkg+https://pkg.freebsd.org/${ABI}/kmods_quarterly_${VERSION_MINOR}, for the quarterly package repositories

I'm not sure how precisely the repositories wil change (and if) when 14.0R goes EoL. At least when 14.3-RELEASE is released, expect kmods_latest_3 to appear.

Currently, for amd64, latest, my host shows (drm-61-kmod highlighted as example):

[0-0] # date -u
Sat Feb  1 20:33:32 UTC 2025
[1-0] # pkg sea '^drm-61-kmod'
drm-61-kmod-6.1.92.1401000_3   DRM drivers modules
drm-61-kmod-6.1.92.1402000_3   DRM drivers modules
[2-0] # pkg sea '\.1402' | wc -l
      43
[3-0] #

re:
[...]
* https://github.com/FreshPorts/packages-import/blob/main/get_packagesite.txz_date

As an alternative with only one pipe, you might consider something like:

[1-0] # curl -sI https://pkg.freebsd.org/FreeBSD:14:amd64/latest/packagesite.txz | sed -n -e ' s/Last-modified: //ip'
Wed, 22 Jan 2025 03:35:49 GMT
[2-0] #

(The combination of 'ip' (or 'Ip') is standard; it works equally well with textproc/gsed/)

dlangille · 2025-02-01T22:54:56Z

I started work on kmod for the private dvl site a while back. I see I have more to do there with respect to latest and quarterly - damn, I thought I had it after today.

I'm not sure how precisely the repositories wil change (and if) when 14.0R goes EoL. At least when 14.3-RELEASE is released, expect kmods_latest_3 to appear.

Oh no, I don't have that configured at all. Several things come to mind:

how does FreshPorts know what valid VERSION_MINOR values exist for a given ABI?
data changes - might be trivial, just another package_set

[1-0] # curl -sI https://pkg.freebsd.org/FreeBSD:14:amd64/latest/packagesite.txz | sed -n -e ' s/Last-modified: //ip'
Wed, 22 Jan 2025 03:35:49 GMT

I've made that change in my working copy.

grahamperrin · 2025-02-02T09:27:50Z

#4 (comment)

… Are there any issues here? …

Attention only to the table cells that already show a 2.⋯ version, plus FreeBSD:15:i386:

no issue.

fetches

grahamperrin:~ % mkdir /tmp/616 ; cd /tmp/616
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:13:amd64/latest/All/pkg-2.0.3.pkg
pkg-2.0.3.pkg                                         9865 kB 5215 kBps    02s
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:13:i386/latest/All/pkg-2.0.4.pkg
pkg-2.0.4.pkg                                         9782 kB 5191 kBps    02s
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:14:amd64/latest/All/pkg-2.0.4.pkg
pkg-2.0.4.pkg                                           11 MB 5201 kBps    02s
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:14:i386/latest/All/pkg-2.0.4.pkg
pkg-2.0.4.pkg                                           11 MB 5452 kBps    02s
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:15:aarch64/latest/All/pkg-2.0.3.pkg
pkg-2.0.3.pkg                                           10 MB 5233 kBps    02s
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:15:amd64/latest/All/pkg-2.0.5.pkg
pkg-2.0.5.pkg                                           11 MB 5434 kBps    02s
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:15:armv7/latest/All/pkg-2.0.3.pkg
pkg-2.0.3.pkg                                           11 MB 5571 kBps    02s
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:15:i386/latest/All/pkg-2.0.4.pkg
fetch: https://pkg.freebsd.org/FreeBSD:15:i386/latest/All/pkg-2.0.4.pkg: Not Found
grahamperrin:/tmp/616 % fetch https://pkg.freebsd.org/FreeBSD:15:i386/latest/All/pkg-2.0.5.pkg
pkg-2.0.5.pkg                                           11 MB 5062 kBps    02s
grahamperrin:/tmp/616 % date
Sun  2 Feb 2025 09:23:58 GMT
grahamperrin:/tmp/616 %

Thanks

meta.conf tells us what file to fetch for a given repo update fetch-extract-parse-import-one-abi.sh was meta.conf aware already see fetch-extract-parse-import-one-abi.sh What I've done is move the meta.conf parsing into a new file: fetch-parse-meta.conf fetch-extract-parse-import-one-abi.sh & get_packagesite.txz_date have been amended to use it. Every hour, check_repos_for_new_stuff.py is launched and checks all the repos for new stuff. It invokes get_packagesite.txz_date to get the Last-modified date of the repo in question. It has been checking the now-deprecate file: packagesite.txz With pkg 2.x, meta.conf now indicates we need to fetch packagesite.tzst With this change, check_repos_for_new_stuff.py will now be getting the Last-modified date of the correct file and repo processing will be correct. re: #616 #617

dlangille · 2025-02-02T13:54:31Z

A new release: https://github.com/FreshPorts/packages-import/releases/tag/2.7.1

I'll promote this through dev, test, stage, etc.

grahamperrin · 2025-02-02T16:09:03Z

At www:

HEADS UP

We have a known problem with lists of packages - they are out of date. A fix is in the works: see https://github.com/FreshPorts/freshports/issues/615 & https://github.com/FreshPorts/freshports/issues/616

Maybe change it to mention https://github.com/FreshPorts/packages-import/issues/4 alone. Or, if the heads-up can include a clickable link, https://github.com/FreshPorts/packages-import/commit/1d2840483067c5aa09373850cca1cbccb33ff56a

Thanks

dlangille · 2025-02-02T16:27:05Z

GitHub forwards the old issues to the new issues. The fix is in production now, and the package lists update process will run at the top of the next hour. It might take a while to finish.

edit: I had to edit other announcements, so made the changes. Thanks.

dlangille · 2025-02-02T17:20:31Z

Production packages are fixed.

EHWV · 2025-02-03T12:22:48Z

Some 'pkg repository' questions remain, probably not directly impacting Freshports.

Why are (all) these outdated remote files, like on https://pkg.freebsd.org/FreeBSD:14:amd64/latest/ , still 'active' or at least present?

Why does meta.conf as in for example:
https://pkg.freebsd.org/FreeBSD:14:amd64/latest/meta.conf
does not contain a reference to data.pkg as in
https://pkg.freebsd.org/FreeBSD:14:amd64/latest/data.pkg

meta.conf contains:

version = 2;
packing_format = "tzst";
manifests = "packagesite.yaml";
data = "data";
filesite = "filesite.yaml";
manifests_archive = "packagesite";
filesite_archive = "filesite";

indicating the designated use of 'packagesite.tzst'

This seems not in line with:
(locally /usr/local/share/doc/pkg/NEWS)
https://github.com/freebsd/pkg/blob/release-1.21/NEWS#L25-L27

Changes from 1.20.99.9 to 1.20.99.10
- fix trigger validation schema
- use data.pkg instead of packagesite.pkg if available

and https://github.com/freebsd/pkg/blob/release-2.0/NEWS#L155-L158

Changes from 1.20.99.7 to 1.20.99.8
- fix a race in vulnxml fetch trigger
- add a new data.pkg which contains a fully valid json and aim at replacing
  packagesite.pkg because extensible, continue to produce packagesite.pkg

Note that pkg-update seems to use 'meta.conf' and 'data.pkg':

# pkg -v
1.21.3
# pkg update                                                                                                                     
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD has a wrong packagesite, need to re-create database
Fetching meta.conf: 100%    179 B   0.2kB/s    00:01
Fetching data.pkg: 100%   10 MiB  10.6MB/s    00:01
Processing entries: 100%
FreeBSD repository update completed. 35999 packages processed.
Updating FreeBSD-kmods repository catalogue...
pkg: Repository FreeBSD-kmods has a wrong packagesite, need to re-create database
Fetching meta.conf: 100%    179 B   0.2kB/s    00:01
Fetching data.pkg: 100%   14 KiB  14.4kB/s    00:01
Processing entries: 100%
FreeBSD-kmods repository update completed. 43 packages processed.
All repositories are up to date.
#

(The same behaviour is shown when using pkg 2.0.4 and the local pkg database requires updating.)

EHWV · 2025-02-05T16:52:00Z

As far as I'm concerned, I consider this completed.

dlangille mentioned this issue Feb 1, 2025

Listing of ports-mgt/pkg - pkg(8) - does not reflect current state of package repository? #3

Closed

dlangille self-assigned this Feb 1, 2025

dlangille added the bug Something isn't working label Feb 1, 2025

dlangille mentioned this issue Feb 1, 2025

Kernel models are VERSION_MINOR specific FreshPorts/freshports#617

Open

dlangille transferred this issue from FreshPorts/freshports Feb 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Why is the wrong file being fetched for `packagesite.yaml` #4

Why is the wrong file being fetched for `packagesite.yaml` #4

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025 •

edited

Loading

dlangille commented Feb 1, 2025 •

edited

Loading

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

EHWV commented Feb 1, 2025 •

edited

Loading

dlangille commented Feb 1, 2025

grahamperrin commented Feb 2, 2025 •

edited

Loading

dlangille commented Feb 2, 2025

grahamperrin commented Feb 2, 2025

dlangille commented Feb 2, 2025 •

edited

Loading

dlangille commented Feb 2, 2025

EHWV commented Feb 3, 2025 •

edited

Loading

EHWV commented Feb 5, 2025

Why is the wrong file being fetched for packagesite.yaml #4

Why is the wrong file being fetched for packagesite.yaml #4

Comments

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025 • edited Loading

dlangille commented Feb 1, 2025 • edited Loading

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

dlangille commented Feb 1, 2025

EHWV commented Feb 1, 2025 • edited Loading

dlangille commented Feb 1, 2025

grahamperrin commented Feb 2, 2025 • edited Loading

dlangille commented Feb 2, 2025

grahamperrin commented Feb 2, 2025

dlangille commented Feb 2, 2025 • edited Loading

dlangille commented Feb 2, 2025

EHWV commented Feb 3, 2025 • edited Loading

EHWV commented Feb 5, 2025

Why is the wrong file being fetched for `packagesite.yaml` #4

Why is the wrong file being fetched for `packagesite.yaml` #4

dlangille commented Feb 1, 2025 •

edited

Loading

dlangille commented Feb 1, 2025 •

edited

Loading

EHWV commented Feb 1, 2025 •

edited

Loading

grahamperrin commented Feb 2, 2025 •

edited

Loading

dlangille commented Feb 2, 2025 •

edited

Loading

EHWV commented Feb 3, 2025 •

edited

Loading