Design Rucio queue status as a RFC7285 service

[yryang]

Make Rucio queue service as a cost map (end point cost service)

[yryang]

• Key indexing structure: (src, dst as attributes, and high-level key as "src+dst"
• Standard closeness (distance, cost)
• Multiple metrics that can be distributed: byes, files, queued, mbps
• Multiple measurements methodologies: 1h, 6h, ...

[yryang]

More reference: https://datatracker.ietf.org/doc/rfc8189/

[yryang]

https://datatracker.ietf.org/doc/draft-ietf-alto-performance-metrics/

[yryang]

Conduct initial design how this will change Rucio code.

[yryang]

Q1. Where is the code that generated the queue status file? Upstream of the modification
Q2. How this info is used? // downstream of the modification

[emiapwil]

How C3PO uses the JSON data: https://github.com/rucio/rucio/blob/master/lib/rucio/daemons/c3po/collectors/network_metrics.py#L31

A talk on C3PO: https://indico.cern.ch/event/1028600/contributions/4318999/attachments/2227099/3772807/Dynamic%20Data%20Placement%20in%20Rucio.pdf

on behalf of @MattSlm

[emiapwil]

https://github.com/rucio/rucio/blob/6e27de4e1894a68a6fdeffd6e703b54590336a3d/lib/rucio/daemons/conveyor/submitter.py#L111
->
https://github.com/rucio/rucio/blob/6e27de4e1894a68a6fdeffd6e703b54590336a3d/lib/rucio/daemons/conveyor/common.py#L432
->
https://github.com/rucio/rucio/blob/6e27de4e1894a68a6fdeffd6e703b54590336a3d/lib/rucio/daemons/conveyor/common.py#L464
->
https://github.com/rucio/rucio/blob/6e27de4e1894a68a6fdeffd6e703b54590336a3d/lib/rucio/daemons/conveyor/common.py#L480

What is actually invoked:

https://github.com/rucio/rucio/blob/6e27de4e1894a68a6fdeffd6e703b54590336a3d/lib/rucio/transfertool/fts3.py#L760