WIP SRUopener #682

dr0i · 2025-03-28T12:17:49Z

This is a draft and WIP.
@TobiasNx you can use it for functional testing.

Resolves #510.

TobiasNx · 2025-03-28T13:36:41Z

Nice seems to work. +1
The printed logs are a little bit esoteric:

, startRecord=1, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=1001
, startRecord=1001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=2001
, startRecord=2001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=3001
, startRecord=3001, maximumRecords=1000, istream.length=7865
urlToOpen=https://services.dnb.de/sru/zdb?query=dnb.isil%3DDE-Sol1&operation=searchRetrieve&recordSchema=MARC21plus-xml&version=1.1&maximumRecords=1000&startRecord=4001
, startRecord=4001, maximumRecords=1000, istream.length=437

TobiasNx · 2025-04-10T08:55:57Z

@dr0i is still in review?

dr0i · 2025-04-10T08:59:18Z

As we found out in #510 this PR needs a complete redesign.

Works hereby in the "stream" fashion. - add to flux-command

dr0i · 2025-06-02T14:37:44Z

@TobiasNx can you do functional tests before I go on here? Have a look at the @Description to see how it works (hint: "stream" based, i.e. other than the OAI-PMH opener works atm.)
I've added the class to flux-commands.
[edit]: and ignore the failing editorconfigChecker for now.

TobiasNx · 2025-06-04T09:54:41Z

@dr0i I tried to install the dist: https://metafacture.github.io/metafacture-documentation/docs/flux/Flux-User-Guide.html#build-from-local-distribution to try the runner for functional testing

but it runs into errors:

$ ./gradlew installDist

> Configure project :
HEAD has no annotated tags
No SCM tag found. Making a snapshot build
Feature branch found
Version is feature-510-addSruOpener-SNAPSHOT

[Incubating] Problems report is available at: file:///home/user/git/metafacture-core/build/reports/problems/problems-report.html

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.13/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

When I test the flux.sh then it outputs the following:

$ /home/user/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh
Exception in thread "main" java.lang.ExceptionInInitializerError
        at org.metafacture.runner.Flux.main(Flux.java:62)
Caused by: org.metafacture.commons.reflection.ReflectionException: Class not found: org.metafacture.io.
        at org.metafacture.commons.reflection.ReflectionUtil.loadClass(ReflectionUtil.java:70)
        at org.metafacture.commons.reflection.ObjectFactory.loadClassesFromMap(ObjectFactory.java:57)
        at org.metafacture.flux.parser.FluxProgramm.<clinit>(FluxProgramm.java:54)
        ... 1 more
Caused by: java.lang.ClassNotFoundException: org.metafacture.io.
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:594)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at org.metafacture.commons.reflection.ReflectionUtil.loadClass(ReflectionUtil.java:67)
        ... 3 more

Can you help? (I tested the current master to compare, there $ ./gradlew installDist and $ /home/user/git/metafacture-core/metafacture-runner/build/install/metafacture-core/flux.sh works)

dr0i · 2025-06-05T07:43:05Z

Ah, I accidently removed the TarReader.
Try again please.

TobiasNx · 2025-06-18T07:08:48Z

Current version stucks in an endless SRU request loop starting by 1 again after finishing all request does not matter if a total number of records is given or not:

e.g.

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10",total="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

Both result in, see that recordPosition 1 is turning up again after the expected last recordPosition 8:

<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
    </datafield>
    <datafield ind1=" " ind2=" " tag="035">
      <subfield code="a">(DE-101)042278333</subfield>
...
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">k</subfield>
      <subfield code="a">Internationaler Sozialistenkongress</subfield>
      <subfield code="0">(DE-588c)4021089-3</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>8</recordPosition></record></records><echoedSearchRetrieveRequest><version>1.1</version><query>WOE=sozialistenkongress and COD=s</query><xQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/><startRecord>6</startRecord><maximumRecords>5</maximumRecords><recordSchema>MARC21plus-xml</recordSchema></echoedSearchRetrieveRequest></searchRetrieveResponse>
<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
...
    </datafield>
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">c</subfield>
      <subfield code="a">Bern / Internationaler Sozialistenkongress &lt;1919&gt;</subfield>
      <subfield code="0">(DE-588c)4227833-8</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>1</recordPosition></record><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">1267605979</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20230329111229.0</controlfield>
    <controlfield tag="008">220908n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">1267605979</subfield>
      <subfield code="0">http://d-nb.info/gnd/1267605979</subfield>
      <subfield code="2">gnd</subfield>
...

TobiasNx

It seems that SRU opener stucks in infinite loop. See: #682 (comment)

dr0i · 2025-06-20T09:35:43Z

The inifinite loop should be fixed with b92238b, please try again @TobiasNx .

TobiasNx

Nice looks good! :) +1

TobiasNx · 2025-06-20T11:42:03Z

Current version stucks in an endless SRU request loop starting by 1 again after finishing all request does not matter if a total number of records is given or not:

e.g.

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

"https://services.dnb.de/sru/authorities"
| open-sru(recordSchema="MARC21plus-xml", query="WOE%3Dsozialistenkongress%20and%20COD%3Ds",version="1.1",maximumRecords="10",total="10")
| object-batch-log(batchsize="10")
| as-records
| write(FLUX_DIR + "result.txt")
;

Both result in, see that recordPosition 1 is turning up again after the expected last recordPosition 8:

<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
    </datafield>
    <datafield ind1=" " ind2=" " tag="035">
      <subfield code="a">(DE-101)042278333</subfield>
...
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">k</subfield>
      <subfield code="a">Internationaler Sozialistenkongress</subfield>
      <subfield code="0">(DE-588c)4021089-3</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>8</recordPosition></record></records><echoedSearchRetrieveRequest><version>1.1</version><query>WOE=sozialistenkongress and COD=s</query><xQuery xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:nil="true"/><startRecord>6</startRecord><maximumRecords>5</maximumRecords><recordSchema>MARC21plus-xml</recordSchema></echoedSearchRetrieveRequest></searchRetrieveResponse>
<?xml version="1.0" encoding="UTF-8"?><searchRetrieveResponse xmlns="http://www.loc.gov/zing/srw/"><version>1.1</version><numberOfRecords>8</numberOfRecords><records><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">042278333</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20110429135047.0</controlfield>
    <controlfield tag="008">900305n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">4227833-8</subfield>
      <subfield code="0">http://d-nb.info/gnd/4227833-8</subfield>
      <subfield code="2">gnd</subfield>
...
    </datafield>
    <datafield ind1=" " ind2=" " tag="913">
      <subfield code="S">swd</subfield>
      <subfield code="i">c</subfield>
      <subfield code="a">Bern / Internationaler Sozialistenkongress &lt;1919&gt;</subfield>
      <subfield code="0">(DE-588c)4227833-8</subfield>
    </datafield>
  </record>
</collection></recordData><recordPosition>1</recordPosition></record><record><recordSchema>MARC21plus-xml</recordSchema><recordPacking>xml</recordPacking><recordData><collection xmlns="http://www.loc.gov/MARC21/slim">
  <record type="Authority">
    <leader>00000nz  a2200000nc 4500</leader>
    <controlfield tag="001">1267605979</controlfield>
    <controlfield tag="003">DE-101</controlfield>
    <controlfield tag="005">20230329111229.0</controlfield>
    <controlfield tag="008">220908n||azznnaabn           | ana    |c</controlfield>
    <datafield ind1="7" ind2=" " tag="024">
      <subfield code="a">1267605979</subfield>
      <subfield code="0">http://d-nb.info/gnd/1267605979</subfield>
      <subfield code="2">gnd</subfield>
...

Both now logs only one object batches and only 8 records are fetched. This is good!

TobiasNx · 2025-06-20T14:40:08Z

One think that just came to my mind is that the sru opener needs the provide to provide a user agent

dr0i · 2025-06-23T09:37:54Z

Code review: @fsteeg

dr0i added 2 commits March 27, 2025 15:22

WIP SRUopener

3986e5c

WIP

1f2cb3d

github-project-automation bot added this to Metafacture Mar 28, 2025

dr0i requested a review from TobiasNx March 28, 2025 12:18

dr0i changed the title ~~WIP SRUopener (#510)~~ WIP SRUopener Mar 28, 2025

dr0i moved this to Review in Metafacture Mar 28, 2025

dr0i added Enhancement Flux Conform To Catmandu labels Mar 28, 2025

TobiasNx approved these changes Mar 28, 2025

View reviewed changes

TobiasNx mentioned this pull request Mar 31, 2025

Add SRU opener / open-sru #510

Open

Refactor WIP

c3f3ad6

dr0i force-pushed the 510-addSruOpener branch from ecd9c8c to c3f3ad6 Compare April 10, 2025 13:32

dr0i added 5 commits April 24, 2025 15:59

WIP and working

26640fa

WIP parsing working

33aeead

WIP #510 (comment)

1896c17

WIP testen

767e8d0

WIP every single output is a valid XML by itself

3dc0416

Works hereby in the "stream" fashion. - add to flux-command

dr0i force-pushed the 510-addSruOpener branch from 84d6845 to 3dc0416 Compare June 2, 2025 14:24

dr0i requested a review from TobiasNx June 2, 2025 14:38

dr0i assigned TobiasNx Jun 2, 2025

TobiasNx assigned dr0i and unassigned TobiasNx Jun 4, 2025

dr0i assigned TobiasNx Jun 5, 2025

dr0i removed their assignment Jun 5, 2025

dr0i added 2 commits June 10, 2025 14:58

Readd TarReader to flux-commands.md

54c0d69

WIP Catch Exception

ac80718

dr0i force-pushed the 510-addSruOpener branch from 4a00838 to ac80718 Compare June 10, 2025 13:02

dr0i added 2 commits June 10, 2025 16:37

WIP imporove

120f601

WIP 2

831e0fb

TobiasNx assigned dr0i and unassigned TobiasNx Jun 18, 2025

TobiasNx requested changes Jun 20, 2025

View reviewed changes

Add fallback if elements are missing in the data

b92238b

dr0i assigned TobiasNx and unassigned dr0i Jun 20, 2025

TobiasNx self-requested a review June 20, 2025 11:37

TobiasNx approved these changes Jun 20, 2025

View reviewed changes

dr0i assigned dr0i and unassigned TobiasNx Jun 23, 2025

dr0i moved this from Review to Working in Metafacture Jun 23, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

WIP SRUopener #682

WIP SRUopener #682

Uh oh!

dr0i commented Mar 28, 2025 •

edited

Loading

Uh oh!

TobiasNx commented Mar 28, 2025

Uh oh!

TobiasNx commented Apr 10, 2025

Uh oh!

dr0i commented Apr 10, 2025

Uh oh!

dr0i commented Jun 2, 2025 •

edited

Loading

Uh oh!

TobiasNx commented Jun 4, 2025 •

edited

Loading

Uh oh!

dr0i commented Jun 5, 2025

Uh oh!

TobiasNx commented Jun 18, 2025 •

edited

Loading

Uh oh!

TobiasNx left a comment •

edited

Loading

Uh oh!

dr0i commented Jun 20, 2025

Uh oh!

TobiasNx left a comment

Uh oh!

TobiasNx commented Jun 20, 2025

Uh oh!

TobiasNx commented Jun 20, 2025

Uh oh!

dr0i commented Jun 23, 2025

Uh oh!

Uh oh!

WIP SRUopener #682

Are you sure you want to change the base?

WIP SRUopener #682

Uh oh!

Conversation

dr0i commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx commented Mar 28, 2025

Uh oh!

TobiasNx commented Apr 10, 2025

Uh oh!

dr0i commented Apr 10, 2025

Uh oh!

dr0i commented Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dr0i commented Jun 5, 2025

Uh oh!

TobiasNx commented Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TobiasNx left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dr0i commented Jun 20, 2025

Uh oh!

TobiasNx left a comment

Choose a reason for hiding this comment

Uh oh!

TobiasNx commented Jun 20, 2025

Uh oh!

TobiasNx commented Jun 20, 2025

Uh oh!

dr0i commented Jun 23, 2025

Uh oh!

Uh oh!

dr0i commented Mar 28, 2025 •

edited

Loading

dr0i commented Jun 2, 2025 •

edited

Loading

TobiasNx commented Jun 4, 2025 •

edited

Loading

TobiasNx commented Jun 18, 2025 •

edited

Loading

TobiasNx left a comment •

edited

Loading