Issue with Importing Data and Empty Database in OpenCRE

[sivanahamer]

Hi!
This is my first time using OpenCRE, and I am attempting to export the data available in OpenCRE. However, I am unable to import the data correctly, which prevents me from exporting it.

Steps to Reproduce:

1. Follow the installation steps as outlined in the documentation:

   docker run -p 5000:5000 ghcr.io/owasp/opencre/opencre:latest

2. Inside the container, try to import all the data using:

   python cre.py --import_external_projects

   Result:

   [========================================] 0 in 0.1s (0.00/s)
   INFO:application.utils.external_project_parsers.base_parser:imported 0 standards in 0.12843418900047254 seconds

3. Inside the container, create a directory for export:

   mkdir exportdata

4. Attempt to export the data:

   python cre.py --export exportdata

   Result:

   Folder is empty

5. Trying to manually import the data for one framework (CWE).

   python cre.py --cwe_in

   Error Message:

   /code/application/__init__.py:50: UserWarning: Flask-Caching: CACHE_TYPE is set to null, caching is effectively disabled.
     cache.init_app(app)
   ERROR:application.prompt_client.prompt_client:cannot instantiate ai client, neither OPENAI_API_KEY nor SERVICE_ACCOUNT_CREDENTIALS are set 
   Traceback (most recent call last):
     File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
       self.dialect.do_execute(
     File "/usr/local/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 941, in do_execute
       cursor.execute(statement, parameters)
   sqlite3.OperationalError: no such table: node

Additional Information:

• I also tried following the [OpenCRE API setup guide](https://github.com/OWASP/OpenCRE/blob/main/docs/my-opencre-user-guide.md) but still face the same issue.
• The database remains empty, and I am unable to import or export data correctly.

Questions:

• How can I get a minimal setup running with the imported data to export the data?
• Is an OPENAI_API_KEY or SERVICE_ACCOUNT_CREDENTIALS required? If so, how should I set them up?

[northdpole]

Thanks for the submission! Looks like a docs issue. Will look into it within a week.

[northdpole]

Update: you tried to do a couple conflicting things:

To solve the original question
To just import from upstream you don't have to do anything other than run the container. The first time it runs it will download the DB. So just running: docker run -p 5000:5000 ghcr.io/owasp/opencre/opencre:latest should be enough.

To export: the export functionality from cli is deprecated and has been removed
You can instead do one of 3 things in increasing order of complexity;

• export all cres from the api via curl <cre host>/rest/v1/cre_csv
• get a list of supported standards via curl <cre host>/rest/v1/standards then for each export any standard from the api via curl /rest/v1/standard/?format=json`
• Last you can get the whole database from the container, it's called "standards_cache.sqlite" and you can dump it with sqlite3 standards_cache.sqlite .dump
  (hint: the above also works with opencre.org, you don't need to run it locally)

What you tried to do:

• Running import from upstream twice does nothing the second time as the data should exist
• python cre.py --cwe_in is an import from CWE upstream, this requires redis and neo4j and some further setup. I wouldn't go down that path. The data on opencre.org is already imported and correlated.
• The error you got about missing tables means: there are no tables in the database, you haven't ran make migrate-upgrade first
• I am following the myopencre setup guide now and it works, can you please try again one last time?
• in general you shouldn't need to run anything inside the container yourself. Do the docs mention that you need to have a terminal inside the container? Please let me know if so, feedback is always welcomed so if it doesn't run please let me know so i can fix it.

To answer your questions:
When I run docker run -p 5000:5000 ghcr.io/owasp/opencre/opencre:latest and I wait a bit i get a database that is full. I can then export with the curl commands above

Is an OPENAI_API_KEY or SERVICE_ACCOUNT_CREDENTIALS required? If so, how should I set them up?
They are only used for running upstream opencre.org.

• OPENAI_KEY is used only for the chatbot and only for when you don't want to use Gemini.
• SERVICE_ACCOUNT_CREDENTIALS is only used for accessing Gemini

Leaving this ticket open until you report success

[sivanahamer]

Thank you! We were able to gather the data from the web page :D