[Bug]: Visiting settings/admin/logging consumes CPU/RAM and triggers out of memory crash when Live is on

[Lawkss]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Visiting settings/admin/logging crashes Edge and Firefox.

You can scroll up and down a bit then CPU load increases dramatically. The size of the logfile is 26MB.

RAM usage increases by several GB in seconds

and then the browser tab crashes: "Out of Memory"

Firefox survives sometimes:

For temporary fix clearing the logfile helps.

Steps to reproduce

1. Uprade NC 28 to NC 29
2. Visit Logging page

Expected behavior

No crashes.

Installation method

Community Web installer on a VPS or web space

Nextcloud Server version

29

Operating system

RHEL/CentOS

PHP engine version

PHP 8.3

Web server

Apache (supported)

Database engine version

MariaDB

Is this bug present after an update or on a fresh install?

Upgraded to a MAJOR version (ex. 22 to 23)

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "vom-bruch.com"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "29.0.1.1",
        "overwrite.cli.url": "https:\/\/vom-bruch.com\/cloud",
        "htaccess.RewriteBase": "\/cloud",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_sendmailmode": "smtp",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "memcache.local": "\\OC\\Memcache\\APCu",
        "filelocking.enabled": true,
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 0,
            "timeout": 0
        },
        "default_phone_region": "AT",
        "simpleSignUpLink.shown": false,
        "app_install_overwrite": [
            "unsplash"
        ],
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "maintenance_window_start": 1,
        "updater.release.channel": "stable"

List of activated Apps

Enabled:
  - activity: 2.21.1
  - bruteforcesettings: 2.9.0
  - calendar: 4.7.4
  - circles: 29.0.0-dev
  - cloud_federation_api: 1.12.0
  - comments: 1.19.0
  - contacts: 6.0.0
  - contactsinteraction: 1.10.0
  - dashboard: 7.9.0
  - dav: 1.30.1
  - federatedfilesharing: 1.19.0
  - federation: 1.19.0
  - files: 2.1.0
  - files_downloadlimit: 2.0.0
  - files_pdfviewer: 2.10.0
  - files_reminders: 1.2.0
  - files_sharing: 1.21.0
  - files_trashbin: 1.19.0
  - files_versions: 1.22.0
  - firstrunwizard: 2.18.0
  - logreader: 2.14.0
  - lookup_server_connector: 1.17.0
  - mail: 3.6.1
  - nextcloud_announcements: 1.18.0
  - notes: 4.10.0
  - notifications: 2.17.0
  - oauth2: 1.17.0
  - passman: 2.4.9
  - password_policy: 1.19.0
  - photos: 2.5.0
  - privacy: 1.13.0
  - provisioning_api: 1.19.0
  - recommendations: 2.1.0
  - related_resources: 1.4.0
  - richdocuments: 8.4.2
  - richdocumentscode: 24.4.201
  - serverinfo: 1.19.0
  - settings: 1.12.0
  - sharebymail: 1.19.0
  - spreed: 19.0.1
  - support: 1.12.0
  - survey_client: 1.17.0
  - suspicious_login: 7.0.0
  - systemtags: 1.19.0
  - text: 3.10.0
  - theming: 2.4.0
  - twofactor_backupcodes: 1.18.0
  - twofactor_totp: 11.0.0-dev
  - updatenotification: 1.19.1
  - user_status: 1.9.0
  - viewer: 2.3.0
  - weather_status: 1.9.0
  - workflowengine: 2.11.0
Disabled:
  - admin_audit: 1.19.0
  - encryption: 2.17.0
  - files_external: 1.21.0
  - user_ldap: 1.20.0

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

No response

Additional info

No response

[Lawkss]

this is still happening in 29.0.3

just visiting logging page will immediately ramp up CPU fan and load.

[LM-vb]

The freezing/crashing does not happen to me any longer, since the update to 29.0.4. Can someone cross-check?
The details of an entry not showing up, see #1275, however, still happens.

[Ornanovitch]

It is still happening with NC 29.0.4 on FF on my side...

[LM-vb]

No freezes on Chrome (127.0.6533.73), Firefox (128.0) or Safari on my side. That's weird.

[baermb]

FWIW, this has been a problem for me on two different NC installations: AIO on Debian; straight install on Ubuntu. And its been happening at least since 29.0.3 and possibly before (currently: the AIO is 29.0.4 and the other is 29.0.6). I've mostly tested with Firefox 115.14.0esr and a little chrome 128.0.6613.119. The tab will freezes up either immediately or after some use, and sometimes the whole window freezes. On occasions parts of the window are clickable and parts are not, even outside of the tab.
In my case, I have access to the log files so this is less of a priority. But looking at logs in the browser is just darkly ironic.

[Ornanovitch]

I tried to analyze this problem a bit more today, and I maybe have a clue. It doesn't freeze systematically, but when it does, this is because the poll?lastReqId request asks for way too much entries (hundreds), eating all my memory. It doesn't wait for me to scroll and to limit the entries with log?offset=50&query=

[spyro1248]

I am experiencing the same bug. As usual, it appears to be nextcloud making some kind of enormous memory allocation.

This has been happening since at least (from memory NC26, and persists in the current version (29.0.7 at the time of writing).

The logging page (when it has worked) has always been incredibly slow and prone to crashing my browser (often taking down my entire laptop - I'm not rich and havent got much RAM.)

In spite of being lowish on RAM, No other website I browse is this abysmally sluggish, and I can normally have over 50 tabs open on the machine on various sites.

[WadohS]

i experienced the same problem, restart apache2.service make off the problem, but reappear between 2-3 minutes after restarting

[Lawkss]

still happening in nextcloud 30.0.1 RC1

[joshtrichards]

I tried to analyze this problem a bit more today, and I maybe have a clue. It doesn't freeze systematically, but when it does, this is because the poll?lastReqId request asks for way too much entries (hundreds), eating all my memory. It doesn't wait for me to scroll and to limit the entries with log?offset=50&query=

Is your system generating lots of new log entries every second? Are the in-browser Log reader settings set to Debug perhaps?

The poll query only asks for new queries since the last ReqId the front-end knows about. It'll poll every 10s, but the lastReqId=xxx value shouldn't change unless there have been new entries. And the entries within the response should be minimal since, at most, they'd cover a 10s period.

Those reporting this matter:

• What are your Log reader settings? Specifically:
  • Filters
  • Log level (note: it's possible for this to be a different number than you see in your config.php)
  • Time format
  • Live view
• How many new log entries are you seeing your log file typically over a one minute period?

Also, in case of interactions with browser extensions, try reproducing with extensions off (and maybe in private browsing mode) just to see if it changes behavior.

[Ornanovitch]

Thanks @joshtrichards

My log reader settings is set on the Warning level as well as the filters (Warning, Error, Fatal). The local time format is checked as well as the live view.I don't have a lot of entries over a one minute period, sometime only 2 or 3! As I said, the reader doesn't freeze systematically, but when it does it doesn't looks like a number of entries problem

[joshtrichards]

Reproduced, though I'm not sure how/what precisely triggers it at the moment. It more sort of just... started happening.

[Lawkss]

I am not generating that many entries. Sometimes non per day. It also doesnt want to display the details of log entries.

for me it always freezes almost immediately. I dont know what you mean by time format I use 24h clock system

I really hope it can be debugged.

[baermb]

Because of the above thread, I took another look at logging on two instances (AIO docker NC 29.0.4 and a stand alone NC 29.07). If I turn off Live-view/Polling in Log reader's settings, both of them seem to function without an issue. If I turn it back on again, the window freezes.
From the hand full of attempts I did, I generally needed to get into the Logging settings dialog and turn the Live-view setting off fairly quickly, before the window froze. And even after that, that window would freeze and I'd need to open a new window and go to NC's Logging again. If I successfully turned polling off, the logging screen then didn't cause any issues for me (unless I was foolish enough to turn Live-view back on to test it anyway :) ).

[denppa]

Same here, turning on live view causes a run away thread and freezes tab. Turning it off resolves the crash.

[Lawkss]

Seems fixed for me in 30.0.1 stable branch.

[denppa]

Waiting for the docker images to update, currently using NC30.0.0. Will report back once the release is made.

[Lawkss]

the crashes are now happening again. No idea why it worked for a brief while.

[joshtrichards]

Perhaps we need to introduce a delay in the live view polling.

[punxxx-me]

Still there in 30.0.2

Tested with Chrome and Firefox

[Cees2439867]

Still happens in 30.0.3 but not so fast as previous. I thought it was fixed, but no luck.

[Lawkss]

Still happens in 30.0.3