Hello,
I've updated recently Nav on debian from 5.5.2-1 to 5.6 I have an error 500 when I try to reach the nav gui website
Here is some errors on /var/log/apache2/error.log [Tue Jan 24 11:18:16.723757 2023] [mpm_prefork:notice] [pid 22855] AH00169: caught SIGTERM, shutting down [Tue Jan 24 11:23:07.716778 2023] [mpm_prefork:notice] [pid 1600] AH00163: Apache/(version removed) (Debian) mod_fcgid/(version removed) OpenSSL/1.1.1n mod_wsgi/(version removed) Python/3.7 configured -- resuming normal operations [Tue Jan 24 11:23:07.881160 2023] [core:notice] [pid 1600] AH00094: Command line: '/usr/sbin/apache2' [Tue Jan 24 11:27:21.186436 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] mod_wsgi (pid=1602): Exception occurred processing WSGI script '/opt/venvs/nav/lib/python3.7/site-packages/nav/wsgi.py'. [Tue Jan 24 11:27:21.187713 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] mod_wsgi (pid=1603): Exception occurred processing WSGI script '/opt/venvs/nav/lib/python3.7/site-packages/nav/wsgi.py'. [Tue Jan 24 11:27:21.212539 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] Traceback (most recent call last): [Tue Jan 24 11:27:21.212543 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] Traceback (most recent call last): [Tue Jan 24 11:27:21.212725 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] File "/opt/venvs/nav/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 133, in __call__ [Tue Jan 24 11:27:21.212731 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] File "/opt/venvs/nav/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 133, in __call__ [Tue Jan 24 11:27:21.212737 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] response = self.get_response(request)one
Apparently, uwsgi Proxy is only used by graphite. No WSGI or uwsgi error found in /var/log/apache2/graphite-web_error.log The uwsgi configuration has not been touched since installation of nav and the other vhosts are working.
When I downgrade nav to previous version, nav is working again and I can see updated traffic graphs (no gaps or missing graphs since the error)
Regards,
Ludovic F. Faculty of pharmacy - Strasbourg FRANCE
On Tue, 24 Jan 2023 16:04:53 +0100 (CET) "FOUR Ludovic (EXT)" ludovic.four@unistra.fr wrote:
I've updated recently Nav on debian from 5.5.2-1 to 5.6 I have an error 500 when I try to reach the nav gui website
Hi, Ludovic!
Apologies for the late response. It seems I never received this mail from the mailing list software - it's nowhere to be found in any of my folders, not spam or otherwise. I was only made aware of it by Davide Miccones recent response. It was clearly sent out on the list, though, as I can find in the web archives - and another list member was kind enough to forward me the original after my query yesterday.
Here is some errors on /var/log/apache2/error.log [Tue Jan 24 11:18:16.723757 2023] [mpm_prefork:notice] [pid 22855] AH00169: caught SIGTERM, shutting down [Tue Jan 24 11:23:07.716778 2023] [mpm_prefork:notice] [pid 1600] AH00163: Apache/(version removed) (Debian) mod_fcgid/(version removed) OpenSSL/1.1.1n mod_wsgi/(version removed) Python/3.7 configured -- resuming normal operations [Tue Jan 24 11:23:07.881160 2023] [core:notice] [pid 1600] AH00094: Command line: '/usr/sbin/apache2' [Tue Jan 24 11:27:21.186436 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] mod_wsgi (pid=1602): Exception occurred processing WSGI script '/opt/venvs/nav/lib/python3.7/site-packages/nav/wsgi.py'. [Tue Jan 24 11:27:21.187713 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] mod_wsgi (pid=1603): Exception occurred processing WSGI script '/opt/venvs/nav/lib/python3.7/site-packages/nav/wsgi.py'. [Tue Jan 24 11:27:21.212539 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] Traceback (most recent call last): [Tue Jan 24 11:27:21.212543 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] Traceback (most recent call last): [Tue Jan 24 11:27:21.212725 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] File "/opt/venvs/nav/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 133, in __call__ [Tue Jan 24 11:27:21.212731 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] File "/opt/venvs/nav/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 133, in __call__ [Tue Jan 24 11:27:21.212737 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] response = self.get_response(request)one
Apparently, uwsgi Proxy is only used by graphite. No WSGI or uwsgi error found in /var/log/apache2/graphite-web_error.log The uwsgi configuration has not been touched since installation of nav and the other vhosts are working.
Both NAV and Graphite can be hosted directly in Apache's mod_wsgi, or any other WSGI server, like for example uwsgi. NAV's config examples default to using mod_wsgi. Our howto guides for integrating Graphite with NAV presents both the mod_wsgi and the uwsgi options, so I guess you chose the latter for Graphite while sticking to mod_wsgi for NAV (which is fine, and at one point it was necessary, since NAV and Graphite needed to run on different versions of Python).
The errors from the log you posted indicate a problem with running the NAV web application itself inside Apache's mod_wsgi container (which only affects the NAV user interface. The back-end collection processes should be unaffected, and so should Graphite).
However, it seems to me like you did not provide the full traceback message from the logs, so it's hard to say what the actual error is. This could be an undiscovered error in NAV 5.6 that we aren't seing because we actually don't use mod_wsgi ourselves - we deploy both NAV and Graphite using uwsgi in production.
I could check if the NAV appliance behaves in this way, since it uses the default setup. I realize it's been a while since you tried NAV 5.6 by now, but if you still have the Apache logs from Jan 24, I would also like you to check if there are more lines of those traceback messages.
On Thu, 09 Mar 2023 11:00:49 +0100 Morten Brekkevold morten.brekkevold@sikt.no wrote:
I could check if the NAV appliance behaves in this way, since it uses the default setup.
I tried, but was unable to reproduce the problem, even when attempting an upgrade from a clean 5.5.2-1 install to 5.6.0, so I would still be dependent on more log output to make an educated guess as to what's going wrong in your or Davide's installations.
Hi Morten,
Sorry for delays. I've resolved my nav upgrade problem. In the error logs, I found this : [Wed Mar 22 [timestamp] 2023] [wsgi:error] [pid xxxx] [remote <IP xxx>:<port xxx>] pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '7.5.3-deb10u1' (package: reportbug)
The problem was the installed package reportbug (https://wiki.debian.org/reportbug) This package is automatically installed with python3
I don't know how it can trigger an error 500 (maybe add some apache module), but in my case, the problem was solved by removing this package + apt autoremove
Regards,
Ludovic F. DR10 CNRS France /Faculté de Pharmacie
----- Mail original ----- De: "Morten Brekkevold" morten.brekkevold@sikt.no À: "FOUR Ludovic / PHA" ludovic.four@unistra.fr Cc: "nav-users" nav-users@uninett.no Envoyé: Jeudi 9 Mars 2023 11:00:49 Objet: Re: [Nav-users] Error 500 when upgrading to nav 5.6
On Tue, 24 Jan 2023 16:04:53 +0100 (CET) "FOUR Ludovic (EXT)" ludovic.four@unistra.fr wrote:
I've updated recently Nav on debian from 5.5.2-1 to 5.6 I have an error 500 when I try to reach the nav gui website
Hi, Ludovic!
Apologies for the late response. It seems I never received this mail from the mailing list software - it's nowhere to be found in any of my folders, not spam or otherwise. I was only made aware of it by Davide Miccones recent response. It was clearly sent out on the list, though, as I can find in the web archives - and another list member was kind enough to forward me the original after my query yesterday.
Here is some errors on /var/log/apache2/error.log [Tue Jan 24 11:18:16.723757 2023] [mpm_prefork:notice] [pid 22855] AH00169: caught SIGTERM, shutting down [Tue Jan 24 11:23:07.716778 2023] [mpm_prefork:notice] [pid 1600] AH00163: Apache/(version removed) (Debian) mod_fcgid/(version removed) OpenSSL/1.1.1n mod_wsgi/(version removed) Python/3.7 configured -- resuming normal operations [Tue Jan 24 11:23:07.881160 2023] [core:notice] [pid 1600] AH00094: Command line: '/usr/sbin/apache2' [Tue Jan 24 11:27:21.186436 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] mod_wsgi (pid=1602): Exception occurred processing WSGI script '/opt/venvs/nav/lib/python3.7/site-packages/nav/wsgi.py'. [Tue Jan 24 11:27:21.187713 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] mod_wsgi (pid=1603): Exception occurred processing WSGI script '/opt/venvs/nav/lib/python3.7/site-packages/nav/wsgi.py'. [Tue Jan 24 11:27:21.212539 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] Traceback (most recent call last): [Tue Jan 24 11:27:21.212543 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] Traceback (most recent call last): [Tue Jan 24 11:27:21.212725 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] File "/opt/venvs/nav/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 133, in __call__ [Tue Jan 24 11:27:21.212731 2023] [wsgi:error] [pid 1603] [remote <IP removed>:55043] File "/opt/venvs/nav/lib/python3.7/site-packages/django/core/handlers/wsgi.py", line 133, in __call__ [Tue Jan 24 11:27:21.212737 2023] [wsgi:error] [pid 1602] [remote <IP removed>:55041] response = self.get_response(request)one
Apparently, uwsgi Proxy is only used by graphite. No WSGI or uwsgi error found in /var/log/apache2/graphite-web_error.log The uwsgi configuration has not been touched since installation of nav and the other vhosts are working.
Both NAV and Graphite can be hosted directly in Apache's mod_wsgi, or any other WSGI server, like for example uwsgi. NAV's config examples default to using mod_wsgi. Our howto guides for integrating Graphite with NAV presents both the mod_wsgi and the uwsgi options, so I guess you chose the latter for Graphite while sticking to mod_wsgi for NAV (which is fine, and at one point it was necessary, since NAV and Graphite needed to run on different versions of Python).
The errors from the log you posted indicate a problem with running the NAV web application itself inside Apache's mod_wsgi container (which only affects the NAV user interface. The back-end collection processes should be unaffected, and so should Graphite).
However, it seems to me like you did not provide the full traceback message from the logs, so it's hard to say what the actual error is. This could be an undiscovered error in NAV 5.6 that we aren't seing because we actually don't use mod_wsgi ourselves - we deploy both NAV and Graphite using uwsgi in production.
I could check if the NAV appliance behaves in this way, since it uses the default setup. I realize it's been a while since you tried NAV 5.6 by now, but if you still have the Apache logs from Jan 24, I would also like you to check if there are more lines of those traceback messages.
On Wed, 22 Mar 2023 12:32:46 +0100 (CET) "FOUR Ludovic (EXT)" ludovic.four@unistra.fr wrote:
Sorry for delays. I've resolved my nav upgrade problem.
Great to hear!
In the error logs, I found this : [Wed Mar 22 [timestamp] 2023] [wsgi:error] [pid xxxx] [remote <IP xxx>:<port xxx>] pkg_resources.extern.packaging.version.InvalidVersion: Invalid version: '7.5.3-deb10u1' (package: reportbug)
The problem was the installed package reportbug (https://wiki.debian.org/reportbug) This package is automatically installed with python3
I don't know how it can trigger an error 500 (maybe add some apache module), but in my case, the problem was solved by removing this package + apt autoremove
Ah, I've seen this reported before, and it appears to be a very Debian-specific problem.
The Debian package `reportbug` is a Python package that has what Python's packaging system considers to be an invalid version number, and for some reason, this triggers an error whenever the Python package `pkg_resources` is used to query about Python package resources on the system.
It isn't entirely clear to me whether the problem is with Debian or Python itself. It might be a problem that goes away once we stop supporting Python 3.7 in NAV and can replace `pkg_resources` with `importlib.metadata`, which is "the new thing" in Python.
But the only workaround for now is the one you've discovered: Uninstall the `reportbug` package.