Hi,
upgraded to 3.6.0 yesterday - it looks great!
But there is a small problem: it seems mcc won't handle norwegian characters in port descriptions.
2010-10-09 09:35:58,708: [ERROR] nav.mcc.utils - Could not encode KVM Str�mskinne 1 to latin-1: 'latin-1' codec can't encode character u'\ufffd' in position 7: ordinal not in range(256)
I don't really care about the description beeing correctly displayed, but mcc stops after this and no more cricket config is generated.
Looks like this happens when convert_unicode_to_latin1 in nav/mcc/utils.py returns the unicode object instead of a latin-1 string.
My workaround is to just return an empty string instead of the unicode object.
Thanks for all the work you guys put into NAV, truly exceptional software solving everyday problems for a lot of us!
Cheers,
-Sigurd
On Sat, Oct 09, 2010 at 10:40:21AM +0200, Sigurd Mytting wrote:
upgraded to 3.6.0 yesterday - it looks great!
Thanks :)
But there is a small problem: it seems mcc won't handle norwegian characters in port descriptions.
2010-10-09 09:35:58,708: [ERROR] nav.mcc.utils - Could not encode KVM Str�mskinne 1 to latin-1: 'latin-1' codec can't encode character u'\ufffd' in position 7: ordinal not in range(256)
The unicode character U+FFFD is not a Norwegian character. It's description is "Replacement character", and for me it renders like a diamond with a question mark in it. It has no equivalent character in the ISO-8859-1 (Latin-1) character set, and that is why the encoding fails. Cricket's configuration will only accept Latin-1 characters.
I don't really care about the description beeing correctly displayed, but mcc stops after this and no more cricket config is generated.
Since this is only for display purposes, I think a more elegant solution would be to just ignore characters that can't be encoded as latin-1. This can easily be achieved by replacing encode('latin-1') with encode('latin-1', 'ignore').
Could you please report this bug at https://bugs.launchpad.net/nav/+filebug ?
On 2010-10-11 11:47, Morten Brekkevold wrote:
Could you please report this bug at https://bugs.launchpad.net/nav/+filebug ?
ACK.
I'm also getting a lot of errors in ipdevpoll.log, just one examle:
2010-10-11 19:46:43,686 [ERROR] [nav.ipdevpoll.jobs.jobhandler.inventory.(12a-sw-vts.vegvesen.no)] Caught exception during save. Last object = Interface(netbox= Netbox(id=351, type=NetboxType(id=20, vendor=Vendor(id=u'cisco'), name=u'catalyst37xxStack', sysobjectid=u'1.3.6.1.4.1.9.1.516', cdp=True, tftp=True, cs_at_vlan=True, chassis=True, description=u'Catalyst 3750'), device=Device(id=370, serial ='FOC1227W4M0', hardware_version='V05', firmware_version='12.2(35)SE5', software_version='12.2(35)SE5'), sysname='12a-sw-vts.vegvesen.no'), ifindex=10117L, ifname='Gi1/0/17', ifdescr='GigabitEthernet1/0/17', iftype=6, speed=10.0, ifphysaddress='00:22:56:34:9b:11', ifadminstatus=1, ifoperstatus=1, ifconnectorpresent=True, ifalias='Temp Overv\xe5ker', baseport=17L, vlan=1, trunk=False, duplex='h', gone_since=None). Last model: Gi1/0/17 at 12a-sw-vts Traceback (most recent call last): File "/var/lib/python-support/python2.5/nav/ipdevpoll/jobs.py", line 334, in perform_save obj_model.save() File "/var/lib/python-support/python2.5/django/db/models/base.py", line 311, in save self.save_base(force_insert=force_insert, force_update=force_update) File "/var/lib/python-support/python2.5/django/db/models/base.py", line 383, in save_base result = manager._insert(values, return_id=update_pk) File "/var/lib/python-support/python2.5/django/db/models/manager.py", line 138, in _insert return insert_query(self.model, values, **kwargs) File "/var/lib/python-support/python2.5/django/db/models/query.py", line 894, in insert_query return query.execute_sql(return_id) File "/var/lib/python-support/python2.5/django/db/models/sql/subqueries.py", line 309, in execute_sql cursor = super(InsertQuery, self).execute_sql(None) File "/var/lib/python-support/python2.5/django/db/models/sql/query.py", line 1734, in execute_sql cursor.execute(sql, params) DataError: invalid byte sequence for encoding "UTF8": 0xe56b65 HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
Is there any way I can avoid putting non latin-1 characters into the database as a workaround?
Cheers,
-Sigurd
On Mon, Oct 11, 2010 at 07:59:41PM +0200, Sigurd Mytting wrote:
I'm also getting a lot of errors in ipdevpoll.log, just one examle:
[snip]
"/var/lib/python-support/python2.5/django/db/models/sql/query.py", line 1734, in execute_sql cursor.execute(sql, params) DataError: invalid byte sequence for encoding "UTF8": 0xe56b65 HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
Is there any way I can avoid putting non latin-1 characters into the database as a workaround?
Well, this error has avoided it for you ;-)
Apparently, you've set the description of Gi1/0/17 on 12a-sw-vts to "Temperatur Overvåker", using a latin-1 encoding.
Mostly, ipdevpoll assumes everything it collects is either ASCII or decodeable as UTF-8. For hardwired text strings, ASCII will almost always be correct, but for user-configured values it should probably try multiple encodings before giving up. This error message says the string could not be saved to the database, since it's not UTF-8, and that ensures that none of 12a-sw-vts' data will be saved at all.
Please report this to the bugtracker, and we'll try to fix it in 3.6.1 (scheduled next week, but possibly delayed over the weekend due to some saved-up vacation time).
On Tue, 12 Oct 2010 09:40:27 +0200, Morten Brekkevold morten.brekkevold@uninett.no wrote:
On Mon, Oct 11, 2010 at 07:59:41PM +0200, Sigurd Mytting wrote:
I'm also getting a lot of errors in ipdevpoll.log, just one examle:
[snip]
"/var/lib/python-support/python2.5/django/db/models/sql/query.py", line 1734, in execute_sql cursor.execute(sql, params) DataError: invalid byte sequence for encoding "UTF8": 0xe56b65 HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
Is there any way I can avoid putting non latin-1 characters into the database as a workaround?
Well, this error has avoided it for you ;-)
Yeah, I kind of got that after poking around in the code after I'd sent the reply ;-) What I should have asked is how to remove all non ASCII at the time data is fetched from the box.
Apparently, you've set the description of Gi1/0/17 on 12a-sw-vts to "Temperatur Overvåker", using a latin-1 encoding.
It's almost unavoidable when having more than 800 devices on 240 different locations all over the country and more than 30 people doing this their own way. I would of course never even dream of using anything other than ASCII (one of the perks with growing up with the DOS codepage-hell in the 1980s.. ;)
Please report this to the bugtracker, and we'll try to fix it in 3.6.1 (scheduled next week, but possibly delayed over the weekend due to some saved-up vacation time).
Reported.
Cheers,
-Sigurd
-
Morten Brekkevold -
Sigurd Mytting