Text encoding

[rowan]

Recently I've noticed text such as this in posts
"It´s the *FEEL* of the N64 games you loose in the process of emulation."

The problem's after the first 2 characters, "´" - during some upgrade, did UTF-8 text get incorrectly decoded as ISO8859-1, Windows-1252 or similar?

[rowan]

A better example is this thread
http://www.yakyak.org/viewtopic.php?f=20&t=63783

The thread title "Fjölnir" is correct but the subject has mojibake "Fjölnir" and there's plenty more in the code sample & subsequent posts

[Muttley]

Looks fine here (Chrome)

[pogo]

Looks fine here too (Safari). I was able to reproduce your problem by changing the text encoding (in Safari's View menu) from 'Default' (i.e. whatever the server tells the browser) to 'Western (ISO Latin I)'. So check your browser isn't similarly overriding the default. If you can't fix it as e.g. you're using an old version of IE consider upgrading to a more modern browser. It's probably this line of the HTML that your browser is ignoring:

Code: Select all

<meta http-equiv="content-type" content="text/html; charset=UTF-8">

[rowan]

How utterly bizarre. Thanks for the answers. No I'm not overriding any encoding, it's on automatic & even tried to force UTF-8 with browser setting.
I'm using Opera 12.16 and can also replicate in IE 8.

I'll give it a bash from another machine with same version of Opera.

edit: In fact just for the hell of it let me stick some chars here..

ë
©
你好

edit2: Yep, they're all fine, what the bejeezus is going on with my machine??

If I view source for this thread, those non-ASCII chars I just put in this post are shown exactly as you see them above, not as multiple-incorrect-chars-per-character.
If I view source for the thread I linked to, it shows multiple-incorrect-chars-per-character. That suggests to me that it's what I originally suspected / asked about in original post.
I'm guessing Safari & Chrome are somehow patching over this?

[pogo]

OK, looking at the thread again:

http://www.yakyak.org/viewtopic.php?f=20&t=63783

It's broken. Not the encoding but the content. From the page source here's the thread heading

Code: Select all

<a href="./viewtopic.php?f=20&t=63783">Fjölnir</a>

But here's the first post heading

Code: Select all

<a href="#p1107304">Fjölnir</a>

Which mirrors what I see with default or UTF-8 text encoding (If I switch to Latin 1 it get even funkier. The thread heading becomes Fjölnir. The post heading becomes Fjölnir)

When you create a new thread there's no option to post different things to the thread heading and post heading. So it looks like the BBS software is doing different things with them; and as one is correct while the other is what happens when UTF-8 is mistransformed into Latin 1 it looks like that's what's happening.

[rowan]

Yes. Sorry I didn't mean the encoding of the webpage, but the bytes in the database itself & I'm not going mad.
I'm guessing somehow they got hosed during some DB upgrade / migration / backup+restore. It's something of a massive time sink dealing with this stuff at work right at the moment & I'm over-sensitive to noticing it....

[pogo]

I've just tried testing it here, creating another thread here, but could not reproduce it. Very odd. I don't know if there's something different about how these things are handled in the programming forum.

[Little stoned kiwi]

My browser does the same (Chrome on Debian), but not often enough for it to irritate me.

There's a stackoverflow thread I found interesting: http://stackoverflow.com/questions/2477 ... instead-of
A bit down there's talk about encoding used in a database and the encoding used for entering data in it. Perhaps that's part of the puzzle?

[rowan]

No, it's not just Programming, it's in other forums.

I'm guessing at some point there was a backup or export process which dumps the DB to a text file. And that text file was written in UTF-8, then it was restored / imported to a DB, reading the text file as an 8bit encoding such as ISO8859-1 or Win1252.

[pogo]

Ah, another thought. That programming thread is very old, almost seven years old. So whatever problem there was is presumably fixed now. That's broken data in the database which is very unlikely to get fixed. Someone could manually repair the heading but the source code is probably unfixable unless someone has an undamaged copy to hand.

[rowan]

Yeah - once the data is read in incorrectly with the multiple characters encoded as correct UTF-8 it becomes very hard to know whether the character sequence "ö" really is "ö" or was supposed to be "ö"

[rowan]

That's broken data in the database which is very unlikely to get fixed.

Ooh, hmm hrr.. Just had a thought after a few rums and managed to turn the source code from that thread into this in a couple of seconds

Code: Select all

;;bottles.fjo
;;Dagur Páll Ammendrup <dagurp at heimsnet dot is>
;;Written on January 10th 2004

"bottles" < main
{
    main ->
        stef(;)
        staðvær i,b
        stofn
            fyrir( i := 100; i > 1; i := i-1) lykkja
                bottles(;i-1),
                skrifastreng(;" on the wall, "),
                bottles(;i-1),
                skrifastreng(;",\ntake one down, pass it around, \n"),
                bottles(;i-2),
                skrifastreng(;" on the wall.\n"),
            lykkjulok,
        stofnlok
}
*
{
    bottles ->
        stef(;n)
        stofn
            val n úr
            kostur 0 þá
                skrifastreng(;"no more bottles"),
            kostur 1 þá
                skrifastreng(;"1 bottle of beer"),
            annars
                skrifa(;n),
                skrifastreng(;" bottles of beer"),
            vallok
        stofnlok
}
*
"GRUNNUR"
;

Hello fixed data....

[JQW]

The affected posts date back to when this site was running phpBB 2.0, which didn't support UTF-8. Although the upgrade to phpBB 3.0 added support, the conversion script most likely didn't convert everything cleanly.

[gumboot]

Looks like the board doesn't support anything outside of BMP. eg.: http://www.alanwood.net/unicode/shavian.html or http://www.fileformat.info/info/unicode ... ertest.htm

Can preview, cannot post.