Vanilla 1.1 & Character Encoding

icouto

@Mark: I've upgraded to Vanilla 1.1, and am unable to see where I'm supposed to set the character encoding for my database. I see in the details for 1.1 that you stated: "Character Encoding Changes For those Vanilla users who have been having problems with characters being garbled by MySQL, you can now specify which character set MySQL should use when setting or retrieving data." I do not see how to specify the character encoding anywhere, and as a consequence, ever since I update to 1.1 the characters in my forum are horribly garbled! Please, help!!!

icouto

This preference may be unadvertently missing from the interface, as so are others, as has been reported in this discussion.

Mark, please, please, give us a fix!

All my characters are garbled, my extensions are spewing errors, it's CHAOS here!!!

Tex

Why don't you use your backup files until there is a solution?

icouto

@Tex: so, are we to assume that Mark made a mistake by releasing this version, and that we are actually supposed to go back and reinstall the previous version, even though it has reported security risks?... I would have thought that this error reported here is so major, and so glaringly obvious - going against even what Mark himself has reported we should be able to do - that there *must* be a very obvious way to fix it. Perhaps he forgot to uncomment something in one of the files, or accidentaly packaged the wrong version of a file in the final. I just find it hard to believe that Mark would have released something so problematic intentionally. This has not been his modus operandi in the past.

Tex

My characters (including some German umlauts) are fine. I just thought you might want to use your old files to keep your board running until you got a solution. Sorry if that didn't help.

I am sure Mark didn't do any mistake here.

Dinoboff

I think you need to add $Configuration['DATABASE_CHARACTER_ENCODING'] = 'utf8' in /conf/settings.php.

icouto

@Tex: thank you for your well-intentioned suggestion. Indeed, I followed it, reverting my forum back to 1.0.3. I guess the problems that I experienced with incorrect characters being displayed are due to the fact that I had already modified my forum, so that the entire system was correctly running in utf-8, as per instructions given by Max in another discussion. Indeed, if I view the source of the pages being outputted by Vanilla 1.1, there is actually no character encoding specified anywhere, and therefore the browsers will revert to the default english iso. I am guessing, that Mark intended for us to be able to specify the character encoding somewhere in the admin interface, but that somehow, that part of the interface has gone 'missing' (just like the entire "Discussion Labels" part, which used to be in 1.0.3, and no longer appears in 1.1). I guess I will wait for 1.1.1! :-)

Dinoboff

I don't think mark will fix it in 1.1.1. He make it that way so people who upgrade can still use their previous encoding. He did thought of the people who had installed Max_B patch.

Max_B will tell you what to do.

icouto

@Dino: I am certain that this was Mark's intention. However, I upgraded, having already a fully functional system, using the patches described by Max_B, and the upgrade broke my system, instead of helping it. From Mark's 'what's new' page on the new version, I get the feeling that there should be a place in the Admin interface somewhere where we should be able to specify or select the character encoding we wish to use (or are already using) both in the interface and the database. Nevertheless, I cannot find that anywhere in the interface. If you know how to make this change, please post it here, as it will be useful to others.

Mark

He already did post how to change it: http://lussumo.com/community/?CommentID=60851 Obviously it is never my intention to piss people off. But there are going to be problems for some people. I just try to make that number as small as possible and offer help to those who do run into issues. Considering that this isn't costing the users any money - only time - I would hope that they remain calm and refrain from acting out.

icouto

@Mark: I am sure we all are, indeed, very thankfull for the incredible amount of work you seem to be putting into producing such a fantastic product - and for free, too! What attracted me to Vanilla was precisely the fact that it looked well thought out, well designed, and simpler to manage. Reading the postings in this forum, it seemed to me that this was your philosophy: keeping things simple, uncomplicated for the end-user, and very modular for the programmer. That seemed to me to be a stark contrast to other bloated systems like phpBB, and it was the ease of implementation and administration that attracted me to Vanilla in the first place. Now, with this latest release, you seem to be expecting your admins to manually handle config files with a text editor - something we were not required to do before. This to me is a detriment to Vanilla, and makes it look like an unfinished product, which it isn't. The character encoding changes you are announcing for this new version sound like a great step forward for Vanilla, and I thank you for it. However, it seems to me, that the entire point of this new feature, is that it would make it easier for us, end users, to 'tame' the character encoding issues between database, controller script and interface. I expected that you would provide us with settings that would allow us to select or specify via the Admin interface what encodings we were using/wished to use. However, the lack of ANY interface control of the new feature, coupled with the lack of any direct guidelines from you as to how to handle this new feature, does not make it easier for us. From my perspective, I already had a fully- working, utf-8 compliant version of Vanilla, which did take me a lot of time and effort to sort out. This version, I would assume, would help others not have to jump through the same hoops, and have to hack the code, like I did, following Max_B's instructions. However, after installing 1.1, much to my dismay, things did not seem to be any easier or friendlier than before. Please be aware, that reading the postings above, while it may seem perfectly clear to you what someone is supposed to do to enable full UTF-8 in Vanilla 1.1(.1), let me tell you, for many of us, it isn't. Am I supposed to open the settings.php file and add the line, as described by Dinoboff? Why isn't Vanilla specifying *any* character encoding at all in its html files? Will that force a character encoding for the html as well as for the database queries? Do I have to change anything else, anywhere else? At the very least, basic, clear instructions are needed. Unfortunately, the lack of interface and implementation documentation makes this feel more like a hack, than the well thought-out and much discussed feature that I believe it really is. It would have been easy enough for you to put a text field or drop-down menu in the "Application Settings", and make it easy, helpful, and looking like a finished product. This would give this feature the polished look that it deserves, and would do justice to all your effort and thinking that went into it.

Max_B

@icouto: I was not "aboard" to try help you in time but dino and mark did answer. So:

- Yes, the utf-8 feature in 1.1 asks for a "manual" text edit of settings. This is anyway a step forward, a setting is more comfortable than a core patch.
Add the line to your conf/settings.php file.
This is not big deal IMO, if it's documented. I promised to write a wiki page this week, and I'll put it in there.

- This release was made to face a security hole and improved extensions management. Languages and encoding facility are not listed on release improvement.

- Don't mix MySQL encoding setting and (x)html encoding. Vanilla files include a header.php which issues a charset information and in such respect, conforms to standard. XHTML default to utf-8 by specs so when (if) we serve files as application/xhtml+xml there is no need to encoding information.

icouto

@Max: thank you for your feedback, and thank you for putting in the time to put together the how-to page on the wiki! Yes, this is certainly needed! I agree with you, that the new version *does* seem to be a big step forward, as far as the character encoding issues are concerned. Personally, I am not too scared of making a manual edit in a config file - this is certainly a lot 'cleaner' than what we had to do before. Nevertheless, please note, that Mark DOES list "character encoding changes" in the list of improvements: "Character Encoding Changes For those Vanilla users who have been having problems with characters being garbled by MySQL, you can now specify which character set MySQL should use when setting or retrieving data." And also, the problem I experienced was not *only* in the database query. It was also in the xhtml header. The header was no longer outputting "charset="utf-8". If I viewed the source of my forum home page, for instance, and on that source did a search for the string "utf-8", it came up with nothing. The charset was not being specified, and as a consequence, the browser was defaulting to english iso.

Mark

The charset is specified as a configuration setting now as well. The default is utf-8. Look for this in appg/settings.php:

$Configuration['CHARSET'] = 'utf-8';

I'm not arguing that the documentation is lacking, but this is an open-source community, and I made a documentation wiki for a reason. If at any time you want to jump in and help out with some documentation, feel free to do so. You've got to stop thinking of this like a customer service forum and start thinking of it like a community of people who help each other.

Max_B

About (x)html encoding information.
Icouto, you didn't get my point about this. Looking at the source of the displayed page does NOT show you headers, but meta. These are two different ways for setting encoding. If you want to track headers, you need a dedicated tool. Or look at the php source. Here are line 19-20 of headers.php (1.1.1) : // PROPERLY ENCODE THE CONTENT header ('content-type: text/html; charset='.$Configuration['CHARSET']);
Meta is more portable, a page saved locally will display correctly, but it is optional for utf-8 encoding in xhtml. Header is output to ensure that the file will be correctly displayed when served as text/html.

icouto

@Max_B: my pages are not displaying correctly in Vanilla 1.1.2, and I have not changed the default characterset as noted by Mark in his posting above. I have tried copying the line "$Configuration['CHARSET'] = 'utf-8';" from appg/settings.php into my conf/settings.php, hoping that perhaps it was just a matter of the configuration not being read. But that did not help the least. My browser (Safari) is displaying the pages with a 'default' character set - which for Safari is ISO Latin-1. However, I can force Safari to re-interpret the page as UTF-8, and when I do so, the characters *still* do NOT display properly - ie., all my accented characters show as "?". I know that my database is perfectly fine - everything in UTF-8, and I can see that with phpMyAdmin. I am guessing that I may have a problem in the php/sql query, and/or on the html interface side (while Vanilla 1.0.3 output 'charset' attributes in its pages, version 1.1.2 does not - if I do a search for "charset" on the source of any page in my forum, I get no results. Are there alterations to the core needed? Can anyone give me a suggestion?

icouto

I have now tried adding the following line to conf/settings.php, as well: $Configuration['DATABASE_CHARACTER_ENCODING'] = 'utf-8'; Now my conf/settings.php file has the 2 following lines at the end: $Configuration['CHARSET'] = 'utf-8'; $Configuration['DATABASE_CHARACTER_ENCODING'] = 'utf-8'; And still, all my accented characters are showing as "?". Can anyone suggest anything I might try?

icouto

Ok, after much sweating and desperation, I've managed to fix the problem. In the conf/settings.php file, the character encoding of the database has to be set like this: $Configuration['DATABASE_CHARACTER_ENCODING'] = 'utf8'; Note that it does NOT have a hyphen between utf and 8. Simply removing this hyphen made the entire system work in Vanilla 1.1.2. Those who are updating the manual, please make a note of this: it seems that in order for php to query your database using utf-8, you must add the line above to your conf/settings.php file. Mark: is this a bug? If the rest of Vanilla is now set to automatically default to utf-8, shouldn't the queries default to that as well? (I noticed that the default setting for DATABASE_CHARACTER_ENCODING is "").

Max_B

@Icouto:
all your fighting should have been saved if you read the wiki page I wrote on the matter after your first issues when switching to 1.1.
The spelling of different utf-8 case is also outlined.
You sort it, brave alone, but a simple search on the community forum would have made the trick.