#58: UTF-16 buffering problem ---------------------+------------------------------------------------------ Reporter: rtoy | Owner: Type: defect | Status: new Priority: major | Milestone: Component: Unicode | Version: 2012-04 Keywords: | ---------------------+------------------------------------------------------
Comment(by rtoy):
The issue is caused by the BOM (byte-order mark) that is inserted in the test file. This is ok, but when reading the file back in, the fast stream buffering code is confused because for all intents and purposes the BOM doesn't exist. But the buffering code needs to know that the BOM was there so that the internal buffers can be updated correctly.
The easiest solution is to disable the fast buffering code for utf16 and utf32. The BOM is not used for other encodings.