Lighthouse has a new layout. Prefer the old one? Return to the old layout, and switch back any time from the link at the top of each page.

libplist utf-8 conversion bugginess

#322

Environment:
Testing on osx snow leopard using libplist-1.10, also tested on libplist-0.8

Problem:

Certain 'apparently legal' utf-8 sequences are not properly converted by libplist. These sequences convert correctly using apple's plutil -convert xml1, but not using plistutil (or the c library directly -- tested via python and ruby bindings)

The resulting plist conversion does not work when parsing by other apple tools.

To reproduce:

See the attached files. sample.bin.plist (original binary src plist) and sample.xml.plist (converted using apple's plutil).

Here's a plist dump of the sample

{
    ProblematicValue = "\Ud83c\Udf1fCute utf-8 art\Ud83c\Udf1f";
}

Try converting sample.bin.plist to xml with plistutil and examining the output with apples plutil:

$ ./plistutil-1.10 -i sample.bin.plist -o sample.lpl.plist
$ plutil -p sample.bin.plist 
{
  "ProblematicValue" => "🌟Cute utf-8 art🌟"
}
$ plutil -p sample.xml.plist 
{
  "ProblematicValue" => "🌟Cute utf-8 art🌟"
}
binbag:plistutil monti$ plutil -p sample.lpl.plist 
sample.lpl.plist: Unable to convert string to correct encoding

$ diff sample.xml.plist sample.lpl.plist 
6c6
<    <string>🌟Cute utf-8 art🌟</string>
---
>    <string>??????Cute utf-8 art??????</string>

Note, you can't see it in this forum, but the string "Cute utf-8 art" is enclosed on either side by a utf-8 star icon. using the sequence: "\Ud83c\Udf1f"

Reported by Eric Monti · March 19th, 2013 @ 09:18 PM

State: resolved
Milestone: 1.2.0 Release
Assigned to: Nikias Bassen Nikias Bassen

Activity

  1. Eric Monti
    Eric Monti

    more context: found the offending codepage -- also note: surrounding values also exhibit the same problem

    http://www.utf8-chartable.de/unicode-utf8-table.pl?start=127775

    March 19th, 2013 @ 09:47 PM

  2. Martin S.
    Martin S.
    • Tag set to conversion, encoding, libplist, utf-8
    • Assigned user set to Nikias Bassen

    I was able to reproduce the problem, however this is related to libxml2 freaking out on the characters:

    parser error : Char 0xD83C out of allowed range
    parser error : PCDATA invalid Char value 55356
    parser error : Char 0xDF1F out of allowed range
    parser error : PCDATA invalid Char value 57119
    
    Not sure yet who is wrong or right here.

    July 1st, 2013 @ 04:21 PM

  3. Shane Garrett
    Shane Garrett

    Ran across the same issue, it looks like the problem is with the plist_utf16_to_utf8() function. It doesn't correctly handle UTF-8 encoding of UTF-16 surrogate pairs, which the sample plist has.

    October 2nd, 2013 @ 11:22 PM

  4. Martin S.
    Martin S.
    • State changed from new to resolved
    • Milestone set to 1.2.0 Release

    The issue is fixed in git HEAD master of libplist.
    Downstream ticket:
    https://github.com/libimobiledevice/libplist/pull/2

    October 19th, 2013 @ 10:10 AM

Please Sign in or create a free account to add a new ticket.

With your very own profile, you can contribute to projects, track your activity, watch tickets, receive and update tickets through your email and much more.