)]}'
{
  "commit": "0716ae21a1e7ab6b4ef73428c0c8fff49685d057",
  "tree": "3793e47c1eb0a11f50d1b8da8bbf3d0d06e2eccb",
  "parents": [
    "6549385d53d9efd546f4b35f6992e6cc9f552657"
  ],
  "author": {
    "name": "Anish Athalye",
    "email": "me@anishathalye.com",
    "time": "Sun Dec 08 11:18:04 2019 -0500"
  },
  "committer": {
    "name": "Tina Müller",
    "email": "cpan2@tinita.de",
    "time": "Fri Dec 20 20:38:46 2019 +0100"
  },
  "message": "Fix reader for Unicode code points over 0xFFFF (#351)\n\nThis patch fixes the handling of inputs with Unicode code points over\r\n0xFFFF when running on a Python 2 that does not have UCS-4 support\r\n(which certain distributions still ship, e.g. macOS).\r\n\r\nWhen Python is compiled without UCS-4 support, it uses UCS-2. In this\r\nsituation, non-BMP Unicode characters, which have code points over\r\n0xFFFF, are represented as surrogate pairs. For example, if we take\r\nu\u0027\\U0001f3d4\u0027, it will be represented as the surrogate pair\r\nu\u0027\\ud83c\\udfd4\u0027. This can be seen by running, for example:\r\n\r\n    [i for i in u\u0027\\U0001f3d4\u0027]\r\n\r\nIn PyYAML, the reader uses a function `check_printable` to validate\r\ninputs, making sure that they only contain printable characters. Prior\r\nto this patch, on UCS-2 builds, it incorrectly identified surrogate\r\npairs as non-printable.\r\n\r\nIt would be fairly natural to write a regular expression that captures\r\nstrings that contain only *printable* characters, as opposed to\r\n*non-printable* characters (as identified by the old code, so not\r\nexcluding surrogate pairs):\r\n\r\n    PRINTABLE \u003d re.compile(u\u0027^[\\x09\\x0A\\x0D\\x20-\\x7E\\x85\\xA0-\\uD7FF\\uE000-\\uFFFD]*$\u0027)\r\n\r\nAdding support for surrogate pairs to this would be straightforward,\r\nadding the option of having a surrogate high followed by a surrogate low\r\n(`[\\uD800-\\uDBFF][\\uDC00-\\uDFFF]`):\r\n\r\n    PRINTABLE \u003d re.compile(u\u0027^(?:[\\x09\\x0A\\x0D\\x20-\\x7E\\x85\\xA0-\\uD7FF\\uE000-\\uFFFD]|[\\uD800-\\uDBFF][\\uDC00-\\uDFFF])*$\u0027)\r\n\r\nThen, this regex could be used as follows:\r\n\r\n    def check_printable(self, data):\r\n        if not self.PRINTABLE.match(data):\r\n            raise ReaderError(...)\r\n\r\nHowever, matching printable strings, rather than searching for\r\nnon-printable characters as the code currently does, would have the\r\ndisadvantage of not identifying the culprit character (we wouldn\u0027t get\r\nthe position and the actual non-printable character from a lack of a\r\nregex match).\r\n\r\nInstead, we can modify the NON_PRINTABLE regex to allow legal surrogate\r\npairs. We do this by removing surrogate pairs from the existing\r\ncharacter set and adding the following options for illegal uses of\r\nsurrogate code points:\r\n\r\n- Surrogate low that doesn\u0027t follow a surrogate high (either a surrogate\r\n  low at the start of a string, or a surrogate low that follows a\r\n  character that\u0027s not a surrogate high):\r\n\r\n    (?:^|[^\\uD800-\\uDBFF])[\\uDC00-\\uDFFF]\r\n\r\n- Surrogate high that isn\u0027t followed by a surrogate low (either a\r\n  surrogate high at the end of a string, or a surrogate high that is\r\n  followed by a character that\u0027s not a surrogate low):\r\n\r\n    [\\uD800-\\uDBFF](?:[^\\uDC00-\\uDFFF]|$)\r\n\r\nThe behavior of this modified regex should match the one that is used\r\nwhen Python is built with UCS-4 support.",
  "tree_diff": [
    {
      "type": "modify",
      "old_id": "b2f10b09123686b0aa9dbaab37406906511deba7",
      "old_mode": 33188,
      "old_path": "lib/yaml/reader.py",
      "new_id": "4b377d61e29b35544933513867252f8c4ccc779b",
      "new_mode": 33188,
      "new_path": "lib/yaml/reader.py"
    }
  ]
}