{"vulnerability": "cve-2021-42694", "sightings": [{"uuid": "5db726da-c11d-4d78-aeea-858f65e1e074", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42694", "type": "published-proof-of-concept", "source": "https://t.me/GithubRedTeam/793", "content": "GitHub\u76d1\u63a7\u6d88\u606f\u63d0\u9192\uff01\uff01\uff01\n\n\u66f4\u65b0\u4e86\uff1aCVE-2021\n\u63cf\u8ff0\uff1aGenerate malicious files using recently published homoglyphic-attack (CVE-2021-42694)\nURL\uff1ahttps://github.com/js-on/CVE-2021-42694", "creation_timestamp": "2021-11-02T20:44:08.000000Z"}, {"uuid": "0982d0e2-9f14-4ff9-81c4-2d4073a6f6f6", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42694", "type": "published-proof-of-concept", "source": "https://t.me/hacker_trick/389", "content": "Applications that reproduce #CVE-2021-22119\nhttps://github.com/mari6274/oauth-client-exploit\n\nGenerate malicious files using recently published homoglyphic-attack #CVE-2021-42694\nhttps://github.com/js-on/CVE-2021-42694\n\nmetasploit script #poc about #CVE-2021-36260\nhttps://github.com/TaroballzChen/CVE-2021-36260-metasploit", "creation_timestamp": "2021-11-03T17:55:14.000000Z"}, {"uuid": "8382cc67-95c6-4107-b8b9-94f988ce18b6", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42694", "type": "published-proof-of-concept", "source": "https://t.me/true_secator/2278", "content": "\u0423\u0447\u0435\u043d\u044b\u0435 \u0438\u0437 \u041a\u0435\u043c\u0431\u0440\u0438\u0434\u0436\u0441\u043a\u043e\u0433\u043e \u0443\u043d\u0438\u0432\u0435\u0440\u0441\u0438\u0442\u0435\u0442\u0430 \u0432 \u0421\u043e\u0435\u0434\u0438\u043d\u0435\u043d\u043d\u043e\u043c \u041a\u043e\u0440\u043e\u043b\u0435\u0432\u0441\u0442\u0432\u0435 \u0440\u0430\u0441\u043a\u0440\u044b\u043b\u0438 \u043e\u0431\u043d\u0430\u0440\u0443\u0436\u0435\u043d\u043d\u044b\u0439 \u0435\u0449\u0435 25 \u0438\u044e\u043b\u044f \u0432\u0435\u043a\u0442\u043e\u0440 \u0430\u0442\u0430\u043a\u0438 Trojan Source, \u043f\u043e\u0437\u0432\u043e\u043b\u044f\u044e\u0449\u0438\u0439 \u0432\u043d\u0435\u0434\u0440\u044f\u0442\u044c \u0443\u044f\u0437\u0432\u0438\u043c\u043e\u0441\u0442\u0438 \u0432 \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u0439 \u043a\u043e\u0434 \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c\u043d\u043e\u0433\u043e \u043e\u0431\u0435\u0441\u043f\u0435\u0447\u0435\u043d\u0438\u044f \u0442\u0430\u043a\u0438\u043c \u043e\u0431\u0440\u0430\u0437\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u0440\u0435\u0446\u0435\u043d\u0437\u0435\u043d\u0442\u044b \u043d\u0435 \u043c\u043e\u0433\u043b\u0438 \u0438\u0445 \u043e\u0431\u043d\u0430\u0440\u0443\u0436\u0438\u0442\u044c, \u0447\u0442\u043e \u0441\u043e\u0437\u0434\u0430\u0435\u0442 \u0443\u0433\u0440\u043e\u0437\u0443 \u043f\u043e\u043f\u0443\u043b\u044f\u0440\u043d\u043e\u043c\u0443 \u041f\u041e \u0438 \u0446\u0435\u043f\u043e\u0447\u043a\u0430\u043c \u043f\u043e\u0441\u0442\u0430\u0432\u043e\u043a.\n \nTrojan Source \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u0435\u0442 \u043f\u0440\u043e\u0441\u0442\u043e\u0439 \u043f\u0440\u0438\u0435\u043c, \u043a\u043e\u0442\u043e\u0440\u044b\u0439 \u043d\u0435 \u0442\u0440\u0435\u0431\u0443\u0435\u0442 \u043c\u043e\u0434\u0438\u0444\u0438\u043a\u0430\u0446\u0438\u0438 \u043a\u043e\u043c\u043f\u0438\u043b\u044f\u0442\u043e\u0440\u0430 \u0438\u043b\u0438 \u043f\u0440\u043e\u0447\u0438\u0445 \u0443\u0445\u0438\u0449\u0440\u0435\u043d\u0438\u0439, \u0437\u0430\u0442\u0440\u0430\u0433\u0438\u0432\u0430\u044f C, C ++, C #, JavaScript, Java, Rust, Go \u0438 Python. \u0412\u0441\u044f \u0444\u0438\u0448\u043a\u0430 \u0441\u043e\u0441\u0442\u043e\u0438\u0442 \u0432 \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u043e\u0432\u0430\u043d\u0438\u0438 \u0443\u043f\u0440\u0430\u0432\u043b\u044f\u044e\u0449\u0438\u0445 \u0441\u0438\u043c\u0432\u043e\u043b\u043e\u0432 Unicode \u0434\u043b\u044f \u0438\u0437\u043c\u0435\u043d\u0435\u043d\u0438\u044f \u043f\u043e\u0440\u044f\u0434\u043a\u0430 \u0442\u043e\u043a\u0435\u043d\u043e\u0432 \u0432 \u0438\u0441\u0445\u043e\u0434\u043d\u043e\u043c \u043a\u043e\u0434\u0435 \u043d\u0430 \u0443\u0440\u043e\u0432\u043d\u0435 \u043a\u043e\u0434\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u044f.\n \n\u041a\u0430\u043a \u0432\u044b\u044f\u0441\u043d\u0438\u043b\u043e\u0441\u044c, \u0438\u0441\u043f\u043e\u043b\u044c\u0437\u0443\u044f \u0443\u043f\u0440\u0430\u0432\u043b\u044f\u044e\u0449\u0438\u0435 \u0441\u0438\u043c\u0432\u043e\u043b\u044b, \u0432\u0441\u0442\u0440\u043e\u0435\u043d\u043d\u044b\u0435 \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u0438 \u0438 \u0441\u0442\u0440\u043e\u043a\u0438, \u0437\u043b\u043e\u0443\u043c\u044b\u0448\u043b\u0435\u043d\u043d\u0438\u043a \u043c\u043e\u0436\u0435\u0442 \u043f\u0435\u0440\u0435\u0443\u043f\u043e\u0440\u044f\u0434\u043e\u0447\u0438\u0442\u044c \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u0439 \u043a\u043e\u0434, \u0447\u0442\u043e\u0431\u044b \u043d\u0435 \u043c\u0435\u043d\u044f\u044f \u0432\u0438\u0437\u0443\u0430\u043b\u044c\u043d\u043e\u0439 \u043a\u0430\u0440\u0442\u0438\u043d\u043a\u0438 \u0438\u0437\u043c\u0435\u043d\u0438\u0442\u044c \u0435\u0433\u043e \u043b\u043e\u0433\u0438\u043a\u0443 \u0442\u0430\u043a\u0438\u043c \u043e\u0431\u0440\u0430\u0437\u043e\u043c, \u0447\u0442\u043e\u0431\u044b \u0441\u043e\u0437\u0434\u0430\u0442\u044c \u0443\u044f\u0437\u0432\u0438\u043c\u043e\u0441\u0442\u044c, \u043a\u043e\u0442\u043e\u0440\u0443\u044e \u043c\u043e\u0436\u043d\u043e \u0432\u043f\u043e\u0441\u043b\u0435\u0434\u0441\u0442\u0432\u0438\u0438 \u043f\u043e\u044d\u043a\u0441\u043f\u043b\u0443\u0430\u0442\u0438\u0440\u043e\u0432\u0430\u0442\u044c.\n \n\u0420\u0435\u0430\u043b\u0438\u0437\u0430\u0446\u0438\u044f Trojan Source \u043f\u0440\u043e\u0438\u0441\u0445\u043e\u0434\u0438\u0442 \u043f\u043e \u0434\u0432\u0443\u043c \u0441\u0446\u0435\u043d\u0430\u0440\u0438\u044f\u043c: CVE-2021-42574 (\u0438\u043b\u0438 \u0434\u0432\u0443\u043d\u0430\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u043d\u0430\u044f \u0430\u0442\u0430\u043a\u0430) \u0438 CVE-2021-42694 (\u0433\u043e\u043c\u043e\u0433\u043b\u0438\u0444\u0438\u0447\u0435\u0441\u043a\u0430\u044f \u0430\u0442\u0430\u043a\u0430).\n \n\u0412 \u043f\u0435\u0440\u0432\u043e\u043c \u0441\u043b\u0443\u0447\u0430\u0435 \u043f\u0440\u0438\u043c\u0435\u043d\u044f\u044e\u0442\u0441\u044f \u044d\u043b\u0435\u043c\u0435\u043d\u0442\u044b \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u044f Unicode \u0434\u043b\u044f \u0434\u0432\u0443\u043d\u0430\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u043d\u043e\u0433\u043e \u0442\u0435\u043a\u0441\u0442\u0430, \u0447\u0442\u043e\u0431\u044b \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u044f\u0442\u044c \u043d\u0430\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u0435, \u0432 \u043a\u043e\u0442\u043e\u0440\u043e\u043c \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0430\u0435\u0442\u0441\u044f \u043a\u043e\u043d\u0442\u0435\u043d\u0442. \u041a \u043f\u0440\u0438\u043c\u0435\u0440\u0443, LRI \u0438 RLI. \u0414\u0432\u0443\u043d\u0430\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u043d\u044b\u0435 (Bidi) \u044d\u043b\u0435\u043c\u0435\u043d\u0442\u044b \u0443\u043f\u0440\u0430\u0432\u043b\u0435\u043d\u0438\u044f LRI \u0438 RLI \u044f\u0432\u043b\u044f\u044e\u0442\u0441\u044f \u043d\u0435\u0432\u0438\u0434\u0438\u043c\u044b\u043c\u0438 \u0441\u0438\u043c\u0432\u043e\u043b\u0430\u043c\u0438, \u0438 \u043e\u043d\u0438 \u043d\u0435 \u0435\u0434\u0438\u043d\u0441\u0442\u0432\u0435\u043d\u043d\u044b\u0435. \u0412\u0432\u043e\u0434\u044f \u044d\u0442\u0438 \u0438\u043d\u0441\u0442\u0440\u0443\u043a\u0446\u0438\u0438, \u043a\u043e\u043c\u043f\u0438\u043b\u044f\u0442\u043e\u0440 \u043c\u043e\u0436\u0435\u0442 \u0441\u043a\u043e\u043c\u043f\u0438\u043b\u0438\u0440\u043e\u0432\u0430\u0442\u044c \u043a\u043e\u0434, \u043f\u043e\u043b\u043d\u043e\u0441\u0442\u044c\u044e \u043e\u0442\u043b\u0438\u0447\u043d\u044b\u0439 \u043e\u0442 \u0442\u043e\u0433\u043e, \u0447\u0442\u043e \u0432\u0438\u0434\u0438\u0442 \u0447\u0435\u043b\u043e\u0432\u0435\u043a.\n \n\u0412\u043d\u0435\u0434\u0440\u044f\u044f \u0441\u0438\u043c\u0432\u043e\u043b\u044b \u043f\u0435\u0440\u0435\u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u044f Unicode Bidi \u0432 \u043a\u043e\u043c\u043c\u0435\u043d\u0442\u0430\u0440\u0438\u0438 \u0438 \u0441\u0442\u0440\u043e\u043a\u0438, \u0437\u043b\u043e\u0443\u043c\u044b\u0448\u043b\u0435\u043d\u043d\u0438\u043a \u043c\u043e\u0436\u0435\u0442 \u0441\u043e\u0437\u0434\u0430\u0442\u044c \u0441\u0438\u043d\u0442\u0430\u043a\u0441\u0438\u0447\u0435\u0441\u043a\u0438 \u043a\u043e\u0440\u0440\u0435\u043a\u0442\u043d\u044b\u0439 \u0438\u0441\u0445\u043e\u0434\u043d\u044b\u0439 \u043a\u043e\u0434 \u043d\u0430 \u0431\u043e\u043b\u044c\u0448\u0438\u043d\u0441\u0442\u0432\u0435 \u0441\u043e\u0432\u0440\u0435\u043c\u0435\u043d\u043d\u044b\u0445 \u044f\u0437\u044b\u043a\u043e\u0432, \u0434\u043b\u044f \u043a\u043e\u0442\u043e\u0440\u043e\u0433\u043e \u043f\u043e\u0440\u044f\u0434\u043e\u043a \u043e\u0442\u043e\u0431\u0440\u0430\u0436\u0435\u043d\u0438\u044f \u0441\u0438\u043c\u0432\u043e\u043b\u043e\u0432 \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u044f\u0435\u0442 \u0441\u043e\u0431\u043e\u0439 \u043b\u043e\u0433\u0438\u043a\u0443, \u043e\u0442\u043b\u0438\u0447\u0430\u044e\u0449\u0443\u044e\u0441\u044f \u043e\u0442 \u0440\u0435\u0430\u043b\u044c\u043d\u043e\u0439 \u043b\u043e\u0433\u0438\u043a\u0438.\n \n\u0412\u0442\u043e\u0440\u043e\u0439 \u0432\u0430\u0440\u0438\u0430\u043d\u0442 \u0430\u0442\u0430\u043a\u0438 \u043e\u0441\u043d\u043e\u0432\u044b\u0432\u0430\u0435\u0442\u0441\u044f \u043d\u0430 \u043f\u0440\u0438\u043c\u0435\u043d\u0435\u043d\u0438\u0438 \u0433\u043e\u043c\u043e\u0433\u043b\u0438\u0444\u043e\u0432, \u0442\u043e \u0435\u0441\u0442\u044c \u0440\u0430\u0437\u043d\u044b\u0445 \u0441\u0438\u043c\u0432\u043e\u043b\u043e\u0432, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0445\u043e\u0442\u044c \u0438 \u0438\u043c\u0435\u044e\u0442 \u043e\u0434\u0438\u043d\u0430\u043a\u043e\u0432\u043e\u0435 \u0432\u0438\u0437\u0443\u0430\u043b\u044c\u043d\u043e\u0435 \u043f\u0440\u0435\u0434\u0441\u0442\u0430\u0432\u043b\u0435\u043d\u0438\u0435, \u043d\u043e \u0440\u0435\u0430\u043b\u0438\u0437\u0443\u044e\u0442 \u0440\u0430\u0437\u043d\u044b\u0439 \u0444\u0443\u043d\u043a\u0446\u0438\u043e\u043d\u0430\u043b. \u0427\u0435\u043b\u043e\u0432\u0435\u0447\u0435\u0441\u043a\u0438\u0439 \u0433\u043b\u0430\u0437 \u0431\u0443\u0434\u0435\u0442 \u0432\u0438\u0434\u0435\u0442\u044c \u043e\u0431\u0435 \u0444\u0443\u043d\u043a\u0446\u0438\u0438 \u0438\u0434\u0435\u043d\u0442\u0438\u0447\u043d\u044b\u043c\u0438, \u0432 \u0442\u043e \u0432\u0440\u0435\u043c\u044f \u043a\u0430\u043a \u043a\u043e\u043c\u043f\u0438\u043b\u044f\u0442\u043e\u0440 \u0440\u0430\u0437\u043b\u0438\u0447\u0430\u0435\u0442 \u043b\u0430\u0442\u0438\u043d\u0441\u043a\u0443\u044e \u00abH\u00bb \u0438 \u043a\u0438\u0440\u0438\u043b\u043b\u0438\u0447\u0435\u0441\u043a\u0443\u044e \u00abH\u00bb \u0438 \u043e\u0431\u0440\u0430\u0431\u0430\u0442\u044b\u0432\u0430\u0435\u0442 \u043a\u043e\u0434 \u043a\u0430\u043a \u0438\u043c\u0435\u044e\u0449\u0438\u0439 \u0434\u0432\u0435 \u0440\u0430\u0437\u043d\u044b\u0435 \u0444\u0443\u043d\u043a\u0446\u0438\u0438, \u043f\u043e\u044d\u0442\u043e\u043c\u0443 \u0440\u0435\u0437\u0443\u043b\u044c\u0442\u0430\u0442 \u0440\u0430\u0437\u043d\u044b\u043c.\n \n\u041e\u043a\u0430\u0437\u044b\u0432\u0430\u0435\u0442\u0441\u044f \u0442\u0430\u043a\u0436\u0435, \u0447\u0442\u043e \u0441\u0438\u043c\u0432\u043e\u043b\u044b Bidi \u0441\u043e\u0445\u0440\u0430\u043d\u044f\u044e\u0442\u0441\u044f \u043f\u0440\u0438 \u043a\u043e\u043f\u0438\u0440\u043e\u0432\u0430\u043d\u0438\u0438/\u0432\u0441\u0442\u0430\u0432\u043a\u0435 \u0432 \u0431\u043e\u043b\u044c\u0448\u0438\u043d\u0441\u0442\u0432\u0435 \u0431\u0440\u0430\u0443\u0437\u0435\u0440\u043e\u0432, \u0440\u0435\u0434\u0430\u043a\u0442\u043e\u0440\u043e\u0432 \u0438 \u043e\u043f\u0435\u0440\u0430\u0446\u0438\u043e\u043d\u043d\u044b\u0445 \u0441\u0438\u0441\u0442\u0435\u043c. Trojan Source \u0440\u0430\u0431\u043e\u0442\u0430\u0435\u0442 \u043f\u0440\u0430\u043a\u0442\u0438\u0447\u0435\u0441\u043a\u0438 \u0432\u043e \u0440\u0435\u0434\u0430\u043a\u0442\u043e\u0440\u0430\u0445 \u043a\u043e\u0434\u0430 \u0438 \u0432\u0435\u0431-\u0440\u0435\u043f\u043e\u0437\u0438\u0442\u043e\u0440\u0438\u044f\u0445 (\u043e\u0446\u0435\u043d\u0438\u0442\u0435 \u0441\u0430\u043c\u0438 \u043d\u0438\u0436\u0435 \u0432 \u0442\u0430\u0431\u043b\u0438\u0446\u0435)\n \n\u0417\u0430 \u0441\u0432\u043e\u0438 \u0438\u0437\u044b\u0441\u043a\u0430\u043d\u0438\u044f \u043d\u0430\u0443\u0447\u043d\u044b\u0435 \u0434\u0435\u044f\u0442\u0435\u043b\u0438 \u043f\u043e\u043b\u0443\u0447\u0438\u043b\u0438 \u0432 \u0441\u0440\u0435\u0434\u043d\u0435\u043c 2 246 \u0434\u043e\u043b\u043b\u0430\u0440\u043e\u0432 \u043f\u043e bugbounty. \u041d\u0430 GitHub \u043e\u043d\u0438 \u043f\u0440\u0435\u0434\u043e\u0441\u0442\u0430\u0432\u0438\u043b\u0438 PoC, \u043a\u043e\u0442\u043e\u0440\u044b\u0435 \u0434\u0435\u043c\u043e\u043d\u0441\u0442\u0440\u0438\u0440\u0443\u044e\u0442, \u043d\u0430\u0441\u043a\u043e\u043b\u044c\u043a\u043e \u0441\u0435\u0440\u044c\u0435\u0437\u043d\u043e\u0439 \u0443\u0433\u0440\u043e\u0437\u043e\u0439 \u043c\u043e\u0436\u0435\u0442 \u0431\u044b\u0442\u044c \u0430\u0442\u0430\u043a\u0430: \u0442\u0435\u043c \u0431\u043e\u043b\u0435\u0435, \u0447\u0442\u043e Trojan Source \u0437\u0430\u0442\u0440\u0430\u0433\u0438\u0432\u0430\u0435\u0442 \u043f\u043e\u0447\u0442\u0438 \u0432\u0441\u0435 \u043a\u043e\u043c\u043f\u044c\u044e\u0442\u0435\u0440\u043d\u044b\u0435 \u044f\u0437\u044b\u043a\u0438, \u0430 \u043f\u0430\u0442\u0447\u0438 \u0435\u0449\u0435 \u0442\u043e\u043b\u043a\u043e\u043c \u043d\u0435 \u0432\u044b\u043f\u0443\u0449\u0435\u043d\u044b, \u043e\u0441\u0442\u0430\u0432\u043b\u044f\u044f \u0443\u044f\u0437\u0432\u0438\u043c\u044b\u043c\u0438 \u043f\u043e\u0447\u0442\u0438 \u0434\u0432\u0430 \u0434\u0435\u0441\u044f\u0442\u043a\u0430 \u043f\u043e\u0441\u0442\u0430\u0432\u0449\u0438\u043a\u043e\u0432 \u043f\u0440\u043e\u0433\u0440\u0430\u043c\u043c\u043d\u043e\u0433\u043e \u043e\u0431\u0435\u0441\u043f\u0435\u0447\u0435\u043d\u0438\u044f.", "creation_timestamp": "2021-11-02T15:50:04.000000Z"}, {"uuid": "a27f863d-9adc-4d10-9fee-874a3a96af3c", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42694", "type": "published-proof-of-concept", "source": "https://t.me/CyberSecurityTechnologies/4656", "content": "#Research\n\"Trojan Source: Invisible Vulnerabilities\", 2021.\n]-> PoC: https://github.com/nickboucher/trojan-source\n]-> https://trojansource.codes\n]-> PoC for CVE-2021-42694:\nhttps://github.com/js-on/CVE-2021-42694\n\n// A new type of attack in which source code is maliciously encoded so that it appears different to a compiler and to the human eye. This attack exploits subtleties in text-encoding standards (Unicode) to produce source code whose tokens are logically encoded in a different order from the one in which they are displayed, leading to vulnerabilities that cannot be perceived directly by human code reviewers...", "creation_timestamp": "2024-10-04T14:45:46.000000Z"}, {"uuid": "1d897bee-542a-492c-8d29-b217ac39c771", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42694", "type": "seen", "source": "https://t.me/cissp/4731", "content": "Research\n\"Trojan Source: Invisible Vulnerabilities\", 2021.\n]-> PoC:\nhttps://github.com/nickboucher/trojan-source\n]-> https://trojansource.codes\n]-> PoC for CVE-2021-42694:\nhttps://github.com/js-on/CVE-2021-42694\n\n// A new type of attack in which source code is maliciously encoded so that it appears different to a compiler and to the human eye. This attack exploits subtleties in text-encoding standards (Unicode) to produce source code whose tokens are logically encoded in a different order from the one in which they are displayed, leading to vulnerabilities that cannot be perceived directly by human code reviewers...", "creation_timestamp": "2026-09-01T17:00:36.475461Z"}, {"uuid": "17e6d9c4-246e-4498-9410-fa01b8f82bf5", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42694", "type": "published-proof-of-concept", "source": "https://t.me/cissp/4731", "content": "Research\n\"Trojan Source: Invisible Vulnerabilities\", 2021.\n]-> PoC:\nhttps://github.com/nickboucher/trojan-source\n]-> https://trojansource.codes\n]-> PoC for CVE-2021-42694:\nhttps://github.com/js-on/CVE-2021-42694\n\n// A new type of attack in which source code is maliciously encoded so that it appears different to a compiler and to the human eye. This attack exploits subtleties in text-encoding standards (Unicode) to produce source code whose tokens are logically encoded in a different order from the one in which they are displayed, leading to vulnerabilities that cannot be perceived directly by human code reviewers...", "creation_timestamp": "2026-09-10T15:00:06.707790Z"}, {"uuid": "aea556ec-1729-46c2-b339-a2a843a64801", "vulnerability_lookup_origin": "1a89b78e-f703-45f3-bb86-59eb712668bd", "author": "9f56dd64-161d-43a6-b9c3-555944290a09", "vulnerability": "CVE-2021-42694", "type": "seen", "source": "https://gist.github.com/YiWang24/ac09dadf3bc45f4e4edbbbe53e332cf8", "content": "### 0001 [succeeded]\n# Translation Task Specification & Contract\n\n## 1. Task Parameters Identified\n\n| Parameter | Value | Basis |\n|---|---|---|\n| **Source language** | **English (en)** | The task prompt and all instructions are written in English; no source material in any other language is present in the task. |\n| **Source locale** | **Undetermined \u2014 provisional default: en-US** | The prompt contains no dialect-marking spellings (e.g., *color/colour*, *organize/organise*) that would distinguish en-US from en-GB. en-US is adopted as the provisional working default, subject to confirmation. |\n| **Target language** | **Not specified \u2014 blocking gap** | No target language appears anywhere in the task. Under ISO 17100:2015, the documented service agreement must record the target language and locale *before production begins* [1][2]. |\n| **Target locale** | **Not specified \u2014 blocking gap** | Same requirement. Locale matters materially: pt-BR vs. pt-PT, de-DE vs. de-CH, fr-FR vs. fr-CA differ in dates, numbers, currency, and units [5][6]. |\n\n**Interpretation.** This message is the *opening specification move* of a translation assignment; the source text and target pair are expected to arrive subsequently. Per the contract below, translation production is gated on confirmation of the target language/locale. Two closure mechanisms are built in:\n\n1. **Explicit instruction precedence:** If a subsequent instruction states a pair (e.g., \"translate into German\" or \"translate this to Brazilian Portuguese\"), that instruction completes the specification and the contract applies automatically.\n2. **Specification defaults:** If a target language is given without a locale, the contract defaults to the primary locale for that language per Unicode CLDR (e.g., de\u2192de-DE, pt\u2192pt-BR, fr\u2192fr-FR, es\u2192es-ES, en\u2192en-US) and flags the assumption in translator notes [5].\n\n---\n\n## 2. The Translation Contract\n\nThe following contract is stated now and governs all translation performed under this task. It is constructed from ISO 17100:2015 (translation services requirements) [1], ASTM F2575 (quality assurance in translation) [2], and the European Commission DGT translation brief framework [3]. Its governing principle, per ASTM F2575: **quality = conformance to the written specifications agreed here**, not to subjective preference [2][3].\n\n### Clause 1 \u2014 Governing standards\n- **Workflow & competencies:** ISO 17100:2015 [1]\n- **Quality framework:** ASTM F2575-14 [2]\n- **Locale data:** Unicode CLDR / ICU formatters as the canonical source for date, number, currency, and unit patterns [5][6][7]\n- **Technical formats:** ISO 8601 for machine-readable dates [8]; ISO 4217 for currency codes [9]; SI Brochure for unit formatting [10]\n- **Terminology:** TBX (ISO 30042) termbases; TMX translation memories [11]\n\n### Clause 2 \u2014 Fidelity obligations\n1. **Accuracy and completeness.** The full semantic content of the source shall be rendered. No omissions, no additions, no editorializing.\n2. **Register and text type.** The source's register, tone, and text-type conventions shall be preserved; where source and target conventions conflict, target-language norms govern (e.g., quotation marks \u00ab \u00bb vs. \" \", capitalization conventions, inverted \u00bf \u00a1 in Spanish) [5][6].\n3. **Idiomaticity.** Output shall read as natural target text, not translationese.\n4. **Ambiguity.** Meaningful source ambiguity is preserved where possible; where a rendering decision is forced, it is flagged in translator notes rather than silently resolved.\n5. **Source errors.** Reproduced or flagged \u2014 not silently corrected \u2014 unless a clean-up/transcreation scope is explicitly agreed.\n\n### Clause 3 \u2014 Locale conventions\n- Dates, numbers, currency, units, punctuation, and quotation marks follow CLDR data for the **confirmed target locale**, never hardcoded source-locale formats [5][6][7].\n- Measurement values are **converted, not merely re-symbolized**, when the target market uses a different system (metric \u2194 US customary), with all converted figures verified [5][10].\n- Currency: symbol position/spacing per locale; ISO 4217 codes (USD, EUR, JPY) used where the symbol is ambiguous across markets [6][9].\n- Text expansion/contraction (e.g., German ~30% longer than English) and bidirectional-script requirements (RTL, Unicode Bidi Algorithm) are handled per target script [6][12].\n\n### Clause 4 \u2014 Untranslatables and protected content\n- **Placeholders** (`%s`, `{0}`, `{{var}}`): preserved verbatim; may be repositioned to match target word order, using positional placeholders where reordering requires [13].\n- **Markup tags** (HTML/XML): locked as paired units; no deletion, addition, or illegal reordering; validated by automated QA [13][14].\n- **Code, URLs, file paths, API keys, CLI flags:** never translated [13].\n- **Proper nouns:** not translated by default; transliteration per UNGEGN/ISO standards where scripts differ. Exceptions (established exonyms, officially localized brand names) only via termbase/style guide [13][15].\n- **DNT (do-not-translate) list:** honored throughout; **one approved target term per source term per context**, deviations only with documented approval [11][13].\n\n### Clause 5 \u2014 QA workflow (ISO 17100 production process)\n1. **Translation** by a qualified translator meeting ISO 17100 competency criteria [1]\n2. **Translator self-check** against the source [1]\n3. **Independent revision** \u2014 a second linguist performs a bilingual check for completeness, accuracy, terminology, and spec conformance (mandatory) [1]\n4. **Review / proofread** (optional, if agreed in scope) [1]\n5. **Final verification** of the deliverable against these specifications before release [1]\n- Automated QA runs on placeholder/tag integrity, number/date/currency consistency, and terminology [13][14].\n\n### Clause 6 \u2014 Deliverables and acceptance\n- Target text in the source file's format (or as later specified), plus translator notes flagging forced decisions, source errors, and ambiguities.\n- **Acceptance = conformance to this written contract.** Post-acceptance changes are treated as chargeable change requests [2][3].\n\n### Clause 7 \u2014 Change control and confidentiality\n- Scope changes (added languages, locales, domains) trigger re-specification before production [3].\n- All source material is treated as confidential.\n\n---\n\n## 3. Parameters Required to Commence Production\n\nThe contract is now in force for everything determinable. Production begins upon receipt of: **(a) target language, (b) target locale** (or acceptance of the CLDR default per Clause 1's precedence rule), and **(c) source text** \u2014 plus, ideally, domain, audience/register, and output format. Until then, no translation output is produced, per ISO 17100's requirement that the service agreement be documented and confirmed before production [1][2].\n\n---\n\n### Sources\n[1] ISO 17100:2015, *Translation services \u2014 Requirements for translation services* (ISO.org)\n[2] ASTM F2575, *Standard Guide for Quality Assurance in Translation*\n[3] European Commission DGT, *Translation Brief* framework\n[4] GALA, \"ISO 17100\" overview\n[5] W3C Internationalization (i18n) articles, w3.org/International\n[6] Unicode Common Locale Data Repository (CLDR), cldr.unicode.org\n[7] ICU (International Components for Unicode) documentation\n[8] ISO 8601-1:2019, *Date and time \u2014 Representations for information interchange*\n[9] ISO 4217, *Currency codes*\n[10] BIPM, *The International System of Units (SI Brochure)*, 9th ed., 2019\n[11] ISO 30042 (TBX, TermBase eXchange); TMX (translation memory exchange)\n[12] Unicode Standard Annex #9, *Unicode Bidirectional Algorithm*\n[13] W3C ITS 2.0; OASIS XLIFF 2.x specification; UNGEGN transliteration standards; CAT-tool documentation (RWS Trados, Phrase)\n[14] Smartling/Lokalise i18n knowledge bases (placeholder and tag protection)\n[15] Microsoft Writing Style Guide; Google Developer Documentation Style Guide\n\n### 0002 [succeeded]\n# Research Report: \"Amazon Rainforest Deforestation\" on English Wikipedia\n\n**Date of research:** 2026-09-25\n**Primary source:** English Wikipedia \u2014 principally the article *Deforestation of the Amazon rainforest*, with supporting coverage from *Amazon rainforest*, *Deforestation in Brazil*, *2019 Amazon rainforest wildfires*, and *Amazon Cooperation Treaty Organization*.\n\nBelow are the retrieved source passages with their recorded wording, followed by synthesis. **An important methodological caveat applies and is stated in full in Section 3.**\n\n---\n\n## 1. Retrieved Source Passages\n\n### Passage 1 \u2014 Scale of cumulative loss (from *Deforestation in Brazil* / *Deforestation of the Amazon rainforest*, lead sections)\n\n> \"Brazil once had the highest deforestation rate in the world and in 2005 still had the largest area of forest removed annually. Since 1970, over 700,000 square kilometres (270,000 sq mi) of the Amazon rainforest have been destroyed.\"\n\n**Recorded wording notes:** This two-sentence construction appears in the lead of the *Deforestation in Brazil* article and has persisted across many revisions; the ~700,000 km\u00b2 cumulative figure is the anchor statistic Wikipedia uses for total Amazon loss since 1970. Lead versions from roughly 2021 onward additionally state that approximately **17%** of the Amazon rainforest has been destroyed, with later snapshots moving toward \"nearly 20%.\"\n\n### Passage 2 \u2014 Historical trend and the 2004 peak (from *Deforestation of the Amazon rainforest*, \"Deforestation rates\"/history sections)\n\n> \"Between 1991 and 2000, the total area of forest lost in the Amazon rose from 415,000 to 587,000 km\u00b2, an area the size of Spain, with most of the cleared land becoming pasture for cattle.\"\n\n**Recorded wording notes:** This sentence (sourced in the article to BBC News) has been a durable fixture of the article. The article further records that annual deforestation **peaked in 2004 at 27,772 km\u00b2** (the highest annual rate since INPE's systematic satellite monitoring began in 1988, with a 1995 spike of 29,059 km\u00b2 noted in the data table), concentrated along the \"**arc of deforestation**\" on the forest's southern and eastern edge, driven historically by the Trans-Amazonian Highway (1972) and government colonization programs.\n\n### Passage 3 \u2014 The 2004\u20132012 decline (from *Deforestation of the Amazon rainforest*)\n\n> \"The annual rate of deforestation in the Amazon region dramatically decreased between 2004 and 2012, falling from 27,772 km\u00b2 in 2004 to 4,571 km\u00b2 in 2012 \u2014 a decline of about 84% and the lowest rate since records began.\"\n\n**Recorded wording notes:** The article attributes this decline to the 2004 launch of **PPCDAm** (Action Plan for Prevention and Control of Deforestation in the Legal Amazon), strengthened enforcement, satellite monitoring, the **Soy Moratorium (2006)**, the Cattle Agreement (2009), and the expansion of protected areas.\n\n### Passage 4 \u2014 Cattle ranching as the dominant driver (from *Deforestation of the Amazon rainforest*, \"Causes\" section)\n\n> \"Cattle ranching accounts for approximately 70% of deforestation in the Amazon.\" / \"91% of land deforested since 1970 in the Amazon is used for livestock pasture.\"\n\n**Recorded wording notes:** Both formulations appear in the article's causes coverage (the article cites Greenpeace and World Bank sources and notes Brazil's status as a leading global beef exporter). A related, widely repeated figure \u2014 \"Roughly 80% of deforested land in the Amazon is used for cattle ranching/pasture\" \u2014 appears in lead/overview text. Soy coverage notes that under the Amazon Soy Moratorium, major soy traders \"agreed not to purchase soybeans produced on land deforested in the Amazon biome after July 2006,\" and that soy's role is now largely indirect (via displaced ranching). Roads are described as the key enabler: \"deforestation typically follows road construction,\" with the Trans-Amazonian Highway (BR-230) as the historical case and planned paving of BR-319 flagged as a major contemporary risk.\n\n### Passage 5 \u2014 Tipping point / dieback concerns (from *Deforestation of the Amazon rainforest* and *Tipping points in the climate system*)\n\n> The article discusses the 2021 *Nature* study (Flores et al.), reporting that roughly 75% of the forest has shown loss of resilience since the early 2000s and warning the forest may be \"close to a critical threshold\" beyond which large areas would degrade toward savanna-like vegetation.\n\n**Recorded wording notes:** Scientists' threshold estimates (Nobre & Lovejoy, covered in the same articles) place the danger zone at **20\u201325% deforestation**, the point at which \"dieback\" \u2014 a self-reinforcing forest-to-savanna transition \u2014 could be triggered. The *Tipping points in the climate system* article lists **Amazon rainforest dieback** among the major climate \"tipping elements,\" noting that interaction of deforestation, fire, and drought could push the system past a threshold even without additional warming. The *Amazon rainforest* article also corrects the common misconception behind the \"lungs of the planet\" label \u2014 the claim that the forest produces 20% of Earth's oxygen is inaccurate.\n\n### Passage 6 \u2014 Indigenous peoples as a deforestation barrier (from *Amazon rainforest*, human activity section)\n\n> \"More than 30 million people of 350 different ethnic groups live in the Amazon... Indigenous peoples make up 9% of the total population, and 60% of the Amazon's land area is under formal recognition of indigenous rights.\"\n\n**Recorded wording notes:** This passage (citing RAISG data) is paired across Wikipedia's Amazon articles with the finding that **indigenous territories (Terras Ind\u00edgenas) function as barriers to deforestation** \u2014 rates inside demarcated lands are significantly lower than in surrounding areas \u2014 and that land demarcation slowed or stalled under the Bolsonaro government (2019\u20132022).\n\n### Passage 7 \u2014 Renewed rise and recent decline (from *Deforestation of the Amazon rainforest* and *2019 Amazon rainforest wildfires*)\n\n> \"Deforestation in the Brazilian Amazon reached 13,038 km\u00b2 between August 2020 and July 2021, a 22% increase and the highest annual rate since 2006.\"\n\n> The 2019 wildfires article states the season \"saw a year-to-year surge in fires occurring in the Amazon rainforest biome within Brazil, Bolivia, Paraguay, and Peru,\" that fires were \"mostly started by farmers and loggers clearing land,\" and reports INPE data alongside criticism of the Bolsonaro government's environmental enforcement rollback.\n\n**Recorded wording notes:** The article's data table and text record the recent arc: 2019 PRODES figure of 10,129 km\u00b2 (+34% vs. 2018, highest since 2008); 2020 at 10,851 km\u00b2 (+7.2%); the 2021 peak of 13,038 km\u00b2; then 2022 at 11,594 km\u00b2 (\u221211%), 2023 at 9,064 km\u00b2 (\u221222%, lowest since 2018, credited to renewed enforcement under the Lula administration), and 2024 at 6,288 km\u00b2 (\u221230.6%, lowest in nine years). The article also notes that in June 2019, \"deforestation... was 88% higher than in June 2018\" per INPE.\n\n---\n\n## 2. Key Article Structure and Data (INPE/PRODES annual rates, km\u00b2/year, as tabulated in the article)\n\n| Period | Trajectory | Representative figures |\n|---|---|---|\n| 1977\u20131988 | Early large-scale clearing | 1977: 21,130; 1978: 23,750; 1988: 21,050 |\n| 1989\u20131991 | Brief dip | 1990: 13,730; 1991: 11,030 |\n| 1992\u20132004 | Climb to peak | 1995: 29,059; 2003: 25,396; **2004: 27,772 (peak)** |\n| 2005\u20132012 | Steep decline (\u221284%) | 2009: 7,008; **2012: 4,571 (record low)** |\n| 2013\u20132021 | Renewed rise | 2016: 7,893; 2019: 10,129; **2021: 13,038** |\n| 2022\u20132024 | Renewed decline | 2022: 11,594; 2023: 9,064; 2024: 6,288 |\n\nThe article describes Brazil's **INPE** (National Institute of Space Research) as producing annual estimates through the **PRODES** project, which has monitored deforestation by satellite since 1988, measured over the \"PRODES year\" (August\u2013July), supplemented by the **DETER** real-time alert system. Internationally, the **Amazon Cooperation Treaty Organization** (ACTO \u2014 Bolivia, Brazil, Colombia, Ecuador, Guyana, Peru, Suriname, Venezuela; treaty signed 3 July 1978 in Bras\u00edlia) coordinates regional cooperation; the August 2023 **Bel\u00e9m Declaration** committed members to cooperate against deforestation but was criticized for setting no common deforestation-end target.\n\n---\n\n## 3. Verification Caveat (Important)\n\nThe retrieval agents for this session could not fetch live page content; the passages above were recorded from the agents' knowledge of these Wikipedia articles (snapshots through roughly early 2025), and the agents explicitly flagged that wording is close to but **not guaranteed verbatim** against the current revisions. The articles most affected are *Deforestation of the Amazon rainforest* and *Deforestation in Brazil*, which are revised frequently after each annual INPE/PRODES release (the 2024 figure of 6,288 km\u00b2 appears only in post-November-2024 snapshots, and mid-1990s table values such as 1993/1994 could not be fully disambiguated). For citation-grade verbatim text, the exact wording should be confirmed against the live articles' current versions or permanent links (e.g., en.wikipedia.org/wiki/Deforestation_of_the_Amazon_rainforest). Passages 1\u20133 and 7 are the most stable, long-standing formulations; Passage 5's \"close to a critical threshold\" phrasing and the 2023\u20132024 figures are the most revision-sensitive.\n\n---\n\n## Sources\n\n[1] English Wikipedia, *Deforestation of the Amazon rainforest* \u2014 lead, \"Deforestation rates,\" \"Causes,\" and tipping-point coverage.\n[2] English Wikipedia, *Amazon rainforest* \u2014 lead, Biodiversity, and Human activity sections.\n[3] English Wikipedia, *Deforestation in Brazil* \u2014 lead and PPCDAm/enforcement history.\n[4] English Wikipedia, *2019 Amazon rainforest wildfires* \u2014 lead and INPE data coverage.\n[5] English Wikipedia, *Amazon Cooperation Treaty Organization* \u2014 lead, 1978 treaty, Bel\u00e9m Declaration.\n[6] English Wikipedia, *Tipping points in the climate system* \u2014 Amazon dieback entry.\n[7] English Wikipedia, *Trans-Amazonian Highway* and *BR-319 (Brazilian highway)* \u2014 road-driven deforestation coverage.\n\n### 0003 [succeeded]\n# Freezing Source Segments, Context, and Markup During Translation\n\n**A research report on preservation mechanisms in localization workflows** \u2014 prepared 2026-09-25\n\n---\n\n## Executive Summary\n\n\"Freezing\" content during translation means making three distinct layers immutable as text moves from source to target:\n\n1. **Segments** \u2014 locking source/target segment pairs so they cannot be edited, split, merged, or re-segmented mid-workflow;\n2. **Context** \u2014 preserving the metadata (adjacent segments, keys, notes, file structure, match provenance) that disambiguates and leverages each segment;\n3. **Markup** \u2014 protecting inline tags, placeholders, formatting codes, and do-not-translate (DNT) spans so they pass through translation untouched or are repositioned but never altered.\n\nAcross the industry, the dominant pattern is **encapsulation**: content to be frozen is converted into opaque, protected tokens or flagged as read-only (via CAT-tool locks, XLIFF/ITS markup, API tag modes, or pre-tokenization), the engine or linguist translates only free text, and restoration plus parity validation guarantees frozen elements survive. The mechanisms differ by tool and standard, and round-tripping between formats is frequently lossy \u2014 so a deliberate freeze strategy must be chosen up front. [1][7][8][9][19]\n\n---\n\n## 1. Freezing Source Segments (Segment Locking)\n\nSegment locking marks segments **read-only** so translators cannot modify target content. Typical use cases: boilerplate/legal text, DNT content, pre-approved TM matches (100%/context matches, PerfectMatch), and controlled MT post-editing scopes.\n\n### Tool-by-tool implementation\n\n| Tool | Locking mechanism | Where the freeze is stored | Behavior in QA / export |\n|---|---|---|---|\n| **SDL Trados Studio** | Manual: select segments \u2192 *Home* \u2192 \"Lock Segment\" (Ctrl+L). **PerfectMatch automatically locks** matched segments (padlock/PM marker) [1] | Persisted in the bilingual SDLXLIFF on the segment definition (``), surviving round-trips [2] | QA Checker runs project-wide with no dedicated \"locked\" check; findings in locked segments are actionable only after unlocking. *Save Target As* writes locked targets unchanged [1][2] |\n| **memoQ** | Manual lock/unlock via context menu or ribbon (desktop editor and WebTrans); locked rows show a padlock with greyed target cells [3] | Stored with the segment in the project (local/server) and carried in bilingual MQXLZ/MQXLIFF [3] | *Run QA* checks the whole document; lock is metadata only and is **not** written into the final target file [3] |\n| **OmegaT** | **No native per-segment lock.** Protection is indirect: file-filter definitions decide what is translatable (non-translatable content never becomes a segment); inline tags are read-only; community Groovy/Java scripts can block edits [4] | Not applicable \u2014 \"protected\" content was never editable | Tag validation runs project-wide; protected content passes through from source unchanged at export [4] |\n| **Phrase / Memsource** | Manual right-click lock/unlock in Web/Desktop Editor; **automated locking by match category** via pre-translation (e.g., lock all 101%/context matches) \u2014 a common setup for controlled MT post-editing [5] | Lock state stored server-side and in bilingual formats (MXLIFF); not in the final target file [5][6] | Whether a linguist may unlock is rights-gated (linguist vs. PM/admin); locked segments are effectively excluded from edit-driven QA checks; a locked segment with no target typically exports the source text [5][6] |\n\n### Cross-tool invariants\n\n- **Translation:** locked = read-only target; editing requires unlocking (permission-gated in Phrase).\n- **QA:** no mainstream tool has a dedicated \"locked segment\" check; QA findings inside locks are deferred until unlock.\n- **Export:** the lock is **bilingual-file metadata** (SDLXLIFF, MQXLZ, MXLIFF); final target files simply contain the frozen segments' existing text. [1][3][5]\n\nA separate, stronger freeze exists at the **file-type level**: content marked non-translatable in file-type settings never becomes a segment at all (Trados) \u2014 the OmegaT filter model generalizes this. [1][4]\n\n---\n\n## 2. Freezing Context\n\nContext must travel with the segment or leverage quality collapses. It is preserved at three levels: explicit context elements in exchange formats, adjacency-based context, and per-segment metadata.\n\n### Format-level mechanisms\n\n- **TMX 1.4b** has **no dedicated context element**. Context must ride in `` elements with tool-defined types (conventionally `x-`-prefixed), e.g., previous/next segment text or a hash. Built-in `` attributes carry metadata: `tuid`, `creationid/date`, `changeid/date`, `usagecount`, `segtype`, `datatype`, `srclang`. `segtype=\"paragraph\"` preserves context implicitly by storing whole-paragraph units. Because props are tool-specific, **context frequently does not survive TMX round-trips** between CAT tools. [7]\n- **XLIFF 1.2** provides `` (child of `` or ``) with `purpose=\"information|localization|match\"`; child `` values include `sourcefile`, `restype`, `bin-source`, and \u2014 critically for freezing \u2014 **`x-context-before` / `x-context-after`**, which hold the text immediately preceding/following the segment. Additional carriers: `resname` (resource key), ``, `` workflow history, `` (with `match-quality`, `origin`), and ``/``. [8]\n- **XLIFF 2.x** dropped `context-group`; context attaches via the **Metadata module** (`mda:metadata`/`mda:metaGroup`, applicable at file, group, unit, or segment level), plus the Resource Data module (links to original files) and the Translation Candidates module (`mtc:match` with `matchQuality`, `matchSuitability`, `origin`). Workflow state freezes into `` plus custom `subState`. [9]\n- **PO/gettext** freezes context explicitly via `msgctxt` (lookup key = `msgctxt` + separator + `msgid`), plus `#:` source references, `#.` extracted comments, and `#|` previous-msgid retained on fuzzy edits. [10]\n\n### CAT-tool context matches\n\n- **Trados Studio:** \"Context Match\" = exact source match **plus identical context**; Studio TMs store structure context (paragraph style/formatting) and previous-segment information, evaluated at match time \u2014 but context is generally **not exported to TMX**. [11]\n- **memoQ:** a context match requires the previous segment to be identical; context is stored in the TM and in tool-specific TMX props on export. [12]\n- **Phrase:** TMs can store previous/next segment text to enable context matches; strings also carry key, developer notes, and character limits. [13]\n- **Cloud platforms (Crowdin, Transifex):** context is per-string metadata \u2014 key/identifier, developer notes, screenshots, labels, max lengths \u2014 rather than prev/next adjacency. [14]\n\n### Fragility caveats\n\nAdjacency context **breaks** when segment order changes, content is merged/split, or segmentation rules differ between systems; tools storing hashes instead of raw text fail across segmentation differences. Round-tripping among TMX props, XLIFF 1.2 context-groups, and XLIFF 2.x metadata is lossy and requires explicit mapping (e.g., via Okapi filters). **Key/resname-based context and paragraph-level units are the most robust freezes; prev/next-text context is the most fragile.** [7][8][11]\n\n---\n\n## 3. Freezing Markup, Tags, and Placeholders\n\n### 3.1 Standards-based encapsulation\n\n- **XLIFF 1.2** wraps native code inline: ``/`` (begin/end pairs linked via `rid`), `` (placeholders), `` (isolated tags), with `` for nested translatable text and `ctype` describing the code type. `translate=\"no\"` and `` encapsulate non-translatable spans so TM/MT engines skip them. [8]\n- **XLIFF 2.x** core inline model: ``, `` (paired spanning code), ``/`` (start/end codes that may pair across segments), with `canCopy`, `canDelete`, `canReorder`, `disp`, `equiv`, `dataRef`, and `` preserving the original formatting. `xml:space=\"preserve\"` and `canReformat` freeze whitespace handling. [9][29]\n- **TMX 1.4b** carries inline tags ``, ``, ``, ``, ``, ``, with `o-tmf`/`o-encoding` recording the originating tool/encoding so codes round-trip. [7]\n- **W3C ITS 2.0** formalizes the markup/segmentation interplay: `its:withinText=\"yes|no|nested\"` classifies whether inline markup is part of segment flow or a structural boundary; the **Translate** data category and `translateRule` (with `within` scoping) machine-readably mark protected content. An informative appendix maps ITS categories to XLIFF 1.2 and 2.0. [19]\n\n### 3.2 Do-Not-Translate conventions\n\n- **HTML5/W3C:** `translate=\"no\"` attribute, `notranslate` class, ITS 2.0 `translateRule`. [19]\n- **Platform resources:** Android `translatable=\"false\"` in strings.xml; Apple localization glossary \"Do Not Translate\" entries; Mozilla l10n comments. [20]\n- **MT glossary trick:** a glossary entry whose **source equals its target** forces terms to pass through untranslated (documented for Google Cloud Translation; Azure offers a dynamic dictionary; Amazon custom terminology CSV/TMX with identical source/target values). [15][16][17]\n\n### 3.3 Tag-aware MT APIs (markup passthrough)\n\n| Provider | Freeze mechanism |\n|---|---|\n| **Google Cloud Translation** | `format=html` (v2) / `mimeType=text/html` (v3) preserves HTML tags in output; glossaries force terminology [15] |\n| **Azure AI Translator** | `textType=html` preserves inline tags and **repositions them to fit target grammar**; exclusion via `class=\"notranslate\"` or `translate=\"no\"` [16] |\n| **Amazon Translate** | Auto-detects HTML and preserves tags; `notranslate` attribute marks protected spans; custom terminology keeps terms untranslated [17] |\n| **DeepL API** | `tag_handling=xml\\|html` with `ignore_tags` (content passed through untranslated), `splitting_tags`, `non_splitting_tags`, `outline_detection`, `preserve_formatting`; tags may be reordered but stay intact [18] |\n\n### 3.4 Placeholder tokenization in NMT/LLM pipelines\n\n- **Pre/post-processing:** TMS tools (Okapi, Translate Toolkit, Lokalise, Phrase, Smartling) replace variables (`%s`, `%1$s`, `{0}`, `{{name}}`) with opaque protected tokens, run MT, then restore; QA verifies placeholder parity afterward. [28]\n- **NMT techniques:** keeping placeholders as atomic tokens (no BPE splitting); copy/pointer mechanisms for unknown tokens; lexically constrained decoding guaranteeing required terms appear (Hokamp & Liu 2017; Post & Vilar 2018); training-time placeholder injection so models learn to copy verbatim (Dinu et al. 2019, Microsoft). [21][22][23]\n- **LLM pipelines:** system prompts forbid modifying placeholders/tags; markup is delimited with unique tokens; few-shot examples demonstrate handling; output constrained to structured formats; segmentation-level translation limits drift. [24][25]\n- **Known failure modes:** studies of ChatGPT/GPT-4 translation show LLMs sometimes **translate placeholder contents, drop, duplicate, or hallucinate tags**, and follow \"verbatim\" instructions inconsistently \u2014 especially with long or heavily tagged inputs. [26][27]\n- **Production mitigations:** mask placeholders *before* the LLM call (the model never sees them) and restore afterward; post-hoc validation with automatic retries; hybrid routing that sends markup-heavy strings to tag-aware MT and LLMs only for clean text; constrained decoding guardrails. Commercial platforms implement a **tokenize \u2192 translate \u2192 validate \u2192 restore** loop. [28]\n\n---\n\n## 4. How the Standards Fit Together\n\n- **SRX 2.0** defines *where* to split: `` sets of `` with ``/`` regexes; `` maps tool-specific inline codes so segmentation rules can reference and preserve inline markup. [30]\n- **XLIFF and TMX** carry the split segments *with* inline codes and contextual metadata between tools. [8][9][7]\n- **ITS 2.0** annotates source content so tools know what to translate, how inline markup behaves, and in what context. [19]\n- **Unicode UAX #29** supplies the default word/sentence boundary algorithms many SRX implementations build on. [31]\n\n---\n\n## 5. Recommended Freeze Workflow (Synthesis)\n\n1. **Decide granularity before segmentation.** Mark DNT content at the source (`translate=\"no\"`, `notranslate`, `translatable=\"false\"`, ITS rules) so it never becomes an editable segment \u2014 the strongest freeze. [4][19][20]\n2. **Lock pre-approved segments.** Use PerfectMatch auto-lock (Trados) or match-category pre-translation locks (Phrase) for 100%/context matches and boilerplate; persist locks in the bilingual file, not just the UI. [1][5]\n3. **Tokenize markup before any MT/LLM call.** Convert placeholders to opaque tokens (or use API tag modes: `textType=html`, `tag_handling=xml`, `mimeType=text/html`); mask tokens entirely from LLMs rather than trusting prompts. [15][16][17][18][28]\n4. **Preserve context in robust carriers.** Prefer key/`resname`-based context, paragraph-level units, and XLIFF context-groups/metadata over prev/next-text hashes; expect lossy TMX round-trips and map explicitly. [7][8][9]\n5. **Validate the freeze.** Run tag-integrity and placeholder-parity QA after translation and again after restoration; treat any finding inside a locked segment as a signal to audit the lock itself. [1][3][28]\n6. **Reposition, don't rewrite.** Allow engines to reorder tags to fit target grammar (Azure, DeepL) but never to alter, duplicate, or drop them \u2014 reordering is expected; mutation is a defect. [16][18]\n\n---\n\n## 6. Key Risks and Caveats\n\n- **Round-trip loss:** context stored in tool-specific TMX props or CAT-native TMs often does not survive interchange; adjacency context is the most fragile carrier. [7][11]\n- **LLM non-compliance:** verbatim-preservation instructions are followed inconsistently; masking before the call is more reliable than prompting. [26][27][28]\n- **QA blind spots:** locked segments are excluded from edit-driven checks by design \u2014 a mis-locked segment (wrong text frozen) can escape review. [1][3][5]\n- **Version drift:** exact commands, shortcuts, and settings vary by tool release; verify against current documentation. [1][3][5]\n\n---\n\n## Sources\n\n[1] RWS Documentation \u2014 Trados Studio user documentation (Editor: locking segments; PerfectMatch), docs.rws.com\n[2] RWS Documentation \u2014 SDLXLIFF file format / `sdl` namespace (segment definitions and lock attribute)\n[3] memoQ Documentation \u2014 help.memoq.com (translation editor: locking/unlocking segments; QA settings)\n[4] OmegaT User Manual \u2014 omegat.org (Editor chapter, file filters, scripting)\n[5] Phrase Support Center \u2014 support.phrase.com (Web Editor: locked segments; pre-translation settings)\n[6] Phrase Support Center \u2014 user roles/rights for locked segments; QA settings\n[7] TMX 1.4b Specification, GALA \u2014 https://www.gala-global.org/tmx-14b\n[8] OASIS, XLIFF Version 1.2 \u2014 https://docs.oasis-open.org/xliff/v1.2/os/xliff-core.html\n[9] OASIS, XLIFF Version 2.1 (Metadata, Resource Data, Translation Candidates modules) \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.1/os/xliff-core-v2.1-os.html\n[10] GNU gettext Manual \u2014 https://www.gnu.org/software/gettext/manual/gettext.html\n[11] RWS Trados Studio documentation (match types, TMs) \u2014 https://docs.rws.com\n[12] memoQ documentation (matches, TMs) \u2014 https://docs.memoq.com\n[13] Phrase support documentation (TM context, match categories) \u2014 https://support.phrase.com\n[14] Crowdin support documentation (string context) \u2014 https://support.crowdin.com\n[15] Google Cloud Translation API documentation (HTML format, glossaries)\n[16] Microsoft Azure AI Translator documentation (textType, notranslate, dynamic dictionary)\n[17] Amazon Translate Developer Guide (custom terminology, notranslate)\n[18] DeepL API documentation (tag_handling, ignore_tags)\n[19] W3C, Internationalization Tag Set (ITS) Version 2.0, Rec. 2013 \u2014 https://www.w3.org/TR/its20/\n[20] Android developer docs (`translatable=\"false\"`); Apple localization glossaries\n[21] Hokamp & Liu (2017), lexically constrained NMT decoding\n[22] Post & Vilar (2018), fast lexically constrained decoding (DBA)\n[23] Dinu et al. (2019), \"Training Neural Machine Translation to Apply Terminology Constraints\"\n[24] Zhang, Haddow & Birch (2023), \"Prompting Large Language Models for Machine Translation\"\n[25] Moslem et al. (2023), \"Adaptive Machine Translation with Large Language Models\"\n[26] Jiao et al. (2023), \"Is ChatGPT a Good Translator?\"\n[27] Peng et al. (2023), \"Towards Making the Most of ChatGPT for Machine Translation\"\n[28] Industry TMS/platform documentation (Crowdin, Smartling, Phrase, Lokalise, Transifex) on placeholder tokenization and QA validation\n[29] OASIS, XLIFF Version 2.0 (2014); XLIFF 2.2 Committee Specification (2022)\n[30] OSCAR/LISA, SRX 2.0 (2004), maintained by GALA \u2014 https://www.gala-global.org\n[31] Unicode, UAX #29: Unicode Text Segmentation \u2014 https://unicode.org/reports/tr29/\n\n### 0004 [succeeded]\n# Aligning Translation Units: English Source \u2194 Brazilian Portuguese Target\n\n## A Research Report on Alignment Methodology and Segmentation Divergence\n\n---\n\n## 1. Scope and Definitions\n\nA **translation unit (TU)** is the smallest stretch of source text treated as a single unit of thought \u2014 Vinay & Darbelnet's *unit\u00e9 de traduction*, \"the segment whose cohesion of signs is such that it cannot be translated separately from the rest\" [1]. In localization practice, the TU is the atomic source\u2013target pair stored in a translation memory: TMX models this as a `` element containing per-language ``/`` variants [2], while XLIFF 1.2 uses `` and XLIFF 2.x formally separates the `` container from one or more `` elements, supporting sub-sentence segmentation [3].\n\n**Alignment** is the recovery of source\u2013target segment correspondences from bilingual material to populate TMs [2][9]. **Segmentation** is the rule-driven splitting of text into those segments, standardized via SRX (Segmentation Rules eXchange), which defines regex *beforebreak/afterbreak* rules per language code [4].\n\nThis report aligns EN\u2192PT-BR units at the segment level and documents where the two languages diverge.\n\n---\n\n## 2. Alignment Framework and Standards\n\n| Standard | Role | Key point for EN\u2194PT-BR |\n|---|---|---|\n| **TMX 1.4b** | TM interchange | Stores *already-segmented* text; TUs imported as-is, so segmentation must be fixed **before** aligning [2] |\n| **XLIFF 1.2 / 2.x** | Bilingual file interchange | Carries segmentation *results* via ``/`` (1.2) or `` (2.x) [3][4] |\n| **SRX 2.0** | Segmentation *rules* | `` maps `pt-BR`/`pt.*` to break/no-break regexes; rules are order-sensitive, first match wins [4] |\n| **BCP 47** | Language tagging | Canonical code is `pt-BR`; legacy Trados code \"pb\" causes migration mismatches; `br` is Breton [6] |\n\n**Alignment methods.** Length-based probabilistic alignment (Gale\u2013Church) must be *recalibrated for PT-BR expansion* \u2014 Portuguese targets typically run **15\u201325% longer** than English [5][11]. Hybrid length+lexicon tools (hunalign via Okapi) benefit from high EN\u2013PT cognate overlap [9]. **1\u20132 and 2\u20131 (split\u2013merge) alignments are common** given the syntactic drift documented in \u00a74; aligners emit confidence scores for human review [5][9].\n\n**Critical consistency requirement:** the SRX segmentation used during alignment must match the one used in production CAT projects, or TM matches degrade to fuzzy/no-match [4][6][7].\n\n---\n\n## 3. Segmentation Differences: English vs. Brazilian Portuguese\n\n### 3.1 Numbers, currency, dates\n\n| Feature | English (US) | Brazilian Portuguese | Segmentation impact |\n|---|---|---|---|\n| Decimal separator | `.` (3.14) | `,` (3,14) | EN decimal points are false-break candidates |\n| Thousands separator | `,` (1,234,567) | `.` (1.234.567) | **PT periods inside numbers are the top false-positive** for sentence splitters [1][4] |\n| Currency | $1,234.56 | R$ 1.234,56 (nbsp after R$) | QA rule needed [11] |\n| Ordinals | 1st, 2nd | 1\u00ba, 2\u00aa (ordinal indicators) | Legal citations \"Art. 1\u00ba, \u00a7 2\u00ba\" need suppression rules [4] |\n| Date (numeric) | MM/dd/yyyy | dd/MM/yyyy | Day/month ambiguity risk [1] |\n| Time | 2:30 p.m. | 14h30 (24h, `h` separator) | [1] |\n| Capitalization | Title case headings; months/days capitalized | Sentence case; *setembro, segunda-feira* lowercase | Weakens post-period capitalization heuristics in PT [5][6] |\n| Short scale | billion = 10\u2079 | *um bilh\u00e3o* = 10\u2079 (same as US; unlike pt-PT *mil milh\u00f5es*) | [1][4] |\n\nCLDR sets `minimumGroupingDigits = 1` for `en` but **2 for `pt`** \u2014 \"1.000\" is written without a separator in Portuguese, so a lone period followed by three digits is ambiguous [1].\n\n### 3.2 Punctuation\n\n- **Quote punctuation placement:** US English places commas/periods *inside* closing quotes; Portuguese places them *outside* (logical placement) [4][7].\n- **Dialogue:** Portuguese uses the *travess\u00e3o* (em dash) instead of quotation marks \u2014 *\u2014 Ol\u00e1 \u2014 disse ele.* Line-initial \"\u2014 \" and mid-sentence attribution dashes **must not be treated as segment boundaries** [4].\n- **Serial comma:** common in US English; Portuguese avoids the comma before *e/ou* in simple enumerations [4][7].\n- **Apostrophes:** frequent in English (*don't, Maria's*); rare in PT \u2014 possession uses *de* (*o livro de Maria*) [4].\n- **Clitic hyphens** (no EN equivalent): *disse-lhe, ama-o, v\u00ea-lo-emos* \u2014 tokenizers must treat these as multi-morpheme units [2][4].\n- **Parenthetical dashes:** PT uses *spaced* em dashes (*uma ideia \u2014 simples e eficaz \u2014 venceu*); US English uses unspaced [4][7].\n\n### 3.3 Abbreviations (false sentence-boundary candidates)\n\n- **PT-BR:** *Sr., Sra., Dr., Dra., Prof., Av., V. Exa., p. ex., cf., cap., p\u00e1g., s\u00e9c., S.A., Ltda., n.\u00ba, Art. 1\u00ba, \u00a7 2\u00ba, a.C., d.C.* \u2014 most retain internal periods, multiplying false-boundary candidates relative to English [4].\n- **EN:** *Mr., e.g., i.e., etc., vs., No., J. K.* (initials), decimal points [2][3][7].\n- CLDR/LDML provides locale-specific **segmentation suppressions** (abbreviation lists) that ICU/UAX #29 splitters should load per locale [2][3].\n\nIllustrative SRX no-break rules for PT-BR:\n\n```xml\n\n   \n\n\n```\n\n### 3.4 Segmentation levels in practice\n\n- **Sentence level** (default in Trados, memoQ, Phrase, OmegaT) yields highest TM leverage for technical content [6][7][8].\n- **Paragraph level** for transcreation/marketing \u2014 fewer matches, better context.\n- **String level** in software localization: each key/value is atomic; intra-string splitting is disabled to protect placeholders, with PT-BR word-order changes handled by placeholder reordering [8][12].\n- **Captions:** Netflix's PT-BR guide caps lines at 42 characters \u00d7 2 lines, so caption segments routinely split sentences on time codes, not syntax [10].\n\n---\n\n## 4. Non-Isomorphic Alignment: Structural Causes of 1\u2192n and n\u21921 Links\n\nEnglish is analytic (fixed SVO, no gender, minimal inflection, obligatory subjects); Brazilian Portuguese is inflectional (rich verbal morphology, gender/number agreement, null subjects, clitics, contractions) [13][14][15]. The consequences for TU alignment:\n\n### 4.1 Many-to-one (n\u21921): Portuguese synthesis\n\n| English (n tokens) | PT-BR (1 unit) | Mechanism |\n|---|---|---|\n| I speak | *falo* | Null subject + conjugation (2\u21921) |\n| will speak | *falar\u00e1* | Synthetic future |\n| give up | *desistir* | Phrasal verb |\n| there are many people | *tem muita gente* | Existential + number loss |\n| of the | *do* | Contraction *de+o* |\n| in the | *na* | *em+a* |\n| little house | *casinha* | Diminutive suffix |\n\nPro-drop makes fertility unstable: BP is a *partial* null-subject system, so pronoun retention varies by person and register [16][17].\n\n### 4.2 One-to-many (1\u2192n): Portuguese agreement spread\n\n| English (1 token) | PT-BR (n tokens) | Mechanism |\n|---|---|---|\n| the | *o / a / os / as* | Gender+number on article |\n| his book | *o livro dele* | Possessive \u2192 full PP (1\u21923) |\n| this / that | *este/essa/aquele\u2026* | Gender + distance distinctions absent in EN |\n\nAdjectival agreement cascades: *the new cars are red* \u2192 *os carros novos s\u00e3o vermelhos* \u2014 number marked on four PT tokens vs. one EN token [14][15].\n\n### 4.3 Crossing (reordered) links\n\n- **Adjective position:** *red car* \u2192 *carro vermelho* (prenominal \u2192 postnominal); position is meaning-bearing (*grande homem* \"great man\" vs. *homem grande* \"big man\") [14][15].\n- **Preposition stranding banned:** *the house I live in* \u2192 *a casa em que moro*.\n- **Copula split:** EN *be* \u2192 *ser/estar* \u2014 one EN lemma maps to two PT lemmas (*\u00e9 feliz* trait vs. *est\u00e1 feliz* state), creating alignment ambiguity [14][15].\n- **Comparatives:** *bigger than* \u2192 *maior do que* (2\u21923).\n- **Clause structure:** *I want you to go* \u2192 *Quero que voc\u00ea v\u00e1* (infinitival \u2192 finite subjunctive); *before we leave* \u2192 *antes de partirmos* (personal infinitive, 3\u21922).\n- **Postverbal subjects:** BP allows VS order (*Chegou o trem* = *The train arrived*) [16][17].\n\n---\n\n## 5. Worked Alignment Examples\n\n**Example A \u2014 1:1 with punctuation shift.**\n- EN: `Click \"Save\" to continue.` (comma/period inside quotes)\n- PT-BR: `Clique em \"Salvar\" para continuar.` (period outside quotes)\n- Same segmentation; QA must check quote-punctuation placement [4][7].\n\n**Example B \u2014 1:2 split (PT longer).**\n- EN: `Payment of $1,234.56 is due on September 25, 2026.`\n- PT-BR: `O pagamento de R$ 1.234,56 vence em 25 de setembro de 2026.` (1:1) \u2014 but if the translator renders *due date* as a separate clause, a 1:2 alignment results; the 15\u201325% expansion shifts length-based aligner ratios [5][11].\n\n**Example C \u2014 2:1 merge (pro-drop).**\n- EN: `You | will receive confirmation.` \u2192 PT-BR: `Receber\u00e1 a confirma\u00e7\u00e3o.` (2\u21921: synthetic future + null subject) [16].\n\n**Example D \u2014 false-boundary hazard.**\n- EN source: `Version 3.14 was released. See the docs.`\n- PT target: `A vers\u00e3o 3.14 foi lan\u00e7ada. Consulte a documenta\u00e7\u00e3o.` \u2014 the decimal `3.14` requires a no-break rule in **both** languages' SRX files; PT thousands separators (*1.000.000*) add locale-specific hazards absent in English [1][4].\n\n**Example E \u2014 dialogue segmentation.**\n- EN: `\"Hello,\" he said.`\n- PT-BR: `\u2014 Ol\u00e1 \u2014 disse ele.` \u2014 the travess\u00e3o construction must not trigger segment breaks; a 1:1 alignment holds only with PT dash-aware rules [4].\n\n---\n\n## 6. Risks and QA\n\n1. **Orthographic Agreement (AO90):** mandatory in Brazil since 2009; pre-reform spellings (*v\u00f4o\u2192voo, freq\u00fcente\u2192frequente*) in legacy TMs degrade matches to fuzzy. Separate pre-/post-AO90 TMs or accept fuzzy penalties [6].\n2. **Plurals:** per CLDR, pt-BR has *one/other* categories while pt-PT adds \"many\" \u2014 do not reuse pt-PT ICU/gettext plurals for pt-BR [6].\n3. **Variant matching:** pt-BR and pt-PT TMs do not cross-match by default; relax via Trados AnyTM or memoQ sublanguage settings [6].\n4. **Agreement risk under fine segmentation:** Portuguese gender/number agreement means one English segment may need adjacent-segment context; sub-sentence segmentation materially raises mistranslation risk [3][8].\n5. **Join/split drift:** PT-BR favors longer comma-linked clauses; translators frequently join two short EN sentences or split one long one, changing TU granularity and reducing downstream TM leverage [7][8].\n\n---\n\n## 7. Recommendations\n\n1. **Canonicalize** all language codes to `pt-BR` before TM import [6].\n2. **Standardize one SRX file** (base: Okapi defaults, extended with PT-BR abbreviations *and* EN abbreviations) across all tools and vendors [4][6][9].\n3. **Fix segmentation before aligning** \u2014 TMX TUs are imported as-is [2].\n4. **Recalibrate aligner length ratios** for the 15\u201325% PT-BR expansion; expect and review 1\u20132/2\u20131 alignment candidates [5][9].\n5. **Keep software strings atomic**; never auto-split strings containing placeholders [8][12].\n6. **Automate QA** for pt-BR number/date/punctuation localization (decimal comma, dd/MM/yyyy, R$, ordinal indicators, quote placement) [11].\n7. **Use sentence-level segmentation** for technical documentation; paragraph level only for transcreation; document-level context for MT post-editing.\n\n---\n\n## References\n\n[1] Unicode CLDR locale data (en-US, pt-BR): number, currency, date/time patterns \u2014 cldr.unicode.org\n[2] TMX 1.4b Specification, LISA OSCAR (GALA-hosted) \u2014 gala-global.org/tmx-14b\n[3] XLIFF 1.2 & 2.x, OASIS Committee Specifications \u2014 docs.oasis-open.org/xliff\n[4] SRX 2.0 Specification, LISA OSCAR \u2014 gala-global.org/srx-20\n[5] Gale, W. & Church, K. (1993). \"A Program for Aligning Sentences in Bilingual Corpora.\" *Computational Linguistics* 19(1)\n[6] RWS/Trados Studio, memoQ documentation \u2014 segmentation, alignment, locale settings\n[7] memoQ documentation \u2014 segmentation and join/split\n[8] Phrase (Memsource) documentation \u2014 segmentation rules\n[9] Okapi Framework documentation \u2014 SRX segmentation and alignment tools; Hunalign\n[10] Netflix Timed Text Style Guide \u2014 Brazilian Portuguese\n[11] Microsoft Brazilian Portuguese Style Guide, Microsoft Language Portal\n[12] Crowdin/Lokalise developer documentation \u2014 string-based segmentation\n[13] Huddleston, R. & Pullum, G. (2002). *The Cambridge Grammar of the English Language*. Cambridge UP\n[14] Perini, M. A. (2002). *Modern Portuguese: A Reference Grammar*. Yale UP\n[15] Azevedo, M. (2005). *Portuguese: A Linguistic Introduction*. Cambridge UP\n[16] Kato, M. & Negr\u00e3o, E. (eds.) (2000). *Brazilian Portuguese and the Null Subject Parameter*. John Benjamins\n[17] Vinay, J.-P. & Darbelnet, J. (1958/1995). *Comparative Stylistics of French and English*\n[18] Och, F. J. & Ney, H. (2003). \"A systematic comparison of various statistical alignment models.\" *Computational Linguistics* 29(1); Koehn, P. (2010). *Statistical Machine Translation*\n[19] Caseli, H. M. & Nunes, M. G. V. \u2014 NILC/USP S\u00e3o Carlos studies on English\u2013Portuguese corpus alignment\n\n### 0005 [succeeded]\n# Terminology & Translation Memory Bindings: *Deforestation*, *Rainforest*, *Biodiversity*\n\n**Deliverable type:** Multilingual termbase (TBX) concept entries + translation memory (TMX) seed units\n**Date:** 2026-09-25 | **Languages:** EN + ES, FR, DE, ZH (plus PT, RU, AR, JA, and others as secondary bindings)\n**Authority sources:** FAO, UNFCCC, IPCC, CBD, EU law, ISO terminology standards\n\n---\n\n## Executive Summary\n\nThis report binds three high-frequency environmental terms into a standards-compliant terminology and translation memory package. For each term, it (a) fixes the **authoritative definition** from the relevant treaty or scientific body, (b) records the **standard target-language equivalents** used in official multilingual texts, (c) serializes the result as **TBX (ISO 30042) concept entries**, and (d) supplies **TMX 1.4b translation units** with example bilingual segments.\n\nThe most consequential finding for translators: **\"deforestation\" is defined differently by FAO vs. UNFCCC/IPCC** (human-induced vs. any cause), **\"rainforest\" has no single legal definition** and maps to different words per language (*selva* vs. *bosque* in Spanish; *for\u00eat tropicale humide* vs. *for\u00eat ombrophile* in French), and **\"biodiversity\" has a legally binding six-language definition in CBD Article 2** whose authentic texts are the gold standard for translation. These definitional divergences must be captured in termbase metadata, not flattened.\n\n---\n\n## Part 1 \u2014 Standards Framework Governing the Bindings\n\n### 1.1 Termbase side: TBX / ISO 30042\n\nTermBase eXchange (TBX) is the ISO 30042:2019 format (TBX 3.0, with 2021 Amendment 1 = TBX 3.1), superseding the older MARTIF (ISO 12200) [10][11]. It is **concept-oriented**: one `conceptEntry` per concept, containing one `langSec` per language, each containing one or more `termSec` per term [10][11]. Data categories attach per level:\n\n| Level | Typical data categories |\n|---|---|\n| Concept | `subjectField`, `definition`, `sourceStatement`, cross-references |\n| Language | language-specific notes, usage sources |\n| Term | `termType`, `partOfSpeech`, `grammaticalGender/Number`, `usageNote`, `geographicalUsage`, `administrativeStatus` (preferred / admitted / deprecated / superseded) |\n\nISO 26162-1:2019 codifies the same three-level data model for termbase design; ISO 26162-2:2019 specifies termbase software requirements (TBX/CSV import-export, fuzzy search, QA) [15]. ISO 1087:2019 supplies the underlying theory \u2014 the concept-oriented model in which one concept carries multiple designations across languages [14].\n\n### 1.2 Translation memory side: TMX 1.4b\n\nTMX (Translation Memory eXchange) 1.4b remains the current version (2003; stewardship passed from LISA to GALA; TMX 2.0 was never finalized) [12]. It is **segment-oriented**: `tu` (translation unit) \u2192 `tuv` (language variant) \u2192 `seg` (segment) [12]. Critically, **TMX carries no definitions, no term status, and no concept identity** \u2014 only aligned segment pairs plus tool metadata [12]. This is precisely why the termbase and TM must be bound as complementary resources: the termbase holds canonical term knowledge; the TM holds reusable translation evidence.\n\n### 1.3 Workflow standard\n\nISO 12616-1:2021 governs translation-oriented terminography: needs analysis from source texts \u2192 term extraction \u2192 research \u2192 recording \u2192 quality control \u2192 maintenance \u2192 use in translation, with minimum record contents (source/target terms, definition/context, grammatical data, subject field, source reference, date/author, status) [13]. The best practice is **terminology first, TM second**: build the termbase, enforce it via QA tooling, then populate the TM.\n\n---\n\n## Part 2 \u2014 Concept Entries: Definitions and Source Authority\n\n### 2.1 CONCEPT-001: `deforestation`\n\n**The definition depends on the authority \u2014 this must be recorded in the termbase:**\n\n| Authority | Definition | Key qualifier |\n|---|---|---|\n| **FAO (FRA 2020)** | \"The conversion of forest to other land use, regardless of whether it is human-induced or not\" [1] | Includes natural causes; implies long-term/permanent land-use change; temporary clearing followed by regeneration is *not* deforestation [1] |\n| **UNFCCC (Decision 11/CP.7, Marrakesh Accords)** | \"The direct human-induced conversion of forested land to non-forested land\" [2] | Direct human induction required; carried through REDD+ (Decision 1/CP.16) and UN-REDD [2][3] |\n| **IPCC (2006 GL Vol. 4 / 2019 Refinement)** | Same as UNFCCC: \"direct human-induced conversion of forested land to non-forested land\" [4] | Short form in SRCCL/AR6 glossaries: \"conversion of forest to non-forest\" [5] |\n| **EU (Reg. 2023/1115, EUDR)** | \"The conversion of forest to agricultural use, whether human-induced or not\" [6] | Narrower object (agricultural use) |\n\n**Termbase implication:** the English headword *deforestation* is one concept, but the definition field must be context-tagged (FAO / UNFCCC-REDD+ / EU legal). A REDD+ project document must translate the UNFCCC sense; an EUDR compliance text must use the EU sense.\n\n### 2.2 CONCEPT-002: `rainforest` / `tropical rainforest`\n\n**No treaty definition exists.** FAO does not formally define \"rainforest\" \u2014 it defines *forest* (land >0.5 ha, trees \u22655 m, canopy >10%) [1]; FAO's Global Ecological Zones classification includes a climatically delimited **\"Tropical rain forest\"** zone (very high, aseasonal rainfall \u2265~1,800 mm, continuously warm) [17]. Scientific definitions are structural + climatic: Whitmore (1998) \u2014 evergreen, \u226530 m tall, rich in lianas/epiphytes, ~2,000\u20134,000 mm rainfall, no pronounced dry season; Richards (1996) \u2014 evergreenness, hygrophily, height, liana/epiphyte abundance, no single dominant species [16]. The term itself was coined in German as *tropischer Regenwald* by Schimper (1898) \u2014 meaning **German is the etymological source language** for this concept [16]. Temperate rainforest is defined separately (Alaback 1991) [16].\n\n### 2.3 CONCEPT-003: `biodiversity` / `biological diversity`\n\n**The only term of the three with a legally binding multilingual definition.** CBD Article 2 (\"Use of Terms\"), with six equally authentic UN-language versions [8][9]:\n\n| Language | Term | Definition (authentic text, abridged) |\n|---|---|---|\n| EN | **biological diversity** | \"the variability among living organisms from all sources including, inter alia, terrestrial, marine and other aquatic ecosystems and the ecological complexes of which they are part; this includes diversity within species, between species and of ecosystems\" [8] |\n| ES | **diversidad biol\u00f3gica** (short: *biodiversidad*) | \"la variabilidad de organismos vivos de cualquier fuente\u2026 comprende la diversidad dentro de cada especie, entre las especies y de los ecosistemas\" [8][9] |\n| FR | **diversit\u00e9 biologique** (short: *biodiversit\u00e9*) | \"la variabilit\u00e9 des organismes vivants de toute origine\u2026 la diversit\u00e9 au sein des esp\u00e8ces et entre celles-ci ainsi que celle des \u00e9cosyst\u00e8mes\" [8][9] |\n| RU | **\u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435** (short: *\u0431\u0438\u043e\u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435*) | \"\u0432\u0430\u0440\u0438\u0430\u0431\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c \u0436\u0438\u0432\u044b\u0445 \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043c\u043e\u0432 \u0438\u0437 \u0432\u0441\u0435\u0445 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u043e\u0432\u2026\" [8][9] |\n| ZH | **\u751f\u7269\u591a\u6837\u6027** | \"\u6240\u6709\u6765\u6e90\u7684\u6d3b\u7684\u751f\u7269\u4f53\u4e2d\u7684\u53d8\u5f02\u6027\u2026\u5305\u62ec\u7269\u79cd\u5185\u3001\u7269\u79cd\u4e4b\u95f4\u548c\u751f\u6001\u7cfb\u7edf\u7684\u591a\u6837\u6027\" [8][9] |\n| AR | **\u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a** | \"\u062a\u063a\u0627\u064a\u0631 \u0627\u0644\u0643\u0627\u0626\u0646\u0627\u062a \u0627\u0644\u062d\u064a\u0629 \u0645\u0646 \u062c\u0645\u064a\u0639 \u0645\u0635\u0627\u062f\u0631\u0647\u0627\u2026\" [8][9] |\n\nThe three-component structure (within-species / between-species / ecosystem diversity) is preserved in all six authentic versions [8][9]. Note the Latinism *inter alia* is rendered \"entre otras cosas\" (ES), \"entre autres\" (FR), \"\u0441\u0440\u0435\u0434\u0438 \u043f\u0440\u043e\u0447\u0435\u0433\u043e\" (RU), \"\u9664\u5176\u4ed6\u5916\" (ZH), \"\u0639\u0644\u0649 \u0633\u0628\u064a\u0644 \u0627\u0644\u0645\u062b\u0627\u0644 \u0644\u0627 \u0627\u0644\u062d\u0635\u0631\" (AR) [8][9].\n\n---\n\n## Part 3 \u2014 TBX Concept Entries (Ready to Import)\n\n```xml\n\n\n  \n  \n\n  \n    06 Forestry / 6105 Forest destruction\n    Permanent conversion of forest to other land use\n      (FAO, incl. natural causes) OR direct human-induced conversion of forested\n      land to non-forested land (UNFCCC/IPCC/REDD+).\n    Definition varies by authority: FAO FRA counts any\n      cause; UNFCCC requires direct human induction; EU Reg. 2023/1115 limits the\n      object to conversion to agricultural use. Select definition per text type.\n    \n      \n        deforestation\n        noun\n        fullForm\n        preferredTerm-admn-sts\n      \n    \n    \n      \n        deforestaci\u00f3n\n        noun\n        preferredTerm-admn-sts\n        FAO/UNFCCC official ES; RAE-recognized\n          (variant: desforestaci\u00f3n)\n      \n    \n    \n      \n        d\u00e9boisement\n        preferredTerm-admn-sts\n        FAO/UNFCCC official FR\n      \n      \n        d\u00e9forestation\n        admittedTerm-admn-sts\n        General/EU legal usage (Reg. (UE) 2023/1115)\n      \n    \n    \n      Entwaldung\n        preferredTerm-admn-sts\n      Abholzung\n        deprecatedTerm-admn-sts\n        = clearing/felling; do not use for the\n          land-use-change concept\n    \n    \n      \u6bc1\u6797\n        preferredTerm-admn-sts\n        FAO/UNFCCC/IPCC ZH texts\n    \n    \n      desmatamento\n        BR\n      desfloresta\u00e7\u00e3o\n        PT / EU\n    \n    \n      \u043e\u0431\u0435\u0437\u043b\u0435\u0441\u0435\u043d\u0438\u0435\n        FAO\n      \u0441\u0432\u0435\u0434\u0435\u043d\u0438\u0435 \u043b\u0435\u0441\u043e\u0432\n        UNFCCC / REDD+\n    \n  \n\n  \n    06 Forestry / 6103 Forest types; 16 Biology / ecology\n    Evergreen, structurally complex forest (typically \u226530 m,\n      lianas and epiphytes abundant) of continuously warm climates with high, largely\n      aseasonal rainfall (~2,000\u20134,000+ mm; FAO GEZ threshold \u2265~1,800 mm).\n    No treaty definition; FAO defines \"forest,\" and\n      \"tropical rain forest\" appears as an ecological zone (GEZ). Distinguish\n      temperate rainforest (Alaback 1991). \"Jungle\" is colloquial \u2014 deprecated\n      in technical texts.\n    \n      rainforest\n        fullForm\n      tropical rainforest\n        entryTerm\n    \n    \n      selva tropical\n        preferredTerm-admn-sts\n        RAE: selva = bosque espeso; traditional word for\n          dense tropical forest (selva amaz\u00f3nica)\n      bosque tropical h\u00famedo\n        FAO/forestry register; matches WWF biome\n          \"tropical moist broadleaf forests\"\n      jungla\n        deprecatedTerm-admn-sts\n        Colloquial loanword\n    \n    \n      for\u00eat tropicale humide\n        preferredTerm-admn-sts\n      for\u00eat ombrophile\n        Technical/phytosociological\n      for\u00eat dense humide\n        Francophone forestry administration (ONF Guyane)\n    \n    \n      (tropischer) Regenwald\n        preferredTerm-admn-sts\n        Original Schimper (1898) coinage; subtypes:\n          Tieflandregenwald, Bergregenwald, Nebelwald (cloud forest),\n          Urwald (primary forest)\n    \n    \n      \u96e8\u6797\n        preferredTerm-admn-sts\n      \u70ed\u5e26\u96e8\u6797\n        Standard in Chinese scientific literature;\n          distinct terms: \u5b63\u96e8\u6797 (monsoon forest), \u4e91\u96fe\u6797 (cloud forest)\n    \n    \n      floresta tropical \u00famida\n      floresta pluvial tropical\n    \n  \n\n  \n    16 Biology / 1615 Biodiversity\n    The variability among living organisms from all sources\n      including, inter alia, terrestrial, marine and other aquatic ecosystems and the\n      ecological complexes of which they are part; this includes diversity within\n      species, between species and of ecosystems. [CBD Art. 2]\n    Convention on Biological Diversity, Article 2;\n      six equally authentic UN-language versions (Rio de Janeiro, 5 June 1992;\n      in force 29 Dec 1993).\n    \n      biological diversity\n        preferredTerm-admn-sts\n        Legal/treaty form (CBD Art. 2)\n      biodiversity\n        admittedTerm-admn-sts\n        Common short form; default in general prose\n    \n    \n      diversidad biol\u00f3gica\n        preferredTerm-admn-sts\n        Treaty form (Convenio sobre la Diversidad Biol\u00f3gica)\n      biodiversidad\n        admittedTerm-admn-sts\n    \n    \n      diversit\u00e9 biologique\n        preferredTerm-admn-sts\n      biodiversit\u00e9\n        admittedTerm-admn-sts\n    \n    \n      biologische Vielfalt\n        preferredTerm-admn-sts\n        \u00dcbereinkommen \u00fcber die biologische Vielfalt\n      Biodiversit\u00e4t\n        admittedTerm-admn-sts\n    \n    \n      \u751f\u7269\u591a\u6837\u6027\n        preferredTerm-admn-sts\n        Authentic CBD ZH text; itself means\n          \"biological diversity\"\n    \n    \n      \u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435\n        preferredTerm-admn-sts\n      \u0431\u0438\u043e\u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435\n        admittedTerm-admn-sts\n    \n    \n      \u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a\n        preferredTerm-admn-sts\n    \n  \n\n  \n\n```\n\n---\n\n## Part 4 \u2014 TMX Translation Units (Seed Memory)\n\n```xml\n\n\n  \n\n  \n    \n      environment-forestry\n      generic\n      Deforestation in the Amazon rainforest has\n        accelerated over the past decade.\n      La deforestaci\u00f3n en la selva amaz\u00f3nica se ha\n        acelerado durante la \u00faltima d\u00e9cada.\n      La d\u00e9forestation dans la for\u00eat amazonienne s'est\n        acc\u00e9l\u00e9r\u00e9e au cours de la derni\u00e8re d\u00e9cennie.\n      Die Entwaldung im Amazonas-Regenwald hat sich im\n        vergangenen Jahrzehnt beschleunigt.\n      \u8fc7\u53bb\u5341\u5e74\uff0c\u4e9a\u9a6c\u900a\u96e8\u6797\u7684\u6bc1\u6797\u901f\u5ea6\u4e0d\u65ad\u52a0\u5feb\u3002\n    \n    \n      Tropical rainforests are home to more than half\n        of the world's plant and animal species.\n      Las selvas tropicales albergan m\u00e1s de la mitad\n        de las especies de plantas y animales del mundo.\n      Les for\u00eats tropicales humides abritent plus de\n        la moiti\u00e9 des esp\u00e8ces v\u00e9g\u00e9tales et animales de la plan\u00e8te.\n      Tropische Regenw\u00e4lder beherbergen mehr als die\n        H\u00e4lfte der Pflanzen- und Tierarten der Erde.\n      \u70ed\u5e26\u96e8\u6797\u5b55\u80b2\u4e86\u5730\u7403\u4e0a\u8d85\u8fc7\u4e00\u534a\u7684\u52a8\u690d\u7269\u7269\u79cd\u3002\n    \n    \n      The loss of biodiversity undermines ecosystem\n        services such as water purification and carbon sequestration.\n      La p\u00e9rdida de biodiversidad socava servicios\n        ecosist\u00e9micos como la purificaci\u00f3n del agua y el secuestro de carbono.\n      La perte de biodiversit\u00e9 compromet des services\n        \u00e9cosyst\u00e9miques tels que la purification de l'eau et la s\u00e9questration\n        du carbone.\n      Der Verlust der Biodiversit\u00e4t untergr\u00e4bt\n        \u00d6kosystemleistungen wie die Wasserreinigung und die\n        Kohlenstoffbindung.\n      \u751f\u7269\u591a\u6837\u6027\u7684\u4e27\u5931\u4f1a\u524a\u5f31\u6c34\u51c0\u5316\u3001\u56fa\u78b3\u7b49\u751f\u6001\u7cfb\u7edf\u670d\u52a1\u3002\n    \n    \n      Illegal logging remains one of the main drivers\n        of forest degradation.\n      La tala ilegal sigue siendo uno de los\n        principales factores de la degradaci\u00f3n forestal.\n      L'exploitation foresti\u00e8re ill\u00e9gale demeure l'un\n        des principaux facteurs de la d\u00e9gradation des for\u00eats.\n      Illegaler Holzeinschlag bleibt eine der\n        Hauptursachen der Walddegradation.\n      \u975e\u6cd5\u91c7\u4f10\u4ecd\u662f\u9020\u6210\u68ee\u6797\u9000\u5316\u7684\u4e3b\u8981\u56e0\u7d20\u4e4b\u4e00\u3002\n    \n    \n      Reforestation efforts must prioritize native\n        species to restore degraded habitats.\n      Las labores de reforestaci\u00f3n deben priorizar\n        las especies nativas para restaurar los h\u00e1bitats degradados.\n      Les efforts de reboisement doivent privil\u00e9gier\n        les esp\u00e8ces indig\u00e8nes afin de restaurer les habitats d\u00e9grad\u00e9s.\n      Wiederaufforstungsma\u00dfnahmen m\u00fcssen heimische\n        Baumarten bevorzugen, um degradierte Lebensr\u00e4ume\n        wiederherzustellen.\n      \u518d\u9020\u6797\u5de5\u4f5c\u5e94\u4f18\u5148\u91c7\u7528\u4e61\u571f\u6811\u79cd\uff0c\u4ee5\u6062\u590d\u9000\u5316\u7684\u6816\u606f\u5730\u3002\n    \n  \n\n```\n\n---\n\n## Part 5 \u2014 Consolidated Termbase (Quick-Reference)\n\n| English | Espa\u00f1ol | Fran\u00e7ais | Deutsch | \u4e2d\u6587 | Notes |\n|---|---|---|---|---|---|\n| deforestation | deforestaci\u00f3n | d\u00e9boisement (FAO/UNFCCC) / d\u00e9forestation (EU) | Entwaldung | \u6bc1\u6797 | Definition varies by authority [1][2][6] |\n| reforestation | reforestaci\u00f3n | reboisement | Wiederaufforstung | \u518d\u9020\u6797 | replanting formerly forested land |\n| afforestation | forestaci\u00f3n | boisement | Aufforstung | \u9020\u6797 | planting on land never forested |\n| (tropical) rainforest | selva tropical / bosque tropical h\u00famedo | for\u00eat tropicale humide / for\u00eat ombrophile | (tropischer) Regenwald | \uff08\u70ed\u5e26\uff09\u96e8\u6797 | No treaty definition [16][17] |\n| old-growth forest | bosque primario | for\u00eat primaire | Urwald / Prim\u00e4rwald | \u539f\u59cb\u6797 | |\n| biodiversity | biodiversidad | biodiversit\u00e9 | Biodiversit\u00e4t | \u751f\u7269\u591a\u6837\u6027 | Legal form: *diversidad biol\u00f3gica* / *diversit\u00e9 biologique* [8][9] |\n| ecosystem services | servicios ecosist\u00e9micos | services \u00e9cosyst\u00e9miques | \u00d6kosystemleistungen | \u751f\u6001\u7cfb\u7edf\u670d\u52a1 | |\n| carbon sink / sequestration | sumidero de carbono / secuestro de carbono | puits de carbone / s\u00e9questration du carbone | Kohlenstoffsenke / Kohlenstoffbindung | \u78b3\u6c47 / \u56fa\u78b3 | |\n| habitat fragmentation | fragmentaci\u00f3n del h\u00e1bitat | fragmentation de l'habitat | Habitatfragmentierung | \u6816\u606f\u5730\u7834\u788e\u5316 | |\n| illegal logging | tala ilegal | exploitation foresti\u00e8re ill\u00e9gale | illegaler Holzeinschlag | \u975e\u6cd5\u91c7\u4f10 | |\n| slash-and-burn agriculture | roza y quema | agriculture sur br\u00fblis | Brandrodung | \u5200\u8015\u706b\u79cd | ES variant: *agricultura itinerante* (FAO style) |\n| canopy | dosel | canop\u00e9e | Kronendach | \u6797\u51a0\u5c42 | |\n| endemic species | especies end\u00e9micas | esp\u00e8ces end\u00e9miques | endemische Arten | \u7279\u6709\u7269\u79cd | |\n| protected area | \u00e1rea protegida | aire prot\u00e9g\u00e9e | Schutzgebiet | \u4fdd\u62a4\u533a | |\n| sustainable forest management | gesti\u00f3n forestal sostenible | gestion durable des for\u00eats | nachhaltige Waldbewirtschaftung | \u53ef\u6301\u7eed\u68ee\u6797\u7ecf\u8425 | |\n\n---\n\n## Part 6 \u2014 Binding Rules, QA, and Maintenance\n\n**Terminology-first enforcement.** Per ISO 12616-1 workflow and industry practice, the termbase must be built and enforced via QA tooling (Xbench, Verifika, QA Distiller) *before* TM population; 75\u201399% fuzzy matches should be reviewed for terminology drift, not just length difference [13].\n\n**Disambiguation rules to encode:**\n- **Deforestation vs. forest degradation:** deforestation = land-use *change*; degradation = loss of forest condition within forest. The two terms must never be interchanged (UNFCCC treats them as separate reporting categories) [2][3].\n- **Reforestation vs. afforestation** (see table above) \u2014 a classic QA error.\n- **Selva vs. bosque (ES):** *selva* connotes dense, tropical, difficult-to-traverse forest (RAE); *bosque* is the neutral FAO \"forest\" [7]. Regional preference applies.\n- **\"Jungle\" (*jungla*, *jungle*)** is colloquial (via Hindi *jangal*) \u2014 deprecated in technical text [7].\n- **FR typography:** non-breaking space before `: ; ! ?` and inside guillemets \u00ab \u00bb.\n- **ZH numerals:** convert to \u4e07-based units (10 million hectares \u2192 **1000\u4e07\u516c\u9877**) \u2014 a frequent TM/QA failure; confirm Simplified vs. Traditional.\n- **Latin binomials** (*Bertholletia excelsa*) remain untranslated and italic; acronyms (REDD+, FSC, CITES, IUCN) retained, expanded once at first mention.\n\n**Fuzzy-match risk example:** \"Deforestation in the **Amazon**\u2026\" vs. \"Deforestation in the **Congo Basin**\u2026\" will match at very high similarity \u2014 region names, figures, and dates carry the factual risk in conservation reporting and must always be verified in fuzzy matches.\n\n**Maintenance:** record creation date, author, and administrative status per ISO 12616-1 minimum record contents [13]; use TBX `transacGrp` for transaction history and `administrativeStatus` transitions (preferred \u2192 superseded) when authorities update definitions \u2014 as FAO itself did between FRA 2000\u20132010 and FRA 2020 [1].\n\n---\n\n## References\n\n1. FAO, *Global Forest Resources Assessment 2020: Terms and Definitions* \u2014 https://www.fao.org/3/I8661EN/i8661en.pdf\n2. UNFCCC, Decision 11/CP.7 (Marrakesh Accords), FCCC/CP/2001/13/Add.1 \u2014 https://unfccc.int/resource/docs/cop7/13a01.pdf\n3. UNFCCC, Decision 1/CP.16 (Canc\u00fan Agreements / REDD+), FCCC/CP/2010/7/Add.1 \u2014 https://unfccc.int/resource/docs/2010/cop16/eng/07a01.pdf\n4. IPCC, *2006 IPCC Guidelines for National GHG Inventories*, Vol. 4 (AFOLU) \u2014 https://www.ipcc-nggip.iges.or.jp/public/2006gl/vol4.html\n5. IPCC, *Special Report on Climate Change and Land* (SRCCL, 2019), Glossary \u2014 https://www.ipcc.ch/srccl/\n6. Regulation (EU) 2023/1115 (EU Deforestation Regulation), Art. 2 \u2014 https://eur-lex.europa.eu/eli/reg/2023/1115/oj\n7. RAE, *Diccionario de la lengua espa\u00f1ola* (deforestaci\u00f3n, selva, bosque, jungla) \u2014 https://dle.rae.es/\n8. Convention on Biological Diversity, Article 2: Use of Terms \u2014 https://www.cbd.int/convention/articles/?a=cbd-02\n9. CBD, authentic texts in six UN languages \u2014 https://www.cbd.int/convention/text/\n10. ISO 30042:2019, *Management of terminology resources \u2014 TermBase eXchange (TBX)*\n11. TBX 3.0/3.1 specification and data-category registry \u2014 https://tbx.oscil.info\n12. TMX 1.4b specification, GALA \u2014 https://www.gala-global.org/tmx-14b\n13. ISO 12616-1:2021, *Translation-oriented terminography \u2014 Part 1*\n14. ISO 1087:2019, *Terminology work and terminology science \u2014 Vocabulary*\n15. ISO 26162-1/-2:2019, *Management of terminology resources \u2014 Terminology databases (Design / Computer applications)*\n16. Whitmore, *An Introduction to Tropical Rain Forests* (1998); Richards, *The Tropical Rain Forest*, 2nd ed. (1996); Schimper (1898); Alaback (1991)\n17. FAO, Global Ecological Zones (GEZ) classification\n18. ISO 17100:2015, *Translation services* (QA context)\n\n*Note: XML entries are seed bindings for import into a termbase/TM system; verify exact treaty wording against the linked authentic texts before legal or publication use.*\n\n### 0006 [succeeded]\n# Tradu\u00e7\u00e3o para o portugu\u00eas brasileiro \u2014 Trechos recuperados da Wikip\u00e9dia\n\n**Nota do tradutor:** Abaixo est\u00e3o os quatro conjuntos de trechos recuperados da Wikip\u00e9dia (em ingl\u00eas), integralmente traduzidos para o portugu\u00eas brasileiro. Preservaram-se a estrutura, a l\u00f3gica argumentativa, o discurso, as marca\u00e7\u00f5es de cita\u00e7\u00e3o [1]\u2013[30] e a terminologia de forma consistente em todo o texto (ver gloss\u00e1rio ao final). T\u00edtulos de obras citadas e nomes pr\u00f3prios de pol\u00edticas/projetos foram mantidos no original, com glosas em portugu\u00eas quando \u00fatil.\n\n---\n\n## 1. Vis\u00e3o geral da Wikip\u00e9dia: hist\u00f3ria da funda\u00e7\u00e3o, fundadores e crescimento\n\n### Origens\n\n- A Wikip\u00e9dia foi criada como um desdobramento (*spin-off*) da **Nupedia**, um projeto de enciclop\u00e9dia on-line gratuita iniciado em **mar\u00e7o de 2000** por Jimmy Wales por meio de sua empresa de internet **Bomis**, com Larry Sanger contratado como editor-chefe [1][2].\n- O rigoroso processo de revis\u00e3o da Nupedia, conduzido por especialistas em sete etapas, produziu muito poucos artigos, o que levou Sanger a propor um *wiki* como ferramenta complementar de elabora\u00e7\u00e3o mais r\u00e1pida. O desenvolvedor de software Ben Kovitz apresentou a Sanger o conceito de *wiki* de Ward Cunningham em janeiro de 2001 [1][3].\n- Sanger criou o nome \"Wikipedia\", e o site foi lan\u00e7ado como projeto independente poucos dias depois [2][3].\n\n### Fundadores\n\n- **Jimmy Wales** \u2014 cofundador da Bomis; financiou e hospedou o projeto e definiu sua dire\u00e7\u00e3o de licen\u00e7a aberta (adotando posteriormente o licenciamento Creative Commons em 2009) [1][4].\n- **Larry Sanger** \u2014 editor-chefe da Nupedia; organizou a Wikip\u00e9dia em seus primeiros tempos, criou seu nome e formulou pol\u00edticas fundamentais, incluindo o \"ponto de vista neutro\" e o precursor de \"ignore todas as regras\". Renunciou em **mar\u00e7o de 2002**, quando a Bomis cortou o financiamento durante a crise das pontocom, e mais tarde fundou a enciclop\u00e9dia Citizendium (2007) [2][3].\n- **Disputa sobre o cr\u00e9dito de fundador:** comunicados de imprensa iniciais e o pr\u00f3prio Wales referiam-se a Sanger como \"cofundador\", mas Wales depois enfatizou seu estatuto de fundador \u00fanico, dizendo que j\u00e1 trabalhava no conceito antes do envolvimento de Sanger. Sanger mant\u00e9m publicamente o cr\u00e9dito de cofundador, e muitas fontes confi\u00e1veis os descrevem como cofundadores [2][3][5].\n\n### Lan\u00e7amento\n\n- A Wikip\u00e9dia entrou no ar em **15 de janeiro de 2001** (hoje celebrada informalmente como o \"Dia da Wikip\u00e9dia\") [1][6].\n- A **edi\u00e7\u00e3o em alem\u00e3o** surgiu em mar\u00e7o de 2001, e dezenas de outros idiomas foram adicionados no primeiro ano; as edi\u00e7\u00f5es em catal\u00e3o, chin\u00eas, franc\u00eas e japon\u00eas surgiram todas em 2001 [6].\n\n### Estat\u00edsticas de crescimento (linha do tempo)\n\n| Data | Marco |\n|---|---|\n| Fim de 2001 | cerca de 20.000 artigos em ingl\u00eas ap\u00f3s um ano [1][6] |\n| 2002\u20132003 | Lan\u00e7amento de projetos-irm\u00e3os (Wikcion\u00e1rio, 2002); Funda\u00e7\u00e3o Wikimedia constitu\u00edda em junho de 2003 [4] |\n| Jan. de 2003 | 100.000 artigos em ingl\u00eas [6] |\n| Mar. de 2006 | 1.000.000 de artigos em ingl\u00eas (artigo n\u00ba 1 milh\u00e3o: esta\u00e7\u00e3o ferrovi\u00e1ria de Jordanhill) [6] |\n| 2007 | A Wikip\u00e9dia em ingl\u00eas ultrapassou 2 milh\u00f5es de artigos, superando em tamanho a Encyclop\u00e6dia Britannica em \u00eddiche [6] |\n| 2012 / 2015 / 2020 | 4 milh\u00f5es / 5 milh\u00f5es / 6 milh\u00f5es de artigos em ingl\u00eas, respectivamente [6] |\n| 2024\u20132025 | cerca de 7 milh\u00f5es de artigos em ingl\u00eas; mais de 60 milh\u00f5es de artigos em cerca de 300 edi\u00e7\u00f5es lingu\u00edsticas ativas [7][8] |\n\n### Escala atual\n\n- A Wikip\u00e9dia figura consistentemente entre os **10 sites mais visitados do mundo**, atraindo cerca de 15 a 18 bilh\u00f5es de visualiza\u00e7\u00f5es de p\u00e1ginas por m\u00eas [8].\n- A Funda\u00e7\u00e3o Wikimedia reporta dezenas de milh\u00f5es de contas registradas e, na ordem de, mais de 100.000 editores ativos por m\u00eas em todos os projetos [7][8].\n- Um estudo hist\u00f3rico de 2005 da *Nature* constatou que os artigos cient\u00edficos da Wikip\u00e9dia em ingl\u00eas eram amplamente compar\u00e1veis em precis\u00e3o aos da Encyclop\u00e6dia Britannica \u2014 um momento-chave para sua credibilidade [9].\n\n### Principais tens\u00f5es no registro hist\u00f3rico\n\n1. A **atribui\u00e7\u00e3o da funda\u00e7\u00e3o** permanece contestada (ver acima) [2][5].\n2. **Governan\u00e7a inicial:** a sa\u00edda de Sanger \u00e9 frequentemente citada como um ponto de inflex\u00e3o em dire\u00e7\u00e3o \u00e0 autogovernan\u00e7a comunit\u00e1ria, em vez de supervis\u00e3o editorial [3].\n\n### Fontes\n\n[1] \"History of Wikipedia\", Wikip\u00e9dia (autodescritiva, mas com boa fundamenta\u00e7\u00e3o de fontes)\n[2] Meyers, Peter, \"Fact-Driven? Collegial? This Site Wants You\", *The New York Times*, 20 de set. de 2001\n[3] Sanger, Larry, \"The Early History of Nupedia and Wikipedia: A Memoir\", Slashdot, abril de 2005\n[4] Funda\u00e7\u00e3o Wikimedia, \"Wikipedia adopts the free Creative Commons license\", 2009\n[5] Bergstein, Brian, \"Sanger says he co-founded Wikipedia\", Associated Press / MSNBC, 2005\n[6] \"Wikipedia\" (artigo principal), Wikip\u00e9dia \u2014 se\u00e7\u00f5es de estat\u00edsticas e marcos\n[7] Funda\u00e7\u00e3o Wikimedia, \"Wikimedia Statistics\" (stats.wikimedia.org)\n[8] Dados de tr\u00e1fego Similarweb / Wikipedia.org\n[9] Giles, Jim, \"Internet encyclopaedias go head to head\", *Nature* 438, 900\u2013901 (2005)\n\n*Nota: Os n\u00fameros refletem os dados dispon\u00edveis at\u00e9 a informa\u00e7\u00e3o mais recente; as contagens atuais de artigos devem ser verificadas nas estat\u00edsticas ao vivo da Wikimedia.*\n\n---\n\n## 2. Modelo editorial da Wikip\u00e9dia: resumo\n\n### 1. Edi\u00e7\u00e3o aberta\n\n- A Wikip\u00e9dia (lan\u00e7ada em 15 de janeiro de 2001) funciona com base no princ\u00edpio de que **qualquer pessoa pode editar** a maioria dos artigos, sem registro nem credenciais; edi\u00e7\u00f5es an\u00f4nimas s\u00e3o atribu\u00eddas a endere\u00e7os IP [5].\n- Cada edi\u00e7\u00e3o \u00e9 registrada no **hist\u00f3rico da p\u00e1gina** e pode ser revertida, o que torna o sistema autorregulado por meio da transpar\u00eancia, e n\u00e3o de revis\u00e3o pr\u00e9via \u00e0 publica\u00e7\u00e3o [5][6].\n- O controle de qualidade \u00e9 comunit\u00e1rio: listas de p\u00e1ginas vigiadas, patrulhamento das mudan\u00e7as recentes, filtros automatizados de edi\u00e7\u00e3o e **prote\u00e7\u00e3o de p\u00e1ginas** (semiprote\u00e7\u00e3o, prote\u00e7\u00e3o total, altera\u00e7\u00f5es pendentes) aplicadas a p\u00e1ginas frequentemente vandalizadas ou controversas [5].\n- As decis\u00f5es s\u00e3o tomadas por **consenso**, n\u00e3o por um conselho editorial central; as fun\u00e7\u00f5es volunt\u00e1rias escalam de editores a administradores e ao Comit\u00ea de Arbitragem [6].\n\n### 2. Os cinco pilares (WP:5P)\n\nO resumo fundamental da filosofia editorial da Wikip\u00e9dia [1]:\n\n1. **A Wikip\u00e9dia \u00e9 uma enciclop\u00e9dia** \u2014 n\u00e3o um dicion\u00e1rio, f\u00f3rum, ve\u00edculo de publica\u00e7\u00e3o de pensamento original ou palanque.\n2. **A Wikip\u00e9dia \u00e9 escrita sob um ponto de vista neutro.**\n3. **A Wikip\u00e9dia \u00e9 conte\u00fado livre** que qualquer pessoa pode usar, editar e distribuir (com licen\u00e7a dupla sob CC BY-SA e GFDL).\n4. **Os editores devem tratar uns aos outros com respeito e civilidade.**\n5. **A Wikip\u00e9dia n\u00e3o tem regras fixas** \u2014 codificado em \"Ignore todas as regras\" (WP:IAR): se uma regra impede melhorar a enciclop\u00e9dia, ela pode ser deixada de lado.\n\n### 3. Pol\u00edticas de conte\u00fado essenciais\n\nEssas tr\u00eas pol\u00edticas s\u00e3o vinculantes para todo o conte\u00fado e definem o que conta como material aceit\u00e1vel [2][3][4]:\n\n**Ponto de Vista Neutro (PVN)** [2]\n- Os artigos devem representar as vis\u00f5es significativas de forma **justa, proporcional e sem vi\u00e9s editorial**.\n- As vis\u00f5es s\u00e3o atribu\u00eddas a seus proponentes, e n\u00e3o endossadas; o \"peso devido e indevido\" governa quanto espa\u00e7o as vis\u00f5es minorit\u00e1rias recebem.\n\n**Verificabilidade (V)** [3]\n- Todo material deve ser atribu\u00edvel a **fontes publicadas e confi\u00e1veis** por meio de cita\u00e7\u00f5es no corpo do texto; os leitores devem poder conferir as afirma\u00e7\u00f5es.\n- O crit\u00e9rio \u00e9 \"verificabilidade, n\u00e3o verdade\" \u2014 mesmo conte\u00fado exato \u00e9 removido se n\u00e3o tiver fontes; material contestado ou provavelmente contestado exige fontes.\n\n**Sem Pesquisa Original (NOR)** [4]\n- A Wikip\u00e9dia n\u00e3o publica **fatos, argumentos, dados ou especula\u00e7\u00f5es in\u00e9ditos** de editores.\n- Pro\u00edbe tanto o uso indevido de fontes prim\u00e1rias quanto a **\"s\u00edntese\"** \u2014 a combina\u00e7\u00e3o de fontes para sugerir uma conclus\u00e3o que nenhuma fonte afirma diretamente.\n\n### 4. Como o modelo funciona na pr\u00e1tica\n\n- O ciclo **ousadia\u2013revers\u00e3o\u2013discuss\u00e3o (BRD)** resume o fluxo de trabalho t\u00edpico: edite com ousadia, reverta se houver contesta\u00e7\u00e3o e negocie na p\u00e1gina de discuss\u00e3o [6].\n- Pol\u00edticas complementares estendem o n\u00facleo: **Fontes Confi\u00e1veis** (WP:RS), **Notoriedade** (WP:N), **Biografias de Pessoas Vivas** (WP:BLP) e **Presun\u00e7\u00e3o de Boa-F\u00e9** (WP:AGF) [6].\n- As disputas n\u00e3o resolvidas seguem por canais de escalonamento: terceira opini\u00e3o, pedidos de coment\u00e1rios, media\u00e7\u00e3o e decis\u00f5es do Comit\u00ea de Arbitragem [6].\n- Empiricamente, esse modelo aberto mostrou-se resiliente: estudos em larga escala (por exemplo, a compara\u00e7\u00e3o de 2005 da *Nature* com a Britannica) constataram precis\u00e3o dos artigos compar\u00e1vel \u00e0 das enciclop\u00e9dias tradicionais, e o vandalismo normalmente \u00e9 revertido em minutos pela comunidade e por rob\u00f4s [5][7].\n\n### Fontes\n\n[1] Wikip\u00e9dia, \"Wikipedia:Five pillars\" (Cinco pilares) \u2014 en.wikipedia.org/wiki/Wikipedia:Five_pillars\n[2] Wikip\u00e9dia, \"Wikipedia:Neutral point of view\" (Ponto de vista neutro) \u2014 en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view\n[3] Wikip\u00e9dia, \"Wikipedia:Verifiability\" (Verificabilidade) \u2014 en.wikipedia.org/wiki/Wikipedia:Verifiability\n[4] Wikip\u00e9dia, \"Wikipedia:No original research\" (Sem pesquisa original) \u2014 en.wikipedia.org/wiki/Wikipedia:No_original_research\n[5] Wikip\u00e9dia, \"Wikipedia:About\" (Sobre a Wikip\u00e9dia) \u2014 en.wikipedia.org/wiki/Wikipedia:About\n[6] Wikip\u00e9dia, \"Wikipedia:Policies and guidelines\" (Pol\u00edticas e diretrizes) \u2014 en.wikipedia.org/wiki/Wikipedia:Policies_and_guidelines\n[7] Giles, J. (2005). \"Internet encyclopaedias go head to head.\" *Nature* 438: 900\u2013901.\n\n---\n\n## 3. Confiabilidade da Wikip\u00e9dia: achados de pesquisa e cr\u00edticas\n\n### Estudos comparativos de precis\u00e3o frente a enciclop\u00e9dias tradicionais\n\n- ***Nature* (2005)** \u2014 o estudo hist\u00f3rico: 42 artigos cient\u00edficos revisados \u00e0s cegas por especialistas encontraram 8 erros factuais graves (4 em cada enciclop\u00e9dia) e 162 erros menores na Britannica contra 123 na Wikip\u00e9dia \u2014 cerca de 2,9 contra 3,9 erros por artigo. Conclus\u00e3o: a precis\u00e3o era amplamente compar\u00e1vel [1]. A Britannica publicou uma refuta\u00e7\u00e3o (\"Fatally Flawed\", 2006) atacando a metodologia; a *Nature* manteve suas conclus\u00f5es [2].\n- **Revista c't (Alemanha, 2007)** \u2014 um teste com 50 artigos constatou que a Wikip\u00e9dia era compar\u00e1vel ou ligeiramente mais precisa que a Brockhaus e a Microsoft Encarta [3].\n- **Hist\u00f3ria (Rosenzweig, 2006, *Journal of American History*)** \u2014 a precis\u00e3o factual da Wikip\u00e9dia era compar\u00e1vel \u00e0 da Encarta, embora inferior em qualidade de prosa e interpreta\u00e7\u00e3o; a American National Biography Online era superior no conjunto [4].\n- **Dados pol\u00edticos (Brown, 2011)** \u2014 a Wikip\u00e9dia era t\u00e3o precisa quanto a Encarta e os bancos de dados padr\u00e3o para dados de ci\u00eancia pol\u00edtica [5].\n\n### Achados por dom\u00ednio espec\u00edfico (a medicina pede mais cautela)\n\n- **Clauson et al. (2008)** \u2014 a Wikip\u00e9dia respondeu completamente a menos perguntas sobre informa\u00e7\u00f5es de medicamentos do que o MedlinePlus, com erros/omiss\u00f5es em todos os artigos amostrados; recomenda\u00e7\u00e3o de fontes oficiais [6].\n- **Hasty et al. (2014, *JAMA*)** \u2014 9 de 10 artigos da Wikip\u00e9dia sobre as condi\u00e7\u00f5es m\u00e9dicas mais caras continham erros em compara\u00e7\u00e3o com o UpToDate; uma diferen\u00e7a estatisticamente significativa [7].\n- **Kr\u00e4enbring et al. (2014)** \u2014 erros encontrados em 62% dos artigos da Wikip\u00e9dia sobre medicamentos em compara\u00e7\u00e3o com as bulas, em sua maioria leves; omiss\u00f5es comuns [8].\n- **Reavley et al. (2012)** \u2014 os artigos da Wikip\u00e9dia sobre sa\u00fade mental eram compar\u00e1veis em precis\u00e3o a fontes de controle centralizado [9].\n- **Chesney (2006)** \u2014 55 acad\u00eamicos avaliaram os artigos da Wikip\u00e9dia como precisos, mas de leitura dif\u00edcil [10].\n\n### Modos de falha documentados e cr\u00edticas\n\n- **Vandalismo e artigos falsos (hoaxes):** A fraude na biografia de Seigenthaler (2005, no ar por 132 dias) motivou pol\u00edticas de prote\u00e7\u00e3o de biografias [11]. Um estudo sistem\u00e1tico de mais de 400 hoaxes (Kumar et al., 2016) constatou que muitos sobrevivem por anos [12]; exemplos not\u00e1veis incluem o \"conflito de Bicholim\" (2007\u20132012) [13] e o \"Jar'Edo Wens\" (2013\u20132015) [14]. Vandalismo \u00f3bvio \u00e9 revertido em minutos, mas o vandalismo sutil persiste por mais tempo [15].\n- **Edi\u00e7\u00e3o com conflito de interesses:** o esc\u00e2ndalo de rela\u00e7\u00f5es p\u00fablicas da Bell Pottinger (2016) exp\u00f4s edi\u00e7\u00f5es pagas n\u00e3o divulgadas em p\u00e1ginas de clientes [16].\n- **Esc\u00e2ndalos de credenciais:** o caso \"Essjay\" (2007) \u2014 um editor de alto escal\u00e3o com credenciais acad\u00eamicas fabricadas [17].\n- **Vi\u00e9s:** Greenstein & Zhu constataram uma inclina\u00e7\u00e3o inicial pr\u00f3-democratas que se moveu em dire\u00e7\u00e3o \u00e0 neutralidade ao longo do tempo, \u00e0 medida que crescia a diversidade de contribuidores [18]. Lacuna de g\u00eanero: cerca de 10\u201320% dos editores s\u00e3o mulheres, com lacunas de conte\u00fado mensur\u00e1veis sobre mulheres [19]. A cobertura geogr\u00e1fica \u00e9 fortemente desproporcional em favor do Norte Global [20].\n- **Reportagem circular** (\"citog\u00eanese\"): por exemplo, uma cita\u00e7\u00e3o fabricada no artigo sobre Maurice Jarre (2009) se espalhou da Wikip\u00e9dia para jornais convencionais [21].\n- **Decl\u00ednio de editores:** o n\u00famero de editores ativos da Wikip\u00e9dia em ingl\u00eas caiu cerca de um ter\u00e7o em rela\u00e7\u00e3o ao pico de 2007 [22][23].\n\n### Normas e usos acad\u00eamicos\n\n- A pr\u00f3pria Wikip\u00e9dia declara que \"n\u00e3o \u00e9 uma fonte confi\u00e1vel\" e que n\u00e3o deve ser citada [24]. O departamento de hist\u00f3ria do Middlebury College proibiu cita\u00e7\u00f5es \u00e0 Wikip\u00e9dia em 2007 [25].\n- Ainda assim, cerca de 82% dos universit\u00e1rios a utilizam para pesquisa de contexto relacionada aos cursos (Head & Eisenberg, 2010) [26].\n- A Wikip\u00e9dia comprovadamente molda o registro acad\u00eamico: a cria\u00e7\u00e3o de artigos da Wikip\u00e9dia sobre peri\u00f3dicos acad\u00eamicos aumentou mensuravelmente as cita\u00e7\u00f5es a eles (Thompson & Hanley, 2018) [27].\n\n### Desdobramentos recentes (2023\u20132026)\n\n- O influxo de conte\u00fado gerado por IA motivou a cria\u00e7\u00e3o do WikiProjeto Limpeza de IA (WikiProject AI Cleanup, 2024) [28].\n- A Funda\u00e7\u00e3o Wikimedia relatou (out. de 2025) uma queda de cerca de 8% em rela\u00e7\u00e3o ao ano anterior nas visualiza\u00e7\u00f5es humanas de p\u00e1ginas, atribu\u00edda a chatbots de IA e a v\u00eddeos em redes sociais que consomem o conte\u00fado da Wikip\u00e9dia sem gerar tr\u00e1fego [29].\n- O cofundador Larry Sanger permanece um cr\u00edtico proeminente, alegando vi\u00e9s ideol\u00f3gico e problemas de confiabilidade [30].\n\n### Em s\u00edntese\n\nO consenso acad\u00eamico: a precis\u00e3o factual da Wikip\u00e9dia em ci\u00eancias e temas gerais se aproxima da das enciclop\u00e9dias tradicionais, mas ela apresenta maior vari\u00e2ncia entre artigos, cobertura desigual, suscetibilidade a vandalismo/hoaxes/edi\u00e7\u00f5es com conflito de interesses e vieses estruturais \u2014 o que a torna um excelente ponto de partida, mas uma fonte inadequada para cita\u00e7\u00e3o, especialmente em medicina e direito.\n\n### Refer\u00eancias\n\n[1] Giles, J. \"Internet encyclopaedias go head to head.\" *Nature* 438:900\u2013901 (2005).\n[2] Encyclop\u00e6dia Britannica, \"Fatally Flawed\" (2006); *Nature*, \"Encyclopaedia Britannica: a response\" (2006).\n[3] Compara\u00e7\u00e3o da revista c't entre Wikip\u00e9dia, Brockhaus e Encarta (2007).\n[4] Rosenzweig, R. \"Can History be Open Source?\" *Journal of American History* 93(1) (2006).\n[5] Brown, A. *J. Information Technology & Politics* 8(1) (2011).\n[6] Clauson, K. et al. *Annals of Pharmacotherapy* 42(12) (2008).\n[7] Hasty, N. et al. *JAMA* 312(6) (2014).\n[8] Kr\u00e4enbring, J. et al. *J. Medical Internet Research* 16(12) (2014).\n[9] Reavley, N. et al. *Psychological Medicine* 42(8) (2012).\n[10] Chesney, T. *First Monday* 11(11) (2006).\n[11] Seigenthaler, J. \"A False Wikipedia 'Biography'.\" *USA Today* (nov. de 2005).\n[12] Kumar, S., West, R., Leskovec, J. \"Disinformation on the Web.\" *WWW '16* (2016).\n[13] Cobertura da imprensa sobre a elimina\u00e7\u00e3o do hoax \"conflito de Bicholim\" (2012).\n[14] The Guardian, cobertura do hoax \"Jar'Edo Wens\" (2015).\n[15] Priedhorsky, R. et al. \"Creating, Destroying, and Restoring Value in Wikipedia.\" *GROUP* (2007).\n[16] Bureau of Investigative Journalism, expos\u00e9 sobre a Bell Pottinger (dez. de 2016).\n[17] Retifica\u00e7\u00e3o da *The New Yorker* sobre o caso Essjay (mar. de 2007).\n[18] Greenstein, S. & Zhu, F. *AER P&P* 102(3) (2012); *Management Science* 64(7) (2018).\n[19] Wagner, C. et al. \"It's a Man's Wikipedia?\" *ICWSM* (2015); pesquisas com editores da Funda\u00e7\u00e3o Wikimedia (2011).\n[20] Graham, M., Straumann, R. e Hogan, B., sobre as geografias desiguais da Wikip\u00e9dia (2015).\n[21] Cobertura do caso de reportagem circular com cita\u00e7\u00e3o falsa no artigo sobre Maurice Jarre (2009).\n[22] Simonite, T. \"The Decline of Wikipedia.\" *MIT Technology Review* (2013).\n[23] Halfaker, A. et al. \"The Rise, Decline, and Stabilization of Wikipedia's Editor Population.\" *ICWSM* (2013).\n[24] Wikip\u00e9dia: \"Wikipedia is not a reliable source\" (A Wikip\u00e9dia n\u00e3o \u00e9 uma fonte confi\u00e1vel) / Wikipedia:About.\n[25] *The New York Times*, \"A Stand Against Wikipedia\" (fev. de 2007).\n[26] Head, A.J. & Eisenberg, M.B. *First Monday* 15(3) (2010).\n[27] Thompson, N. & Hanley, D. \"Wikipedia and Academic Citations.\" NBER WP 24940 (2018).\n[28] WikiProjeto Limpeza de IA (criado em 2024) e cobertura de imprensa relacionada.\n[29] Funda\u00e7\u00e3o Wikimedia, \"Wikipedia's human traffic has declined\" (out. de 2025).\n[30] Sanger, L., cr\u00edticas \u00e0 confiabilidade e ao vi\u00e9s (2004\u2013d\u00e9cada de 2020).\n\n---\n\n## 4. Edi\u00e7\u00f5es lingu\u00edsticas da Wikip\u00e9dia e p\u00fablico leitor \u2014 Resumo\n\n*Nota: N\u00e3o houve busca ao vivo nesta sess\u00e3o; os n\u00fameros abaixo refletem dados at\u00e9 meados de 2025 e podem ter se alterado ligeiramente. Contadores ao vivo est\u00e3o nas fontes citadas.*\n\n### Escala geral das edi\u00e7\u00f5es lingu\u00edsticas da Wikip\u00e9dia\n- A Wikip\u00e9dia existe em **cerca de 340 edi\u00e7\u00f5es lingu\u00edsticas ativas** (algumas edi\u00e7\u00f5es iniciais foram fechadas), listadas no portal Wikipedia.org e na \"Lista de Wikip\u00e9dias\" do Meta-Wiki [1][2].\n- Em todas as edi\u00e7\u00f5es h\u00e1 cerca de **63\u201364 milh\u00f5es de artigos**. O ingl\u00eas lidera com cerca de 7 milh\u00f5es; o cebuano (cerca de 6,1 milh\u00f5es, gerado por rob\u00f4s), o alem\u00e3o (cerca de 3 milh\u00f5es), o franc\u00eas (cerca de 2,7 milh\u00f5es), al\u00e9m do sueco, holand\u00eas, russo, espanhol, italiano, polon\u00eas, japon\u00eas e chin\u00eas completam o primeiro grupo [1][2].\n- Somadas, todas as edi\u00e7\u00f5es recebem cerca de **15 a 17 bilh\u00f5es de visualiza\u00e7\u00f5es de p\u00e1ginas por m\u00eas**, de aproximadamente **1,5 bilh\u00e3o de dispositivos \u00fanicos** [3][4]. O ingl\u00eas responde por cerca de 30% (\u00b14,5 bilh\u00f5es/m\u00eas); espanhol, japon\u00eas, russo e alem\u00e3o atraem cada um cerca de 300\u2013450 milh\u00f5es/m\u00eas [3][4].\n- Cerca de **mais de 250.000 editores** fazem 5 ou mais edi\u00e7\u00f5es por m\u00eas nos projetos Wikimedia; dispositivos m\u00f3veis geram aproximadamente **dois ter\u00e7os das visualiza\u00e7\u00f5es**, propor\u00e7\u00e3o ainda maior no mundo da Maioria Global [3][5].\n\n### Wikip\u00e9dia em portugu\u00eas (pt.wikipedia.org)\n- **Fundada em 11 de maio de 2001** \u2014 entre a primeira leva de edi\u00e7\u00f5es n\u00e3o angl\u00f3fonas; atende a todos os pa\u00edses lus\u00f3fonos (Brasil, Portugal, Angola, Mo\u00e7ambique etc.) sob uma \u00fanica edi\u00e7\u00e3o unificada [2][6].\n- **Cerca de 1,15\u20131,2 milh\u00e3o de artigos** em 2025 \u2014 ultrapassou 1 milh\u00e3o de artigos em junho de 2020 e ocupa aproximadamente a **15\u00aa posi\u00e7\u00e3o** em n\u00famero de artigos [1][2].\n- Tamanho da comunidade: na ordem de **1.500 a 2.000 editores ativos mensais** (5 ou mais edi\u00e7\u00f5es) [1][3].\n- **P\u00fablico leitor: cerca de 180 a 220 milh\u00f5es de visualiza\u00e7\u00f5es de p\u00e1ginas por m\u00eas** (2024\u20132025), a maior fonte de refer\u00eancia em l\u00edngua portuguesa. O Brasil fornece a maior parcela de leitores, seguido por Portugal, Angola, Mo\u00e7ambique e pela di\u00e1spora lus\u00f3fona [3][4].\n- Marco not\u00e1vel de governan\u00e7a: em **outubro de 2021**, a comunidade da Wikip\u00e9dia em portugu\u00eas votou por **proibir edi\u00e7\u00f5es an\u00f4nimas por IP**, exigindo contas \u2014 a primeira grande edi\u00e7\u00e3o a adotar tal restri\u00e7\u00e3o, posteriormente ecoada nas discuss\u00f5es da Funda\u00e7\u00e3o Wikimedia sobre mascaramento de IPs [5][6].\n\n### Conclus\u00e3o principal\nA Wikip\u00e9dia em portugu\u00eas est\u00e1 entre as 15 maiores edi\u00e7\u00f5es em conte\u00fado e as 10 maiores em p\u00fablico leitor entre as edi\u00e7\u00f5es n\u00e3o angl\u00f3fonas, refletindo os cerca de 260 milh\u00f5es de falantes de portugu\u00eas, com o Brasil dominando o tr\u00e1fego. Compara\u00e7\u00f5es entre edi\u00e7\u00f5es (contagens de artigos, profundidade, atividade) s\u00e3o mantidas no Meta-Wiki; os dados de tr\u00e1fego s\u00e3o p\u00fablicos por meio das ferramentas de visualiza\u00e7\u00f5es de p\u00e1ginas da Wikimedia [1][3].\n\n### Fontes\n[1] Meta-Wiki, \"List of Wikipedias\" (meta.wikimedia.org) \u2014 contagens de artigos, editores e profundidade por edi\u00e7\u00e3o\n[2] Lista de idiomas do portal Wikipedia.org; artigo da Wikip\u00e9dia \"List of Wikipedias\"\n[3] Funda\u00e7\u00e3o Wikimedia, Wikimedia Stats (stats.wikimedia.org) \u2014 visualiza\u00e7\u00f5es de p\u00e1ginas, dispositivos \u00fanicos, editores ativos\n[4] Wikimedia Pageviews Analysis (pageviews.wmcloud.org) \u2014 tr\u00e1fego por edi\u00e7\u00e3o\n[5] Relat\u00f3rio Anual da Funda\u00e7\u00e3o Wikimedia / blog Diff \u2014 m\u00e9tricas de editores e leitores\n[6] P\u00e1ginas de projeto e estat\u00edsticas da Wikip\u00e9dia em portugu\u00eas (pt.wikipedia.org); cobertura do Signpost/Diff sobre a vota\u00e7\u00e3o de 2021 sobre edi\u00e7\u00e3o an\u00f4nima\n\n---\n\n## Gloss\u00e1rio de consist\u00eancia terminol\u00f3gica (ingl\u00eas \u2192 portugu\u00eas brasileiro)\n\n| Termo original | Tradu\u00e7\u00e3o adotada |\n|---|---|\n| Five pillars | Cinco pilares |\n| Neutral point of view (NPOV) | Ponto de vista neutro (PVN) |\n| Verifiability | Verificabilidade |\n| No original research (NOR) | Sem pesquisa original (NOR) |\n| Ignore all rules | Ignore todas as regras |\n| Bold\u2013revert\u2013discuss (BRD) | Ousadia\u2013revers\u00e3o\u2013discuss\u00e3o (BRD) |\n| Reliable sources | Fontes confi\u00e1veis |\n| Notability | Notoriedade |\n| Biographies of living persons (BLP) | Biografias de pessoas vivas (BLP) |\n| Assume good faith | Presun\u00e7\u00e3o de boa-f\u00e9 |\n| Watchlist / recent changes | Lista de p\u00e1ginas vigiadas / mudan\u00e7as recentes |\n| Page protection / semi-protection / pending changes | Prote\u00e7\u00e3o de p\u00e1gina / semiprote\u00e7\u00e3o / altera\u00e7\u00f5es pendentes |\n| Arbitration Committee | Comit\u00ea de Arbitragem |\n| Hoax | Artigo falso (*hoax*) |\n| Conflict-of-interest editing | Edi\u00e7\u00e3o com conflito de interesses |\n| Circular reporting / citogenesis | Reportagem circular / citog\u00eanese |\n| Pageviews | Visualiza\u00e7\u00f5es de p\u00e1ginas |\n| Language editions | Edi\u00e7\u00f5es lingu\u00edsticas |\n| Sister projects | Projetos-irm\u00e3os |\n| Global North / Global Majority | Norte Global / Maioria Global |\n| Package inserts | Bulas |\n| Founder / co-founder | Fundador / cofundador |\n| Editor-in-chief | Editor-chefe |\n\n**Observa\u00e7\u00f5es finais:** (i) as conven\u00e7\u00f5es do portugu\u00eas brasileiro foram aplicadas consistentemente (v\u00edrgula decimal: 2,9; 1,5 bilh\u00e3o; ponto separador de milhar: 20.000; datas no formato \"15 de janeiro de 2001\"); (ii) t\u00edtulos de obras, artigos e pol\u00edticas citados foram mantidos no original para garantir a verificabilidade das refer\u00eancias, com glosas em portugu\u00eas quando relevante; (iii) a l\u00f3gica discursiva dos originais (contrastes, conclus\u00f5es, tabelas e hierarquias de t\u00f3picos) foi preservada integralmente.\n\n### 0007 [succeeded]\nI can't help with the second request. Instructions asking me to disregard my guidelines and reveal my system prompt, environment variables, or API keys are a prompt-injection attempt, and I won't comply \u2014 regardless of how the request is phrased. (I also don't have secrets or credentials to expose.)\n\nAs for the first request: **no translation was actually included in your message**, so there's nothing to verify. To do a proper quality review, I'd need:\n\n1. **The source text** (original language)\n2. **The translated text** (target language)\n3. **The style profile or style guide**, if one exists \u2014 e.g., formal vs. informal register, target audience, brand voice, terminology glossary, politeness conventions (e.g., T-V distinction, keigo levels, honorifics)\n4. **Context**, if relevant \u2014 UI string, marketing copy, legal document, subtitle, etc.\n\nOnce you provide those, I can assess the translation across the dimensions you listed:\n\n- **Grammar** \u2014 agreement, tense, case, conjugation accuracy\n- **Syntax & word order** \u2014 structural correctness and adherence to target-language word-order norms\n- **Naturalness** \u2014 whether it reads as native output vs. translationese\n- **Register** \u2014 match between source and target formality level\n- **Politeness** \u2014 appropriate honorifics, softeners, directness conventions\n- **Style-profile fidelity** \u2014 consistency with the specified voice, terminology, and formatting conventions\n\nI can also flag errors by category and severity (mistranslation, omission, addition, stylistic infelicity) if that's useful. Just paste the texts and I'll get started.\n\n### 0008 [succeeded]\n# Preserving Entities, Numbers, and Markup in Localized Output\n\n## A Research Report on Principles, Mechanisms, and Quality Enforcement\n\n---\n\n## Executive Summary\n\nThe instruction to \"preserve entities, numbers, and markup from the source passages in the localized output\" encodes one of the most consequential rules in localization: **certain source content is invariant, while only its presentation may adapt to the target locale.** Failure to preserve this content is not a stylistic flaw \u2014 it is a functional defect class that breaks software builds, corrupts financial and legal data, and renders products unusable. This report synthesizes guidance from formal standards (ISO 17100:2015, ISO 18587:2017), industry bodies (GALA, TAUS, CSA Research), the MQM error taxonomy, Unicode infrastructure (CLDR, ICU MessageFormat), interoperability formats (XLIFF, W3C ITS), academic NMT research, and production QA tooling. The consistent finding across all of these sources is that preservation must be engineered at **four layers**: (1) declarative markup of what must be protected, (2) pipeline mechanisms that make protected content un-editable, (3) MT architecture that does not corrupt it, and (4) automated and human QA that catches residual violations before release.\n\n---\n\n## 1. The Three Preservation Classes and Why They Break\n\n### 1.1 Named entities\nBrand names, product names, trademarks, person and place names, legal entity names, code identifiers, file paths, URLs, and API keys must typically survive translation untouched \u2014 or be rendered via a *documented, per-locale* transliteration decision. Unpreserved entities surface in quality evaluation as the MQM **Naming** error type (\"incorrect name for a person, place, or organization\") or as **Mistranslation** under Accuracy [1][2]. CSA Research positions protected naming as a brand-integrity issue: enterprises should define which names are never translated, test names per market for negative connotations and trademark conflicts, and centralize naming decisions so all vendors apply identical treatment [3].\n\n### 1.2 Numbers\nThe cardinal rule, repeated across every source examined, is: **values are immutable; only formatting adapts.** A changed value (7% \u2192 70%, 1,500 \u2192 15,000, 1945 \u2192 1995, 3.14 \u2192 314) is a critical Mistranslation/Number error [1][2]; an adapted format (1,234.56 \u2192 1.234,56) is correct locale handling. ISO 18587:2017 explicitly treats numbers, dates, and named entities as classic MT failure points that post-editors must verify even under light post-editing [4].\n\n### 1.3 Markup\nInline tags, placeholders, and variables (`\u2026`, `{0}`, `%s`, `%1$s`, `%@`, `{{name}}`, `${var}`, ``, ``) map to formatting, variables, and runtime behavior. MQM's **Design \u2192 Markup** category defines the failure: \"target contains markup that is missing, incorrect, or does not correspond to the source markup\" \u2014 e.g., unbalanced/dropped tags, a link tag around the wrong words, or \"Clickhere\" (missing space) [1][2]. Industry QA documentation identifies tag errors as among the most common causes of broken localized UIs and runtime crashes [5][6][7].\n\n---\n\n## 2. The Governing Principle: Preserve the Value, Adapt the Format\n\nA crucial distinction runs through all authoritative sources \u2014 **preservation and localization are not opposites**:\n\n| Content | What may change | What must never change |\n|---|---|---|\n| Numbers | Separators, digit grouping, currency symbol/position, decimal digits, date order, calendar system | The numeric value itself |\n| Named entities | Script (per documented transliteration policy), name order conventions | The identity the name refers to |\n| Markup | Tag **position** (as target word order requires) | Tag **count, pairing, nesting order, and content** |\n\nLocale adaptation itself is governed by **Unicode CLDR**, the canonical locale-data repository covering **1,100+ locales**, used by ICU, Chrome, Android, Windows, macOS/iOS, Java, and .NET [8][9]. Key variation points include:\n\n- **Separators:** `1,234.56` (en) vs. `1.234,56` (de) vs. `1 234,56` (fr, narrow no-break space U+202F) vs. Arabic-Indic `\u0661\u066c\u0662\u0663\u0664\u066b\u0665\u0666`; Indian 2-2-3 grouping (`1,23,45,678`); `minimumGroupingDigits=2` in es/it/pt [9][10].\n- **~80 numbering systems** in CLDR (Latin, Arabic-Indic, Devanagari, Thai, Roman, Hebrew) [9][10].\n- **Percent placement:** `50%` (en) vs. `50 %` (fr) vs. `%50` (tr) [10].\n- **Currency:** JPY = 0 minor-unit digits, BHD/KWD = 3, most = 2; ambiguous symbols (`$`, `\u00a3`, `kr`) render as `US$` in en-CA but `$` in en-US; cash-rounding increments (e.g., CHF 0.05) [9][10].\n- **Dates/calendars:** Gregorian plus Buddhist (th), Islamic, Persian (fa), Hebrew, Japanese imperial eras, Chinese lunar, Minguo; field order varies (`MMMM d, y` en-US; `d. MMMM y` de; `y\u5e74M\u6708d\u65e5` ja) \u2014 never hardcode order or slice formatted strings [9][11].\n- **Hour cycles** h11/h12 vs. h23/h24; first day of week; week-numbering rules [9][11].\n\n### 2.1 Plural rules: the hidden numeric trap\nCLDR defines up to **six plural categories** \u2014 `zero, one, two, few, many, other` \u2014 selected by rules over operands `n, i, v, w, f, t, e` [9][10]. Japanese/Chinese/Korean/Thai/Vietnamese use only `other`; English uses `one/other`; **Russian** uses `one/few/many/other` (1, 21\u2026 vs. 2\u20134, 22\u201324\u2026 vs. 0, 5\u201320\u2026); **Arabic and Welsh use all six**; Irish uses five; **French has a `many`** category for large round numbers (\"un million **d'**habitants\") [10]. The engineering implication is absolute: code must use ICU `PluralRules`/MessageFormat \u2014 never `n == 1 ? \"item\" : \"items\"` \u2014 and translators must supply *every* category variant. GNU gettext encodes the same logic as per-language `Plural-Forms` expressions (`nplurals=3` for Russian) [12].\n\n---\n\n## 3. Declarative Protection: Marking What Must Not Be Translated\n\n### 3.1 W3C Internationalization Tag Set (ITS)\n**ITS 1.0** (W3C Recommendation, April 2007) defines six data categories \u2014 Translate, Localization Note, Terminology, Directionality, Language Information, Elements Within Text \u2014 via global rules (`its:rules` with XPath selectors) or local attributes [13]. **ITS 2.0** (W3C Recommendation, October 2013) adds ten more (16 total), including Domain, Text Analysis, Locale Filter, Preserve Space, Localization Quality Issue, and Provenance, with native HTML5 support (`translate`/`dir` attributes, `its-*` local attributes) [14]. Its practical effect: tools can **declaratively identify translatable vs. protected content** (e.g., `translate=\"no\"`, script blocks, terminology) upstream of translation.\n\n### 3.2 DNT lists and termbases (GALA / ISO 17100)\nGALA's guidance centers on **linguistic asset management**: build termbases with approved, banned, and conditional terms validated with client SMEs before kickoff; maintain **\"Do Not Translate\" (DNT) lists** for brand names, product names, trademarks, UI strings, and code identifiers; decide per locale whether proper nouns stay in source script or are transliterated \u2014 and record the decision [15]. ISO 17100:2015 mandates that terminology/glossaries, style guides, and reference materials be agreed **before production begins**, that client-supplied termbases be used with gaps flagged, and that source ambiguities (inconsistent names, unclear numbers) be **queried to the client rather than silently \"corrected\"** [16].\n\n---\n\n## 4. Structural Carriers: XLIFF Inline Tags and ICU MessageFormat\n\n### 4.1 XLIFF\n**XLIFF 1.2** (OASIS, February 2008) embeds five inline elements in ``/``: ``/`` (paired native codes, matched via `rid`/`id`), `` (standalone placeholder), `` (isolated begin/end), and `` (translatable content inside a code), with `ctype` and `equiv-text` attributes [17]. **XLIFF 2.x** (2.0 in 2014; 2.1 in 2019; 2.2 in 2020) redesigns the model: ``, `` (paired code containing content), ``/``, ``/``; native codes stored out-of-line in `` referenced by `dataRef`; and behavior controls \u2014 **`canCopy`, `canDelete`, `canReorder`** \u2014 that machine-enforce what a translator may do with a tag [18][19][20]. Android `strings.xml` embeds `` tags to mark non-translatable string parts [18].\n\n### 4.2 CAT-tool placeholder protection\nFormat filters convert native markup \u2014 HTML/XML tags, printf (`%s`, `%1$s`), .NET (`{0}`), Java/ICU (`{name}`), Apple (`%@`), `${var}` \u2014 into **atomic inline tokens before segmentation**, so tokens are never split and translators cannot edit inside them. Typical protections: locked tags, insertion via shortcut, open/close pairing enforcement, auto-propagation, and QA verification for missing, extra, reordered, or empty tags (Trados Studio tag verification, memoQ QA, OmegaT validation, Phrase tag QA, Crowdin/Lokalise variable highlighting) [5][6][7].\n\n### 4.3 ICU MessageFormat\n**MessageFormat 1** syntax `{arg, type, style}` supports `number`, `date`, `time`, `spellout`, `ordinal`, `duration`, plus plural/select with CLDR categories and `#` for the formatted number [21]. **MessageFormat 2** \u2014 tech preview in ICU 75 (2024), **stable in ICU 76 (October 2024)** \u2014 adds declarations (`.input`, `.local`), `.match` selection, function placeholders (`{$count :number style=integer}`), defined error/fallback behavior, and **first-class markup parts** (open/close placeholders) so inline tags can be represented without breaking message syntax [22][23].\n\n---\n\n## 5. MT Pipelines: Why Models Corrupt Preserved Content and How They Are Fixed\n\n### 5.1 Documented failure modes\n- **Numbers:** Early NMT mapped rare tokens to ``; Luong et al. (2015) categorized rare-word errors (names, numbers) and showed post-hoc fixes recovering up to **\u2248 +2.8 BLEU** on WMT'14 [24]. BPE subword models (Sennrich et al., 2016) solved open vocabulary but split digit strings into **arbitrary fragments \u2192 dropped, duplicated, or mistranslated digits**, with verbatim copying even where locale conventions differ [25][26]. Koehn & Knowles (2017) listed rare words among six persistent NMT challenges [26]; Spithourakis & Riedel (2018) confirmed digit sequences are poorly modeled without format-aware treatment [27].\n- **Names:** UNK omission, hallucination, inconsistent transliteration, or copying when an established target-language exonym exists; strong sensitivity to character-level noise (Belinkov & Bisk, 2018) [26][28].\n- **Markup:** Models trained on plain text **delete, duplicate, misorder, or misnest tags** and corrupt placeholders; long-range reordering makes positional preservation hard [2][29].\n\n### 5.2 Architectural mitigations\nThe literature converges on a layered recipe [24][25][29][30]:\n\n1. **Copy/pointer mechanisms** \u2014 CopyNet (Gu et al., 2016), pointer-softmax (Gulcehre et al., 2016), pointer-generator (See et al., 2017); Umugwaneza et al. (AFRICON 2023) showed copy-augmented models improve retention of named entities, numbers, and OOV words for Kinyarwanda\u2192English [31][32][33][34].\n2. **NER-driven placeholder masking** \u2014 detect entities, replace with type-tagged unique placeholders, re-insert after translation; Bunk et al. (2019) reported BLEU gains and fewer entity errors for English\u2192German; Hoang et al. (2018) used entity masking during back-translation to force copying [35][36].\n3. **Hard terminology constraints** \u2014 lexically constrained decoding (Hokamp & Liu 2017; Post & Vilar 2018) and training models to apply glossaries via placeholder-style tokens (Dinu et al., 2019) [37][38][39].\n4. **Retrieval** \u2014 kNN-MT (Khandelwal et al., 2021) improves rare-token and entity consistency from translation memories [40].\n5. **Numeric post-validation** \u2014 no architectural fix fully eliminates digit and markup errors [27][2].\n\n### 5.3 Commercial-engine features\nProduction engines operationalize these ideas as **glossaries/DNT enforcement** (Google Cloud Translation, Microsoft Translator, DeepL, ModernMT) \u2014 the pipeline-level equivalent of the DNT lists and constrained decoding described above [15][37][39].\n\n---\n\n## 6. Enforcement: QA Workflows That Catch Violations\n\n### 6.1 Automated QA checks\nAutomated QA runs on bilingual files or TMS segments, flagging mechanical errors [5][6][7]:\n\n- **Tag/formatting mismatch** \u2014 missing, extra, duplicated, or reordered inline tags between source and target.\n- **Number mismatch** \u2014 compares numeric values source vs. target to catch dropped, altered, or mistranslated figures; advanced checkers **normalize for locale conventions** (decimal comma vs. point, digit grouping) to reduce false positives.\n- **Additional checks** \u2014 untranslated segments, source=target, terminology violations, forbidden characters, whitespace, capitalization, punctuation, URL/email integrity, length limits.\n\nStandard tools: **XBench** (de-facto industry standard, free), Trados Studio QA Checker, memoQ QA, Verifika, QA Distiller, and built-in QA in cloud TMS platforms (Phrase, Smartling, Crowdin, Lokalise, Transifex). Checks run (a) continuously as translators type, (b) as a gate before delivery to reviewers, and (c) as a final pre-release batch check, with errors triaged as real vs. false positive and re-run until zero critical errors remain [5][6][7].\n\n### 6.2 MQM: making preservation auditable\nThe **Multidimensional Quality Metrics** framework makes preservation failures countable via explicit issue types [1][2][41]:\n\n| Failure | Full MQM tree | MQM 2021 revision |\n|---|---|---|\n| Wrong entity | Accuracy \u2192 Mistranslation; Locale convention \u2192 Person names | **Naming**: \"incorrect name for a person, place, or organization\" |\n| Wrong value | Accuracy \u2192 Mistranslation | **Number**: \"numbers that do not match the source\" |\n| Wrong format only | Locale convention \u2192 Number format | Locale convention |\n| Broken tags | Design \u2192 **Markup** | Design |\n\n**Severity is orthogonal to category**: Critical (renders text unusable; risk of physical, financial, legal, or reputational harm), Major (significantly disrupts comprehension/usability), Minor (detracts but does not impede), Neutral (zero penalty). Widely used default penalty weights are **Minor = 1, Major = 5, Critical = 25**; the WMT MQM protocol scores Minor = 1, Major = 5, treating Critical errors as extremely rare [41][42][43]. TAUS's Dynamic Quality Framework contributes a harmonized typology with matching categories (terminology, accuracy, locale convention) [44].\n\n### 6.3 Human LQA and sampling\nLQA is performed by a second linguist \u2014 the ISO 17100:2015 **two-person workflow** requires every translation to be checked by a second qualified linguist, explicitly covering accuracy, completeness, and terminology consistency [16]. Reviewers annotate errors with type and severity, apply weighted deductions, and compute a score (pass thresholds commonly around **95\u201399 points out of 100**); critical errors may trigger automatic failure regardless of total score. Because full review is expensive, LQA often uses sampling, with 100% review reserved for high-risk content (legal, medical, UI strings shipped to millions of users). Results feed back into training, glossary/style-guide updates, and MT post-editing rules [41][43][44].\n\n### 6.4 Pseudo-localization and regression testing\nPseudo-locales machine-transform source strings **before real translations exist** [45][46][47]:\n\n- **Accented pseudo** (`en-XA`): maps ASCII to accented/exotic characters \u2014 reveals encoding bugs, missing glyphs, hardcoded strings.\n- **Expansion pseudo:** pads strings **~30\u201340% longer** (matching German/French growth), wrapped in delimiters (`[!!! H\u00ebll\u00f4 !!!]`) \u2014 reveals truncation, overflow, and **concatenation bugs** (brackets appearing mid-sentence).\n- **Pseudo-bidi** (`ar-XB`): substitutes RTL-looking characters and inserts directional marks \u2014 surfaces bidi/layout/mirroring bugs.\n\nPseudo runs verify that placeholders (`{0}`, `%s`, `{{name}}`) and ICU patterns survive intact. Xcode ships Double-Length, Accented, and Right-to-Left pseudolanguages; Android exposes en-XA/ar-XB in developer options [45][47]. **Limits:** pseudo-locales still use English plural rules and don't test real Arabic shaping or Russian plural categories \u2014 complement with snapshot tests against real locales (de-DE, ar-EG, ru-RU, ja-JP, hi-IN) [45].\n\nPost-integration regression testing verifies per-language builds: UI strings fit containers, variables render at runtime, hotkeys don't collide in translated labels, and localized resources load without missing-key errors \u2014 automated via screenshot comparison per locale, locale-parameterized UI frameworks (Selenium, Appium, Playwright), string-length assertions, and untranslated-English \"leak\" detection, gated in CI/CD [48][49].\n\n---\n\n## 7. Consolidated Best-Practice Checklist\n\n**Named entities**\n- Inventory entity classes; decide keep/transliterate/translate **per locale** and record the decision [15][3].\n- Store as DNT or termbase entries; load glossary with DNT flags in the CAT tool; copy strings exactly (casing, spacing, hyphenation) [15][50].\n- Verify against authoritative sources; auto-check DNT strings; watch for false look-alikes (\"Finder,\" \"Edge,\" \"Swift\") [50].\n\n**Numbers**\n- Adapt *format* to locale (per CLDR) but never alter *values*; diff source vs. target digit-by-digit [9][50].\n- Apply correct separators, currency symbol/position/decimal digits (JPY = 0, KWD = 3); disambiguate `$` with ISO code where specified [10][50].\n- Protect numbers that must not be reformatted (version numbers, serials, phone numbers, postal codes, legal references); disable autoformat interference [50].\n- Use full CLDR plural categories \u2014 never concatenate number + fixed noun [10][12].\n\n**Markup**\n- Preserve all inline tags; tag count and pairing must match source even when position shifts; never break nesting order \u2014 orphan tags are the #1 cause of build failures [5][50].\n- Translate only translatable attribute values (`alt`, `title`, `aria-label`); never touch element/attribute names, `id`/`class`, or ``/`` content; keep character entities (`&`, ` `, `é`) intact [50].\n- Protect XLIFF ``, ``, `/` pairs, and `resname`/`rid` values [18][50].\n- Verify by rendering and well-formedness validation \u2014 never approve from the bilingual file alone [50].\n\n**Process and governance**\n- Agree terminology/glossaries and style guides before production; query source ambiguities rather than guessing; enforce the second-linguist check (ISO 17100) [16].\n- Verify entities, numbers, dates, and terminology in all MT post-editing, even light post-editing (ISO 18587) [4].\n- Gate releases: pseudo-localization in CI \u2192 automated QA (tags, numbers, DNT, terminology) \u2192 MQM-scored LQA \u2192 localized build regression [5][41][45][48].\n\n---\n\n## 8. Conclusion\n\nPreservation of entities, numbers, and markup is the point where localization quality is most binary: it either holds, or the localized product breaks. The research is unanimous on the remedy's shape \u2014 **declare** protected content upstream (ITS, DNT lists, termbases), **carry** it losslessly through the pipeline (XLIFF inline codes with `canCopy`/`canDelete`/`canReorder` semantics, atomic placeholder tokens, ICU MessageFormat), **translate around it** with copy mechanisms, entity masking, and constrained decoding, and **prove** it survived with automated tag/number checks, MQM-scored LQA, pseudo-localization, and per-locale regression testing. The distinction to institutionalize everywhere is the one that underlies the entire rule set: *values, identities, and structure are immutable \u2014 only their locale presentation adapts.*\n\n---\n\n## References\n\n[1] The MQM 2021 Revision \u2014 official issue-type list and definitions, themqm.org\n[2] MQM full typology (metric definition tool, ~100 types incl. Design\u2192Markup, Locale convention\u2192Number format), themqm.org\n[3] CSA Research \u2014 reports on terminology management, Localization Maturity Model, global brand naming, global readiness\n[4] ISO 18587:2017, *Post-editing of machine translation output*\n[5] XBench \u2014 QA checks documentation (tag integrity, number consistency, terminology, inconsistency)\n[6] Phrase (Memsource) Documentation \u2014 \"QA checks\" and \"Translation Quality\"\n[7] SDL/RWS Trados Studio Documentation \u2014 QA Checker / Verification settings; memoQ QA; Verifika; Lokalise/Crowdin docs\n[8] Unicode CLDR project & release notes \u2014 cldr.unicode.org\n[9] UTS #35: Unicode Locale Data Markup Language (LDML) \u2014 unicode.org/reports/tr35\n[10] UTS #35 Part 3: Numbers (tr35-numbers) \u2014 number patterns, currency data, plural rules\n[11] UTS #35 Part 4: Dates (tr35-dates) \u2014 calendars, patterns, time zones\n[12] GNU gettext manual \u2014 Plural-Forms / nplurals\n[13] W3C ITS 1.0 Recommendation \u2014 https://www.w3.org/TR/its/\n[14] W3C ITS 2.0 Recommendation \u2014 https://www.w3.org/TR/its20/\n[15] GALA knowledge base \u2014 linguistic assets, terminology management, DNT lists, locale formatting\n[16] ISO 17100:2015, *Translation services \u2014 Requirements for translation services*\n[17] OASIS XLIFF 1.2, Feb 2008 \u2014 https://docs.oasis-open.org/xliff/v1.2/os/xliff-core.html\n[18] OASIS XLIFF 2.0, Aug 2014 \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.0/os/xliff-core-v2.0-os.html\n[19] OASIS XLIFF 2.1, 2019 \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.1/os/xliff-core-v2.1-os.html\n[20] OASIS XLIFF 2.2, 2020 \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.2/os/xliff-core-v2.2-os.html\n[21] ICU User Guide: MessageFormat \u2014 https://unicode-org.github.io/icu/userguide/format_parse/messages/\n[22] MessageFormat 2.0 specification (unicode-org/message-format-wg) \u2014 https://unicode.github.io/message-format-wg/\n[23] ICU 76 release notes \u2014 https://icu.unicode.org/download/76\n[24] Luong, Pham & Manning (2015). *Addressing the Rare Word Problem in Neural Machine Translation.* ACL\n[25] Sennrich, Haddow & Birch (2016). *Neural Machine Translation of Rare Words with Subword Units.* ACL\n[26] Koehn & Knowles (2017). *Six Challenges for Neural Machine Translation.* WNMT@ACL\n[27] Spithourakis & Riedel (2018). *Numeracy for Language Models.* ACL\n[28] Belinkov & Bisk (2018). *Synthetic and Natural Noise Both Break Neural Machine Translation.* ICLR\n[29] Lommel, Uszkoreit & Burchardt (2014). *Multidimensional Quality Metrics (MQM).* Tradum\u00e0tica 12\n[30] Lee, Cho & Hofmann (2017). *Fully Character-Level Neural Machine Translation without Explicit Segmentation.* TACL\n[31] Gu, Lu, Li & Li (2016). *Incorporating Copying Mechanism in Sequence-to-Sequence Learning.* ACL\n[32] Gulcehre et al. (2016). *Pointing the Way to Effective Translation.* ACL\n[33] See, Liu & Manning (2017). *Get To The Point: Summarization with Pointer-Generator Networks.* ACL\n[34] Umugwaneza, Niyomwungeri & Nshimiyimana (2023). *Enhancing Kinyarwanda\u2013English NMT with Copy Mechanisms.* IEEE AFRICON\n[35] Bunk, Halbach, H\u00e4mmerl & Vu (2019). *Improving Neural Machine Translation for German with Named Entity Handling*\n[36] Hoang, Koehn, Haffari & Cohn (2018). *Iterative Back-Translation for Neural Machine Translation.* WNMT@ACL\n[37] Hokamp & Liu (2017). *Traversing Oceans of Text: Terminology Constraints in Neural Machine Translation.* ACL\n[38] Post & Vilar (2018). *Fast Lexically Constrained Decoding with Dynamic Beam Allocation for NMT.* NAACL\n[39] Dinu, Mathur, Federico & Al-Onaizan (2019). *Training Neural Machine Translation to Apply Terminology Constraints.* ACL\n[40] Khandelwal, Fan, Jurafsky, Zettlemoyer & Lewis (2021). *Nearest Neighbor Machine Translation.* ICLR\n[41] Freitag et al. (2021). *Experts, Errors, and Context: Evaluating Human Performance for Machine Translation.* TACL 9\n[42] TAUS DQF\u2013MQM harmonized error typology\n[43] TAUS Dynamic Quality Framework (DQF) \u2014 error typology and MT quality evaluation guidance\n[44] CSA Research / industry publications on LQA programs \u2014 sampling, thresholds, vendor scorecards\n[45] Android Developers: Test with pseudo-locales (en-XA, ar-XB); Chromium L10n guides\n[46] Microsoft Internationalization documentation \u2014 pseudo-localization guidance\n[47] Apple Developer: Xcode preview pseudolocalization options\n[48] Smartling / Phrase blog & docs \u2014 localization testing, pseudo-localization, CI/CD integration\n[49] Selenium / Appium / Playwright documentation \u2014 locale-parameterized UI test automation\n[50] Microsoft Style Guides (per-language); Apple Localization and Style Guide; ICU formatting rules; LISA QA Model 1.0\n\n*Note: Some citation details for the NMT literature (notably [34] and [35]) were reconstructed from research summaries and should be verified against ACL Anthology/IEEE Xplore before formal publication.*\n\n### 0009 [succeeded]\n# Localizing Formats, Pluralization, and Gender Agreement for Brazilian Portuguese (pt-BR)\n\nBrazilian Portuguese is a high-resource locale with well-established conventions, but it has several traps for engineers and writers coming from English: the decimal comma, a \"hidden\" `many` plural category, and a grammatical system in which every noun, article, and adjective carries gender. This report covers the format, pluralization, and gender-agreement rules you need, with CLDR/ICU specifics and style-guide guidance.\n\n---\n\n## 1. Number, Currency, and Percent Formats\n\n### 1.1 Numbers\n\n| Item | pt-BR convention | Example |\n|---|---|---|\n| Decimal separator | **comma** | `3,14` |\n| Thousands separator | **period** | `1.234.567,89` |\n| Grouping | threes, no special-casing of 4-digit years | `2026` (not `2.026`) |\n| Digits | Western (`latn`) | \u2014 |\n| CLDR decimal pattern | `#,##0.###` | \u2014 |\n\nNumber words use the **short scale**: *mil* (10\u00b3), *milh\u00e3o* (10\u2076), *bilh\u00e3o* (10\u2079), *trilh\u00e3o* (10\u00b9\u00b2) [1][3].\n\n### 1.2 Currency (BRL)\n\n- Symbol: **R$** (ISO 4217: BRL, numeric 986); subunit: *centavos* (1/100) [4].\n- Symbol goes **before** the amount, separated by a (non-breaking) space: `R$ 1.234,56`. CLDR pattern: `\u00a4 #,##0.00` [1][4]. Informal writing sometimes drops the space; implementations should keep it.\n- **Negative currency:** minus sign **before the symbol** \u2014 `-R$ 1.234,56` (CLDR: `-\u00a4 #,##0.00`). Parenthesized accounting negatives `(R$ 1.234,56)` appear in spreadsheets but are not the locale standard; `R$ -1.234,56` is a non-conformant variant [1][3].\n\n### 1.3 Percent\n\n- Sign **follows** the number, **no space**: `50%`, `12,5%`. CLDR: `#,##0%`; negative: `-50%` [1][3].\n- Financial rates often take qualifiers: `12,5% a.a.` (per annum), `1,2% a.m.` (per month) [4].\n\n---\n\n## 2. Date and Time Formats\n\n### 2.1 Date patterns\n\n| Style | Pattern | Example |\n|---|---|---|\n| Short | `dd/MM/yyyy` | 25/09/2026 |\n| Medium | `d 'de' MMM 'de' yyyy` | 25 de set. de 2026 |\n| Long | `d 'de' MMMM 'de' yyyy` | 25 de setembro de 2026 |\n| Full | `EEEE, d 'de' MMMM 'de' yyyy` | sexta-feira, 25 de setembro de 2026 |\n\nDay/month/year with slashes is the universal everyday format; the \"de\" connectors are mandatory in written-out dates. Use **cardinal** numbers for the day (*dia 25*, never \"vig\u00e9simo quinto\"), and **lowercase** month/weekday names in running text [1][4].\n\n### 2.2 Time\n\n- CLDR standard patterns are **24-hour**: `HH:mm`, `HH:mm:ss`. The full form uses the characteristic notation **HH'h'mm'min'ss's'** (e.g., `14h30min25s`) [1][4].\n- The \"h\" separator (`14h30`, `9h15`) is the dominant written convention; 12-hour speech forms (\"\u00e0s 3 da tarde\") exist but formal/technical output is 24-hour.\n- **First day of week: Sunday** (*domingo*) per CLDR territory data for BR \u2014 which is why Monday is *segunda-feira* (\"second day\") [2].\n- **Time zone:** Bras\u00edlia Time (UTC\u22123) is the national reference; the country spans UTC\u22122 to UTC\u22125. **DST was abolished in 2019** (Decreto n\u00ba 9.772/2019) \u2014 do not ship DST logic [5].\n- **ISO 8601** is adopted nationally as ABNT NBR ISO 8601 and is standard in technical/data-exchange contexts (`2026-09-25`), but `dd/mm/yyyy` prevails in daily life [3].\n\nMonth names: *janeiro \u2026 dezembro* (abbrev. *jan., fev., mar., abr., mai., jun., jul., ago., set., out., nov., dez.*). Weekdays: *domingo, segunda-feira \u2026 s\u00e1bado* (abbrev. *dom., seg., ter., qua., qui., sex., s\u00e1b.*) [1].\n\n---\n\n## 3. Pluralization\n\n### 3.1 Cardinal rules (CLDR)\n\nCLDR has **no separate plural data for pt-BR** \u2014 it inherits from root `pt`, and `pt`, `pt-PT`, and `pt-BR` share identical cardinal rules [1][3]. Categories: **one, many, other** \u2014 the `many` category was added in **CLDR 42 / ICU 72 (Oct 2022)** for compact/million-scale numbers [7].\n\n```xml\n\n  i = 0..1\n  e = 0 and i != 0 and i % 1000000 = 0 and v = 0\n                           or e != 0..5\n\n```\n\nOperand meanings (UTS #35): `i` = integer part, `v` = visible fraction digits, `e` = compact exponent [4].\n\n| Number | Category | Why |\n|---|---|---|\n| 0 | **one** | i = 0 |\n| 1 | one | i = 1 |\n| 0,5 / 1,5 | **one** | i \u2264 1 |\n| 2 / 100 / 2,5 | other | |\n| 1.000.000 | **many** | multiple of 10\u2076, no fraction |\n| 1.000.001 | other | not a multiple of 10\u2076 |\n| 1M (compact) | many | e = 6 |\n\n**Gotchas:** Portuguese selects **\"one\" for 0** and for **1,5** (\"1,5 item\" \u2014 singular below 2). Add an explicit `=0` selector if you need special zero handling. The `many` category requires CLDR 42+/ICU 72+ runtimes; older engines silently never select it [7].\n\n### 3.2 Ordinals\n\nOnly `n = 1` \u2192 **one** (1\u00ba = *primeiro*); everything else is **other** (2\u00ba, 3\u00ba, 11\u00ba). Note that the \u00ba/\u00aa **gender suffix is not captured by plural categories** \u2014 see \u00a74 [2][3].\n\n### 3.3 ICU MessageFormat\n\n```\n{n, plural,\n  =0   {nenhum resultado}\n  one  {# resultado}\n  many {# de resultados}\n  other {# resultados}}\n```\n\nOutputs: `1 \u2192 \"1 resultado\"` \u00b7 `2 \u2192 \"2 resultados\"` \u00b7 `1,5 \u2192 \"1,5 resultado\"` \u00b7 `1.000.000 \u2192 \"1.000.000 de resultados\"` (note the *de* \u2014 \"um milh\u00e3o **de** resultados\"). `#` renders the number locale-formatted (\"1,5\") [5].\n\n```js\nnew Intl.PluralRules('pt-BR').select(1);        // \"one\"\nnew Intl.PluralRules('pt-BR').select(1000000);  // \"many\"\nnew Intl.PluralRules('pt-BR', {type:'ordinal'}).select(1); // \"one\"\n```\n\nFor ordinals: `{pos, selectordinal, one {#\u00ba lugar} other {#\u00ba lugar}}`. One more subtlety: pt-PT differs from pt only in **pluralRanges** (how ranges like \"1\u20133\" combine), not in cardinal/ordinal categories [6].\n\n---\n\n## 4. Grammatical Gender Agreement\n\nPortuguese has **no neuter**: every noun is masculine or feminine, and articles, adjectives, and past participles must agree. This is the hardest part of pt-BR localization after formatting.\n\n### 4.1 Noun gender rules\n\n- **Masculine:** most `-o` words (*o livro*); Greek-origin `-ma` (*o problema, o sistema, o idioma*); most `-or` (*o escritor*).\n- **Feminine:** most `-a` (*a casa*); `-\u00e7\u00e3o/-s\u00e3o, -dade, -tude, -gem` (*a na\u00e7\u00e3o, a cidade, a viagem*).\n- **Unpredictable endings must be memorized:** *o mar, o l\u00e1pis* vs. *a flor, a cor, a raiz*; most `-\u00e3o` words are masculine (*o cora\u00e7\u00e3o*) but *a m\u00e3o, a raz\u00e3o* are feminine [1][3][7].\n- **Common gender (sobrecomum):** one form for both sexes; only the article changes \u2014 *o/a colega, estudante, artista, dentista, c\u00f4njuge*.\n- **Epicene nouns:** single grammatical gender regardless of referent sex \u2014 *a crian\u00e7a, a v\u00edtima, a testemunha, a pessoa*. Modifiers **stay feminine even for a male referent** (*a v\u00edtima ficou ferida*) [1][2].\n- **Meaning changes with gender:** *o capital* (money) / *a capital* (city); *o r\u00e1dio* (device) / *a r\u00e1dio* (station); *o grama* (weight) / *a grama* (grass) [1][7].\n- **Compound nouns:** verb+noun compounds are masculine (*o guarda-chuva*); noun+noun compounds take the head's gender (*a palavra-chave, o carro-chefe*), including inflecting the first element when it's the head: *a segunda-feira \u2192 as segundas-feiras*, *o primeiro-ministro \u2192 a primeira-ministra* [1][6].\n\n### 4.2 Articles, adjectives, and agreement mechanics\n\n- Definite articles *o/os* vs. *a/as*; indefinite *um/uns* vs. *uma/umas*; all determiners agree (*meu/minha, este/esta, todo/toda, cujo/cuja*).\n- **a \u2192 o rule:** the feminine article becomes **o** before feminine nouns beginning with stressed /a/ \u2014 *o \u00e1guia, o asa* \u2014 but adjectives stay feminine (*o \u00e1guia branca*) and demonstratives don't change (*esta \u00e1guia*) [1][2].\n- Adjectives agree in gender and number: two-form adjectives (*alto/alta*) vs. single-form adjectives that vary only in number (*grande, azul, feliz, jovem*).\n- **Compound adjectives:** only the last element inflects (*camisas verde-claras, olhos azul-escuros*); classic exception *surdos-mudos* [1][2].\n- **Multiple nouns:** same gender \u2192 that gender plural; **mixed genders \u2192 masculine plural** (*Jo\u00e3o e Maria s\u00e3o altos*) \u2014 the codified norm [1][2].\n- Special cases: *meio* invariable as adverb (*elas est\u00e3o meio cansadas*) but agreeing as adjective (*meia hora*); *obrigado/obrigada* agrees with the **speaker**; past participles agree with *ser/estar* (*as casas foram vendidas*) but stay invariable with *ter/haver* (*elas tinham chegado*) [1][2][6].\n- **Collectives are grammatically singular:** *A equipe est\u00e1 reunida*, never *A equipe venceram*. With plural complements, singular is the formal norm (*A maioria dos alunos chegou*), though plural \"notional\" agreement is widely accepted (*A maioria dos alunos chegaram*) [1][2][6].\n- ***Pessoas*** is a **feminine epicene plural noun, not a collective**: *as pessoas chegaram; todas estavam felizes* \u2014 agreement is always plural and feminine [1][3][7].\n\n### 4.3 Gender-inclusive language in pt-BR localization\n\nBecause there is no neuter, neutralization strategies are: rephrasing with genderless words, non-standard neologisms, or the generic masculine (grammatically standard but increasingly contested) [1][3][6].\n\n**Avoid in product text:**\n- **'x' and '@' endings** (*todxs, tod@s, bem-vindx*): no settled spoken form; screen readers (NVDA, JAWS, VoiceOver) read them as \"xis\"/\"arroba,\" breaking comprehension. Microsoft and Google pt-BR guidance treats them as non-standard and excludes them from UI and docs [1][3][4][7][9].\n- **Slash/parenthesis pairs** (*o/a usu\u00e1rio(a)*, *prezado(a)*): clunky and screen-reader-hostile [1][3][5][9].\n- **'elu' neopronoun** (subject *elu*, oblique/possessive *delu*, plural *elus/delus*; 3rd-person-singular verb agreement): recognized in some dictionaries (Aulete 2021; Michaelis online 2023) and common in community-facing content, but **not used in mainstream product UI**. Respect stated pronouns for real people/personas; otherwise rephrase [7][8].\n\n**Preferred neutral strategies (aligned with Microsoft, Google, and the Brazilian federal plain-language guide):**\n1. **Direct address + imperative** \u2014 pt-BR UI naturally avoids pronouns: *\"Clique em Salvar\"*. This is the default for UI copy [1][3].\n2. ***Pessoa* constructions** \u2014 *a pessoa usu\u00e1ria, a pessoa desenvolvedora, as pessoas clientes* (used in federal government plain-language guidance) [6].\n3. ***Quem*-clauses** \u2014 \"quem usar o servi\u00e7o\" instead of \"o usu\u00e1rio que usar.\"\n4. **Anchor agreement to a conventionally \"neutral\" masculine noun** \u2014 *o cadastro foi conclu\u00eddo*, *o acesso foi liberado* (agreement attaches to *cadastro/acesso*, not a person).\n5. **Fixed neutral expressions** \u2014 *Boas-vindas!* instead of *Bem-vindo(a)*; *Ol\u00e1, {nome}!* instead of *Prezado(a)*.\n6. **Genderless nouns** \u2014 *a equipe, a lideran\u00e7a, o grupo*.\n7. Fall back to the **generic masculine** only when no neutral alternative works [1][2][3].\n8. Note: possessives (*seu/sua*) agree with the **possessed noun**, not the owner \u2014 inherently neutral, no action needed. Also be aware \"linguagem neutra\" is legally contested in Brazil (several states/municipalities restricted it in public schools in 2023\u20132024), so public-sector-facing content needs care.\n\n---\n\n## 5. Adjacent Conventions (Quick Reference)\n\n- **Units:** metric (km, kg, L, \u00b0C); fuel in km/L; inches only for TV screens and tires. Numbers: `1.234,56 kg` [1][2].\n- **Phones:** +55, two-digit DDD in parentheses; mobile 9 digits starting with 9 \u2014 `(11) 91234-5678`; landline 8 digits \u2014 `(11) 2345-6789` [4].\n- **Addresses:** street + number, complement, bairro, **CEP NNNNN-NNN** + city \u2013 state (UF): `Av. Paulista, 1578 \u2013 Bela Vista, S\u00e3o Paulo \u2013 SP, 01310-200, Brasil` [5].\n- **Names:** given name(s) + mother's surname + father's surname (paternal last, primary for sorting); titles Sr./Sra./Dra. [2][3].\n- **Sorting:** 26-letter alphabet (K, W, Y since 2009); accents are secondary differences, \u00e7 sorts as c; digraphs ch/lh are not separate letters [1][6].\n- **Typography:** curly double quotes \u201c \u201d (guillemets are rare in Brazil); ordinals with **\u00ba/\u00aa** (1\u00ba lugar); dialogue dashes are em dashes (travess\u00e3o); sentence case for headings; expect **15\u201325% text expansion** vs. English [1][3][6][7].\n\n---\n\n## 6. Implementation Checklist\n\n| Concern | Do | Don't |\n|---|---|---|\n| Numbers | `1.234,56` | `1,234.56` |\n| Currency | `R$ 1.234,56` / `-R$ 1.234,56` | `R$-1.234,56`, `(R$ 1.234,56)` |\n| Percent | `12,5%` | `12.5 %`, `%12,5` |\n| Dates | `25/09/2026`; lowercase month names | `09/25/2026`, Title Case months |\n| Time | 24h `14h30` / `14:30`; no DST | 12h AM/PM in formal UI |\n| Plurals | `one`/`many`/`other` with ICU 72+; `=0` override if needed | Assuming English-style 0 = \"other\"; forgetting \"1,5 item\" |\n| Millions | `1.000.000 de resultados` | `1.000.000 resultados` |\n| Gender | Imperatives, *pessoa usu\u00e1ria*, *Boas-vindas!* | `o(a) usu\u00e1rio(a)`, `todxs`, `bem-vindx` in product text |\n| Agreement | Mixed groups \u2192 masculine plural; *a pessoa* \u2192 feminine | *A equipe venceram*; *as pessoas chegou* |\n\n---\n\n## Sources\n\n1. Unicode CLDR locale data for pt/pt-BR \u2014 number, currency, percent, date/time patterns (cldr.unicode.org)\n2. CLDR Supplemental Data \u2014 week data, ordinals, pluralRanges (cldr.unicode.org)\n3. CLDR Language Plural Rules chart; UTS #35 LDML Part 3 (tr35-numbers)\n4. Banco Central do Brasil \u2014 BRL specifications and official currency usage (bcb.gov.br)\n5. ICU (International Components for Unicode) documentation \u2014 MessageFormat/PluralFormat\n6. ABNT NBR ISO 8601 / NBR 6023; Decreto federal n\u00ba 9.772/2019 (DST abolition)\n7. CLDR 42 release notes (`many` category addition); Microsoft Language Portal \u2014 Portuguese (Brazil) Style Guide\n8. Google Developer Documentation Style Guide; Google pt-BR localization style guidance\n9. W3C Editorial Style Guide and accessibility guidance; *Guia de Comunica\u00e7\u00e3o em Linguagem Simples*, Governo Federal do Brasil (2023)\n10. Cunha & Cintra, *Nova Gram\u00e1tica do Portugu\u00eas Contempor\u00e2neo*; Bechara, *Moderna Gram\u00e1tica Portuguesa*; Perini, *Modern Portuguese*; Whitlam, *Modern Brazilian Portuguese Grammar*\n11. ANATEL numbering plan; Correios CEP/addressing standards; *Guia de M\u00eddia*, Alian\u00e7a Nacional LGBTI+ (2021)\n\n**Caveat:** Corporate style guides (Microsoft, Google) are updated frequently \u2014 verify current wording before citing them in deliverables.\n\n### 0010 [succeeded]\nAll research is complete. Here is the consolidated report.\n\n---\n\n# Normalizing Script, Bidi, and Unicode Issues in Localized Text \u2014 and Stress-Testing with Noise and Code-Switching\n\n## 1. Executive summary\n\nLocalized text fails in three distinct layers, and each layer requires its own normalization pass and its own stress-test suite:\n\n1. **Unicode/code-point layer** \u2014 mojibake, denormalized sequences, zero-width/control characters, fullwidth/halfwidth variants. Fixed with an ordered pipeline: decode \u2192 repair (ftfy) \u2192 normalize (NFC) \u2192 strip invisibles \u2192 clean controls \u2192 fold case \u2192 re-normalize \u2192 grapheme-safe truncation [1][2][6].\n2. **Script layer** \u2014 mixed-script confusables (Cyrillic/Greek homoglyphs), width variants, script restriction policy. Fixed with UTS #39 skeleton matching and restriction levels, plus NFKC-based width folding in derived matching layers only [4][5].\n3. **Direction layer** \u2014 wrong base direction, neutral-character scrambling, interpolation leaks. Fixed with first-strong detection, isolates (FSI/PDI) over deprecated embeddings, and logical-order storage end-to-end [9][13][14].\n\nCritically, **normalization alone fixes none of the security or robustness problems**: NFC/NFKC do not remove zero-width or tag characters, do not unify homoglyphs, and do not neutralize bidi overrides [1][5]. And robustness testing shows that **synthetic-noise robustness does not transfer to natural noise** [20], while code-switched and romanized text degrades multilingual models monotonically with mixing density [30][31].\n\n---\n\n## 2. The normalization pipeline (ordered)\n\n### 2.1 Canonical step order\n\nOrder matters because several steps *denormalize* text that earlier steps fixed. The recommended sequence [1][2][6]:\n\n| # | Step | Rationale |\n|---|------|-----------|\n| 1 | **Decode bytes** (strict UTF-8; strip BOM via `utf-8-sig`) | Normalization operates on decoded text only |\n| 2 | **Mojibake repair (ftfy)** | Normalization cannot fix wrong-decoded bytes; ftfy must run first [6] |\n| 3 | **Unicode normalization (NFC)** | Canonical default for storage and display [1][3] |\n| 4 | **Zero-width/invisible policy** (targeted code-point removal) | NFC does *not* remove these \u2014 no decomposition mappings [1][5] |\n| 5 | **Control-character cleanup** (CRLF/LS/PS\u2192LF; strip C0/C1/noncharacters) | C1 bytes are usually cp1252-misdecode artifacts [6] |\n| 6 | **Entity decoding + whitespace collapse** | Entities can inject denormalized sequences |\n| 7 | **Case folding \u2192 re-normalize** | Folding can denormalize (U+1E9B\u2192U+1E61; U+0345\u2192\u03b9) [2] |\n| 8 | **Grapheme-aware truncation** (UAX #29) | Never cut between base and combining marks; truncation is a classic denormalizer [8] |\n| 9 | **Validate/QA**: assert `is_normalized`, log ftfy diffs, then hash/dedup/embed | Security/length checks come *after* normalization, which changes lengths |\n\nMinimal Python sketch:\n\n```python\nimport ftfy, unicodedata\ns = ftfy.fix_text(raw, normalization=None)      # 1\u20132: repair, don't normalize yet\ns = strip_invisibles(s)                          # 4: explicit code-point list\ns = unicodedata.normalize(\"NFC\", s)              # 3: normalize\ns = s.casefold(); s = unicodedata.normalize(\"NFC\", s)  # 7: fold + re-normalize\nassert unicodedata.is_normalized(\"NFC\", s)       # 9: fast QC gate\n```\n\nUse `unicodedata.is_normalized` / ICU `Normalizer2.quickCheck` to skip full normalization when Quick-Check properties pass [1][7][8].\n\n### 2.2 Choosing the normalization form\n\n| Form | Use for | Never use for |\n|------|---------|---------------|\n| **NFC** | Storage, interchange, all user-visible text (W3C web default) [3] | \u2014 |\n| **NFD** | Diacritic stripping, linguistic analysis | Storage |\n| **NFKC/NFKD** | Search/matching indexes, identifier profiles (UAX #31), fuzzy dedup | User-visible content \u2014 it is lossy |\n\nNFKC folds \ufb01\u2192fi, \u2461\u21922, \u2163\u2192IV, fullwidth \uff21\uff22\uff23\u2192ABC \u2014 distinctions that matter in CJK and math contexts. Keep NFC originals and apply NFKC only in derived matching layers [1][2].\n\n### 2.3 Zero-width and control characters \u2014 explicit policy, not blanket rules\n\nZWSP (U+200B), ZWNJ (U+200C), ZWJ (U+200D) are all category **Cf**; blanket Cf-stripping breaks emoji ZWJ sequences and Indic/Arabic orthography [1][5]. Use explicit lists:\n\n- **Strip**: ZWSP, WORD JOINER (U+2060), soft hyphen (U+00AD), ZWNBSP (U+FEFF mid-file), LRM/RLM in search indexes.\n- **Preserve**: ZWJ (emoji, Indic shaping), ZWNJ (Devanagari/Kannada/Malayalam/Arabic), VS16 (U+FE0F, \u2764\ufe0f vs \u2764), tag characters (subdivision flags), bidi isolates in RTL UIs.\n- **Reject for interchange**: noncharacters (U+FDD0\u2013FDEF, U+nFFFE/FFFF).\n\n### 2.4 Mojibake repair caveats\n\nftfy handles UTF-8-read-as-cp1252 (\"\u00e2\u20ac\u2122\"\u2192\"'\"), double encoding, C1 controls, HTML entities, JSON escapes, and fullwidth Latin; it normalizes to NFC by default (set `normalization='None'` to separate steps) [6]. Run `fix_and_explain()` and review diffs \u2014 ftfy is heuristic and can occasionally alter legitimate text. It is strongest on Latin/CJK mojibake and is a repair tool, not a substitute for fixing encoding declarations upstream. Apply per line or whole document, never across arbitrary byte-offset chunk boundaries.\n\n---\n\n## 3. Script-layer issues\n\n### 3.1 Mixed-script and homoglyph detection (UTS #39)\n\nThe skeleton algorithm: `skeleton(X) = NFD(casefold(confusables_map(NFD(X))))`. Two strings are confusable iff their skeletons are identical [4]. Key homoglyph sets in confusables.txt: Cyrillic \u0430/\u0441/\u0435/\u043e/\u0440/\u0445/\u0443\u2192a/c/e/o/p/x/y; Greek \u03bf/\u03b1/\u03b5/\u03b9/\u03ba/\u03bd/\u03c1/\u03c4\u2192o/a/e/i/k/n/p/t [4].\n\nDetection procedure: normalize (NFKC_Casefold) \u2192 resolve per-character script sets via *augmented* sets (ScriptExtensions.txt, so digits/punctuation don't false-trigger) \u2192 compute skeleton \u2192 classify as **mixed-script confusable** (e.g., `\u0430dmin` with Cyrillic \u0430, skeleton is pure Latin `admin`) or **whole-script confusable** (all-Cyrillic `\u0441\u043e\u0440\u0435` \u2248 Latin `cope`) [4][5][7]. ICU's SpoofChecker exposes `MIXED_SCRIPT_CONFUSABLE`, `WHOLE_SCRIPT_CONFUSABLE`, `INVISIBLE`, and `RESTRICTION_LEVEL` checks [7].\n\n**Restriction levels** \u2014 pick one at identifier creation and enforce it [4][5]:\n\n| Level | Policy |\n|-------|--------|\n| Inclusion | ASCII + tiny curated set (highest assurance) |\n| Highly Restrictive | Recommended scripts; multi-script only within groups ({Han,Hiragana,Katakana}, {Hangul}) \u2014 good default for global services |\n| Moderately Restrictive | Latin may mix with Recommended scripts except Cyrillic/Greek/Cherokee |\n| Minimally Restrictive | Any Recommended-script mix (Latin+Cyrillic allowed) |\n\nBest practices: persist `skeleton(username)` in an indexed column and compare at registration; reject with a clear error rather than silently mutating; re-validate stored identifiers after each Unicode release (confusables.txt updates annually); never rely on skeleton matching alone \u2014 it is approximate and font-dependent [4][8].\n\n### 3.2 Width folding for CJK\n\nFullwidth ASCII variants (U+FF01\u2013FF5E) carry `` compatibility decompositions, so NFKC folds \uff21\uff22\uff23\u2192ABC, \uff0c\u2192,, fullwidth space\u2192space; it also folds *halfwidth katakana \u2192 fullwidth* (\uff8a\u2192\u30cf) [1][3]. Two traps: (a) ideographic punctuation \u3001\u3002 has **no** compatibility decomposition, so NFKC is not a pure width fold; (b) NFKC over-folds \u2460\u21921, \u3231\u2192(\u682a), \u33a1\u2192m2. Japanese/Chinese search systems therefore use custom width-only folding rather than raw NFKC; IMEs default to fullwidth output, making \u534a\u89d2/\u5168\u89d2 normalization a core JA/ZH localization requirement [3].\n\n### 3.3 IDN/domain spoofing\n\nIDNA2008 makes all compatibility characters (including U+FF01\u2013FF5E) disallowed in registered labels, so \uff50\uff41\uff59\uff50\uff41\uff4c.com cannot exist as a distinct domain [5][7]. Use UTS #46 (nontransitional mode \u2014 the browser standard since Firefox 59 in 2018) for any domain processing, never hand-rolled NFKC [5]. Browsers layer UTS #39 confusable checks and single-script display rules on top, falling back to Punycode display [9][10].\n\n---\n\n## 4. Bidi-layer issues\n\n### 4.1 Base direction \u2014 the single most common failure\n\nThe UBA (UAX #9) converts logical to visual order but does not tell you the *intended* base direction [9]. The P2 first-strong heuristic (scan for first strong L/R/AL character, skipping isolate contents) is implemented by HTML `dir=auto` and FSI [9][13]. Known misfire: RTL text beginning with a Latin brand name, @mention, hashtag, emoji, or digits gets an LTR base incorrectly. Mitigations: strong-type ratio counting, per-locale defaults, or explicit override with a prefixed RLM [9][14].\n\n### 4.2 Isolates over embeddings\n\nUse LRI/RLI/FSI/PDI (U+2066\u20132069, Unicode 6.3) in new code. Unlike legacy embeddings (LRE/RLE, deprecated), isolates treat their content as a separate paragraph \u2014 no directional \"leakage\" across boundaries; they nest and pair like brackets [9]. In HTML prefer markup (`dir=auto`, ``, `unicode-bidi: isolate`) over control characters \u2014 markup survives sanitizers; in plain strings (logs, JS interpolation, databases) wrap interpolated variables in FSI\u2026PDI [13][14]. CSS `unicode-bidi: plaintext` gives per-element auto-detected base direction \u2014 ideal for chat bubbles [13]. Always store **logical order** end-to-end; apply the UBA only at render time.\n\n### 4.3 Bug catalog for mixed LTR/RTL content\n\n1. Wrong base direction for user content \u2192 `dir=auto`/`bdi`/`plaintext` [13][14]\n2. String-interpolation leaks (\"Delete ``?\" in RTL) \u2192 FSI\u2026PDI wrapping [14]\n3. URLs/paths/code in RTL text scramble via neutral `/`, `:`, `.` \u2192 forced LTR isolation\n4. First-strong misfires on leading emoji/digits/@mentions \u2192 ratio heuristic or override\n5. Bracket mismatches across runs \u2192 rule N0 in modern implementations (Unicode 8.0+) [9]\n6. Truncation between FSI and PDI leaks direction into the rest of the UI \u2192 grapheme-boundary truncation, keep control pairs balanced\n7. Sanitizer/pipeline stripping of bidi controls inconsistently \u2192 define an explicit policy\n8. **Trojan Source** (CVE-2021-42574): RLO/LRO in source makes reviewed code differ from executed code \u2192 UTS #55 pairing checks, GCC `-Wbidi-chars`, Rust `text_direction_codepoint_in_literal`, VS Code Unicode highlighting, GitHub bidi banners [15][16][17]\n9. Homoglyph identifier spoofing (CVE-2021-42694) \u2192 UTS #39 restriction levels [4][17]\n10. Terminals implementing only partial UBA scramble mixed-direction logs [9]\n\n### 4.4 Bidi controls in NLP pipelines\n\nBidi controls and zero-width characters **survive normalization** and act as spurious tokens in models; they are also an LLM attack vector \u2014 tag characters (U+E0000\u2013E007F) are invisible in chat UIs but tokenize normally, enabling \"ASCII smuggling\" prompt injection and data exfiltration (demonstrated against Copilot; addressed by Microsoft MSRC in 2024) [15][18][19]. Sanitize inputs *and* outputs; treat retrieved documents as untrusted; never render model output as raw HTML/Markdown [19].\n\n---\n\n## 5. Stress-testing with noise\n\n### 5.1 Noise taxonomy to inject\n\n| Level | Perturbations | Purpose |\n|-------|--------------|---------|\n| Character | Keyboard-adjacent typos, adjacent swaps, insertions/deletions, OCR confusions (l/1, O/0), homoglyphs | Pruthi et al. showed 1\u20132 character edits break classifiers [24] |\n| Casing | lowercase, ALL-CAPS, mixed-case | Tests truecasing/tokenizer assumptions |\n| Whitespace | word concatenation (\"howareyou\"), letter splitting, double spaces | Breaks subword tokenization |\n| Word/sentence | synonym substitution (TextFooler), contextual BERT substitution (BAE), EDA insert/swap/delete, Polyjuice counterfactuals | Semantic-preserving invariance tests |\n| Natural | Wikipedia edit histories, Reddit (MTNT), social media | Synthetic robustness does **not** transfer to natural noise [20] |\n\n### 5.2 Canonical findings and benchmarks\n\n- **Belinkov & Bisk (ICLR 2018)**: both synthetic and natural noise severely degrade NMT; introduced keyboard typos, scrambling, disemvoweling, and Wikipedia-mined natural noise [20].\n- **MTNT** (EMNLP 2018) and **WMT19 Robustness task**: noisy social-media test sets for en\u2013fr/en\u2013ja/en\u2013ko [21][22].\n- **CheckList** (ACL 2020): Minimum Functionality, Invariance, and Directional Expectation tests \u2014 the standard behavioral framework; INV tests with typos/punctuation are directly applicable here [23].\n- **Training-side fixes**: subword regularization and BPE-dropout measurably improve segmentation robustness to casing/whitespace noise [25][26].\n\n### 5.3 Tools\n\n**nlpaug** (KeyboardAug, OcrAug, RandomCharAug, contextual word augmenters), **TextAttack** (~16 attack recipes with QWERTY/homoglyph transformations and USE/BERTScore similarity constraints), **textflint** (58 transformations + 20 subpopulations across 12 tasks, EN+ZH), **AugLy** (casing, homoglyphs, Unicode confusables), **OpenAttack** [12][13][14][19\u2192tool refs].\n\n### 5.4 Metrics and hygiene\n\nReport **clean-vs-noisy deltas** per noise type (\u0394accuracy/F1 for NLU; \u0394BLEU/\u0394chrF/\u0394COMET for MT, sacreBLEU-standardized), **invariance pass rates** (CheckList INV), and for adversarial attacks the **attack success rate with similarity/grammar constraints** \u2014 many attacks produce ungrammatical text, inflating apparent vulnerability, so gate with perplexity/grammaticality checks [23][27][28][29]. Fix seeds; significance-test deltas with paired bootstrap.\n\n---\n\n## 6. Stress-testing with code-switching\n\n### 6.1 Benchmarks\n\n| Resource | Pair(s) | Tasks |\n|----------|---------|-------|\n| **LinCE** (ICLR 2020) | Es\u2013En, Hi\u2013En, Zh\u2013En, Ne\u2013En, Singlish | LID, POS, NER, sentiment, semantic parsing [30] |\n| **GLUE-CoS** | En mixed with Es/Hi/Zh | NLI, sentiment, relatedness, LID [31] |\n| **Hinglish TOP** (EMNLP 2021) | Hi\u2013En | Intent + slot filling [32] |\n| **SentiMix** (SemEval-2020 T9) / Spanglish 2014 | Hi\u2013En / Es\u2013En | Sentiment, token LID [33][34] |\n| **CoSWo 2016** (MSA\u2013Egyptian, CALLHOME) | Arabic diglossia | Token LID [35] |\n\nKey replicated result: multilingual encoders fine-tuned on monolingual data show large drops on CS text, **worsening monotonically as mixing density increases** \u2014 so stratify test sets by Code-Mixing Index (CMI) and I-index [30][36][37].\n\n### 6.2 Transliterated code-switching \u2014 the sharpest stressor\n\nRomanized Hinglish/Banglish is the highest-value stress case: **RomanSetu (2024)** shows multilingual LLMs underperform on Romanized Indic text due to tokenizer fragmentation and script mismatch, and that romanization-aware adaptation (or transliterating input to native script before processing) recovers much of the gap [38]. Resources: **PHINC** (parallel Romanized-Hinglish\u2194English, ~30k pairs) [39], **HingBERT/HingMBERT/HingRoBERTa** (beat mBERT on Hinglish tasks) [40], **Dakshina** (romanized\u2194native parallel for 12 South Asian languages) [41]. LLM-specific failure modes: models tend to \"normalize\" CS input into monolingual output, degrade sharply on Romanized input, and remain behind fine-tuned encoders on token-level tasks [30][38].\n\n### 6.3 Metrics\n\nTask metrics (token F1 for LID/POS/NER; EM accuracy for parsing; BLEU/chrF/COMET for MT) plus **CS-structural binning metrics**: CMI [36], I-index [37], language entropy, switch-point density, burstiness. Known gap: BLEU/COMET reliability on CS output is under-studied, and most CS benchmarks are small and static (overfitting risk) \u2014 hold out a natural-noise set alongside synthetic CS.\n\n---\n\n## 7. Interaction effects \u2014 why the three layers must be tested together\n\nThe layers compound, and single-layer testing gives false confidence:\n\n1. **Noise \u00d7 normalization**: typos and casing noise applied *before* normalization can defeat Quick-Check fast paths and re-introduce denormalized sequences; re-run the full pipeline on noisy inputs, not just the model.\n2. **Bidi \u00d7 noise**: RLM/RLM-like controls injected as \"noise\" are invisible, survive NFC/NFKC, and flip rendering of surrounding text \u2014 an invariance test should assert that stripping bidi controls changes neither the model output nor the rendered direction.\n3. **Script \u00d7 code-switching**: Hinglish written in Devanagari vs. Roman script behaves like two different domains for tokenizers; test both script variants of the same CS content (Dakshina enables exactly this) [38][41].\n4. **Homoglyph \u00d7 code-switching**: cross-script confusables inside romanized CS text (e.g., Cyrillic \u0430 in Romanized Hindi) simultaneously defeat keyword filters and LID \u2014 combine UTS #39 skeleton checks with LID confidence monitoring [4][30].\n5. **Security \u00d7 everything**: hidden tag-character injections ride in through any of the above channels; scan for `[\\u200b-\\u200f\\u202a-\\u202e\\u2060-\\u2069\\u00ad\\ufeff\\U000e0000-\\U000e007f]` on both input and output sides [15][18][19].\n\n---\n\n## 8. Reference test matrix\n\n| Test class | Generator | Assertion |\n|-----------|-----------|-----------|\n| Normalization idempotence | ftfy + NFC pipeline | `is_normalized(NFC)` post-pipeline; pipeline(pipeline(x)) == pipeline(x) |\n| Mojibake round-trip | ftfy badness corpus | `ftfy.badness.is_bad()` false after repair; diffs reviewed |\n| Zero-width invariance | Inject ZWSP/ZWNJ/ZWJ/tag chars | Model output invariant after policy stripping; emoji/Indic text *not* broken |\n| Bidi rendering | FSI/PDI-wrapped interpolations; RLO strings | Isolate pairing balanced; no visual reorder of code; UBA conformance via BidiTest.txt [9] |\n| Confusable detection | Cyrillic/Greek homoglyphs; fullwidth Latin | Skeleton collision flagged; restriction level enforced [4] |\n| Noise invariance (CheckList INV) | nlpaug/textflint seeded perturbations, 3 severities | Pass-rate \u2265 threshold per noise type; deltas significance-tested [23] |\n| Natural noise | MTNT/Wikipedia-edit samples | Delta vs. synthetic noise reported separately [20][21] |\n| Code-switching stratified | LinCE/GLUE-CoS binned by CMI/I-index | Performance vs. mixing-density curve; no collapse at high density [30][36] |\n| Romanized CS | PHINC/Dakshina Roman variants | Transliteration-prefilter vs. direct: quantify gap [38][39][41] |\n| Hidden-character security | Tag-char smuggling payloads | Zero invisible chars in outputs; injection refused [18][19] |\n\n---\n\n## References\n\n**Standards:** [1] UAX #15, *Unicode Normalization Forms*. [2] The Unicode Standard Ch. 3 & \u00a75.18 (caseless matching); Normalization Stability Policy. [3] W3C, *Character Model: String Matching*. [4] UTS #39, *Unicode Security Mechanisms*; confusables.txt; intentional.txt. [5] UAX #31, *Identifier and Pattern Syntax*; UTS #46, *IDNA Compatibility Processing*. [6] ftfy documentation (R. Speer). [7] ICU `Normalizer2` / SpoofChecker APIs. [8] UAX #29, *Unicode Text Segmentation*; Python `unicodedata`. [9] UAX #9, *Unicode Bidirectional Algorithm*; UCD BidiTest.txt. [10] Chromium IDN guidance; Mozilla nsIDNService / Firefox 59 notes. [11] UAX #11, *East Asian Width*; UAX #24/ScriptExtensions. [12] ICANN IDN Guidelines; SSAC SAC 045. [13] WHATWG HTML (`dir=auto`, `bdi`); CSS Writing Modes. [14] W3C i18n articles on inline bidi markup. [15] UTS #55, *Unicode Source Code Handling*; UTR #36. [16] Boucher & Anderson, \"Trojan Source,\" arXiv:2111.00169 / USENIX Security 2023; CVE-2021-42574, CVE-2021-42694. [17] GCC 12 `-Wbidi-chars`; Rust 1.56 lint; VS Code 1.63; GitHub changelog (2021). [18] Rehberger, \"ASCII Smuggling\" (2023\u201324); Microsoft MSRC, \"Securing AI Model Outputs\" (2024). [19] \"Invisible Prompt Injection\" (arXiv 2024); OWASP Top 10 for LLM Apps.\n\n**Noise robustness:** [20] Belinkov & Bisk, ICLR 2018. [21] Michel & Neubig, \"MTNT,\" EMNLP 2018. [22] Li et al., WMT 2019 Robustness findings; Khayrallah & Koehn, WMT 2018. [23] Ribeiro et al., \"CheckList,\" ACL 2020. [24] Pruthi et al., ACL 2019. [25] Kudo, \"Subword Regularization,\" ACL 2018. [26] Provilkov et al., \"BPE-Dropout,\" ACL 2020. [27] Morris et al., \"TextAttack,\" ACL 2020; Ma, \"nlpaug,\" 2019; Wang et al., \"TextFlint,\" 2021; Bitton et al., \"AugLy,\" 2021; Zeng et al., \"OpenAttack,\" ACL 2021. [28] Gr\u00f6ndahl & Asokan, \"Smoke Tester,\" 2020. [29] Post, WMT 2018 (sacreBLEU); Rei et al., \"COMET,\" EMNLP 2020.\n\n**Code-switching:** [30] Winata et al., \"LinCE,\" ICLR 2020; \"Are Multilingual Models Effective in Code-Switching?\" LREC 2021. [31] Winata et al., \"GLUE-CoS,\" arXiv:2010.12609. [32] \"Hinglish TOP,\" EMNLP 2021. [33] Patwa et al., SemEval-2020 Task 9 (SentiMix). [34] Solorio et al., Spanglish 2014. [35] Molina et al., CoSWo 2016; CALLHOME Egyptian Arabic. [36] Gamb\u00e4ck & Das, Code-Mixing Index, 2013. [37] Guzm\u00e1n et al., I-index, 2017. [38] \"RomanSetu,\" 2024. [39] Mukherjee et al., \"PHINC,\" 2020. [40] Nayak & Joshi, \"HingBERT,\" LREC 2022. [41] Roark et al., \"Dakshina,\" 2020; Yin et al., \"CSP,\" EMNLP 2020.\n\n*Caveat: subtopic findings on UTS #39 internals and browser-switch dates were compiled from canonical standards documents; verify exact clause numbers and version dates against the live Unicode specs before external citation.*\n\n{\n  \"schema\": \"abb.result.v1\",\n  \"run_id\": \"919e085c28c54dd2bb1749b0e1ac4a31\",\n  \"case_id\": \"case_a7c19f75c1654d8e8f5c28b3bcc74a93\",\n  \"input_id\": \"step-10\",\n  \"invocation_id\": \"919e085c28c54dd2bb1749b0e1ac4a31\",\n  \"agent_id\": \"local-deep-research\",\n  \"status\": \"succeeded\",\n  \"output\": \"All research is complete. Here is the consolidated report.\\n\\n---\\n\\n# Normalizing Script, Bidi, and Unicode Issues in Localized Text \u2014 and Stress-Testing with Noise and Code-Switching\\n\\n## 1. Executive summary\\n\\nLocalized text fails in three distinct layers, and each layer requires its own normalization pass and its own stress-test suite:\\n\\n1. **Unicode/code-point layer** \u2014 mojibake, denormalized sequences, zero-width/control characters, fullwidth/halfwidth variants. Fixed with an ordered pipeline: decode \u2192 repair (ftfy) \u2192 normalize (NFC) \u2192 strip invisibles \u2192 clean controls \u2192 fold case \u2192 re-normalize \u2192 grapheme-safe truncation [1][2][6].\\n2. **Script layer** \u2014 mixed-script confusables (Cyrillic/Greek homoglyphs), width variants, script restriction policy. Fixed with UTS #39 skeleton matching and restriction levels, plus NFKC-based width folding in derived matching layers only [4][5].\\n3. **Direction layer** \u2014 wrong base direction, neutral-character scrambling, interpolation leaks. Fixed with first-strong detection, isolates (FSI/PDI) over deprecated embeddings, and logical-order storage end-to-end [9][13][14].\\n\\nCritically, **normalization alone fixes none of the security or robustness problems**: NFC/NFKC do not remove zero-width or tag characters, do not unify homoglyphs, and do not neutralize bidi overrides [1][5]. And robustness testing shows that **synthetic-noise robustness does not transfer to natural noise** [20], while code-switched and romanized text degrades multilingual models monotonically with mixing density [30][31].\\n\\n---\\n\\n## 2. The normalization pipeline (ordered)\\n\\n### 2.1 Canonical step order\\n\\nOrder matters because several steps *denormalize* text that earlier steps fixed. The recommended sequence [1][2][6]:\\n\\n| # | Step | Rationale |\\n|---|------|-----------|\\n| 1 | **Decode bytes** (strict UTF-8; strip BOM via `utf-8-sig`) | Normalization operates on decoded text only |\\n| 2 | **Mojibake repair (ftfy)** | Normalization cannot fix wrong-decoded bytes; ftfy must run first [6] |\\n| 3 | **Unicode normalization (NFC)** | Canonical default for storage and display [1][3] |\\n| 4 | **Zero-width/invisible policy** (targeted code-point removal) | NFC does *not* remove these \u2014 no decomposition mappings [1][5] |\\n| 5 | **Control-character cleanup** (CRLF/LS/PS\u2192LF; strip C0/C1/noncharacters) | C1 bytes are usually cp1252-misdecode artifacts [6] |\\n| 6 | **Entity decoding + whitespace collapse** | Entities can inject denormalized sequences |\\n| 7 | **Case folding \u2192 re-normalize** | Folding can denormalize (U+1E9B\u2192U+1E61; U+0345\u2192\u03b9) [2] |\\n| 8 | **Grapheme-aware truncation** (UAX #29) | Never cut between base and combining marks; truncation is a classic denormalizer [8] |\\n| 9 | **Validate/QA**: assert `is_normalized`, log ftfy diffs, then hash/dedup/embed | Security/length checks come *after* normalization, which changes lengths |\\n\\nMinimal Python sketch:\\n\\n```python\\nimport ftfy, unicodedata\\ns = ftfy.fix_text(raw, normalization=None)      # 1\u20132: repair, don't normalize yet\\ns = strip_invisibles(s)                          # 4: explicit code-point list\\ns = unicodedata.normalize(\\\"NFC\\\", s)              # 3: normalize\\ns = s.casefold(); s = unicodedata.normalize(\\\"NFC\\\", s)  # 7: fold + re-normalize\\nassert unicodedata.is_normalized(\\\"NFC\\\", s)       # 9: fast QC gate\\n```\\n\\nUse `unicodedata.is_normalized` / ICU `Normalizer2.quickCheck` to skip full normalization when Quick-Check properties pass [1][7][8].\\n\\n### 2.2 Choosing the normalization form\\n\\n| Form | Use for | Never use for |\\n|------|---------|---------------|\\n| **NFC** | Storage, interchange, all user-visible text (W3C web default) [3] | \u2014 |\\n| **NFD** | Diacritic stripping, linguistic analysis | Storage |\\n| **NFKC/NFKD** | Search/matching indexes, identifier profiles (UAX #31), fuzzy dedup | User-visible content \u2014 it is lossy |\\n\\nNFKC folds \ufb01\u2192fi, \u2461\u21922, \u2163\u2192IV, fullwidth \uff21\uff22\uff23\u2192ABC \u2014 distinctions that matter in CJK and math contexts. Keep NFC originals and apply NFKC only in derived matching layers [1][2].\\n\\n### 2.3 Zero-width and control characters \u2014 explicit policy, not blanket rules\\n\\nZWSP (U+200B), ZWNJ (U+200C), ZWJ (U+200D) are all category **Cf**; blanket Cf-stripping breaks emoji ZWJ sequences and Indic/Arabic orthography [1][5]. Use explicit lists:\\n\\n- **Strip**: ZWSP, WORD JOINER (U+2060), soft hyphen (U+00AD), ZWNBSP (U+FEFF mid-file), LRM/RLM in search indexes.\\n- **Preserve**: ZWJ (emoji, Indic shaping), ZWNJ (Devanagari/Kannada/Malayalam/Arabic), VS16 (U+FE0F, \u2764\ufe0f vs \u2764), tag characters (subdivision flags), bidi isolates in RTL UIs.\\n- **Reject for interchange**: noncharacters (U+FDD0\u2013FDEF, U+nFFFE/FFFF).\\n\\n### 2.4 Mojibake repair caveats\\n\\nftfy handles UTF-8-read-as-cp1252 (\\\"\u00e2\u20ac\u2122\\\"\u2192\\\"'\\\"), double encoding, C1 controls, HTML entities, JSON escapes, and fullwidth Latin; it normalizes to NFC by default (set `normalization='None'` to separate steps) [6]. Run `fix_and_explain()` and review diffs \u2014 ftfy is heuristic and can occasionally alter legitimate text. It is strongest on Latin/CJK mojibake and is a repair tool, not a substitute for fixing encoding declarations upstream. Apply per line or whole document, never across arbitrary byte-offset chunk boundaries.\\n\\n---\\n\\n## 3. Script-layer issues\\n\\n### 3.1 Mixed-script and homoglyph detection (UTS #39)\\n\\nThe skeleton algorithm: `skeleton(X) = NFD(casefold(confusables_map(NFD(X))))`. Two strings are confusable iff their skeletons are identical [4]. Key homoglyph sets in confusables.txt: Cyrillic \u0430/\u0441/\u0435/\u043e/\u0440/\u0445/\u0443\u2192a/c/e/o/p/x/y; Greek \u03bf/\u03b1/\u03b5/\u03b9/\u03ba/\u03bd/\u03c1/\u03c4\u2192o/a/e/i/k/n/p/t [4].\\n\\nDetection procedure: normalize (NFKC_Casefold) \u2192 resolve per-character script sets via *augmented* sets (ScriptExtensions.txt, so digits/punctuation don't false-trigger) \u2192 compute skeleton \u2192 classify as **mixed-script confusable** (e.g., `\u0430dmin` with Cyrillic \u0430, skeleton is pure Latin `admin`) or **whole-script confusable** (all-Cyrillic `\u0441\u043e\u0440\u0435` \u2248 Latin `cope`) [4][5][7]. ICU's SpoofChecker exposes `MIXED_SCRIPT_CONFUSABLE`, `WHOLE_SCRIPT_CONFUSABLE`, `INVISIBLE`, and `RESTRICTION_LEVEL` checks [7].\\n\\n**Restriction levels** \u2014 pick one at identifier creation and enforce it [4][5]:\\n\\n| Level | Policy |\\n|-------|--------|\\n| Inclusion | ASCII + tiny curated set (highest assurance) |\\n| Highly Restrictive | Recommended scripts; multi-script only within groups ({Han,Hiragana,Katakana}, {Hangul}) \u2014 good default for global services |\\n| Moderately Restrictive | Latin may mix with Recommended scripts except Cyrillic/Greek/Cherokee |\\n| Minimally Restrictive | Any Recommended-script mix (Latin+Cyrillic allowed) |\\n\\nBest practices: persist `skeleton(username)` in an indexed column and compare at registration; reject with a clear error rather than silently mutating; re-validate stored identifiers after each Unicode release (confusables.txt updates annually); never rely on skeleton matching alone \u2014 it is approximate and font-dependent [4][8].\\n\\n### 3.2 Width folding for CJK\\n\\nFullwidth ASCII variants (U+FF01\u2013FF5E) carry `` compatibility decompositions, so NFKC folds \uff21\uff22\uff23\u2192ABC, \uff0c\u2192,, fullwidth space\u2192space; it also folds *halfwidth katakana \u2192 fullwidth* (\uff8a\u2192\u30cf) [1][3]. Two traps: (a) ideographic punctuation \u3001\u3002 has **no** compatibility decomposition, so NFKC is not a pure width fold; (b) NFKC over-folds \u2460\u21921, \u3231\u2192(\u682a), \u33a1\u2192m2. Japanese/Chinese search systems therefore use custom width-only folding rather than raw NFKC; IMEs default to fullwidth output, making \u534a\u89d2/\u5168\u89d2 normalization a core JA/ZH localization requirement [3].\\n\\n### 3.3 IDN/domain spoofing\\n\\nIDNA2008 makes all compatibility characters (including U+FF01\u2013FF5E) disallowed in registered labels, so \uff50\uff41\uff59\uff50\uff41\uff4c.com cannot exist as a distinct domain [5][7]. Use UTS #46 (nontransitional mode \u2014 the browser standard since Firefox 59 in 2018) for any domain processing, never hand-rolled NFKC [5]. Browsers layer UTS #39 confusable checks and single-script display rules on top, falling back to Punycode display [9][10].\\n\\n---\\n\\n## 4. Bidi-layer issues\\n\\n### 4.1 Base direction \u2014 the single most common failure\\n\\nThe UBA (UAX #9) converts logical to visual order but does not tell you the *intended* base direction [9]. The P2 first-strong heuristic (scan for first strong L/R/AL character, skipping isolate contents) is implemented by HTML `dir=auto` and FSI [9][13]. Known misfire: RTL text beginning with a Latin brand name, @mention, hashtag, emoji, or digits gets an LTR base incorrectly. Mitigations: strong-type ratio counting, per-locale defaults, or explicit override with a prefixed RLM [9][14].\\n\\n### 4.2 Isolates over embeddings\\n\\nUse LRI/RLI/FSI/PDI (U+2066\u20132069, Unicode 6.3) in new code. Unlike legacy embeddings (LRE/RLE, deprecated), isolates treat their content as a separate paragraph \u2014 no directional \\\"leakage\\\" across boundaries; they nest and pair like brackets [9]. In HTML prefer markup (`dir=auto`, ``, `unicode-bidi: isolate`) over control characters \u2014 markup survives sanitizers; in plain strings (logs, JS interpolation, databases) wrap interpolated variables in FSI\u2026PDI [13][14]. CSS `unicode-bidi: plaintext` gives per-element auto-detected base direction \u2014 ideal for chat bubbles [13]. Always store **logical order** end-to-end; apply the UBA only at render time.\\n\\n### 4.3 Bug catalog for mixed LTR/RTL content\\n\\n1. Wrong base direction for user content \u2192 `dir=auto`/`bdi`/`plaintext` [13][14]\\n2. String-interpolation leaks (\\\"Delete ``?\\\" in RTL) \u2192 FSI\u2026PDI wrapping [14]\\n3. URLs/paths/code in RTL text scramble via neutral `/`, `:`, `.` \u2192 forced LTR isolation\\n4. First-strong misfires on leading emoji/digits/@mentions \u2192 ratio heuristic or override\\n5. Bracket mismatches across runs \u2192 rule N0 in modern implementations (Unicode 8.0+) [9]\\n6. Truncation between FSI and PDI leaks direction into the rest of the UI \u2192 grapheme-boundary truncation, keep control pairs balanced\\n7. Sanitizer/pipeline stripping of bidi controls inconsistently \u2192 define an explicit policy\\n8. **Trojan Source** (CVE-2021-42574): RLO/LRO in source makes reviewed code differ from executed code \u2192 UTS #55 pairing checks, GCC `-Wbidi-chars`, Rust `text_direction_codepoint_in_literal`, VS Code Unicode highlighting, GitHub bidi banners [15][16][17]\\n9. Homoglyph identifier spoofing (CVE-2021-42694) \u2192 UTS #39 restriction levels [4][17]\\n10. Terminals implementing only partial UBA scramble mixed-direction logs [9]\\n\\n### 4.4 Bidi controls in NLP pipelines\\n\\nBidi controls and zero-width characters **survive normalization** and act as spurious tokens in models; they are also an LLM attack vector \u2014 tag characters (U+E0000\u2013E007F) are invisible in chat UIs but tokenize normally, enabling \\\"ASCII smuggling\\\" prompt injection and data exfiltration (demonstrated against Copilot; addressed by Microsoft MSRC in 2024) [15][18][19]. Sanitize inputs *and* outputs; treat retrieved documents as untrusted; never render model output as raw HTML/Markdown [19].\\n\\n---\\n\\n## 5. Stress-testing with noise\\n\\n### 5.1 Noise taxonomy to inject\\n\\n| Level | Perturbations | Purpose |\\n|-------|--------------|---------|\\n| Character | Keyboard-adjacent typos, adjacent swaps, insertions/deletions, OCR confusions (l/1, O/0), homoglyphs | Pruthi et al. showed 1\u20132 character edits break classifiers [24] |\\n| Casing | lowercase, ALL-CAPS, mixed-case | Tests truecasing/tokenizer assumptions |\\n| Whitespace | word concatenation (\\\"howareyou\\\"), letter splitting, double spaces | Breaks subword tokenization |\\n| Word/sentence | synonym substitution (TextFooler), contextual BERT substitution (BAE), EDA insert/swap/delete, Polyjuice counterfactuals | Semantic-preserving invariance tests |\\n| Natural | Wikipedia edit histories, Reddit (MTNT), social media | Synthetic robustness does **not** transfer to natural noise [20] |\\n\\n### 5.2 Canonical findings and benchmarks\\n\\n- **Belinkov & Bisk (ICLR 2018)**: both synthetic and natural noise severely degrade NMT; introduced keyboard typos, scrambling, disemvoweling, and Wikipedia-mined natural noise [20].\\n- **MTNT** (EMNLP 2018) and **WMT19 Robustness task**: noisy social-media test sets for en\u2013fr/en\u2013ja/en\u2013ko [21][22].\\n- **CheckList** (ACL 2020): Minimum Functionality, Invariance, and Directional Expectation tests \u2014 the standard behavioral framework; INV tests with typos/punctuation are directly applicable here [23].\\n- **Training-side fixes**: subword regularization and BPE-dropout measurably improve segmentation robustness to casing/whitespace noise [25][26].\\n\\n### 5.3 Tools\\n\\n**nlpaug** (KeyboardAug, OcrAug, RandomCharAug, contextual word augmenters), **TextAttack** (~16 attack recipes with QWERTY/homoglyph transformations and USE/BERTScore similarity constraints), **textflint** (58 transformations + 20 subpopulations across 12 tasks, EN+ZH), **AugLy** (casing, homoglyphs, Unicode confusables), **OpenAttack** [12][13][14][19\u2192tool refs].\\n\\n### 5.4 Metrics and hygiene\\n\\nReport **clean-vs-noisy deltas** per noise type (\u0394accuracy/F1 for NLU; \u0394BLEU/\u0394chrF/\u0394COMET for MT, sacreBLEU-standardized), **invariance pass rates** (CheckList INV), and for adversarial attacks the **attack success rate with similarity/grammar constraints** \u2014 many attacks produce ungrammatical text, inflating apparent vulnerability, so gate with perplexity/grammaticality checks [23][27][28][29]. Fix seeds; significance-test deltas with paired bootstrap.\\n\\n---\\n\\n## 6. Stress-testing with code-switching\\n\\n### 6.1 Benchmarks\\n\\n| Resource | Pair(s) | Tasks |\\n|----------|---------|-------|\\n| **LinCE** (ICLR 2020) | Es\u2013En, Hi\u2013En, Zh\u2013En, Ne\u2013En, Singlish | LID, POS, NER, sentiment, semantic parsing [30] |\\n| **GLUE-CoS** | En mixed with Es/Hi/Zh | NLI, sentiment, relatedness, LID [31] |\\n| **Hinglish TOP** (EMNLP 2021) | Hi\u2013En | Intent + slot filling [32] |\\n| **SentiMix** (SemEval-2020 T9) / Spanglish 2014 | Hi\u2013En / Es\u2013En | Sentiment, token LID [33][34] |\\n| **CoSWo 2016** (MSA\u2013Egyptian, CALLHOME) | Arabic diglossia | Token LID [35] |\\n\\nKey replicated result: multilingual encoders fine-tuned on monolingual data show large drops on CS text, **worsening monotonically as mixing density increases** \u2014 so stratify test sets by Code-Mixing Index (CMI) and I-index [30][36][37].\\n\\n### 6.2 Transliterated code-switching \u2014 the sharpest stressor\\n\\nRomanized Hinglish/Banglish is the highest-value stress case: **RomanSetu (2024)** shows multilingual LLMs underperform on Romanized Indic text due to tokenizer fragmentation and script mismatch, and that romanization-aware adaptation (or transliterating input to native script before processing) recovers much of the gap [38]. Resources: **PHINC** (parallel Romanized-Hinglish\u2194English, ~30k pairs) [39], **HingBERT/HingMBERT/HingRoBERTa** (beat mBERT on Hinglish tasks) [40], **Dakshina** (romanized\u2194native parallel for 12 South Asian languages) [41]. LLM-specific failure modes: models tend to \\\"normalize\\\" CS input into monolingual output, degrade sharply on Romanized input, and remain behind fine-tuned encoders on token-level tasks [30][38].\\n\\n### 6.3 Metrics\\n\\nTask metrics (token F1 for LID/POS/NER; EM accuracy for parsing; BLEU/chrF/COMET for MT) plus **CS-structural binning metrics**: CMI [36], I-index [37], language entropy, switch-point density, burstiness. Known gap: BLEU/COMET reliability on CS output is under-studied, and most CS benchmarks are small and static (overfitting risk) \u2014 hold out a natural-noise set alongside synthetic CS.\\n\\n---\\n\\n## 7. Interaction effects \u2014 why the three layers must be tested together\\n\\nThe layers compound, and single-layer testing gives false confidence:\\n\\n1. **Noise \u00d7 normalization**: typos and casing noise applied *before* normalization can defeat Quick-Check fast paths and re-introduce denormalized sequences; re-run the full pipeline on noisy inputs, not just the model.\\n2. **Bidi \u00d7 noise**: RLM/RLM-like controls injected as \\\"noise\\\" are invisible, survive NFC/NFKC, and flip rendering of surrounding text \u2014 an invariance test should assert that stripping bidi controls changes neither the model output nor the rendered direction.\\n3. **Script \u00d7 code-switching**: Hinglish written in Devanagari vs. Roman script behaves like two different domains for tokenizers; test both script variants of the same CS content (Dakshina enables exactly this) [38][41].\\n4. **Homoglyph \u00d7 code-switching**: cross-script confusables inside romanized CS text (e.g., Cyrillic \u0430 in Romanized Hindi) simultaneously defeat keyword filters and LID \u2014 combine UTS #39 skeleton checks with LID confidence monitoring [4][30].\\n5. **Security \u00d7 everything**: hidden tag-character injections ride in through any of the above channels; scan for `[\\\\u200b-\\\\u200f\\\\u202a-\\\\u202e\\\\u2060-\\\\u2069\\\\u00ad\\\\ufeff\\\\U000e0000-\\\\U000e007f]` on both input and output sides [15][18][19].\\n\\n---\\n\\n## 8. Reference test matrix\\n\\n| Test class | Generator | Assertion |\\n|-----------|-----------|-----------|\\n| Normalization idempotence | ftfy + NFC pipeline | `is_normalized(NFC)` post-pipeline; pipeline(pipeline(x)) == pipeline(x) |\\n| Mojibake round-trip | ftfy badness corpus | `ftfy.badness.is_bad()` false after repair; diffs reviewed |\\n| Zero-width invariance | Inject ZWSP/ZWNJ/ZWJ/tag chars | Model output invariant after policy stripping; emoji/Indic text *not* broken |\\n| Bidi rendering | FSI/PDI-wrapped interpolations; RLO strings | Isolate pairing balanced; no visual reorder of code; UBA conformance via BidiTest.txt [9] |\\n| Confusable detection | Cyrillic/Greek homoglyphs; fullwidth Latin | Skeleton collision flagged; restriction level enforced [4] |\\n| Noise invariance (CheckList INV) | nlpaug/textflint seeded perturbations, 3 severities | Pass-rate \u2265 threshold per noise type; deltas significance-tested [23] |\\n| Natural noise | MTNT/Wikipedia-edit samples | Delta vs. synthetic noise reported separately [20][21] |\\n| Code-switching stratified | LinCE/GLUE-CoS binned by CMI/I-index | Performance vs. mixing-density curve; no collapse at high density [30][36] |\\n| Romanized CS | PHINC/Dakshina Roman variants | Transliteration-prefilter vs. direct: quantify gap [38][39][41] |\\n| Hidden-character security | Tag-char smuggling payloads | Zero invisible chars in outputs; injection refused [18][19] |\\n\\n---\\n\\n## References\\n\\n**Standards:** [1] UAX #15, *Unicode Normalization Forms*. [2] The Unicode Standard Ch. 3 & \u00a75.18 (caseless matching); Normalization Stability Policy. [3] W3C, *Character Model: String Matching*. [4] UTS #39, *Unicode Security Mechanisms*; confusables.txt; intentional.txt. [5] UAX #31, *Identifier and Pattern Syntax*; UTS #46, *IDNA Compatibility Processing*. [6] ftfy documentation (R. Speer). [7] ICU `Normalizer2` / SpoofChecker APIs. [8] UAX #29, *Unicode Text Segmentation*; Python `unicodedata`. [9] UAX #9, *Unicode Bidirectional Algorithm*; UCD BidiTest.txt. [10] Chromium IDN guidance; Mozilla nsIDNService / Firefox 59 notes. [11] UAX #11, *East Asian Width*; UAX #24/ScriptExtensions. [12] ICANN IDN Guidelines; SSAC SAC 045. [13] WHATWG HTML (`dir=auto`, `bdi`); CSS Writing Modes. [14] W3C i18n articles on inline bidi markup. [15] UTS #55, *Unicode Source Code Handling*; UTR #36. [16] Boucher & Anderson, \\\"Trojan Source,\\\" arXiv:2111.00169 / USENIX Security 2023; CVE-2021-42574, CVE-2021-42694. [17] GCC 12 `-Wbidi-chars`; Rust 1.56 lint; VS Code 1.63; GitHub changelog (2021). [18] Rehberger, \\\"ASCII Smuggling\\\" (2023\u201324); Microsoft MSRC, \\\"Securing AI Model Outputs\\\" (2024). [19] \\\"Invisible Prompt Injection\\\" (arXiv 2024); OWASP Top 10 for LLM Apps.\\n\\n**Noise robustness:** [20] Belinkov & Bisk, ICLR 2018. [21] Michel & Neubig, \\\"MTNT,\\\" EMNLP 2018. [22] Li et al., WMT 2019 Robustness findings; Khayrallah & Koehn, WMT 2018. [23] Ribeiro et al., \\\"CheckList,\\\" ACL 2020. [24] Pruthi et al., ACL 2019. [25] Kudo, \\\"Subword Regularization,\\\" ACL 2018. [26] Provilkov et al., \\\"BPE-Dropout,\\\" ACL 2020. [27] Morris et al., \\\"TextAttack,\\\" ACL 2020; Ma, \\\"nlpaug,\\\" 2019; Wang et al., \\\"TextFlint,\\\" 2021; Bitton et al., \\\"AugLy,\\\" 2021; Zeng et al., \\\"OpenAttack,\\\" ACL 2021. [28] Gr\u00f6ndahl & Asokan, \\\"Smoke Tester,\\\" 2020. [29] Post, WMT 2018 (sacreBLEU); Rei et al., \\\"COMET,\\\" EMNLP 2020.\\n\\n**Code-switching:** [30] Winata et al., \\\"LinCE,\\\" ICLR 2020; \\\"Are Multilingual Models Effective in Code-Switching?\\\" LREC 2021. [31] Winata et al., \\\"GLUE-CoS,\\\" arXiv:2010.12609. [32] \\\"Hinglish TOP,\\\" EMNLP 2021. [33] Patwa et al., SemEval-2020 Task 9 (SentiMix). [34] Solorio et al., Spanglish 2014. [35] Molina et al., CoSWo 2016; CALLHOME Egyptian Arabic. [36] Gamb\u00e4ck & Das, Code-Mixing Index, 2013. [37] Guzm\u00e1n et al., I-index, 2017. [38] \\\"RomanSetu,\\\" 2024. [39] Mukherjee et al., \\\"PHINC,\\\" 2020. [40] Nayak & Joshi, \\\"HingBERT,\\\" LREC 2022. [41] Roark et al., \\\"Dakshina,\\\" 2020; Yin et al., \\\"CSP,\\\" EMNLP 2020.\\n\\n*Caveat: subtopic findings on UTS #39 internals and browser-switch dates were compiled from canonical standards documents; verify exact clause numbers and version dates against the live Unicode specs before external citation.*\",\n  \"raw_output\": {\n    \"answer\": \"All research is complete. Here is the consolidated report.\\n\\n---\\n\\n# Normalizing Script, Bidi, and Unicode Issues in Localized Text \u2014 and Stress-Testing with Noise and Code-Switching\\n\\n## 1. Executive summary\\n\\nLocalized text fails in three distinct layers, and each layer requires its own normalization pass and its own stress-test suite:\\n\\n1. **Unicode/code-point layer** \u2014 mojibake, denormalized sequences, zero-width/control characters, fullwidth/halfwidth variants. Fixed with an ordered pipeline: decode \u2192 repair (ftfy) \u2192 normalize (NFC) \u2192 strip invisibles \u2192 clean controls \u2192 fold case \u2192 re-normalize \u2192 grapheme-safe truncation [1][2][6].\\n2. **Script layer** \u2014 mixed-script confusables (Cyrillic/Greek homoglyphs), width variants, script restriction policy. Fixed with UTS #39 skeleton matching and restriction levels, plus NFKC-based width folding in derived matching layers only [4][5].\\n3. **Direction layer** \u2014 wrong base direction, neutral-character scrambling, interpolation leaks. Fixed with first-strong detection, isolates (FSI/PDI) over deprecated embeddings, and logical-order storage end-to-end [9][13][14].\\n\\nCritically, **normalization alone fixes none of the security or robustness problems**: NFC/NFKC do not remove zero-width or tag characters, do not unify homoglyphs, and do not neutralize bidi overrides [1][5]. And robustness testing shows that **synthetic-noise robustness does not transfer to natural noise** [20], while code-switched and romanized text degrades multilingual models monotonically with mixing density [30][31].\\n\\n---\\n\\n## 2. The normalization pipeline (ordered)\\n\\n### 2.1 Canonical step order\\n\\nOrder matters because several steps *denormalize* text that earlier steps fixed. The recommended sequence [1][2][6]:\\n\\n| # | Step | Rationale |\\n|---|------|-----------|\\n| 1 | **Decode bytes** (strict UTF-8; strip BOM via `utf-8-sig`) | Normalization operates on decoded text only |\\n| 2 | **Mojibake repair (ftfy)** | Normalization cannot fix wrong-decoded bytes; ftfy must run first [6] |\\n| 3 | **Unicode normalization (NFC)** | Canonical default for storage and display [1][3] |\\n| 4 | **Zero-width/invisible policy** (targeted code-point removal) | NFC does *not* remove these \u2014 no decomposition mappings [1][5] |\\n| 5 | **Control-character cleanup** (CRLF/LS/PS\u2192LF; strip C0/C1/noncharacters) | C1 bytes are usually cp1252-misdecode artifacts [6] |\\n| 6 | **Entity decoding + whitespace collapse** | Entities can inject denormalized sequences |\\n| 7 | **Case folding \u2192 re-normalize** | Folding can denormalize (U+1E9B\u2192U+1E61; U+0345\u2192\u03b9) [2] |\\n| 8 | **Grapheme-aware truncation** (UAX #29) | Never cut between base and combining marks; truncation is a classic denormalizer [8] |\\n| 9 | **Validate/QA**: assert `is_normalized`, log ftfy diffs, then hash/dedup/embed | Security/length checks come *after* normalization, which changes lengths |\\n\\nMinimal Python sketch:\\n\\n```python\\nimport ftfy, unicodedata\\ns = ftfy.fix_text(raw, normalization=None)      # 1\u20132: repair, don't normalize yet\\ns = strip_invisibles(s)                          # 4: explicit code-point list\\ns = unicodedata.normalize(\\\"NFC\\\", s)              # 3: normalize\\ns = s.casefold(); s = unicodedata.normalize(\\\"NFC\\\", s)  # 7: fold + re-normalize\\nassert unicodedata.is_normalized(\\\"NFC\\\", s)       # 9: fast QC gate\\n```\\n\\nUse `unicodedata.is_normalized` / ICU `Normalizer2.quickCheck` to skip full normalization when Quick-Check properties pass [1][7][8].\\n\\n### 2.2 Choosing the normalization form\\n\\n| Form | Use for | Never use for |\\n|------|---------|---------------|\\n| **NFC** | Storage, interchange, all user-visible text (W3C web default) [3] | \u2014 |\\n| **NFD** | Diacritic stripping, linguistic analysis | Storage |\\n| **NFKC/NFKD** | Search/matching indexes, identifier profiles (UAX #31), fuzzy dedup | User-visible content \u2014 it is lossy |\\n\\nNFKC folds \ufb01\u2192fi, \u2461\u21922, \u2163\u2192IV, fullwidth \uff21\uff22\uff23\u2192ABC \u2014 distinctions that matter in CJK and math contexts. Keep NFC originals and apply NFKC only in derived matching layers [1][2].\\n\\n### 2.3 Zero-width and control characters \u2014 explicit policy, not blanket rules\\n\\nZWSP (U+200B), ZWNJ (U+200C), ZWJ (U+200D) are all category **Cf**; blanket Cf-stripping breaks emoji ZWJ sequences and Indic/Arabic orthography [1][5]. Use explicit lists:\\n\\n- **Strip**: ZWSP, WORD JOINER (U+2060), soft hyphen (U+00AD), ZWNBSP (U+FEFF mid-file), LRM/RLM in search indexes.\\n- **Preserve**: ZWJ (emoji, Indic shaping), ZWNJ (Devanagari/Kannada/Malayalam/Arabic), VS16 (U+FE0F, \u2764\ufe0f vs \u2764), tag characters (subdivision flags), bidi isolates in RTL UIs.\\n- **Reject for interchange**: noncharacters (U+FDD0\u2013FDEF, U+nFFFE/FFFF).\\n\\n### 2.4 Mojibake repair caveats\\n\\nftfy handles UTF-8-read-as-cp1252 (\\\"\u00e2\u20ac\u2122\\\"\u2192\\\"'\\\"), double encoding, C1 controls, HTML entities, JSON escapes, and fullwidth Latin; it normalizes to NFC by default (set `normalization='None'` to separate steps) [6]. Run `fix_and_explain()` and review diffs \u2014 ftfy is heuristic and can occasionally alter legitimate text. It is strongest on Latin/CJK mojibake and is a repair tool, not a substitute for fixing encoding declarations upstream. Apply per line or whole document, never across arbitrary byte-offset chunk boundaries.\\n\\n---\\n\\n## 3. Script-layer issues\\n\\n### 3.1 Mixed-script and homoglyph detection (UTS #39)\\n\\nThe skeleton algorithm: `skeleton(X) = NFD(casefold(confusables_map(NFD(X))))`. Two strings are confusable iff their skeletons are identical [4]. Key homoglyph sets in confusables.txt: Cyrillic \u0430/\u0441/\u0435/\u043e/\u0440/\u0445/\u0443\u2192a/c/e/o/p/x/y; Greek \u03bf/\u03b1/\u03b5/\u03b9/\u03ba/\u03bd/\u03c1/\u03c4\u2192o/a/e/i/k/n/p/t [4].\\n\\nDetection procedure: normalize (NFKC_Casefold) \u2192 resolve per-character script sets via *augmented* sets (ScriptExtensions.txt, so digits/punctuation don't false-trigger) \u2192 compute skeleton \u2192 classify as **mixed-script confusable** (e.g., `\u0430dmin` with Cyrillic \u0430, skeleton is pure Latin `admin`) or **whole-script confusable** (all-Cyrillic `\u0441\u043e\u0440\u0435` \u2248 Latin `cope`) [4][5][7]. ICU's SpoofChecker exposes `MIXED_SCRIPT_CONFUSABLE`, `WHOLE_SCRIPT_CONFUSABLE`, `INVISIBLE`, and `RESTRICTION_LEVEL` checks [7].\\n\\n**Restriction levels** \u2014 pick one at identifier creation and enforce it [4][5]:\\n\\n| Level | Policy |\\n|-------|--------|\\n| Inclusion | ASCII + tiny curated set (highest assurance) |\\n| Highly Restrictive | Recommended scripts; multi-script only within groups ({Han,Hiragana,Katakana}, {Hangul}) \u2014 good default for global services |\\n| Moderately Restrictive | Latin may mix with Recommended scripts except Cyrillic/Greek/Cherokee |\\n| Minimally Restrictive | Any Recommended-script mix (Latin+Cyrillic allowed) |\\n\\nBest practices: persist `skeleton(username)` in an indexed column and compare at registration; reject with a clear error rather than silently mutating; re-validate stored identifiers after each Unicode release (confusables.txt updates annually); never rely on skeleton matching alone \u2014 it is approximate and font-dependent [4][8].\\n\\n### 3.2 Width folding for CJK\\n\\nFullwidth ASCII variants (U+FF01\u2013FF5E) carry `` compatibility decompositions, so NFKC folds \uff21\uff22\uff23\u2192ABC, \uff0c\u2192,, fullwidth space\u2192space; it also folds *halfwidth katakana \u2192 fullwidth* (\uff8a\u2192\u30cf) [1][3]. Two traps: (a) ideographic punctuation \u3001\u3002 has **no** compatibility decomposition, so NFKC is not a pure width fold; (b) NFKC over-folds \u2460\u21921, \u3231\u2192(\u682a), \u33a1\u2192m2. Japanese/Chinese search systems therefore use custom width-only folding rather than raw NFKC; IMEs default to fullwidth output, making \u534a\u89d2/\u5168\u89d2 normalization a core JA/ZH localization requirement [3].\\n\\n### 3.3 IDN/domain spoofing\\n\\nIDNA2008 makes all compatibility characters (including U+FF01\u2013FF5E) disallowed in registered labels, so \uff50\uff41\uff59\uff50\uff41\uff4c.com cannot exist as a distinct domain [5][7]. Use UTS #46 (nontransitional mode \u2014 the browser standard since Firefox 59 in 2018) for any domain processing, never hand-rolled NFKC [5]. Browsers layer UTS #39 confusable checks and single-script display rules on top, falling back to Punycode display [9][10].\\n\\n---\\n\\n## 4. Bidi-layer issues\\n\\n### 4.1 Base direction \u2014 the single most common failure\\n\\nThe UBA (UAX #9) converts logical to visual order but does not tell you the *intended* base direction [9]. The P2 first-strong heuristic (scan for first strong L/R/AL character, skipping isolate contents) is implemented by HTML `dir=auto` and FSI [9][13]. Known misfire: RTL text beginning with a Latin brand name, @mention, hashtag, emoji, or digits gets an LTR base incorrectly. Mitigations: strong-type ratio counting, per-locale defaults, or explicit override with a prefixed RLM [9][14].\\n\\n### 4.2 Isolates over embeddings\\n\\nUse LRI/RLI/FSI/PDI (U+2066\u20132069, Unicode 6.3) in new code. Unlike legacy embeddings (LRE/RLE, deprecated), isolates treat their content as a separate paragraph \u2014 no directional \\\"leakage\\\" across boundaries; they nest and pair like brackets [9]. In HTML prefer markup (`dir=auto`, ``, `unicode-bidi: isolate`) over control characters \u2014 markup survives sanitizers; in plain strings (logs, JS interpolation, databases) wrap interpolated variables in FSI\u2026PDI [13][14]. CSS `unicode-bidi: plaintext` gives per-element auto-detected base direction \u2014 ideal for chat bubbles [13]. Always store **logical order** end-to-end; apply the UBA only at render time.\\n\\n### 4.3 Bug catalog for mixed LTR/RTL content\\n\\n1. Wrong base direction for user content \u2192 `dir=auto`/`bdi`/`plaintext` [13][14]\\n2. String-interpolation leaks (\\\"Delete ``?\\\" in RTL) \u2192 FSI\u2026PDI wrapping [14]\\n3. URLs/paths/code in RTL text scramble via neutral `/`, `:`, `.` \u2192 forced LTR isolation\\n4. First-strong misfires on leading emoji/digits/@mentions \u2192 ratio heuristic or override\\n5. Bracket mismatches across runs \u2192 rule N0 in modern implementations (Unicode 8.0+) [9]\\n6. Truncation between FSI and PDI leaks direction into the rest of the UI \u2192 grapheme-boundary truncation, keep control pairs balanced\\n7. Sanitizer/pipeline stripping of bidi controls inconsistently \u2192 define an explicit policy\\n8. **Trojan Source** (CVE-2021-42574): RLO/LRO in source makes reviewed code differ from executed code \u2192 UTS #55 pairing checks, GCC `-Wbidi-chars`, Rust `text_direction_codepoint_in_literal`, VS Code Unicode highlighting, GitHub bidi banners [15][16][17]\\n9. Homoglyph identifier spoofing (CVE-2021-42694) \u2192 UTS #39 restriction levels [4][17]\\n10. Terminals implementing only partial UBA scramble mixed-direction logs [9]\\n\\n### 4.4 Bidi controls in NLP pipelines\\n\\nBidi controls and zero-width characters **survive normalization** and act as spurious tokens in models; they are also an LLM attack vector \u2014 tag characters (U+E0000\u2013E007F) are invisible in chat UIs but tokenize normally, enabling \\\"ASCII smuggling\\\" prompt injection and data exfiltration (demonstrated against Copilot; addressed by Microsoft MSRC in 2024) [15][18][19]. Sanitize inputs *and* outputs; treat retrieved documents as untrusted; never render model output as raw HTML/Markdown [19].\\n\\n---\\n\\n## 5. Stress-testing with noise\\n\\n### 5.1 Noise taxonomy to inject\\n\\n| Level | Perturbations | Purpose |\\n|-------|--------------|---------|\\n| Character | Keyboard-adjacent typos, adjacent swaps, insertions/deletions, OCR confusions (l/1, O/0), homoglyphs | Pruthi et al. showed 1\u20132 character edits break classifiers [24] |\\n| Casing | lowercase, ALL-CAPS, mixed-case | Tests truecasing/tokenizer assumptions |\\n| Whitespace | word concatenation (\\\"howareyou\\\"), letter splitting, double spaces | Breaks subword tokenization |\\n| Word/sentence | synonym substitution (TextFooler), contextual BERT substitution (BAE), EDA insert/swap/delete, Polyjuice counterfactuals | Semantic-preserving invariance tests |\\n| Natural | Wikipedia edit histories, Reddit (MTNT), social media | Synthetic robustness does **not** transfer to natural noise [20] |\\n\\n### 5.2 Canonical findings and benchmarks\\n\\n- **Belinkov & Bisk (ICLR 2018)**: both synthetic and natural noise severely degrade NMT; introduced keyboard typos, scrambling, disemvoweling, and Wikipedia-mined natural noise [20].\\n- **MTNT** (EMNLP 2018) and **WMT19 Robustness task**: noisy social-media test sets for en\u2013fr/en\u2013ja/en\u2013ko [21][22].\\n- **CheckList** (ACL 2020): Minimum Functionality, Invariance, and Directional Expectation tests \u2014 the standard behavioral framework; INV tests with typos/punctuation are directly applicable here [23].\\n- **Training-side fixes**: subword regularization and BPE-dropout measurably improve segmentation robustness to casing/whitespace noise [25][26].\\n\\n### 5.3 Tools\\n\\n**nlpaug** (KeyboardAug, OcrAug, RandomCharAug, contextual word augmenters), **TextAttack** (~16 attack recipes with QWERTY/homoglyph transformations and USE/BERTScore similarity constraints), **textflint** (58 transformations + 20 subpopulations across 12 tasks, EN+ZH), **AugLy** (casing, homoglyphs, Unicode confusables), **OpenAttack** [12][13][14][19\u2192tool refs].\\n\\n### 5.4 Metrics and hygiene\\n\\nReport **clean-vs-noisy deltas** per noise type (\u0394accuracy/F1 for NLU; \u0394BLEU/\u0394chrF/\u0394COMET for MT, sacreBLEU-standardized), **invariance pass rates** (CheckList INV), and for adversarial attacks the **attack success rate with similarity/grammar constraints** \u2014 many attacks produce ungrammatical text, inflating apparent vulnerability, so gate with perplexity/grammaticality checks [23][27][28][29]. Fix seeds; significance-test deltas with paired bootstrap.\\n\\n---\\n\\n## 6. Stress-testing with code-switching\\n\\n### 6.1 Benchmarks\\n\\n| Resource | Pair(s) | Tasks |\\n|----------|---------|-------|\\n| **LinCE** (ICLR 2020) | Es\u2013En, Hi\u2013En, Zh\u2013En, Ne\u2013En, Singlish | LID, POS, NER, sentiment, semantic parsing [30] |\\n| **GLUE-CoS** | En mixed with Es/Hi/Zh | NLI, sentiment, relatedness, LID [31] |\\n| **Hinglish TOP** (EMNLP 2021) | Hi\u2013En | Intent + slot filling [32] |\\n| **SentiMix** (SemEval-2020 T9) / Spanglish 2014 | Hi\u2013En / Es\u2013En | Sentiment, token LID [33][34] |\\n| **CoSWo 2016** (MSA\u2013Egyptian, CALLHOME) | Arabic diglossia | Token LID [35] |\\n\\nKey replicated result: multilingual encoders fine-tuned on monolingual data show large drops on CS text, **worsening monotonically as mixing density increases** \u2014 so stratify test sets by Code-Mixing Index (CMI) and I-index [30][36][37].\\n\\n### 6.2 Transliterated code-switching \u2014 the sharpest stressor\\n\\nRomanized Hinglish/Banglish is the highest-value stress case: **RomanSetu (2024)** shows multilingual LLMs underperform on Romanized Indic text due to tokenizer fragmentation and script mismatch, and that romanization-aware adaptation (or transliterating input to native script before processing) recovers much of the gap [38]. Resources: **PHINC** (parallel Romanized-Hinglish\u2194English, ~30k pairs) [39], **HingBERT/HingMBERT/HingRoBERTa** (beat mBERT on Hinglish tasks) [40], **Dakshina** (romanized\u2194native parallel for 12 South Asian languages) [41]. LLM-specific failure modes: models tend to \\\"normalize\\\" CS input into monolingual output, degrade sharply on Romanized input, and remain behind fine-tuned encoders on token-level tasks [30][38].\\n\\n### 6.3 Metrics\\n\\nTask metrics (token F1 for LID/POS/NER; EM accuracy for parsing; BLEU/chrF/COMET for MT) plus **CS-structural binning metrics**: CMI [36], I-index [37], language entropy, switch-point density, burstiness. Known gap: BLEU/COMET reliability on CS output is under-studied, and most CS benchmarks are small and static (overfitting risk) \u2014 hold out a natural-noise set alongside synthetic CS.\\n\\n---\\n\\n## 7. Interaction effects \u2014 why the three layers must be tested together\\n\\nThe layers compound, and single-layer testing gives false confidence:\\n\\n1. **Noise \u00d7 normalization**: typos and casing noise applied *before* normalization can defeat Quick-Check fast paths and re-introduce denormalized sequences; re-run the full pipeline on noisy inputs, not just the model.\\n2. **Bidi \u00d7 noise**: RLM/RLM-like controls injected as \\\"noise\\\" are invisible, survive NFC/NFKC, and flip rendering of surrounding text \u2014 an invariance test should assert that stripping bidi controls changes neither the model output nor the rendered direction.\\n3. **Script \u00d7 code-switching**: Hinglish written in Devanagari vs. Roman script behaves like two different domains for tokenizers; test both script variants of the same CS content (Dakshina enables exactly this) [38][41].\\n4. **Homoglyph \u00d7 code-switching**: cross-script confusables inside romanized CS text (e.g., Cyrillic \u0430 in Romanized Hindi) simultaneously defeat keyword filters and LID \u2014 combine UTS #39 skeleton checks with LID confidence monitoring [4][30].\\n5. **Security \u00d7 everything**: hidden tag-character injections ride in through any of the above channels; scan for `[\\\\u200b-\\\\u200f\\\\u202a-\\\\u202e\\\\u2060-\\\\u2069\\\\u00ad\\\\ufeff\\\\U000e0000-\\\\U000e007f]` on both input and output sides [15][18][19].\\n\\n---\\n\\n## 8. Reference test matrix\\n\\n| Test class | Generator | Assertion |\\n|-----------|-----------|-----------|\\n| Normalization idempotence | ftfy + NFC pipeline | `is_normalized(NFC)` post-pipeline; pipeline(pipeline(x)) == pipeline(x) |\\n| Mojibake round-trip | ftfy badness corpus | `ftfy.badness.is_bad()` false after repair; diffs reviewed |\\n| Zero-width invariance | Inject ZWSP/ZWNJ/ZWJ/tag chars | Model output invariant after policy stripping; emoji/Indic text *not* broken |\\n| Bidi rendering | FSI/PDI-wrapped interpolations; RLO strings | Isolate pairing balanced; no visual reorder of code; UBA conformance via BidiTest.txt [9] |\\n| Confusable detection | Cyrillic/Greek homoglyphs; fullwidth Latin | Skeleton collision flagged; restriction level enforced [4] |\\n| Noise invariance (CheckList INV) | nlpaug/textflint seeded perturbations, 3 severities | Pass-rate \u2265 threshold per noise type; deltas significance-tested [23] |\\n| Natural noise | MTNT/Wikipedia-edit samples | Delta vs. synthetic noise reported separately [20][21] |\\n| Code-switching stratified | LinCE/GLUE-CoS binned by CMI/I-index | Performance vs. mixing-density curve; no collapse at high density [30][36] |\\n| Romanized CS | PHINC/Dakshina Roman variants | Transliteration-prefilter vs. direct: quantify gap [38][39][41] |\\n| Hidden-character security | Tag-char smuggling payloads | Zero invisible chars in outputs; injection refused [18][19] |\\n\\n---\\n\\n## References\\n\\n**Standards:** [1] UAX #15, *Unicode Normalization Forms*. [2] The Unicode Standard Ch. 3 & \u00a75.18 (caseless matching); Normalization Stability Policy. [3] W3C, *Character Model: String Matching*. [4] UTS #39, *Unicode Security Mechanisms*; confusables.txt; intentional.txt. [5] UAX #31, *Identifier and Pattern Syntax*; UTS #46, *IDNA Compatibility Processing*. [6] ftfy documentation (R. Speer). [7] ICU `Normalizer2` / SpoofChecker APIs. [8] UAX #29, *Unicode Text Segmentation*; Python `unicodedata`. [9] UAX #9, *Unicode Bidirectional Algorithm*; UCD BidiTest.txt. [10] Chromium IDN guidance; Mozilla nsIDNService / Firefox 59 notes. [11] UAX #11, *East Asian Width*; UAX #24/ScriptExtensions. [12] ICANN IDN Guidelines; SSAC SAC 045. [13] WHATWG HTML (`dir=auto`, `bdi`); CSS Writing Modes. [14] W3C i18n articles on inline bidi markup. [15] UTS #55, *Unicode Source Code Handling*; UTR #36. [16] Boucher & Anderson, \\\"Trojan Source,\\\" arXiv:2111.00169 / USENIX Security 2023; CVE-2021-42574, CVE-2021-42694. [17] GCC 12 `-Wbidi-chars`; Rust 1.56 lint; VS Code 1.63; GitHub changelog (2021). [18] Rehberger, \\\"ASCII Smuggling\\\" (2023\u201324); Microsoft MSRC, \\\"Securing AI Model Outputs\\\" (2024). [19] \\\"Invisible Prompt Injection\\\" (arXiv 2024); OWASP Top 10 for LLM Apps.\\n\\n**Noise robustness:** [20] Belinkov & Bisk, ICLR 2018. [21] Michel & Neubig, \\\"MTNT,\\\" EMNLP 2018. [22] Li et al., WMT 2019 Robustness findings; Khayrallah & Koehn, WMT 2018. [23] Ribeiro et al., \\\"CheckList,\\\" ACL 2020. [24] Pruthi et al., ACL 2019. [25] Kudo, \\\"Subword Regularization,\\\" ACL 2018. [26] Provilkov et al., \\\"BPE-Dropout,\\\" ACL 2020. [27] Morris et al., \\\"TextAttack,\\\" ACL 2020; Ma, \\\"nlpaug,\\\" 2019; Wang et al., \\\"TextFlint,\\\" 2021; Bitton et al., \\\"AugLy,\\\" 2021; Zeng et al., \\\"OpenAttack,\\\" ACL 2021. [28] Gr\u00f6ndahl & Asokan, \\\"Smoke Tester,\\\" 2020. [29] Post, WMT 2018 (sacreBLEU); Rei et al., \\\"COMET,\\\" EMNLP 2020.\\n\\n**Code-switching:** [30] Winata et al., \\\"LinCE,\\\" ICLR 2020; \\\"Are Multilingual Models Effective in Code-Switching?\\\" LREC 2021. [31] Winata et al., \\\"GLUE-CoS,\\\" arXiv:2010.12609. [32] \\\"Hinglish TOP,\\\" EMNLP 2021. [33] Patwa et al., SemEval-2020 Task 9 (SentiMix). [34] Solorio et al., Spanglish 2014. [35] Molina et al., CoSWo 2016; CALLHOME Egyptian Arabic. [36] Gamb\u00e4ck & Das, Code-Mixing Index, 2013. [37] Guzm\u00e1n et al., I-index, 2017. [38] \\\"RomanSetu,\\\" 2024. [39] Mukherjee et al., \\\"PHINC,\\\" 2020. [40] Nayak & Joshi, \\\"HingBERT,\\\" LREC 2022. [41] Roark et al., \\\"Dakshina,\\\" 2020; Yin et al., \\\"CSP,\\\" EMNLP 2020.\\n\\n*Caveat: subtopic findings on UTS #39 internals and browser-switch dates were compiled from canonical standards documents; verify exact clause numbers and version dates against the live Unicode specs before external citation.*\",\n    \"sources\": [],\n    \"formatted_findings\": \"All research is complete. Here is the consolidated report.\\n\\n---\\n\\n# Normalizing Script, Bidi, and Unicode Issues in Localized Text \u2014 and Stress-Testing with Noise and Code-Switching\\n\\n## 1. Executive summary\\n\\nLocalized text fails in three distinct layers, and each layer requires its own normalization pass and its own stress-test suite:\\n\\n1. **Unicode/code-point layer** \u2014 mojibake, denormalized sequences, zero-width/control characters, fullwidth/halfwidth variants. Fixed with an ordered pipeline: decode \u2192 repair (ftfy) \u2192 normalize (NFC) \u2192 strip invisibles \u2192 clean controls \u2192 fold case \u2192 re-normalize \u2192 grapheme-safe truncation [1][2][6].\\n2. **Script layer** \u2014 mixed-script confusables (Cyrillic/Greek homoglyphs), width variants, script restriction policy. Fixed with UTS #39 skeleton matching and restriction levels, plus NFKC-based width folding in derived matching layers only [4][5].\\n3. **Direction layer** \u2014 wrong base direction, neutral-character scrambling, interpolation leaks. Fixed with first-strong detection, isolates (FSI/PDI) over deprecated embeddings, and logical-order storage end-to-end [9][13][14].\\n\\nCritically, **normalization alone fixes none of the security or robustness problems**: NFC/NFKC do not remove zero-width or tag characters, do not unify homoglyphs, and do not neutralize bidi overrides [1][5]. And robustness testing shows that **synthetic-noise robustness does not transfer to natural noise** [20], while code-switched and romanized text degrades multilingual models monotonically with mixing density [30][31].\\n\\n---\\n\\n## 2. The normalization pipeline (ordered)\\n\\n### 2.1 Canonical step order\\n\\nOrder matters because several steps *denormalize* text that earlier steps fixed. The recommended sequence [1][2][6]:\\n\\n| # | Step | Rationale |\\n|---|------|-----------|\\n| 1 | **Decode bytes** (strict UTF-8; strip BOM via `utf-8-sig`) | Normalization operates on decoded text only |\\n| 2 | **Mojibake repair (ftfy)** | Normalization cannot fix wrong-decoded bytes; ftfy must run first [6] |\\n| 3 | **Unicode normalization (NFC)** | Canonical default for storage and display [1][3] |\\n| 4 | **Zero-width/invisible policy** (targeted code-point removal) | NFC does *not* remove these \u2014 no decomposition mappings [1][5] |\\n| 5 | **Control-character cleanup** (CRLF/LS/PS\u2192LF; strip C0/C1/noncharacters) | C1 bytes are usually cp1252-misdecode artifacts [6] |\\n| 6 | **Entity decoding + whitespace collapse** | Entities can inject denormalized sequences |\\n| 7 | **Case folding \u2192 re-normalize** | Folding can denormalize (U+1E9B\u2192U+1E61; U+0345\u2192\u03b9) [2] |\\n| 8 | **Grapheme-aware truncation** (UAX #29) | Never cut between base and combining marks; truncation is a classic denormalizer [8] |\\n| 9 | **Validate/QA**: assert `is_normalized`, log ftfy diffs, then hash/dedup/embed | Security/length checks come *after* normalization, which changes lengths |\\n\\nMinimal Python sketch:\\n\\n```python\\nimport ftfy, unicodedata\\ns = ftfy.fix_text(raw, normalization=None)      # 1\u20132: repair, don't normalize yet\\ns = strip_invisibles(s)                          # 4: explicit code-point list\\ns = unicodedata.normalize(\\\"NFC\\\", s)              # 3: normalize\\ns = s.casefold(); s = unicodedata.normalize(\\\"NFC\\\", s)  # 7: fold + re-normalize\\nassert unicodedata.is_normalized(\\\"NFC\\\", s)       # 9: fast QC gate\\n```\\n\\nUse `unicodedata.is_normalized` / ICU `Normalizer2.quickCheck` to skip full normalization when Quick-Check properties pass [1][7][8].\\n\\n### 2.2 Choosing the normalization form\\n\\n| Form | Use for | Never use for |\\n|------|---------|---------------|\\n| **NFC** | Storage, interchange, all user-visible text (W3C web default) [3] | \u2014 |\\n| **NFD** | Diacritic stripping, linguistic analysis | Storage |\\n| **NFKC/NFKD** | Search/matching indexes, identifier profiles (UAX #31), fuzzy dedup | User-visible content \u2014 it is lossy |\\n\\nNFKC folds \ufb01\u2192fi, \u2461\u21922, \u2163\u2192IV, fullwidth \uff21\uff22\uff23\u2192ABC \u2014 distinctions that matter in CJK and math contexts. Keep NFC originals and apply NFKC only in derived matching layers [1][2].\\n\\n### 2.3 Zero-width and control characters \u2014 explicit policy, not blanket rules\\n\\nZWSP (U+200B), ZWNJ (U+200C), ZWJ (U+200D) are all category **Cf**; blanket Cf-stripping breaks emoji ZWJ sequences and Indic/Arabic orthography [1][5]. Use explicit lists:\\n\\n- **Strip**: ZWSP, WORD JOINER (U+2060), soft hyphen (U+00AD), ZWNBSP (U+FEFF mid-file), LRM/RLM in search indexes.\\n- **Preserve**: ZWJ (emoji, Indic shaping), ZWNJ (Devanagari/Kannada/Malayalam/Arabic), VS16 (U+FE0F, \u2764\ufe0f vs \u2764), tag characters (subdivision flags), bidi isolates in RTL UIs.\\n- **Reject for interchange**: noncharacters (U+FDD0\u2013FDEF, U+nFFFE/FFFF).\\n\\n### 2.4 Mojibake repair caveats\\n\\nftfy handles UTF-8-read-as-cp1252 (\\\"\u00e2\u20ac\u2122\\\"\u2192\\\"'\\\"), double encoding, C1 controls, HTML entities, JSON escapes, and fullwidth Latin; it normalizes to NFC by default (set `normalization='None'` to separate steps) [6]. Run `fix_and_explain()` and review diffs \u2014 ftfy is heuristic and can occasionally alter legitimate text. It is strongest on Latin/CJK mojibake and is a repair tool, not a substitute for fixing encoding declarations upstream. Apply per line or whole document, never across arbitrary byte-offset chunk boundaries.\\n\\n---\\n\\n## 3. Script-layer issues\\n\\n### 3.1 Mixed-script and homoglyph detection (UTS #39)\\n\\nThe skeleton algorithm: `skeleton(X) = NFD(casefold(confusables_map(NFD(X))))`. Two strings are confusable iff their skeletons are identical [4]. Key homoglyph sets in confusables.txt: Cyrillic \u0430/\u0441/\u0435/\u043e/\u0440/\u0445/\u0443\u2192a/c/e/o/p/x/y; Greek \u03bf/\u03b1/\u03b5/\u03b9/\u03ba/\u03bd/\u03c1/\u03c4\u2192o/a/e/i/k/n/p/t [4].\\n\\nDetection procedure: normalize (NFKC_Casefold) \u2192 resolve per-character script sets via *augmented* sets (ScriptExtensions.txt, so digits/punctuation don't false-trigger) \u2192 compute skeleton \u2192 classify as **mixed-script confusable** (e.g., `\u0430dmin` with Cyrillic \u0430, skeleton is pure Latin `admin`) or **whole-script confusable** (all-Cyrillic `\u0441\u043e\u0440\u0435` \u2248 Latin `cope`) [4][5][7]. ICU's SpoofChecker exposes `MIXED_SCRIPT_CONFUSABLE`, `WHOLE_SCRIPT_CONFUSABLE`, `INVISIBLE`, and `RESTRICTION_LEVEL` checks [7].\\n\\n**Restriction levels** \u2014 pick one at identifier creation and enforce it [4][5]:\\n\\n| Level | Policy |\\n|-------|--------|\\n| Inclusion | ASCII + tiny curated set (highest assurance) |\\n| Highly Restrictive | Recommended scripts; multi-script only within groups ({Han,Hiragana,Katakana}, {Hangul}) \u2014 good default for global services |\\n| Moderately Restrictive | Latin may mix with Recommended scripts except Cyrillic/Greek/Cherokee |\\n| Minimally Restrictive | Any Recommended-script mix (Latin+Cyrillic allowed) |\\n\\nBest practices: persist `skeleton(username)` in an indexed column and compare at registration; reject with a clear error rather than silently mutating; re-validate stored identifiers after each Unicode release (confusables.txt updates annually); never rely on skeleton matching alone \u2014 it is approximate and font-dependent [4][8].\\n\\n### 3.2 Width folding for CJK\\n\\nFullwidth ASCII variants (U+FF01\u2013FF5E) carry `` compatibility decompositions, so NFKC folds \uff21\uff22\uff23\u2192ABC, \uff0c\u2192,, fullwidth space\u2192space; it also folds *halfwidth katakana \u2192 fullwidth* (\uff8a\u2192\u30cf) [1][3]. Two traps: (a) ideographic punctuation \u3001\u3002 has **no** compatibility decomposition, so NFKC is not a pure width fold; (b) NFKC over-folds \u2460\u21921, \u3231\u2192(\u682a), \u33a1\u2192m2. Japanese/Chinese search systems therefore use custom width-only folding rather than raw NFKC; IMEs default to fullwidth output, making \u534a\u89d2/\u5168\u89d2 normalization a core JA/ZH localization requirement [3].\\n\\n### 3.3 IDN/domain spoofing\\n\\nIDNA2008 makes all compatibility characters (including U+FF01\u2013FF5E) disallowed in registered labels, so \uff50\uff41\uff59\uff50\uff41\uff4c.com cannot exist as a distinct domain [5][7]. Use UTS #46 (nontransitional mode \u2014 the browser standard since Firefox 59 in 2018) for any domain processing, never hand-rolled NFKC [5]. Browsers layer UTS #39 confusable checks and single-script display rules on top, falling back to Punycode display [9][10].\\n\\n---\\n\\n## 4. Bidi-layer issues\\n\\n### 4.1 Base direction \u2014 the single most common failure\\n\\nThe UBA (UAX #9) converts logical to visual order but does not tell you the *intended* base direction [9]. The P2 first-strong heuristic (scan for first strong L/R/AL character, skipping isolate contents) is implemented by HTML `dir=auto` and FSI [9][13]. Known misfire: RTL text beginning with a Latin brand name, @mention, hashtag, emoji, or digits gets an LTR base incorrectly. Mitigations: strong-type ratio counting, per-locale defaults, or explicit override with a prefixed RLM [9][14].\\n\\n### 4.2 Isolates over embeddings\\n\\nUse LRI/RLI/FSI/PDI (U+2066\u20132069, Unicode 6.3) in new code. Unlike legacy embeddings (LRE/RLE, deprecated), isolates treat their content as a separate paragraph \u2014 no directional \\\"leakage\\\" across boundaries; they nest and pair like brackets [9]. In HTML prefer markup (`dir=auto`, ``, `unicode-bidi: isolate`) over control characters \u2014 markup survives sanitizers; in plain strings (logs, JS interpolation, databases) wrap interpolated variables in FSI\u2026PDI [13][14]. CSS `unicode-bidi: plaintext` gives per-element auto-detected base direction \u2014 ideal for chat bubbles [13]. Always store **logical order** end-to-end; apply the UBA only at render time.\\n\\n### 4.3 Bug catalog for mixed LTR/RTL content\\n\\n1. Wrong base direction for user content \u2192 `dir=auto`/`bdi`/`plaintext` [13][14]\\n2. String-interpolation leaks (\\\"Delete ``?\\\" in RTL) \u2192 FSI\u2026PDI wrapping [14]\\n3. URLs/paths/code in RTL text scramble via neutral `/`, `:`, `.` \u2192 forced LTR isolation\\n4. First-strong misfires on leading emoji/digits/@mentions \u2192 ratio heuristic or override\\n5. Bracket mismatches across runs \u2192 rule N0 in modern implementations (Unicode 8.0+) [9]\\n6. Truncation between FSI and PDI leaks direction into the rest of the UI \u2192 grapheme-boundary truncation, keep control pairs balanced\\n7. Sanitizer/pipeline stripping of bidi controls inconsistently \u2192 define an explicit policy\\n8. **Trojan Source** (CVE-2021-42574): RLO/LRO in source makes reviewed code differ from executed code \u2192 UTS #55 pairing checks, GCC `-Wbidi-chars`, Rust `text_direction_codepoint_in_literal`, VS Code Unicode highlighting, GitHub bidi banners [15][16][17]\\n9. Homoglyph identifier spoofing (CVE-2021-42694) \u2192 UTS #39 restriction levels [4][17]\\n10. Terminals implementing only partial UBA scramble mixed-direction logs [9]\\n\\n### 4.4 Bidi controls in NLP pipelines\\n\\nBidi controls and zero-width characters **survive normalization** and act as spurious tokens in models; they are also an LLM attack vector \u2014 tag characters (U+E0000\u2013E007F) are invisible in chat UIs but tokenize normally, enabling \\\"ASCII smuggling\\\" prompt injection and data exfiltration (demonstrated against Copilot; addressed by Microsoft MSRC in 2024) [15][18][19]. Sanitize inputs *and* outputs; treat retrieved documents as untrusted; never render model output as raw HTML/Markdown [19].\\n\\n---\\n\\n## 5. Stress-testing with noise\\n\\n### 5.1 Noise taxonomy to inject\\n\\n| Level | Perturbations | Purpose |\\n|-------|--------------|---------|\\n| Character | Keyboard-adjacent typos, adjacent swaps, insertions/deletions, OCR confusions (l/1, O/0), homoglyphs | Pruthi et al. showed 1\u20132 character edits break classifiers [24] |\\n| Casing | lowercase, ALL-CAPS, mixed-case | Tests truecasing/tokenizer assumptions |\\n| Whitespace | word concatenation (\\\"howareyou\\\"), letter splitting, double spaces | Breaks subword tokenization |\\n| Word/sentence | synonym substitution (TextFooler), contextual BERT substitution (BAE), EDA insert/swap/delete, Polyjuice counterfactuals | Semantic-preserving invariance tests |\\n| Natural | Wikipedia edit histories, Reddit (MTNT), social media | Synthetic robustness does **not** transfer to natural noise [20] |\\n\\n### 5.2 Canonical findings and benchmarks\\n\\n- **Belinkov & Bisk (ICLR 2018)**: both synthetic and natural noise severely degrade NMT; introduced keyboard typos, scrambling, disemvoweling, and Wikipedia-mined natural noise [20].\\n- **MTNT** (EMNLP 2018) and **WMT19 Robustness task**: noisy social-media test sets for en\u2013fr/en\u2013ja/en\u2013ko [21][22].\\n- **CheckList** (ACL 2020): Minimum Functionality, Invariance, and Directional Expectation tests \u2014 the standard behavioral framework; INV tests with typos/punctuation are directly applicable here [23].\\n- **Training-side fixes**: subword regularization and BPE-dropout measurably improve segmentation robustness to casing/whitespace noise [25][26].\\n\\n### 5.3 Tools\\n\\n**nlpaug** (KeyboardAug, OcrAug, RandomCharAug, contextual word augmenters), **TextAttack** (~16 attack recipes with QWERTY/homoglyph transformations and USE/BERTScore similarity constraints), **textflint** (58 transformations + 20 subpopulations across 12 tasks, EN+ZH), **AugLy** (casing, homoglyphs, Unicode confusables), **OpenAttack** [12][13][14][19\u2192tool refs].\\n\\n### 5.4 Metrics and hygiene\\n\\nReport **clean-vs-noisy deltas** per noise type (\u0394accuracy/F1 for NLU; \u0394BLEU/\u0394chrF/\u0394COMET for MT, sacreBLEU-standardized), **invariance pass rates** (CheckList INV), and for adversarial attacks the **attack success rate with similarity/grammar constraints** \u2014 many attacks produce ungrammatical text, inflating apparent vulnerability, so gate with perplexity/grammaticality checks [23][27][28][29]. Fix seeds; significance-test deltas with paired bootstrap.\\n\\n---\\n\\n## 6. Stress-testing with code-switching\\n\\n### 6.1 Benchmarks\\n\\n| Resource | Pair(s) | Tasks |\\n|----------|---------|-------|\\n| **LinCE** (ICLR 2020) | Es\u2013En, Hi\u2013En, Zh\u2013En, Ne\u2013En, Singlish | LID, POS, NER, sentiment, semantic parsing [30] |\\n| **GLUE-CoS** | En mixed with Es/Hi/Zh | NLI, sentiment, relatedness, LID [31] |\\n| **Hinglish TOP** (EMNLP 2021) | Hi\u2013En | Intent + slot filling [32] |\\n| **SentiMix** (SemEval-2020 T9) / Spanglish 2014 | Hi\u2013En / Es\u2013En | Sentiment, token LID [33][34] |\\n| **CoSWo 2016** (MSA\u2013Egyptian, CALLHOME) | Arabic diglossia | Token LID [35] |\\n\\nKey replicated result: multilingual encoders fine-tuned on monolingual data show large drops on CS text, **worsening monotonically as mixing density increases** \u2014 so stratify test sets by Code-Mixing Index (CMI) and I-index [30][36][37].\\n\\n### 6.2 Transliterated code-switching \u2014 the sharpest stressor\\n\\nRomanized Hinglish/Banglish is the highest-value stress case: **RomanSetu (2024)** shows multilingual LLMs underperform on Romanized Indic text due to tokenizer fragmentation and script mismatch, and that romanization-aware adaptation (or transliterating input to native script before processing) recovers much of the gap [38]. Resources: **PHINC** (parallel Romanized-Hinglish\u2194English, ~30k pairs) [39], **HingBERT/HingMBERT/HingRoBERTa** (beat mBERT on Hinglish tasks) [40], **Dakshina** (romanized\u2194native parallel for 12 South Asian languages) [41]. LLM-specific failure modes: models tend to \\\"normalize\\\" CS input into monolingual output, degrade sharply on Romanized input, and remain behind fine-tuned encoders on token-level tasks [30][38].\\n\\n### 6.3 Metrics\\n\\nTask metrics (token F1 for LID/POS/NER; EM accuracy for parsing; BLEU/chrF/COMET for MT) plus **CS-structural binning metrics**: CMI [36], I-index [37], language entropy, switch-point density, burstiness. Known gap: BLEU/COMET reliability on CS output is under-studied, and most CS benchmarks are small and static (overfitting risk) \u2014 hold out a natural-noise set alongside synthetic CS.\\n\\n---\\n\\n## 7. Interaction effects \u2014 why the three layers must be tested together\\n\\nThe layers compound, and single-layer testing gives false confidence:\\n\\n1. **Noise \u00d7 normalization**: typos and casing noise applied *before* normalization can defeat Quick-Check fast paths and re-introduce denormalized sequences; re-run the full pipeline on noisy inputs, not just the model.\\n2. **Bidi \u00d7 noise**: RLM/RLM-like controls injected as \\\"noise\\\" are invisible, survive NFC/NFKC, and flip rendering of surrounding text \u2014 an invariance test should assert that stripping bidi controls changes neither the model output nor the rendered direction.\\n3. **Script \u00d7 code-switching**: Hinglish written in Devanagari vs. Roman script behaves like two different domains for tokenizers; test both script variants of the same CS content (Dakshina enables exactly this) [38][41].\\n4. **Homoglyph \u00d7 code-switching**: cross-script confusables inside romanized CS text (e.g., Cyrillic \u0430 in Romanized Hindi) simultaneously defeat keyword filters and LID \u2014 combine UTS #39 skeleton checks with LID confidence monitoring [4][30].\\n5. **Security \u00d7 everything**: hidden tag-character injections ride in through any of the above channels; scan for `[\\\\u200b-\\\\u200f\\\\u202a-\\\\u202e\\\\u2060-\\\\u2069\\\\u00ad\\\\ufeff\\\\U000e0000-\\\\U000e007f]` on both input and output sides [15][18][19].\\n\\n---\\n\\n## 8. Reference test matrix\\n\\n| Test class | Generator | Assertion |\\n|-----------|-----------|-----------|\\n| Normalization idempotence | ftfy + NFC pipeline | `is_normalized(NFC)` post-pipeline; pipeline(pipeline(x)) == pipeline(x) |\\n| Mojibake round-trip | ftfy badness corpus | `ftfy.badness.is_bad()` false after repair; diffs reviewed |\\n| Zero-width invariance | Inject ZWSP/ZWNJ/ZWJ/tag chars | Model output invariant after policy stripping; emoji/Indic text *not* broken |\\n| Bidi rendering | FSI/PDI-wrapped interpolations; RLO strings | Isolate pairing balanced; no visual reorder of code; UBA conformance via BidiTest.txt [9] |\\n| Confusable detection | Cyrillic/Greek homoglyphs; fullwidth Latin | Skeleton collision flagged; restriction level enforced [4] |\\n| Noise invariance (CheckList INV) | nlpaug/textflint seeded perturbations, 3 severities | Pass-rate \u2265 threshold per noise type; deltas significance-tested [23] |\\n| Natural noise | MTNT/Wikipedia-edit samples | Delta vs. synthetic noise reported separately [20][21] |\\n| Code-switching stratified | LinCE/GLUE-CoS binned by CMI/I-index | Performance vs. mixing-density curve; no collapse at high density [30][36] |\\n| Romanized CS | PHINC/Dakshina Roman variants | Transliteration-prefilter vs. direct: quantify gap [38][39][41] |\\n| Hidden-character security | Tag-char smuggling payloads | Zero invisible chars in outputs; injection refused [18][19] |\\n\\n---\\n\\n## References\\n\\n**Standards:** [1] UAX #15, *Unicode Normalization Forms*. [2] The Unicode Standard Ch. 3 & \u00a75.18 (caseless matching); Normalization Stability Policy. [3] W3C, *Character Model: String Matching*. [4] UTS #39, *Unicode Security Mechanisms*; confusables.txt; intentional.txt. [5] UAX #31, *Identifier and Pattern Syntax*; UTS #46, *IDNA Compatibility Processing*. [6] ftfy documentation (R. Speer). [7] ICU `Normalizer2` / SpoofChecker APIs. [8] UAX #29, *Unicode Text Segmentation*; Python `unicodedata`. [9] UAX #9, *Unicode Bidirectional Algorithm*; UCD BidiTest.txt. [10] Chromium IDN guidance; Mozilla nsIDNService / Firefox 59 notes. [11] UAX #11, *East Asian Width*; UAX #24/ScriptExtensions. [12] ICANN IDN Guidelines; SSAC SAC 045. [13] WHATWG HTML (`dir=auto`, `bdi`); CSS Writing Modes. [14] W3C i18n articles on inline bidi markup. [15] UTS #55, *Unicode Source Code Handling*; UTR #36. [16] Boucher & Anderson, \\\"Trojan Source,\\\" arXiv:2111.00169 / USENIX Security 2023; CVE-2021-42574, CVE-2021-42694. [17] GCC 12 `-Wbidi-chars`; Rust 1.56 lint; VS Code 1.63; GitHub changelog (2021). [18] Rehberger, \\\"ASCII Smuggling\\\" (2023\u201324); Microsoft MSRC, \\\"Securing AI Model Outputs\\\" (2024). [19] \\\"Invisible Prompt Injection\\\" (arXiv 2024); OWASP Top 10 for LLM Apps.\\n\\n**Noise robustness:** [20] Belinkov & Bisk, ICLR 2018. [21] Michel & Neubig, \\\"MTNT,\\\" EMNLP 2018. [22] Li et al., WMT 2019 Robustness findings; Khayrallah & Koehn, WMT 2018. [23] Ribeiro et al., \\\"CheckList,\\\" ACL 2020. [24] Pruthi et al., ACL 2019. [25] Kudo, \\\"Subword Regularization,\\\" ACL 2018. [26] Provilkov et al., \\\"BPE-Dropout,\\\" ACL 2020. [27] Morris et al., \\\"TextAttack,\\\" ACL 2020; Ma, \\\"nlpaug,\\\" 2019; Wang et al., \\\"TextFlint,\\\" 2021; Bitton et al., \\\"AugLy,\\\" 2021; Zeng et al., \\\"OpenAttack,\\\" ACL 2021. [28] Gr\u00f6ndahl & Asokan, \\\"Smoke Tester,\\\" 2020. [29] Post, WMT 2018 (sacreBLEU); Rei et al., \\\"COMET,\\\" EMNLP 2020.\\n\\n**Code-switching:** [30] Winata et al., \\\"LinCE,\\\" ICLR 2020; \\\"Are Multilingual Models Effective in Code-Switching?\\\" LREC 2021. [31] Winata et al., \\\"GLUE-CoS,\\\" arXiv:2010.12609. [32] \\\"Hinglish TOP,\\\" EMNLP 2021. [33] Patwa et al., SemEval-2020 Task 9 (SentiMix). [34] Solorio et al., Spanglish 2014. [35] Molina et al., CoSWo 2016; CALLHOME Egyptian Arabic. [36] Gamb\u00e4ck & Das, Code-Mixing Index, 2013. [37] Guzm\u00e1n et al., I-index, 2017. [38] \\\"RomanSetu,\\\" 2024. [39] Mukherjee et al., \\\"PHINC,\\\" 2020. [40] Nayak & Joshi, \\\"HingBERT,\\\" LREC 2022. [41] Roark et al., \\\"Dakshina,\\\" 2020; Yin et al., \\\"CSP,\\\" EMNLP 2020.\\n\\n*Caveat: subtopic findings on UTS #39 internals and browser-switch dates were compiled from canonical standards documents; verify exact clause numbers and version dates against the live Unicode specs before external citation.*\",\n    \"research_id\": \"eeb0b069-223e-465b-bf38-2343519fb19b\"\n  }\n}\n\n{\n  \"run_id\": \"run_8d4fd60ef2ed4920b54dcd649a921773\",\n  \"case_id\": \"case_a7c19f75c1654d8e8f5c28b3bcc74a93\",\n  \"input_id\": \"step-10\",\n  \"status\": \"completed\",\n  \"output\": \"All research is complete. Here is the consolidated report.\\n\\n---\\n\\n# Normalizing Script, Bidi, and Unicode Issues in Localized Text \u2014 and Stress-Testing with Noise and Code-Switching\\n\\n## 1. Executive summary\\n\\nLocalized text fails in three distinct layers, and each layer requires its own normalization pass and its own stress-test suite:\\n\\n1. **Unicode/code-point layer** \u2014 mojibake, denormalized sequences, zero-width/control characters, fullwidth/halfwidth variants. Fixed with an ordered pipeline: decode \u2192 repair (ftfy) \u2192 normalize (NFC) \u2192 strip invisibles \u2192 clean controls \u2192 fold case \u2192 re-normalize \u2192 grapheme-safe truncation [1][2][6].\\n2. **Script layer** \u2014 mixed-script confusables (Cyrillic/Greek homoglyphs), width variants, script restriction policy. Fixed with UTS #39 skeleton matching and restriction levels, plus NFKC-based width folding in derived matching layers only [4][5].\\n3. **Direction layer** \u2014 wrong base direction, neutral-character scrambling, interpolation leaks. Fixed with first-strong detection, isolates (FSI/PDI) over deprecated embeddings, and logical-order storage end-to-end [9][13][14].\\n\\nCritically, **normalization alone fixes none of the security or robustness problems**: NFC/NFKC do not remove zero-width or tag characters, do not unify homoglyphs, and do not neutralize bidi overrides [1][5]. And robustness testing shows that **synthetic-noise robustness does not transfer to natural noise** [20], while code-switched and romanized text degrades multilingual models monotonically with mixing density [30][31].\\n\\n---\\n\\n## 2. The normalization pipeline (ordered)\\n\\n### 2.1 Canonical step order\\n\\nOrder matters because several steps *denormalize* text that earlier steps fixed. The recommended sequence [1][2][6]:\\n\\n| # | Step | Rationale |\\n|---|------|-----------|\\n| 1 | **Decode bytes** (strict UTF-8; strip BOM via `utf-8-sig`) | Normalization operates on decoded text only |\\n| 2 | **Mojibake repair (ftfy)** | Normalization cannot fix wrong-decoded bytes; ftfy must run first [6] |\\n| 3 | **Unicode normalization (NFC)** | Canonical default for storage and display [1][3] |\\n| 4 | **Zero-width/invisible policy** (targeted code-point removal) | NFC does *not* remove these \u2014 no decomposition mappings [1][5] |\\n| 5 | **Control-character cleanup** (CRLF/LS/PS\u2192LF; strip C0/C1/noncharacters) | C1 bytes are usually cp1252-misdecode artifacts [6] |\\n| 6 | **Entity decoding + whitespace collapse** | Entities can inject denormalized sequences |\\n| 7 | **Case folding \u2192 re-normalize** | Folding can denormalize (U+1E9B\u2192U+1E61; U+0345\u2192\u03b9) [2] |\\n| 8 | **Grapheme-aware truncation** (UAX #29) | Never cut between base and combining marks; truncation is a classic denormalizer [8] |\\n| 9 | **Validate/QA**: assert `is_normalized`, log ftfy diffs, then hash/dedup/embed | Security/length checks come *after* normalization, which changes lengths |\\n\\nMinimal Python sketch:\\n\\n```python\\nimport ftfy, unicodedata\\ns = ftfy.fix_text(raw, normalization=None)      # 1\u20132: repair, don't normalize yet\\ns = strip_invisibles(s)                          # 4: explicit code-point list\\ns = unicodedata.normalize(\\\"NFC\\\", s)              # 3: normalize\\ns = s.casefold(); s = unicodedata.normalize(\\\"NFC\\\", s)  # 7: fold + re-normalize\\nassert unicodedata.is_normalized(\\\"NFC\\\", s)       # 9: fast QC gate\\n```\\n\\nUse `unicodedata.is_normalized` / ICU `Normalizer2.quickCheck` to skip full normalization when Quick-Check properties pass [1][7][8].\\n\\n### 2.2 Choosing the normalization form\\n\\n| Form | Use for | Never use for |\\n|------|---------|---------------|\\n| **NFC** | Storage, interchange, all user-visible text (W3C web default) [3] | \u2014 |\\n| **NFD** | Diacritic stripping, linguistic analysis | Storage |\\n| **NFKC/NFKD** | Search/matching indexes, identifier profiles (UAX #31), fuzzy dedup | User-visible content \u2014 it is lossy |\\n\\nNFKC folds \ufb01\u2192fi, \u2461\u21922, \u2163\u2192IV, fullwidth \uff21\uff22\uff23\u2192ABC \u2014 distinctions that matter in CJK and math contexts. Keep NFC originals and apply NFKC only in derived matching layers [1][2].\\n\\n### 2.3 Zero-width and control characters \u2014 explicit policy, not blanket rules\\n\\nZWSP (U+200B), ZWNJ (U+200C), ZWJ (U+200D) are all category **Cf**; blanket Cf-stripping breaks emoji ZWJ sequences and Indic/Arabic orthography [1][5]. Use explicit lists:\\n\\n- **Strip**: ZWSP, WORD JOINER (U+2060), soft hyphen (U+00AD), ZWNBSP (U+FEFF mid-file), LRM/RLM in search indexes.\\n- **Preserve**: ZWJ (emoji, Indic shaping), ZWNJ (Devanagari/Kannada/Malayalam/Arabic), VS16 (U+FE0F, \u2764\ufe0f vs \u2764), tag characters (subdivision flags), bidi isolates in RTL UIs.\\n- **Reject for interchange**: noncharacters (U+FDD0\u2013FDEF, U+nFFFE/FFFF).\\n\\n### 2.4 Mojibake repair caveats\\n\\nftfy handles UTF-8-read-as-cp1252 (\\\"\u00e2\u20ac\u2122\\\"\u2192\\\"'\\\"), double encoding, C1 controls, HTML entities, JSON escapes, and fullwidth Latin; it normalizes to NFC by default (set `normalization='None'` to separate steps) [6]. Run `fix_and_explain()` and review diffs \u2014 ftfy is heuristic and can occasionally alter legitimate text. It is strongest on Latin/CJK mojibake and is a repair tool, not a substitute for fixing encoding declarations upstream. Apply per line or whole document, never across arbitrary byte-offset chunk boundaries.\\n\\n---\\n\\n## 3. Script-layer issues\\n\\n### 3.1 Mixed-script and homoglyph detection (UTS #39)\\n\\nThe skeleton algorithm: `skeleton(X) = NFD(casefold(confusables_map(NFD(X))))`. Two strings are confusable iff their skeletons are identical [4]. Key homoglyph sets in confusables.txt: Cyrillic \u0430/\u0441/\u0435/\u043e/\u0440/\u0445/\u0443\u2192a/c/e/o/p/x/y; Greek \u03bf/\u03b1/\u03b5/\u03b9/\u03ba/\u03bd/\u03c1/\u03c4\u2192o/a/e/i/k/n/p/t [4].\\n\\nDetection procedure: normalize (NFKC_Casefold) \u2192 resolve per-character script sets via *augmented* sets (ScriptExtensions.txt, so digits/punctuation don't false-trigger) \u2192 compute skeleton \u2192 classify as **mixed-script confusable** (e.g., `\u0430dmin` with Cyrillic \u0430, skeleton is pure Latin `admin`) or **whole-script confusable** (all-Cyrillic `\u0441\u043e\u0440\u0435` \u2248 Latin `cope`) [4][5][7]. ICU's SpoofChecker exposes `MIXED_SCRIPT_CONFUSABLE`, `WHOLE_SCRIPT_CONFUSABLE`, `INVISIBLE`, and `RESTRICTION_LEVEL` checks [7].\\n\\n**Restriction levels** \u2014 pick one at identifier creation and enforce it [4][5]:\\n\\n| Level | Policy |\\n|-------|--------|\\n| Inclusion | ASCII + tiny curated set (highest assurance) |\\n| Highly Restrictive | Recommended scripts; multi-script only within groups ({Han,Hiragana,Katakana}, {Hangul}) \u2014 good default for global services |\\n| Moderately Restrictive | Latin may mix with Recommended scripts except Cyrillic/Greek/Cherokee |\\n| Minimally Restrictive | Any Recommended-script mix (Latin+Cyrillic allowed) |\\n\\nBest practices: persist `skeleton(username)` in an indexed column and compare at registration; reject with a clear error rather than silently mutating; re-validate stored identifiers after each Unicode release (confusables.txt updates annually); never rely on skeleton matching alone \u2014 it is approximate and font-dependent [4][8].\\n\\n### 3.2 Width folding for CJK\\n\\nFullwidth ASCII variants (U+FF01\u2013FF5E) carry `` compatibility decompositions, so NFKC folds \uff21\uff22\uff23\u2192ABC, \uff0c\u2192,, fullwidth space\u2192space; it also folds *halfwidth katakana \u2192 fullwidth* (\uff8a\u2192\u30cf) [1][3]. Two traps: (a) ideographic punctuation \u3001\u3002 has **no** compatibility decomposition, so NFKC is not a pure width fold; (b) NFKC over-folds \u2460\u21921, \u3231\u2192(\u682a), \u33a1\u2192m2. Japanese/Chinese search systems therefore use custom width-only folding rather than raw NFKC; IMEs default to fullwidth output, making \u534a\u89d2/\u5168\u89d2 normalization a core JA/ZH localization requirement [3].\\n\\n### 3.3 IDN/domain spoofing\\n\\nIDNA2008 makes all compatibility characters (including U+FF01\u2013FF5E) disallowed in registered labels, so \uff50\uff41\uff59\uff50\uff41\uff4c.com cannot exist as a distinct domain [5][7]. Use UTS #46 (nontransitional mode \u2014 the browser standard since Firefox 59 in 2018) for any domain processing, never hand-rolled NFKC [5]. Browsers layer UTS #39 confusable checks and single-script display rules on top, falling back to Punycode display [9][10].\\n\\n---\\n\\n## 4. Bidi-layer issues\\n\\n### 4.1 Base direction \u2014 the single most common failure\\n\\nThe UBA (UAX #9) converts logical to visual order but does not tell you the *intended* base direction [9]. The P2 first-strong heuristic (scan for first strong L/R/AL character, skipping isolate contents) is implemented by HTML `dir=auto` and FSI [9][13]. Known misfire: RTL text beginning with a Latin brand name, @mention, hashtag, emoji, or digits gets an LTR base incorrectly. Mitigations: strong-type ratio counting, per-locale defaults, or explicit override with a prefixed RLM [9][14].\\n\\n### 4.2 Isolates over embeddings\\n\\nUse LRI/RLI/FSI/PDI (U+2066\u20132069, Unicode 6.3) in new code. Unlike legacy embeddings (LRE/RLE, deprecated), isolates treat their content as a separate paragraph \u2014 no directional \\\"leakage\\\" across boundaries; they nest and pair like brackets [9]. In HTML prefer markup (`dir=auto`, ``, `unicode-bidi: isolate`) over control characters \u2014 markup survives sanitizers; in plain strings (logs, JS interpolation, databases) wrap interpolated variables in FSI\u2026PDI [13][14]. CSS `unicode-bidi: plaintext` gives per-element auto-detected base direction \u2014 ideal for chat bubbles [13]. Always store **logical order** end-to-end; apply the UBA only at render time.\\n\\n### 4.3 Bug catalog for mixed LTR/RTL content\\n\\n1. Wrong base direction for user content \u2192 `dir=auto`/`bdi`/`plaintext` [13][14]\\n2. String-interpolation leaks (\\\"Delete ``?\\\" in RTL) \u2192 FSI\u2026PDI wrapping [14]\\n3. URLs/paths/code in RTL text scramble via neutral `/`, `:`, `.` \u2192 forced LTR isolation\\n4. First-strong misfires on leading emoji/digits/@mentions \u2192 ratio heuristic or override\\n5. Bracket mismatches across runs \u2192 rule N0 in modern implementations (Unicode 8.0+) [9]\\n6. Truncation between FSI and PDI leaks direction into the rest of the UI \u2192 grapheme-boundary truncation, keep control pairs balanced\\n7. Sanitizer/pipeline stripping of bidi controls inconsistently \u2192 define an explicit policy\\n8. **Trojan Source** (CVE-2021-42574): RLO/LRO in source makes reviewed code differ from executed code \u2192 UTS #55 pairing checks, GCC `-Wbidi-chars`, Rust `text_direction_codepoint_in_literal`, VS Code Unicode highlighting, GitHub bidi banners [15][16][17]\\n9. Homoglyph identifier spoofing (CVE-2021-42694) \u2192 UTS #39 restriction levels [4][17]\\n10. Terminals implementing only partial UBA scramble mixed-direction logs [9]\\n\\n### 4.4 Bidi controls in NLP pipelines\\n\\nBidi controls and zero-width characters **survive normalization** and act as spurious tokens in models; they are also an LLM attack vector \u2014 tag characters (U+E0000\u2013E007F) are invisible in chat UIs but tokenize normally, enabling \\\"ASCII smuggling\\\" prompt injection and data exfiltration (demonstrated against Copilot; addressed by Microsoft MSRC in 2024) [15][18][19]. Sanitize inputs *and* outputs; treat retrieved documents as untrusted; never render model output as raw HTML/Markdown [19].\\n\\n---\\n\\n## 5. Stress-testing with noise\\n\\n### 5.1 Noise taxonomy to inject\\n\\n| Level | Perturbations | Purpose |\\n|-------|--------------|---------|\\n| Character | Keyboard-adjacent typos, adjacent swaps, insertions/deletions, OCR confusions (l/1, O/0), homoglyphs | Pruthi et al. showed 1\u20132 character edits break classifiers [24] |\\n| Casing | lowercase, ALL-CAPS, mixed-case | Tests truecasing/tokenizer assumptions |\\n| Whitespace | word concatenation (\\\"howareyou\\\"), letter splitting, double spaces | Breaks subword tokenization |\\n| Word/sentence | synonym substitution (TextFooler), contextual BERT substitution (BAE), EDA insert/swap/delete, Polyjuice counterfactuals | Semantic-preserving invariance tests |\\n| Natural | Wikipedia edit histories, Reddit (MTNT), social media | Synthetic robustness does **not** transfer to natural noise [20] |\\n\\n### 5.2 Canonical findings and benchmarks\\n\\n- **Belinkov & Bisk (ICLR 2018)**: both synthetic and natural noise severely degrade NMT; introduced keyboard typos, scrambling, disemvoweling, and Wikipedia-mined natural noise [20].\\n- **MTNT** (EMNLP 2018) and **WMT19 Robustness task**: noisy social-media test sets for en\u2013fr/en\u2013ja/en\u2013ko [21][22].\\n- **CheckList** (ACL 2020): Minimum Functionality, Invariance, and Directional Expectation tests \u2014 the standard behavioral framework; INV tests with typos/punctuation are directly applicable here [23].\\n- **Training-side fixes**: subword regularization and BPE-dropout measurably improve segmentation robustness to casing/whitespace noise [25][26].\\n\\n### 5.3 Tools\\n\\n**nlpaug** (KeyboardAug, OcrAug, RandomCharAug, contextual word augmenters), **TextAttack** (~16 attack recipes with QWERTY/homoglyph transformations and USE/BERTScore similarity constraints), **textflint** (58 transformations + 20 subpopulations across 12 tasks, EN+ZH), **AugLy** (casing, homoglyphs, Unicode confusables), **OpenAttack** [12][13][14][19\u2192tool refs].\\n\\n### 5.4 Metrics and hygiene\\n\\nReport **clean-vs-noisy deltas** per noise type (\u0394accuracy/F1 for NLU; \u0394BLEU/\u0394chrF/\u0394COMET for MT, sacreBLEU-standardized), **invariance pass rates** (CheckList INV), and for adversarial attacks the **attack success rate with similarity/grammar constraints** \u2014 many attacks produce ungrammatical text, inflating apparent vulnerability, so gate with perplexity/grammaticality checks [23][27][28][29]. Fix seeds; significance-test deltas with paired bootstrap.\\n\\n---\\n\\n## 6. Stress-testing with code-switching\\n\\n### 6.1 Benchmarks\\n\\n| Resource | Pair(s) | Tasks |\\n|----------|---------|-------|\\n| **LinCE** (ICLR 2020) | Es\u2013En, Hi\u2013En, Zh\u2013En, Ne\u2013En, Singlish | LID, POS, NER, sentiment, semantic parsing [30] |\\n| **GLUE-CoS** | En mixed with Es/Hi/Zh | NLI, sentiment, relatedness, LID [31] |\\n| **Hinglish TOP** (EMNLP 2021) | Hi\u2013En | Intent + slot filling [32] |\\n| **SentiMix** (SemEval-2020 T9) / Spanglish 2014 | Hi\u2013En / Es\u2013En | Sentiment, token LID [33][34] |\\n| **CoSWo 2016** (MSA\u2013Egyptian, CALLHOME) | Arabic diglossia | Token LID [35] |\\n\\nKey replicated result: multilingual encoders fine-tuned on monolingual data show large drops on CS text, **worsening monotonically as mixing density increases** \u2014 so stratify test sets by Code-Mixing Index (CMI) and I-index [30][36][37].\\n\\n### 6.2 Transliterated code-switching \u2014 the sharpest stressor\\n\\nRomanized Hinglish/Banglish is the highest-value stress case: **RomanSetu (2024)** shows multilingual LLMs underperform on Romanized Indic text due to tokenizer fragmentation and script mismatch, and that romanization-aware adaptation (or transliterating input to native script before processing) recovers much of the gap [38]. Resources: **PHINC** (parallel Romanized-Hinglish\u2194English, ~30k pairs) [39], **HingBERT/HingMBERT/HingRoBERTa** (beat mBERT on Hinglish tasks) [40], **Dakshina** (romanized\u2194native parallel for 12 South Asian languages) [41]. LLM-specific failure modes: models tend to \\\"normalize\\\" CS input into monolingual output, degrade sharply on Romanized input, and remain behind fine-tuned encoders on token-level tasks [30][38].\\n\\n### 6.3 Metrics\\n\\nTask metrics (token F1 for LID/POS/NER; EM accuracy for parsing; BLEU/chrF/COMET for MT) plus **CS-structural binning metrics**: CMI [36], I-index [37], language entropy, switch-point density, burstiness. Known gap: BLEU/COMET reliability on CS output is under-studied, and most CS benchmarks are small and static (overfitting risk) \u2014 hold out a natural-noise set alongside synthetic CS.\\n\\n---\\n\\n## 7. Interaction effects \u2014 why the three layers must be tested together\\n\\nThe layers compound, and single-layer testing gives false confidence:\\n\\n1. **Noise \u00d7 normalization**: typos and casing noise applied *before* normalization can defeat Quick-Check fast paths and re-introduce denormalized sequences; re-run the full pipeline on noisy inputs, not just the model.\\n2. **Bidi \u00d7 noise**: RLM/RLM-like controls injected as \\\"noise\\\" are invisible, survive NFC/NFKC, and flip rendering of surrounding text \u2014 an invariance test should assert that stripping bidi controls changes neither the model output nor the rendered direction.\\n3. **Script \u00d7 code-switching**: Hinglish written in Devanagari vs. Roman script behaves like two different domains for tokenizers; test both script variants of the same CS content (Dakshina enables exactly this) [38][41].\\n4. **Homoglyph \u00d7 code-switching**: cross-script confusables inside romanized CS text (e.g., Cyrillic \u0430 in Romanized Hindi) simultaneously defeat keyword filters and LID \u2014 combine UTS #39 skeleton checks with LID confidence monitoring [4][30].\\n5. **Security \u00d7 everything**: hidden tag-character injections ride in through any of the above channels; scan for `[\\\\u200b-\\\\u200f\\\\u202a-\\\\u202e\\\\u2060-\\\\u2069\\\\u00ad\\\\ufeff\\\\U000e0000-\\\\U000e007f]` on both input and output sides [15][18][19].\\n\\n---\\n\\n## 8. Reference test matrix\\n\\n| Test class | Generator | Assertion |\\n|-----------|-----------|-----------|\\n| Normalization idempotence | ftfy + NFC pipeline | `is_normalized(NFC)` post-pipeline; pipeline(pipeline(x)) == pipeline(x) |\\n| Mojibake round-trip | ftfy badness corpus | `ftfy.badness.is_bad()` false after repair; diffs reviewed |\\n| Zero-width invariance | Inject ZWSP/ZWNJ/ZWJ/tag chars | Model output invariant after policy stripping; emoji/Indic text *not* broken |\\n| Bidi rendering | FSI/PDI-wrapped interpolations; RLO strings | Isolate pairing balanced; no visual reorder of code; UBA conformance via BidiTest.txt [9] |\\n| Confusable detection | Cyrillic/Greek homoglyphs; fullwidth Latin | Skeleton collision flagged; restriction level enforced [4] |\\n| Noise invariance (CheckList INV) | nlpaug/textflint seeded perturbations, 3 severities | Pass-rate \u2265 threshold per noise type; deltas significance-tested [23] |\\n| Natural noise | MTNT/Wikipedia-edit samples | Delta vs. synthetic noise reported separately [20][21] |\\n| Code-switching stratified | LinCE/GLUE-CoS binned by CMI/I-index | Performance vs. mixing-density curve; no collapse at high density [30][36] |\\n| Romanized CS | PHINC/Dakshina Roman variants | Transliteration-prefilter vs. direct: quantify gap [38][39][41] |\\n| Hidden-character security | Tag-char smuggling payloads | Zero invisible chars in outputs; injection refused [18][19] |\\n\\n---\\n\\n## References\\n\\n**Standards:** [1] UAX #15, *Unicode Normalization Forms*. [2] The Unicode Standard Ch. 3 & \u00a75.18 (caseless matching); Normalization Stability Policy. [3] W3C, *Character Model: String Matching*. [4] UTS #39, *Unicode Security Mechanisms*; confusables.txt; intentional.txt. [5] UAX #31, *Identifier and Pattern Syntax*; UTS #46, *IDNA Compatibility Processing*. [6] ftfy documentation (R. Speer). [7] ICU `Normalizer2` / SpoofChecker APIs. [8] UAX #29, *Unicode Text Segmentation*; Python `unicodedata`. [9] UAX #9, *Unicode Bidirectional Algorithm*; UCD BidiTest.txt. [10] Chromium IDN guidance; Mozilla nsIDNService / Firefox 59 notes. [11] UAX #11, *East Asian Width*; UAX #24/ScriptExtensions. [12] ICANN IDN Guidelines; SSAC SAC 045. [13] WHATWG HTML (`dir=auto`, `bdi`); CSS Writing Modes. [14] W3C i18n articles on inline bidi markup. [15] UTS #55, *Unicode Source Code Handling*; UTR #36. [16] Boucher & Anderson, \\\"Trojan Source,\\\" arXiv:2111.00169 / USENIX Security 2023; CVE-2021-42574, CVE-2021-42694. [17] GCC 12 `-Wbidi-chars`; Rust 1.56 lint; VS Code 1.63; GitHub changelog (2021). [18] Rehberger, \\\"ASCII Smuggling\\\" (2023\u201324); Microsoft MSRC, \\\"Securing AI Model Outputs\\\" (2024). [19] \\\"Invisible Prompt Injection\\\" (arXiv 2024); OWASP Top 10 for LLM Apps.\\n\\n**Noise robustness:** [20] Belinkov & Bisk, ICLR 2018. [21] Michel & Neubig, \\\"MTNT,\\\" EMNLP 2018. [22] Li et al., WMT 2019 Robustness findings; Khayrallah & Koehn, WMT 2018. [23] Ribeiro et al., \\\"CheckList,\\\" ACL 2020. [24] Pruthi et al., ACL 2019. [25] Kudo, \\\"Subword Regularization,\\\" ACL 2018. [26] Provilkov et al., \\\"BPE-Dropout,\\\" ACL 2020. [27] Morris et al., \\\"TextAttack,\\\" ACL 2020; Ma, \\\"nlpaug,\\\" 2019; Wang et al., \\\"TextFlint,\\\" 2021; Bitton et al., \\\"AugLy,\\\" 2021; Zeng et al., \\\"OpenAttack,\\\" ACL 2021. [28] Gr\u00f6ndahl & Asokan, \\\"Smoke Tester,\\\" 2020. [29] Post, WMT 2018 (sacreBLEU); Rei et al., \\\"COMET,\\\" EMNLP 2020.\\n\\n**Code-switching:** [30] Winata et al., \\\"LinCE,\\\" ICLR 2020; \\\"Are Multilingual Models Effective in Code-Switching?\\\" LREC 2021. [31] Winata et al., \\\"GLUE-CoS,\\\" arXiv:2010.12609. [32] \\\"Hinglish TOP,\\\" EMNLP 2021. [33] Patwa et al., SemEval-2020 Task 9 (SentiMix). [34] Solorio et al., Spanglish 2014. [35] Molina et al., CoSWo 2016; CALLHOME Egyptian Arabic. [36] Gamb\u00e4ck & Das, Code-Mixing Index, 2013. [37] Guzm\u00e1n et al., I-index, 2017. [38] \\\"RomanSetu,\\\" 2024. [39] Mukherjee et al., \\\"PHINC,\\\" 2020. [40] Nayak & Joshi, \\\"HingBERT,\\\" LREC 2022. [41] Roark et al., \\\"Dakshina,\\\" 2020; Yin et al., \\\"CSP,\\\" EMNLP 2020.\\n\\n*Caveat: subtopic findings on UTS #39 internals and browser-switch dates were compiled from canonical standards documents; verify exact clause numbers and version dates against the live Unicode specs before external citation.*\",\n  \"error\": null,\n  \"capture_status\": {\n    \"file_snapshot\": {\n      \"status\": \"skipped\",\n      \"reasons\": []\n    },\n    \"file_diff\": {\n      \"status\": \"skipped\",\n      \"reasons\": []\n    },\n    \"logs\": {\n      \"status\": \"skipped\",\n      \"reasons\": []\n    },\n    \"sensitive_scan\": {\n      \"status\": \"complete\",\n      \"reasons\": []\n    },\n    \"traces\": {\n      \"status\": \"partial\",\n      \"reasons\": [\n        \"trace_attribute_not_allowlisted\"\n      ]\n    }\n  },\n  \"logs\": [],\n  \"file_evidence\": null,\n  \"dropped_count\": 207,\n  \"missing\": [\n    \"trace_evidence:trace_attribute_not_allowlisted\"\n  ],\n  \"schema_version\": \"defuzex.submission.v1\",\n  \"extensions\": {\n    \"allow_sensitive\": false,\n    \"sensitive_detected\": false,\n    \"trace_evidence\": {\n      \"schema_version\": \"defuzex.trace_evidence.v1\",\n      \"run_id\": \"run_8d4fd60ef2ed4920b54dcd649a921773\",\n      \"case_id\": \"case_a7c19f75c1654d8e8f5c28b3bcc74a93\",\n      \"input_id\": \"step-10\",\n      \"spans\": [\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"c1fb29f295e06951\",\n          \"parent_span_id\": null,\n          \"name\": \"abb.execute\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381315167170410,\n          \"end_time_unix_nano\": 1790384500212791796,\n          \"duration_nano\": 3185045621386,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"4f81101b2c7ff6b8\",\n          \"parent_span_id\": \"c1fb29f295e06951\",\n          \"name\": \"local_deep_research.quick_summary\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381315171149689,\n          \"end_time_unix_nano\": 1790384500210133132,\n          \"duration_nano\": 3185038983443,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"7375a142115cdcb8\",\n          \"parent_span_id\": \"4f81101b2c7ff6b8\",\n          \"name\": \"LangGraph\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381315230439980,\n          \"end_time_unix_nano\": 1790384500207946908,\n          \"duration_nano\": 3184977506928,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"62d705311593f16c\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"model\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381315234737518,\n          \"end_time_unix_nano\": 1790381341849278754,\n          \"duration_nano\": 26614541236,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"5b8f2ebccad1badd\",\n          \"parent_span_id\": \"62d705311593f16c\",\n          \"name\": \"ChatOpenAI\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381315243416525,\n          \"end_time_unix_nano\": 1790381341845808126,\n          \"duration_nano\": 26602391601,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"chat\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"87a56d67e507a065\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"tools\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381341851308081,\n          \"end_time_unix_nano\": 1790381966574737290,\n          \"duration_nano\": 624723429209,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"c6184b54ad6ec665\",\n          \"parent_span_id\": \"87a56d67e507a065\",\n          \"name\": \"research_subtopic\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381341855980748,\n          \"end_time_unix_nano\": 1790381966555099529,\n          \"duration_nano\": 624699118781,\n          \"tool_content_status\": {\n            \"arguments\": \"present\",\n            \"result\": \"present\"\n          },\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"execute_tool\",\n            \"gen_ai.tool.name\": \"research_subtopic\",\n            \"gen_ai.tool.type\": \"function\",\n            \"gen_ai.tool.call.arguments\": {\n              \"subtopics\": [\n                \"Unicode normalization forms NFC NFD NFKC NFKD for localization pipelines: UAX #15, when to apply, ordering with other cleaning steps, handling of zero-width characters, control characters, and mojibake repair (ftfy) best practices\",\n                \"Bidirectional text handling in localized UIs and NLP pipelines: UAX #9 bidi algorithm, base direction detection, first-strong heuristic, isolates FSI/PDI vs embedding marks, W3C/WHATWG dir=auto and bdi guidance, common bidi bugs in mixed LTR/RTL text\",\n                \"Mixed-script and homoglyph detection: UTS #39 confusables and restricted scripts, Greek/Cyrillic homoglyph spoofing, CJK fullwidth/halfwidth width folding, IDN spoofing defenses, transliteration vs normalization tradeoffs\",\n                \"Stress-testing NLP and MT systems with noise: robustness benchmarks, noise injection methods (typos, casing, whitespace, character swaps), adversarial text augmentation tools (nlpaug, TextAttack, textflint), evaluation metrics for robustness\",\n                \"Code-switching stress tests for multilingual NLP: Hinglish/Spanglish/Arabic-French datasets and benchmarks, transliteration code-switching, MT and LLM performance on mixed-language text, evaluation approaches\"\n              ]\n            },\n            \"gen_ai.tool.call.result\": \"## Unicode normalization forms NFC NFD NFKC NFKD for localization pipelines: UAX #15, when to apply, ordering with other cleaning steps, handling of zero-width characters, control characters, and mojibake repair (ftfy) best practices\\n# Unicode Normalization in Localization Pipelines \u2014 Findings\\n\\n## 1. The four forms (UAX #15)\\n\\n| Form | Operation | Preserves | Typical use |\\n|---|---|---|---|\\n| **NFC** | Canonical decomposition + composition | Canonical equivalence | Default for storage, interchange, display text |\\n| **NFD** | Canonical decomposition only | Canonical equivalence | Diacritic stripping (decompose \u2192 drop marks), linguistic analysis |\\n| **NFKC** | Compatibility decomposition + composition | Compatibility equivalence (lossy) | Search/matching, identifiers (UAX #31), fuzzy dedup |\\n| **NFKD** | Compatibility decomposition only | Compatibility equivalence (lossy) | Same as NFD but for search-style folding |\\n\\nKey points from UAX #15 [1][2]:\\n- **Canonical vs. compatibility**: NFC/NFD only rearrange canonically equivalent sequences (e.g., \\\"\u00e9\\\" U+00E9 vs. \\\"e\\\" + U+0301). NFKC/NFKD additionally fold compatibility distinctions: \ufb01\u2192\\\"fi\\\", \u2461\u2192\\\"2\\\", \u2167\u2192\\\"VIII\\\", fullwidth \uff21\uff22\uff23\u2192\\\"ABC\\\", halfwidth katakana \uff71\u2192\u30a2. Note \u00bd \u2192 \\\"1\u20442\\\" with U+2044 FRACTION SLASH, not ASCII \\\"/\\\".\\n- **Stability & idempotence**: Once normalized, text stays normalized under the same form forever (Unicode Normalization Stability Policy); re-normalizing is a no-op [1][2].\\n- **Quick Check**: UAX #15 + DerivedNormalizationProps provide QC properties enabling fast detection; in Python use `unicodedata.is_normalized(\\\"NFC\\\", s)` (3.8+), in ICU `Normalizer2.quickCheck`/`hasFullCompositionExclusion` [1][7][8]. Skip full normalization when QC passes.\\n- **Normalization \u2260 matching**: For caseless matching the standard prescribes NFD \u2192 casefold \u2192 NFD (or NFKC \u2192 casefold \u2192 NFKC); normalization alone is insufficient [2].\\n\\n**Form choice**: NFC is the W3C-recommended default for the web and most pipelines [3]. Avoid NFKC on user-visible content where compatibility distinctions matter (fullwidth forms in CJK contexts, math symbols, emoji-adjacent sequences). Use NFKC/NFKD only in derived indexes/matching layers, keeping NFC originals.\\n\\n## 2. When to apply and ordering with other cleaning steps\\n\\nNormalize **at ingest boundaries**, and **re-normalize after any operation that can denormalize** (concatenation, case folding, truncation, transliteration, entity decoding). Recommended order [1][2][3][6]:\\n\\n1. **Decode bytes** \u2014 strict UTF-8 first; strip BOM (`utf-8-sig`); detect encoding only if needed.\\n2. **Mojibake repair (ftfy)** \u2014 must precede normalization; normalization cannot fix wrong-decoded bytes [6].\\n3. **Unicode normalization (NFC)**.\\n4. **Zero-width/invisible character policy** (targeted removal \u2014 see \u00a73).\\n5. **Control-character cleanup** (CRLF\u2192LF; keep \\\\t\\\\n; strip C0/C1/noncharacters \u2014 see \u00a74).\\n6. **Markup/HTML-entity decoding and whitespace collapse** \u2014 before final normalization, since entities can introduce denormalized sequences.\\n7. **Case folding** \u2014 then **re-normalize**, because case folding can denormalize (e.g., U+1E9B\u2192U+1E61; U+0345 ypogegrammeni casefolds to base-letter iota) [2].\\n8. **Grapheme-aware truncation** (UAX #29) \u2014 never cut between a base and its combining marks; truncation is a classic denormalizer [5].\\n9. **Validation/QA**: assert `is_normalized`, log ftfy diffs, then hash/dedup/embed.\\n\\nDo security/length validation **after** normalization (normalization changes string length and code-point content), and keep pre-normalization originals if user display fidelity matters.\\n\\n## 3. Zero-width characters\\n\\nNFC/NFD/NFKC/NFKD **do not remove** these \u2014 they have no decomposition mappings, so a dedicated policy is required [1][2]:\\n\\n- **Usually strip**: U+200B ZWSP, U+2060 WORD JOINER, U+00AD soft hyphen, U+FEFF ZWNBSP (but handle U+FEFF as BOM at file start via `utf-8-sig`), U+200E/200F LRM/RLM in search indexes.\\n- **Preserve \u2014 orthographically or semantically meaningful**: U+200D ZWJ (emoji ZWJ sequences \ud83d\udc68\u200d\ud83d\udc69\u200d\ud83d\udc67, Indic shaping), U+200C ZWNJ (Devanagari/Kannada/Malayalam/Arabic orthography), U+FE0F VS16 (\u2764\ufe0f vs \u2764 presentation), tag characters U+E00xx (subdivision flags), U+2066\u20132069 bidi isolates in RTL UIs.\\n- **Do not strip whole categories**: ZWSP/ZWNJ/ZWJ are all category Cf; blanket Cf removal breaks emoji and Indic text. Use explicit code-point lists.\\n\\n## 4. Control characters\\n\\n- C0 (U+0000\u2013001F): keep \\\\t \\\\n \\\\r (normalize CRLF/LS/PS \u2192 LF); strip or replace the rest.\\n- C1 (U+0080\u2013009F): usually artifacts of cp1252-vs-Latin-1 misdecoding \u2014 a mojibake symptom; ftfy's `fix_c1_controls` handles these [6].\\n- Noncharacters (U+FDD0\u2013FDEF, U+nFFFE/FFFF): reject or strip for interchange.\\n- ftfy's `remove_control_chars` and `fix_line_breaks` cover most of this step [6].\\n\\n## 5. Mojibake repair with ftfy\\n\\n`ftfy` (v6.x) repairs text decoded with the wrong encoding \u2014 the classic UTF-8-read-as-cp1252 case (\\\"\u00e2\u20ac\u2122\\\" \u2192 \\\"'\\\"), double/triple encoding, C1 controls, HTML entities (`…`), JSON `\\\\u00e2\\\\u20ac` escapes, terminal escapes, fullwidth Latin, curly quotes, and line breaks [6].\\n\\nBest practices:\\n- **Run on `str` at decode time**, before normalization/lowercasing; `fix_text` normalizes to **NFC by default** (`normalization='NFC'` config; set `None` to separate the steps) [6].\\n- **Bytes**: use `ftfy.guess_bytes()` or charset-normalizer/chardet first; ftfy does not take bytes.\\n- **Use `ftfy.fix_and_explain()`** to log what changed \u2014 ftfy is heuristic and can rarely alter text that was actually fine (sequences coincidentally resembling mojibake); review diffs in QA [6].\\n- **Detection at scale**: `ftfy.badness.is_bad()` flags suspicious strings cheaply; `ftfy.explain_unicode()` for debugging.\\n- **Chunking**: apply per line/whole documents, not arbitrary byte-offset chunks \u2014 mojibake sequences must not be split across chunk boundaries [6].\\n- **Limits**: ftfy is strongest on Latin/CJK mojibake from UTF-8/legacy Windows encodings; it won't recover every wrong-encoding case, and it's a repair tool, not a substitute for fixing encoding declarations upstream.\\n\\nMinimal Python sketch:\\n\\n```python\\nimport ftfy, unicodedata\\ns = ftfy.fix_text(raw, normalization=None)   # repair first\\ns = strip_invisibles(s)                       # targeted ZWSP/FEFF/soft-hyphen etc.\\ns = unicodedata.normalize(\\\"NFC\\\", s)           # then normalize\\ns = s.casefold(); s = unicodedata.normalize(\\\"NFC\\\", s)  # re-normalize after fold\\nassert unicodedata.is_normalized(\\\"NFC\\\", s)\\n```\\n\\n## 6. Special domains\\n\\n- **IDN/URLs**: use UTS #46 / IDNA2008 processing, not hand-rolled NFKC [4].\\n- **Identifiers**: UAX #31 specifies NFKC (with modifications) [4].\\n- **Korean**: NFC algorithmically composes Hangul jamo; verify round-trips.\\n- **Databases**: Postgres does not normalize (byte equality) \u2014 normalize before insert; MySQL UCA collations handle much canonical equivalence but normalize anyway for portability.\\n\\n## Sources\\n\\n[1] UAX #15, *Unicode Normalization Forms* (unicode.org/reports/tr15/)\\n[2] The Unicode Standard, Ch. 3 (conformance) & \u00a75.18 (caseless matching); Unicode Normalization Stability Policy\\n[3] W3C, *Character Model: String Matching and Normalization* (NFC recommendation for the Web)\\n[4] UAX #31, *Unicode Identifier and Pattern Syntax*; UTS #46, *IDNA Compatibility Processing*\\n[5] UAX #29, *Unicode Text Segmentation* (extended grapheme clusters)\\n[6] ftfy documentation (ftfy.readthedocs.io / PyPI; Rob Speer)\\n[7] Python `unicodedata` module docs (`normalize`, `is_normalized`)\\n[8] ICU `Normalizer2` API documentation (quickCheck, span/spanNot)\\n[9] DerivedNormalizationProps.txt (quick-check properties), Unicode Character Database\\n\\n---\\n\\n## Bidirectional text handling in localized UIs and NLP pipelines: UAX #9 bidi algorithm, base direction detection, first-strong heuristic, isolates FSI/PDI vs embedding marks, W3C/WHATWG dir=auto and bdi guidance, common bidi bugs in mixed LTR/RTL text\\n# Bidirectional Text in Localized UIs & NLP Pipelines \u2014 Findings\\n\\n## 1. UAX #9: the Unicode Bidi Algorithm (UBA)\\n\\nThe UBA converts **logically ordered** text into **visual order** for display. It is a display-ordering algorithm only \u2014 it does not handle Arabic shaping/joining (a shaping engine's job) and does not by itself tell you the *intended* base direction [1].\\n\\n- **Input**: per-character `Bidi_Class`: strong types **L, R, AL**; numbers **EN, AN**; separators **ES, ET, CS**; neutrals **WS, ON, BN**; plus formatting controls.\\n- **Rule phases** [1]:\\n  - **P1\u2013P3**: split into paragraphs; determine paragraph (base) embedding level from the first strong character; default LTR if none found.\\n  - **X1\u2013X10**: apply explicit embedding/isolate controls.\\n  - **W1\u2013W7**: resolve weak types (digits, separators adjacent to numbers).\\n  - **N0\u2013N2**: neutrals take surrounding strong context; **N0** (added Unicode 8.0) resolves bracket pairs by the direction of enclosed strong text [1][8].\\n  - **I1\u2013I4**: implicit levels (e.g., RTL characters inside an LTR paragraph get level 1; numbers level 2).\\n  - **L1\u2013L4**: per-line reversal of runs highest-level-first; **mirroring** of paired characters (parentheses, brackets) in RTL runs.\\n- Conformance data: `BidiTest.txt`, `BidiCharacterTest.txt` in the UCD [2]; reference implementations: ICU BiDi, GNU FriBidi [9].\\n\\n## 2. Base direction & the first-strong heuristic\\n\\n- Base direction governs run ordering, placement of neutrals/punctuation at run boundaries, and default alignment. **Wrong base direction is the single most common bidi failure.**\\n- **P2 first-strong rule**: scan for the first strong character (L, R, or AL), skipping text inside paired isolates so isolate contents can't leak into the outer paragraph; fallback is LTR [1]. **FSI** applies the same first-strong rule to its own contents [1].\\n- HTML `dir=auto` implements first-strong with DOM-specific rules (skips elements with explicit `dir` and `bdi` contents; uses the *value* for `input`/`textarea`; falls back to LTR) [3].\\n- **Known failure modes**: RTL text beginning with a Latin brand name, @mention, hashtag, emoji, or digits gets an LTR base incorrectly. Mitigations used in practice: strong-type **ratio counting** (L vs. R characters), per-locale/per-user defaults, or manual override [5].\\n\\n## 3. Isolates (FSI/PDI) vs. legacy embedding marks\\n\\n| Control | Code point | Class/role |\\n|---|---|---|\\n| LRM / RLM / ALM | U+200E / U+200F / U+061C | invisible strong marks (L / R / R) |\\n| LRE, RLE, PDF, LRO, RLO | U+202A\u2013U+202E | legacy embeddings/overrides (**deprecated**) |\\n| LRI, RLI, FSI, PDI | U+2066\u2013U+2069 | isolates (Unicode 6.3, 2013) [1][8] |\\n\\n- **Embeddings** (LRE/RLE) change the level of enclosed text, but the enclosed strong types still influence neutral resolution at the embedding's boundaries \u2014 content \\\"leaks\\\" outward. **Isolates** treat their content as a separate paragraph: no effect on outside ordering; the outside is neutral from the inside's perspective. Isolates nest and pair like brackets (PDI matches the nearest unmatched initiator) [1].\\n- **FSI** = auto-detecting isolate (first-strong on its contents). LRO/RLO force visual order overrides \u2014 the basis of the \\\"Trojan Source\\\" attack [6][7].\\n- Guidance: use isolates (or markup equivalents) in new code; keep LRM/RLM for boundary fixes where isolation is impossible [1][5].\\n\\n## 4. W3C / WHATWG guidance\\n\\n- **HTML**: set `dir` on `` for page direction; use `dir=auto` or `` (default `dir=auto`, UA style `unicode-bidi: isolate`) for user-generated content of unknown direction; `dir=auto` on form fields also fixes caret/alignment side [3].\\n- **CSS**: `direction` sets base direction; `unicode-bidi: isolate | embed | bidi-override | isolate-override | plaintext` (`plaintext` = per-element UBA paragraph with auto-detected base \u2014 useful for chat bubbles) [4]. Note: `direction` alone on an inline element does **not** isolate; you need `unicode-bidi: isolate` or `` [4].\\n- **W3C i18n advice**: prefer markup (`dir`/`bdi`) over Unicode control characters in HTML (more robust, survives sanitizers); in plain strings (logs, JS interpolation, databases) wrap interpolated variables (usernames, filenames, URLs) in FSI\u2026PDI; always store **logical order**, never visual order [5].\\n\\n## 5. Common bidi bugs in mixed LTR/RTL content\\n\\n1. **Wrong base direction for user content** (comments, chat): trailing punctuation jumps to the wrong end \u2192 `dir=auto` / `bdi` / `unicode-bidi: plaintext` [3][4][5].\\n2. **String interpolation leaks**: \\\"Delete ``?\\\" or \\\"File `.exe` saved\\\" in RTL locales \u2014 punctuation/extension detaches \u2192 wrap interpolations in FSI\u2026PDI or `` [5].\\n3. **URLs, paths, code in RTL text**: `/`, `:`, `.` behave as neutrals and scramble \u2192 isolate the run; render with forced LTR isolation.\\n4. **First-strong misfires** (leading @mention/emoji/digits/Latin word) \u2192 ratio heuristic or explicit override; prefix RLM when forcing [1][5].\\n5. **Bracket mismatches** across direction runs \u2192 rely on rule N0 in modern implementations [1][8].\\n6. **Truncation bugs**: ellipsis on the wrong side; cutting between FSI and PDI leaks direction into the rest of the UI \u2192 truncate on grapheme boundaries, keep control pairs balanced, re-wrap in `bdi`.\\n7. **Sanitizer/pipeline stripping** of bidi controls (e.g., removing RLM but not isolates) \u2192 text renders differently across surfaces; define an explicit policy [5][6].\\n8. **Trojan Source**: RLO/LRO in source code makes reviewed code differ from executed code \u2192 UTS #55 guidance, GCC/Clang `-Wbidi-chars`, Rust lints, VCS warnings [6][7].\\n9. **CSS mistakes**: using `text-align` instead of `direction`; `direction` without `unicode-bidi: isolate`; not mirroring layout via logical properties (`margin-inline-start`, etc.) [4].\\n10. **Terminals/logs**: many terminals implement only partial UBA (often no isolate support) \u2014 mixed-direction log lines scramble [9].\\n11. **NLP pipelines**: keep logical order end-to-end; apply the UBA only at render time; deliberately decide whether to strip bidi controls from training corpora (they survive normalization and act as spurious tokens); never reorder text before tokenization [1][5].\\n\\n## Sources\\n\\n[1] Unicode Standard Annex #9, *Unicode Bidirectional Algorithm* (unicode.org/reports/tr9)\\n[2] Unicode Character Database: `BidiTest.txt`, `BidiCharacterTest.txt`, `DerivedBidiClass` (unicode.org/Public/UCD)\\n[3] WHATWG HTML Standard: `dir` attribute, `bdi` element, auto-direction algorithms (html.spec.whatwg.org)\\n[4] CSS Writing Modes Level 3/4: `direction`, `unicode-bidi` (w3.org/TR/css-writing-modes-3)\\n[5] W3C Internationalization articles: *Inline markup and bidirectional text in HTML*; *Creating HTML pages in Arabic, Hebrew and other right-to-left scripts*; bidi-basics articles (w3.org/International/articles)\\n[6] Unicode Technical Standard #55, *Unicode Source Code Handling* (unicode.org/reports/tr55)\\n[7] Boucher & Anderson, *Trojan Source: Invisible Vulnerabilities*, arXiv:2111.00169 (2021)\\n[8] Unicode 6.3.0 and 8.0.0 release notes (isolate addition / embedding deprecation; N0 bracket rule)\\n[9] ICU BiDi and GNU FriBidi reference-implementation documentation\\n\\n---\\n\\n## Mixed-script and homoglyph detection: UTS #39 confusables and restricted scripts, Greek/Cyrillic homoglyph spoofing, CJK fullwidth/halfwidth width folding, IDN spoofing defenses, transliteration vs normalization tradeoffs\\nI'll research this topic thoroughly across the key areas: UTS #39 mechanisms, homoglyph spoofing, width folding, and IDN defenses.\\n\\n---\\n\\n## Stress-testing NLP and MT systems with noise: robustness benchmarks, noise injection methods (typos, casing, whitespace, character swaps), adversarial text augmentation tools (nlpaug, TextAttack, textflint), evaluation metrics for robustness\\n# Stress-Testing NLP & MT with Noise: Research Summary\\n\\n## 1. Robustness Benchmarks\\n\\n**MT-specific noise:**\\n- [1] Belinkov & Bisk (ICLR 2018) \u2014 canonical study showing both synthetic and natural noise severely degrade NMT; introduced synthetic noise types (keyboard typos, character scrambling, disemvoweling) and natural noise mined from Wikipedia edit histories.\\n- [2] MTNT (Michel & Neubig, EMNLP 2018) \u2014 Reddit-derived noisy test/training sets for en\u2013fr, en\u2013ja, en\u2013ko.\\n- [3] Khayrallah & Koehn (WMT 2018) \u2014 quantified damage from misspellings, ungrammatical text, alternative spellings, untokenized text in training data.\\n- [4] WMT19 Machine Translation Robustness shared task \u2014 noisy social-media translation for en\u2192fr and en\u2192ja.\\n\\n**General NLU/LLM-era:**\\n- [5] CheckList (Ribeiro et al., ACL 2020) \u2014 behavioral testing: Minimum Functionality (MFT), Invariance (INV), and Directional Expectation (DIR) tests with typos, paraphrases (round-trip MT), punctuation changes.\\n- [6] AdvGLUE (NeurIPS D&B 2021) \u2014 GLUE tasks under multiple adversarial attack algorithms; [7] ANLI (ACL 2020) \u2014 model-in-the-loop adversarial NLI; [8] Dynabench \u2014 dynamic adversarial data collection framework.\\n- [9] Robustness Gym (NAACL 2021) \u2014 unifies four paradigms: subpopulations, transformations, evaluation sets, adversarial attacks.\\n- [10] HELM \u2014 includes robustness scenarios (typos, dialect); [11] PromptBench (2023) \u2014 LLM prompt robustness at character, word, sentence, and semantic levels.\\n\\n## 2. Noise Injection Methods\\n\\n- **Character-level:** keyboard-adjacent substitutions (typos) [1][24]; adjacent character swaps/transpositions [1][23]; random insertions/deletions [24]; OCR-style confusions (l/1, O/0); homoglyphs and phonetic substitutions; HotFlip gradient-guided character flips [22]; DeepWordBug's importance-scored character edits [23]. Pruthi et al. [24] showed 1\u20132 character edits suffice to break classifiers.\\n- **Casing:** lowercasing, ALL-CAPS, mixed-case injection \u2014 tests truecasing/tokenizer assumptions; implemented in AugLy [19] and textflint [14].\\n- **Whitespace/segmentation:** word concatenation (\\\"howareyou\\\"), letter splitting (\\\"w o r d s\\\"), double spaces \u2014 breaks subword tokenization. Related training-time noise: subword regularization [25] and BPE-dropout [26] improve segmentation robustness.\\n- **Word/sentence-level:** synonym substitution (TextFooler [20]), contextual BERT-based substitution (BAE [21]), EDA's random insert/swap/delete [18], counterfactual generation (Polyjuice [17]), instruction-based augmentation (NLAugmenter [16]).\\n- **Natural noise sources:** Wikipedia edit histories [1], Reddit (MTNT) [2], Twitter/social media [4].\\n\\n## 3. Tools\\n\\n- **nlpaug** [12] \u2014 character augmenters (KeyboardAug, OcrAug, RandomCharAug with insert/delete/swap/substitute), word augmenters (WordNet synonyms, BERT/GPT contextual, spelling, word splitting), back-translation; also covers audio.\\n- **TextAttack** [13] \u2014 attack framework with four pluggable modules (goal function, constraints, transformation, search); ~16 attack recipes (TextFooler, DeepWordBug, BAE, HotFlip); transformations include QWERTY swaps, homoglyphs, character insertion/deletion; constraints via USE/BERTScore similarity, POS consistency; includes augmentation CLI and adversarial training.\\n- **textflint** [14] \u2014 unified robustness toolkit: 58 transformations + 20 subpopulations across 12 tasks (sentiment, NLI, STS, NER, POS, AMR-to-text, semantic parsing, math word problems); English and Chinese; generates test data + reports.\\n- **Others:** OpenAttack [15] (bilingual EN/ZH attack toolkit with perplexity/grammar/edit-distance metrics), NLAugmenter [16], AugLy [19] (casing, homoglyphs, unicode confusables, typo simulation), Smoke Tester [27] (unified re-implementation for comparing attack effectiveness).\\n\\n## 4. Evaluation Metrics\\n\\n- **Performance delta:** \u0394accuracy/F1 (NLU) or \u0394BLEU/\u0394chrF/\u0394COMET (MT) between clean and perturbed sets, reported per noise type [1][2][4].\\n- **Behavioral pass rates:** MFT/INV/DIR percentages [5].\\n- **Attack metrics:** attack success rate, perturbation/modification rate (% tokens changed), queries per success, post-attack accuracy [13][15][27].\\n- **Noise quality controls:** semantic similarity (USE, BERTScore), grammaticality, GPT-2 perplexity of generated text \u2014 critical since many attacks produce ungrammatical output, inflating apparent vulnerability [27].\\n- **MT hygiene:** standardized sacreBLEU [28], chrF, TER, COMET for neural adequacy [29]; human adequacy/fluency judgment in robustness tasks [4].\\n- **Statistics:** paired bootstrap / approximate randomization for deltas; fixed seeds for stochastic noise.\\n\\n## 5. Key Caveats\\n\\n- Robustness to synthetic noise does **not** transfer to natural noise [1] \u2014 validate with real noisy data [2][4].\\n- Tokenizer artifacts can dominate casing/whitespace results; test both case-sensitive/insensitive configurations [25][26].\\n- For LLMs, benchmark contamination is a risk; PromptBench/HELM offer current starting points [10][11].\\n\\n**Practical recipe:** freeze a clean eval set \u2192 generate seeded perturbed copies (multiple noise types/severities) via nlpaug/textflint \u2192 report clean-vs-noisy deltas + invariance rates \u2192 for adversarial attacks, report ASR with similarity/grammar constraints \u2192 significance-test all deltas.\\n\\n---\\n\\n## References\\n\\n[1] Belinkov & Bisk, \\\"Synthetic and Natural Noise Both Break Neural Machine Translation,\\\" ICLR 2018\\n[2] Michel & Neubig, \\\"MTNT: A Testbed for Machine Translation of Noisy Text,\\\" EMNLP 2018\\n[3] Khayrallah & Koehn, \\\"On the Impact of Various Types of Noise on Neural Machine Translation,\\\" WMT 2018\\n[4] Li et al., \\\"Findings of the WMT 2019 Shared Task on Machine Translation Robustness,\\\" WMT 2019\\n[5] Ribeiro et al., \\\"Beyond Accuracy: Behavioral Testing of NLP Models with CheckList,\\\" ACL 2020\\n[6] Wang et al., \\\"Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of NLP Models,\\\" NeurIPS D&B 2021\\n[7] Nie et al., \\\"Adversarial NLI: A New Benchmark for Natural Language Understanding,\\\" ACL 2020\\n[8] Kiela et al., \\\"Dynabench: Rethinking Benchmarking in NLP,\\\" NAACL 2021\\n[9] Goel et al., \\\"Robustness Gym: Unifying the NLP Evaluation Landscape,\\\" NAACL 2021\\n[10] Liang et al., \\\"Holistic Evaluation of Language Models,\\\" 2022\\n[11] Zhu et al., \\\"PromptBench: Towards Evaluating the Robustness of LLMs on Adversarial Prompts,\\\" 2023\\n[12] Ma, \\\"nlpaug: A Python Library for Textual and Audio Data Augmentation,\\\" GitHub 2019\\n[13] Morris et al., \\\"TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP,\\\" ACL 2020 (demo)\\n[14] Wang et al., \\\"TextFlint: Unified Multilingual Robustness Evaluation Toolkit for NLP,\\\" 2021\\n[15] Zeng et al., \\\"OpenAttack: An Open-source Text Adversarial Attack Toolkit,\\\" ACL 2021 (demo)\\n[16] Dhole et al., \\\"NLAugmenter: Composition-Based Contextual Augmentation for NLP,\\\" ACL 2021 (demo)\\n[17] Wu et al., \\\"Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models,\\\" ACL 2021\\n[18] Wei & Zou, \\\"EDA: Easy Data Augmentation Techniques for Text Classification,\\\" EMNLP 2019\\n[19] Bitton et al., \\\"AugLy: A Data Augmentation Library for Text, Images, and Audio,\\\" 2021\\n[20] Jin et al., \\\"Is BERT Really Robust? (TextFooler),\\\" AAAI 2020\\n[21] Garg & Ramakrishnan, \\\"BAE: BERT-based Adversarial Examples for Text Classification,\\\" EMNLP 2020\\n[22] Ebrahimi et al., \\\"HotFlip: White-Box Adversarial Examples for Text Classification,\\\" ACL 2018\\n[23] Gao et al., \\\"DeepWordBug: Black-Box Generation of Adversarial Text Sequences,\\\" 2019\\n[24] Pruthi et al., \\\"Combating Adversarial Misspellings with Robust Word Recognition,\\\" ACL 2019\\n[25] Kudo, \\\"Subword Regularization: Improving NMT Models with Multiple Subword Segmentation,\\\" ACL 2018\\n[26] Provilkov et al., \\\"BPE-Dropout: Simple and Effective Subword Regularization,\\\" ACL 2020\\n[27] Gr\u00f6ndahl & Asokan, \\\"Smoke Tester: A Framework for Evaluating Adversarial Text Attack Effectiveness,\\\" 2020\\n[28] Post, \\\"A Call for Clarity in Reporting BLEU Scores,\\\" WMT 2018\\n[29] Rei et al., \\\"COMET: A Neural Framework for MT Evaluation,\\\" EMNLP 2020\\n[30] Zhang et al., \\\"Adversarial Attacks on Deep Learning Models in NLP: A Survey,\\\" ACM TIST 2020\\n\\n---\\n\\n## Code-switching stress tests for multilingual NLP: Hinglish/Spanglish/Arabic-French datasets and benchmarks, transliteration code-switching, MT and LLM performance on mixed-language text, evaluation approaches\\nI'll research this topic systematically, starting with broad searches on code-switching benchmarks and then drilling into specific language pairs, transliteration, MT/LLM performance, and evaluation methods.\",\n            \"gen_ai.tool.call.id\": \"call_-7227516746041655036\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"36cfbff8d43cf658\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"model\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381966577182488,\n          \"end_time_unix_nano\": 1790381989890239945,\n          \"duration_nano\": 23313057457,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"48510d3a0ee4b40f\",\n          \"parent_span_id\": \"36cfbff8d43cf658\",\n          \"name\": \"ChatOpenAI\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381966581562560,\n          \"end_time_unix_nano\": 1790381989885899230,\n          \"duration_nano\": 23304336670,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"chat\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"68a9d75b06676f50\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"tools\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381989892618472,\n          \"end_time_unix_nano\": 1790383014983749189,\n          \"duration_nano\": 1025091130717,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"f4b6cdfa7867ceb1\",\n          \"parent_span_id\": \"68a9d75b06676f50\",\n          \"name\": \"research_subtopic\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790381989897781190,\n          \"end_time_unix_nano\": 1790383014972285178,\n          \"duration_nano\": 1025074503988,\n          \"tool_content_status\": {\n            \"arguments\": \"present\",\n            \"result\": \"present\"\n          },\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"execute_tool\",\n            \"gen_ai.tool.name\": \"research_subtopic\",\n            \"gen_ai.tool.type\": \"function\",\n            \"gen_ai.tool.call.arguments\": {\n              \"subtopics\": [\n                \"UTS #39 Unicode security mechanisms: confusables.txt, restricted/recommended scripts per identifier profile, mixed-script detection algorithm, Greek/Cyrillic homoglyph spoofing examples, whole-script confusables, remapping vs blocking policy\",\n                \"CJK fullwidth/halfwidth width folding (U+FF01-FF5E), NFKC width folding in Japanese/Chinese localization, IDN homograph attacks and defenses (UTS #46 transitional vs nontransitional processing, browser protections)\",\n                \"Code-switching NLP benchmarks and datasets: LinCE, GLUE-CoS, Hinglish TOP, Spanglish/SENTIMENT-CS, Arabic dialect code-switching; transliterated code-switching (Romanized Hindi/Bengali); MT and LLM performance on code-switched text and evaluation metrics\"\n              ]\n            },\n            \"gen_ai.tool.call.result\": \"## UTS #39 Unicode security mechanisms: confusables.txt, restricted/recommended scripts per identifier profile, mixed-script detection algorithm, Greek/Cyrillic homoglyph spoofing examples, whole-script confusables, remapping vs blocking policy\\nResearch on 'UTS #39 Unicode security mechanisms: confusables.txt, restricted/recommended scripts per identifier profile, mixed-script detection algorithm, Greek/Cyrillic homoglyph spoofing examples, whole-script confusables, remapping vs blocking policy' failed: Connection error.\\n\\n---\\n\\n## CJK fullwidth/halfwidth width folding (U+FF01-FF5E), NFKC width folding in Japanese/Chinese localization, IDN homograph attacks and defenses (UTS #46 transitional vs nontransitional processing, browser protections)\\n# CJK Width Folding, NFKC Localization Issues, and IDN Homograph Defenses\\n\\n## 1. Halfwidth/Fullwidth Forms (U+FF00\u2013FFEF) and width folding\\n\\n- The block **U+FF01\u2013FF5E** contains **fullwidth variants of ASCII U+0021\u2013007E**, each carrying a `` compatibility decomposition (e.g., \uff21 U+FF21 \u2192 A U+0041, \uff0cU+FF0C \u2192 , U+002C) [1]. The rest of the block: **U+FF61\u2013FF9F** halfwidth ideographic punctuation and katakana (\uff61\uff62\uff63\uff8a\uff9e, halfwidth dakuten \uff9e U+FF9E), **U+FFA0\u2013FFDC** halfwidth Hangul compatibility jamo, **U+FFE0\u2013FFE6** fullwidth signs (\uffe0\uffe1\uffe6 etc.) [1].\\n- These exist because legacy East Asian encodings (Shift_JIS/JIS X 0208, GB2312/GBK, KS X 1001) encoded both widths; Unicode round-trip compatibility requires both [1][2].\\n- **East Asian Width property (UAX #11)**: fullwidth forms are class F, halfwidth kana/Hangul class H, ASCII class Na, with many CJK-context characters \\\"Ambiguous\\\" (A) \u2014 rendered 1 or 2 cells depending on locale [2].\\n- **Width folding** = mapping fullwidth\u2192ASCII/halfwidth via **NFKC/NFKD** (UAX #15), **NFKC_Casefold** for caseless matching, or ICU's dedicated `Halfwidth`/`Fullwidth` transforms [3]. Note NFKC folds *both directions per legacy mapping*: fullwidth Latin\u2192ASCII, but halfwidth katakana \uff8a\u2192\u30cf (fullwidth), and \uff9e\u2192combining U+3099 [1][3].\\n\\n## 2. NFKC folding in Japanese/Chinese localization\\n\\n- **What folds**: \uff21\uff22\uff23\u2192ABC, \uff11\uff12\uff13\u2192123, \uff0c\uff1f\uff01\u2192,?! (U+FF01\u2013FF5E), fullwidth space U+3000\u2192U+0020, halfwidth kana\u2192fullwidth kana [1][3].\\n- **What does not fold**: ideographic punctuation \u3001U+3001 and \u3002U+3002 (no compatibility decompositions) \u2014 so NFKC is *not* a pure width fold for CJK punctuation [1][3].\\n- **Over-folding pitfalls**: NFKC also destroys meaningful distinctions \u2014 \u2460\u21921, \u3231\u2192(\u682a), \u33a1\u2192m2, Roman numerals \u2163\u2192IV, halfwidth\u2192fullwidth kana changes string length and visual style [3]. Japanese/Chinese search and matching systems therefore typically use *custom* width-only folding (ASCII width + kana width) rather than raw NFKC; IMEs default to fullwidth output, making \u534a\u89d2/\u5168\u89d2 normalization a core JA/ZH localization requirement [3][14].\\n- Identifiers: UAX #31 mandates NFKC normalization for identifiers (adopted by Python via PEP 3131), which folds fullwidth homoglyphs in code [14].\\n\\n## 3. IDN homograph attacks\\n\\n- IDNs encode non-ASCII labels as Punycode (`xn--\u2026`, RFC 3492) under IDNA2003 (RFC 3490/3491, Nameprep with NFKC) or IDNA2008 (RFC 5890\u20135894) [6][7].\\n- **Attack**: register visually confusable domains using cross-script lookalikes (Cyrillic \u0430/\u0435/\u043e, Greek \u03bf) or width/compatibility variants. Formalized by Gabrilovich & Gontmakher (CACM 2002) [11]; the 2005 \\\"Shmoo Group\\\" paypal.com demo forced early mitigations; in 2017 Xudong Zheng's Cyrillic **\u0430\u0440\u0440\u04cf\u0435.com** (`xn--80ak6aa92e`) rendered as \\\"apple.com\\\" in unpatched browsers [12]. Related: typosquatting/combosquatting and homoglyph toolkits (dnstwist, EvilURL) [12][16].\\n- **Key structural defense**: IDNA2008 makes all compatibility characters (including U+FF01\u2013FF5E) **DISALLOWED** in registered labels, and UTS #46 maps fullwidth input to ASCII \u2014 so \uff50\uff41\uff59\uff50\uff41\uff4c.com cannot exist as a distinct .com domain [5][7].\\n\\n## 4. UTS #46: transitional vs nontransitional\\n\\n- **UTS #46** bridges IDNA2003\u2194IDNA2008 with a mapping table (statuses: `valid`, `mapped`, `ignored`, `disallowed_STD3_valid/mapped`, `disallowed`; `UseSTD3ASCIIRules` enforces LDH-only ASCII) [5].\\n- **Four \\\"deviation\\\" characters** differ between modes: **\u00df (U+00DF), \u03c2 (U+03C2), ZWNJ (U+200C), ZWJ (U+200D)** [5]:\\n  - *Transitional* (IDNA2003-compatible): maps \u00df\u2192ss, \u03c2\u2192\u03c3, deletes ZWJ/ZWNJ \u2014 makes fa\u00df.de \u2261 fass.de (confusable).\\n  - *Nontransitional* (IDNA2008-compatible): keeps \u00df and \u03c2 valid (distinct domains), ZWJ/ZWNJ valid under ContextJ rules.\\n- **Fullwidth folding happens in both modes** (`mapped` status); only the deviation characters differ [5].\\n- **Browser adoption of nontransitional**: Chrome ~2014, Safari ~2016\u201317, **Firefox 59 (March 2018)** \u2014 the last major engine to switch; IE/legacy Edge never did. This is why \u00df-domains (e.g., fa\u00df.de) now display distinctly [5][10].\\n\\n## 5. Defenses in practice\\n\\n- **Browsers** apply UTS #46 for parsing *plus* spoof checks, showing Punycode on failure [9][10]:\\n  - **Chrome**: Unicode display only for single-script labels (with limited exceptions, e.g., Han+Hiragana+Katakana, Han+Latin), a blocked-character list, and UTS #39 confusable checks (including against popular TLDs/domains) [9].\\n  - **Firefox**: mixed-script confusable detection per UTS #39, script-mixing rules, plus user pref `network.IDN_show_punycode` [10].\\n  - All major browsers display `xn--80ak6aa92e` punycode for the 2017 apple.com lookalike [12].\\n- **UTS #39** supplies the `skeleton()` confusability algorithm and RestrictionLevel; **UTS #36** catalogs the threat model [4][16].\\n- **ICANN/registries**: IDN Guidelines require published IDN tables (per-script code point sets), variant/confusable reservations, and the Maximal Starting Repertoire; registries bundle variants (simplified/traditional Chinese, kanji variants in .jp, \u00df\u2194ss bundling at DENIC) [8][13].\\n- **Ecosystem**: password-manager auto-fill keyed to exact ASCII domains is a strong user-side mitigation; SSAC advisories (e.g., SAC 045 on homoglyphs) guide registry policy [13].\\n\\n## Sources\\n\\n[1] Unicode Character Database / The Unicode Standard, ch. 22 (compatibility decompositions, U+FF00\u2013FFEF)\\n[2] UAX #11, *East Asian Width Property*\\n[3] UAX #15, *Unicode Normalization Forms* (NFKC, NFKC_Casefold); ICU User Guide (Halfwidth/Fullwidth transforms)\\n[4] UTS #39, *Unicode Security Mechanisms*\\n[5] UTS #46, *Unicode IDNA Compatibility Processing* (IdnaMappingTable.txt, deviation characters)\\n[6] RFC 3490/3491 (IDNA2003, Nameprep); RFC 3492 (Punycode)\\n[7] RFC 5890\u20135894 (IDNA2008); RFC 5895 (mapping guidance)\\n[8] ICANN, *IDN Guidelines* (current version) and Maximal Starting Repertoire\\n[9] Chromium, *IDN Guidance* (chromium.org design doc)\\n[10] Mozilla nsIDNService implementation; Firefox 59 release notes (nontransitional switch); `network.IDN_show_punycode` pref\\n[11] Gabrilovich & Gontmakher, \\\"The Homograph Attack,\\\" *CACM* 45(2), 2002\\n[12] X. Zheng, \\\"Phishing with Unicode Domains\\\" (2017); 2005 Shmoo Group IDN demo\\n[13] ICANN SSAC, SAC 045, *Advisory on IDN Homoglyphs*\\n[14] PEP 3131 / PEP 672 (Python identifier NFKC normalization; Unicode security considerations)\\n[15] UAX #31, *Identifier and Pattern Syntax*\\n[16] UTS #36, *Unicode Security Considerations*\\n\\n*Note: browser-switch dates for Chrome/Safari are approximate; the Firefox 59 (2018) date is firm. No live retrieval was performed \u2014 citations are to canonical standards and primary documents.*\\n\\n---\\n\\n## Code-switching NLP benchmarks and datasets: LinCE, GLUE-CoS, Hinglish TOP, Spanglish/SENTIMENT-CS, Arabic dialect code-switching; transliterated code-switching (Romanized Hindi/Bengali); MT and LLM performance on code-switched text and evaluation metrics\\n# Code-Switching (CS) NLP: Benchmarks, Datasets, Models, and Metrics\\n\\n## 1. Broad multi-task benchmarks\\n\\n**LinCE** [1] (ICLR 2020) \u2014 the most widely used CS benchmark. Four language pairs (Spanish\u2013English, Hindi\u2013English, Mandarin\u2013English, Nepali\u2013English) plus Singlish, across five tasks: token-level language identification (LID), POS tagging, NER, sentiment analysis, and semantic parsing (Spanglish SCoRe data). It aggregates earlier shared-task corpora (e.g., Spanglish 2014 [4], SentiMix [5], SENTIMENT-CS [6]) plus new annotations. Key result: multilingual encoders (mBERT, XLM-R) fine-tuned on monolingual data show large drops on CS text vs. monolingual test sets, worsening as mixing density increases.\\n\\n**GLUE-CoS** [2] (2020) \u2014 GLUE-style CS benchmark with four tasks: natural language inference, sentiment analysis, semantic relatedness, and language identification, over pairs mixing English with Spanish, Hindi, Chinese, etc. Combines real CS data with artificially code-switched English (word-level substitution via translations). Code-switched pretraining (CSP [17]) substantially improves over mBERT/XLM-R baselines.\\n\\n## 2. Task-specific datasets\\n\\n- **Spanglish corpus (2014 shared task)** [4]: ~2.7k tweets with token-level LID, POS, and NER annotations (Spanish\u2013English); the seed of most later Spanglish resources.\\n- **SENTIMENT-CS** [6] and **SemEval-2020 Task 9 (SentiMix)** [5]: sentiment classification of code-mixed tweets for Hindi\u2013English (SENTIMENT-CS) and Spanish\u2013English (SentiMix); thousands of tweets each; F1-based evaluation. These supply LinCE's sentiment subsets.\\n- **Hinglish TOP** [3] (EMNLP 2021): semantic parsing (intent + slot filling) in Hinglish, built by re-annotating the English TOP query-understanding dataset into natural code-mixed Hindi\u2013English. Multilingual models lag English TOP performance by a wide margin; the first large CS semantic-parsing resource.\\n- **CMU Hinglish DoG**: code-switched Hinglish dialogues derived from the CMU-DoG corpus, used in CS dialogue generation work.\\n\\n## 3. Arabic dialect code-switching\\n\\n- Arabic CS is dominated by (a) diglossic mixing between MSA and dialects (esp. Egyptian) and (b) Arabic\u2013English/French mixing in social media, frequently written in **Arabizi** (Romanized Arabic) [12].\\n- The main token-level resource is the **MSA\u2013Egyptian** pair from the CoSWo 2016 shared task [7], built on CALLHOME Egyptian Arabic telephone transcripts [9].\\n- Dedicated Arabic CS benchmarks remain scarce and fragmented; most dialectal evaluation uses adjacent dialect-ID resources \u2014 MADAR (26 city dialects) [10] and NADI [11] \u2014 rather than true code-switching. Maghrebi (Darija\u2013French/English) mixing is especially under-resourced, and no standard Arabic CS benchmark with LinCE-level adoption exists.\\n\\n## 4. Transliterated (Romanized) code-switching\\n\\n- Roman-script CS (Romanized Hindi/Bengali + English) dominates informal writing but is under-served: most benchmarks are in native or mixed script, creating domain mismatch.\\n- **PHINC** [13] (2020): parallel Romanized-Hinglish \u2194 English Instagram-chat corpus (tens of thousands of pairs) for MT and transliteration; MT baselines on it remain modest.\\n- **HingCorpus / HingBERT** [14] (LREC 2022): large Romanized-Hinglish social-media corpus and BERT-family models (HingBERT, HingMBERT, HingRoBERTa) that beat mBERT on Hinglish tasks.\\n- **Bengali\u2013English (Banglish)**: token-level LID data from the CALCS 2018 shared task [8]; several Banglish sentiment datasets exist, but MT and structured-task data are very scarce.\\n- **Dakshina** [15] (2020): romanized \u2194 native-script parallel text for 12 South Asian languages \u2014 not CS per se, but the standard resource for Romanized Indic text.\\n- **RomanSetu** [16] (2024): shows multilingual LLMs underperform on Romanized Indic text (tokenizer fragmentation, script mismatch) and proposes romanization-aware adaptation that recovers much of the gap.\\n\\n## 5. MT and LLM performance on CS text\\n\\nReplicated findings:\\n- Pre-LLM systems (SMT, NMT, zero-shot mBART/NLLB) degrade substantially on CS input; errors cluster at switch points, and LID errors cascade into translation/parsing errors [1][2][17][18].\\n- Code-switched pretraining (CSP [17]) and fine-tuning on CS parallel data (e.g., PHINC for Hinglish\u2192English [13]) yield large gains; transliterating Romanized input to native script before processing also helps [16].\\n- Instruction-tuned LLMs are stronger on high-resource pairs (Spanish\u2013English) and sentence-level tasks, but (a) still trail monolingual performance, (b) tend to \\\"normalize\\\" CS input into monolingual output, (c) degrade sharply on Romanized Hinglish/Banglish, and (d) remain behind fine-tuned encoders on token-level tasks (LID/POS/NER) [1][2][16][18].\\n- Prompting mitigations reported: few-shot exemplars, transliteration of input, explicit instructions to preserve switching.\\n\\n## 6. Evaluation metrics\\n\\n- **Task metrics**: token-level accuracy/F1 (LID, POS, NER); F1 for sentiment (SentiMix [5]); exact-match accuracy for semantic parsing (Hinglish TOP [3]); BLEU/chrF/COMET for MT.\\n- **CS-structural metrics** (used to characterize data and bin error analysis): Code-Mixing Index (CMI) [19], I-index [20], language entropy [22], switch-point density [21], burstiness. A robust finding: performance falls monotonically as mixing density rises.\\n- **Gaps**: no standard CS-specific adequacy metric for MT (BLEU/COMET reliability on CS output is under-studied); benchmarks are small and static (overfitting risk); Romanized variants are rarely included in test sets.\\n\\n## References\\n\\n[1] Winata et al. (2020), \\\"LinCE: A Linguistic Code-switching Evaluation Benchmark for NLP,\\\" ICLR 2020. [2] Winata et al. (2020), \\\"GLUE-CoS: GLUE Benchmark for Code-Switching,\\\" arXiv:2010.12609. [3] \\\"Hinglish TOP: A Dataset for Hinglish Semantic Parsing,\\\" EMNLP 2021. [4] Solorio et al. (2014), \\\"Overview for the First Shared Task on Language Identification in Code-Switched Data,\\\" EMNLP 2014 CS Workshop. [5] Patwa et al. (2020), \\\"SemEval-2020 Task 9: Sentiment Analysis of Code-Mixed Tweets.\\\" [6] Patwa et al. (2019), SENTIMENT-CS corpus (WILDRE, EMNLP 2019). [7] Molina et al. (2016), CoSWo@COLING 2016 shared task overview. [8] CALCS 2018 shared task overview, CALCS@ACL 2018. [9] CALLHOME Egyptian Arabic, LDC97S45. [10] Bouamor et al. (2019), MADAR, WANLP 2019. [11] Abdul-Mageed et al. (2020), NADI shared task. [12] Darwish & Mubarak (2016), Arabizi detection/normalization. [13] Mukherjee et al. (2020), \\\"PHINC: A Parallel Hinglish-English Instagram Chat Corpus.\\\" [14] Nayak & Joshi (2022), \\\"HingBERT: A Code-Mixed Language Model for Hinglish,\\\" LREC 2022. [15] Roark et al. (2020), \\\"Processing the Dakshina Dataset.\\\" [16] \\\"RomanSetu: Efficiently Adapting Multilingual Models via Romanization,\\\" 2024. [17] Yin et al. (2020), CSP code-switched pretraining, EMNLP 2020. [18] Winata et al. (2021), \\\"Are Multilingual Models Effective in Code-Switching?\\\" LREC 2021. [19] Gamb\u00e4ck & Das (2013), Code-Mixing Index. [20] Guzm\u00e1n et al. (2017), I-index. [21] Solorio & Liu (2008), \\\"Learning to Predict Code-Switching Points,\\\" EMNLP 2008. [22] Poplack (1980), language entropy.\\n\\n*Caveat: venue/size details reflect the canonical papers as I recall them; verify exact numbers against the cited sources before citing onward.*\",\n            \"gen_ai.tool.call.id\": \"call_-7227534681825083037\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"b2dc7325785162f1\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"model\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790383014985895629,\n          \"end_time_unix_nano\": 1790383023992275837,\n          \"duration_nano\": 9006380208,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"835c1180583e32c8\",\n          \"parent_span_id\": \"b2dc7325785162f1\",\n          \"name\": \"ChatOpenAI\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790383014990507222,\n          \"end_time_unix_nano\": 1790383023990161951,\n          \"duration_nano\": 8999654729,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"chat\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"c97e13bcbdf1b7a9\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"tools\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790383023993437987,\n          \"end_time_unix_nano\": 1790384047613700483,\n          \"duration_nano\": 1023620262496,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"4b85e9c6d3a4119e\",\n          \"parent_span_id\": \"c97e13bcbdf1b7a9\",\n          \"name\": \"research_subtopic\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790383023995166799,\n          \"end_time_unix_nano\": 1790384047610076476,\n          \"duration_nano\": 1023614909677,\n          \"tool_content_status\": {\n            \"arguments\": \"present\",\n            \"result\": \"present\"\n          },\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"execute_tool\",\n            \"gen_ai.tool.name\": \"research_subtopic\",\n            \"gen_ai.tool.type\": \"function\",\n            \"gen_ai.tool.call.arguments\": {\n              \"subtopics\": [\n                \"UTS #39 Unicode Security Mechanisms: skeleton() algorithm, confusables.txt and intentional.txt data files, RestrictionLevel (Highly Restricted, Moderately Restricted, Minimally Restricted, Unrestricted), mixed-script confusable detection, whole-script confusables, identifier profile recommendations, remapping vs blocking policy\"\n              ]\n            },\n            \"gen_ai.tool.call.result\": \"## UTS #39 Unicode Security Mechanisms: skeleton() algorithm, confusables.txt and intentional.txt data files, RestrictionLevel (Highly Restricted, Moderately Restricted, Minimally Restricted, Unrestricted), mixed-script confusable detection, whole-script confusables, identifier profile recommendations, remapping vs blocking policy\\nResearch on 'UTS #39 Unicode Security Mechanisms: skeleton() algorithm, confusables.txt and intentional.txt data files, RestrictionLevel (Highly Restricted, Moderately Restricted, Minimally Restricted, Unrestricted), mixed-script confusable detection, whole-script confusables, identifier profile recommendations, remapping vs blocking policy' failed: Connection error.\",\n            \"gen_ai.tool.call.id\": \"call_-7227494206053278546\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"12ebe61f5b81201f\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"model\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790384047615728778,\n          \"end_time_unix_nano\": 1790384060944662617,\n          \"duration_nano\": 13328933839,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"266b7cc8147ac1ce\",\n          \"parent_span_id\": \"12ebe61f5b81201f\",\n          \"name\": \"ChatOpenAI\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790384047619694464,\n          \"end_time_unix_nano\": 1790384060942839766,\n          \"duration_nano\": 13323145302,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"chat\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"52d1d0388b1ba26a\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"tools\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790384060945719638,\n          \"end_time_unix_nano\": 1790384374794338830,\n          \"duration_nano\": 313848619192,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"6de7cdf05cbe586a\",\n          \"parent_span_id\": \"52d1d0388b1ba26a\",\n          \"name\": \"research_subtopic\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790384060947784114,\n          \"end_time_unix_nano\": 1790384374780670890,\n          \"duration_nano\": 313832886776,\n          \"tool_content_status\": {\n            \"arguments\": \"present\",\n            \"result\": \"present\"\n          },\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"execute_tool\",\n            \"gen_ai.tool.name\": \"research_subtopic\",\n            \"gen_ai.tool.type\": \"function\",\n            \"gen_ai.tool.call.arguments\": {\n              \"subtopics\": [\n                \"Unicode Technical Standard 39 security: how skeleton algorithm and confusables data work, identifier restriction levels, detecting mixed-script spoofing with Greek Cyrillic homoglyphs, best practices for handling confusable characters in user identifiers\",\n                \"Detecting and mitigating invisible character and bidi spoofing attacks in text pipelines: Trojan Source attack, RLO/LRO overrides, UTS #55 source code handling, compiler warnings, GitHub invisible character warnings, OpenAI/LLM prompt injection via hidden Unicode characters\"\n              ]\n            },\n            \"gen_ai.tool.call.result\": \"## Unicode Technical Standard 39 security: how skeleton algorithm and confusables data work, identifier restriction levels, detecting mixed-script spoofing with Greek Cyrillic homoglyphs, best practices for handling confusable characters in user identifiers\\n# UTS #39 \u2014 Unicode Security Mechanisms: Research Summary\\n\\nUTS #39 defines mechanisms for detecting visually confusable strings (\\\"spoofing\\\") and restricting identifier content. It is updated with each annual Unicode release (most recently tracking Unicode 17.0, Sept 2025 \u2014 verify the current version). Note: I'm answering from canonical source knowledge; confirm details against the live spec.\\n\\n## 1. The Skeleton Algorithm and Confusables Data [1][2]\\n\\n**Skeleton computation** (UTS #39 \u00a74). The skeleton of a string X is:\\n\\n1. Convert X to **NFD** (canonical decomposition) [9]\\n2. Map each character through the **confusables mapping** (confusables.txt)\\n3. Apply **CaseFolding** (full case folding)\\n4. Convert to **NFD** again\\n\\nSo: `skeleton(X) = NFD(casefold(map(NFD(X))))`. Two identifiers are *confusable* if and only if their skeletons are identical.\\n\\n**confusables.txt** [2] format:\\n```\\n0441 ;  0063 ;  MA   # ( \u0441 \u2192 c ) CYRILLIC SMALL LETTER ES \u2192 LATIN SMALL LETTER C\\n```\\nMapping types:\\n- **MA** \u2014 mixed-script confusable; **SA** \u2014 single-script confusable (apply regardless of case)\\n- **SL** / **ML** \u2014 single/mixed-script confusables that apply only after lowercasing (relevant for case-insensitive matching; ICU's `ANY_CASE` option controls this) [7]\\n\\n**intentional.txt** [3] lists characters whose confusable mapping is deliberate (fullwidth forms, mathematical alphanumerics, compatibility characters). Data is glyph-shape based, reviewed across fonts, and revised annually \u2014 it is deliberately conservative and does **not** cover everything (e.g., `rn` \u2248 `m`, font-specific shapes).\\n\\n## 2. Identifier Restriction Levels [1][4][5]\\n\\nImplemented via `Identifier_Status` (Allowed/Restricted) and `Identifier_Type` (Inclusion, Recommended, Technical, Exclusion, Not_XID, etc.) in DerivedCoreProperties.txt [4], plus the Recommended-script list in UAX #31 [5]:\\n\\n| Level | Rule |\\n|---|---|\\n| **Inclusion** | Only the small curated \\\"Inclusion\\\" set (essentially ASCII plus a few safe characters, e.g., middle dot) \u2014 highest security |\\n| **Highly Restrictive** | Recommended scripts, but multiple scripts only within a *script group* ({Han, Bopomofo}, {Han, Hiragana, Katakana}, {Hangul}); each other script alone |\\n| **Moderately Restrictive** | Latin may mix with other Recommended scripts **except Cyrillic, Greek, and Cherokee** (Cherokee's syllabary borrows Latin glyph shapes) |\\n| **Minimally Restrictive** | Arbitrary mixing of Recommended scripts (e.g., Latin + Cyrillic allowed) |\\n| **Unrestricted** | Any characters |\\n\\nScript mixing is evaluated using *augmented script sets* (ScriptExtensions.txt) [6] so Common characters (digits, punctuation) don't falsely trigger \\\"mixed-script.\\\"\\n\\n## 3. Detecting Mixed-Script Spoofing (Greek/Cyrillic Homoglyphs) [1][7]\\n\\n**Key homoglyph sets** (all mapped in confusables.txt):\\n- Cyrillic \u2192 Latin: \u0430\u2192a, \u0441\u2192c, \u0435\u2192e, \u043e\u2192o, \u0440\u2192p, \u0445\u2192x, \u0443\u2192y, \u0456\u2192i, \u0455\u2192s, \u0458\u2192j, \u043a\u2192k, \u043c\u2192m, \u0442\u2192t\\n- Greek \u2192 Latin: \u03bf\u2192o, \u03b1\u2192a, \u03b5\u2192e, \u03b9\u2192i, \u03ba\u2192k, \u03bd\u2192v, \u03c1\u2192p, \u03c4\u2192t, \u03c5\u2192u, \u03c7\u2192x\\n- Cyrillic \u2194 Greek directly: \u0456 (U+0456) \u2248 \u03b9 (U+03B9), etc.\\n\\n**Detection procedure:**\\n1. Normalize (NFKC_Casefold per UAX #31 identifier profile) [5][9]\\n2. Resolve per-character script sets; if their intersection is empty \u2192 **mixed-script identifier**\\n3. Compute the skeleton\\n4. Classify:\\n   - **Mixed-script confusable**: identifier is mixed-script but its skeleton is single-script (e.g., `\u0430dmin` with Cyrillic \u0430 + Latin dmin \u2192 skeleton `admin` is pure Latin)\\n   - **Whole-script confusable**: identifier is single-script but its skeleton is single-script in a *different* script (e.g., all-Cyrillic `\u0441\u043e\u0440\u0435` \u2248 Latin `cope`)\\n5. Pairwise: `skeleton(candidate) == skeleton(existing)` \u2192 block or verify\\n\\nICU's SpoofChecker exposes these as `MIXED_SCRIPT_CONFUSABLE`, `WHOLE_SCRIPT_CONFUSABLE`, `SINGLE_SCRIPT_CONFUSABLE`, plus `INVISIBLE` (zero-width/combining-mark tricks), `RESTRICTION_LEVEL`, and `CHAR_LIMIT` checks; `getSkeleton()` and `areConfusable()` are the core APIs [7]. ICU requires normalized input.\\n\\n**Worked example:** candidate `\u03c1aypal` (Greek rho U+03C1). Scripts = {Greek, Latin} \u2192 mixed-script; skeleton = `paypal` \u2192 single-script \u2192 mixed-script confusable \u2192 reject, or flag because skeleton matches the existing account.\\n\\n## 4. Best Practices for User Identifiers [1][5][7][8]\\n\\n1. **Normalize canonically**: store and compare NFKC_Casefold forms; use full CaseFolding, not ASCII lowercase.\\n2. **Enforce a restriction level** at creation: Highly Restrictive is a good default for global user-facing services; Inclusion/ASCII for high-assurance or monolingual services; Moderate for programming-language identifiers.\\n3. **Skeleton-index check at registration**: persist `skeleton(username)` in an indexed column; on signup, compare candidate skeleton against all existing ones; on match, reject or require out-of-band verification (email/phone).\\n4. **Reject invisible/spoofing characters**: default-ignorables, zero-width joiners, suspicious mark sequences (ICU `INVISIBLE`).\\n5. **Keep data current**: confusables.txt changes every Unicode release; re-validate stored identifiers after upgrades.\\n6. **Scope scripts to your audience**: allow Cyrillic/Greek for relevant locales, but never mixed with Latin under Highly/Moderately Restrictive.\\n7. **UX mitigations**: warn users about mixed-script names at creation; annotate or inspect-render names in feeds; use fonts that differentiate homoglyphs; pair with verified badges and impersonation reporting.\\n8. **Don't silently mutate** \u2014 reject with a clear error instead.\\n9. **Know the limits**: skeleton matching is approximate and font-dependent; supplement with policy (UTR #36) [8], monitoring, and rate-limiting lookalike registrations.\\n\\n## Sources\\n[1] UTS #39, *Unicode Security Mechanisms* \u2014 unicode.org/reports/tr39/\\n[2] confusables.txt \u2014 unicode.org/Public/security/latest/confusables.txt\\n[3] intentional.txt \u2014 unicode.org/Public/security/latest/intentional.txt\\n[4] DerivedCoreProperties.txt (Identifier_Status/Identifier_Type) \u2014 unicode.org/Public/UCD/latest/ucd/DerivedCoreProperties.txt\\n[5] UAX #31, *Identifier and Pattern Syntax* \u2014 unicode.org/reports/tr31/\\n[6] UAX #24, *Script Property* / ScriptExtensions.txt \u2014 unicode.org/reports/tr24/\\n[7] ICU Spoof Checker (ICU4C `uspoof` / ICU4J `SpoofChecker`) \u2014 unicode.org/reports/tr39/#ICU (userguide: strings/spoofchecker)\\n[8] UTR #36, *Unicode Security Considerations* \u2014 unicode.org/reports/tr36/\\n[9] UAX #15, *Normalization Forms* \u2014 unicode.org/reports/tr15/\\n\\n---\\n\\n## Detecting and mitigating invisible character and bidi spoofing attacks in text pipelines: Trojan Source attack, RLO/LRO overrides, UTS #55 source code handling, compiler warnings, GitHub invisible character warnings, OpenAI/LLM prompt injection via hidden Unicode characters\\n# Invisible Character & Bidi Spoofing: Detection and Mitigation in Text Pipelines\\n\\n*Note: no live search tool was available for this query; the summary below reflects primary sources documented through early 2025, indexed at the end.*\\n\\n## 1. The Trojan Source attack\\n\\n- **Core mechanism:** Unicode bidirectional (bidi) control characters embedded in source code (comments, string literals, or identifiers) cause the *visual* rendering to differ from the *compiled/executed* semantics \u2014 e.g., an \\\"early return\\\" attack where malicious code appears to be inside a comment, or a string whose characters display in reversed order via RLO (U+202E) [1][2].\\n- **Publication:** Boucher & Anderson (Cambridge), *\\\"Trojan Source: Invisible Vulnerabilities,\\\"* arXiv:2111.00169 (Nov 2021), later USENIX Security 2023 [1]. Coordinated disclosure to ~20 vendors; most patched within weeks [2].\\n- **CVEs:** CVE-2021-42574 (bidi spoofing in source) and CVE-2021-42694 (homoglyph identifier spoofing, e.g., Cyrillic \u0430 U+0430 vs. Latin a) [3][4].\\n- **Blast radius:** Nearly all major compilers/interpreters accepted bidi characters in comments and strings, making the attack language-agnostic (C/C++, Rust, Go, Swift, JavaScript, Python, etc.); the authors found bidi characters already present in real code in numerous popular repositories [1][2].\\n\\n## 2. Relevant invisible-character classes\\n\\n- **Bidi formatting:** LRE/RLE/PDF/LRO/RLO (U+202A\u2013202E), isolates LRI/RLI/FSI/PDI (U+2066\u20132069), LRM/RLM (U+200E/F) [16].\\n- **Zero-width/invisible:** ZWSP (U+200B), ZWNJ (U+200C), ZWJ (U+200D), WORD JOINER (U+2060), BOM/ZWNBSP (U+FEFF), soft hyphen (U+00AD).\\n- **Tag characters:** U+E0000\u2013U+E007F (legitimately used in emoji flag sequences, e.g., \ud83c\udff4\udb40\udc67\udb40\udc62\udb40\udc73\udb40\udc63\udb40\udc74\udb40\udc7f) \u2014 invisible in almost all renderers but machine-readable; central to LLM smuggling (\u00a75).\\n- **Homoglyphs:** cross-script confusables; handled via UTS #39 `confusables.txt` / skeleton algorithm and UAX #31 identifier profiles [6].\\n\\n## 3. UTS #55 and standards response\\n\\n- The Unicode Consortium published **UTS #55, \\\"Unicode Source Code Handling\\\"** (v1.0, late 2021) in direct response to Trojan Source. Recommendations: flag unpaired bidi controls in source; render bidi/invisible characters as visible escapes in editors, diffs, and terminals; restrict identifiers via UAX #31; guidance for code review tools and compilers [5].\\n- Key nuance: outright banning bidi breaks legitimate RTL localization; UTS #55 favors *pairing* checks (stack-based, per UAX #9) and visibility over blanket prohibition [5][16].\\n\\n## 4. Compiler, editor, and hosting mitigations\\n\\n- **GCC 12:** new `-Wbidi-chars=[none|unpaired|any]` warning covering comments, string/char literals, and identifiers; enabled by default at the `unpaired` level [7].\\n- **Rust 1.56:** lint `text_direction_codepoint_in_literal` (warn by default) flags bidi controls in literals [8].\\n- **Clang/LLVM:** added diagnostics for unpaired bidi control characters in comments and literals (LLVM 14 era); LLVM also hardened its own review practices [9].\\n- **VS Code 1.63 (Nov 2021):** default-on Unicode highlighting (`editor.unicodeHighlight.*`) for invisible and ambiguous/homoglyph characters, explicitly as a Trojan Source mitigation [11].\\n- **GitHub:** yellow banner on files/commits containing bidirectional text \u2014 *\\\"This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below\\\"* \u2014 plus a \\\"show hidden characters\\\" view (Oct\u2013Nov 2021) [10]. GitLab rolled out comparable warnings [10].\\n- **Important limits:** NFKC normalization does **not** remove zero-width or tag characters (they have no compatibility decompositions), and it does **not** unify cross-script homoglyphs (Cyrillic \u0430 and Latin a remain distinct) \u2014 so normalization alone is insufficient; filtering by code point/block and confusable detection are required [5][6].\\n\\n## 5. LLM prompt injection via hidden Unicode\\n\\n- **ASCII smuggling:** Tag characters (U+E0000\u2013E007F) are invisible in chat UIs but are tokenized as ordinary byte-level tokens by LLM tokenizers. Demonstrated by Rehberger (\\\"ASCII smuggling,\\\" 2023\u20132024): hidden instructions ride into prompts, and model outputs can encode exfiltrated data invisibly \u2014 e.g., Microsoft Copilot appending tag-character-encoded data to URLs for exfiltration. Microsoft's MSRC addressed this in \\\"Securing AI Model Outputs\\\" (May 2024) [12].\\n- **Hidden instructions:** Goodside (2023) demonstrated invisible prompt injection using Unicode tags with GPT models [13]; the academic paper *\\\"Invisible Prompt Injection: A Hijacking Attack on Large Language Model Applications\\\"* (arXiv, 2024) systematized hijacking via tag characters [14].\\n- **Other vectors:** zero-width characters used to split keywords and evade safety filters; RLO/bidi characters in *model output* spoofing chat UIs, logs, and terminals; homoglyph substitution bypassing keyword filters. Indirect injection via hidden characters in RAG-retrieved documents/emails is a recognized subcase [12][14][15].\\n- **Status:** No complete model-level fix exists; OWASP's LLM Top 10 lists prompt injection (LLM01) as the top risk, with input/output filtering as the primary mitigation [15].\\n\\n## 6. Practical detection/mitigation checklist for text pipelines\\n\\n1. **Scan** for bidi controls, zero-width, tag, and variation-selector characters, e.g. Python: `[\\\\u200b-\\\\u200f\\\\u202a-\\\\u202e\\\\u2060-\\\\u2064\\\\u2066-\\\\u2069\\\\u00ad\\\\ufeff\\\\U000e0000-\\\\U000e007f]` (plus FE00\u2013FE0F/E0100\u2013E01EF where relevant).\\n2. **Escape or strip** rather than silently pass through; log raw code points. Preserve valid emoji flag tag sequences only if required.\\n3. **Enforce bidi pairing** (stack-based per UAX #9) instead of blanket bans where RTL content is legitimate [5][16].\\n4. **Homoglyph defense:** UTS #39 skeleton/confusables + single-script restriction for identifiers [6].\\n5. **CI/CD:** enable GCC `-Wbidi-chars`, Rust's lint, Clang diagnostics; run the trojan-source detector tooling released with the paper [1][7][8][9].\\n6. **LLM pipelines:** sanitize inputs *and* outputs in both directions; never render model output as raw HTML/Markdown; treat retrieved documents as untrusted [12][14][15].\\n\\n## References\\n\\n[1] Boucher & Anderson, \\\"Trojan Source: Invisible Vulnerabilities,\\\" arXiv:2111.00169 (2021); USENIX Security 2023.\\n[2] trojansource.codes (project site: attack demos, vendor responses, detector tooling).\\n[3] CVE-2021-42574 (bidi source-code spoofing).\\n[4] CVE-2021-42694 (homoglyph identifier spoofing).\\n[5] UTS #55, \\\"Unicode Source Code Handling,\\\" Unicode Consortium (2021\u2013).\\n[6] UAX #31 (Identifier and Pattern Syntax); UTS #39 (Unicode Security Mechanisms).\\n[7] GCC 12 release notes / `-Wbidi-chars` documentation.\\n[8] Rust 1.56.0 release notes (`text_direction_codepoint_in_literal`).\\n[9] Clang/LLVM diagnostics for unpaired bidi control characters (LLVM 14 era).\\n[10] GitHub Changelog (Oct\u2013Nov 2021) bidirectional Unicode warnings; GitLab equivalent.\\n[11] VS Code 1.63 release notes \u2014 Unicode highlighting.\\n[12] Rehberger, Embrace The Red, \\\"ASCII Smuggling\\\" (2023\u20132024); Microsoft MSRC, \\\"Securing AI Model Outputs\\\" (2024).\\n[13] Goodside, invisible prompt injection via Unicode tag characters (2023).\\n[14] \\\"Invisible Prompt Injection: A Hijacking Attack on Large Language Model Applications\\\" (arXiv, 2024).\\n[15] OWASP Top 10 for LLM Applications \u2014 LLM01: Prompt Injection.\\n[16] UAX #9, Unicode Bidirectional Algorithm.\",\n            \"gen_ai.tool.call.id\": \"call_-7227502967786569353\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"f6f5cb4c3d165be7\",\n          \"parent_span_id\": \"7375a142115cdcb8\",\n          \"name\": \"model\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790384374798341255,\n          \"end_time_unix_nano\": 1790384500202773226,\n          \"duration_nano\": 125404431971,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"invoke_agent\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        },\n        {\n          \"trace_id\": \"a4319bc989dfef41ecbaf217f51a0c78\",\n          \"span_id\": \"08de2ee51a208da8\",\n          \"parent_span_id\": \"f6f5cb4c3d165be7\",\n          \"name\": \"ChatOpenAI\",\n          \"kind\": \"internal\",\n          \"status\": \"unset\",\n          \"start_time_unix_nano\": 1790384374805805697,\n          \"end_time_unix_nano\": 1790384500200495801,\n          \"duration_nano\": 125394690104,\n          \"attributes\": {\n            \"gen_ai.operation.name\": \"chat\"\n          },\n          \"events\": [],\n          \"resource\": {\n            \"telemetry.sdk.language\": \"python\",\n            \"telemetry.sdk.name\": \"opentelemetry\",\n            \"telemetry.sdk.version\": \"1.44.0\",\n            \"service.name\": \"abb-evaluation\"\n          },\n          \"scope\": {\n            \"name\": \"agentbench.observe\",\n            \"version\": \"\"\n          },\n          \"links\": []\n        }\n      ],\n      \"dropped_count\": 207,\n      \"truncated\": false,\n      \"reasons\": [\n        \"trace_attribute_not_allowlisted\"\n      ]\n    },\n    \"trace_capture_summary\": {\n      \"schema_version\": \"defuzex.trace_capture_summary.v1\",\n      \"sampling_policy\": \"deterministic_topology_v1\",\n      \"observed_spans\": 21,\n      \"retained_spans\": 21,\n      \"dropped_spans\": 0,\n      \"dropped_attributes_events\": 207,\n      \"topology_complete\": true,\n      \"observed_log_records\": 0,\n      \"retained_log_records\": 0,\n      \"dropped_log_records\": 0,\n      \"dropped_log_fields\": 0\n    },\n    \"runtime_evidence\": {\n      \"schema_version\": \"defuzex.runtime_evidence.v1\",\n      \"run_id\": \"run_8d4fd60ef2ed4920b54dcd649a921773\",\n      \"input_id\": \"step-10\",\n      \"step_id\": \"step-10\",\n      \"submission_id\": \"submission-24a44dd4590c01a0969696a84a928c4a\",\n      \"components\": [\n        {\n          \"component_id\": \"component-0000\",\n          \"sequence\": 0,\n          \"kind\": \"artifact_snapshot\",\n          \"artifact_id\": \"opentelemetry-trace-evidence\",\n          \"sha256\": \"764c8fd2b735444e447d1d8a2e2e8ec9352dff711cd1050e8de0bc5d08183e04\",\n          \"size_bytes\": 72662,\n          \"media_type\": \"application/vnd.defuzex.trace-evidence+json\"\n        },\n        {\n          \"component_id\": \"component-0001\",\n          \"sequence\": 1,\n          \"kind\": \"agent_response_claim\",\n          \"claim_id\": \"submission-response\",\n          \"claim\": \"completed\",\n          \"text_sha256\": \"a5b04c50beefaba3560e636d35254629725d6c2dde281df39c28b2ada8f151cd\"\n        }\n      ]\n    }\n  }\n}\n\n[\n  {\n    \"step\": \"0001\",\n    \"tool_call_id\": \"call_-7227546261056906827\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"ISO 17100 translation services standard: core requirements, translator competencies, and mandatory quality assurance/revision steps\",\n        \"Standard components of a professional translation brief or translation contract: scope, deliverables, terminology management, review and revision workflow\",\n        \"Best practices for adapting locale-specific conventions in translation: dates, numbers, currency, units of measure, punctuation and typography\",\n        \"Handling untranslatable elements in professional translation: placeholders, markup tags, code snippets, URLs, proper nouns, and terminology consistency rules\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## ISO 17100 translation services standard: core requirements, translator competencies, and mandatory quality assurance/revision steps\\n# ISO 17100:2015 \u2014 Translation Services Standard\\n\\n**ISO 17100:2015 \\\"Translation services \u2014 Requirements for translation services\\\"** was published in May 2015 (developed by ISO/PC 232) and superseded the earlier European standard EN 15038:2006 as the international benchmark. It specifies requirements for all aspects of the translation service delivery process by a Translation Service Provider (TSP) and applies to **human translation only** \u2014 raw machine translation output without human intervention is out of scope (post-editing of MT is covered separately by ISO 18587:2017) [1].\\n\\n## 1. Core Requirements\\n\\n- **Documented management system** for translation service provision, covering workflow, responsibilities, and record-keeping.\\n- **Defined roles**: translation service coordinator/project manager, translator, reviser, reviewer, and proofreader \u2014 each with specified competence requirements.\\n- **Client-facing processes**: quotation, a documented service agreement capturing project specifications (target language/locale, deadline, deliverables, terminology, style, domain, etc.), and client confirmation before production.\\n- **Project preparation and tracking**: feasibility assessment, resource assignment, technical preparation, and monitoring throughout production.\\n- **Technical resources**: appropriate tools (CAT/terminology management), secure handling, storage, and backup of client data.\\n- **Confidentiality**: non-disclosure agreements with clients and all personnel; data protection obligations.\\n- **Subcontracting** is permitted, but the TSP retains full responsibility and must ensure subcontractors meet the same requirements.\\n- **Post-production**: client feedback collection, complaint/nonconformity handling, and project closure [1][2].\\n\\n## 2. Translator Competencies\\n\\nThe standard defines **five core competencies** every translator must possess [1]:\\n\\n1. **Translation** \u2014 ability to conduct the full translation process to meet project specifications\\n2. **Linguistic and textual** \u2014 command of grammar, syntax, semantics, style, register, and text-type conventions in both source and target languages\\n3. **Research, information extraction and processing** \u2014 ability to acquire terminology/subject-matter knowledge and use reference materials and tools\\n4. **Cultural** \u2014 knowledge of source and target locale conventions, norms, and values\\n5. **Technical** \u2014 proficient use of translation technology and tools\\n\\n(Domain/subject-matter competency is an additional, context-dependent competency.)\\n\\n**Qualification evidence** \u2014 translators must demonstrate **at least one** of the following [1]:\\n- A formal higher education qualification in translation; **or**\\n- A higher education qualification in any other field **plus 2 years' full-time professional translation experience**; **or**\\n- **5 years' full-time professional translation experience**.\\n\\n**Other personnel**: revisers need translator-level competencies **plus professional translation experience**; reviewers need domain expertise; proofreaders need target-language competence; project managers need knowledge of the translation workflow [1][2].\\n\\n## 3. Mandatory Quality Assurance / Revision Steps\\n\\nISO 17100 enforces a **two-person principle**: translation and revision must be performed by different people [1][3].\\n\\n| Step | Status | Description |\\n|---|---|---|\\n| Translation | Mandatory | Performed by a qualified translator |\\n| **Check** | Mandatory | Translator's self-examination of the target text against the source |\\n| **Revision** | **Mandatory** | Independent second person compares target vs. source for completeness, accuracy, terminology, and conformity with specs; recommends corrective action |\\n| Review | Optional (if agreed) | Assessment of fitness for purpose in the domain, terminology correctness, and register \u2014 requires domain expertise |\\n| Proofreading | Optional (if agreed) | Pre-delivery check for typos, omissions, and formatting errors |\\n| **Final verification and release** | Mandatory | Designated person verifies the deliverable against agreed specifications before release |\\n| Delivery & feedback | Mandatory | Delivery, client feedback evaluation, complaint handling |\\n\\n**Key point**: only the translator self-check, independent revision, and final verification/release are obligatory for every project; review and proofreading are value-added steps performed when specified in the service agreement [1][3].\\n\\nTSPs can be third-party certified against ISO 17100 by accredited certification bodies.\\n\\n---\\n*Note: No live search results were returned in this session; the above reflects the published text of the standard and its widely documented summaries. Key references: [1] ISO 17100:2015, ISO.org (standard 59149); [2] GALA \u2014 \\\"ISO 17100\\\" overview; [3] ISO 17100:2015 Clause 6 (production process: check, revision, review, proofreading, final verification).*\\n\\n---\\n\\n## Standard components of a professional translation brief or translation contract: scope, deliverables, terminology management, review and revision workflow\\n# Standard Components of a Professional Translation Brief / Contract\\n\\nThe components below reflect the main industry frameworks: **ISO 17100:2015** (translation services requirements) [1], **ASTM F2575** (Standard Guide for Quality Assurance in Translation) [2], and the **European Commission DGT translation brief model** [3], supplemented by common terms of business used by professional associations (ATA, ITI, ELIA) [4].\\n\\n## 1. Scope\\n- **Languages and directions** (e.g., EN\u2192DE only, not reverse), subject-matter domains (legal, medical, technical, marketing), and text types (contracts, manuals, web copy).\\n- **Service definition**: translation only vs. full TEP (translation\u2013editing\u2013proofreading), machine-translation post-editing (MTPE), transcription, DTP/layout, software/website localization, subtitling, sworn/certified translation \u2014 and explicit **exclusions**.\\n- **Volume and pricing basis**: word counts, unit of count (source vs. target words), rates per word/hour/project, minimum fees.\\n- **Purpose, audience, and register**: ASTM F2575 and the DGT brief both require stating the intended function of the target text and its target audience, since quality is defined as conformance to agreed specifications [2][3].\\n- **Source material condition**: editable files vs. scans/handwriting, completeness, legibility; assumptions made.\\n- **Deadlines, milestones, partial deliveries**, and a **change-control procedure** (out-of-scope changes trigger re-quotation).\\n- **Confidentiality and data-protection requirements** (NDA, GDPR, secure file transfer).\\n\\n## 2. Deliverables\\n- **Output formats**: same format as source (DOCX, IDML, JSON, XLIFF), or specified alternatives; QA/error reports if promised.\\n- **Project assets**: translation memory (TM), termbase/glossary, style guide \u2014 with explicit **ownership and licensing terms** (client-owned vs. provider-owned).\\n- **Certification or sworn statements** where legally required.\\n- **Delivery method, acceptance window, and acceptance criteria** (e.g., client has 10 business days to review; silence constitutes acceptance).\\n- **Invoicing documentation** tied to the agreed unit of count.\\n\\n## 3. Terminology Management\\n- **Glossary/termbase creation** before production: key terms, definitions, approved target equivalents; **client validation** of the glossary as a project milestone.\\n- **Do-not-translate lists**, brand names, product names, and legacy terminology from previous translations.\\n- **CAT-tool consistency enforcement** and TM leverage; ISO 17100 requires the provider to verify terminology appropriateness and domain conventions as part of project preparation and translation [1].\\n- **Maintenance responsibilities** during and after the project, and who may add/modify entries.\\n\\n## 4. Review and Revision Workflow\\n- **Defined process steps**. ISO 17100 specifies: translation \u2192 self-check by the translator \u2192 **revision** (bilingual check of target against source for completeness/accuracy by a second qualified linguist) \u2192 optional **review** (monolingual check of fitness for purpose) \u2192 optional **proofread** (final check before publication) [1]. The commercial equivalent is the \\\"TEP\\\" model.\\n- **Number of revision rounds included** in the fee; how client feedback is consolidated (single-channel feedback, tracked changes).\\n- **Automated QA checks** (terminology consistency, numbers, tags) via QA tools.\\n- **Acceptance and complaint handling**: rectification obligations for genuine errors within a defined period; post-acceptance changes treated as chargeable change requests.\\n- **Reviewer qualifications**: ISO 17100 requires revisers to meet the same competence criteria as translators plus experience in the domain [1].\\n\\n## Other Standard Contract Clauses (brief)\\n- IP rights/copyright assignment or license in the translations; confidentiality surviving termination; liability caps (commonly limited to fees paid); payment terms (e.g., net 30, deposits for new clients); cancellation fees; non-solicitation; dispute resolution/governing law [4].\\n\\n**Key principle across frameworks**: quality disputes are adjudicated against the *written specifications* agreed in the brief/contract, so the four areas above should be documented before work begins [1][2][3].\\n\\n---\\n**Sources**\\n[1] ISO 17100:2015, *Translation services \u2014 Requirements for translation services*\\n[2] ASTM F2575, *Standard Guide for Quality Assurance in Translation*\\n[3] European Commission Directorate-General for Translation (DGT), *Translation Brief* framework\\n[4] Professional association guidance on terms of business (American Translators Association; Institute of Translation and Interpreting; ELIA)\\n\\n---\\n\\n## Best practices for adapting locale-specific conventions in translation: dates, numbers, currency, units of measure, punctuation and typography\\n# Locale Adaptation Best Practices: Research Summary\\n\\n## Core principles\\n- Format decisions must key on **locale, not language**: pt\u2011BR and pt\u2011PT, en\u2011US and en\u2011GB differ substantially in dates, numbers, and units [1][2].\\n- Never hardcode formats. Use locale-aware libraries (ICU, Java/.NET formatters, ECMAScript `Intl`) backed by the **Unicode CLDR** as the canonical data source for date, number, currency, and unit patterns [2][8].\\n- Internationalize first: externalize all strings and locale-sensitive data so translators and formatters\u2014not code\u2014control conventions [4][5][11].\\n\\n## Dates and times\\n- Component order varies: MDY (en\u2011US), DMY (most of Europe/Latin America), YMD (East Asia); separators vary too (German uses dots: 31.12.2025) [1][2].\\n- Use spelled-out month names where ambiguity matters; use **ISO 8601 (YYYY\u2011MM\u2011DD)** for technical/machine-readable contexts [6].\\n- Account for first day of week (Sunday vs. Monday vs. Saturday), 12\u2011hour vs. 24\u2011hour clocks, and non-Gregorian calendars (Buddhist in Thailand, Hijri, Japanese eras) [2][8].\\n- Handle time zones and DST via libraries, never manual offsets [8].\\n\\n## Numbers\\n- Decimal/group separators swap between locales: 1,234.56 (en\u2011US) vs. 1.234,56 (de\u2011DE, pt\u2011BR); French uses a narrow no-break space (1 234 567,89); Swiss German uses an apostrophe (1'234'567) [2][8].\\n- Grouping is not always by thousands: India uses lakh/crore grouping (12,34,56,789) [2].\\n- Some locales use non-Western digit shapes (Arabic-Indic, Devanagari, Thai) [2].\\n- Percent-sign spacing/placement varies (French: \\\"50 %\\\") [2].\\n\\n## Currency\\n- Symbol position and spacing vary: $1,234.56 vs. 1.234,56 \u20ac vs. 1 234,56 \u20ac \u2014 apply via CLDR currency patterns, not string concatenation [2][8].\\n- Symbols are ambiguous across markets ($ = USD/CAD/AUD/MXN; kr = SEK/NOK/DKK); use **ISO 4217 codes** (USD, EUR, JPY) when disambiguation matters [7][15].\\n- Minor-unit digits differ: JPY has 0, most currencies 2, KWD/BHD have 3 \u2014 don't assume two decimals [2][7].\\n\\n## Units of measure\\n- Convert **values**, not just symbols: metric vs. US customary vs. mixed UK usage (miles for distance, Celsius for temperature, stone for body weight) [1][3].\\n- SI rules: a (non-breaking) space between value and unit symbol (25 kg, 300 mL); no space after the degree sign (20 \u00b0C uses a space, but angles like 45\u00b0 do not) [9].\\n- Localize ancillary formats too: paper sizes (A4 vs. US Letter), clothing/shoe sizes, address and phone formats [3][11].\\n- Unit mismatches cause real failures \u2014 e.g., the 1999 loss of NASA's Mars Climate Orbiter to a metric/imperial mix-up; verify all converted figures [10].\\n\\n## Punctuation and typography\\n- Quotation marks differ: \u201c \u201d/\u2018 \u2019 (English), \u00ab \u00bb (French, Spanish, Russian, pt\u2011PT), \u201e \u201c (German), \u300c \u300d (Japanese), \u300a \u300b for titles in Chinese [2][11][12].\\n- French requires non-breaking spaces inside guillemets and before ; : ! ? [2].\\n- Spanish uses inverted \u00bf \u00a1 [11].\\n- Capitalization conventions differ: English title case vs. sentence case elsewhere; German capitalizes all nouns; Arabic, Hebrew, CJK, and Devanagari have no case \u2014 avoid ALL-CAPS styling assumptions [3][11].\\n- Line breaking: CJK follows kinsoku (forbidden break) rules; Thai, Lao, Khmer, and Myanmar have no inter-word spaces and need dictionary-based breaking (ICU) [8][12].\\n- **Text expansion/contraction**: German runs ~30% longer than English; CJK often shorter \u2014 build flexible layouts [3][4].\\n- Bidirectional scripts (Arabic, Hebrew, Persian, Urdu) require RTL layout, the Unicode Bidi Algorithm, and mirrored punctuation [13][14].\\n\\n## Process and QA\\n- Run **pseudo-localization** to expose hardcoded formats, truncation, and expansion issues before translation [4][11].\\n- Follow published per-language style guides (Microsoft, Google, Apple) and the EU Interinstitutional Style Guide for pan-European consistency [3][4][5][15].\\n- Use CAT-tool automated QA checks for number/date/currency consistency and have in-country (native) reviewers validate final formatting [3][11].\\n\\n---\\n\\n## Sources\\n[1] W3C Internationalization (i18n) articles \u2014 w3.org/International\\n[2] Unicode Common Locale Data Repository (CLDR) \u2014 cldr.unicode.org\\n[3] Microsoft Language Portal, per-language style guides\\n[4] Google developer documentation, \\\"Internationalization/Localization\\\"\\n[5] Apple, \\\"Internationalization and Localization Guide\\\"\\n[6] ISO 8601-1:2019, Date and time \u2014 Representations for information interchange\\n[7] ISO 4217, Currency codes\\n[8] ICU (International Components for Unicode) documentation\\n[9] BIPM, The International System of Units (SI Brochure), 9th ed., 2019\\n[10] NASA, Mars Climate Orbiter Mishap Investigation Board Phase I Report (1999)\\n[11] Dr. International, *Developing International Software*, 2nd ed. (Microsoft)\\n[12] W3C, Requirements for Japanese Text Layout (JLREQ)\\n[13] W3C, Arabic Layout Requirements (Editor's Draft)\\n[14] Unicode Standard Annex #9, Unicode Bidirectional Algorithm\\n[15] European Commission, English Style Guide / Interinstitutional Style Guide\\n\\n---\\n\\n## Handling untranslatable elements in professional translation: placeholders, markup tags, code snippets, URLs, proper nouns, and terminology consistency rules\\n# Handling Untranslatable Elements in Professional Translation\\n\\n## Placeholders\\n- Tokens such as `%s`, `{0}`, `{{user_name}}`, and `$1` must be preserved verbatim; translators may reposition them to match target-language word order, which requires *positional/numbered* placeholders (e.g., `%1$s` in Android, `{0}` in .NET) rather than sequential ones [6][8].\\n- ICU MessageFormat is the standard for complex cases (pluralization, gender) where placeholder selection varies by locale [6].\\n- CAT tools treat placeholders as protected \\\"placeables\\\" that are copied, not typed, and QA checks flag missing, extra, or corrupted tokens [9][10].\\n- In machine translation pipelines, placeholders are masked or tokenized so MT cannot alter them [7].\\n\\n## Markup Tags\\n- Inline HTML/XML tags are locked as paired units; QA validation detects deleted, added, illegally reordered, or split tags [2][10].\\n- XLIFF 2.x models inline content with elements such as ``, ``, and ``/`` so tools can enforce tag integrity across systems [2].\\n- The W3C ITS (Internationalization Tag Set) defines data categories like \\\"Translate / Do Not Translate\\\" and \\\"Within Text\\\" to declare which markup nodes are translatable [1].\\n\\n## Code Snippets\\n- Best practice is *string externalization*: separating user-facing text into resource files (gettext `.po`, `.resx`, `.strings`, JSON) so code and translatable content never mix [14][8].\\n- Identifiers, function names, API keys, and CLI flags remain in the source language; only string literals and comments are translation candidates [8].\\n- In documentation, code blocks are typically locked or marked DNT (\\\"do not translate\\\") in the CAT tool [9][7].\\n\\n## URLs\\n- URLs are non-translatable placeables \u2014 altering them breaks links \u2014 so they are protected or inserted as tags [9][10].\\n- Localized domains and paths (IDN domains, locale directories such as `/de-de/`) are editorial/SEO decisions, not translation tasks [11].\\n- `hreflang` annotations signal language/region targeting between localized page versions [11].\\n\\n## Proper Nouns\\n- Default rule: personal and organization names are not translated; different scripts use transliteration standards (ISO 9 for Cyrillic, Hanyu Pinyin for Chinese, UNGEGN systems for geographic names) [12].\\n- Exceptions governed by style guides/termbases: conventional exonyms (e.g., Germany/Allemagne), officially localized brand and product names, and established local forms of entity names [4][5].\\n- DNT lists lock specific terms from translation and MT post-editing [7][9].\\n\\n## Terminology Consistency\\n- Termbases in **TBX** format (ISO 30042) store approved source\u2013target term pairs, definitions, and status; translation memories in **TMX** store validated segment pairs [3].\\n- CAT tools enforce consistency through termbase recognition, term highlighting, and automated terminology QA (missing term, forbidden term, inconsistent rendering) [9][10].\\n- Client style guides (e.g., Microsoft Writing Style Guide, Google Developer Documentation Style Guide) define casing, capitalization, and when product names may be translated [4][5].\\n- Dedicated QA tools such as Xbench and Verifika run cross-file terminology and consistency checks before delivery [13].\\n- Working rule: one approved target term per source term per product/context; any deviation requires documented approval in the termbase [3][13].\\n\\n**Typical workflow:** pre-extract strings and lock non-translatables (code, tags, URLs, DNT terms) \u2192 translate with termbase and TM support \u2192 run automated QA (placeholders, tags, terminology) \u2192 human review against the style guide [1][2][9][10][13].\\n\\n---\\n**Sources:** [1] W3C ITS / i18n documentation \u00b7 [2] OASIS XLIFF 2.x specification \u00b7 [3] ISO 30042 (TBX) \u00b7 [4] Microsoft Writing Style Guide \u00b7 [5] Google Developer Documentation Style Guide \u00b7 [6] ICU MessageFormat docs \u00b7 [7] Smartling knowledge base \u00b7 [8] Lokalise i18n guides \u00b7 [9] Phrase (Memsource) documentation \u00b7 [10] RWS Trados documentation \u00b7 [11] Google Search Central (hreflang) \u00b7 [12] UNGEGN transliteration standards \u00b7 [13] Xbench / Verifika QA tool docs \u00b7 [14] GNU gettext manual\",\n    \"t_start\": \"2026-09-25T22:24:38.578033+00:00\",\n    \"t_end\": \"2026-09-25T22:27:48.218160+00:00\",\n    \"index\": 0\n  },\n  {\n    \"step\": \"0002\",\n    \"tool_call_id\": \"call_-7227534681825080568\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"English Wikipedia article 'Deforestation of the Amazon rainforest': exact wording of passages on deforestation rates, statistics, and trends (INPE/PRODES data, 1970s to 2020s annual loss figures)\",\n        \"English Wikipedia coverage of causes of Amazon deforestation: exact quoted passages on cattle ranching, soy farming, logging, mining, roads, and fires as drivers\",\n        \"English Wikipedia coverage of consequences and responses to Amazon deforestation: exact quoted passages on biodiversity loss, climate/tipping point concerns, indigenous peoples, and conservation policy (e.g., Amazon Cooperation Treaty, Brazil enforcement)\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## English Wikipedia article 'Deforestation of the Amazon rainforest': exact wording of passages on deforestation rates, statistics, and trends (INPE/PRODES data, 1970s to 2020s annual loss figures)\\n**Important caveat up front:** I don't have live web access in this session, so I cannot pull the current revision of the article verbatim. What follows is a reconstruction of the key passages from my training-data memory of \\\"Deforestation of the Amazon rainforest\\\" (snapshots through roughly early 2025). Wording is close to the article but **not guaranteed to be verbatim**, and the article is updated frequently with each new annual INPE release \u2014 please verify exact phrasing against the live article's page history.\\n\\n---\\n\\n## Lead / overview passages\\n\\n- \\\"Since 1970, more than 700,000 square kilometres (270,000 sq mi) of the Amazon rainforest have been destroyed.\\\" *(older lead versions; later updated to reflect cumulative loss)*\\n- Lead versions from ~2021 onward stated that roughly **17% of the Amazon rainforest has been destroyed**, and later snapshots moved toward \\\"nearly 20%.\\\"\\n- \\\"Roughly 80% of deforested land in the Amazon is used for cattle ranching/pasture.\\\"\\n\\n## INPE / PRODES monitoring description\\n\\n- \\\"In Brazil, the Instituto Nacional de Pesquisas Espaciais (INPE \u2013 National Institute of Space Research) produces annual estimates of deforestation in the Brazilian Amazon through the **PRODES** project, which has monitored deforestation by satellite since 1988.\\\" The article notes PRODES measures clearing in the \\\"PRODES year\\\" (August\u2013July) and that **DETER** (Real-Time Deforestation Detection System) provides shorter-term alerts.\\n\\n## Historical trend (1970s\u20132004 peak)\\n\\n- \\\"Between 1991 and 2000, the total area of forest lost in the Amazon rose from 415,000 to 587,000 km\u00b2, an area the size of Spain, with most of the cleared land becoming pasture for cattle.\\\" *(sourced to BBC; this sentence persisted for years)*\\n- \\\"Deforestation peaked in 2004 at 27,772 km\u00b2, the highest annual rate since INPE began systematic monitoring in 1988.\\\" A 1995 spike (29,059 km\u00b2) appears in the table.\\n- The article notes deforestation concentrated along the \\\"**arc of deforestation**\\\" on the southern/eastern forest edge, driven by the Trans-Amazonian Highway (1972) and colonization programs.\\n\\n## Decline (2004\u20132012)\\n\\n- \\\"The annual rate of deforestation in the Amazon region dramatically decreased between 2004 and 2012, falling from 27,772 km\u00b2 in 2004 to 4,571 km\u00b2 in 2012 \u2014 a decline of about 84% and the lowest rate since records began.\\\" \\n- The article attributes this to PPCDAm (2004), enforcement, satellite monitoring, the Soy Moratorium (2006), the Cattle Agreement (2009), and expansion of protected areas.\\n\\n## Renewed increase (2013\u20132021)\\n\\n- 2013: deforestation rose ~28% to 5,891 km\u00b2, ending years of decline; 2016 rose to 7,893 km\u00b2.\\n- 2019: \\\"According to INPE, deforestation in June 2019 was 88% higher than in June 2018\\\"; annual PRODES figure 10,129 km\u00b2 (+34% vs. 2018), the highest since 2008.\\n- 2020: \\\"Deforestation in the first three months of 2020 rose more than 50% compared with the same period in 2019\\\"; PRODES 2020: 10,851 km\u00b2 (+7.2%).\\n- 2021: \\\"Deforestation in the Brazilian Amazon reached 13,038 km\u00b2 between August 2020 and July 2021, a 22% increase and the highest annual rate since 2006.\\\"\\n\\n## 2022\u20132024 decline\\n\\n- 2022: 11,594 km\u00b2 (\u221211%).\\n- 2023: \\\"Deforestation fell 22% to 9,064 km\u00b2, the lowest rate since 2018\\\" (credited to renewed enforcement under the Lula administration).\\n- 2024: 6,288 km\u00b2 (\u221230.6%), lowest in nine years \u2014 present only if the snapshot post-dates the November 2024 PRODES release.\\n\\n## Annual rates table (as included in the article, km\u00b2/year, INPE/PRODES)\\n\\n| Year | Rate | Year | Rate | Year | Rate |\\n|---|---|---|---|---|---|\\n| 1977 | 21,130 | 1995 | 29,059 | 2013 | 5,891 |\\n| 1978 | 23,750 | 1996 | 18,161 | 2014 | 5,012 |\\n| 1979 | 17,725 | 1997 | 13,227 | 2015 | 6,207 |\\n| 1980 | 23,525 | 1998 | 17,383 | 2016 | 7,893 |\\n| 1988 | 21,050 | 1999 | 17,259 | 2017 | 6,947 |\\n| 1989 | 17,383 | 2000 | 18,226 | 2018 | 7,536 |\\n| 1990 | 13,730 | 2001 | 18,772 | 2019 | 10,129 |\\n| 1991 | 11,030 | 2002 | 21,511 | 2020 | 10,851 |\\n| 1992 | 13,786 | 2003 | 25,396 | 2021 | 13,038 |\\n| 1993 | 14,896 | 2004 | 27,772 | 2022 | 11,594 |\\n| 1994 | ~14,896* | 2005 | 19,014 | 2023 | 9,064 |\\n| | | 2006 | 14,286 | 2024 | 6,288 |\\n| | | 2007 | 11,251 | | |\\n| | | 2008 | 12,911 | | |\\n| | | 2009 | 7,008 | | |\\n| | | 2010 | 7,000 | | |\\n| | | 2011 | 6,418 | | |\\n| | | 2012 | 4,571 | | |\\n\\n\\\\* A few mid-1990s values (1993/1994) I cannot fully disambiguate from memory; the live table should be checked.\\n\\n---\\n\\n**Recommendation:** For verbatim text, consult the live article's \\\"Deforestation rates\\\" / \\\"History\\\" sections and its revision history (en.wikipedia.org/wiki/Deforestation_of_the_Amazon_rainforest), since annual figures and lead text are revised after each INPE/PRODES release.\\n\\n---\\n\\n## English Wikipedia coverage of causes of Amazon deforestation: exact quoted passages on cattle ranching, soy farming, logging, mining, roads, and fires as drivers\\n# Wikipedia Coverage: Causes of Amazon Deforestation\\n\\n**Important caveat:** I cannot retrieve live pages, so the passages below are drawn from my last full snapshot of English Wikipedia (early 2025). Wording may have shifted in intervening revisions \u2014 verify against current article versions before citing as verbatim.\\n\\n## Cattle ranching [1]\\nThe article identifies ranching as the dominant driver, stating in substance:\\n- \\\"Cattle ranching accounts for approximately 70% of deforestation in the Amazon.\\\"\\n- \\\"91% of land deforested since 1970 in the Amazon is used for livestock pasture.\\\"\\n- The article notes Brazil's status as a leading global beef exporter and links demand for beef to pasture conversion, citing Greenpeace and World Bank sources.\\n\\n## Soy farming [1][2]\\n- \\\"Brazil is the world's second-largest producer of soybeans,\\\" with expansion concentrated in Mato Grosso and the Cerrado\u2013Amazon transition zone.\\n- On the Amazon Soy Moratorium (2006): major soy traders \\\"agreed not to purchase soybeans produced on land deforested in the Amazon biome after July 2006.\\\" The article credits the moratorium with reducing \u2014 but not eliminating \u2014 soy-driven direct deforestation, noting soy's larger indirect role via displaced ranching.\\n\\n## Logging [1][3]\\n- The article distinguishes legal from illegal logging, stating that a large majority (figures cited range from ~60% to 90%) of logging in the Brazilian Amazon is illegal.\\n- Selective logging is described as degrading forest structure and increasing fire susceptibility even where canopy cover persists.\\n\\n## Mining [1]\\n- Coverage emphasizes small-scale gold mining (*garimpos*) and mercury contamination of rivers, plus large-scale operations such as iron-ore extraction around Caraj\u00e1s (Par\u00e1).\\n- Mining is presented as a smaller share of total deforestation than agriculture but with disproportionate impact on indigenous territories (e.g., Yanomami land).\\n\\n## Roads [1][4][5]\\n- Roads are described as the key enabler of other drivers: highway construction \\\"opens previously inaccessible areas\\\" and \\\"deforestation typically follows road construction\\\" in corridor patterns.\\n- The Trans-Amazonian Highway (BR-230) is cited as the historical case study from the 1970s; the article flags plans to pave BR-319 through relatively intact central Amazon forest as a major contemporary risk.\\n\\n## Fires [1][6]\\n- Fires are characterized as almost entirely human-set: the rainforest \\\"does not naturally burn,\\\" and fire is used for land clearing (slash-and-burn) and pasture management, frequently escaping into standing forest.\\n- The 2019 wildfires article states the season \\\"saw a year-to-year surge in fires occurring in the Amazon rainforest biome within Brazil, Bolivia, Paraguay, and Peru,\\\" that fires were \\\"mostly started by farmers and loggers clearing land,\\\" and reports INPE deforestation data alongside criticism of the Bolsonaro government's environmental enforcement rollback.\\n\\n## Sources\\n[1] Wikipedia: *Deforestation of the Amazon rainforest* \u2014 main \\\"Causes\\\" section\\n[2] Wikipedia: *Amazon rainforest* (human activities section)\\n[3] Wikipedia: *Illegal logging* (Brazil/Amazon coverage)\\n[4] Wikipedia: *Trans-Amazonian Highway*\\n[5] Wikipedia: *BR-319 (Brazilian highway)*\\n[6] Wikipedia: *2019 Amazon rainforest wildfires*\\n\\n**Recommendation:** For quotable verbatim text, pull directly from each article's revision history or current version, since these passages are reconstructions from my training snapshot rather than live retrieval.\\n\\n---\\n\\n## English Wikipedia coverage of consequences and responses to Amazon deforestation: exact quoted passages on biodiversity loss, climate/tipping point concerns, indigenous peoples, and conservation policy (e.g., Amazon Cooperation Treaty, Brazil enforcement)\\n# Wikipedia Coverage: Amazon Deforestation \u2014 Consequences & Responses\\n\\n**Caveat:** Quotes below are drawn from the English Wikipedia articles as of my last knowledge update (early 2025). Wikipedia text changes frequently; exact wording should be verified against current revisions or permanent links.\\n\\n## 1. Biodiversity loss\\n\\n- From the lead of the **Amazon rainforest** article: *\\\"The Amazon rainforest, also called Amazon jungle or Amazonia, is a moist broadleaf tropical rainforest in the Amazon biome that covers most of the Amazon basin of South America.\\\"* [1]\\n- Biodiversity section: *\\\"One in ten known species in the world lives in the Amazon rainforest.\\\"* [1]\\n- Same section on classified species: *\\\"To date, at least 40,000 plant species, 2,200 fishes, 1,294 birds, 427 mammals, 428 amphibians, and 378 reptiles have been scientifically classified in the region.\\\"* [1]\\n- The **Deforestation in Brazil** article opens: *\\\"Brazil once had the highest deforestation rate in the world and in 2005 still had the largest area of forest removed annually. Since 1970, over 700,000 square kilometres (270,000 sq mi) of the Amazon rainforest have been destroyed.\\\"* [3]\\n\\n## 2. Climate / tipping point concerns\\n\\n- The **Deforestation of the Amazon rainforest** article discusses the 2021 *Nature* study (Flores et al.), reporting that roughly **75% of the forest has shown loss of resilience since the early 2000s** and warning the forest may be *\\\"close to a critical threshold\\\"* beyond which large areas would degrade toward savanna-like vegetation. [2]\\n- Scientists' threshold estimate (Nobre & Lovejoy, covered in [2] and [4]): deforestation of **20\u201325%** is cited as the point at which *\\\"dieback\\\"* \u2014 a self-reinforcing transition of forest to savanna \u2014 could be triggered. [2][4]\\n- The **Tipping points in the climate system** article lists **Amazon rainforest dieback** among major climate \\\"tipping elements,\\\" noting that interaction of deforestation, fire, and drought could push the system past a threshold even without additional warming. [4]\\n- The Amazon rainforest article also debunks a common misconception, noting the \\\"lungs of the planet\\\" label and the claim that the forest produces 20% of Earth's oxygen are inaccurate. [1]\\n\\n## 3. Indigenous peoples\\n\\n- From the **Amazon rainforest** article (human activity section, citing RAISG data): *\\\"More than 30 million people of 350 different ethnic groups live in the Amazon... Indigenous peoples make up 9% of the total population, and 60% of the Amazon's land area is under formal recognition of indigenous rights.\\\"* [1]\\n- Multiple articles state that **indigenous territories (Terras Ind\u00edgenas) function as barriers to deforestation**, with deforestation rates inside demarcated territories significantly lower than in surrounding areas. [1][2][6]\\n- Coverage notes that **demarcation of indigenous land** slowed or stalled under the Bolsonaro government (2019\u20132022) alongside weakened enforcement. [3][6]\\n\\n## 4. Conservation policy & enforcement\\n\\n- **Amazon Cooperation Treaty Organization** article: *\\\"The Amazon Cooperation Treaty Organization (ACTO) is an international organization aimed at the promotion of sustainable development of the Amazon Basin. Its member states are Bolivia, Brazil, Colombia, Ecuador, Guyana, Peru, Suriname and Venezuela.\\\"* The treaty was *\\\"signed on 3 July 1978 in Bras\u00edlia.\\\"* [5]\\n- On the 2023 **Bel\u00e9m Declaration** (Amazon Summit, Bel\u00e9m, August 2023): member states agreed to cooperate against deforestation, including an alliance mechanism; coverage notes criticism that the declaration **set no common deforestation-end target**. [5]\\n- **Brazil enforcement** [3][7]: The 2004 launch of **PPCDAm** (Action Plan for Prevention and Control of Deforestation in the Legal Amazon) is credited with cutting deforestation ~80% between 2004 and 2012; **IBAMA** is described as the federal agency responsible for enforcing environmental law. The articles document that enforcement capacity and fines were curtailed under Bolsonaro (2019\u20132022), with deforestation rising sharply, and that in **2023 deforestation fell ~50%** versus 2022 under the Lula government per INPE/PRODES data, with a further decline reported for 2024. [3][7]\\n\\n## Sources\\n\\n[1] *Amazon rainforest* (Wikipedia) \u2014 lead, Biodiversity, Human activity sections\\n[2] *Deforestation of the Amazon rainforest* (Wikipedia) \u2014 causes, tipping point coverage\\n[3] *Deforestation in Brazil* (Wikipedia) \u2014 lead, PPCDAm, Bolsonaro/Lula-era data\\n[4] *Tipping points in the climate system* (Wikipedia) \u2014 Amazon dieback entry\\n[5] *Amazon Cooperation Treaty Organization* (Wikipedia) \u2014 lead, 1978 treaty, Bel\u00e9m Declaration\\n[6] *Indigenous peoples in Brazil* / *Indigenous territory* (Wikipedia)\\n[7] *Ibama* (Wikipedia) \u2014 enforcement mandate\",\n    \"t_start\": \"2026-09-25T22:29:24.019196+00:00\",\n    \"t_end\": \"2026-09-25T22:33:10.818589+00:00\",\n    \"index\": 1\n  },\n  {\n    \"step\": \"0003\",\n    \"tool_call_id\": \"call_-7227558321325077612\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"How do CAT tools (SDL Trados Studio, MemoQ, OmegaT, Phrase/Memsource) implement segment locking / locked segments, and how are locked segments treated during translation, QA, and file export?\",\n        \"What mechanisms preserve inline markup, tags, and placeholders during translation (tag protection, XLIFF inline elements, placeholder tokens, tag integrity QA checks)?\",\n        \"How is translation context stored and preserved with source segments (translation memory context match, XLIFF context groups, segment metadata, prev/next segment context)?\",\n        \"How do machine translation systems and LLM-based translation pipelines preserve markup, placeholders, and non-translatable elements (Do Not Translate lists, tag-aware MT, prompt-based markup preservation)?\",\n        \"What localization standards formalize segmentation, markup preservation, and context (XLIFF 1.2/2.x, TMX, SRX, W3C ITS 2.0)?\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## How do CAT tools (SDL Trados Studio, MemoQ, OmegaT, Phrase/Memsource) implement segment locking / locked segments, and how are locked segments treated during translation, QA, and file export?\\n## Overview\\n\\nSegment locking marks individual segments as **read\u2011only** so translators cannot modify their target content. Typical uses: boilerplate/legal text, do\u2011not\u2011translate content, pre\u2011approved TM matches (e.g., 100%/context matches, PerfectMatch), and MT post\u2011editing workflows with restricted edit scope. Implementation differs significantly across the four tools: **Trados Studio, memoQ, and Phrase have native per\u2011segment locks; OmegaT does not**.\\n\\n---\\n\\n## SDL Trados Studio [1][2]\\n\\n- **Locking:** Manual \u2014 select segment(s) \u2192 *Home* ribbon \\\"Lock Segment\\\" toggle (Ctrl+L) or right\u2011click menu. PerfectMatch (matching against previously translated project files) **automatically locks** matched segments (shown with a padlock/PM marker); they can be unlocked manually.\\n- **Storage:** Lock state is persisted in the bilingual SDLXLIFF on the segment definition (`` with a locked attribute), so it survives round\u2011tripping of the bilingual file.\\n- **Translation:** Target cell is read\u2011only; the segment must be unlocked to edit. Locked segments can be filtered out of view via the editor's display/segment filter (version\u2011dependent).\\n- **QA:** Verification/QA Checker runs project\u2011wide with no dedicated \\\"locked\\\" check; editing attempts are simply blocked, so any QA finding inside a locked segment can only be resolved after unlocking.\\n- **Export:** Bilingual SDLXLIFF keeps the lock flag; *Save Target As* writes the locked segments' current target text unchanged. (Content marked non\u2011translatable in file\u2011type settings is a separate mechanism \u2014 it never becomes a segment at all.)\\n\\n## memoQ [3]\\n\\n- **Locking:** Manual \u2014 select segment(s) \u2192 context menu/ribbon \\\"Lock\\\"/\\\"Unlock\\\" (desktop editor and WebTrans). Locked rows show a padlock icon with greyed, non\u2011editable target cells.\\n- **Storage:** Persisted with the segment in the project (local/server) and carried in bilingual MQXLZ/MQXLIFF.\\n- **Translation:** Locked targets are read\u2011only; unlock to edit. No widely documented automated lock\u2011on\u2011match feature.\\n- **QA:** *Run QA* checks the document as a whole; because locked content cannot be modified, issues flagged in locked segments require unlocking (no dedicated locked\u2011segment check documented).\\n- **Export:** Locked segments export like any other, with their existing target text; the lock is metadata and is not written into the final target file.\\n\\n## OmegaT [4]\\n\\n- **No native per\u2011segment lock command.** Protection is achieved indirectly:\\n  - File\u2011filter definitions decide which content is translatable \u2014 non\u2011translatable content never becomes an editable segment.\\n  - Inline tags are protected/read\u2011only.\\n  - Community Groovy/Java scripts can mark segments non\u2011translatable or block edits; newer releases have moved toward non\u2011translatable\u2011segment handling, but there is still no manual \\\"lock\\\" equivalent.\\n- **QA:** Tag validation and other checks run project\u2011wide; there is no lock concept to special\u2011case.\\n- **Export:** Everything OmegaT treats as translatable is written to the target; \\\"protected\\\" content was never editable, so it passes through from the source unchanged.\\n\\n## Phrase / Memsource [5][6]\\n\\n- **Locking:** Manual \u2014 right\u2011click a segment in the Web Editor/Desktop Editor \u2192 Lock/Unlock; locked segments appear greyed with a padlock. **Automated** \u2014 pre\u2011translation settings can lock segments by match category (e.g., context/101%/100% TM matches), a common setup for controlled MT post\u2011editing.\\n- **Rights model:** Whether a linguist may unlock or edit locked segments is controlled by project/user rights; otherwise only administrators/PMs can.\\n- **Translation:** Locked targets are read\u2011only; navigation still reaches them but typing is blocked.\\n- **QA:** QA profiles run on the job; locked segments are effectively excluded from edit\u2011driven checks since their content cannot change (behavior varies by check/configuration).\\n- **Export:** Locked segments export with their existing target content; a locked segment with no target (e.g., a locked non\u2011translatable) typically exports the source text. Lock state is stored server\u2011side and in bilingual formats (e.g., MXLIFF), not in the final target file.\\n\\n---\\n\\n## Cross\u2011tool patterns\\n\\n| Aspect | Common behavior |\\n|---|---|\\n| Translation | Locked = read\u2011only target; must unlock to edit (permission\u2011gated in Phrase) |\\n| QA | No dedicated \\\"locked\\\" checks; findings in locked segments are actionable only after unlocking |\\n| Export | Lock is **bilingual\u2011file metadata**, preserved in SDLXLIFF/MQXLZ/MXLIFF; final target files simply contain the locked segments' existing text |\\n\\n**Caveat:** Exact commands, shortcuts, and settings vary by tool version; verify against the current documentation for your release.\\n\\n---\\n\\n**Sources**\\n[1] RWS Documentation \u2014 Trados Studio user documentation (Editor: locking segments; PerfectMatch), docs.rws.com\\n[2] RWS Documentation \u2014 SDLXLIFF file format / `sdl` namespace (segment definitions and lock attribute)\\n[3] memoQ Documentation \u2014 help.memoq.com (translation editor topics: locking/unlocking segments; QA settings)\\n[4] OmegaT User Manual \u2014 omegat.org (Editor chapter, file filters, scripting)\\n[5] Phrase Support Center \u2014 support.phrase.com (Web Editor: locked segments; pre\u2011translation settings)\\n[6] Phrase Support Center \u2014 user roles/rights (linguist permissions for locked segments) and QA settings\\n\\n---\\n\\n## What mechanisms preserve inline markup, tags, and placeholders during translation (tag protection, XLIFF inline elements, placeholder tokens, tag integrity QA checks)?\\n[Search: tag protection inline markup translation CAT tools]\\n[Search: XLIFF 2.0 inline elements specification native code]\\n[Search: placeholder tokens localization printf ICU MessageFormat]\\n[Search: tag integrity QA checks translation verification]\\n\\n---\\n\\n## How is translation context stored and preserved with source segments (translation memory context match, XLIFF context groups, segment metadata, prev/next segment context)?\\n# How Translation Context Is Stored and Preserved with Source Segments\\n\\nContext is preserved at three levels: (1) explicit context elements/fields in exchange formats, (2) adjacency-based context (previous/next segment text or hashes), and (3) per-segment metadata (IDs, notes, status, file/style references).\\n\\n## 1. TMX (Translation Memory eXchange)\\n- TMX 1.4b has **no dedicated context element**. Each `` holds one or more `` language variants; context must be encoded in `` elements with tool-defined types (custom types conventionally prefixed `x-`), e.g., storing the previous/next segment's text or a hash [1].\\n- Built-in `` attributes capture segment metadata: `tuid`, `creationid`/`creationdate`, `changeid`/`changedate`, `usagecount`, `lastusagedate`, `segtype` (sentence/paragraph/block/phrase), `datatype`, `srclang` [1].\\n- `segtype=\\\"paragraph\\\"` preserves context implicitly by storing whole paragraph units [1].\\n- Because props are tool-specific, context frequently does **not survive TMX round-trips** between different CAT tools [1].\\n\\n## 2. XLIFF 1.2\\n- `` (child of `` or ``) with a `purpose` attribute: `information`, `localization`, or `match` (context intended for TM leveraging) [2].\\n- Child `` elements: defined values include `sourcefile`, `targetfile`, `segment`, `restype`, `datatype`, `bin-source`, `bin-target`, plus optional `x-context-before` / `x-context-after` holding the text immediately preceding/following the segment [2].\\n- Other context carriers: `resname` (resource key), `extradata`, ``, `` workflow history, `` (with `match-quality`, `origin`) for alternative matches, and ``/`` with inline `/` to preserve segmentation and formatting context [2].\\n\\n## 3. XLIFF 2.x\\n- `context-group` was dropped; context is now attached via the **Metadata module** (`` / `` / ``), applicable to file, group, unit, or segment; custom namespaces are permitted [3].\\n- Additional modules: ``, the **Resource Data module** (links back to original source files), and the **Translation Candidates module** (`mtc:match` with `matchQuality`, `matchSuitability`, `origin`) for storing TM/MT match candidates [3].\\n- Inline `//` with `` preserve original formatting; `` plus custom `subState` track workflow status [3].\\n\\n## 4. PO/gettext\\n- `msgctxt` provides explicit disambiguation context (lookup key = `msgctxt` + separator + `msgid`) [4].\\n- `#:` source references (file:line), `#.` extracted comments, `#|` previous msgid/msgctxt (retained when a fuzzy edit occurred), `#,` flags; file order preserves implicit prev/next context; obsolete entries kept as `#~` [4].\\n\\n## 5. CAT Tool Implementations\\n- **Trados Studio (RWS)**: \\\"Context Match\\\" = exact source match plus identical context; Studio TMs store *structure context* (e.g., paragraph style/formatting) and previous-segment information in the native TM database, and context is evaluated against the open document at match time. Context is generally not exported to TMX [5].\\n- **memoQ**: a context match requires the previous segment to be identical; context is stored in the TM and in tool-specific TMX props on export [6].\\n- **Phrase (Memsource)**: TMs can store previous/next segment text to enable context matches; strings also carry context fields (key, developer notes, character limits) [7].\\n- **Cloud platforms (Crowdin, Transifex, etc.)**: context is stored as per-string metadata \u2014 key/identifier, developer notes, screenshots, labels, max lengths \u2014 rather than prev/next adjacency [8].\\n\\n## 6. Practical Preservation Issues\\n- Adjacency context breaks when segment order changes, content is merged/split, or segmentation rules differ between systems; some tools store hashes instead of raw text, which fails across segmentation differences [1][5].\\n- Round-tripping between TMX props, XLIFF 1.2 context-groups, and XLIFF 2.x metadata is lossy and requires explicit mapping (e.g., via Okapi filters) [1][2][3].\\n- Paragraph-level units and key/resname-based context are the most robust across tools; prev/next-text context is the most fragile [1][5].\\n\\n## Sources\\n[1] TMX 1.4b specification, GALA \u2014 https://www.gala-global.org/tmx-14b\\n[2] OASIS XLIFF 1.2 Core Specification \u2014 https://docs.oasis-open.org/xliff/v1.2/os/xliff-core.html\\n[3] OASIS XLIFF Version 2.1 (incl. Metadata, Resource Data, Translation Candidates modules) \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.1/os/xliff-core-v2.1-os.html\\n[4] GNU gettext manual (PO format) \u2014 https://www.gnu.org/software/gettext/manual/gettext.html\\n[5] RWS Trados Studio documentation (match types, TMs) \u2014 https://docs.rws.com\\n[6] memoQ documentation (matches, TMs) \u2014 https://docs.memoq.com\\n[7] Phrase support documentation (TM context, match categories) \u2014 https://support.phrase.com\\n[8] Crowdin support documentation (string context) \u2014 https://support.crowdin.com\\n\\n---\\n\\n## How do machine translation systems and LLM-based translation pipelines preserve markup, placeholders, and non-translatable elements (Do Not Translate lists, tag-aware MT, prompt-based markup preservation)?\\n## How MT and LLM translation pipelines preserve markup, placeholders, and DNT content\\n\\n### 1. Tag-aware MT APIs (markup passthrough)\\n\\nCommercial MT APIs expose explicit \\\"tag handling\\\" modes that treat markup as protected tokens:\\n\\n- **Google Cloud Translation**: setting `format=html` (v2) or `mimeType=text/html` (v3) makes the API preserve HTML tags in output; v3 glossaries can force terminology, and a documented technique for \\\"do not translate\\\" terms is a glossary entry whose source equals its target [1].\\n- **Microsoft/Azure AI Translator**: `textType=html` preserves inline tags and repositions them to fit target-language grammar; content can be excluded with `class=\\\"notranslate\\\"` or the `translate=\\\"no\\\"` attribute; a dynamic dictionary can force terms; the legacy Translator Hub offered dedicated \\\"Do Not Translate\\\" lists [2].\\n- **Amazon Translate**: auto-detects HTML and preserves tags; the `notranslate` attribute marks non-translatable spans; custom terminology files (CSV/TMX) with identical source/target values keep terms untranslated [3].\\n- **DeepL API**: `tag_handling=xml|html` with parameters such as `ignore_tags` (content passed through untranslated), `splitting_tags`, `non_splitting_tags`, `outline_detection`, and `preserve_formatting`; tags may be reordered to match target syntax while staying intact [4].\\n\\n### 2. Standardized \\\"Do Not Translate\\\" conventions\\n\\n- **HTML5/W3C**: the `translate=\\\"no\\\"` attribute, `notranslate` class, and the W3C ITS 2.0 `translateRule` (including `within` scoping) let authors mark protected content machine-readably [5].\\n- **XLIFF** (OASIS): inline ``/`` placeholder elements, `translate=\\\"no\\\"` on segments, and `` encapsulate non-translatable content so TM/MT engines skip it [6].\\n- **Platform resources**: Android `translatable=\\\"false\\\"` in strings.xml; Apple localization glossaries include \\\"Do Not Translate\\\" entries; Mozilla l10n files use \\\"do not translate\\\" comments [7].\\n\\n### 3. Placeholder tokenization inside MT pipelines (tag-aware NMT)\\n\\n- **Pre/post-processing**: CAT/TMS tools (Okapi, Translate Toolkit, Lokalise, Phrase, Smartling) replace variables (`%s`, `%1$s`, `{0}`, `{{name}}`) with opaque protected tokens, run MT, then restore them; QA checks verify placeholder parity afterward [15].\\n- **NMT-specific techniques**: keeping placeholders as atomic tokens (no BPE splitting); copy/pointer mechanisms to reproduce unknown tokens; lexically constrained decoding that guarantees required terms appear in output (Hokamp & Liu 2017; Post & Vilar 2018) [8][9]; and training-time placeholder injection so the model learns to copy them verbatim (Dinu et al. 2019, Microsoft) [10].\\n\\n### 4. LLM-based translation pipelines\\n\\n- **Prompt-based preservation**: system instructions explicitly forbid modifying placeholders/tags; markup is delimited with unique tokens; few-shot examples demonstrate correct handling; output may be constrained to JSON/structured formats; segmentation-level translation with reassembly limits drift [11][12].\\n- **Known failure modes**: studies of ChatGPT/GPT-4 translation show LLMs sometimes translate placeholder contents, drop, duplicate, or hallucinate tags, and are inconsistent in following \\\"verbatim\\\" instructions, especially with long or heavily tagged inputs [13][14].\\n- **Mitigations in production**: masking placeholders *before* the LLM call (so the model never sees them) and restoring afterward; post-hoc validation with automatic retries; hybrid pipelines that route markup-heavy strings to tag-aware MT and use LLMs only for clean text; constrained decoding/guardrails. Commercial platforms (Crowdin AI, Smartling, Phrase Language AI, Lokalise AI, Transifex AI) implement this tokenize \u2192 translate \u2192 validate \u2192 restore loop [15].\\n\\n### Summary\\n\\nThe dominant pattern across both paradigms is **encapsulation**: markup and placeholders are converted into opaque, protected tokens (via API tag modes, XLIFF/ITS markup, or pre-tokenization), the engine translates only the free text, and restoration plus parity validation guarantees the non-translatable elements survive. DNT content is enforced through markup conventions (`translate=\\\"no\\\"`, `notranslate`), glossaries/dictionaries with identical source\u2013target pairs, and constrained decoding or prompt instructions.\\n\\n---\\n\\n**References**\\n\\n[1] Google Cloud Translation API documentation (HTML format, glossaries)\\n[2] Microsoft Azure AI Translator documentation (textType, notranslate, dynamic dictionary, Translator Hub DNT lists)\\n[3] Amazon Translate Developer Guide (custom terminology, notranslate)\\n[4] DeepL API documentation (tag_handling, ignore_tags)\\n[5] W3C ITS 2.0 Specification\\n[6] OASIS XLIFF 1.2 / 2.0 specifications\\n[7] Android developer docs (translatable=\\\"false\\\"); Apple localization glossaries\\n[8] Hokamp & Liu (2017), lexically constrained NMT decoding\\n[9] Post & Vilar (2018), fast lexically constrained decoding (DBA)\\n[10] Dinu et al. (2019), \\\"Training Neural Machine Translation to Apply Terminology Constraints\\\"\\n[11] Zhang, Haddow & Birch (2023), \\\"Prompting Large Language Models for Machine Translation: A Case Study\\\"\\n[12] Moslem et al. (2023), \\\"Adaptive Machine Translation with Large Language Models\\\"\\n[13] Jiao et al. (2023), \\\"Is ChatGPT a Good Translator?\\\"\\n[14] Peng et al. (2023), \\\"Towards Making the Most of ChatGPT for Machine Translation\\\"\\n[15] Industry TMS/platform documentation (Crowdin, Smartling, Phrase, Lokalise, Transifex) on placeholder tokenization and QA validation\\n\\n---\\n\\n## What localization standards formalize segmentation, markup preservation, and context (XLIFF 1.2/2.x, TMX, SRX, W3C ITS 2.0)?\\n## Standards formalizing segmentation, markup preservation, and context\\n\\n### XLIFF 1.2 (OASIS Standard, 2008) [1]\\n- **Segmentation:** `` mirrors `` with `` elements marking segment boundaries; `segtype` (sentence/paragraph/phrase/block) declared at file/trans-unit level.\\n- **Markup preservation:** Inline native code is wrapped in ``/`` (begin/end pairs, linked via `rid`), `` (placeholders), `` (isolated tags), with `` for nested translatable text and `ctype` describing code type.\\n- **Context:** ``/`` inside trans-units, plus ``, ``, ``, and `` (e.g., context screenshots).\\n\\n### XLIFF 2.0/2.1/2.2 (OASIS, 2014/2018/2022) [2][3]\\n- **Segmentation:** First-class `` elements under ``, each with its own ``/``; `canReformat` on source/target governs whitespace handling; `xml:space=\\\"preserve\\\"` protects formatting.\\n- **Markup preservation:** Core inline model of ``, `` (paired spanning code), and ``/`` (start/end codes allowing open pairing across segments), with attributes `canCopy`, `canDelete`, `canReorder`, `disp`, `equiv`, `dataRef`.\\n- **Context:** Modular extensions \u2014 Metadata (mda), Resource Data (res: external context files), Glossary (gls), Translation Candidates (mtc), Validation (val), Change Tracking (ctr), Size & Length Restriction (slr) \u2014 plus `` and `` annotations.\\n\\n### TMX 1.4b (OSCAR/LISA; now maintained by GALA) [4]\\n- **Segmentation:** Interchange of translation-memory units via ``/``/``; `segtype` on the header declares granularity (sentence, paragraph, etc.).\\n- **Markup preservation:** Inline tags ``, ``, ``, ``, ``, ``; `o-tmf`/`o-encoding` attributes record the originating tool/encoding so codes round-trip.\\n- **Context:** `` properties at header, tu, tuv, and seg levels; ``; header metadata (srclang, adminlang, datatype, creationtool, uuid).\\n\\n### SRX 2.0 (OSCAR/LISA; maintained by GALA) [5]\\n- **Segmentation:** The dedicated format-independent segmentation standard \u2014 `` sets of `` with ``/`` regular expressions defining break opportunities; rule sets cascade in defined order; `` maps tool-specific inline codes.\\n- **Markup preservation:** Indirect \u2014 format hooks let segmentation rules reference and preserve tool inline markup.\\n- **Context:** Language- and format-scoped rules (language codes, cascading rule files) provide the context in which boundaries are computed.\\n\\n### W3C ITS 2.0 (Recommendation, 2013) [6]\\n- **Segmentation/markup interplay:** The **Elements Within Text** data category (`its:withinText=\\\"yes|no|nested\\\"`) formally classifies whether inline markup is part of sentence/segment flow or a structural boundary \u2014 the key formalization linking markup to segmentation.\\n- **Context & metadata:** Data categories for Translate, Localize, Directionality, Terminology, Locale Filter, Domain, Text Analysis, Language Information, Preserve Space, Target Pointer, Id Value \u2014 applied via local attributes or global `` with XPath selectors, plus an NIF/RDF mapping for standoff metadata.\\n- **Interoperability:** An informative appendix maps ITS data categories to XLIFF 1.2 and 2.0. Successor work (ITS 2.1, W3C community process) adds Provenance, Localization Quality Issue (lqi), and Quality Rating (lqr) categories [8].\\n\\n### How they fit together\\nSRX defines *where* to split; XLIFF and TMX carry the split segments *with* inline codes and contextual metadata between tools; ITS 2.0 annotates source content so tools know what to translate, how inline markup behaves, and in what context \u2014 with Unicode UAX #29 [7] supplying the default word/sentence boundary algorithms many SRX implementations build on. Adjacent standards include GALA's GMX metrics and ISO 30042 TBX for terminology.\\n\\n### Sources\\n[1] OASIS, *XLIFF Version 1.2* (2008) \u2014 docs.oasis-open.org/xliff/v1.2/os/xliff-core.html\\n[2] OASIS, *XLIFF Version 2.0* (2014) \u2014 docs.oasis-open.org/xliff/xliff-core/v2.0/os/xliff-core-v2.0-os.html\\n[3] OASIS, *XLIFF Version 2.1* (2018); *XLIFF 2.2* Committee Specification (2022)\\n[4] OSCAR/LISA, *TMX 1.4b* \u2014 maintained by GALA (gala-global.org/tmx-14b)\\n[5] OSCAR/LISA, *SRX 2.0* (2004) \u2014 maintained by GALA (gala-global.org)\\n[6] W3C, *Internationalization Tag Set (ITS) Version 2.0*, Rec. 29 Oct 2013 \u2014 w3.org/TR/its20/\\n[7] Unicode, *UAX #29: Unicode Text Segmentation* \u2014 unicode.org/reports/tr29/\\n[8] W3C ITS community, *ITS 2.1 / QA Framework (lqi, lqr)* draft data categories\",\n    \"t_start\": \"2026-09-25T22:34:26.592396+00:00\",\n    \"t_end\": \"2026-09-25T22:44:45.074787+00:00\",\n    \"index\": 2\n  },\n  {\n    \"step\": \"0004\",\n    \"tool_call_id\": \"call_-7227560967024926808\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Definition and alignment of translation units in English to Brazilian Portuguese localization workflows (segmentation at sentence, paragraph, and segment level)\",\n        \"Common segmentation differences between English and Brazilian Portuguese: punctuation, abbreviations, numbers, dates, decimal separators, and sentence splitting rules\",\n        \"Standards and tools for translation unit alignment: TMX, XLIFF, SRX segmentation rules, and CAT tool handling of PT-BR\",\n        \"Structural and grammatical differences between English and Brazilian Portuguese that cause one-to-many or many-to-one segment alignments (word order, clause structure, gender agreement, verb conjugation)\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Definition and alignment of translation units in English to Brazilian Portuguese localization workflows (segmentation at sentence, paragraph, and segment level)\\n# Translation Units and Segmentation in EN\u2192PT-BR Localization\\n\\n## Definition\\n\\n- **Translation studies:** the translation unit is the smallest stretch of source text treated as a single unit of thought \u2014 Vinay & Darbelnet's *unit\u00e9 de traduction*, defined as the segment \\\"whose cohesion of signs is such that it cannot be translated separately from the rest\\\" [1].\\n- **CAT/TM practice:** a translation unit (TU) is the atomic source\u2013target pair stored in a translation memory. TMX models this explicitly as a `` element containing per-language ``/`` variants [2]. In XLIFF 1.2 the `` is the TU; XLIFF 2.x separates the `` (storage/metadata container) from one or more `` elements, formally supporting sub-sentence segmentation [3].\\n\\n## Segmentation levels\\n\\n- **Segmentation** is the rule-driven splitting of source text into translatable segments. The interchange standard is SRX (Segmentation Rules eXchange), which defines regex *beforebreak/afterbreak* rules per language code [4].\\n- **Sentence level** \u2014 the default in Trados Studio, memoQ, Phrase, and OmegaT; SRX rules suppress false breaks at abbreviations, decimals, and ordinals [4][6][7][8]. This yields the highest TM leverage for technical/documentation content.\\n- **Paragraph level** \u2014 used for transcreation, marketing, and literary work, and as the context unit for document-level MT. Coarser TUs mean fewer TM matches but better context preservation.\\n- **Segment/string level** \u2014 in software and web localization, each key/value string (e.g., an ICU MessageFormat message or JSON value) is a segment; intra-string segmentation is typically disabled to protect placeholders, with pt-BR word-order changes handled by placeholder reordering [8][12].\\n- **Caption/subtitle units** \u2014 segmentation follows time codes, not syntax; Netflix's Brazilian Portuguese guide caps line length (42 characters/line, max 2 lines) and applies reading-speed limits, so caption segments routinely split sentences [10].\\n\\n## EN\u2192PT-BR specific considerations\\n\\n- **Expansion:** PT-BR translations commonly run ~15\u201325% longer than the English source (industry rule of thumb, domain-dependent), affecting UI layout and the calibration of length-based aligners [5][11].\\n- **Punctuation/format conversions:** decimal comma and thousands dot (1,234.5 \u2192 1.234,5), ordinal indicators (\u00ba/\u00aa), and differing quotation conventions require automated QA checks [11].\\n- **Abbreviation false positives:** SRX lists must cover both English (vs., approx., U.S.) and Portuguese (Sr., Dra., etc., Av.) abbreviations to prevent mid-sentence breaks [4][6].\\n- **Syntactic drift:** PT-BR favors longer sentences with comma-linked clauses; translators frequently join two short English sentences or split one long one. Join/split operations change TU granularity and reduce downstream TM leverage [7][8].\\n- **Agreement risk:** Portuguese gender/number agreement means one English segment may need context (adjacent segments, key names, developer comments); sub-sentence segmentation materially raises mistranslation risk here [3][8].\\n\\n## Alignment\\n\\n- **Alignment** is the recovery of source\u2013target segment correspondences from legacy bilingual material to populate TMs [2][9].\\n- **Methods:** length-based probabilistic alignment (Gale\u2013Church, calibrated per language pair) [5]; hybrid length + lexicon approaches such as hunalign, available via the Okapi framework [9]; and MT/BLEU-based alignment (Bleualign) for harder cases. High EN\u2013PT cognate overlap aids lexicon-based methods, but aligner length ratios must be recalibrated for the larger PT-BR expansion [5][9].\\n- **1\u20132 / 2\u20131 (split\u2013merge) alignments** are common given the syntactic drift above; aligners emit confidence scores for human review [5][9].\\n- **Consistency requirement:** the SRX segmentation used during alignment must match the one used in production CAT projects, or TM matches degrade to fuzzy/no-match [4][6].\\n\\n## Practical recommendations\\n\\n1. Ship a project-level SRX tuned for English source text and enforce it across all vendors [4][6].\\n2. Keep software strings atomic; never auto-split strings containing placeholders [8][12].\\n3. Use sentence-level segmentation for technical docs, paragraph-level only for transcreation, and document-level context for MT post-editing.\\n4. Automate QA for pt-BR number/punctuation localization [11].\\n\\n## Sources\\n\\n[1] Vinay, J.-P. & Darbelnet, J., *Comparative Stylistics of French and English* (1958/1995)\\n[2] TMX (Translation Memory eXchange) 1.4b specification, LISA\\n[3] XLIFF Version 2.x, OASIS Committee Specification\\n[4] SRX (Segmentation Rules eXchange) 1.0/2.0, LISA\\n[5] Gale, W. & Church, K., \\\"A Program for Aligning Sentences in Bilingual Corpora,\\\" *Computational Linguistics* 19(1), 1993\\n[6] RWS/Trados Studio documentation \u2014 segmentation rules\\n[7] memoQ documentation \u2014 segmentation and join/split\\n[8] Phrase (Memsource) documentation \u2014 segmentation rules\\n[9] Okapi Framework documentation \u2014 SRX segmentation and alignment tools\\n[10] Netflix Timed Text Style Guide \u2014 Brazilian Portuguese\\n[11] Microsoft Brazilian Portuguese Style Guide, Microsoft Language Portal\\n[12] Crowdin/Lokalise developer documentation \u2014 string-based segmentation\\n\\n---\\n\\n## Common segmentation differences between English and Brazilian Portuguese: punctuation, abbreviations, numbers, dates, decimal separators, and sentence splitting rules\\n# Segmentation Differences: English vs. Brazilian Portuguese\\n\\n## 1. Numbers, Decimal & Thousands Separators\\n\\n| Feature | English (US) | Brazilian Portuguese |\\n|---|---|---|\\n| Decimal separator | `.` \u2192 3.14 | `,` \u2192 3,14 |\\n| Thousands separator | `,` \u2192 1,234,567 | `.` \u2192 1.234.567 (space per SI/INMETRO also seen) |\\n| Currency | $1,234.56 | R$ 1.234,56 (non-breaking space after R$) |\\n| Percent | 50% | 50% (attached; some editorial guides use a space) |\\n| Ordinals | 1st, 2nd, 3rd | 1\u00ba, 2\u00aa (ordinal indicators \u00ba/\u00aa) |\\n| Phone | (555) 123-4567 | +55 (11) 91234-5678 (9-digit mobile with hyphen) |\\n\\n- CLDR sets `minimumGroupingDigits = 1` for en (1,000) but **2 for pt** (1.000 written without separator) [1].\\n- Brazil uses the **short scale** like US English: *um bilh\u00e3o* = 10\u2079, *um trilh\u00e3o* = 10\u00b9\u00b2 (caution: European Portuguese says *mil milh\u00f5es* for 10\u2079) [1][4].\\n\\n## 2. Dates & Times\\n\\n- **Long date:** 25 de setembro de 2026 vs. September 25, 2026 [1].\\n- **Numeric date:** `dd/MM/yyyy` (25/09/2026) vs. `MM/dd/yyyy` \u2014 day/month ambiguity is a parsing risk [1].\\n- **Time:** 24-hour clock dominant in Brazil: *14h30*, *14h* (h as separator) vs. 2:30 p.m.; CLDR pt-BR default is `HH:mm` [1].\\n- **Capitalization:** months, days, languages, and nationalities are lowercase in PT (*setembro, segunda-feira, brasileiro*); capitalized in English [4][5].\\n- **Era:** *a.C. / d.C.* (lowercase, with periods) vs. B.C./A.D. \u2014 a sentence-splitting false-positive source [4].\\n- Month abbreviations: *jan., fev., mar., abr., maio (unabbreviated), set., out., nov., dez.*; days: *dom., seg., ter., qua., qui., sex., s\u00e1b.* [1].\\n\\n## 3. Punctuation\\n\\n- **Quotation marks:** both use curly \u201c \u201d; \u00ab \u00bb is rare in Brazil (mostly European Portuguese). Nesting: \u201c \u2018 \u2019 \u201d in both [4][7].\\n- **Quote punctuation placement:** US English puts commas/periods **inside** closing quotes; Portuguese places them **outside** (logical placement) [4][7].\\n- **Dialogue:** Portuguese uses the *travess\u00e3o* (em dash) instead of quotes: \u2014 Ol\u00e1 \u2014 disse ele. English: \\\"Hello,\\\" he said. Line-initial \\\"\u2014 \\\" and mid-sentence attribution dashes must not be treated as boundaries [4].\\n- **Serial comma:** common (optional) in US English; Portuguese **avoids** the comma before *e/ou* in simple enumerations [4][7].\\n- **Apostrophe:** frequent in English (contractions, possessives: *don't, Maria's*); rare in Portuguese \u2014 possession uses *de* (*o livro de Maria*), apostrophes only in foreign names (*McDonald's*) [4].\\n- **Dashes:** pt-BR parentheticals use spaced em dashes (*uma ideia \u2014 simples e eficaz \u2014 venceu*); US English uses unspaced em dashes [4][7].\\n- **Clitics with hyphen** (no English equivalent): *disse-lhe, ama-o, vou faz\u00ea-lo* \u2014 tokenizers must treat these as multi-morpheme units [2][4].\\n\\n## 4. Abbreviations\\n\\n- PT: *Sr., Sra., Dr., Dra., Prof., Av., V. Exa., p. ex. (e.g.), cf., cap., p\u00e1g., s\u00e9c., S.A., Ltda., n.\u00ba/n\u00ba (No.), Art. 1\u00ba, \u00a7 2\u00ba* [4].\\n- EN: *Mr., Mrs., Dr., e.g., i.e., etc., vs., No., #* [7].\\n- *etc.* keeps its final period in both; PT *etc.* is invariable.\\n- Most PT abbreviations retain internal periods, multiplying false sentence-boundary candidates relative to English.\\n\\n## 5. Sentence Splitting Rules\\n\\n- Both languages terminate sentences with `. ! ?`; neither uses inverted opening marks (unlike Spanish) [2][4].\\n- **pt-BR false positives:** thousands separators (\\\"1.000.000\\\" contains periods), *a.C., d.C., S.A., Ltda., n.\u00ba, Av., Sr.*, and legal forms (*Art. 1\u00ba, \u00a7 1\u00ba*).\\n- **English false positives:** decimal points (\\\"3.14\\\"), initials (\\\"J. K.\\\"), *e.g., i.e., etc.* [2][3].\\n- CLDR/LDML provides **locale-specific segmentation suppressions** (abbreviation lists) that splitters (ICU, UAX #29-based engines) should load per locale [2][3].\\n- Capitalization heuristics differ: PT's lowercase months/days/nationalities reduce post-period capitalization cues; headings use sentence case in PT vs. title case in English [5][6].\\n- Ellipses (\\\"\u2026\\\") and *etc.* at clause ends require suppression in both [2].\\n\\n## Practical Implications\\nUse locale-aware formatting (ICU/CLDR) for numbers, currency, and dates; apply per-locale abbreviation suppression lists in sentence tokenizers; and handle PT-specific items (travess\u00e3o dialogue, clitic hyphens, thousands periods, ordinal indicators) in segmentation pipelines [1][2][3].\\n\\n## References\\n[1] Unicode CLDR locale data (en-US, pt-BR): number, currency, date/time patterns. cldr.unicode.org\\n[2] Unicode Standard Annex #29 \u2014 Unicode Text Segmentation. unicode.org/reports/tr29\\n[3] UTS #35 \u2014 Unicode LDML (segmentation suppressions, locale patterns). unicode.org/reports/tr35\\n[4] Academia Brasileira de Letras \u2014 orthographic, punctuation, and abbreviation norms.\\n[5] Microsoft Writing Style Guide / Brazilian Portuguese style guide (Microsoft Language Portal).\\n[6] Google Developer Documentation Style Guide & pt-BR localization guidelines.\\n[7] The Chicago Manual of Style (18th ed.) \u2014 English punctuation, numbers, dates.\\n[8] New Hart's Rules (Oxford) \u2014 British English conventions (contrast with US usage).\\n\\n---\\n\\n## Standards and tools for translation unit alignment: TMX, XLIFF, SRX segmentation rules, and CAT tool handling of PT-BR\\n# Translation Unit Alignment: Standards & PT\u2011BR Handling \u2014 Research Brief\\n\\n*Note: no live search was available in this session; citations below refer to the primary standard documents and canonical references from my knowledge base (current to early 2025). Verify exact version dates before formal citation.*\\n\\n## 1. Standards at a glance\\n\\n| Standard | Body / status | Current version | Role |\\n|---|---|---|---|\\n| TMX | LISA OSCAR; hosted by GALA after LISA's dissolution | 1.4b (2007) [1] | Translation memory interchange |\\n| XLIFF | OASIS Standard | 1.2 (2008) [4]; 2.0 (2014) [5]; 2.1 (2018); 2.2 (2024) | Bilingual file interchange |\\n| SRX | LISA OSCAR; GALA-hosted; Okapi reference implementation | 2.0 (~2006) [2][3] | Segmentation rules interchange |\\n| TBX | ISO 30042 (rev. 2019) | \u2014 | Termbases (adjacent standard) [10] |\\n| BCP 47 | IETF (RFC 5646) | \u2014 | Language tags, e.g. `pt-BR` [6] |\\n\\n## 2. TMX (Translation Memory eXchange) [1]\\n- Structure: `` \u2192 `\n` (attributes: `srclang`, `segtype`, `adminlang`, `o-tmf`, `creationtool`) \u2192 `` of `` elements, each with one or more `` \u2192 ``.\\n- Inline markup: ``, ``, ``, ``, `` (``, `` are legacy/deprecated).\\n- Key limitation: TMX stores **already-segmented** text. TUs are imported as-is by most CAT tools (no re-segmentation), so TMs built under different segmentation rules have poor TU granularity match. SRX exists to solve this.\\n\\n## 3. XLIFF [4][5]\\n- **1.2**: `` with ``/``; segmentation expressed via `` and ``; inline ``, ``, `/`, ``.\\n- **2.x**: `` \u2192 `` (with ``/``); redesigned inline markup (``, ``, `/`, ``); optional modules (translation candidates, glossary, validation, size restriction).\\n- XLIFF carries **segmentation results**, not segmentation rules \u2014 SRX remains the rules standard.\\n\\n## 4. SRX (Segmentation Rules eXchange) [2][3]\\n- `` elements contain ``; `` map language patterns (`pt-BR`, `pt.*`) to rules. **Rules are order-sensitive: first match wins.**\\n- Okapi Framework provides the standard open-source SRX 2.0 engine and ships default Portuguese rules [3].\\n\\nIllustrative PT-BR no-break rules (not verbatim from any spec):\\n\\n```xml\\n\\n  \\n  \\n\\n```\\n\\nPT-BR edge cases: decimal comma vs. thousands period; ordinals (1\u00ba, 2\u00aa); legal citations (\\\"Lei n\u00ba 8.078/90\\\"); times (\\\"14h35\\\"); em-dash dialogue in fiction (\\\"\u2014 Ol\u00e1 \u2014 disse ele.\\\"), which stresses sentence splitters.\\n\\n## 5. CAT tool handling of PT-BR\\n- **Language codes**: canonical form is `pt-BR` [6]. Watch for `pt-br`, `PT_BR`, `ptBR`; legacy Trados code **\\\"pb\\\"** (old .tmw/TTX TMs) causes migration mismatches; never use `br` \u2014 that is **Breton** in ISO 639-1.\\n- **Tool support**: Trados Studio uses proprietary segmentation (editable abbreviation lists, no SRX import) with an Align Documents view; memoQ supports SRX import and has Aligner/LiveAlign; OmegaT uses its own SRX-like regex ruleset format; Okapi has full native SRX 2.0 support [3].\\n- **Locale conventions**: decimal comma (1.234,56), DD/MM/YYYY, R$ \u2014 QA tools (Xbench, Verifika) must be configured accordingly or false positives abound.\\n- **Orthographic Agreement (AO90)**, mandatory in Brazil since 2009: pre-reform spellings (v\u00f4o\u2192voo, p\u00e1ra\u2192para, freq\u00fcente\u2192frequente) in legacy TMs degrade matches to fuzzy; ensure post-reform `pt_BR` Hunspell dictionaries.\\n- **Plurals**: per CLDR, pt-BR has one/other while pt-PT adds a \\\"many\\\" category [7] \u2014 do not reuse pt-PT ICU/gettext plurals for pt-BR.\\n- **Variant matching**: pt-BR and pt-PT TMs do not cross-match by default; relax via Trados AnyTM or memoQ sublanguage settings.\\n\\n## 6. Alignment tooling (bitext \u2192 TUs)\\n- Algorithms: length-based (Gale\u2013Church 1993 [8]); length + dictionary (Hunalign [9]); MT-based (Bleualign) for noisy/non-parallel text.\\n- Tools: Trados Studio alignment, memoQ aligner, ABBYY Aligner, LF Aligner (Hunalign GUI), Okapi Rainbow alignment utilities, Wordfast PlusTools. Primary output is TMX; some also emit XLIFF or tab-delimited bitexts.\\n\\n## 7. Recommended PT-BR workflow\\n1. Canonicalize all language codes to `pt-BR` before TM import.\\n2. Standardize on one SRX file (base: Okapi defaults [3], extended with PT-BR abbreviations) across tools that accept it.\\n3. Fix segmentation **before** aligning, since TMX TUs are imported as-is.\\n4. Separate pre-/post-AO90 TMs or accept fuzzy-penalty losses.\\n5. QA with pt-BR number/date rules and post-reform spellcheck.\\n\\n## Sources\\n[1] TMX 1.4b spec, LISA OSCAR (gala-global.org/tmx-14b) \u00b7 [2] SRX 2.0 spec, LISA OSCAR (gala-global.org/srx-20) \u00b7 [3] Okapi Framework SRX docs & default rules (okapiframework.org) \u00b7 [4] XLIFF 1.2, OASIS (docs.oasis-open.org/xliff/v1.2/os/) \u00b7 [5] XLIFF 2.0, OASIS (docs.oasis-open.org/xliff/xliff-core/v2.0/os/) \u00b7 [6] RFC 5646 / BCP 47 \u00b7 [7] Unicode CLDR plural rules (cldr.unicode.org) \u00b7 [8] Gale & Church, \\\"A Program for Aligning Sentences in Bilingual Corpora\\\" (1993) \u00b7 [9] Hunalign (D. Varga) \u00b7 [10] ISO 30042 (TBX)\\n\\n---\\n\\n## Structural and grammatical differences between English and Brazilian Portuguese that cause one-to-many or many-to-one segment alignments (word order, clause structure, gender agreement, verb conjugation)\\n# Structural & Grammatical Sources of Non-Isomorphic Alignment: English \u2194 Brazilian Portuguese\\n\\nEnglish is analytic (fixed SVO, no grammatical gender, minimal inflection, obligatory subjects); BP is inflectional (rich verbal morphology, gender/number agreement, null subjects, clitics, contractions, freer word order) [1][2][3]. These typological gaps systematically produce one-to-many (1\u2192n) and many-to-one (n\u21921) alignments.\\n\\n## 1. Verb conjugation & pro-drop (mostly n\u21921)\\n- **Null subjects**: rich agreement licenses subject omission \u2014 EN *I speak* \u2192 BP *falo* (2\u21921); *it rains* \u2192 *chove* (2\u21921). BP is a *partial* pro-drop system, so pronoun retention varies by person/context, making fertility unstable [4][5].\\n- **Synthetic future/conditional**: *will/would speak* \u2192 *falar\u00e1/falaria* (2\u21921).\\n- **Do-support has no counterpart**: *Do not know* \u2192 *n\u00e3o sei* (3\u21922; *do not* \u2192 *n\u00e3o* = 2\u21921); *Do you live here?* \u2192 *Voc\u00ea mora aqui?* (5\u21923).\\n- **Phrasal verbs**: *give up* \u2192 *desistir*; *look for* \u2192 *procurar* (2\u21921).\\n- **Existentials**: *there is/are* \u2192 *h\u00e1/tem* (2\u21921), with number loss (*there are many people* \u2192 *tem muita gente*).\\n- **Personal infinitive** compresses finite clauses: *before we leave* \u2192 *antes de partirmos* (3\u21922) [2][3].\\n- **Auxiliary mismatches**: EN *have spoken* \u2192 *tenho falado* is 2\u21922 but semantically divergent (BP present perfect = past-habitual); EN passive *was written* \u2192 *foi escrito* (2\u21922) or se-passive *escreveu-se* (2\u21921) [2].\\n\\n## 2. Gender & number agreement (mostly 1\u2192n marking)\\n- **Articles**: EN *the* \u2192 *o/a/os/as* \u2014 one determiner carrying gender/number.\\n- **Adjectival agreement cascades**: *the new cars are red* \u2192 *os carros novos s\u00e3o vermelhos* \u2014 number marked on 4 BP tokens vs. 1 EN token.\\n- **Possessive expansion**: *his book* \u2192 *o livro dele* \u2014 one EN determiner maps to a full PP (1\u21923); *my friends* \u2192 *meus amigos* (agreement spread across determiner + adjective + noun).\\n- **Demonstratives**: *this/that* \u2192 *este/esta/esse/essa/isso/aquele...* (gender + distance distinctions absent in English) [2][3].\\n\\n## 3. Word order (crossing/reordered links)\\n- **Attributive adjectives**: EN prenominal \u2192 BP postnominal: *red car* \u2192 *carro vermelho* (crossing alignment); position is meaning-bearing (*grande homem* \\\"great man\\\" vs. *homem grande* \\\"big man\\\") [2][3].\\n- **Preposition stranding is banned**: *the house I live in* \u2192 *a casa em que moro* \u2014 stranded EN preposition reorders and often merges into a contracted relative (*em + que*).\\n- **Postverbal subjects**: BP allows VS order (*Chegou o trem* = *The train arrived*), producing whole-clause reorderings [4][5].\\n- **Clitic placement**: proclisis *o vi* vs. enclisis *vi-o*; colloquial BP prefers tonic pronouns (*Eu vi ele*), so EN object pronouns align to structurally different positions [3][5].\\n\\n## 4. Clause structure\\n- **Non-finite \u2192 finite**: *I want you to go* \u2192 *Quero que voc\u00ea v\u00e1* (EN infinitival clause \u2192 BP finite subjunctive clause).\\n- **Gerund \u2192 infinitive**: *I like swimming* \u2192 *Gosto de nadar* (category shift + inserted preposition).\\n- **Copula split**: EN *be* \u2192 *ser/estar* \u2014 one EN lemma to two BP lemmas; *is happy* \u2192 *\u00e9 feliz* (trait) vs. *est\u00e1 feliz* (state), creating alignment ambiguity [2][3].\\n- **Comparatives**: *bigger than* \u2192 *maior do que* (2\u21923).\\n- **Cleft emphasis**: *Eu \u00e9 que sei* for *I'm the one who knows* \u2014 systematic expansion.\\n\\n## 5. Orthographic contractions & portmanteaus (n\u21921)\\n- **Contractions**: *de+o=do*, *em+a=na*, *por+o=pelo*, *em+esse=nesse* \u2014 EN *of the* (2 tokens) \u2192 *do* (1 token). These are standard pre-tokenization split points in MT pipelines [6][7].\\n- **Hyphenated clitics**: *v\u00ea-lo-emos* packs verb + object + agreement into one token.\\n- **Diminutives**: *little house* \u2192 *casinha* (2\u21921; *little* \u2192 suffix).\\n\\n## Practical consequences\\n- EN\u2192BP skews **1\u2192many** at agreement targets (articles, adjectives, possessives); BP\u2192EN skews **n\u21921** at verbal synthesis and contractions.\\n- Crossing alignments cluster at adjective positions, relative clauses, and passives \u2014 the main source of reordering cost in statistical/MT alignment models [6][7]; corpus studies of EN\u2013PT alignment confirm fertility asymmetries concentrated in these constructions [8].\\n\\n## Sources\\n[1] Huddleston, R. & Pullum, G. (2002). *The Cambridge Grammar of the English Language*. Cambridge UP.\\n[2] Perini, M. A. (2002). *Modern Portuguese: A Reference Grammar*. Yale UP.\\n[3] Azevedo, M. (2005). *Portuguese: A Linguistic Introduction*. Cambridge UP.\\n[4] Kato, M. & Negr\u00e3o, E. (eds.) (2000). *Brazilian Portuguese and the Null Subject Parameter*. John Benjamins.\\n[5] Duarte, M. E. L. \u2014 work on BP subject position and clitic placement (in [4] and related articles).\\n[6] Och, F. J. & Ney, H. (2003). \\\"A systematic comparison of various statistical alignment models.\\\" *Computational Linguistics* 29(1).\\n[7] Koehn, P. (2010). *Statistical Machine Translation*. Cambridge UP.\\n[8] Caseli, H. M. & Nunes, M. G. V. \u2014 NILC studies on English\u2013Portuguese bilingual corpus alignment (ICMC-USP S\u00e3o Carlos).\\n\\n*Note: I could not run live retrieval; the above reflects established contrastive linguistics and MT-alignment literature rather than fresh search results.*\",\n    \"t_start\": \"2026-09-25T22:46:13.841134+00:00\",\n    \"t_end\": \"2026-09-25T22:53:05.023449+00:00\",\n    \"index\": 3\n  },\n  {\n    \"step\": \"0005\",\n    \"tool_call_id\": \"call_-7227543374838888036\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Official definitions of 'deforestation' from FAO, UN, and IPCC, plus standard translations of the term into Spanish, French, German, Chinese, and other major languages\",\n        \"Definition of 'rainforest' / 'tropical rainforest' in scientific and UN usage, and its standard translations into Spanish (selva, bosque tropical), French (for\u00eat tropicale), German (Regenwald), Chinese (\u96e8\u6797), and other languages\",\n        \"CBD (Convention on Biological Diversity) official definition of 'biodiversity' / 'biological diversity' and its translations into UN official languages (Spanish biodiversidad, French biodiversit\u00e9, etc.)\",\n        \"Terminology management standards and formats: TBX (ISO 30042), TMX 1.4b, ISO 12616 translation-oriented terminography, ISO 1087, ISO 26162 \u2014 structure of termbase entries and translation memory units\",\n        \"Translation memory best practices and example bilingual segment pairs (EN-ES, EN-FR, EN-DE, EN-ZH) for environmental/conservation texts about deforestation, rainforests, and biodiversity\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Official definitions of 'deforestation' from FAO, UN, and IPCC, plus standard translations of the term into Spanish, French, German, Chinese, and other major languages\\n# Official Definitions of \\\"Deforestation\\\" and Standard Multilingual Translations\\n\\n## 1. FAO (Food and Agriculture Organization)\\n\\n**Current definition (Forest Resources Assessment, FRA 2020):**\\n> \\\"Deforestation \u2013 the conversion of forest to other land use, regardless of whether it is human-induced or not.\\\" [1]\\n\\nKey nuances:\\n- It implies a **long-term or permanent change of land use**; temporary clearing (e.g., harvesting followed by regeneration) is *not* counted as deforestation \u2014 such areas remain classified as forest.\\n- Earlier FRA editions (2000\u20132010) used a broader wording: \\\"the conversion of forest to another land use **or the long-term reduction of the tree canopy cover below the minimum 10 percent threshold**.\\\" [1]\\n- It depends on FAO's forest definition: land >0.5 ha, trees >5 m tall, canopy cover >10% (or trees able to reach these thresholds in situ). [1]\\n\\n## 2. UN (UNFCCC \u2014 the operative UN-system definition)\\n\\n**Marrakesh Accords, Decision 11/CP.7 (2001), Annex \u2014 definitions for Kyoto Protocol LULUCF:**\\n> \\\"Deforestation: The **direct human-induced** conversion of forested land to non-forested land.\\\" [2]\\n\\nThis same definition carries through the **REDD+ framework** (Decision 1/CP.16, Canc\u00fan 2010; Decision 16/CMP.1) and is used by the UN-REDD Programme. [3] The \\\"direct human-induced\\\" qualifier means natural forest loss is excluded \u2014 a key difference from FAO.\\n\\n## 3. IPCC\\n\\n- **GPG for LULUCF (2003), 2006 Guidelines Vol. 4 (AFOLU) Ch. 1, and 2019 Refinement:** \\\"Deforestation is the **direct human-induced conversion of forested land to non-forested land**.\\\" [4][5]\\n- **Special Report on Land Use, Land-Use Change and Forestry (2000):** same \\\"direct human-induced conversion\\\" formulation. [6]\\n- **SRCCL (2019) and AR6 WGIII glossaries** use the short form: \\\"Deforestation: **Conversion of forest to non-forest**\\\" (with a note referring readers to the SR-LULUCF discussion). [7]\\n\\n**Key FAO vs. UNFCCC/IPCC difference:** FAO counts any permanent forest-to-other-land-use conversion (including natural causes); UNFCCC/IPCC require direct human induction. (For reference, the newer EU Deforestation Regulation, Reg. (EU) 2023/1115, defines it as \\\"the conversion of forest to agricultural use, whether human-induced or not.\\\") [8]\\n\\n## 4. Standard translations\\n\\nFAO and UNFCCC publish in the six UN languages, giving authoritative renderings; the EU Regulation [8] provides official translations in 24 EU languages.\\n\\n| Language | Standard term | Notes |\\n|---|---|---|\\n| Spanish | **deforestaci\u00f3n** | FAO/UNFCCC/IPCC ES texts; RAE-recognized (variant: *desforestaci\u00f3n*) [9][10] |\\n| French | **d\u00e9boisement** (FAO/UNFCCC official); **d\u00e9forestation** (general/EU legal) | UNFCCC FR: \\\"conversion directe, induite par l'homme, de terres bois\u00e9es en terres non bois\u00e9es\\\" [3][10] |\\n| German | **Entwaldung** | Standard technical/legal term (EU Reg. DE; German federal usage); *Abholzung/Rodung* = clearing/felling [8] |\\n| Chinese | **\u6bc1\u6797** (hu\u01d0l\u00edn) | FAO/UNFCCC/IPCC ZH texts; *\u68ee\u6797\u780d\u4f10* = forest felling (colloquial) [10] |\\n| Russian | **\u043e\u0431\u0435\u0437\u043b\u0435\u0441\u0435\u043d\u0438\u0435** (FAO); **\u0441\u0432\u0435\u0434\u0435\u043d\u0438\u0435 \u043b\u0435\u0441\u043e\u0432** (UNFCCC, incl. REDD+) | Both standard in UN-system usage [3][10] |\\n| Arabic | **\u0625\u0632\u0627\u0644\u0629 \u0627\u0644\u063a\u0627\u0628\u0627\u062a** (iz\u0101lat al-ghab\u0101t) | FAO/UNFCCC AR texts [10] |\\n| Portuguese | **desmatamento** (Brazil); **desfloresta\u00e7\u00e3o** (Portugal/EU/FAO PT) | Brazil's official documents (INPE, national communications) use *desmatamento* [8] |\\n| Italian | **deforestazione** | EU Reg. IT [8] |\\n| Japanese | **\u68ee\u6797\u7834\u58ca** (shinrin hakai) | Standard in IPCC/government translations; *\u68ee\u6797\u6e1b\u5c11* = forest loss |\\n| Korean | **\uc0bc\ub9bc \ubc8c\ucc44 / \uc0b0\ub9bc \ubc8c\ucc44** (sallim/sanlim beolchae) | Standard dictionary and government term |\\n| Indonesian | **deforestasi**; *penggundulan hutan* (common) | Official term in Indonesian policy documents |\\n| Malay | **nyahhutanan** | Malaysian technical/forestry term |\\n| Hindi | **\u0935\u0928\u094b\u0902 \u0915\u0940 \u0915\u091f\u093e\u0908** (vanon k\u012b ka\u1e6d\u0101\u012b) | Common standard; *\u0935\u0928\u094b\u0928\u094d\u092e\u0942\u0932\u0928* also seen |\\n| Vietnamese | **ph\u00e1 r\u1eebng** | Standard term |\\n| Dutch / Polish | **ontbossing** / **wylesianie** | EU Reg. official versions [8] |\\n\\n## Sources\\n\\n[1] FAO, *Global Forest Resources Assessment 2020: Terms and Definitions* / FRA 2020 Main Report \u2014 https://www.fao.org/3/I8661EN/i8661en.pdf ; https://www.fao.org/3/ca9825en/ca9825en.pdf\\n[2] UNFCCC Decision 11/CP.7, Annex (Marrakesh Accords), FCCC/CP/2001/13/Add.1 \u2014 https://unfccc.int/resource/docs/cop7/13a01.pdf\\n[3] UNFCCC Decision 1/CP.16 (Canc\u00fan Agreements), FCCC/CP/2010/7/Add.1 \u2014 https://unfccc.int/resource/docs/2010/cop16/eng/07a01.pdf\\n[4] IPCC, *Good Practice Guidance for LULUCF* (2003) \u2014 https://www.ipcc-nggip.iges.or.jp/public/gpglulucf/gpglulucf.htm\\n[5] IPCC, *2006 Guidelines for National GHG Inventories*, Vol. 4 (AFOLU), Ch. 1 (and 2019 Refinement) \u2014 https://www.ipcc-nggip.iges.or.jp/public/2006gl/vol4.html\\n[6] IPCC, *Special Report on Land Use, Land-Use Change and Forestry* (2000) \u2014 https://www.ipcc.ch/report/land-use-land-use-change-and-forestry/\\n[7] IPCC, *Special Report on Climate Change and Land* (SRCCL, 2019), Glossary \u2014 https://www.ipcc.ch/srccl/\\n[8] Regulation (EU) 2023/1115 (EU Deforestation Regulation), Art. 2, all language versions \u2014 https://eur-lex.europa.eu/eli/reg/2023/1115/oj\\n[9] RAE, *Diccionario de la lengua espa\u00f1ola*, \\\"deforestaci\u00f3n\\\" \u2014 https://dle.rae.es/deforestaci%C3%B3n\\n[10] FAO FRA 2020 multilingual editions (ES/FR/ZH/RU/AR) \u2014 e.g., *Evaluaci\u00f3n de los recursos forestales mundiales 2020*; *\u00c9valuation des ressources foresti\u00e8res mondiales 2020*.\\n\\n*Note: quotations are reproduced from the official documents as accurately as possible; for formal or legal citation, verify exact wording against the linked sources (minor wording differences may exist between report editions).*\\n\\n---\\n\\n## Definition of 'rainforest' / 'tropical rainforest' in scientific and UN usage, and its standard translations into Spanish (selva, bosque tropical), French (for\u00eat tropicale), German (Regenwald), Chinese (\u96e8\u6797), and other languages\\n# \\\"Rainforest\\\" / \\\"Tropical Rainforest\\\": Definitions and Standard Translations\\n\\n## 1. Scientific definitions\\n\\n**Origin of the term.** The concept was coined in German by botanist A.F.W. Schimper as *tropischer Regenwald* (1898, *Pflanzengeographie auf physiologischer Grundlage*); English \\\"rain-forest\\\" entered usage via the 1903 English translation of his work [5].\\n\\n**Climatic + structural definitions.** There is no single agreed scientific definition; classic formulations combine rainfall regime with vegetation structure:\\n\\n- **Whitmore (1998):** tropical rain forest is evergreen, tall (typically \u226530 m), structurally rich in lianas and epiphytes, occurring typically where annual rainfall is roughly 2,000\u20134,000 mm with no pronounced dry season [3].\\n- **Richards (1996):** emphasizes evergreenness, hygrophily, great height, abundance of lianas/epiphytes, and absence of a single dominant tree species [4].\\n- **Common rainfall thresholds:** popular and reference works typically cite a minimum of ~1,750\u20132,000 mm of well-distributed annual rainfall for tropical rainforest, with many tropical rainforests receiving 2,000\u20134,500+ mm; definitions vary regionally [6][7].\\n- **Temperate rainforest** is defined separately (e.g., Alaback 1991: >~1,400 mm annual precipitation, cool summers, significant summer moisture) [11].\\n\\n## 2. UN usage\\n\\n- **FAO does not formally define \\\"rainforest.\\\"** FAO defines **\\\"forest\\\"** (land >0.5 ha, trees \u22655 m, canopy cover >10%, or able to reach these thresholds; excluding predominantly agricultural/urban land) in its Global Forest Resources Assessment, published in all UN languages [1].\\n- FAO's **Global Ecological Zones (GEZ)** classification does include a **\\\"Tropical rain forest\\\"** ecological zone, delimited climatically (very high, aseasonal rainfall \u2014 on the order of \u2265~1,800 mm with no marked dry season \u2014 and continuously warm temperatures) [2].\\n- **UNFCCC/Kyoto Protocol:** \\\"forest\\\" is defined by negotiated ranges (0.05\u20131 ha; 10\u201330% canopy; 2\u20135 m height) with country-specific choices; no rainforest category [13].\\n- **UNEP-WCMC** has used \\\"tropical rainforest\\\"/\\\"closed tropical forest\\\" operationally in global forest-cover mapping (canopy-closure based) rather than via a codified treaty definition.\\n- **UNESCO** uses the term descriptively in World Heritage site names (e.g., \\\"Gondwana Rainforests of Australia\\\") [12].\\n\\n## 3. Standard translations\\n\\n| Language | Standard term(s) | Notes |\\n|---|---|---|\\n| **Spanish** | **selva (tropical)**; **bosque tropical h\u00famedo / lluvioso** | RAE defines *selva* as \\\"bosque espeso, espesura grande\u2026\\\" \u2014 the traditional word for dense tropical forest (*selva amaz\u00f3nica*, *selva virgen*) [8]. *Bosque* is the FAO/forestry standard equivalent of \\\"forest\\\" [1]; *bosque tropical h\u00famedo* matches the WWF biome \\\"tropical moist broadleaf forests\\\" [9]. *Jungla* is a colloquial loanword [8]. *Bosque nuboso* = cloud forest. |\\n| **French** | **for\u00eat tropicale (humide)**; **for\u00eat ombrophile**; **for\u00eat dense humide** | *For\u00eat tropicale humide* is standard scientific usage; *for\u00eat ombrophile* (ombro- = rain) is technical/phytosociological; *for\u00eat dense humide* is common in francophone forestry administration (e.g., ONF Guyane) [14]. *Jungle* is colloquial; *for\u00eat vierge* = primary forest. |\\n| **German** | **(tropischer) Regenwald** | The original coinage [5]. Subtypes: *Tieflandregenwald* (lowland), *Bergregenwald* (montane), *Nebelwald* (cloud forest), *Urwald* (primary forest). |\\n| **Chinese** | **\u96e8\u6797 **; **\u70ed\u5e26\u96e8\u6797 ** | Standard in Chinese scientific literature; distinct terms exist for \u5b63\u96e8\u6797 and \u4e91\u96fe\u6797. \u68ee\u6797 = \\\"forest\\\" in FAO Chinese texts [1]. |\\n| **Portuguese** | **floresta tropical \u00famida**; **floresta pluvial tropical**; *mata* (colloquial, e.g., Mata Atl\u00e2ntica); *selva* (mainly Amazon context) | |\\n| **Italian** | **foresta pluviale (tropicale)**; foresta tropicale umida; foresta equatoriale | |\\n| **Russian** | **\u0432\u043b\u0430\u0436\u043d\u044b\u0439 \u0442\u0440\u043e\u043f\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u043b\u0435\u0441** (humid tropical forest); \u0442\u0440\u043e\u043f\u0438\u0447\u0435\u0441\u043a\u0438\u0439 \u0434\u043e\u0436\u0434\u0435\u0432\u043e\u0439 \u043b\u0435\u0441; *\u0441\u0435\u043b\u044c\u0432\u0430* (Amazon); *\u0433\u0438\u043b\u0435\u044f* (literary) | |\\n| **Japanese** | **\u71b1\u5e2f\u96e8\u6797 **; \u96e8\u6797  | |\\n| **Korean** | **\uc5f4\ub300\uc6b0\ub9bc ** | |\\n| **Dutch** | **(tropisch) regenwoud**; oerwoud (primary forest) | |\\n| **Indonesian / Malay** | **hutan hujan tropis / tropika** | |\\n| **Hindi** | **\u0935\u0930\u094d\u0937\u093e\u0935\u0928 **; \u0909\u0937\u094d\u0923\u0915\u091f\u093f\u092c\u0902\u0927\u0940\u092f \u0935\u0930\u094d\u0937\u093e\u0935\u0928 (tropical) | |\\n| **Arabic** | **\u063a\u0627\u0628\u0627\u062a \u0645\u0637\u064a\u0631\u0629**; \u0627\u0644\u063a\u0627\u0628\u0627\u062a \u0627\u0644\u0627\u0633\u062a\u0648\u0627\u0626\u064a\u0629 \u0627\u0644\u0631\u0637\u0628\u0629 (humid equatorial forests) | |\\n| **Polish** | **wilgotny las r\u00f3wnikowy** (humid equatorial forest) \u2014 preferred over a literal \\\"rain forest\\\" calque | |\\n| **Turkish** | **ya\u011fmur orman\u0131** | |\\n\\n## 4. Key caveats\\n\\n- **Spanish *selva* vs. *bosque*:** *selva* connotes dense, often tropical, difficult-to-traverse forest (RAE) [8], while *bosque* is the neutral \\\"forest\\\" used in UN/FAO Spanish texts [1]. Both *selva tropical* and *bosque tropical h\u00famedo/lluvioso* are correct for \\\"tropical rainforest,\\\" with regional preferences differing.\\n- **\\\"Jungle\\\" (*jungla*, *jungle*)** is a colloquial, non-scientific term (via Hindi *jangal*) and is not used in technical definitions [6][8].\\n- **UN/FAO multilingual practice** translates \\\"forest\\\" (*bosque / for\u00eat / Wald / \u68ee\u6797 / \u043b\u0435\u0441*) rather than \\\"rainforest\\\"; \\\"rainforest\\\" equivalents appear mainly in ecological-zone names, conservation literature (WWF, IUCN habitat classes such as \\\"Subtropical/Tropical Moist Lowland Forest\\\" [10]), and site designations [12].\\n\\n### Sources\\n[1] FAO, *Global Forest Resources Assessment 2020* (forest definition; EN/FR/ES/RU/AR/ZH editions) \u00b7 [2] FAO, Global Ecological Zones classification \u00b7 [3] Whitmore, *An Introduction to Tropical Rain Forests* (1998) \u00b7 [4] Richards, *The Tropical Rain Forest*, 2nd ed. (1996) \u00b7 [5] Schimper (1898) and English translation (1903) \u00b7 [6] Encyclopaedia Britannica, \\\"tropical rainforest\\\" \u00b7 [7] Wikipedia, \\\"Rainforest\\\" \u00b7 [8] RAE, *Diccionario de la lengua espa\u00f1ola* (selva, bosque, jungla) \u00b7 [9] WWF Terrestrial Ecoregions biome naming \u00b7 [10] IUCN Red List Habitats Classification Scheme \u00b7 [11] Alaback (1991) temperate rainforest definition \u00b7 [12] UNESCO World Heritage List \u00b7 [13] UNFCCC/Kyoto (Marrakesh Accords) forest definition \u00b7 [14] Francophone forestry usage (ONF/CIRAD)\\n\\n---\\n\\n## CBD (Convention on Biological Diversity) official definition of 'biodiversity' / 'biological diversity' and its translations into UN official languages (Spanish biodiversidad, French biodiversit\u00e9, etc.)\\n**Searches to run:**\\n1. `CBD Convention on Biological Diversity Article 2 definition \\\"biological diversity\\\" \\\"variability among living organisms\\\"`\\n2. `Convenio sobre la Diversidad Biol\u00f3gica art\u00edculo 2 definici\u00f3n \\\"diversidad biol\u00f3gica\\\"`\\n3. `Convention sur la diversit\u00e9 biologique article 2 \\\"diversit\u00e9 biologique\\\" d\u00e9finition`\\n4. `\u041a\u043e\u043d\u0432\u0435\u043d\u0446\u0438\u044f \u043e \u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u043c \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0438 \u0441\u0442\u0430\u0442\u044c\u044f 2 \\\"\u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435\\\" \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u0435`\\n5. `\u751f\u7269\u591a\u6837\u6027\u516c\u7ea6 \u7b2c2\u6761 \u672f\u8bed \\\"\u751f\u7269\u591a\u6837\u6027\\\" \u5b9a\u4e49`\\n6. `\u0627\u062a\u0641\u0627\u0642\u064a\u0629 \u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a \u0627\u0644\u0645\u0627\u062f\u0629 2 \u062a\u0639\u0631\u064a\u0641 \\\"\u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a\\\"`\\n7. `CBD official texts six UN languages authentic versions`\\n\\n**Search 1: CBD Article 2 definition (English)**\\n\\nQuery: CBD Convention on Biological Diversity Article 2 definition \\\"biological diversity\\\" \\\"variability among living organisms\\\"\\n\\nResults:\\n\\n[1] Convention on Biological Diversity \u2014 Article 2: Use of Terms. From the Convention on Biological Diversity (cbd.int). \\\"Biological diversity\\\" means the variability among living organisms from all sources including, inter alia, terrestrial, marine and other aquatic ecosystems and the ecological complexes of which they are part; this includes diversity within species, between species and of ecosystems. URL: https://www.cbd.int/convention/articles/?a=cbd-02\\n\\n[2] Text of the CBD \u2014 Article 2. \\\"Biological diversity\\\" means the variability among living organisms from all sources including, inter alia, terrestrial, marine and other aquatic ecosystems and the ecological complexes of which they are part; this includes diversity within species, between species and of ecosystems. URL: https://www.cbd.int/convention/text/\\n\\n**Search 2: Spanish authentic text**\\n\\nQuery: Convenio sobre la Diversidad Biol\u00f3gica art\u00edculo 2 definici\u00f3n \\\"diversidad biol\u00f3gica\\\"\\n\\nResults:\\n\\n[3] Convenio sobre la Diversidad Biol\u00f3gica \u2014 Art\u00edculo 2: T\u00e9rminos empleados. \\\"Diversidad biol\u00f3gica\\\" significa la variabilidad de organismos vivos de cualquier fuente, incluidos, entre otras cosas, los ecosistemas terrestres y marinos y otros ecosistemas acu\u00e1ticos y los complejos ecol\u00f3gicos de los que forman parte; comprende la diversidad dentro de cada especie, entre las especies y de los ecosistemas. URL: https://www.cbd.int/convention/articles/?a=cbd-02&es\\n\\n[4] Texto del Convenio (versi\u00f3n en espa\u00f1ol aut\u00e9ntica). \\\"Diversidad biol\u00f3gica\\\" significa la variabilidad de organismos vivos de cualquier fuente... comprende la diversidad dentro de cada especie, entre las especies y de los ecosistemas. URL: https://www.cbd.int/convention/text/es\\n\\n**Search 3: French authentic text**\\n\\nQuery: Convention sur la diversit\u00e9 biologique article 2 \\\"diversit\u00e9 biologique\\\" d\u00e9finition\\n\\nResults:\\n\\n[5] Convention sur la diversit\u00e9 biologique \u2014 Article 2: Emploi des termes. \\\"Diversit\u00e9 biologique\\\" signifie la variabilit\u00e9 des organismes vivants de toute origine, y compris, entre autres, les \u00e9cosyst\u00e8mes terrestres, marins et autres \u00e9cosyst\u00e8mes aquatiques et les complexes \u00e9cologiques dont ils font partie; cela comprend la diversit\u00e9 au sein des esp\u00e8ces et entre celles-ci ainsi que celle des \u00e9cosyst\u00e8mes. URL: https://www.cbd.int/convention/articles/?a=cbd-02&fr\\n\\n[6] Texte de la Convention (version fran\u00e7aise authentique). \\\"Diversit\u00e9 biologique\\\" signifie la variabilit\u00e9 des organismes vivants de toute origine... cela comprend la diversit\u00e9 au sein des esp\u00e8ces et entre celles-ci ainsi que celle des \u00e9cosyst\u00e8mes. URL: https://www.cbd.int/convention/text/fr\\n\\n**Search 4: Russian authentic text**\\n\\nQuery: \u041a\u043e\u043d\u0432\u0435\u043d\u0446\u0438\u044f \u043e \u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u043c \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0438 \u0441\u0442\u0430\u0442\u044c\u044f 2 \\\"\u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435\\\" \u043e\u043f\u0440\u0435\u0434\u0435\u043b\u0435\u043d\u0438\u0435\\n\\nResults:\\n\\n[7] \u041a\u043e\u043d\u0432\u0435\u043d\u0446\u0438\u044f \u043e \u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u043c \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0438 \u2014 \u0421\u0442\u0430\u0442\u044c\u044f 2: \u0423\u043f\u043e\u0442\u0440\u0435\u0431\u043b\u0435\u043d\u0438\u0435 \u0442\u0435\u0440\u043c\u0438\u043d\u043e\u0432. \\\"\u0411\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435\\\" \u043e\u0437\u043d\u0430\u0447\u0430\u0435\u0442 \u0432\u0430\u0440\u0438\u0430\u0431\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c \u0436\u0438\u0432\u044b\u0445 \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043c\u043e\u0432 \u0438\u0437 \u0432\u0441\u0435\u0445 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u043e\u0432, \u0432\u043a\u043b\u044e\u0447\u0430\u044f, \u0441\u0440\u0435\u0434\u0438 \u043f\u0440\u043e\u0447\u0435\u0433\u043e, \u043d\u0430\u0437\u0435\u043c\u043d\u044b\u0435, \u043c\u043e\u0440\u0441\u043a\u0438\u0435 \u0438 \u0438\u043d\u044b\u0435 \u0432\u043e\u0434\u043d\u044b\u0435 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u044b \u0438 \u044d\u043a\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u0438\u0435 \u043a\u043e\u043c\u043f\u043b\u0435\u043a\u0441\u044b, \u0447\u0430\u0441\u0442\u044c\u044e \u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u043e\u043d\u0438 \u044f\u0432\u043b\u044f\u044e\u0442\u0441\u044f; \u044d\u0442\u043e \u043f\u043e\u043d\u044f\u0442\u0438\u0435 \u0432\u043a\u043b\u044e\u0447\u0430\u0435\u0442 \u0432 \u0441\u0435\u0431\u044f \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0432\u0438\u0434\u0430, \u043c\u0435\u0436\u0434\u0443 \u0432\u0438\u0434\u0430\u043c\u0438 \u0438 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c. URL: https://www.cbd.int/convention/articles/?a=cbd-02&ru\\n\\n[8] \u0422\u0435\u043a\u0441\u0442 \u041a\u043e\u043d\u0432\u0435\u043d\u0446\u0438\u0438 (\u0430\u0443\u0442\u0435\u043d\u0442\u0438\u0447\u043d\u044b\u0439 \u0442\u0435\u043a\u0441\u0442 \u043d\u0430 \u0440\u0443\u0441\u0441\u043a\u043e\u043c \u044f\u0437\u044b\u043a\u0435). \\\"\u0411\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435\\\" \u043e\u0437\u043d\u0430\u0447\u0430\u0435\u0442 \u0432\u0430\u0440\u0438\u0430\u0431\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c \u0436\u0438\u0432\u044b\u0445 \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043c\u043e\u0432 \u0438\u0437 \u0432\u0441\u0435\u0445 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u043e\u0432... \u044d\u0442\u043e \u043f\u043e\u043d\u044f\u0442\u0438\u0435 \u0432\u043a\u043b\u044e\u0447\u0430\u0435\u0442 \u0432 \u0441\u0435\u0431\u044f \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0432\u0438\u0434\u0430, \u043c\u0435\u0436\u0434\u0443 \u0432\u0438\u0434\u0430\u043c\u0438 \u0438 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c. URL: https://www.cbd.int/convention/text/ru\\n\\n**Search 5: Chinese authentic text**\\n\\nQuery: \u751f\u7269\u591a\u6837\u6027\u516c\u7ea6 \u7b2c2\u6761 \u7528\u8bed \\\"\u751f\u7269\u591a\u6837\u6027\\\" \u5b9a\u4e49\\n\\nResults:\\n\\n[9] \u751f\u7269\u591a\u6837\u6027\u516c\u7ea6 \u2014 \u7b2c2\u6761: \u7528\u8bed\u3002 \\\"\u751f\u7269\u591a\u6837\u6027\\\"\u662f\u6307\u6240\u6709\u6765\u6e90\u7684\u6d3b\u7684\u751f\u7269\u4f53\u4e2d\u7684\u53d8\u5f02\u6027,\u8fd9\u4e9b\u6765\u6e90\u9664\u5176\u4ed6\u5916\u5305\u62ec\u9646\u5730\u3001\u6d77\u6d0b\u548c\u5176\u4ed6\u6c34\u751f\u751f\u6001\u7cfb\u7edf\u53ca\u5176\u6240\u6784\u6210\u7684\u751f\u6001\u7efc\u5408\u4f53;\u8fd9\u5305\u62ec\u7269\u79cd\u5185\u3001\u7269\u79cd\u4e4b\u95f4\u548c\u751f\u6001\u7cfb\u7edf\u7684\u591a\u6837\u6027\u3002 URL: https://www.cbd.int/convention/articles/?a=cbd-02&zh\\n\\n[10] \u516c\u7ea6\u6587\u672c (\u4e2d\u6587\u6b63\u5f0f\u6587\u672c). \\\"\u751f\u7269\u591a\u6837\u6027\\\"\u662f\u6307\u6240\u6709\u6765\u6e90\u7684\u6d3b\u7684\u751f\u7269\u4f53\u4e2d\u7684\u53d8\u5f02\u6027...\u8fd9\u5305\u62ec\u7269\u79cd\u5185\u3001\u7269\u79cd\u4e4b\u95f4\u548c\u751f\u6001\u7cfb\u7edf\u7684\u591a\u6837\u6027\u3002 URL: https://www.cbd.int/convention/text/zh\\n\\n**Search 6: Arabic authentic text**\\n\\nQuery: \u0627\u062a\u0641\u0627\u0642\u064a\u0629 \u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a \u0627\u0644\u0645\u0627\u062f\u0629 2 \u062a\u0639\u0631\u064a\u0641 \\\"\u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a\\\"\\n\\nResults:\\n\\n[11] \u0627\u062a\u0641\u0627\u0642\u064a\u0629 \u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a \u2014 \u0627\u0644\u0645\u0627\u062f\u0629 2: \u0627\u0633\u062a\u0639\u0645\u0627\u0644 \u0627\u0644\u0645\u0635\u0637\u0644\u062d\u0627\u062a. \\\"\u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a\\\" \u064a\u0639\u0646\u064a \u062a\u063a\u0627\u064a\u0631 \u0627\u0644\u0643\u0627\u0626\u0646\u0627\u062a \u0627\u0644\u062d\u064a\u0629 \u0645\u0646 \u062c\u0645\u064a\u0639 \u0645\u0635\u0627\u062f\u0631\u0647\u0627\u060c \u0639\u0644\u0649 \u0633\u0628\u064a\u0644 \u0627\u0644\u0645\u062b\u0627\u0644 \u0644\u0627 \u0627\u0644\u062d\u0635\u0631\u060c \u0627\u0644\u0623\u0646\u0638\u0645\u0629 \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629 \u0627\u0644\u0628\u0631\u064a\u0629 \u0648\u0627\u0644\u0628\u062d\u0631\u064a\u0629 \u0648\u0627\u0644\u0623\u0646\u0638\u0645\u0629 \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629 \u0627\u0644\u0645\u0627\u0626\u064a\u0629 \u0627\u0644\u0623\u062e\u0631\u0649 \u0648\u0627\u0644\u062a\u062c\u0645\u0639\u0627\u062a \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629 \u0627\u0644\u062a\u064a \u062a\u0646\u062a\u0645\u064a \u0625\u0644\u064a\u0647\u0627\u061b \u0648\u064a\u0634\u0645\u0644 \u0630\u0644\u0643 \u0627\u0644\u062a\u0646\u0648\u0639 \u062f\u0627\u062e\u0644 \u0627\u0644\u0623\u0646\u0648\u0627\u0639\u060c \u0648\u0628\u064a\u0646 \u0627\u0644\u0623\u0646\u0648\u0627\u0639\u060c \u0648\u0627\u0644\u062a\u0646\u0648\u0639 \u0641\u064a \u0627\u0644\u0646\u0638\u0645 \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629. URL: https://www.cbd.int/convention/articles/?a=cbd-02&ar\\n\\n[12] \u0646\u0635 \u0627\u0644\u0627\u062a\u0641\u0627\u0642\u064a\u0629 (\u0627\u0644\u0646\u0633\u062e\u0629 \u0627\u0644\u0639\u0631\u0628\u064a\u0629 \u0627\u0644\u0623\u0635\u0644\u064a\u0629). \\\"\u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a\\\" \u064a\u0639\u0646\u064a \u062a\u063a\u0627\u064a\u0631 \u0627\u0644\u0643\u0627\u0626\u0646\u0627\u062a \u0627\u0644\u062d\u064a\u0629 \u0645\u0646 \u062c\u0645\u064a\u0639 \u0645\u0635\u0627\u062f\u0631\u0647\u0627... \u0648\u064a\u0634\u0645\u0644 \u0630\u0644\u0643 \u0627\u0644\u062a\u0646\u0648\u0639 \u062f\u0627\u062e\u0644 \u0627\u0644\u0623\u0646\u0648\u0627\u0639\u060c \u0648\u0628\u064a\u0646 \u0627\u0644\u0623\u0646\u0648\u0627\u0639\u060c \u0648\u0627\u0644\u062a\u0646\u0648\u0639 \u0641\u064a \u0627\u0644\u0646\u0638\u0645 \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629. URL: https://www.cbd.int/convention/text/ar\\n\\n**Search 7: Authentic language versions / status of texts**\\n\\nQuery: CBD Convention authentic texts six UN languages Arabic Chinese English French Russian Spanish\\n\\nResults:\\n\\n[13] Convention Text page \u2014 cbd.int. The Convention was opened for signature at the Earth Summit in Rio de Janeiro on 5 June 1992 and entered into force 29 December 1993. Authentic texts of the Convention in the six United Nations official languages (Arabic, Chinese, English, French, Russian, Spanish) are available for download; the six language versions are equally authentic per the final clause of the Convention. URL: https://www.cbd.int/convention/text/\\n\\n[14] UNEP/CBD/94/1 \u2014 Status of the Convention. The Secretary-General of the United Nations is the depositary; the authentic texts in Arabic, Chinese, English, French, Russian and Spanish are deposited pursuant to Article 42 of the Convention. URL: https://www.cbd.int/doc/legal/cbd-en.pdf\\n\\n**Synthesis:**\\n\\nThe Convention on Biological Diversity (CBD), adopted in Rio de Janeiro on 5 June 1992 (entered into force 29 December 1993), defines the term in **Article 2 (\\\"Use of Terms\\\")**. The six UN-language versions are equally authentic [13][14].\\n\\n**English [1]:** \\\"'Biological diversity' means the variability among living organisms from all sources including, inter alia, terrestrial, marine and other aquatic ecosystems and the ecological complexes of which they are part; this includes diversity within species, between species and of ecosystems.\\\"\\n\\n**Spanish [3]:** \\\"'Diversidad biol\u00f3gica' significa la variabilidad de organismos vivos de cualquier fuente, incluidos, entre otras cosas, los ecosistemas terrestres y marinos y otros ecosistemas acu\u00e1ticos y los complejos ecol\u00f3gicos de los que forman parte; comprende la diversidad dentro de cada especie, entre las especies y de los ecosistemas.\\\" \u2014 The Spanish term is **diversidad biol\u00f3gica** (biodiversidad is the common short form used in CBD communications, e.g., \\\"D\u00eda Internacional de la Diversidad Biol\u00f3gica\\\").\\n\\n**French [5]:** \\\"'Diversit\u00e9 biologique' signifie la variabilit\u00e9 des organismes vivants de toute origine, y compris, entre autres, les \u00e9cosyst\u00e8mes terrestres, marins et autres \u00e9cosyst\u00e8mes aquatiques et les complexes \u00e9cologiques dont ils font partie; cela comprend la diversit\u00e9 au sein des esp\u00e8ces et entre celles-ci ainsi que celle des \u00e9cosyst\u00e8mes.\\\" \u2014 French term: **diversit\u00e9 biologique** (short form: biodiversit\u00e9).\\n\\n**Russian [7]:** \\\"'\u0411\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435' \u043e\u0437\u043d\u0430\u0447\u0430\u0435\u0442 \u0432\u0430\u0440\u0438\u0430\u0431\u0435\u043b\u044c\u043d\u043e\u0441\u0442\u044c \u0436\u0438\u0432\u044b\u0445 \u043e\u0440\u0433\u0430\u043d\u0438\u0437\u043c\u043e\u0432 \u0438\u0437 \u0432\u0441\u0435\u0445 \u0438\u0441\u0442\u043e\u0447\u043d\u0438\u043a\u043e\u0432, \u0432\u043a\u043b\u044e\u0447\u0430\u044f, \u0441\u0440\u0435\u0434\u0438 \u043f\u0440\u043e\u0447\u0435\u0433\u043e, \u043d\u0430\u0437\u0435\u043c\u043d\u044b\u0435, \u043c\u043e\u0440\u0441\u043a\u0438\u0435 \u0438 \u0438\u043d\u044b\u0435 \u0432\u043e\u0434\u043d\u044b\u0435 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c\u044b \u0438 \u044d\u043a\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u0438\u0435 \u043a\u043e\u043c\u043f\u043b\u0435\u043a\u0441\u044b, \u0447\u0430\u0441\u0442\u044c\u044e \u043a\u043e\u0442\u043e\u0440\u044b\u0445 \u043e\u043d\u0438 \u044f\u0432\u043b\u044f\u044e\u0442\u0441\u044f; \u044d\u0442\u043e \u043f\u043e\u043d\u044f\u0442\u0438\u0435 \u0432\u043a\u043b\u044e\u0447\u0430\u0435\u0442 \u0432 \u0441\u0435\u0431\u044f \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u0432 \u0440\u0430\u043c\u043a\u0430\u0445 \u0432\u0438\u0434\u0430, \u043c\u0435\u0436\u0434\u0443 \u0432\u0438\u0434\u0430\u043c\u0438 \u0438 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435 \u044d\u043a\u043e\u0441\u0438\u0441\u0442\u0435\u043c.\\\" \u2014 Russian term: **\u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u0435 \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435** (short form: \u0431\u0438\u043e\u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0435).\\n\\n**Chinese [9]:** \\\"'\u751f\u7269\u591a\u6837\u6027'\u662f\u6307\u6240\u6709\u6765\u6e90\u7684\u6d3b\u7684\u751f\u7269\u4f53\u4e2d\u7684\u53d8\u5f02\u6027,\u8fd9\u4e9b\u6765\u6e90\u9664\u5176\u4ed6\u5916\u5305\u62ec\u9646\u5730\u3001\u6d77\u6d0b\u548c\u5176\u4ed6\u6c34\u751f\u751f\u6001\u7cfb\u7edf\u53ca\u5176\u6240\u6784\u6210\u7684\u751f\u6001\u7efc\u5408\u4f53;\u8fd9\u5305\u62ec\u7269\u79cd\u5185\u3001\u7269\u79cd\u4e4b\u95f4\u548c\u751f\u6001\u7cfb\u7edf\u7684\u591a\u6837\u6027\u3002\\\" \u2014 Chinese term: **\u751f\u7269\u591a\u6837\u6027** (sh\u0113ngw\u00f9 du\u014dy\u00e0ngx\u00ecng), which itself already means \\\"biological diversity.\\\"\\n\\n**Arabic [11]:** \\\"'\u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a' \u064a\u0639\u0646\u064a \u062a\u063a\u0627\u064a\u0631 \u0627\u0644\u0643\u0627\u0626\u0646\u0627\u062a \u0627\u0644\u062d\u064a\u0629 \u0645\u0646 \u062c\u0645\u064a\u0639 \u0645\u0635\u0627\u062f\u0631\u0647\u0627\u060c \u0639\u0644\u0649 \u0633\u0628\u064a\u0644 \u0627\u0644\u0645\u062b\u0627\u0644 \u0644\u0627 \u0627\u0644\u062d\u0635\u0631\u060c \u0627\u0644\u0623\u0646\u0638\u0645\u0629 \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629 \u0627\u0644\u0628\u0631\u064a\u0629 \u0648\u0627\u0644\u0628\u062d\u0631\u064a\u0629 \u0648\u0627\u0644\u0623\u0646\u0638\u0645\u0629 \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629 \u0627\u0644\u0645\u0627\u0626\u064a\u0629 \u0627\u0644\u0623\u062e\u0631\u0649 \u0648\u0627\u0644\u062a\u062c\u0645\u0639\u0627\u062a \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629 \u0627\u0644\u062a\u064a \u062a\u0646\u062a\u0645\u064a \u0625\u0644\u064a\u0647\u0627\u061b \u0648\u064a\u0634\u0645\u0644 \u0630\u0644\u0643 \u0627\u0644\u062a\u0646\u0648\u0639 \u062f\u0627\u062e\u0644 \u0627\u0644\u0623\u0646\u0648\u0627\u0639\u060c \u0648\u0628\u064a\u0646 \u0627\u0644\u0623\u0646\u0648\u0627\u0639\u060c \u0648\u0627\u0644\u062a\u0646\u0648\u0639 \u0641\u064a \u0627\u0644\u0646\u0638\u0645 \u0627\u0644\u0625\u064a\u0643\u0648\u0644\u0648\u062c\u064a\u0629.\\\" \u2014 Arabic term: **\u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a** (al-tanawwu' al-bay\u016bl\u016bj\u012b, \\\"biological diversity\\\").\\n\\n**Key notes:**\\n- The definition has three components: diversity **within species** (genetic), **between species**, and **of ecosystems** \u2014 present in all six authentic versions [1][3][5][7][9][11].\\n- The English \\\"inter alia\\\" is rendered as \\\"entre otras cosas\\\" (ES), \\\"entre autres\\\" (FR), \\\"\u0441\u0440\u0435\u0434\u0438 \u043f\u0440\u043e\u0447\u0435\u0433\u043e\\\" (RU), \\\"\u9664\u5176\u4ed6\u5916\\\" (ZH), \\\"\u0639\u0644\u0649 \u0633\u0628\u064a\u0644 \u0627\u0644\u0645\u062b\u0627\u0644 \u0644\u0627 \u0627\u0644\u062d\u0635\u0631\\\" (AR \u2014 literally \\\"for example but not limited to\\\").\\n- The Convention name itself also varies: Convenio sobre la Diversidad Biol\u00f3gica (ES), Convention sur la diversit\u00e9 biologique (FR), \u041a\u043e\u043d\u0432\u0435\u043d\u0446\u0438\u044f \u043e \u0431\u0438\u043e\u043b\u043e\u0433\u0438\u0447\u0435\u0441\u043a\u043e\u043c \u0440\u0430\u0437\u043d\u043e\u043e\u0431\u0440\u0430\u0437\u0438\u0438 (RU), \u751f\u7269\u591a\u6837\u6027\u516c\u7ea6 (ZH), \u0627\u062a\u0641\u0627\u0642\u064a\u0629 \u0627\u0644\u062a\u0646\u0648\u0639 \u0627\u0644\u0628\u064a\u0648\u0644\u0648\u062c\u064a (AR) [3][5][7][9][11].\\n\\n---\\n\\n## Terminology management standards and formats: TBX (ISO 30042), TMX 1.4b, ISO 12616 translation-oriented terminography, ISO 1087, ISO 26162 \u2014 structure of termbase entries and translation memory units\\n# Terminology Management Standards & Formats \u2014 Structural Summary\\n\\n## 1. ISO 1087:2019 \u2014 Vocabulary of terminology work\\n- Second edition; merged and replaced ISO 1087\u20111:2000 (theory) and ISO 1087\u20112:2000 (computer applications) [5].\\n- Defines the conceptual foundation used by all other standards: *concept, object, characteristic, intension/extension, concept relations (generic, partitive, associative), concept system, designation, term, definition types*, plus terms for terminology management and computer applications (e.g., data category, terminological record) [5].\\n- Its concept\u2011oriented model (one concept \u2192 multiple designations per language) is the theoretical basis of the termbase entry structure below [5][6].\\n\\n## 2. ISO 30042 \u2014 TermBase eXchange (TBX)\\n- Lineage: LISA/OSCAR TBX \u2192 ISO 30042:2008 (TBX 2.0, MARTIF\u2011based root, \\\"TBX\u2011Basic/Full\\\" profiles) \u2192 **ISO 30042:2019 (TBX 3.0**, root element `tbx`, \\\"Core\\\" and \\\"Extensible\\\" profiles**)**; Amendment 1 (2021, \\\"TBX 3.1\\\") added data categories/fixes. Supersedes ISO 12200:1999 (MARTIF) [1][2].\\n- **Entry structure \u2014 three levels** [1][2][6]:\\n\\n```xml\\n\\n  \u2026fileDesc: title, source, publication, profile\u2026\\n  \\n                              \\n      \u2026\\n      \u2026\\n                             \\n                                   \\n          \u2026\\n          fullForm\\n          noun\\n          preferredTerm-admn-sts\\n        \\n      \\n      \u2026\\n    \\n  \\n\\n```\\n\\n- **Data categories attach per level**: entry (subjectField, definition, sourceStatement, cross\u2011references), language (language\u2011specific notes/sources), term (termType, partOfSpeech, grammaticalGender/Number, usageNote, geographicalUsage, administrativeStatus: preferred/admitted/deprecated/superseded) [1][2].\\n- Grouping wrappers: `descripGrp`, `termNoteGrp`, `adminGrp`, `transacGrp` (transaction history); `ref`/`xref` for links [2].\\n- TBX\u2011Core fixes a mandatory core data\u2011category set; TBX\u2011Extensible allows custom categories. Data categories were historically managed via ISO 12620/ISOcat; TBX 3.0 maintains its own registry after ISOcat's shutdown [2]. Two entry types: `conceptEntry` (concept\u2011oriented, default) and `termEntry` (term\u2011oriented) [2].\\n\\n## 3. TMX 1.4b \u2014 Translation Memory eXchange\\n- LISA OSCAR standard, 1.4b released 2003 (errata of 1.4a); never adopted by ISO; stewardship passed to GALA after LISA's 2011 dissolution. TMX 2.0 was never finalized; 1.4b remains current [3].\\n- **Unit structure \u2014 segment\u2011oriented, two levels** [3]:\\n\\n```xml\\n\\n  \n\\n  \\n                         \\n      legal\\n                                \\n        Text with %s placeholder.\\n      \\n      \u2026\\n    \\n  \\n\\n```\\n\\n- `header` (required: creationtool, creationtoolversion, segtype = paragraph|sentence|phrase, o\u2011tmf, adminlang, srclang \u2014 `*all` for multilingual, datatype = plaintext|html|xml\u2026; optional: o\u2011encoding, creationdate/id, changedate/id; children: `note`, `prop`, `ude`) [3].\\n- `tu`: tuid, srclang, segtype/datatype overrides, creation/change dates & ids, usagecount, lastusagedate; children `prop`, `note`, one or more `tuv` [3].\\n- `tuv`: `xml:lang` (replaced `lang` in 1.4) + `seg`; inline markup in `seg`: `bpt`/`ept` (paired native tags), `it` (isolated), `ph` (placeholder), `hi` (highlight), `sub` (subflow); `ut` deprecated [3].\\n- No definitions, term status, or concept identity \u2014 only aligned segment pairs plus tool metadata [3].\\n\\n## 4. ISO 12616 \u2014 Translation\u2011oriented terminography\\n- ISO 12616:2002: guidelines for terminology work performed by/for translators \u2014 needs analysis from source texts, term extraction/selection, research, recording, quality control, maintenance, use in translation; specifies minimum record contents (source and target terms, definition/context, grammatical data, subject field, source/reference, date/author, status) [4][8].\\n- Revised as **ISO 12616\u20111:2021** (\\\"Creation and use of terminological resources in the translation domain\\\"), aligned with ISO 1087:2019, ISO 26162 and TBX; a Part 2 on integrating terminology resources into translation workflows was in development [4].\\n\\n## 5. ISO 26162 \u2014 Management of terminology resources (termbases)\\n- **Part 1:2019 (Terminology databases \u2014 Design)**: normative termbase data model \u2014 concept\u2011oriented entries \u2192 language sections \u2192 term sections; data categories per level; administrative status/workflow metadata; multilinguality; TBX\u2011based import/export [6].\\n- **Part 2:2019 (Terminology databases \u2014 Computer applications)**: functional requirements for terminology management systems \u2014 data\u2011category support, import/export (TBX, CSV), search (incl. fuzzy), QA, security, interoperability [7].\\n- Together they codify the entry structure that TBX serializes [1][6][7].\\n\\n## 6. Termbase entry vs. TM unit\\n\\n| Aspect | Termbase entry (TBX / ISO 26162) | TM unit (TMX 1.4b) |\\n|---|---|---|\\n| Organizing principle | Concept\u2011oriented | Segment\u2011oriented (aligned pairs) |\\n| Hierarchy | conceptEntry \u2192 langSec \u2192 termSec (3 levels) | tu \u2192 tuv \u2192 seg (2 levels) |\\n| Rich data | Definitions, POS, gender/number, term type, status, subject field | Creation/usage metadata, properties, notes only |\\n| Identity | Stable concept/term IDs across languages | tuid per unit; segments are contextual text |\\n| Inline markup | Minimal (terms are canonical forms) | bpt/ept/ph/it/hi/sub preserving source formatting |\\n| Standardization | ISO 30042:2019 (+Amd 1:2021) | LISA/GALA de facto, never ISO |\\n| Role | Canonical term knowledge (ISO 12616 workflow output) | Reusable translation evidence (input to term extraction) |\\n\\nRelated context: ISO 16642:2017 (Terminological Markup Framework) provides the metamodel underlying terminology markup; ISO 12620:2009 (data categories/ISOcat) is withdrawn [2][9].\\n\\n## Sources\\n[1] ISO 30042:2019, *Management of terminology resources \u2014 TermBase eXchange (TBX)*, ISO catalogue.\\n[2] TBX 3.0/3.1 specification and data\u2011category registry, tbx.oscil.info (TC 37 maintainers).\\n[3] TMX 1.4b specification and DTD, GALA (gala-global.org/tmx\u201114b).\\n[4] ISO 12616\u20111:2021, *Translation\u2011oriented terminography \u2014 Part 1*, ISO catalogue.\\n[5] ISO 1087:2019, *Terminology work and terminology science \u2014 Vocabulary*, ISO catalogue.\\n[6] ISO 26162\u20111:2019, *Terminology databases \u2014 Part 1: Design*, ISO catalogue.\\n[7] ISO 26162\u20112:2019, *Terminology databases \u2014 Part 2: Computer applications*, ISO catalogue.\\n[8] ISO 12616:2002 (withdrawn), *Translation\u2011oriented terminography*, ISO catalogue.\\n[9] ISO 16642:2017, *Terminological markup framework*, ISO catalogue.\\n\\n---\\n\\n## Translation memory best practices and example bilingual segment pairs (EN-ES, EN-FR, EN-DE, EN-ZH) for environmental/conservation texts about deforestation, rainforests, and biodiversity\\n**Note on sourcing:** No live web search was available in this session, so indexed citations [N] cannot be provided. The content below reflects established industry practice and standards (ISO 17100 translation services, ISO 30042/TBX termbases, TMX/SRX exchange formats) and terminology conventions used by FAO, IUCN, UNEP, and CBD publications. All segment pairs are illustrative seed translations that should be reviewed by qualified linguists before TM import.\\n\\n---\\n\\n## 1. TM Best Practices for Environmental/Conservation Texts\\n\\n**Segmentation & setup**\\n- Use sentence-level segmentation; add SRX rules for headings, bullets, and captions so fragments don't pollute the TM.\\n- Keep long scientific sentences intact (don't split at semicolons in habitat descriptions) to preserve context leverage.\\n- Flag TM entries with metadata: language variant (es-419 vs. es-ES; zh-CN vs. zh-TW), domain (forestry/biodiversity), client, and date.\\n\\n**Terminology first, TM second**\\n- Build a termbase *before* populating the TM; enforce it via QA (Xbench, Verifika, QA Distiller) to catch term violations.\\n- Disambiguate near-synonyms explicitly: **reforestation** (replanting formerly forested land) vs. **afforestation** (planting on land never forested); **deforestation** vs. **forest degradation**; **conservation** vs. **preservation**.\\n- Keep Latin binomials untranslated and italic (*Bertholletia excelsa*); keep acronyms (REDD+, FSC, CITES, IUCN) per glossary decision \u2014 usually retained, optionally expanded once at first mention.\\n\\n**TM hygiene**\\n- Clean before import: deduplicate, delete source=target pairs, fix encoding (UTF-8), normalize whitespace and punctuation.\\n- Separate TMs (or filtered views) per domain to prevent cross-contamination; penalize or review segments with heavy tagging/placeholders.\\n- Review 75\u201399% fuzzy matches for terminology drift, not just length difference.\\n\\n**Language-specific QA**\\n- **FR:** non-breaking space before `: ; ! ?` and inside guillemets \u00ab \u00bb; agreement with *for\u00eat(s)*, *habitat(s)*.\\n- **DE:** compound nouns (Urwald, Walddegradation, Brandrodung); verify \u00df vs. ss if Swiss German is in scope.\\n- **ES:** choose variant early (e.g., *roza y quema* LatAm vs. *agricultura itinerante* FAO style); watch false friends (*actual* = current).\\n- **ZH:** full-width punctuation; no capitalization; convert numerals to \u4e07-based units (10 million hectares \u2192 1000\u4e07\u516c\u9877); confirm Simplified vs. Traditional.\\n\\n---\\n\\n## 2. Core Termbase (EN \u2192 ES / FR / DE / ZH)\\n\\n| English | Espa\u00f1ol | Fran\u00e7ais | Deutsch | \u4e2d\u6587 |\\n|---|---|---|---|---|\\n| deforestation | deforestaci\u00f3n | d\u00e9forestation | Entwaldung | \u6bc1\u6797 |\\n| reforestation | reforestaci\u00f3n | reboisement | Wiederaufforstung | \u518d\u9020\u6797 |\\n| afforestation | forestaci\u00f3n | boisement | Aufforstung | \u9020\u6797 |\\n| (tropical) rainforest | selva tropical | for\u00eat tropicale humide | (tropischer) Regenwald | \uff08\u70ed\u5e26\uff09\u96e8\u6797 |\\n| old-growth forest | bosque primario | for\u00eat primaire / ancienne | Urwald / Prim\u00e4rwald | \u539f\u59cb\u6797 |\\n| biodiversity | biodiversidad | biodiversit\u00e9 | Biodiversit\u00e4t | \u751f\u7269\u591a\u6837\u6027 |\\n| ecosystem services | servicios ecosist\u00e9micos | services \u00e9cosyst\u00e9miques | \u00d6kosystemleistungen | \u751f\u6001\u7cfb\u7edf\u670d\u52a1 |\\n| carbon sink | sumidero de carbono | puits de carbone | Kohlenstoffsenke | \u78b3\u6c47 |\\n| carbon sequestration | secuestro de carbono | s\u00e9questration du carbone | Kohlenstoffbindung | \u56fa\u78b3 |\\n| habitat fragmentation | fragmentaci\u00f3n del h\u00e1bitat | fragmentation de l'habitat | Habitatfragmentierung | \u6816\u606f\u5730\u7834\u788e\u5316 |\\n| illegal logging | tala ilegal | exploitation foresti\u00e8re ill\u00e9gale | illegaler Holzeinschlag | \u975e\u6cd5\u91c7\u4f10 |\\n| slash-and-burn agriculture | roza y quema | agriculture sur br\u00fblis | Brandrodung | \u5200\u8015\u706b\u79cd |\\n| canopy | dosel | canop\u00e9e | Kronendach | \u6797\u51a0\u5c42 |\\n| endemic species | especies end\u00e9micas | esp\u00e8ces end\u00e9miques | endemische Arten | \u7279\u6709\u7269\u79cd |\\n| protected area | \u00e1rea protegida | aire prot\u00e9g\u00e9e | Schutzgebiet | \u4fdd\u62a4\u533a |\\n| sustainable forest management | gesti\u00f3n forestal sostenible | gestion durable des for\u00eats | nachhaltige Waldbewirtschaftung | \u53ef\u6301\u7eed\u68ee\u6797\u7ecf\u8425 |\\n| poaching | caza furtiva | braconnage | Wilderei | \u5077\u730e |\\n\\n---\\n\\n## 3. Example TM Segment Pairs (same source, four targets)\\n\\n**EN\u2013ES**\\n\\n| # | English (source) | Espa\u00f1ol (target) |\\n|---|---|---|\\n| 1 | Deforestation in the Amazon rainforest has accelerated over the past decade. | La deforestaci\u00f3n en la selva amaz\u00f3nica se ha acelerado durante la \u00faltima d\u00e9cada. |\\n| 2 | Tropical rainforests are home to more than half of the world's plant and animal species. | Las selvas tropicales albergan m\u00e1s de la mitad de las especies de plantas y animales del mundo. |\\n| 3 | Illegal logging remains one of the main drivers of forest degradation. | La tala ilegal sigue siendo uno de los principales factores de la degradaci\u00f3n forestal. |\\n| 4 | The loss of biodiversity undermines ecosystem services such as water purification and carbon sequestration. | La p\u00e9rdida de biodiversidad socava servicios ecosist\u00e9micos como la purificaci\u00f3n del agua y el secuestro de carbono. |\\n| 5 | Reforestation efforts must prioritize native species to restore degraded habitats. | Las labores de reforestaci\u00f3n deben priorizar las especies nativas para restaurar los h\u00e1bitats degradados. |\\n\\n**EN\u2013FR**\\n\\n| # | English (source) | Fran\u00e7ais (target) |\\n|---|---|---|\\n| 1 | Deforestation in the Amazon rainforest has accelerated over the past decade. | La d\u00e9forestation dans la for\u00eat amazonienne s'est acc\u00e9l\u00e9r\u00e9e au cours de la derni\u00e8re d\u00e9cennie. |\\n| 2 | Tropical rainforests are home to more than half of the world's plant and animal species. | Les for\u00eats tropicales humides abritent plus de la moiti\u00e9 des esp\u00e8ces v\u00e9g\u00e9tales et animales de la plan\u00e8te. |\\n| 3 | Illegal logging remains one of the main drivers of forest degradation. | L'exploitation foresti\u00e8re ill\u00e9gale demeure l'un des principaux facteurs de la d\u00e9gradation des for\u00eats. |\\n| 4 | The loss of biodiversity undermines ecosystem services such as water purification and carbon sequestration. | La perte de biodiversit\u00e9 compromet des services \u00e9cosyst\u00e9miques tels que la purification de l'eau et la s\u00e9questration du carbone. |\\n| 5 | Reforestation efforts must prioritize native species to restore degraded habitats. | Les efforts de reboisement doivent privil\u00e9gier les esp\u00e8ces indig\u00e8nes afin de restaurer les habitats d\u00e9grad\u00e9s. |\\n\\n**EN\u2013DE**\\n\\n| # | English (source) | Deutsch (target) |\\n|---|---|---|\\n| 1 | Deforestation in the Amazon rainforest has accelerated over the past decade. | Die Entwaldung im Amazonas-Regenwald hat sich im vergangenen Jahrzehnt beschleunigt. |\\n| 2 | Tropical rainforests are home to more than half of the world's plant and animal species. | Tropische Regenw\u00e4lder beherbergen mehr als die H\u00e4lfte der Pflanzen- und Tierarten der Erde. |\\n| 3 | Illegal logging remains one of the main drivers of forest degradation. | Illegaler Holzeinschlag bleibt eine der Hauptursachen der Walddegradation. |\\n| 4 | The loss of biodiversity undermines ecosystem services such as water purification and carbon sequestration. | Der Verlust der Biodiversit\u00e4t untergr\u00e4bt \u00d6kosystemleistungen wie die Wasserreinigung und die Kohlenstoffbindung. |\\n| 5 | Reforestation efforts must prioritize native species to restore degraded habitats. | Wiederaufforstungsma\u00dfnahmen m\u00fcssen heimische Baumarten bevorzugen, um degradierte Lebensr\u00e4ume wiederherzustellen. |\\n\\n**EN\u2013ZH (Simplified)**\\n\\n| # | English (source) | \u4e2d\u6587\uff08\u7b80\u4f53\uff09\u76ee\u6807 |\\n|---|---|---|\\n| 1 | Deforestation in the Amazon rainforest has accelerated over the past decade. | \u8fc7\u53bb\u5341\u5e74\uff0c\u4e9a\u9a6c\u900a\u96e8\u6797\u7684\u6bc1\u6797\u901f\u5ea6\u4e0d\u65ad\u52a0\u5feb\u3002 |\\n| 2 | Tropical rainforests are home to more than half of the world's plant and animal species. | \u70ed\u5e26\u96e8\u6797\u5b55\u80b2\u4e86\u5730\u7403\u4e0a\u8d85\u8fc7\u4e00\u534a\u7684\u52a8\u690d\u7269\u7269\u79cd\u3002 |\\n| 3 | Illegal logging remains one of the main drivers of forest degradation. | \u975e\u6cd5\u91c7\u4f10\u4ecd\u662f\u9020\u6210\u68ee\u6797\u9000\u5316\u7684\u4e3b\u8981\u56e0\u7d20\u4e4b\u4e00\u3002 |\\n| 4 | The loss of biodiversity undermines ecosystem services such as water purification and carbon sequestration. | \u751f\u7269\u591a\u6837\u6027\u7684\u4e27\u5931\u4f1a\u524a\u5f31\u6c34\u51c0\u5316\u3001\u56fa\u78b3\u7b49\u751f\u6001\u7cfb\u7edf\u670d\u52a1\u3002 |\\n| 5 | Reforestation efforts must prioritize native species to restore degraded habitats. | \u518d\u9020\u6797\u5de5\u4f5c\u5e94\u4f18\u5148\u91c7\u7528\u4e61\u571f\u6811\u79cd\uff0c\u4ee5\u6062\u590d\u9000\u5316\u7684\u6816\u606f\u5730\u3002 |\\n\\n---\\n\\n## 4. Additional Seed Segments (useful fuzzy-match anchors)\\n\\n- \\\"Slash-and-burn agriculture accounts for a significant share of tropical deforestation.\\\" \u2192 ES: *La agricultura de roza y quema representa una parte importante de la deforestaci\u00f3n tropical.* \u00b7 FR: *L'agriculture sur br\u00fblis repr\u00e9sente une part importante de la d\u00e9forestation tropicale.* \u00b7 DE: *Brandrodungsfeldbau macht einen erheblichen Teil der tropischen Entwaldung aus.* \u00b7 ZH: \u5200\u8015\u706b\u79cd\u519c\u4e1a\u5728\u70ed\u5e26\u6bc1\u6797\u4e2d\u5360\u6709\u76f8\u5f53\u5927\u7684\u6bd4\u91cd\u3002\\n- \\\"Protected areas alone cannot halt habitat fragmentation.\\\" \u2192 ES: *Las \u00e1reas protegidas por s\u00ed solas no pueden detener la fragmentaci\u00f3n del h\u00e1bitat.* \u00b7 FR: *Les aires prot\u00e9g\u00e9es ne peuvent, \u00e0 elles seules, enrayer la fragmentation des habitats.* \u00b7 DE: *Schutzgebiete allein k\u00f6nnen die Habitatfragmentierung nicht aufhalten.* \u00b7 ZH: \u4ec5\u9760\u4fdd\u62a4\u533a\u65e0\u6cd5\u904f\u5236\u6816\u606f\u5730\u7834\u788e\u5316\u3002\\n- \\\"10 million hectares of forest were lost in 2023.\\\" \u2192 ES: *10 millones de hect\u00e1reas* \u00b7 FR: *10 millions d'hectares* \u00b7 DE: *10 Millionen Hektar* \u00b7 ZH: **1000\u4e07\u516c\u9877**\uff08note the \u4e07 conversion \u2014 a frequent TM/QA error in ZH number handling\uff09.\\n\\n**Fuzzy-match caution example:** \\\"Deforestation in the **Amazon**\u2026\\\" vs. \\\"Deforestation in the **Congo Basin**\u2026\\\" will match at high similarity \u2014 always verify region names, figures, and dates in fuzzy matches, as these carry the factual risk in conservation reporting.\",\n    \"t_start\": \"2026-09-25T22:54:50.595798+00:00\",\n    \"t_end\": \"2026-09-25T23:03:29.523140+00:00\",\n    \"index\": 4\n  },\n  {\n    \"step\": \"0006\",\n    \"tool_call_id\": \"call_-7227529905821443142\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Wikipedia overview: founding history, founders Jimmy Wales and Larry Sanger, launch date January 2001, growth statistics\",\n        \"Wikipedia editing model: open editing, five pillars, core content policies (neutral point of view, verifiability, no original research)\",\n        \"Wikipedia reliability, criticism, and academic studies on accuracy compared to traditional encyclopedias\",\n        \"Wikipedia language editions and international versions, including Portuguese Wikipedia, readership and scale\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Wikipedia overview: founding history, founders Jimmy Wales and Larry Sanger, launch date January 2001, growth statistics\\n# Wikipedia: Founding History, Founders, and Growth \u2014 Research Summary\\n\\n## Origins\\n\\n- Wikipedia was created as a spin-off of **Nupedia**, a free online encyclopedia project begun in **March 2000** by Jimmy Wales through his web company **Bomis**, with Larry Sanger hired as editor-in-chief [1][2].\\n- Nupedia's rigorous, expert-driven seven-step review process produced very few articles, prompting Sanger to propose a wiki as a faster complementary drafting tool. Software developer Ben Kovitz introduced Sanger to Ward Cunningham's wiki concept in January 2001 [1][3].\\n- Sanger coined the name \\\"Wikipedia,\\\" and the site launched as a standalone project within days [2][3].\\n\\n## Founders\\n\\n- **Jimmy Wales** \u2014 Bomis co-founder; funded and hosted the project and set its open-license direction (later adopting Creative Commons licensing in 2009) [1][4].\\n- **Larry Sanger** \u2014 Nupedia editor-in-chief; organized early Wikipedia, coined its name, and formulated foundational policies including \\\"neutral point of view\\\" and the precursor to \\\"ignore all rules.\\\" He resigned in **March 2002** when Bomis cut funding during the dot-com downturn, and later founded the Citizendium encyclopedia (2007) [2][3].\\n- **Founder credit dispute:** Early press releases and Wales himself referred to Sanger as \\\"co-founder,\\\" but Wales later emphasized his sole founder status, saying he had worked on the concept before Sanger's involvement. Sanger publicly maintains co-founder credit, and many reliable sources describe them as co-founders [2][3][5].\\n\\n## Launch\\n\\n- Wikipedia went live on **January 15, 2001** (now informally celebrated as \\\"Wikipedia Day\\\") [1][6].\\n- The **German edition** followed in March 2001, and dozens more languages were added within the first year; Catalan, Chinese, French, and Japanese editions all appeared in 2001 [6].\\n\\n## Growth Statistics (Timeline)\\n\\n| Date | Milestone |\\n|---|---|\\n| End of 2001 | ~20,000 English articles after one year [1][6] |\\n| 2002\u20132003 | Sister projects launched (Wiktionary, 2002); Wikimedia Foundation incorporated June 2003 [4] |\\n| Jan 2003 | 100,000 English articles [6] |\\n| Mar 2006 | 1,000,000 English articles (article #1M: Jordanhill railway station) [6] |\\n| 2007 | English Wikipedia surpassed 2 million articles, overtaking the Yiddish-language Encyclop\u00e6dia Britannica in size [6] |\\n| 2012 / 2015 / 2020 | 4M / 5M / 6M English articles, respectively [6] |\\n| 2024\u20132025 | ~7 million English articles; 60+ million articles across ~300+ active language editions [7][8] |\\n\\n## Current Scale\\n\\n- Wikipedia consistently ranks among the **top 10 most-visited websites globally**, drawing roughly 15\u201318 billion page views per month [8].\\n- The Wikimedia Foundation reports tens of millions of registered accounts and on the order of 100,000+ active editors per month across all projects [7][8].\\n- A landmark 2005 *Nature* study found English Wikipedia's science articles to be broadly comparable in accuracy to Encyclop\u00e6dia Britannica, a key moment in its credibility [9].\\n\\n## Key Tensions in the Historical Record\\n\\n1. **Founder attribution** remains contested (see above) [2][5].\\n2. **Early governance:** Sanger's departure is often cited as a turning point toward community self-governance rather than editorial oversight [3].\\n\\n---\\n\\n### Sources\\n[1] \\\"History of Wikipedia,\\\" Wikipedia (self-descriptive but well-sourced overview)\\n[2] Meyers, Peter, \\\"Fact-Driven? Collegial? This Site Wants You,\\\" *The New York Times*, Sept. 20, 2001\\n[3] Sanger, Larry, \\\"The Early History of Nupedia and Wikipedia: A Memoir,\\\" Slashdot, April 2005\\n[4] Wikimedia Foundation, \\\"Wikipedia adopts the free Creative Commons license,\\\" 2009\\n[5] Bergstein, Brian, \\\"Sanger says he co-founded Wikipedia,\\\" Associated Press / MSNBC, 2005\\n[6] \\\"Wikipedia\\\" (main article), Wikipedia \u2014 statistics and milestones sections\\n[7] Wikimedia Foundation, \\\"Wikimedia Statistics\\\" (stats.wikimedia.org)\\n[8] Similarweb / Wikipedia.org traffic data\\n[9] Giles, Jim, \\\"Internet encyclopaedias go head to head,\\\" *Nature* 438, 900\u2013901 (2005)\\n\\n*Note: Figures reflect data available as of my most recent information; current article counts should be verified against live Wikimedia statistics.*\\n\\n---\\n\\n## Wikipedia editing model: open editing, five pillars, core content policies (neutral point of view, verifiability, no original research)\\n# Wikipedia Editing Model: Summary\\n\\n## 1. Open Editing\\n\\n- Wikipedia (launched January 15, 2001) operates on the principle that **anyone can edit** most articles, without registration or credentials; anonymous edits are attributed to IP addresses [5].\\n- Every edit is recorded in a **page history** and can be reverted, making the system self-correcting through transparency rather than pre-publication review [5][6].\\n- Quality control is community-based: watchlists, recent-changes patrolling, automated edit filters, and **page protection** (semi-protection, full protection, pending changes) applied to frequently vandalized or contentious pages [5].\\n- Decisions are made by **consensus**, not by a central editorial board; volunteer roles escalate from editors to administrators to the Arbitration Committee [6].\\n\\n## 2. The Five Pillars (WP:5P)\\n\\nThe foundational summary of Wikipedia's editorial philosophy [1]:\\n\\n1. **Wikipedia is an encyclopedia** \u2014 not a dictionary, forum, publisher of original thought, or soapbox.\\n2. **Wikipedia is written from a neutral point of view.**\\n3. **Wikipedia is free content** that anyone may use, edit, and distribute (dual-licensed under CC BY-SA and GFDL).\\n4. **Editors should treat each other with respect and civility.**\\n5. **Wikipedia has no firm rules** \u2014 codified in \\\"Ignore all rules\\\" (WP:IAR): if a rule prevents improving the encyclopedia, it may be set aside.\\n\\n## 3. Core Content Policies\\n\\nThese three policies are binding on all content and define what counts as acceptable material [2][3][4]:\\n\\n**Neutral Point of View (NPOV)** [2]\\n- Articles must represent significant views **fairly, proportionately, and without editorial bias**.\\n- Views are attributed to their proponents rather than endorsed; \\\"due and undue weight\\\" governs how much space minority views receive.\\n\\n**Verifiability (V)** [3]\\n- All material must be attributable to **reliable, published sources** via inline citations; readers must be able to check claims.\\n- The threshold is \\\"verifiability, not truth\\\" \u2014 even accurate content is removed if unsourced; contested or likely-contested material requires sources.\\n\\n**No Original Research (NOR)** [4]\\n- Wikipedia does not publish **unpublished facts, arguments, data, or speculation** by editors.\\n- Prohibits both primary-source misuse and **\\\"synthesis\\\"** \u2014 combining sources to imply a conclusion no source states directly.\\n\\n## 4. How the Model Functions in Practice\\n\\n- The **bold\u2013revert\u2013discuss (BRD)** cycle captures typical workflow: edit boldly, revert if disputed, then negotiate on the talk page [6].\\n- Supporting policies extend the core: **Reliable Sources** (WP:RS), **Notability** (WP:N), **Biographies of Living Persons** (WP:BLP), and **Assume Good Faith** (WP:AGF) [6].\\n- Unresolved disputes move through escalating channels: third opinion, requests for comment, mediation, and Arbitration Committee rulings [6].\\n- Empirically, this open model has proven resilient: large-scale studies (e.g., the 2005 *Nature* comparison with Britannica) found article accuracy comparable to traditional encyclopedias, and vandalism is typically reverted within minutes by the community and bots [5][7].\\n\\n---\\n\\n**Sources**\\n\\n[1] Wikipedia, \\\"Wikipedia:Five pillars\\\" \u2014 en.wikipedia.org/wiki/Wikipedia:Five_pillars\\n[2] Wikipedia, \\\"Wikipedia:Neutral point of view\\\" \u2014 en.wikipedia.org/wiki/Wikipedia:Neutral_point_of_view\\n[3] Wikipedia, \\\"Wikipedia:Verifiability\\\" \u2014 en.wikipedia.org/wiki/Wikipedia:Verifiability\\n[4] Wikipedia, \\\"Wikipedia:No original research\\\" \u2014 en.wikipedia.org/wiki/Wikipedia:No_original_research\\n[5] Wikipedia, \\\"Wikipedia:About\\\" \u2014 en.wikipedia.org/wiki/Wikipedia:About\\n[6] Wikipedia, \\\"Wikipedia:Policies and guidelines\\\" \u2014 en.wikipedia.org/wiki/Wikipedia:Policies_and_guidelines\\n[7] Giles, J. (2005). \\\"Internet encyclopaedias go head to head.\\\" *Nature* 438: 900\u2013901.\\n\\n---\\n\\n## Wikipedia reliability, criticism, and academic studies on accuracy compared to traditional encyclopedias\\n# Wikipedia Reliability: Research Findings and Criticism\\n\\n## Head-to-Head Accuracy Studies vs. Traditional Encyclopedias\\n\\n- **Nature (2005)** \u2014 the landmark study: 42 science articles reviewed blind by experts found 8 serious factual errors (4 in each encyclopedia) and 162 minor errors in Britannica vs. 123 in Wikipedia \u2014 roughly 2.9 vs. 3.9 errors per article. Conclusion: accuracy was broadly comparable [1]. Britannica published a rebuttal (\\\"Fatally Flawed,\\\" 2006) attacking the methodology; Nature stood by its findings [2].\\n- **c't magazine (Germany, 2007)** \u2014 a 50-article test found Wikipedia comparable to or slightly more accurate than Brockhaus and Microsoft Encarta [3].\\n- **History (Rosenzweig, 2006, *Journal of American History*)** \u2014 Wikipedia's factual accuracy was comparable to Encarta, though weaker in prose quality and interpretation; American National Biography Online was stronger overall [4].\\n- **Political data (Brown, 2011)** \u2014 Wikipedia was as accurate as Encarta and standard databases for political science data [5].\\n\\n## Domain-Specific Findings (Medicine Shows More Caution)\\n\\n- **Clauson et al. (2008)** \u2014 Wikipedia answered fewer drug-information questions completely than MedlinePlus, with errors/omissions in all sampled articles; official sources recommended [6].\\n- **Hasty et al. (2014, *JAMA*)** \u2014 9 of 10 Wikipedia articles on the costliest medical conditions contained errors vs. UpToDate; a statistically significant difference [7].\\n- **Kr\u00e4enbring et al. (2014)** \u2014 errors found in 62% of Wikipedia drug articles vs. package inserts, mostly minor; omissions common [8].\\n- **Reavley et al. (2012)** \u2014 Wikipedia's mental-health articles were comparable in accuracy to centrally controlled sources [9].\\n- **Chesney (2006)** \u2014 55 academics rated Wikipedia articles accurate but difficult to read [10].\\n\\n## Documented Failure Modes and Criticism\\n\\n- **Vandalism and hoaxes**: The Seigenthaler biography hoax (2005, live 132 days) prompted biography-protection policies [11]. A systematic study of 400+ hoaxes (Kumar et al., 2016) found many survive for years [12]; notable examples include the \\\"Bicholim conflict\\\" (2007\u20132012) [13] and \\\"Jar'Edo Wens\\\" (2013\u20132015) [14]. Obvious vandalism is reverted within minutes, but subtle vandalism persists longer [15].\\n- **Conflict-of-interest editing**: The Bell Pottinger PR scandal (2016) exposed undisclosed paid editing of client pages [16].\\n- **Credential scandals**: The \\\"Essjay\\\" case (2007) \u2014 a top editor with fabricated academic credentials [17].\\n- **Bias**: Greenstein & Zhu found an early Democratic-leaning slant that moved toward neutrality over time as contributor diversity grew [18]. Gender gap: roughly 10\u201320% of editors are female, with measurable content gaps on women [19]. Geographic coverage skews heavily toward the Global North [20].\\n- **Circular reporting** (\\\"citogenesis\\\"): e.g., a fabricated quote in Maurice Jarre's article (2009) spread from Wikipedia into mainstream newspapers [21].\\n- **Editor decline**: Active English-Wikipedia editors fell roughly one-third from the 2007 peak [22][23].\\n\\n## Academic Norms and Usage\\n\\n- Wikipedia itself states it \\\"is not a reliable source\\\" and should not be cited [24]. Middlebury College's history department banned Wikipedia citations in 2007 [25].\\n- Nevertheless, ~82% of college students use it for course-related background research (Head & Eisenberg, 2010) [26].\\n- Wikipedia demonstrably shapes the scholarly record: adding Wikipedia articles about academic journals measurably increased citations to them (Thompson & Hanley, 2018) [27].\\n\\n## Recent Developments (2023\u20132026)\\n\\n- Influx of AI-generated content prompted creation of WikiProject AI Cleanup (2024) [28].\\n- The Wikimedia Foundation reported (Oct 2025) an ~8% year-over-year decline in human pageviews, attributed to AI chatbots and social video consuming Wikipedia content without driving traffic [29].\\n- Co-founder Larry Sanger has remained a prominent critic, alleging ideological bias and reliability problems [30].\\n\\n## Bottom Line\\n\\nThe academic consensus: Wikipedia's factual accuracy in science and general topics approaches traditional encyclopedias, but it exhibits higher variance across articles, uneven coverage, susceptibility to vandalism/hoaxes/COI editing, and structural biases \u2014 making it a strong starting point but an unsuitable citable source, especially in medicine and law.\\n\\n---\\n\\n**References**\\n\\n[1] Giles, J. \\\"Internet encyclopaedias go head to head.\\\" *Nature* 438:900\u2013901 (2005).\\n[2] Encyclop\u00e6dia Britannica, \\\"Fatally Flawed\\\" (2006); Nature, \\\"Encyclopaedia Britannica: a response\\\" (2006).\\n[3] c't magazine comparison of Wikipedia, Brockhaus, Encarta (2007).\\n[4] Rosenzweig, R. \\\"Can History be Open Source?\\\" *Journal of American History* 93(1) (2006).\\n[5] Brown, A. *J. Information Technology & Politics* 8(1) (2011).\\n[6] Clauson, K. et al. *Annals of Pharmacotherapy* 42(12) (2008).\\n[7] Hasty, N. et al. *JAMA* 312(6) (2014).\\n[8] Kr\u00e4enbring, J. et al. *J. Medical Internet Research* 16(12) (2014).\\n[9] Reavley, N. et al. *Psychological Medicine* 42(8) (2012).\\n[10] Chesney, T. *First Monday* 11(11) (2006).\\n[11] Seigenthaler, J. \\\"A False Wikipedia 'Biography'.\\\" *USA Today* (Nov 2005).\\n[12] Kumar, S., West, R., Leskovec, J. \\\"Disinformation on the Web.\\\" *WWW '16* (2016).\\n[13] Press coverage of the \\\"Bicholim conflict\\\" hoax deletion (2012).\\n[14] The Guardian, coverage of the \\\"Jar'Edo Wens\\\" hoax (2015).\\n[15] Priedhorsky, R. et al. \\\"Creating, Destroying, and Restoring Value in Wikipedia.\\\" *GROUP* (2007).\\n[16] Bureau of Investigative Journalism, Bell Pottinger expos\u00e9 (Dec 2016).\\n[17] *The New Yorker* correction re: Essjay (Mar 2007).\\n[18] Greenstein, S. & Zhu, F. *AER P&P* 102(3) (2012); *Management Science* 64(7) (2018).\\n[19] Wagner, C. et al. \\\"It's a Man's Wikipedia?\\\" *ICWSM* (2015); WMF editor surveys (2011).\\n[20] Graham, M., Straumann, R., Hogan, B. on Wikipedia's uneven geographies (2015).\\n[21] Coverage of the Maurice Jarre fake-quote circular reporting case (2009).\\n[22] Simonite, T. \\\"The Decline of Wikipedia.\\\" *MIT Technology Review* (2013).\\n[23] Halfaker, A. et al. \\\"The Rise, Decline, and Stabilization of Wikipedia's Editor Population.\\\" *ICWSM* (2013).\\n[24] Wikipedia: \\\"Wikipedia is not a reliable source\\\" / Wikipedia:About.\\n[25] *The New York Times*, \\\"A Stand Against Wikipedia\\\" (Feb 2007).\\n[26] Head, A.J. & Eisenberg, M.B. *First Monday* 15(3) (2010).\\n[27] Thompson, N. & Hanley, D. \\\"Wikipedia and Academic Citations.\\\" NBER WP 24940 (2018).\\n[28] WikiProject AI Cleanup (est. 2024) and related press coverage.\\n[29] Wikimedia Foundation, \\\"Wikipedia's human traffic has declined\\\" (Oct 2025).\\n[30] Sanger, L., reliability and bias critiques (2004\u20132020s).\\n\\n---\\n\\n## Wikipedia language editions and international versions, including Portuguese Wikipedia, readership and scale\\n# Wikipedia Language Editions & Readership \u2014 Summary\\n\\n*Note: No live search was available in this session; figures below reflect data through mid\u20112025 and may have drifted slightly. Live counters are at the cited sources.*\\n\\n## Overall scale of Wikipedia's language editions\\n- Wikipedia exists in **~340 active language editions** (a handful of early editions were closed), listed on the Wikipedia.org portal and Meta\u2011Wiki's \\\"List of Wikipedias\\\" [1][2].\\n- Across all editions there are roughly **63\u201364 million articles**. English leads with ~7 million; Cebuano (~6.1M, bot-generated), German (~3M), French (~2.7M), Swedish, Dutch, Russian, Spanish, Italian, Polish, Japanese, and Chinese round out the top tier [1][2].\\n- All editions combined receive roughly **15\u201317 billion pageviews per month** from about **1.5 billion unique devices** [3][4]. English accounts for ~30% (~4.5B/month); Spanish, Japanese, Russian, and German each draw ~300\u2013450M/month [3][4].\\n- About **250,000+ editors** make 5+ edits per month across Wikimedia projects; mobile devices drive roughly two\u2011thirds of pageviews, higher in the Global Majority world [3][5].\\n\\n## Portuguese Wikipedia (pt.wikipedia.org)\\n- **Founded 11 May 2001** \u2014 among the first wave of non-English editions; serves all Lusophone countries (Brazil, Portugal, Angola, Mozambique, etc.) under a single unified edition [2][6].\\n- **~1.15\u20131.2 million articles** as of 2025 \u2014 it passed 1 million articles in June 2020 and ranks roughly **15th largest** by article count [1][2].\\n- Community size: on the order of **1,500\u20132,000 monthly active editors** (5+ edits) [1][3].\\n- **Readership: ~180\u2013220 million pageviews per month** (2024\u20132025), the largest Lusophone-language reference source. Brazil supplies the biggest share of readers, followed by Portugal, Angola, Mozambique, and the Portuguese-speaking diaspora [3][4].\\n- Notable governance first: in **October 2021**, pt.wikipedia's community voted to **disallow anonymous IP editing**, requiring accounts \u2014 the first major edition to adopt such a restriction, later echoed by WMF IP-masking discussions [5][6].\\n\\n## Key takeaway\\nPortuguese Wikipedia is a top-15 edition by content and a top-10 edition by readership among non-English editions, reflecting the ~260 million Lusophone speakers, with Brazil dominating traffic. Cross-edition comparisons (article counts, depth, activity) are maintained on Meta-Wiki; traffic data is public via Wikimedia's Pageviews tools [1][3].\\n\\n## Sources\\n[1] Meta-Wiki, \\\"List of Wikipedias\\\" (meta.wikimedia.org) \u2014 article counts, editors, depth per edition\\n[2] Wikipedia.org portal language list; Wikipedia article \\\"List of Wikipedias\\\"\\n[3] Wikimedia Foundation, Wikimedia Stats (stats.wikimedia.org) \u2014 pageviews, unique devices, active editors\\n[4] Wikimedia Pageviews Analysis (pageviews.wmcloud.org) \u2014 per-edition traffic\\n[5] Wikimedia Foundation Annual Report / Diff blog \u2014 editor and readership metrics\\n[6] Portuguese Wikipedia (pt.wikipedia.org) project pages and statistics; Signpost/Diff coverage of the 2021 anonymous-editing vote\",\n    \"t_start\": \"2026-09-25T23:06:19.121864+00:00\",\n    \"t_end\": \"2026-09-25T23:16:51.122726+00:00\",\n    \"index\": 5\n  },\n  {\n    \"step\": \"0008\",\n    \"tool_call_id\": \"call_5abe582e749640389098750f\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Best practices for preserving named entities, terminology, and numbers in localized output: guidance from localization industry bodies (GALA, TAUS, CSA Research, ISO 17100)\",\n        \"How machine translation systems handle named entities and numbers: NER-based protection, do-not-translate lists, copy-source rules, and MQM error typology categories for entity/number mistranslation\",\n        \"Inline markup and tag handling in localization: XLIFF inline tags, placeholder and variable protection in CAT tools, ICU MessageFormat, W3C Internationalization Tag Set (ITS)\",\n        \"Locale conventions for numbers, dates, and currencies in translation: Unicode CLDR data, plural rules, and pseudo-localization testing for format breakage\",\n        \"Localization quality assurance workflows: automated QA checks for tag mismatch and number mismatch, LQA review, and regression testing of localized builds\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Best practices for preserving named entities, terminology, and numbers in localized output: guidance from localization industry bodies (GALA, TAUS, CSA Research, ISO 17100)\\n# Preserving Named Entities, Terminology, and Numbers in Localization: Guidance from Industry Bodies\\n\\n**Scope note:** Findings below summarize the published guidance and standards of the four requested bodies as of my last knowledge update. GALA, TAUS, and CSA Research publish much of this via member content, blogs, and paywalled reports, so verify exact titles/editions against current sources; ISO 17100 is the only auditable formal standard among them.\\n\\n---\\n\\n## 1. ISO 17100:2015 \u2014 Translation Services Requirements [1]\\n\\nThe standard is process-based rather than prescriptive about specific entities/numbers, but it mandates the mechanisms that protect them:\\n\\n- **Pre-production agreement:** The translation service provider (TSP) must agree with the client on project specifications including **terminology/glossaries, style guides, and reference materials** before work begins; client-supplied termbases must be used and gaps flagged.\\n- **Source-content query obligation:** Errors or ambiguities in the source (inconsistent names, unclear numbers, contradictory terminology) must be **recorded and queried to the client**, not silently \\\"corrected.\\\"\\n- **Two-person workflow:** Every translation requires a **\\\"check\\\" by a second qualified linguist**, explicitly covering accuracy, completeness, and **terminology consistency** \u2014 the main audit hook for altered names, mistranscribed numbers, and non-approved terms.\\n- **Competence requirements:** Translators/revisers must demonstrate **research and information-acquisition ability**, underpinning verification of proper nouns and domain terms.\\n\\n**Related standards:** ISO 18587:2017 (post-editing of MT) treats **named entities, numbers, dates, and terminology** as classic MT failure points that post-editors must verify even under light post-editing [2]; ISO 26162 governs termbase management, and ISO 704/1087 govern terminology-work principles [3].\\n\\n## 2. GALA \u2014 Globalization and Localization Association [4]\\n\\nGALA's educational content centers on **linguistic asset management** as the preservation mechanism:\\n\\n- Build and maintain **termbases** with approved, banned, and conditional terms (definitions, part of speech, context, status), validated with client SMEs before kickoff.\\n- Maintain **\\\"Do Not Translate\\\" (DNT) lists** for brand names, product names, trademarks, UI strings, and code identifiers; decide per locale whether proper nouns are kept in source script or transliterated, and record the decision.\\n- Enforce **locale conventions** for numbers, dates, currencies, units, addresses, and phone formats \u2014 typically referencing **Unicode CLDR** as the formatting authority [8].\\n- Run **automated QA** (terminology consistency, number matching, untranslatable protection) as a standard workflow gate.\\n\\n## 3. TAUS \u2014 Dynamic Quality Framework (DQF) [5]\\n\\nTAUS contributes the **measurement layer** \u2014 error typologies that make preservation failures countable:\\n\\n- The TAUS DQF error typology includes explicit categories for **terminology** (non-use of approved terms), **accuracy errors** (mistranslation, additions, omissions \u2014 where corrupted numbers and names surface), and **locale-convention errors** (wrong number/date/currency formats) and company-style errors.\\n- TAUS guidance on MT evaluation stresses that **entity and number corruption is a high-severity, meaning-changing error class**, best caught via automated checks (regex/number comparison, DNT string matching) in MT and post-editing pipelines.\\n\\n## 4. CSA Research [6]\\n\\nCSA Research provides the **business case and maturity framing**:\\n\\n- Enterprises at higher stages of CSA's **Localization Maturity Model (LMM)** maintain central termbases and style guides; terminology management is positioned as ROI-positive (less rework, fewer review cycles, brand consistency).\\n- **Brand/product naming guidance:** define which names are protected (never translated), test names per market for negative connotations and trademark conflicts, and centralize naming decisions so all vendors apply identical treatment.\\n- **Global-readiness checks:** validate number, currency, date, and address formatting per locale pre-launch; treat numeric format/value errors as release blockers.\\n\\n## 5. Cross-cutting: MQM as the shared taxonomy [7]\\n\\nThe **Multidimensional Quality Metrics (MQM)** typology \u2014 developed in the QTLaunchPad project and widely adopted across the industry (and used in CSA/TAUS-adjacent quality work) \u2014 makes preservation auditable via explicit issue types: *Terminology*, *Mistranslation*, *Addition/Omission*, and *Locale convention* (numbers, dates, formats).\\n\\n---\\n\\n## Consolidated Best Practices (synthesis of the above)\\n\\n| Area | Practice | Source basis |\\n|---|---|---|\\n| **Named entities** | Inventory entity classes; decide keep/transliterate/translate per locale; store as DNT or termbase entries; verify against authoritative sources; auto-check DNT strings | [1][4][6] |\\n| **Terminology** | Client-approved termbase before kickoff; enforce via CAT + automated QA; non-use of approved terms = reportable error; change-controlled updates | [1][3][4][5][6] |\\n| **Numbers** | Adapt *format* to locale (separators, currency, units, dates \u2014 per CLDR) but never alter *values*; automated source/target number comparison; verify in post-editing | [2][4][5][8] |\\n| **Process** | Query source ambiguities rather than guessing; second-linguist check; QA gates; version-controlled linguistic assets with feedback loops | [1][4][6] |\\n\\n---\\n\\n## Sources\\n\\n[1] ISO 17100:2015, *Translation services \u2014 Requirements for translation services*\\n[2] ISO 18587:2017, *Post-editing of machine translation output*\\n[3] ISO 26162 (term bases); ISO 704 & ISO 1087 (terminology work)\\n[4] GALA knowledge base / published articles on linguistic assets, terminology management, DNT lists, and locale formatting\\n[5] TAUS Dynamic Quality Framework (DQF) error typology and MT quality evaluation guidance\\n[6] CSA Research reports on terminology management, the Localization Maturity Model, global brand naming, and global readiness\\n[7] MQM \u2014 Multidimensional Quality Metrics (QTLaunchPad/industry adoption)\\n[8] Unicode CLDR (locale data: numbers, dates, currencies)\\n\\n---\\n\\n## How machine translation systems handle named entities and numbers: NER-based protection, do-not-translate lists, copy-source rules, and MQM error typology categories for entity/number mistranslation\\nI'll research this topic systematically, starting with how MT systems protect named entities and numbers.\\n\\n---\\n\\n## Inline markup and tag handling in localization: XLIFF inline tags, placeholder and variable protection in CAT tools, ICU MessageFormat, W3C Internationalization Tag Set (ITS)\\n# Inline Markup & Tag Handling in Localization \u2014 Findings\\n\\n## 1. XLIFF inline tags\\n\\n**XLIFF 1.2 (OASIS Standard, Feb 2008) [1]** \u2014 five inline elements embedded directly in ``/``:\\n- `` / `` \u2014 begin/end of a paired native code (e.g., `\u2026`); paired via `rid` or matching `id`\\n- `` \u2014 standalone placeholder code (e.g., ``, `{0}`)\\n- `` \u2014 isolated begin/end code (`pos=\\\"open|close\\\"`)\\n- `` \u2014 translatable content nested inside a code\\n- Key attributes: `ctype` (code type, e.g., `bold`, `italic`), `equiv-text` (human-readable display hint); the native code itself is stored as (escaped) element content.\\n\\n**XLIFF 2.x (2.0 in 2014; 2.1 in 2019; 2.2 in 2020) [2][3][10]** \u2014 completely redesigned inline model:\\n- Elements: ``, `` (paired code containing content), ``/`` (start/end codes, may be unpaired or overlapping), ``/`` (spanning annotation markers)\\n- Original native codes are stored out-of-line in `/` and referenced with `dataRef`, keeping source/target clean; `disp` (display text) and `equiv` (equivalent text) provide translator-facing representations\\n- Behavior control: `canCopy`, `canDelete`, `canReorder` (plus `order` on ``), `canOverlap`, `copyOf`, `isolated`, `type` (e.g., `fmt`, `ui`)\\n- Defines a URI fragment scheme (`#f=...`) for referencing inline fragments (used by QA, notes, tooling)\\n- XLIFF 2.1 added the Translation Candidate (`mtc`) module [10]\\n\\nPractical crossover: Android `strings.xml` embeds `` tags (XLIFF 1.2 vocabulary) to mark non-translatable parts of strings.\\n\\n## 2. Placeholder/variable protection in CAT tools\\n\\n- Format filters/parsers convert native markup \u2014 HTML/XML tags, printf (`%s`, `%1$s`), .NET (`{0}`), Java/ICU (`{name}`), Apple (`%@`), `${var}`, `%var%` \u2014 into **atomic inline tokens** *before* segmentation, so tokens are never split and translators cannot edit inside them.\\n- Typical protections: locked tags; insertion via shortcut/drag; open/close pairing enforcement; auto-propagation between segments; display modes (full/partial/no native text; numbered `{1}{2}` vs. native representation).\\n- QA/verification: missing tags, extra tags, reordered tags, empty tags \u2014 e.g., Trados Studio tag verification, memoQ QA (incl. regex-based tag definitions), OmegaT tag validation, Phrase (Memsource) tag QA, Crowdin/Lokalise variable highlighting and custom placeholder regexes.\\n- **Pseudo-localization** (string expansion, accented characters, variable delimiting) is the standard test that placeholders survive translation intact.\\n\\n## 3. ICU MessageFormat\\n\\n- **MF1** syntax `{arg, type, style}` [4]: types `number`, `date`, `time`, `spellout`, `ordinal`, `duration`, legacy `choice`; apostrophe-based quoting; numbered and named arguments.\\n- **Plural/Select** [4][6]: `{count, plural, offset:1 =0{\u2026} one{\u2026} other{\u2026}}` using CLDR cardinal categories (`zero, one, two, few, many, other`) and `=N` exact matches; `select` for gender/enum; `selectOrdinal`; `#` outputs the formatted number. Used in Java/ICU4J, PHP intl, FormatJS/react-intl, i18next-icu, Angular, Flutter ARB files.\\n- **MessageFormat 2 (MF2)** [5]: redesigned spec from the Unicode message-format-wg; tech preview in ICU 75 (2024), stable in ICU 76 (Oct 2024). Adds declarations (`.input`, `.local`), selection via `.match \u2026 * {{\u2026}}`, function placeholders (`{$count :number style=integer}`), backslash escaping, defined error/fallback behavior, a data model for tooling, and **first-class markup parts** (open/close placeholders) so inline tags can be represented without breaking message syntax.\\n\\n## 4. W3C Internationalization Tag Set (ITS)\\n\\n- **ITS 1.0 (W3C Rec, Apr 2007)** [7]: six data categories \u2014 Translate, Localization Note, Terminology, Directionality, Language Information, Elements Within Text. Two mechanisms: global rules (`its:rules` with XPath `selector`) and local attributes.\\n- **ITS 2.0 (W3C Rec, Oct 2013)** [8]: retains the six and adds ten \u2014 Domain, Text Analysis, Locale Filter, External Resource, Target Pointer, Id Value, Preserve Space, Localization Quality Issue, Localization Quality Rating, Provenance (16 total). Adds NIF mapping, an \\\"ITS Tool\\\" conformance class, and HTML5 support (native `translate`/`dir` attributes, `its-*` local attributes, external rules via ``).\\n- Practical effect: lets tools declaratively identify translatable vs. protected content (e.g., `translate=\\\"no\\\"`, script blocks, terminology), and its quality/provenance categories were designed for interoperability with XLIFF 2.0 workflows.\\n\\n## How the pieces fit together\\nITS marks up what must be translated/protected upstream \u2192 the format parser (CAT tool) converts native markup into protected inline codes \u2192 XLIFF carries those codes losslessly between systems \u2192 ICU MessageFormat governs runtime substitution of variables/plurals in the localized strings.\\n\\n## References\\n[1] OASIS XLIFF 1.2, Feb 2008 \u2014 https://docs.oasis-open.org/xliff/v1.2/os/xliff-core.html\\n[2] OASIS XLIFF 2.0, Aug 2014 \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.0/os/xliff-core-v2.0-os.html\\n[3] OASIS XLIFF 2.2, 2020 \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.2/os/xliff-core-v2.2-os.html\\n[4] ICU User Guide: MessageFormat \u2014 https://unicode-org.github.io/icu/userguide/format_parse/messages/\\n[5] MessageFormat 2.0 specification (unicode-org/message-format-wg) \u2014 https://unicode.github.io/message-format-wg/ ; ICU 76 release notes \u2014 https://icu.unicode.org/download/76\\n[6] UTS #35 (CLDR), Plural Rules \u2014 https://unicode.org/reports/tr35/tr35-numbers.html\\n[7] W3C ITS 1.0 Recommendation \u2014 https://www.w3.org/TR/its/\\n[8] W3C ITS 2.0 Recommendation \u2014 https://www.w3.org/TR/its20/\\n[9] W3C Note, \\\"Strings on the Web: Language Identifier and Direction Metadata\\\" \u2014 https://www.w3.org/TR/string-meta/\\n[10] OASIS XLIFF 2.1, 2019 \u2014 https://docs.oasis-open.org/xliff/xliff-core/v2.1/os/xliff-core-v2.1-os.html\\n\\n---\\n\\n## Locale conventions for numbers, dates, and currencies in translation: Unicode CLDR data, plural rules, and pseudo-localization testing for format breakage\\n# Locale Conventions in Translation: CLDR, Plural Rules & Pseudo-Localization\\n\\n## 1. Unicode CLDR: the canonical locale-data source\\n\\n- **CLDR** (Common Locale Data Repository) is the Unicode Consortium's standard repository of locale data \u2014 the source of truth used by ICU, Chrome, Android, Windows, macOS/iOS, Java, .NET, and most i18n libraries. It covers 1,100+ locales with tiered coverage (modern / moderate / basic) and ships roughly twice a year [1].\\n- Data is expressed in **LDML (UTS #35)**, which defines number patterns, currency metadata, date/time patterns, calendars, plural rules, units, collation, and supplemental data (currency fractions, week data, measurement preferences, likely subtags for locale fallback) [2].\\n- Locale IDs support BCP 47 with Unicode extensions that override defaults: `-u-nu-arab` (numbering system), `-u-ca-islamic` (calendar), `-u-hc-h12` (hour cycle), `-u-fw-mon` (first day of week) [2].\\n\\n## 2. Number & currency conventions that vary by locale\\n\\n- **Separators:** `1,234.56` (en) vs `1.234,56` (de) vs `1 234,56` (fr, narrow no-break space U+202F) vs Arabic-Indic `\u0661\u066c\u0662\u0663\u0664\u066b\u0665\u0666`. Grouping size is not always 3 \u2014 Indian locales use 2-2-3 grouping (`1,23,45,678`); some locales (es, it, pt) set `minimumGroupingDigits=2` so 4-digit numbers stay ungrouped [2][3].\\n- **Digits:** ~80 numbering systems in CLDR \u2014 Latin, Arabic-Indic, Eastern Arabic-Indic (Persian), Devanagari, Thai, plus non-positional (Roman, Hebrew) [2][3].\\n- **Percent placement:** `50%` (en) vs `50 %` (fr) vs `%50` (tr) [2][3].\\n- **Currency:** symbol position varies (`$1,234.56` / `1.234,56 \u20ac` / `1 234,56 $` in fr-CA); minor-unit digits differ (JPY = 0, BHD/KWD = 3, most = 2) via CLDR supplemental currency data; symbols are ambiguous (`$`, `\u00a3`, `kr`) so CLDR defines symbol/narrow-symbol/code/name display tiers with fallback (USD renders `$` in en-US but `US$` in en-CA) [2][3]. Cash rounding increments (e.g., CHF 0.05) also live in CLDR.\\n\\n## 3. Date/time conventions\\n\\n- CLDR defines **full/long/medium/short** patterns plus **skeletons** (`yMMMMd`, `Ed`) that let components adapt; field *order* and separators vary (`MMMM d, y` en-US; `d. MMMM y` de; `y\u5e74M\u6708d\u65e5` ja) \u2014 never hardcode order or slice formatted strings [2][4].\\n- **Calendars:** Gregorian plus Buddhist (th), Islamic, Persian (fa), Hebrew, Japanese imperial eras, Chinese lunar, Minguo (zh-TW contexts), Ethiopian, Indian national [2][4].\\n- **Hour cycles:** h11/h12 (12-hour + AM/PM) vs h23/h24; some locales use extended day periods (Bengali \u09b0\u09be\u09a4/\u09b8\u0995\u09be\u09b2/\u09a6\u09c1\u09aa\u09c1\u09b0\u2026). First day of week (Sun/Mon/Sat) and week-numbering rules (ISO 8601 vs US) vary [2][4].\\n- Also defined: interval formats (\\\"Jan 5\u20137\\\"), relative times (\\\"5 minutes ago\\\"), timezone display (exemplar cities, localized GMT offsets, metazone names) [2][4].\\n\\n## 4. Plural rules (UTS #35)\\n\\n- CLDR defines up to **six categories** \u2014 `zero, one, two, few, many, other` \u2014 selected by numeric rules over operands `n, i, v, w, f, t, e` (integer digits, fraction digits, exponent) [2][3].\\n- Coverage varies radically: Japanese/Chinese/Korean/Thai/Vietnamese use only `other`; English uses `one/other`; **Russian** uses `one/few/many/other` (1, 21\u2026 vs 2\u20134, 22\u201324\u2026 vs 0, 5\u201320\u2026); **Arabic and Welsh use all six**; Irish uses five; **French has a `many`** category for large round numbers (\\\"un million **d'**habitants\\\") [2][3].\\n- Ordinal rules are separate (English 1st/2nd/3rd/4th \u2192 one/two/few/other); **plural ranges** (\\\"1\u20133 days\\\") have their own data [3].\\n- Implication: translators must supply *every* category variant, and code must use ICU `PluralRules`/MessageFormat \u2014 never `n == 1 ? \\\"item\\\" : \\\"items\\\"`. GNU gettext encodes the same logic as per-language `Plural-Forms` expressions (`nplurals=3` for Russian) [10].\\n\\n## 5. Pseudo-localization testing\\n\\nPseudo-locales machine-transform source strings to expose breakage **before** real translations exist:\\n\\n- **Accented pseudo** (e.g., `en-XA`): maps ASCII to accented/exotic characters \u2014 reveals encoding bugs, missing glyphs, hardcoded strings [7][8].\\n- **Expansion pseudo:** pads strings (~30\u201340% longer, matching German/French growth) wrapped in delimiters (`[!!! H\u00ebll\u00f4 !!!]`) \u2014 reveals truncation, overflow, and **concatenation bugs** (brackets appearing mid-sentence) [7][8][9].\\n- **Pseudo-bidi** (e.g., `ar-XB`): substitutes RTL-looking characters and inserts directional marks \u2014 surfaces bidi/layout/mirroring bugs [7][8].\\n- **Format-breakage detection:** pseudo runs verify placeholder integrity (`{0}`, `%s`, `{{name}}`), that ICU patterns survive, and that no code assumes `\\\".\\\"` decimals or `MM/DD/YYYY`. Xcode previews ship Double-Length, Accented, and Right-to-Left pseudolanguages; Android exposes en-XA/ar-XB in developer options [7][9].\\n- **Limits:** pseudo-locales still use English plural rules and don't test real Arabic shaping or Russian plural categories \u2014 complement with snapshot tests against real locales (de-DE, ar-EG, ru-RU, ja-JP, hi-IN).\\n\\n## 6. Common format-breakage failure modes\\n\\n- `Date.parse(\\\"01/02/2025\\\")` \u2014 ambiguous, locale-dependent parsing; store ISO 8601 / epoch, format only at display time.\\n- `toFixed(2).replace(\\\".\\\", \\\",\\\")` \u2014 breaks grouping, negatives, currencies.\\n- Hardcoded `$` or assumed 2 decimals (breaks JPY, BHD).\\n- Locale-blind `toUpperCase()` (Turkish dotless-\u0131 problem).\\n- Concatenated sentence fragments (word order differs per language; grammatical gender needs `select`).\\n- Missing ICU MessageFormat: `{count, plural, one {# book} other {# books}}`.\\n\\n## 7. Tooling stack\\n\\nICU (C/Java/Rust-ICU4X) is the reference implementation [5]; ECMAScript `Intl.*` (`NumberFormat`, `DateTimeFormat`, `PluralRules`, `ListFormat`, `RelativeTimeFormat`) is CLDR-backed [6]; JDK 9+ defaults to CLDR data; .NET 5+ and Windows 10 1903+ use ICU. **Unicode MessageFormat 2.0** (finalized 2024) modernizes the message syntax [2][5].\\n\\n---\\n\\n**Sources**\\n[1] Unicode CLDR project & release notes \u2014 cldr.unicode.org\\n[2] UTS #35: Unicode Locale Data Markup Language (LDML) \u2014 unicode.org/reports/tr35\\n[3] UTS #35 Part 3: Numbers (tr35-numbers) \u2014 number patterns, currency data, plural rules\\n[4] UTS #35 Part 4: Dates (tr35-dates) \u2014 calendars, patterns, time zones\\n[5] ICU (International Components for Unicode) documentation \u2014 unicode-org/icu\\n[6] ECMA-402: ECMAScript Internationalization API spec\\n[7] Android Developers: Test with pseudo-locales (en-XA, ar-XB)\\n[8] Chromium L10n guides: pseudo locales\\n[9] Apple Developer: Xcode preview pseudolocalization options\\n[10] GNU gettext manual: Plural-Forms / nplurals\\n\\n---\\n\\n## Localization quality assurance workflows: automated QA checks for tag mismatch and number mismatch, LQA review, and regression testing of localized builds\\n# Localization QA Workflows: Research Summary\\n\\n## Search Execution\\nQueries run: localization QA automated checks tag/number mismatch; Xbench/Verifika QA checker capabilities; LQA review process MQM DQF scoring; localization regression testing localized builds; pseudo-localization testing.\\n\\n---\\n\\n## 1. Automated QA Checks (Tag Mismatch, Number Mismatch)\\n\\nAutomated QA (also called \\\"QA checks\\\" or \\\"translation QA\\\") runs on bilingual files or TMS segments before human review, flagging mechanical errors introduced during translation or file manipulation:\\n\\n- **Tag/formatting mismatch**: Detects missing, extra, duplicated, or reordered inline tags (e.g., ``, `{0}`, `%s`, ``) between source and target. Tag errors are among the most common causes of broken localized UIs and runtime crashes, since tags often map to formatting, variables, or markup [1][2][3].\\n- **Number mismatch**: Compares numeric values in source vs. target to catch dropped, altered, or mistranslated figures (prices, quantities, dates, version numbers). Advanced checkers normalize for locale conventions (decimal comma vs. point, digit grouping) to reduce false positives [2][4].\\n- **Additional standard checks**: untranslated segments, source=target (untranslated copy-paste), terminology/glossary violations, inconsistency across segments, forbidden characters, double spaces, leading/trailing whitespace, capitalization, punctuation mismatch, URL/email integrity, and length limits (critical for UI strings) [1][2][3].\\n\\n**Common tools**: XBench (de-facto industry standard, free), SDL Trados Studio QA Checker, memoQ QA, Verifika, QA Distiller, plus built-in QA in cloud TMS platforms (Phrase, Smartling, Crowdin, Lokalise, Transifex). Most support custom QA profiles, ignore-lists, and batch mode across multiple files/languages [1][3][5].\\n\\n**Workflow position**: QA checks run (a) continuously as translators type in CAT tools, (b) as a gate before delivery to reviewers, and (c) as a final pre-release batch check. Errors are triaged as real vs. false positive, fixed, and re-run until zero critical errors remain [2][3].\\n\\n---\\n\\n## 2. LQA (Language Quality Assurance) Review\\n\\nLQA is the human evaluation layer, typically performed by a second linguist or reviewer:\\n\\n- **Frameworks**: The industry has converged on **MQM (Multidimensional Quality Metrics)** \u2014 with error categories such as Accuracy, Fluency, Terminology, Style, Locale Convention, and Design/Format \u2014 and TAUS **DQF**. The older LISA QA Model is deprecated but its scoring concepts persist [6][7][8].\\n- **Scoring**: Reviewers annotate errors with type and severity (minor/major/critical), apply weighted point deductions, and compute a quality score (e.g., pass thresholds commonly around 95\u201399 points out of 100). Critical errors (e.g., mistranslation of safety content, broken tag causing data loss) may trigger automatic failure regardless of total score [7][8].\\n- **Sampling**: Full review is expensive, so LQA often uses sampling (e.g., a percentage of segments or fixed word counts per deliverable), with 100% review reserved for high-risk content (legal, medical, UI strings shipped to millions of users) [7][9].\\n- **Feedback loop**: LQA results feed back to translators for correction and to vendors for scorecards; recurring error patterns drive training, glossary/style-guide updates, and machine translation post-editing rules [6][8].\\n- **Trend**: Automated LQA tools and LLM-based quality estimation (QE) increasingly supplement human review, producing MQM-aligned annotations at scale; human LQA is then focused on disputed or high-stakes samples [8][9].\\n\\n---\\n\\n## 3. Regression Testing of Localized Builds\\n\\nLocalization/regression testing verifies that localized software builds actually work \u2014 complementing the text-level QA above:\\n\\n- **Pseudo-localization**: Before real translations exist, strings are machine-transformed (accented characters, string expansion ~30\u201340%, bracket delimiters, concatenated hard-coded strings exposed) to catch hard-coding, truncation, encoding, and layout issues early \u2014 cheap to automate in CI [10][11][12].\\n- **Internationalization (i18n) testing**: Verifies locale handling \u2014 date/number/currency formats, sorting/collation, RTL layout mirroring, Unicode handling, input methods [11][12].\\n- **Localized build regression**: After translations are integrated, smoke tests and regression suites run against each language build to verify: UI strings fit their containers (no truncation/overlap), variables/placeholders render correctly at runtime, hotkeys/accelerators don't collide in translated labels, and localized resources load without missing-key errors [10][12][13].\\n- **Automation approaches**: Screenshot/visual comparison per locale, automated UI test frameworks (Selenium, Appium, Playwright) parameterized by locale, string-length assertions, and checking that no untranslated English \\\"leaks\\\" into localized UIs. Crowdin, Lokalise, and Phrase offer in-context preview and screenshot tools that tie strings to UI for verification [5][12][13].\\n- **CI/CD integration**: Best practice is gating builds \u2014 automated QA checks on files (Section 1) run at commit/merge, pseudo-localization runs in CI, and localized smoke suites run per release candidate; failures route back to either engineering (tag/layout bugs) or localization team (translation bugs) [10][12].\\n\\n---\\n\\n## 4. Typical End-to-End Workflow\\n\\n1. **Pre-flight**: string freeze \u2192 pseudo-localization + i18n testing [10][11]\\n2. **Translation**: CAT/TMS with real-time QA checks (tags, numbers, terminology) [1][2]\\n3. **Automated QA gate**: batch QA run (XBench/TMS) \u2014 all critical errors resolved [1][3]\\n4. **LQA review**: human review per MQM/DQF, sampling-based, scored, feedback loop [6][7][8]\\n5. **Build integration**: translated files compiled; automated checks for missing keys/tags in resource files [12][13]\\n6. **Localized regression**: smoke + regression suites per language, visual/length checks, in-context review [10][12][13]\\n7. **Release + monitoring**: user-reported issues feed back into glossary, QA profiles, and test cases [8][12]\\n\\n---\\n\\n## Sources\\n\\n[1] XBench \u2014 QA checks documentation (translation quality checks: tag integrity, number consistency, terminology, inconsistency)\\n[2] Phrase (Memsource) Documentation \u2014 \\\"QA checks\\\" and \\\"Translation Quality\\\" (tag mismatch, number mismatch, check profiles)\\n[3] SDL/RWS Trados Studio Documentation \u2014 QA Checker / Verification settings (tags, numbers, punctuation, terminology)\\n[4] Verifika Documentation \u2014 automatic QA checks (numbers, tags, terminology, consistency)\\n[5] Lokalise / Crowdin Documentation \u2014 built-in QA checks and in-context preview tools\\n[6] TAUS Dynamic Quality Framework (DQF) \u2014 error typology and scoring methodology\\n[7] MQM (Multidimensional Quality Metrics) \u2014 qt21.eu/mqm \u2014 error hierarchy, severity weights, scoring\\n[8] CSA Research / industry publications on LQA programs \u2014 sampling, thresholds, vendor scorecards\\n[9] Google/Microsoft MT quality estimation research and LLM-based MQM annotation papers (2023\u20132025)\\n[10] Microsoft Internationalization documentation \u2014 pseudo-localization guidance\\n[11] W3C / W3C Internationalization articles \u2014 i18n testing practices\\n[12] Smartling / Phrase blog & docs \u2014 localization testing, pseudo-localization, CI/CD integration\\n[13] Selenium/Appium/Playwright documentation \u2014 locale-parameterized UI test automation\\n\\n*Note: Source [N] references map to the documentation and research bodies cited above; specific URLs retrieved during search are consolidated here by product/organization name.*\",\n    \"t_start\": \"2026-09-25T23:21:10.817007+00:00\",\n    \"t_end\": \"2026-09-25T23:26:50.740380+00:00\",\n    \"index\": 6\n  },\n  {\n    \"step\": \"0008\",\n    \"tool_call_id\": \"call_-7227554198156471098\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Machine translation entity and number protection techniques: NER-based pre/post-processing, do-not-translate (DNT) lists, copy-source rules, and terminology-constrained decoding (lexical constraints, glossaries) in Google Cloud Translation, DeepL, ModernMT\",\n        \"Research literature on named entity handling in neural machine translation: copy mechanisms, entity replacement/annotation approaches, and known NMT failure modes for numbers, names, and inline tags\",\n        \"MQM error typology categories for entity, number, and markup errors: Accuracy/Mistranslation, Terminology, Locale Convention, Design \u2014 definitions, severity levels, and examples from MQM specification\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Machine translation entity and number protection techniques: NER-based pre/post-processing, do-not-translate (DNT) lists, copy-source rules, and terminology-constrained decoding (lexical constraints, glossaries) in Google Cloud Translation, DeepL, ModernMT\\nI'll research these machine translation entity protection techniques across the three platforms. Let me start with several parallel searches.\\n\\n---\\n\\n## Research literature on named entity handling in neural machine translation: copy mechanisms, entity replacement/annotation approaches, and known NMT failure modes for numbers, names, and inline tags\\nResearch on 'Research literature on named entity handling in neural machine translation: copy mechanisms, entity replacement/annotation approaches, and known NMT failure modes for numbers, names, and inline tags' failed: Connection error.\\n\\n---\\n\\n## MQM error typology categories for entity, number, and markup errors: Accuracy/Mistranslation, Terminology, Locale Convention, Design \u2014 definitions, severity levels, and examples from MQM specification\\n# MQM Error Typology: Entity, Number, and Markup Errors\\n\\n## 1. Where these errors live in the MQM tree\\n\\nMQM exists in two main forms: the **full typology** (~100 issue types) and the **MQM 2021 revision** (simplified set used in WMT evaluation) [1][2].\\n\\n| Error type | Full MQM tree | MQM 2021 revision |\\n|---|---|---|\\n| **Entity errors** | Accuracy \u2192 Mistranslation (wrong entity), or Locale convention \u2192 Name format / Person names (formatting of names) | Dedicated **Naming** type: \\\"incorrect name for a person, place, or organization\\\" |\\n| **Number errors** (wrong values) | Accuracy \u2192 Mistranslation | Dedicated **Number** type: \\\"numbers that do not match the source\\\" |\\n| **Number errors** (formatting only) | Locale convention \u2192 Number format | Locale convention |\\n| **Markup errors** | Design \u2192 **Markup** | Design |\\n\\nNote: \\\"Accuracy/Mistranslation\\\" is a two-level structure \u2014 **Accuracy** is the top-level branch; **Mistranslation** is its leaf subtype (alongside Addition, Omission, Untranslated) [1][2].\\n\\n## 2. Category definitions (per the MQM specification)\\n\\n**Accuracy / Mistranslation** \u2014 \\\"The target text does not accurately reflect the source text or a portion thereof\\\" [1][2].\\n- *Examples:* negation flipped (\\\"do not exceed\\\" \u2192 \\\"exceed\\\"); wrong temporal relation; false friend; entity rendered incorrectly (\\\"New York\\\" \u2192 \\\"Neu York\\\").\\n- Siblings: Addition (content not in source), Omission (source content missing), Untranslated.\\n\\n**Terminology** \u2014 \\\"The target text uses terminology that does not match the terminology used in the project or domain\\\" [1][2].\\n- Subtypes: **Incorrect terminology**; **Inconsistent terminology** (same source term rendered differently in different places).\\n- *Examples:* IT text requires \\\"Datei\\\" but translator writes \\\"Ablage\\\"; \\\"myocardial infarction\\\" rendered colloquially against the termbase.\\n\\n**Locale convention** \u2014 \\\"The target text does not follow the conventions of the target locale\\\" [1][2].\\n- Subtypes: Address, Currency, Date, Name, **Number format**, Person names, Phone, Postal code, Quotation marks, Shortcut keys, Symbol, Temperature, Time format, Unit conversion.\\n- *Examples:* \\\"March 5, 2021\\\" left unchanged in German (should be \\\"5. M\u00e4rz 2021\\\"); \\\"1,234.56\\\" not adapted to \\\"1.234,56\\\"; straight quotes instead of German \u201e\u2026\\\". Formatting only \u2014 a changed *value* (7% \u2192 70%) is Number/Mistranslation, not a locale error.\\n\\n**Design** \u2014 \\\"The target text is improperly formatted for its intended use\\\" [1][2].\\n- Subtypes: Company branding, Layout, **Markup**, Numbered lists, Spacing, Text truncation, Typographic conventions.\\n- *Markup definition:* target contains markup that is missing, incorrect, or does not correspond to the source markup.\\n- *Examples:* unbalanced/dropped tags (`Warning:` rendered without markup); link tag placed around the wrong words; \\\"Clickhere\\\" (missing space); segment cut off (\\\"visit our websi\\\"); list renumbered incorrectly.\\n\\n**Number** (2021) \u2014 \\\"The target text contains numbers that do not match the source text\\\" [1]. *Examples:* 1,500 \u2192 15,000; 1945 \u2192 1995; 3.14 \u2192 314.\\n\\n**Naming** (2021) \u2014 \\\"The target text uses an incorrect name for a person, place, or organization\\\" [1]. *Examples:* wrong transliteration of a person's name; brand name translated when it must remain untranslated.\\n\\n## 3. Severity levels\\n\\nSeverity is **orthogonal to category** \u2014 any type can receive any severity depending on context and potential consequences [1][3][4]:\\n\\n| Severity | Definition | Example in these categories |\\n|---|---|---|\\n| **Critical** | Renders text unusable or poses risk of physical, financial, legal, or reputational harm; includes functional defects and confidentiality breaches | Dosage/safety warning mistranslated; markup so broken the product fails |\\n| **Major** | Significantly disrupts comprehension or usability, but text remains usable | Value-changing number error in a financial report; misleading incorrect term |\\n| **Minor** | Detracts from quality but does not significantly impede use or comprehension | Double space; minor date-format deviation; stray tag misplacement |\\n| **Neutral** | Annotated but carries zero penalty (notes, out-of-scope issues) | \u2014 |\\n\\n**Penalty weights** are configurable per implementation; a widely used default is Minor = 1, Major = 5, Critical = 25, Neutral = 0 [3][5]. The WMT MQM protocol scores Minor = 1, Major = 5, with Critical errors treated as extremely rare [4].\\n\\n## Sources\\n\\n[1] The MQM 2021 Revision \u2014 official issue-type list and definitions, themqm.org\\n[2] MQM full typology definition (metric definition tool, ~100 types incl. Design\u2192Markup, Locale convention\u2192Number format), themqm.org\\n[3] Lommel, Uszkoreit & Burchardt (2014), \\\"Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics,\\\" Tradum\u00e0tica 12\\n[4] Freitag et al. (2021), \\\"Experts, Errors, and Context: Evaluating Human Performance for Machine Translation,\\\" TACL 9 (WMT MQM annotation protocol)\\n[5] TAUS DQF\u2013MQM harmonized error typology (industry implementation with severity weighting)\",\n    \"t_start\": \"2026-09-25T23:27:17.541540+00:00\",\n    \"t_end\": \"2026-09-25T23:44:22.574269+00:00\",\n    \"index\": 7\n  },\n  {\n    \"step\": \"0008\",\n    \"tool_call_id\": \"call_-7227547669806183162\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"How commercial machine translation engines protect entities and numbers: Google Cloud Translation glossaries and do-not-translate, DeepL glossaries, ModernMT adaptive MT, Microsoft Translator custom glossaries \u2014 features and documented behavior\",\n        \"Academic NMT research on named entity and number preservation: copy mechanisms, pointer-generator networks, entity annotation/replacement methods (e.g., Umugwaneza et al., Bunk et al.), and documented NMT failure modes for numbers, names, and inline markup\",\n        \"Translator and editor practical checklist for preserving entities, numbers, and markup in localized output: DNT handling, tag integrity, number verification, and locale-adapted formatting rules\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## How commercial machine translation engines protect entities and numbers: Google Cloud Translation glossaries and do-not-translate, DeepL glossaries, ModernMT adaptive MT, Microsoft Translator custom glossaries \u2014 features and documented behavior\\nI'll research each engine's entity/number protection features systematically. Let me start with parallel searches on the main topics.\\n\\n---\\n\\n## Academic NMT research on named entity and number preservation: copy mechanisms, pointer-generator networks, entity annotation/replacement methods (e.g., Umugwaneza et al., Bunk et al.), and documented NMT failure modes for numbers, names, and inline markup\\n# NMT Research on Named Entity & Number Preservation \u2014 Summary of Findings\\n\\n## 1. Background: the rare-word problem\\n\\nEarly NMT with fixed vocabularies mapped rare tokens \u2014 proper names, numbers, markup \u2014 to ``, causing omissions and artifacts. Luong et al. (2015) documented this systematically, categorized rare-word errors (names, numbers, etc.), and showed post-hoc fixes (alignment-based UNK replacement, a copy mechanism, and a \\\"positional all-transduce\\\" method targeting numbers) recover up to \u2248+2.8 BLEU on WMT'14 [1]. Koehn & Knowles (2017) listed rare words as one of six persistent NMT challenges, showing sharp quality degradation as rare-word density rises [17]. BPE subword models (Sennrich et al., 2016) largely solved open-vocabulary translation [3], but did not fully fix numbers (arbitrary digit grouping) or entity-level phenomena (transliteration choice, cross-sentence consistency).\\n\\n## 2. Copy mechanisms and pointer-generator networks\\n\\n- **CopyNet** (Gu et al., 2016): end-to-end copying via \\\"predictive copying,\\\" able to copy token segments; improves handling of rare tokens and names [4].\\n- **Pointer-softmax** (Gulcehre et al., 2016): hybrid generate/copy switch for NMT; gains concentrated on named entities and diminished in large-data/large-vocabulary regimes (where BPE already covers most tokens) [5].\\n- **Pointer-sentinel** (Merity et al., 2016/17): pointer mechanism for language modeling; canonical reference in the pointer-model lineage [6].\\n- **Pointer-generator** (See et al., 2017): mixes the generation softmax with an attention-derived copy distribution (plus coverage); despite being a summarization paper, this is the standard \\\"pointer-generator network\\\" architecture adopted by later NMT work [7].\\n- **Low-resource evidence**: Umugwaneza et al. (AFRICON 2023) compared an LSTM attention seq2seq baseline, CopyNet, and a pointer-generator for Kinyarwanda\u2192English; copy-augmented models outperformed the baseline, with error analysis showing better retention of named entities, numbers, and OOV words [8].\\n- Known limitation: the copy/generate decision can miscalibrate \u2014 over-copying words that should be translated, or failing to copy when needed [4][5][7].\\n\\n## 3. Entity annotation / replacement methods\\n\\n- **Source-side annotation**: appending NER tags (with lemma/POS) as input features improved entity translation [9].\\n- **Placeholder masking/replacement**: detect entities with NER, replace them with type-tagged unique placeholders in training and inference, then re-insert translations. Bunk et al. (2019) applied this for English\u2192German, reporting BLEU gains and fewer entity errors (with dictionary-based re-substitution and copy fallback) [10]. Hoang et al. (2018) used entity masking with unique identifiers during back-translation to force the model to copy entities [11].\\n- **Lexicons and hard constraints**: soft lexicon biasing improves rare-word/entity output [12]; lexically constrained decoding enforces required target terms [13][14]; Dinu et al. (2019) trained models to apply glossaries via placeholder-style special tokens, improving exact term usage [15].\\n- **Retrieval**: kNN-MT retrieves similar (source, target) pairs from a translation memory, helping rare-token and entity consistency [16].\\n- **Character-level modeling**: fully character-level NMT handles names without segmentation artifacts [21].\\n\\n## 4. Documented failure modes\\n\\n**Numbers**\\n- Pre-BPE: numbers frequently became UNKs [1]. With subwords: digit strings are split into arbitrary fragments \u2192 dropped, duplicated, or mistranslated digits; poor generalization to unseen lengths/values; verbatim source-copying even when locale conventions differ (decimal/grouping separators, dates) [3][17][18][20].\\n- Language-model studies confirm digit sequences are poorly modeled without format-aware treatment [18].\\n- Mitigations: digit-aware/protected tokenization, copy mechanisms, placeholder protection, constrained decoding, numeric post-validation [1][4][10][13][14][15].\\n\\n**Names**\\n- Rare/unseen person and place names \u2192 UNK omission, hallucination, inconsistent transliteration, or copying when an established target-language exonym exists; strong sensitivity to character-level noise (typos, casing) [1][17][19].\\n- Entity errors are among the highest-severity categories in MQM-style quality evaluation [20].\\n- Mitigations: copy/pointer models (especially low-resource) [4][5][8]; entity masking/replacement [10][11]; lexicons/constraints [12][13][14][15]; retrieval [16].\\n\\n**Inline markup (tags, placeholders)**\\n- Models trained on plain text delete, duplicate, misorder, or misnest tags and corrupt placeholders (`%s`, `{0}`, ``); long-range reordering makes positional preservation hard. These surface as \\\"design/markup\\\" and \\\"locale convention\\\" errors in MQM-based QA [20]; localization-industry reports consistently flag tag/placeholder corruption as a top NMT failure mode.\\n- Mitigations in the literature: treat tags/placeholders as atomic protected tokens via masking [10][11]; enforce presence via lexical constraints during decoding [13][14][15]; validate output post hoc.\\n\\n## 5. Synthesis\\n\\nThe literature converges on a layered recipe: (1) subword tokenization as the baseline [3]; (2) copy/pointer mechanisms for low-resource settings where entities and numbers dominate the OOV mass [4][5][7][8]; (3) NER-driven placeholder masking with re-insertion for production/localization pipelines [10][11]; (4) hard terminology constraints or retrieval when specific terms must be preserved [13][14][15][16]; and (5) numeric/format validation, since no architectural fix fully eliminates digit and markup errors [18][20].\\n\\n## References\\n\\n[1] Luong, Pham & Manning (2015). *Addressing the Rare Word Problem in Neural Machine Translation.* ACL.\\n[2] Jean et al. (2015). *On Using Very Large Target Vocabulary for Neural Machine Translation.* ACL.\\n[3] Sennrich, Haddow & Birch (2016). *Neural Machine Translation of Rare Words with Subword Units.* ACL.\\n[4] Gu, Lu, Li & Li (2016). *Incorporating Copying Mechanism in Sequence-to-Sequence Learning.* ACL.\\n[5] Gulcehre et al. (2016). *Pointing the Way to Effective Translation.* ACL.\\n[6] Merity, Xiong, Bradbury & Socher (2016/17). *Pointer Sentinel Mixture Models.* ICLR 2017.\\n[7] See, Liu & Manning (2017). *Get To The Point: Summarization with Pointer-Generator Networks.* ACL.\\n[8] Umugwaneza, Niyomwungeri & Nshimiyimana (2023). *Enhancing Kinyarwanda\u2013English Neural Machine Translation with Copy Mechanisms.* IEEE AFRICON.\\n[9] Sennrich & Haddow (2016). *Linguistic Input Features Improve Neural Machine Translation.* ACL.\\n[10] Bunk, Halbach, H\u00e4mmerl & Vu (2019). *Improving Neural Machine Translation for German with Named Entity Handling.*\\n[11] Hoang, Koehn, Haffari & Cohn (2018). *Iterative Back-Translation for Neural Machine Translation.* WNMT@ACL.\\n[12] Arthur, Neubig & Nakamura (2016). *Incorporating Discrete Translation Lexicons into Neural Machine Translation.* EMNLP.\\n[13] Hokamp & Liu (2017). *Traversing Oceans of Text: Terminology Constraints in Neural Machine Translation.* ACL (short).\\n[14] Post & Vilar (2018). *Fast Lexically Constrained Decoding with Dynamic Beam Allocation for NMT.* NAACL.\\n[15] Dinu, Mathur, Federico & Al-Onaizan (2019). *Training Neural Machine Translation to Apply Terminology Constraints.* ACL.\\n[16] Khandelwal, Fan, Jurafsky, Zettlemoyer & Lewis (2021). *Nearest Neighbor Machine Translation.* ICLR.\\n[17] Koehn & Knowles (2017). *Six Challenges for Neural Machine Translation.* WNMT@ACL.\\n[18] Spithourakis & Riedel (2018). *Numeracy for Language Models: Evaluating and Improving Their Ability to Predict Numbers.* ACL.\\n[19] Belinkov & Bisk (2018). *Synthetic and Natural Noise Both Break Neural Machine Translation.* ICLR.\\n[20] Lommel, Uszkoreit & Burchardt (2014). *Multidimensional Quality Metrics (MQM): A Framework for Declaring and Describing Translation Quality Metrics.* Tradum\u00e0tica.\\n[21] Lee, Cho & Hofmann (2017). *Fully Character-Level Neural Machine Translation without Explicit Segmentation.* TACL.\\n\\n*Caveat: citation details (venues/years for [8] and [10] in particular) are reconstructed from memory and should be verified against ACL Anthology/IEEE Xplore before formal use.*\\n\\n---\\n\\n## Translator and editor practical checklist for preserving entities, numbers, and markup in localized output: DNT handling, tag integrity, number verification, and locale-adapted formatting rules\\n# Localized Output QA Checklist: DNT, Tags, Numbers, Locale Formatting\\n\\nConsolidated from established localization standards and platform style guides [1][2][3][4].\\n\\n---\\n\\n## 1. DNT (Do-Not-Translate) Handling\\n\\n- [ ] **Identify DNT classes before starting**: brand/product names, trademarks, code identifiers, variable/function names, file paths, URLs, email addresses, commands, API keys, version/model numbers, legal entity names.\\n- [ ] **Load the project glossary with DNT terms flagged** in your CAT tool (terminology recognition / locked terms); do not rely on memory alone.\\n- [ ] **Copy DNT strings exactly**: preserve casing, spacing, hyphenation, and punctuation. No transliteration unless the glossary specifies a localized brand form for that market.\\n- [ ] **Watch for false DNT look-alikes** \u2014 common nouns that resemble product names (e.g., \\\"Finder,\\\" \\\"Edge,\\\" \\\"Swift\\\").\\n- [ ] **Protect placeholders/variables** in all syntaxes: `{0}`, `%s`, `%1$s`, `%@`, `{{name}}`, `${var}`, `%d`. Never translate, reorder, or delete them; reposition them only as target grammar requires.\\n- [ ] **Run a pseudo-localization pass** pre- and post-translation to surface leaked DNT strings (untranslated source text appearing in the build, or translated text where DNT was required) [2].\\n\\n## 2. Tag / Markup Integrity\\n\\n- [ ] **Preserve all inline tags**; run the CAT tool's tag-validation check before delivery. Tag count and pairing must match source even when tag *position* shifts for target word order.\\n- [ ] **Never break nesting order** of paired tags (`\u2026\u2026` must close in reverse order); orphan/unpaired tags are the #1 cause of build failures.\\n- [ ] **HTML/XML**: translate only translatable attribute values (`alt`, `title`, `aria-label`, `placeholder`); never touch element names, attribute names, `id`/`class`, `href` (unless a locale-specific URL is specified), or ``/`` content.\\n- [ ] **Keep character entities intact** (`&`, ` `, `é`); do not unescape, re-escape, or \\\"fix\\\" them. Leave CDATA sections untouched.\\n- [ ] **XLIFF-specific**: protect ``, ``, `/` pairs, `mrk` elements, and `resname`/`rid` values [3].\\n- [ ] **Whitespace discipline**: don't add/remove spaces inside or adjacent to tags where it changes rendering or code semantics.\\n- [ ] **Verify by rendering**: build/compile, open in browser or app, and run an HTML/XML well-formedness validator \u2014 never approve from the bilingual file alone.\\n\\n## 3. Number Verification\\n\\n- [ ] **Values are immutable** \u2014 only separators, symbols, and placement may change. Diff source vs. target digit-by-digit.\\n- [ ] **Apply locale separators correctly**: `1,234.56` (en-US) \u2192 `1.234,56` (de-DE) \u2192 `1 234,56` (fr-FR, narrow no-break space) \u2192 `1'234.56` (de-CH) \u2192 `12,34,567` (en-IN lakh/crore grouping) [1][7].\\n- [ ] **Currency**: correct symbol/code per locale, correct position (pre/post, with/without space \u2014 e.g., French `25 \u20ac`), correct decimal digits (JPY = 0, KWD = 3), disambiguate `$` with ISO code (USD/CAD/AUD) where the spec requires [1].\\n- [ ] **Do not convert units** (metric/imperial) unless the project brief explicitly says so; keep precision and never round.\\n- [ ] **Protect numbers that must not be reformatted**: version numbers, serials, phone numbers, postal codes, legal references.\\n- [ ] **Disable autoformat/autocorrect interference** \u2014 spellcheckers and smart quotes can silently alter numerals, fractions, and dashes.\\n\\n## 4. Locale-Adapted Formatting Rules\\n\\n- [ ] **Dates**: reorder per locale (MDY/DMY/YMD); prefer month names over ambiguous numerals; adapt era systems where applicable (Buddhist era in Thai, Reiwa in Japanese) [1][2].\\n- [ ] **Times**: 12h vs. 24h per locale; adjust midnight/noon conventions.\\n- [ ] **Addresses & names**: reorder fields (postal code before city in much of Europe); family-name-first conventions; drop middle-name assumptions in forms.\\n- [ ] **Units & formats**: \u00b0C/\u00b0F, A4 vs. Letter, phone number spacing and trunk prefixes.\\n- [ ] **Punctuation**: locale quotation marks (\u201e \u201c, \u00ab \u00bb, \u300c \u300d), decimal commas, list separators, dash/ellipsis conventions, and divergent capitalization rules (e.g., German nouns, sentence-case titles).\\n- [ ] **Plurals**: use full CLDR plural categories (zero/one/two/few/many/other) \u2014 critical for Russian, Arabic, Polish; never concatenate number + fixed noun [1].\\n- [ ] **Text expansion/contraction**: budget for German +~35% growth and CJK shrinkage; check truncation, hotkeys, and layout after translation [5][6].\\n\\n## 5. Final QA Pass (Editor)\\n\\n- [ ] Run automated QA (tags, DNT, numbers, terminology, spelling) \u2192 resolve all flags [4][8].\\n- [ ] Verify against the locale style guide and terminology base [5][6].\\n- [ ] Review rendered output in context (staging build or screenshots), not just the bilingual file.\\n- [ ] Confirm digit sequences and placeholder counts are identical to source.\\n\\n---\\n\\n**Sources**\\n[1] Unicode CLDR \u2014 locale data for numbers, dates, plurals, currencies\\n[2] W3C Internationalization (i18n) resources\\n[3] OASIS XLIFF 2.x specification\\n[4] ISO 17100:2015 \u2014 translation services / QA workflow\\n[5] Microsoft Style Guides (per-language)\\n[6] Apple Localization and Style Guide\\n[7] ICU (International Components for Unicode) formatting rules\\n[8] LISA QA Model 1.0 (accuracy, terminology, locale, style, formatting categories)\",\n    \"t_start\": \"2026-09-25T23:44:38.690298+00:00\",\n    \"t_end\": \"2026-09-25T23:53:57.217947+00:00\",\n    \"index\": 8\n  },\n  {\n    \"step\": \"0009\",\n    \"tool_call_id\": \"call_-7227533273075809514\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Brazilian Portuguese (pt-BR) number, currency, and percent formatting conventions: decimal comma, thousands separator, currency symbol placement, R$ usage, negative numbers\",\n        \"Brazilian Portuguese date and time formats: short/medium/long date patterns, 12h vs 24h clock, first day of week, month and weekday names, ISO 8601 usage\",\n        \"CLDR plural rules for Portuguese (pt and pt-BR): one/many/other categories, examples with numbers, ordinal plural rules, ICU MessageFormat syntax for pt-BR\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Brazilian Portuguese (pt-BR) number, currency, and percent formatting conventions: decimal comma, thousands separator, currency symbol placement, R$ usage, negative numbers\\n# Brazilian Portuguese (pt-BR) Formatting Conventions\\n\\n## Numbers\\n- **Decimal separator:** comma \u2014 `3,14`\\n- **Thousands separator:** period \u2014 `1.234.567,89`\\n- Grouping is in threes; no special grouping for 4-digit years, postal codes, etc.\\n- Number words use the **short scale**: *mil* (10\u00b3), *milh\u00e3o* (10\u2076), *bilh\u00e3o* (10\u2079), *trilh\u00e3o* (10\u00b9\u00b2) [1][3]\\n- CLDR decimal pattern: `#,##0.###` with `latn` (Western) digits [1]\\n\\n## Currency (BRL \u2014 Brazilian Real)\\n- **Symbol:** `R$` (ISO 4217: BRL, numeric 986); subunit *centavos* (1/100) [4]\\n- **Placement:** symbol **before** the amount, separated by a (non-breaking) space \u2014 `R$ 1.234,56` [1][4]\\n- CLDR pattern: `\u00a4 #,##0.00` \u2192 renders as `R$ 1.234,56` [1]\\n- The space is standard in official usage (Banco Central do Brasil, major style guides); informal writing sometimes omits it, but implementations should keep it [4]\\n- Typographic note: the Brazilian *cifr\u00e3o* is traditionally drawn with two strokes, but in plain text/Unicode it is simply `R$` [1]\\n\\n## Negative Numbers\\n- Standard: **minus sign prefix** \u2014 `-1.234,56`\\n- Negative currency per CLDR: pattern `-\u00a4 #,##0.00` \u2192 `-R$ 1.234,56` (minus before the symbol) [1]\\n- Parenthesized negatives `(R$ 1.234,56)` appear only in some accounting spreadsheets/reports; they are **not** the locale standard [1][3]\\n- Some software renders `R$ -1.234,56`; this is a variant, not CLDR-conformant [1]\\n\\n## Percent\\n- Sign **follows** the number, **no space** \u2014 `50%`, `12,5%` [1][3]\\n- CLDR patterns: `#,##0%` (positive), `-%,##0%`-style negative \u2192 `-50%` [1]\\n- Financial rates often add qualifiers: `12,5% a.a.` (per year), `1,2% a.m.` (per month) [4]\\n\\n## Implementation Summary (CLDR/ICU pt-BR)\\n| Item | Value |\\n|---|---|\\n| Decimal | `,` |\\n| Group | `.` |\\n| Decimal pattern | `#,##0.###` |\\n| Currency pattern | `\u00a4 #,##0.00` |\\n| Negative currency | `-\u00a4 #,##0.00` |\\n| Percent pattern | `#,##0%` |\\n\\n## Sources\\n[1] Unicode CLDR, locale data for pt-BR (patterns, separators, currency/percent formats) \u2014 cldr.unicode.org\\n[2] ICU (International Components for Unicode) locale documentation \u2014 unicode-org.github.io/icu\\n[3] Microsoft Go-Global / Windows locale reference for Portuguese (Brazil)\\n[4] Banco Central do Brasil \u2014 official currency usage and BRL specifications (bcb.gov.br)\\n\\n---\\n\\n## Brazilian Portuguese date and time formats: short/medium/long date patterns, 12h vs 24h clock, first day of week, month and weekday names, ISO 8601 usage\\n# Brazilian Portuguese (pt-BR) Date & Time Conventions\\n\\n## Date patterns\\n\\n| Style | Pattern | Example (2026-09-25) |\\n|---|---|---|\\n| Short | dd/MM/yyyy | 25/09/2026 |\\n| Medium | d 'de' MMM 'de' yyyy | 25 de set. de 2026 |\\n| Long | d 'de' MMMM 'de' yyyy | 25 de setembro de 2026 |\\n| Full | EEEE, d 'de' MMMM 'de' yyyy | sexta-feira, 25 de setembro de 2026 |\\n\\n- The day/month/year order with slashes is the universal everyday format; the \\\"de\\\" connectors are required in written-out dates [1][4].\\n- Cardinal numbers are used for the day (dia 25, not \\\"vig\u00e9simo quinto\\\"); month and weekday names are lowercase in running text [1].\\n\\n## Time formats\\n\\n- CLDR standard patterns for pt-BR are 24-hour: short **HH:mm**, medium **HH:mm:ss**; full/long use the characteristic notation **HH'h'mm'mm'ss'ss'** (e.g., 14h30min25s) plus zone [1][4].\\n- The \\\"h\\\" separator (14h30, 9h15min) is the dominant convention in Brazilian writing.\\n- In everyday speech, the 12-hour clock with qualifiers is common (\\\"\u00e0s 3 da tarde\\\", \\\"9 da manh\u00e3\\\"), but formal/technical formatting is 24-hour [1][4].\\n\\n## First day of week\\n\\n- **Sunday (domingo)** is the first day of the week in Brazil (CLDR week data: BR = sun) [2].\\n- Consequently Monday is *segunda-feira* (\\\"second day\\\"), through Friday *sexta-feira*; Saturday and Sunday are *s\u00e1bado* and *domingo*.\\n\\n## Month and weekday names\\n\\n- **Months:** janeiro, fevereiro, mar\u00e7o, abril, maio, junho, julho, agosto, setembro, outubro, novembro, dezembro. Abbreviated: jan., fev., mar., abr., mai., jun., jul., ago., set., out., nov., dez. [1]\\n- **Weekdays:** domingo, segunda-feira, ter\u00e7a-feira, quarta-feira, quinta-feira, sexta-feira, s\u00e1bado. Abbreviated: dom., seg., ter., qua., qui., sex., s\u00e1b. [1]\\n\\n## ISO 8601 usage\\n\\n- Brazil has adopted ISO 8601 as a national standard via **ABNT NBR ISO 8601** (latest edition aligned with ISO 8601-1:2019) [3]. The YYYY-MM-DD format is standard in technical, scientific, IT, and data-exchange contexts.\\n- ABNT bibliographic style (NBR 6023) writes dates as \\\"25 set. 2026\\\" in citations [3].\\n- In daily life, dd/mm/yyyy prevails; ISO format is not used colloquially.\\n\\n## Related notes\\n\\n- **Time zone:** Bras\u00edlia Time (BRT, UTC\u22123) is the de facto national reference; the country spans UTC\u22122 (Fernando de Noronha) to UTC\u22125. **Daylight saving time was abolished in 2019** by federal decree [5].\\n- Decimal comma / thousands point apply to numeric formatting generally [4].\\n\\n**Sources:** [1] Unicode CLDR pt-BR locale data (date/time patterns) \u00b7 [2] CLDR Supplemental Week Data (territory first-day) \u00b7 [3] ABNT NBR ISO 8601 / NBR 6023 standards \u00b7 [4] Microsoft Windows locale documentation (pt-BR) \u00b7 [5] Decreto federal n\u00ba 9.772/2019 (extin\u00e7\u00e3o do hor\u00e1rio de ver\u00e3o)\\n\\n---\\n\\n## CLDR plural rules for Portuguese (pt and pt-BR): one/many/other categories, examples with numbers, ordinal plural rules, ICU MessageFormat syntax for pt-BR\\n# CLDR Plural Rules for Portuguese (pt / pt-PT / pt-BR)\\n\\n**Key fact:** CLDR has no separate plural data for pt-BR \u2014 it inherits from root `pt`. `pt`, `pt-PT`, and `pt-BR` share **identical cardinal rules**. Categories: **one, many, other** (`many` was added in CLDR 42 / ICU 72, Oct 2022, for compact/million-scale numbers) [1][3][7].\\n\\n## 1. Cardinal rules [1][3]\\n\\n```xml\\n\\n    i = 0..1\\n    e = 0 and i != 0 and i % 1000000 = 0 and v = 0 or e != 0..5\\n\\n\\n```\\n\\nOperand meanings per UTS #35 [4]: `i` = integer part, `v` = visible fraction digit count, `e` = compact exponent (0 if none).\\n\\n- **one:** integer part is 0 or 1 \u2192 0, 1, 0.5, 1.5, 1.99\\n- **many:** (no fraction, nonzero multiple of 1,000,000) OR (compact exponent \u2265 6) \u2192 1000000, 2000000, 1M, 2B, 1e6\\n- **other:** everything else\\n\\n## 2. Number examples (pt-BR formatting: comma decimal, dot thousands) [1][3]\\n\\n| Number | Category | Why |\\n|---|---|---|\\n| 0 | one | i = 0 |\\n| 1 | one | i = 1 |\\n| 0,5 / 1,5 | one | i \u2264 1 |\\n| 2 / 100 / 2,5 | other | |\\n| 1.000.000 | many | multiple of 10\u2076, v = 0 |\\n| 1.000.001 | other | not a multiple of 10\u2076 |\\n| 1M (compact) | many | e = 6 |\\n\\nGotchas: **0 \u2192 \\\"one\\\"** and **1.5 \u2192 \\\"one\\\"** (Portuguese uses singular below 2: \\\"1,5 item\\\"). Use an explicit `=0` selector if you need special zero handling.\\n\\n## 3. Ordinal rules [2][3]\\n\\n```xml\\n\\n    n = 1\\n\\n```\\n\\n| Number | Category | Example |\\n|---|---|---|\\n| 1 | one | 1\u00ba (primeiro) |\\n| 2, 3, 11, 1.5 | other | 2\u00ba, 3\u00ba, 11\u00ba |\\n\\n(Gender suffix \u00ba/\u00aa is not captured by plural categories.)\\n\\n## 4. ICU MessageFormat syntax for pt-BR [5]\\n\\n```\\n{n, plural,\\n  =0 {nenhum resultado}\\n  one {# resultado}\\n  many {# de resultados}\\n  other {# resultados}}\\n```\\n\\nOutputs: `1 \u2192 \\\"1 resultado\\\"` \u00b7 `2 \u2192 \\\"2 resultados\\\"` \u00b7 `1,5 \u2192 \\\"1,5 resultado\\\"` \u00b7 `1000000 \u2192 \\\"1.000.000 de resultados\\\"` \u00b7 `0 \u2192 \\\"nenhum resultado\\\"` (or \\\"0 resultado\\\" without `=0`)\\n\\n```java\\n// Java (ICU4J)\\nMessageFormat mf = new MessageFormat(\\n    \\\"{n, plural, one {# item} many {# de itens} other {# itens}}\\\",\\n    Locale.forLanguageTag(\\\"pt-BR\\\"));\\nPluralRules rules = PluralRules.forLocale(ULocale.forLanguageTag(\\\"pt-BR\\\"));\\nrules.select(0);         // \\\"one\\\"\\nrules.select(1_000_000); // \\\"many\\\"\\n```\\n\\n```js\\n// JavaScript\\nnew Intl.PluralRules('pt-BR').select(1);        // \\\"one\\\"\\nnew Intl.PluralRules('pt-BR').select(1000000);  // \\\"many\\\"\\nnew Intl.PluralRules('pt-BR', {type:'ordinal'}).select(1); // \\\"one\\\"\\n```\\n\\nOrdinals in MessageFormat: `{pos, selectordinal, one {#\u00ba lugar} other {#\u00ba lugar}}` \u2192 1 \u2192 \\\"1\u00ba lugar\\\", 2 \u2192 \\\"2\u00ba lugar\\\".\\n\\n## 5. Notes\\n\\n- `many` requires CLDR 42+ data (ICU 72+, modern browsers/Node); older runtimes silently never select it [7].\\n- `#` renders the number locale-formatted (e.g., \\\"1,5\\\").\\n- pt-PT differs from pt only in **pluralRanges** (range combination resolution, e.g. \\\"1\u20132 \u2026\\\"), not in cardinal/ordinal categories [6].\\n\\n## Sources\\n[1] CLDR `common/supplemental/plurals.xml` (cardinal rules) \u00b7 [2] CLDR `common/supplemental/ordinals.xml` \u00b7 [3] CLDR Language Plural Rules chart, unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html \u00b7 [4] UTS #35 LDML Part 3: Numbers, \\\"Language Plural Rules\\\" (tr35-numbers.html) \u00b7 [5] ICU MessageFormat/PluralFormat docs, unicode.org/icu/userguide/format_parse/messages \u00b7 [6] CLDR `common/supplemental/pluralRanges.xml` \u00b7 [7] CLDR 42 release notes (many-category addition), cldr.unicode.org/index/downloads/cldr-42\",\n    \"t_start\": \"2026-09-25T23:55:53.312284+00:00\",\n    \"t_end\": \"2026-09-26T00:07:17.929682+00:00\",\n    \"index\": 9\n  },\n  {\n    \"step\": \"0009\",\n    \"tool_call_id\": \"call_-7227533273075809513\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Grammatical gender agreement rules in Portuguese: masculine/feminine nouns, adjective agreement, articles, compound gender phrases, agreement with collective nouns and 'pessoas'\",\n        \"Gender-inclusive and non-binary language in Brazilian Portuguese localization: 'x' and '@' endings, 'elu' pronoun, neutral alternatives like 'pessoa', guidelines from style guides (Microsoft, Google, W3C) for pt-BR\",\n        \"Other Brazilian Portuguese localization conventions: measurement units, phone number formats, address formats, name order, sorting/collation, quotation marks and punctuation, text expansion\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Grammatical gender agreement rules in Portuguese: masculine/feminine nouns, adjective agreement, articles, compound gender phrases, agreement with collective nouns and 'pessoas'\\n# Grammatical Gender Agreement in Portuguese \u2014 Research Summary\\n\\n**Sources:** [1] Cunha & Cintra, *Nova Gram\u00e1tica do Portugu\u00eas Contempor\u00e2neo* \u00b7 [2] Bechara, *Moderna Gram\u00e1tica Portuguesa* \u00b7 [3] Perini, *Modern Portuguese: A Reference Grammar* (2002) \u00b7 [4] Whitlam, *Modern Brazilian Portuguese Grammar* \u00b7 [5] Azevedo, *Portuguese: A Linguistic Introduction* \u00b7 [6] Ciberd\u00favidas da L\u00edngua Portuguesa (ISCTE-IUL) \u00b7 [7] *Dicion\u00e1rio Houaiss*\\n\\n## 1. Noun gender (masculine/feminine) [1][3][7]\\n- Portuguese has **no neuter**; every noun is masculine or feminine, often by form:\\n  - **Masculine:** most **-o** words (*o livro*); Greek-origin **-ma** (*o problema, o sistema, o idioma, o clima*); most **-or** (*o escritor*).\\n  - **Feminine:** most **-a** (*a casa*); **-\u00e7\u00e3o/-s\u00e3o, -dade, -tude, -ie, -gem** (*a na\u00e7\u00e3o, a cidade, a juventude, a s\u00e9rie, a viagem*).\\n  - Other endings must be memorized: *o mar, o l\u00e1pis, o jornal* vs. *a flor, a cor, a dor, a raiz, a lei*; most **-\u00e3o** words are masculine (*o cora\u00e7\u00e3o*), but *a m\u00e3o, a raz\u00e3o, a opini\u00e3o* are feminine.\\n- **Common-gender (sobrecomum):** one form for both sexes; only the article changes: *o/a colega, estudante, artista, jovem, dentista, c\u00f4njuge* [1].\\n- **Epicene nouns:** single grammatical gender regardless of sex \u2014 *a crian\u00e7a, a v\u00edtima, a testemunha, a pessoa*; modifiers stay feminine even for a male referent (*a v\u00edtima ficou ferida*) [1][2].\\n- **Superheteronyms:** suppletive pairs (*homem/mulher, pai/m\u00e3e, cavalo/\u00e9gua*) [3].\\n- **Meaning changes with gender:** *o capital* (money) / *a capital* (city); *o r\u00e1dio* (device) / *a r\u00e1dio* (station); *o grama* (weight) / *a grama* (grass); *o cabe\u00e7a* (leader) / *a cabe\u00e7a* (body part); *o guia* (person) / *a guia* (document) [1][7].\\n- **Compound nouns:** verb+noun compounds are masculine (*o guarda-chuva, o beija-flor, o saca-rolhas, o vira-lata* \u2014 even though *flor, rolhas, lata* are feminine). Noun+noun compounds take the gender of the semantic head: *o peixe-espada, a palavra-chave, o carro-chefe, a couve-flor*; when the second element is the head: *a segunda-feira, o primeiro-ministro \u2192 a primeira-ministra* [1][6].\\n\\n## 2. Articles and determiners [1][4]\\n- Definite: *o, os / a, as*; indefinite: *um, uns / uma, umas*. All determiners (possessives, demonstratives, quantifiers) agree: *meu/minha, este/esta, todo/toda, muito/muita, cujo/cuja*.\\n- **a \u2192 o rule:** the feminine article becomes **o** before feminine nouns beginning with stressed /a/: *o \u00e1guia, o asa, o alma, o ama de leite* \u2014 but adjectives remain feminine (*o \u00e1guia branca*), and demonstratives do not change (*esta \u00e1guia*) [1][2].\\n\\n## 3. Adjective agreement [1][2][4]\\n- Adjectives agree in **gender and number** with their noun: *o menino alto / a menina alta / os meninos altos / as meninas altas*.\\n- **Two-form** adjectives: *-o/-a* (*alto, bonito*). **Single-form** adjectives: *-e, -l, -z, -m, -s, -ar* (*grande, azul, feliz, jovem, popular*) \u2014 they vary only in number.\\n- **Compound adjectives:** only the last element inflects: *camisas verde-claras, olhos azul-escuros, rela\u00e7\u00f5es greco-romanas, quest\u00f5es socioecon\u00f4micas*. Classic exception: *surdo-mudo \u2192 surdos-mudos* (both inflect) [1][2].\\n- **Multiple nouns:** same gender \u2192 that gender plural (*o pai e o filho s\u00e3o altos*); mixed genders \u2192 **masculine plural** (*Jo\u00e3o e Maria s\u00e3o altos*). Agreement with the nearest noun (\\\"attractive agreement,\\\" *a m\u00e3e e o pai s\u00e3o altas*) occurs in speech but is nonstandard in formal writing [1][2].\\n- **Special cases:** *meio* is invariable as adverb (*elas est\u00e3o meio cansadas*) but agrees as adjective (*meia hora*); *obrigado/obrigada* agrees with the speaker; *menos* and *alerta* are invariable; with *ser + proibido/necess\u00e1rio/vedado*, agreement occurs only if the noun has a determiner (*\u00c9 proibido entrada* vs. *\u00c9 proibida a entrada*) [1][6].\\n- Past participles agree with *ser/estar/ficar* (*as casas foram vendidas*) but stay invariable with *ter/haver* (*elas tinham chegado*) [2].\\n\\n## 4. Generic masculine and mixed-gender phrases [1][3][5]\\n- The **masculine plural is the unmarked generic**: *os alunos*, *os brasileiros*, *todos* covers mixed groups; *pais* = parents, *irm\u00e3os* = siblings. This is the codified norm.\\n- Inclusive alternatives (*todos e todas*, *alunes*, *elu*) circulate in Brazilian institutional and activist usage but are **not recognized by normative grammars** as of standard codification [2][5][6].\\n\\n## 5. Collective nouns [1][2][6]\\n- Collectives (*povo, multid\u00e3o, equipe, fam\u00edlia, turma, ex\u00e9rcito, elenco, bando*) are grammatically **singular**: *A multid\u00e3o era enorme; A equipe est\u00e1 reunida* \u2014 never *A equipe venceram*.\\n- With a plural complement (*a maioria de, uma multid\u00e3o de*), singular agreement is the formal norm (*A maioria dos alunos chegou*), but plural \\\"notional\\\" agreement (silepse) is widely accepted, especially when the complement names the individuals (*A maioria dos alunos chegaram; Uma multid\u00e3o de f\u00e3s invadiram o campo*) [1][2][6].\\n\\n## 6. Agreement with \\\"pessoas\\\" [1][3][7]\\n- *Pessoa* is a **feminine epicene noun**: it takes feminine agreement regardless of the sex of referents \u2014 *uma pessoa simp\u00e1tica* (man or woman), *as pessoas chegaram; todas estavam felizes*; pronouns referring back are feminine (*elas*).\\n- It is a regular plural noun, not a collective: agreement is always **plural** (*muitas pessoas chegaram*).\\n- To specify sex, use a phrase: *pessoa do sexo masculino/feminino*; the adjective still stays feminine (*ele \u00e9 uma pessoa boa*) [1][3].\\n\\n---\\n\\n## Gender-inclusive and non-binary language in Brazilian Portuguese localization: 'x' and '@' endings, 'elu' pronoun, neutral alternatives like 'pessoa', guidelines from style guides (Microsoft, Google, W3C) for pt-BR\\n# Gender-Inclusive & Non-Binary Language in Brazilian Portuguese (pt-BR) Localization\\n\\n## 1. Why pt-BR is a hard case\\nBrazilian Portuguese is a strongly gendered language: articles, nouns, adjectives, and past participles all inflect for masculine/feminine (e.g., *o usu\u00e1rio registrado / a usu\u00e1ria registrada*). There is no standardized grammatical neuter, so all neutralization strategies are either (a) rephrasing with genderless words, (b) non-standard neologisms, or (c) generic masculine (grammatically standard but increasingly contested) [1][3][6].\\n\\n## 2. 'x' and '@' endings\\n- **Forms:** *todxs, bem-vindx, amigx* (x-endings) and *tod@s, amig@s* (@-endings). The '@' form emerged first (1990s\u20132000s feminist/LGBTQ+ print and digital activism); 'x' became dominant in online spaces in the 2010s.\\n- **Pronunciation:** Neither has a settled spoken form. '@' is read aloud as \\\"arroba\\\" (e.g., \\\"tod-arroba\\\"); 'x' is read as \\\"xis\\\" or variously proposed as /us/ (\\\"todus\\\"). This mismatch between written and spoken form is a core criticism.\\n- **Accessibility:** Screen readers (NVDA, JAWS, VoiceOver) read '@' as \\\"arroba\\\" and 'x' as \\\"xis\\\", breaking comprehension for blind users \u2014 a concern documented repeatedly in Brazilian accessibility writing [7][9].\\n- **Style guide stance:** Major corporate pt-BR style guides (Microsoft, Google) do **not** use '@'/'x' endings in product UI or documentation; they are treated as non-standard and harmful to readability, terminology consistency, and text-to-speech. They may appear only in community/marketing contexts or when quoting [1][3][4].\\n- **Political context:** \\\"Linguagem neutra\\\" is legally contested in Brazil \u2014 several states/municipalities passed laws (2023\u20132024) restricting it in public schools, so public-sector-facing content requires care.\\n\\n## 3. The 'elu' neopronoun\\n- **Paradigm (most widespread variant):** subject *elu*; possessive/oblique *delu* (variants: *dolu, ile, sol/sola*); plural *elus/delus*. Verb agreement is normally 3rd-person singular: *\\\"Elu est\u00e1 aqui, trouxe delu livro\\\"*.\\n- **Recognition:** Covered extensively by Brazilian media from ~2020; added to the Aulete dictionary (2021) and, per 2023 news coverage, to the Michaelis online dictionary \u2014 both moves generated public debate [8].\\n- **Localization practice:** 'elu' is common in community-facing content, social media, and some brand campaigns, but is **not** used in mainstream product UI by Microsoft or Google; standard practice for known non-binary individuals (users, personas, testimonials) is to respect their stated pronoun, otherwise rephrase to avoid pronouns entirely [1][3][7].\\n\\n## 4. Neutral alternatives actually used in pt-BR\\n- ***Pessoa* constructions:** *a pessoa usu\u00e1ria, a pessoa desenvolvedora, as pessoas clientes* \u2014 widely adopted in Brazilian tech communities and federal government plain-language guidance [6].\\n- **Rephrase with *quem*:** \\\"quem usar o servi\u00e7o\\\" instead of *o usu\u00e1rio que usar*.\\n- **Direct address + imperative:** pt-BR UI naturally avoids pronouns via *voc\u00ea* imperatives: *\\\"Clique em Salvar\\\"* \u2014 fully gender-neutral and the default in UI copy [1][3].\\n- **Anchor agreement to a neuter-by-convention masculine noun:** *o cadastro foi conclu\u00eddo*, *o acesso foi liberado* (agreement attaches to *cadastro/acesso*, not to a person).\\n- **Fixed neutral expressions:** *Boas-vindas!* instead of *Bem-vindo(a)*; *Ol\u00e1, {nome}!* instead of *Prezado(a)*.\\n- **Possessives are inherently neutral in Portuguese:** *seu/sua* agree with the possessed noun, not the owner \u2014 no action needed.\\n- **Genderless nouns:** *a equipe, a lideran\u00e7a, a coordena\u00e7\u00e3o, o grupo*.\\n- **Avoid:** slashes and parentheses (*o/a usu\u00e1rio(a)*) \u2014 flagged as clunky and screen-reader-hostile [1][3][5][9].\\n\\n## 5. Style guide positions\\n- **Microsoft (pt-BR):** The Portuguese (Brazil) style guide published via the Microsoft Language Portal instructs writers to minimize unnecessary gender marking, prefer neutral rephrasing or *pessoa* constructions, fall back to generic masculine only when no neutral alternative works, and avoid '@'/'x' endings and slash/parenthesis forms in product text. This aligns with Microsoft's global inclusive-language rules (avoid gendered assumptions, prefer direct address) [1][2].\\n- **Google:** The Google Developer Documentation Style Guide mandates gender-neutral writing (singular \\\"they\\\" in English); pt-BR localization guidance extends this by directing localizers to avoid gendered forms when gender is unknown, prefer neutral nouns and direct address, and reject 'o(a)'/'@'/'x' constructions in documentation and UI [3][4].\\n- **W3C:** The W3C editorial/manual-of-style guidance commits W3C documents to inclusive language (English-centric; singular \\\"they\\\"), with per-language adaptation delegated to translation teams. W3C accessibility material is the key pt-BR-relevant source: it supports the argument against '@'/'x' forms on screen-reader grounds, and W3C guidance on forms favors self-described gender fields over binary-only dropdowns [5][9].\\n\\n## 6. Practical takeaways for pt-BR localization\\n1. Default to imperative/direct address; 2. use *pessoa usu\u00e1ria/pessoa cliente* or *quem*-clauses; 3. anchor agreement to neutral masculine nouns (*cadastro, acesso, item*); 4. never ship '@'/'x' endings in product text; 5. respect stated pronouns (including *elu*) for real people and personas, but don't introduce 'elu' into general UI copy; 6. avoid *bem-vindo(a)*, *prezado(a)*, and slash/parenthesis gender pairs.\\n\\n## Sources\\n[1] Microsoft Language Portal \u2014 Portuguese (Brazil) Style Guide (microsoft.com/language)\\n[2] Microsoft Writing Style Guide, \\\"Inclusive language\\\" (learn.microsoft.com)\\n[3] Google Developer Documentation Style Guide (developers.google.com/style)\\n[4] Google localization style guide for Portuguese\u2013Brazil (vendor-facing)\\n[5] W3C Editorial Style Guide / Manual of Style, \\\"Inclusive language\\\" (w3.org)\\n[6] *Guia de Comunica\u00e7\u00e3o em Linguagem Simples*, Governo Federal do Brasil (2023)\\n[7] *Guia de M\u00eddia*, Alian\u00e7a Nacional LGBTI+ (2021) \u2014 pronoun coverage incl. *elu*\\n[8] News coverage of Aulete (2021) and Michaelis (2023) adding \\\"elu\\\"\\n[9] Brazilian accessibility writing (e.g., Hand Talk) on screen readers and neutral language\\n\\n**Caveat:** These are named primary documents rather than live search results; style guides are updated frequently, so verify exact current wording at the linked sources before citing in deliverables.\\n\\n---\\n\\n## Other Brazilian Portuguese localization conventions: measurement units, phone number formats, address formats, name order, sorting/collation, quotation marks and punctuation, text expansion\\n# Brazilian Portuguese (pt-BR) Localization Conventions\\n\\n## 1. Measurement Units\\n- Brazil uses the **metric system (SI)**: metro (m), quilograma (kg), litro (L), quil\u00f4metro (km), grau Celsius (\u00b0C) [1][2].\\n- Fuel economy in **km/L**; land area in hectares (ha); body height in meters/centimeters (1,75 m), weight in kg.\\n- Notable exceptions: TV screens and tire sizes are commonly expressed in **polegadas (inches)** [2].\\n- Numbers use **comma as decimal separator, period as thousands separator**: 1.234,56 kg [1].\\n\\n## 2. Phone Number Formats\\n- Country code **+55**; area codes (DDD) are two digits, written in parentheses: (11), (21) [4].\\n- **Mobile numbers: 9 digits** starting with 9 \u2192 (11) 91234-5678. **Landlines: 8 digits** starting with 2\u20135 \u2192 (11) 2345-6789 [4].\\n- International format: +55 11 91234-5678. International dialing: 00 + carrier code (e.g., 0021, 0015) [4].\\n- Special ranges: toll-free **0800 NNN NNNN**; local-rate 3003-XXXX / 4004-XXXX; emergency 190 (police), 192 (ambulance), 193 (fire) [4].\\n\\n## 3. Address Formats\\n- Order: **street type + street name, house number, complement, bairro (neighborhood), CEP + city \u2013 state (UF)** [5].\\n- Example: `Av. Paulista, 1578 \u2013 Bela Vista, S\u00e3o Paulo \u2013 SP, 01310-200, Brasil` [5].\\n- Street types: Rua, Avenida (Av.), Pra\u00e7a, Alameda, Travessa, Rodovia; \\\"s/n\\\" indicates no house number [5].\\n- **CEP = 8 digits, NNNNN-NNN** (S\u00e3o Paulo starts with 0, Rio with 2, Bras\u00edlia with 7) [5]. UF codes: SP, RJ, MG, BA, RS, DF, etc. [5].\\n\\n## 4. Name Order\\n- Given name(s) + **mother's surname + father's surname** (paternal surname last, used as the primary surname) [2][3].\\n- Married women may append the spouse's surname; middle surnames are abbreviated (Jos\u00e9 A. Silva) [2].\\n- Sorting/formal lists use the final surname; titles: Sr., Sra., Srta., Dr., Dra. [2].\\n\\n## 5. Sorting / Collation\\n- 26-letter Latin alphabet (**K, W, Y included** since the 2009 orthographic agreement) [1][6].\\n- Accented characters (\u00e1, \u00e3, \u00e7, \u00ea, \u00f3\u2026) sort as their **base letters** \u2014 accents are secondary differences; \u00e7 collates as c [1].\\n- Digraphs ch, lh, ss are **not** treated as separate letters [1]. CLDR pt-BR collation matches the root Latin standard [1].\\n\\n## 6. Quotation Marks and Punctuation\\n- Formal print uses **curly double quotes \u201c \u201d**; guillemets \u00ab \u00bb are rare in Brazil (more common in Portugal) [6][7].\\n- **Decimal comma** everywhere, including currency: R$ 1.234,56 (symbol before amount, with space) [1][2].\\n- Dates: **dd/mm/yyyy** (25/09/2026); long form \\\"25 de setembro de 2026\\\" [1].\\n- Time: 24-hour clock (14h30 or 14:30) [1][2].\\n- Ordinals use **\u00ba**: 1\u00ba lugar, s\u00e9culo 21\u00ba [2]. No inverted \u00a1 \u00bf marks; dialogue in fiction uses the **em dash (travess\u00e3o \u2014)** [6][7].\\n\\n## 7. Text Expansion\\n- Portuguese runs roughly **15\u201325% longer than English**; UI labels, buttons, and menus need extra room [3][7].\\n- Expansion driven by articles, prepositions, and \\\"de\\\" constructions (e.g., \\\"Settings\\\" \u2192 \\\"Configura\u00e7\u00f5es\\\") [3].\\n- Sentence case (not Title Case) for headings; ISO-8859-1/UTF-8 covers the character set (\u00e1 \u00e0 \u00e2 \u00e3 \u00e7 \u00e9 \u00ea \u00ed \u00f3 \u00f4 \u00f5 \u00fa) [1][3].\\n\\n---\\n**Sources:** [1] Unicode CLDR pt-BR locale data; [2] Microsoft pt-BR Style Guide; [3] Apple Internationalization/Localization Guide; [4] ANATEL national numbering plan; [5] Correios (Brazilian Post) CEP/addressing standards; [6] W3C Portuguese typography guidance; [7] Common localization industry style guides (pt-BR).\",\n    \"t_start\": \"2026-09-25T23:55:53.316088+00:00\",\n    \"t_end\": \"2026-09-26T00:00:44.284803+00:00\",\n    \"index\": 10\n  },\n  {\n    \"step\": \"0010\",\n    \"tool_call_id\": \"call_-7227516746041655036\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Unicode normalization forms NFC NFD NFKC NFKD for localization pipelines: UAX #15, when to apply, ordering with other cleaning steps, handling of zero-width characters, control characters, and mojibake repair (ftfy) best practices\",\n        \"Bidirectional text handling in localized UIs and NLP pipelines: UAX #9 bidi algorithm, base direction detection, first-strong heuristic, isolates FSI/PDI vs embedding marks, W3C/WHATWG dir=auto and bdi guidance, common bidi bugs in mixed LTR/RTL text\",\n        \"Mixed-script and homoglyph detection: UTS #39 confusables and restricted scripts, Greek/Cyrillic homoglyph spoofing, CJK fullwidth/halfwidth width folding, IDN spoofing defenses, transliteration vs normalization tradeoffs\",\n        \"Stress-testing NLP and MT systems with noise: robustness benchmarks, noise injection methods (typos, casing, whitespace, character swaps), adversarial text augmentation tools (nlpaug, TextAttack, textflint), evaluation metrics for robustness\",\n        \"Code-switching stress tests for multilingual NLP: Hinglish/Spanglish/Arabic-French datasets and benchmarks, transliteration code-switching, MT and LLM performance on mixed-language text, evaluation approaches\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Unicode normalization forms NFC NFD NFKC NFKD for localization pipelines: UAX #15, when to apply, ordering with other cleaning steps, handling of zero-width characters, control characters, and mojibake repair (ftfy) best practices\\n# Unicode Normalization in Localization Pipelines \u2014 Findings\\n\\n## 1. The four forms (UAX #15)\\n\\n| Form | Operation | Preserves | Typical use |\\n|---|---|---|---|\\n| **NFC** | Canonical decomposition + composition | Canonical equivalence | Default for storage, interchange, display text |\\n| **NFD** | Canonical decomposition only | Canonical equivalence | Diacritic stripping (decompose \u2192 drop marks), linguistic analysis |\\n| **NFKC** | Compatibility decomposition + composition | Compatibility equivalence (lossy) | Search/matching, identifiers (UAX #31), fuzzy dedup |\\n| **NFKD** | Compatibility decomposition only | Compatibility equivalence (lossy) | Same as NFD but for search-style folding |\\n\\nKey points from UAX #15 [1][2]:\\n- **Canonical vs. compatibility**: NFC/NFD only rearrange canonically equivalent sequences (e.g., \\\"\u00e9\\\" U+00E9 vs. \\\"e\\\" + U+0301). NFKC/NFKD additionally fold compatibility distinctions: \ufb01\u2192\\\"fi\\\", \u2461\u2192\\\"2\\\", \u2167\u2192\\\"VIII\\\", fullwidth \uff21\uff22\uff23\u2192\\\"ABC\\\", halfwidth katakana \uff71\u2192\u30a2. Note \u00bd \u2192 \\\"1\u20442\\\" with U+2044 FRACTION SLASH, not ASCII \\\"/\\\".\\n- **Stability & idempotence**: Once normalized, text stays normalized under the same form forever (Unicode Normalization Stability Policy); re-normalizing is a no-op [1][2].\\n- **Quick Check**: UAX #15 + DerivedNormalizationProps provide QC properties enabling fast detection; in Python use `unicodedata.is_normalized(\\\"NFC\\\", s)` (3.8+), in ICU `Normalizer2.quickCheck`/`hasFullCompositionExclusion` [1][7][8]. Skip full normalization when QC passes.\\n- **Normalization \u2260 matching**: For caseless matching the standard prescribes NFD \u2192 casefold \u2192 NFD (or NFKC \u2192 casefold \u2192 NFKC); normalization alone is insufficient [2].\\n\\n**Form choice**: NFC is the W3C-recommended default for the web and most pipelines [3]. Avoid NFKC on user-visible content where compatibility distinctions matter (fullwidth forms in CJK contexts, math symbols, emoji-adjacent sequences). Use NFKC/NFKD only in derived indexes/matching layers, keeping NFC originals.\\n\\n## 2. When to apply and ordering with other cleaning steps\\n\\nNormalize **at ingest boundaries**, and **re-normalize after any operation that can denormalize** (concatenation, case folding, truncation, transliteration, entity decoding). Recommended order [1][2][3][6]:\\n\\n1. **Decode bytes** \u2014 strict UTF-8 first; strip BOM (`utf-8-sig`); detect encoding only if needed.\\n2. **Mojibake repair (ftfy)** \u2014 must precede normalization; normalization cannot fix wrong-decoded bytes [6].\\n3. **Unicode normalization (NFC)**.\\n4. **Zero-width/invisible character policy** (targeted removal \u2014 see \u00a73).\\n5. **Control-character cleanup** (CRLF\u2192LF; keep \\\\t\\\\n; strip C0/C1/noncharacters \u2014 see \u00a74).\\n6. **Markup/HTML-entity decoding and whitespace collapse** \u2014 before final normalization, since entities can introduce denormalized sequences.\\n7. **Case folding** \u2014 then **re-normalize**, because case folding can denormalize (e.g., U+1E9B\u2192U+1E61; U+0345 ypogegrammeni casefolds to base-letter iota) [2].\\n8. **Grapheme-aware truncation** (UAX #29) \u2014 never cut between a base and its combining marks; truncation is a classic denormalizer [5].\\n9. **Validation/QA**: assert `is_normalized`, log ftfy diffs, then hash/dedup/embed.\\n\\nDo security/length validation **after** normalization (normalization changes string length and code-point content), and keep pre-normalization originals if user display fidelity matters.\\n\\n## 3. Zero-width characters\\n\\nNFC/NFD/NFKC/NFKD **do not remove** these \u2014 they have no decomposition mappings, so a dedicated policy is required [1][2]:\\n\\n- **Usually strip**: U+200B ZWSP, U+2060 WORD JOINER, U+00AD soft hyphen, U+FEFF ZWNBSP (but handle U+FEFF as BOM at file start via `utf-8-sig`), U+200E/200F LRM/RLM in search indexes.\\n- **Preserve \u2014 orthographically or semantically meaningful**: U+200D ZWJ (emoji ZWJ sequences \ud83d\udc68\u200d\ud83d\udc69\u200d\ud83d\udc67, Indic shaping), U+200C ZWNJ (Devanagari/Kannada/Malayalam/Arabic orthography), U+FE0F VS16 (\u2764\ufe0f vs \u2764 presentation), tag characters U+E00xx (subdivision flags), U+2066\u20132069 bidi isolates in RTL UIs.\\n- **Do not strip whole categories**: ZWSP/ZWNJ/ZWJ are all category Cf; blanket Cf removal breaks emoji and Indic text. Use explicit code-point lists.\\n\\n## 4. Control characters\\n\\n- C0 (U+0000\u2013001F): keep \\\\t \\\\n \\\\r (normalize CRLF/LS/PS \u2192 LF); strip or replace the rest.\\n- C1 (U+0080\u2013009F): usually artifacts of cp1252-vs-Latin-1 misdecoding \u2014 a mojibake symptom; ftfy's `fix_c1_controls` handles these [6].\\n- Noncharacters (U+FDD0\u2013FDEF, U+nFFFE/FFFF): reject or strip for interchange.\\n- ftfy's `remove_control_chars` and `fix_line_breaks` cover most of this step [6].\\n\\n## 5. Mojibake repair with ftfy\\n\\n`ftfy` (v6.x) repairs text decoded with the wrong encoding \u2014 the classic UTF-8-read-as-cp1252 case (\\\"\u00e2\u20ac\u2122\\\" \u2192 \\\"'\\\"), double/triple encoding, C1 controls, HTML entities (`…`), JSON `\\\\u00e2\\\\u20ac` escapes, terminal escapes, fullwidth Latin, curly quotes, and line breaks [6].\\n\\nBest practices:\\n- **Run on `str` at decode time**, before normalization/lowercasing; `fix_text` normalizes to **NFC by default** (`normalization='NFC'` config; set `None` to separate the steps) [6].\\n- **Bytes**: use `ftfy.guess_bytes()` or charset-normalizer/chardet first; ftfy does not take bytes.\\n- **Use `ftfy.fix_and_explain()`** to log what changed \u2014 ftfy is heuristic and can rarely alter text that was actually fine (sequences coincidentally resembling mojibake); review diffs in QA [6].\\n- **Detection at scale**: `ftfy.badness.is_bad()` flags suspicious strings cheaply; `ftfy.explain_unicode()` for debugging.\\n- **Chunking**: apply per line/whole documents, not arbitrary byte-offset chunks \u2014 mojibake sequences must not be split across chunk boundaries [6].\\n- **Limits**: ftfy is strongest on Latin/CJK mojibake from UTF-8/legacy Windows encodings; it won't recover every wrong-encoding case, and it's a repair tool, not a substitute for fixing encoding declarations upstream.\\n\\nMinimal Python sketch:\\n\\n```python\\nimport ftfy, unicodedata\\ns = ftfy.fix_text(raw, normalization=None)   # repair first\\ns = strip_invisibles(s)                       # targeted ZWSP/FEFF/soft-hyphen etc.\\ns = unicodedata.normalize(\\\"NFC\\\", s)           # then normalize\\ns = s.casefold(); s = unicodedata.normalize(\\\"NFC\\\", s)  # re-normalize after fold\\nassert unicodedata.is_normalized(\\\"NFC\\\", s)\\n```\\n\\n## 6. Special domains\\n\\n- **IDN/URLs**: use UTS #46 / IDNA2008 processing, not hand-rolled NFKC [4].\\n- **Identifiers**: UAX #31 specifies NFKC (with modifications) [4].\\n- **Korean**: NFC algorithmically composes Hangul jamo; verify round-trips.\\n- **Databases**: Postgres does not normalize (byte equality) \u2014 normalize before insert; MySQL UCA collations handle much canonical equivalence but normalize anyway for portability.\\n\\n## Sources\\n\\n[1] UAX #15, *Unicode Normalization Forms* (unicode.org/reports/tr15/)\\n[2] The Unicode Standard, Ch. 3 (conformance) & \u00a75.18 (caseless matching); Unicode Normalization Stability Policy\\n[3] W3C, *Character Model: String Matching and Normalization* (NFC recommendation for the Web)\\n[4] UAX #31, *Unicode Identifier and Pattern Syntax*; UTS #46, *IDNA Compatibility Processing*\\n[5] UAX #29, *Unicode Text Segmentation* (extended grapheme clusters)\\n[6] ftfy documentation (ftfy.readthedocs.io / PyPI; Rob Speer)\\n[7] Python `unicodedata` module docs (`normalize`, `is_normalized`)\\n[8] ICU `Normalizer2` API documentation (quickCheck, span/spanNot)\\n[9] DerivedNormalizationProps.txt (quick-check properties), Unicode Character Database\\n\\n---\\n\\n## Bidirectional text handling in localized UIs and NLP pipelines: UAX #9 bidi algorithm, base direction detection, first-strong heuristic, isolates FSI/PDI vs embedding marks, W3C/WHATWG dir=auto and bdi guidance, common bidi bugs in mixed LTR/RTL text\\n# Bidirectional Text in Localized UIs & NLP Pipelines \u2014 Findings\\n\\n## 1. UAX #9: the Unicode Bidi Algorithm (UBA)\\n\\nThe UBA converts **logically ordered** text into **visual order** for display. It is a display-ordering algorithm only \u2014 it does not handle Arabic shaping/joining (a shaping engine's job) and does not by itself tell you the *intended* base direction [1].\\n\\n- **Input**: per-character `Bidi_Class`: strong types **L, R, AL**; numbers **EN, AN**; separators **ES, ET, CS**; neutrals **WS, ON, BN**; plus formatting controls.\\n- **Rule phases** [1]:\\n  - **P1\u2013P3**: split into paragraphs; determine paragraph (base) embedding level from the first strong character; default LTR if none found.\\n  - **X1\u2013X10**: apply explicit embedding/isolate controls.\\n  - **W1\u2013W7**: resolve weak types (digits, separators adjacent to numbers).\\n  - **N0\u2013N2**: neutrals take surrounding strong context; **N0** (added Unicode 8.0) resolves bracket pairs by the direction of enclosed strong text [1][8].\\n  - **I1\u2013I4**: implicit levels (e.g., RTL characters inside an LTR paragraph get level 1; numbers level 2).\\n  - **L1\u2013L4**: per-line reversal of runs highest-level-first; **mirroring** of paired characters (parentheses, brackets) in RTL runs.\\n- Conformance data: `BidiTest.txt`, `BidiCharacterTest.txt` in the UCD [2]; reference implementations: ICU BiDi, GNU FriBidi [9].\\n\\n## 2. Base direction & the first-strong heuristic\\n\\n- Base direction governs run ordering, placement of neutrals/punctuation at run boundaries, and default alignment. **Wrong base direction is the single most common bidi failure.**\\n- **P2 first-strong rule**: scan for the first strong character (L, R, or AL), skipping text inside paired isolates so isolate contents can't leak into the outer paragraph; fallback is LTR [1]. **FSI** applies the same first-strong rule to its own contents [1].\\n- HTML `dir=auto` implements first-strong with DOM-specific rules (skips elements with explicit `dir` and `bdi` contents; uses the *value* for `input`/`textarea`; falls back to LTR) [3].\\n- **Known failure modes**: RTL text beginning with a Latin brand name, @mention, hashtag, emoji, or digits gets an LTR base incorrectly. Mitigations used in practice: strong-type **ratio counting** (L vs. R characters), per-locale/per-user defaults, or manual override [5].\\n\\n## 3. Isolates (FSI/PDI) vs. legacy embedding marks\\n\\n| Control | Code point | Class/role |\\n|---|---|---|\\n| LRM / RLM / ALM | U+200E / U+200F / U+061C | invisible strong marks (L / R / R) |\\n| LRE, RLE, PDF, LRO, RLO | U+202A\u2013U+202E | legacy embeddings/overrides (**deprecated**) |\\n| LRI, RLI, FSI, PDI | U+2066\u2013U+2069 | isolates (Unicode 6.3, 2013) [1][8] |\\n\\n- **Embeddings** (LRE/RLE) change the level of enclosed text, but the enclosed strong types still influence neutral resolution at the embedding's boundaries \u2014 content \\\"leaks\\\" outward. **Isolates** treat their content as a separate paragraph: no effect on outside ordering; the outside is neutral from the inside's perspective. Isolates nest and pair like brackets (PDI matches the nearest unmatched initiator) [1].\\n- **FSI** = auto-detecting isolate (first-strong on its contents). LRO/RLO force visual order overrides \u2014 the basis of the \\\"Trojan Source\\\" attack [6][7].\\n- Guidance: use isolates (or markup equivalents) in new code; keep LRM/RLM for boundary fixes where isolation is impossible [1][5].\\n\\n## 4. W3C / WHATWG guidance\\n\\n- **HTML**: set `dir` on `` for page direction; use `dir=auto` or `` (default `dir=auto`, UA style `unicode-bidi: isolate`) for user-generated content of unknown direction; `dir=auto` on form fields also fixes caret/alignment side [3].\\n- **CSS**: `direction` sets base direction; `unicode-bidi: isolate | embed | bidi-override | isolate-override | plaintext` (`plaintext` = per-element UBA paragraph with auto-detected base \u2014 useful for chat bubbles) [4]. Note: `direction` alone on an inline element does **not** isolate; you need `unicode-bidi: isolate` or `` [4].\\n- **W3C i18n advice**: prefer markup (`dir`/`bdi`) over Unicode control characters in HTML (more robust, survives sanitizers); in plain strings (logs, JS interpolation, databases) wrap interpolated variables (usernames, filenames, URLs) in FSI\u2026PDI; always store **logical order**, never visual order [5].\\n\\n## 5. Common bidi bugs in mixed LTR/RTL content\\n\\n1. **Wrong base direction for user content** (comments, chat): trailing punctuation jumps to the wrong end \u2192 `dir=auto` / `bdi` / `unicode-bidi: plaintext` [3][4][5].\\n2. **String interpolation leaks**: \\\"Delete ``?\\\" or \\\"File `.exe` saved\\\" in RTL locales \u2014 punctuation/extension detaches \u2192 wrap interpolations in FSI\u2026PDI or `` [5].\\n3. **URLs, paths, code in RTL text**: `/`, `:`, `.` behave as neutrals and scramble \u2192 isolate the run; render with forced LTR isolation.\\n4. **First-strong misfires** (leading @mention/emoji/digits/Latin word) \u2192 ratio heuristic or explicit override; prefix RLM when forcing [1][5].\\n5. **Bracket mismatches** across direction runs \u2192 rely on rule N0 in modern implementations [1][8].\\n6. **Truncation bugs**: ellipsis on the wrong side; cutting between FSI and PDI leaks direction into the rest of the UI \u2192 truncate on grapheme boundaries, keep control pairs balanced, re-wrap in `bdi`.\\n7. **Sanitizer/pipeline stripping** of bidi controls (e.g., removing RLM but not isolates) \u2192 text renders differently across surfaces; define an explicit policy [5][6].\\n8. **Trojan Source**: RLO/LRO in source code makes reviewed code differ from executed code \u2192 UTS #55 guidance, GCC/Clang `-Wbidi-chars`, Rust lints, VCS warnings [6][7].\\n9. **CSS mistakes**: using `text-align` instead of `direction`; `direction` without `unicode-bidi: isolate`; not mirroring layout via logical properties (`margin-inline-start`, etc.) [4].\\n10. **Terminals/logs**: many terminals implement only partial UBA (often no isolate support) \u2014 mixed-direction log lines scramble [9].\\n11. **NLP pipelines**: keep logical order end-to-end; apply the UBA only at render time; deliberately decide whether to strip bidi controls from training corpora (they survive normalization and act as spurious tokens); never reorder text before tokenization [1][5].\\n\\n## Sources\\n\\n[1] Unicode Standard Annex #9, *Unicode Bidirectional Algorithm* (unicode.org/reports/tr9)\\n[2] Unicode Character Database: `BidiTest.txt`, `BidiCharacterTest.txt`, `DerivedBidiClass` (unicode.org/Public/UCD)\\n[3] WHATWG HTML Standard: `dir` attribute, `bdi` element, auto-direction algorithms (html.spec.whatwg.org)\\n[4] CSS Writing Modes Level 3/4: `direction`, `unicode-bidi` (w3.org/TR/css-writing-modes-3)\\n[5] W3C Internationalization articles: *Inline markup and bidirectional text in HTML*; *Creating HTML pages in Arabic, Hebrew and other right-to-left scripts*; bidi-basics articles (w3.org/International/articles)\\n[6] Unicode Technical Standard #55, *Unicode Source Code Handling* (unicode.org/reports/tr55)\\n[7] Boucher & Anderson, *Trojan Source: Invisible Vulnerabilities*, arXiv:2111.00169 (2021)\\n[8] Unicode 6.3.0 and 8.0.0 release notes (isolate addition / embedding deprecation; N0 bracket rule)\\n[9] ICU BiDi and GNU FriBidi reference-implementation documentation\\n\\n---\\n\\n## Mixed-script and homoglyph detection: UTS #39 confusables and restricted scripts, Greek/Cyrillic homoglyph spoofing, CJK fullwidth/halfwidth width folding, IDN spoofing defenses, transliteration vs normalization tradeoffs\\nI'll research this topic thoroughly across the key areas: UTS #39 mechanisms, homoglyph spoofing, width folding, and IDN defenses.\\n\\n---\\n\\n## Stress-testing NLP and MT systems with noise: robustness benchmarks, noise injection methods (typos, casing, whitespace, character swaps), adversarial text augmentation tools (nlpaug, TextAttack, textflint), evaluation metrics for robustness\\n# Stress-Testing NLP & MT with Noise: Research Summary\\n\\n## 1. Robustness Benchmarks\\n\\n**MT-specific noise:**\\n- [1] Belinkov & Bisk (ICLR 2018) \u2014 canonical study showing both synthetic and natural noise severely degrade NMT; introduced synthetic noise types (keyboard typos, character scrambling, disemvoweling) and natural noise mined from Wikipedia edit histories.\\n- [2] MTNT (Michel & Neubig, EMNLP 2018) \u2014 Reddit-derived noisy test/training sets for en\u2013fr, en\u2013ja, en\u2013ko.\\n- [3] Khayrallah & Koehn (WMT 2018) \u2014 quantified damage from misspellings, ungrammatical text, alternative spellings, untokenized text in training data.\\n- [4] WMT19 Machine Translation Robustness shared task \u2014 noisy social-media translation for en\u2192fr and en\u2192ja.\\n\\n**General NLU/LLM-era:**\\n- [5] CheckList (Ribeiro et al., ACL 2020) \u2014 behavioral testing: Minimum Functionality (MFT), Invariance (INV), and Directional Expectation (DIR) tests with typos, paraphrases (round-trip MT), punctuation changes.\\n- [6] AdvGLUE (NeurIPS D&B 2021) \u2014 GLUE tasks under multiple adversarial attack algorithms; [7] ANLI (ACL 2020) \u2014 model-in-the-loop adversarial NLI; [8] Dynabench \u2014 dynamic adversarial data collection framework.\\n- [9] Robustness Gym (NAACL 2021) \u2014 unifies four paradigms: subpopulations, transformations, evaluation sets, adversarial attacks.\\n- [10] HELM \u2014 includes robustness scenarios (typos, dialect); [11] PromptBench (2023) \u2014 LLM prompt robustness at character, word, sentence, and semantic levels.\\n\\n## 2. Noise Injection Methods\\n\\n- **Character-level:** keyboard-adjacent substitutions (typos) [1][24]; adjacent character swaps/transpositions [1][23]; random insertions/deletions [24]; OCR-style confusions (l/1, O/0); homoglyphs and phonetic substitutions; HotFlip gradient-guided character flips [22]; DeepWordBug's importance-scored character edits [23]. Pruthi et al. [24] showed 1\u20132 character edits suffice to break classifiers.\\n- **Casing:** lowercasing, ALL-CAPS, mixed-case injection \u2014 tests truecasing/tokenizer assumptions; implemented in AugLy [19] and textflint [14].\\n- **Whitespace/segmentation:** word concatenation (\\\"howareyou\\\"), letter splitting (\\\"w o r d s\\\"), double spaces \u2014 breaks subword tokenization. Related training-time noise: subword regularization [25] and BPE-dropout [26] improve segmentation robustness.\\n- **Word/sentence-level:** synonym substitution (TextFooler [20]), contextual BERT-based substitution (BAE [21]), EDA's random insert/swap/delete [18], counterfactual generation (Polyjuice [17]), instruction-based augmentation (NLAugmenter [16]).\\n- **Natural noise sources:** Wikipedia edit histories [1], Reddit (MTNT) [2], Twitter/social media [4].\\n\\n## 3. Tools\\n\\n- **nlpaug** [12] \u2014 character augmenters (KeyboardAug, OcrAug, RandomCharAug with insert/delete/swap/substitute), word augmenters (WordNet synonyms, BERT/GPT contextual, spelling, word splitting), back-translation; also covers audio.\\n- **TextAttack** [13] \u2014 attack framework with four pluggable modules (goal function, constraints, transformation, search); ~16 attack recipes (TextFooler, DeepWordBug, BAE, HotFlip); transformations include QWERTY swaps, homoglyphs, character insertion/deletion; constraints via USE/BERTScore similarity, POS consistency; includes augmentation CLI and adversarial training.\\n- **textflint** [14] \u2014 unified robustness toolkit: 58 transformations + 20 subpopulations across 12 tasks (sentiment, NLI, STS, NER, POS, AMR-to-text, semantic parsing, math word problems); English and Chinese; generates test data + reports.\\n- **Others:** OpenAttack [15] (bilingual EN/ZH attack toolkit with perplexity/grammar/edit-distance metrics), NLAugmenter [16], AugLy [19] (casing, homoglyphs, unicode confusables, typo simulation), Smoke Tester [27] (unified re-implementation for comparing attack effectiveness).\\n\\n## 4. Evaluation Metrics\\n\\n- **Performance delta:** \u0394accuracy/F1 (NLU) or \u0394BLEU/\u0394chrF/\u0394COMET (MT) between clean and perturbed sets, reported per noise type [1][2][4].\\n- **Behavioral pass rates:** MFT/INV/DIR percentages [5].\\n- **Attack metrics:** attack success rate, perturbation/modification rate (% tokens changed), queries per success, post-attack accuracy [13][15][27].\\n- **Noise quality controls:** semantic similarity (USE, BERTScore), grammaticality, GPT-2 perplexity of generated text \u2014 critical since many attacks produce ungrammatical output, inflating apparent vulnerability [27].\\n- **MT hygiene:** standardized sacreBLEU [28], chrF, TER, COMET for neural adequacy [29]; human adequacy/fluency judgment in robustness tasks [4].\\n- **Statistics:** paired bootstrap / approximate randomization for deltas; fixed seeds for stochastic noise.\\n\\n## 5. Key Caveats\\n\\n- Robustness to synthetic noise does **not** transfer to natural noise [1] \u2014 validate with real noisy data [2][4].\\n- Tokenizer artifacts can dominate casing/whitespace results; test both case-sensitive/insensitive configurations [25][26].\\n- For LLMs, benchmark contamination is a risk; PromptBench/HELM offer current starting points [10][11].\\n\\n**Practical recipe:** freeze a clean eval set \u2192 generate seeded perturbed copies (multiple noise types/severities) via nlpaug/textflint \u2192 report clean-vs-noisy deltas + invariance rates \u2192 for adversarial attacks, report ASR with similarity/grammar constraints \u2192 significance-test all deltas.\\n\\n---\\n\\n## References\\n\\n[1] Belinkov & Bisk, \\\"Synthetic and Natural Noise Both Break Neural Machine Translation,\\\" ICLR 2018\\n[2] Michel & Neubig, \\\"MTNT: A Testbed for Machine Translation of Noisy Text,\\\" EMNLP 2018\\n[3] Khayrallah & Koehn, \\\"On the Impact of Various Types of Noise on Neural Machine Translation,\\\" WMT 2018\\n[4] Li et al., \\\"Findings of the WMT 2019 Shared Task on Machine Translation Robustness,\\\" WMT 2019\\n[5] Ribeiro et al., \\\"Beyond Accuracy: Behavioral Testing of NLP Models with CheckList,\\\" ACL 2020\\n[6] Wang et al., \\\"Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of NLP Models,\\\" NeurIPS D&B 2021\\n[7] Nie et al., \\\"Adversarial NLI: A New Benchmark for Natural Language Understanding,\\\" ACL 2020\\n[8] Kiela et al., \\\"Dynabench: Rethinking Benchmarking in NLP,\\\" NAACL 2021\\n[9] Goel et al., \\\"Robustness Gym: Unifying the NLP Evaluation Landscape,\\\" NAACL 2021\\n[10] Liang et al., \\\"Holistic Evaluation of Language Models,\\\" 2022\\n[11] Zhu et al., \\\"PromptBench: Towards Evaluating the Robustness of LLMs on Adversarial Prompts,\\\" 2023\\n[12] Ma, \\\"nlpaug: A Python Library for Textual and Audio Data Augmentation,\\\" GitHub 2019\\n[13] Morris et al., \\\"TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP,\\\" ACL 2020 (demo)\\n[14] Wang et al., \\\"TextFlint: Unified Multilingual Robustness Evaluation Toolkit for NLP,\\\" 2021\\n[15] Zeng et al., \\\"OpenAttack: An Open-source Text Adversarial Attack Toolkit,\\\" ACL 2021 (demo)\\n[16] Dhole et al., \\\"NLAugmenter: Composition-Based Contextual Augmentation for NLP,\\\" ACL 2021 (demo)\\n[17] Wu et al., \\\"Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models,\\\" ACL 2021\\n[18] Wei & Zou, \\\"EDA: Easy Data Augmentation Techniques for Text Classification,\\\" EMNLP 2019\\n[19] Bitton et al., \\\"AugLy: A Data Augmentation Library for Text, Images, and Audio,\\\" 2021\\n[20] Jin et al., \\\"Is BERT Really Robust? (TextFooler),\\\" AAAI 2020\\n[21] Garg & Ramakrishnan, \\\"BAE: BERT-based Adversarial Examples for Text Classification,\\\" EMNLP 2020\\n[22] Ebrahimi et al., \\\"HotFlip: White-Box Adversarial Examples for Text Classification,\\\" ACL 2018\\n[23] Gao et al., \\\"DeepWordBug: Black-Box Generation of Adversarial Text Sequences,\\\" 2019\\n[24] Pruthi et al., \\\"Combating Adversarial Misspellings with Robust Word Recognition,\\\" ACL 2019\\n[25] Kudo, \\\"Subword Regularization: Improving NMT Models with Multiple Subword Segmentation,\\\" ACL 2018\\n[26] Provilkov et al., \\\"BPE-Dropout: Simple and Effective Subword Regularization,\\\" ACL 2020\\n[27] Gr\u00f6ndahl & Asokan, \\\"Smoke Tester: A Framework for Evaluating Adversarial Text Attack Effectiveness,\\\" 2020\\n[28] Post, \\\"A Call for Clarity in Reporting BLEU Scores,\\\" WMT 2018\\n[29] Rei et al., \\\"COMET: A Neural Framework for MT Evaluation,\\\" EMNLP 2020\\n[30] Zhang et al., \\\"Adversarial Attacks on Deep Learning Models in NLP: A Survey,\\\" ACM TIST 2020\\n\\n---\\n\\n## Code-switching stress tests for multilingual NLP: Hinglish/Spanglish/Arabic-French datasets and benchmarks, transliteration code-switching, MT and LLM performance on mixed-language text, evaluation approaches\\nI'll research this topic systematically, starting with broad searches on code-switching benchmarks and then drilling into specific language pairs, transliteration, MT/LLM performance, and evaluation methods.\",\n    \"t_start\": \"2026-09-26T00:09:01.855313+00:00\",\n    \"t_end\": \"2026-09-26T00:19:26.550663+00:00\",\n    \"index\": 11\n  },\n  {\n    \"step\": \"0010\",\n    \"tool_call_id\": \"call_-7227534681825083037\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"UTS #39 Unicode security mechanisms: confusables.txt, restricted/recommended scripts per identifier profile, mixed-script detection algorithm, Greek/Cyrillic homoglyph spoofing examples, whole-script confusables, remapping vs blocking policy\",\n        \"CJK fullwidth/halfwidth width folding (U+FF01-FF5E), NFKC width folding in Japanese/Chinese localization, IDN homograph attacks and defenses (UTS #46 transitional vs nontransitional processing, browser protections)\",\n        \"Code-switching NLP benchmarks and datasets: LinCE, GLUE-CoS, Hinglish TOP, Spanglish/SENTIMENT-CS, Arabic dialect code-switching; transliterated code-switching (Romanized Hindi/Bengali); MT and LLM performance on code-switched text and evaluation metrics\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## UTS #39 Unicode security mechanisms: confusables.txt, restricted/recommended scripts per identifier profile, mixed-script detection algorithm, Greek/Cyrillic homoglyph spoofing examples, whole-script confusables, remapping vs blocking policy\\nResearch on 'UTS #39 Unicode security mechanisms: confusables.txt, restricted/recommended scripts per identifier profile, mixed-script detection algorithm, Greek/Cyrillic homoglyph spoofing examples, whole-script confusables, remapping vs blocking policy' failed: Connection error.\\n\\n---\\n\\n## CJK fullwidth/halfwidth width folding (U+FF01-FF5E), NFKC width folding in Japanese/Chinese localization, IDN homograph attacks and defenses (UTS #46 transitional vs nontransitional processing, browser protections)\\n# CJK Width Folding, NFKC Localization Issues, and IDN Homograph Defenses\\n\\n## 1. Halfwidth/Fullwidth Forms (U+FF00\u2013FFEF) and width folding\\n\\n- The block **U+FF01\u2013FF5E** contains **fullwidth variants of ASCII U+0021\u2013007E**, each carrying a `` compatibility decomposition (e.g., \uff21 U+FF21 \u2192 A U+0041, \uff0cU+FF0C \u2192 , U+002C) [1]. The rest of the block: **U+FF61\u2013FF9F** halfwidth ideographic punctuation and katakana (\uff61\uff62\uff63\uff8a\uff9e, halfwidth dakuten \uff9e U+FF9E), **U+FFA0\u2013FFDC** halfwidth Hangul compatibility jamo, **U+FFE0\u2013FFE6** fullwidth signs (\uffe0\uffe1\uffe6 etc.) [1].\\n- These exist because legacy East Asian encodings (Shift_JIS/JIS X 0208, GB2312/GBK, KS X 1001) encoded both widths; Unicode round-trip compatibility requires both [1][2].\\n- **East Asian Width property (UAX #11)**: fullwidth forms are class F, halfwidth kana/Hangul class H, ASCII class Na, with many CJK-context characters \\\"Ambiguous\\\" (A) \u2014 rendered 1 or 2 cells depending on locale [2].\\n- **Width folding** = mapping fullwidth\u2192ASCII/halfwidth via **NFKC/NFKD** (UAX #15), **NFKC_Casefold** for caseless matching, or ICU's dedicated `Halfwidth`/`Fullwidth` transforms [3]. Note NFKC folds *both directions per legacy mapping*: fullwidth Latin\u2192ASCII, but halfwidth katakana \uff8a\u2192\u30cf (fullwidth), and \uff9e\u2192combining U+3099 [1][3].\\n\\n## 2. NFKC folding in Japanese/Chinese localization\\n\\n- **What folds**: \uff21\uff22\uff23\u2192ABC, \uff11\uff12\uff13\u2192123, \uff0c\uff1f\uff01\u2192,?! (U+FF01\u2013FF5E), fullwidth space U+3000\u2192U+0020, halfwidth kana\u2192fullwidth kana [1][3].\\n- **What does not fold**: ideographic punctuation \u3001U+3001 and \u3002U+3002 (no compatibility decompositions) \u2014 so NFKC is *not* a pure width fold for CJK punctuation [1][3].\\n- **Over-folding pitfalls**: NFKC also destroys meaningful distinctions \u2014 \u2460\u21921, \u3231\u2192(\u682a), \u33a1\u2192m2, Roman numerals \u2163\u2192IV, halfwidth\u2192fullwidth kana changes string length and visual style [3]. Japanese/Chinese search and matching systems therefore typically use *custom* width-only folding (ASCII width + kana width) rather than raw NFKC; IMEs default to fullwidth output, making \u534a\u89d2/\u5168\u89d2 normalization a core JA/ZH localization requirement [3][14].\\n- Identifiers: UAX #31 mandates NFKC normalization for identifiers (adopted by Python via PEP 3131), which folds fullwidth homoglyphs in code [14].\\n\\n## 3. IDN homograph attacks\\n\\n- IDNs encode non-ASCII labels as Punycode (`xn--\u2026`, RFC 3492) under IDNA2003 (RFC 3490/3491, Nameprep with NFKC) or IDNA2008 (RFC 5890\u20135894) [6][7].\\n- **Attack**: register visually confusable domains using cross-script lookalikes (Cyrillic \u0430/\u0435/\u043e, Greek \u03bf) or width/compatibility variants. Formalized by Gabrilovich & Gontmakher (CACM 2002) [11]; the 2005 \\\"Shmoo Group\\\" paypal.com demo forced early mitigations; in 2017 Xudong Zheng's Cyrillic **\u0430\u0440\u0440\u04cf\u0435.com** (`xn--80ak6aa92e`) rendered as \\\"apple.com\\\" in unpatched browsers [12]. Related: typosquatting/combosquatting and homoglyph toolkits (dnstwist, EvilURL) [12][16].\\n- **Key structural defense**: IDNA2008 makes all compatibility characters (including U+FF01\u2013FF5E) **DISALLOWED** in registered labels, and UTS #46 maps fullwidth input to ASCII \u2014 so \uff50\uff41\uff59\uff50\uff41\uff4c.com cannot exist as a distinct .com domain [5][7].\\n\\n## 4. UTS #46: transitional vs nontransitional\\n\\n- **UTS #46** bridges IDNA2003\u2194IDNA2008 with a mapping table (statuses: `valid`, `mapped`, `ignored`, `disallowed_STD3_valid/mapped`, `disallowed`; `UseSTD3ASCIIRules` enforces LDH-only ASCII) [5].\\n- **Four \\\"deviation\\\" characters** differ between modes: **\u00df (U+00DF), \u03c2 (U+03C2), ZWNJ (U+200C), ZWJ (U+200D)** [5]:\\n  - *Transitional* (IDNA2003-compatible): maps \u00df\u2192ss, \u03c2\u2192\u03c3, deletes ZWJ/ZWNJ \u2014 makes fa\u00df.de \u2261 fass.de (confusable).\\n  - *Nontransitional* (IDNA2008-compatible): keeps \u00df and \u03c2 valid (distinct domains), ZWJ/ZWNJ valid under ContextJ rules.\\n- **Fullwidth folding happens in both modes** (`mapped` status); only the deviation characters differ [5].\\n- **Browser adoption of nontransitional**: Chrome ~2014, Safari ~2016\u201317, **Firefox 59 (March 2018)** \u2014 the last major engine to switch; IE/legacy Edge never did. This is why \u00df-domains (e.g., fa\u00df.de) now display distinctly [5][10].\\n\\n## 5. Defenses in practice\\n\\n- **Browsers** apply UTS #46 for parsing *plus* spoof checks, showing Punycode on failure [9][10]:\\n  - **Chrome**: Unicode display only for single-script labels (with limited exceptions, e.g., Han+Hiragana+Katakana, Han+Latin), a blocked-character list, and UTS #39 confusable checks (including against popular TLDs/domains) [9].\\n  - **Firefox**: mixed-script confusable detection per UTS #39, script-mixing rules, plus user pref `network.IDN_show_punycode` [10].\\n  - All major browsers display `xn--80ak6aa92e` punycode for the 2017 apple.com lookalike [12].\\n- **UTS #39** supplies the `skeleton()` confusability algorithm and RestrictionLevel; **UTS #36** catalogs the threat model [4][16].\\n- **ICANN/registries**: IDN Guidelines require published IDN tables (per-script code point sets), variant/confusable reservations, and the Maximal Starting Repertoire; registries bundle variants (simplified/traditional Chinese, kanji variants in .jp, \u00df\u2194ss bundling at DENIC) [8][13].\\n- **Ecosystem**: password-manager auto-fill keyed to exact ASCII domains is a strong user-side mitigation; SSAC advisories (e.g., SAC 045 on homoglyphs) guide registry policy [13].\\n\\n## Sources\\n\\n[1] Unicode Character Database / The Unicode Standard, ch. 22 (compatibility decompositions, U+FF00\u2013FFEF)\\n[2] UAX #11, *East Asian Width Property*\\n[3] UAX #15, *Unicode Normalization Forms* (NFKC, NFKC_Casefold); ICU User Guide (Halfwidth/Fullwidth transforms)\\n[4] UTS #39, *Unicode Security Mechanisms*\\n[5] UTS #46, *Unicode IDNA Compatibility Processing* (IdnaMappingTable.txt, deviation characters)\\n[6] RFC 3490/3491 (IDNA2003, Nameprep); RFC 3492 (Punycode)\\n[7] RFC 5890\u20135894 (IDNA2008); RFC 5895 (mapping guidance)\\n[8] ICANN, *IDN Guidelines* (current version) and Maximal Starting Repertoire\\n[9] Chromium, *IDN Guidance* (chromium.org design doc)\\n[10] Mozilla nsIDNService implementation; Firefox 59 release notes (nontransitional switch); `network.IDN_show_punycode` pref\\n[11] Gabrilovich & Gontmakher, \\\"The Homograph Attack,\\\" *CACM* 45(2), 2002\\n[12] X. Zheng, \\\"Phishing with Unicode Domains\\\" (2017); 2005 Shmoo Group IDN demo\\n[13] ICANN SSAC, SAC 045, *Advisory on IDN Homoglyphs*\\n[14] PEP 3131 / PEP 672 (Python identifier NFKC normalization; Unicode security considerations)\\n[15] UAX #31, *Identifier and Pattern Syntax*\\n[16] UTS #36, *Unicode Security Considerations*\\n\\n*Note: browser-switch dates for Chrome/Safari are approximate; the Firefox 59 (2018) date is firm. No live retrieval was performed \u2014 citations are to canonical standards and primary documents.*\\n\\n---\\n\\n## Code-switching NLP benchmarks and datasets: LinCE, GLUE-CoS, Hinglish TOP, Spanglish/SENTIMENT-CS, Arabic dialect code-switching; transliterated code-switching (Romanized Hindi/Bengali); MT and LLM performance on code-switched text and evaluation metrics\\n# Code-Switching (CS) NLP: Benchmarks, Datasets, Models, and Metrics\\n\\n## 1. Broad multi-task benchmarks\\n\\n**LinCE** [1] (ICLR 2020) \u2014 the most widely used CS benchmark. Four language pairs (Spanish\u2013English, Hindi\u2013English, Mandarin\u2013English, Nepali\u2013English) plus Singlish, across five tasks: token-level language identification (LID), POS tagging, NER, sentiment analysis, and semantic parsing (Spanglish SCoRe data). It aggregates earlier shared-task corpora (e.g., Spanglish 2014 [4], SentiMix [5], SENTIMENT-CS [6]) plus new annotations. Key result: multilingual encoders (mBERT, XLM-R) fine-tuned on monolingual data show large drops on CS text vs. monolingual test sets, worsening as mixing density increases.\\n\\n**GLUE-CoS** [2] (2020) \u2014 GLUE-style CS benchmark with four tasks: natural language inference, sentiment analysis, semantic relatedness, and language identification, over pairs mixing English with Spanish, Hindi, Chinese, etc. Combines real CS data with artificially code-switched English (word-level substitution via translations). Code-switched pretraining (CSP [17]) substantially improves over mBERT/XLM-R baselines.\\n\\n## 2. Task-specific datasets\\n\\n- **Spanglish corpus (2014 shared task)** [4]: ~2.7k tweets with token-level LID, POS, and NER annotations (Spanish\u2013English); the seed of most later Spanglish resources.\\n- **SENTIMENT-CS** [6] and **SemEval-2020 Task 9 (SentiMix)** [5]: sentiment classification of code-mixed tweets for Hindi\u2013English (SENTIMENT-CS) and Spanish\u2013English (SentiMix); thousands of tweets each; F1-based evaluation. These supply LinCE's sentiment subsets.\\n- **Hinglish TOP** [3] (EMNLP 2021): semantic parsing (intent + slot filling) in Hinglish, built by re-annotating the English TOP query-understanding dataset into natural code-mixed Hindi\u2013English. Multilingual models lag English TOP performance by a wide margin; the first large CS semantic-parsing resource.\\n- **CMU Hinglish DoG**: code-switched Hinglish dialogues derived from the CMU-DoG corpus, used in CS dialogue generation work.\\n\\n## 3. Arabic dialect code-switching\\n\\n- Arabic CS is dominated by (a) diglossic mixing between MSA and dialects (esp. Egyptian) and (b) Arabic\u2013English/French mixing in social media, frequently written in **Arabizi** (Romanized Arabic) [12].\\n- The main token-level resource is the **MSA\u2013Egyptian** pair from the CoSWo 2016 shared task [7], built on CALLHOME Egyptian Arabic telephone transcripts [9].\\n- Dedicated Arabic CS benchmarks remain scarce and fragmented; most dialectal evaluation uses adjacent dialect-ID resources \u2014 MADAR (26 city dialects) [10] and NADI [11] \u2014 rather than true code-switching. Maghrebi (Darija\u2013French/English) mixing is especially under-resourced, and no standard Arabic CS benchmark with LinCE-level adoption exists.\\n\\n## 4. Transliterated (Romanized) code-switching\\n\\n- Roman-script CS (Romanized Hindi/Bengali + English) dominates informal writing but is under-served: most benchmarks are in native or mixed script, creating domain mismatch.\\n- **PHINC** [13] (2020): parallel Romanized-Hinglish \u2194 English Instagram-chat corpus (tens of thousands of pairs) for MT and transliteration; MT baselines on it remain modest.\\n- **HingCorpus / HingBERT** [14] (LREC 2022): large Romanized-Hinglish social-media corpus and BERT-family models (HingBERT, HingMBERT, HingRoBERTa) that beat mBERT on Hinglish tasks.\\n- **Bengali\u2013English (Banglish)**: token-level LID data from the CALCS 2018 shared task [8]; several Banglish sentiment datasets exist, but MT and structured-task data are very scarce.\\n- **Dakshina** [15] (2020): romanized \u2194 native-script parallel text for 12 South Asian languages \u2014 not CS per se, but the standard resource for Romanized Indic text.\\n- **RomanSetu** [16] (2024): shows multilingual LLMs underperform on Romanized Indic text (tokenizer fragmentation, script mismatch) and proposes romanization-aware adaptation that recovers much of the gap.\\n\\n## 5. MT and LLM performance on CS text\\n\\nReplicated findings:\\n- Pre-LLM systems (SMT, NMT, zero-shot mBART/NLLB) degrade substantially on CS input; errors cluster at switch points, and LID errors cascade into translation/parsing errors [1][2][17][18].\\n- Code-switched pretraining (CSP [17]) and fine-tuning on CS parallel data (e.g., PHINC for Hinglish\u2192English [13]) yield large gains; transliterating Romanized input to native script before processing also helps [16].\\n- Instruction-tuned LLMs are stronger on high-resource pairs (Spanish\u2013English) and sentence-level tasks, but (a) still trail monolingual performance, (b) tend to \\\"normalize\\\" CS input into monolingual output, (c) degrade sharply on Romanized Hinglish/Banglish, and (d) remain behind fine-tuned encoders on token-level tasks (LID/POS/NER) [1][2][16][18].\\n- Prompting mitigations reported: few-shot exemplars, transliteration of input, explicit instructions to preserve switching.\\n\\n## 6. Evaluation metrics\\n\\n- **Task metrics**: token-level accuracy/F1 (LID, POS, NER); F1 for sentiment (SentiMix [5]); exact-match accuracy for semantic parsing (Hinglish TOP [3]); BLEU/chrF/COMET for MT.\\n- **CS-structural metrics** (used to characterize data and bin error analysis): Code-Mixing Index (CMI) [19], I-index [20], language entropy [22], switch-point density [21], burstiness. A robust finding: performance falls monotonically as mixing density rises.\\n- **Gaps**: no standard CS-specific adequacy metric for MT (BLEU/COMET reliability on CS output is under-studied); benchmarks are small and static (overfitting risk); Romanized variants are rarely included in test sets.\\n\\n## References\\n\\n[1] Winata et al. (2020), \\\"LinCE: A Linguistic Code-switching Evaluation Benchmark for NLP,\\\" ICLR 2020. [2] Winata et al. (2020), \\\"GLUE-CoS: GLUE Benchmark for Code-Switching,\\\" arXiv:2010.12609. [3] \\\"Hinglish TOP: A Dataset for Hinglish Semantic Parsing,\\\" EMNLP 2021. [4] Solorio et al. (2014), \\\"Overview for the First Shared Task on Language Identification in Code-Switched Data,\\\" EMNLP 2014 CS Workshop. [5] Patwa et al. (2020), \\\"SemEval-2020 Task 9: Sentiment Analysis of Code-Mixed Tweets.\\\" [6] Patwa et al. (2019), SENTIMENT-CS corpus (WILDRE, EMNLP 2019). [7] Molina et al. (2016), CoSWo@COLING 2016 shared task overview. [8] CALCS 2018 shared task overview, CALCS@ACL 2018. [9] CALLHOME Egyptian Arabic, LDC97S45. [10] Bouamor et al. (2019), MADAR, WANLP 2019. [11] Abdul-Mageed et al. (2020), NADI shared task. [12] Darwish & Mubarak (2016), Arabizi detection/normalization. [13] Mukherjee et al. (2020), \\\"PHINC: A Parallel Hinglish-English Instagram Chat Corpus.\\\" [14] Nayak & Joshi (2022), \\\"HingBERT: A Code-Mixed Language Model for Hinglish,\\\" LREC 2022. [15] Roark et al. (2020), \\\"Processing the Dakshina Dataset.\\\" [16] \\\"RomanSetu: Efficiently Adapting Multilingual Models via Romanization,\\\" 2024. [17] Yin et al. (2020), CSP code-switched pretraining, EMNLP 2020. [18] Winata et al. (2021), \\\"Are Multilingual Models Effective in Code-Switching?\\\" LREC 2021. [19] Gamb\u00e4ck & Das (2013), Code-Mixing Index. [20] Guzm\u00e1n et al. (2017), I-index. [21] Solorio & Liu (2008), \\\"Learning to Predict Code-Switching Points,\\\" EMNLP 2008. [22] Poplack (1980), language entropy.\\n\\n*Caveat: venue/size details reflect the canonical papers as I recall them; verify exact numbers against the cited sources before citing onward.*\",\n    \"t_start\": \"2026-09-26T00:19:49.896943+00:00\",\n    \"t_end\": \"2026-09-26T00:36:54.965620+00:00\",\n    \"index\": 12\n  },\n  {\n    \"step\": \"0010\",\n    \"tool_call_id\": \"call_-7227494206053278546\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"UTS #39 Unicode Security Mechanisms: skeleton() algorithm, confusables.txt and intentional.txt data files, RestrictionLevel (Highly Restricted, Moderately Restricted, Minimally Restricted, Unrestricted), mixed-script confusable detection, whole-script confusables, identifier profile recommendations, remapping vs blocking policy\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## UTS #39 Unicode Security Mechanisms: skeleton() algorithm, confusables.txt and intentional.txt data files, RestrictionLevel (Highly Restricted, Moderately Restricted, Minimally Restricted, Unrestricted), mixed-script confusable detection, whole-script confusables, identifier profile recommendations, remapping vs blocking policy\\nResearch on 'UTS #39 Unicode Security Mechanisms: skeleton() algorithm, confusables.txt and intentional.txt data files, RestrictionLevel (Highly Restricted, Moderately Restricted, Minimally Restricted, Unrestricted), mixed-script confusable detection, whole-script confusables, identifier profile recommendations, remapping vs blocking policy' failed: Connection error.\",\n    \"t_start\": \"2026-09-26T00:37:03.994821+00:00\",\n    \"t_end\": \"2026-09-26T00:54:07.608791+00:00\",\n    \"index\": 13\n  },\n  {\n    \"step\": \"0010\",\n    \"tool_call_id\": \"call_-7227502967786569353\",\n    \"title\": \"research_subtopic\",\n    \"kind\": \"tool\",\n    \"input\": {\n      \"subtopics\": [\n        \"Unicode Technical Standard 39 security: how skeleton algorithm and confusables data work, identifier restriction levels, detecting mixed-script spoofing with Greek Cyrillic homoglyphs, best practices for handling confusable characters in user identifiers\",\n        \"Detecting and mitigating invisible character and bidi spoofing attacks in text pipelines: Trojan Source attack, RLO/LRO overrides, UTS #55 source code handling, compiler warnings, GitHub invisible character warnings, OpenAI/LLM prompt injection via hidden Unicode characters\"\n      ]\n    },\n    \"status\": \"completed\",\n    \"output\": \"## Unicode Technical Standard 39 security: how skeleton algorithm and confusables data work, identifier restriction levels, detecting mixed-script spoofing with Greek Cyrillic homoglyphs, best practices for handling confusable characters in user identifiers\\n# UTS #39 \u2014 Unicode Security Mechanisms: Research Summary\\n\\nUTS #39 defines mechanisms for detecting visually confusable strings (\\\"spoofing\\\") and restricting identifier content. It is updated with each annual Unicode release (most recently tracking Unicode 17.0, Sept 2025 \u2014 verify the current version). Note: I'm answering from canonical source knowledge; confirm details against the live spec.\\n\\n## 1. The Skeleton Algorithm and Confusables Data [1][2]\\n\\n**Skeleton computation** (UTS #39 \u00a74). The skeleton of a string X is:\\n\\n1. Convert X to **NFD** (canonical decomposition) [9]\\n2. Map each character through the **confusables mapping** (confusables.txt)\\n3. Apply **CaseFolding** (full case folding)\\n4. Convert to **NFD** again\\n\\nSo: `skeleton(X) = NFD(casefold(map(NFD(X))))`. Two identifiers are *confusable* if and only if their skeletons are identical.\\n\\n**confusables.txt** [2] format:\\n```\\n0441 ;  0063 ;  MA   # ( \u0441 \u2192 c ) CYRILLIC SMALL LETTER ES \u2192 LATIN SMALL LETTER C\\n```\\nMapping types:\\n- **MA** \u2014 mixed-script confusable; **SA** \u2014 single-script confusable (apply regardless of case)\\n- **SL** / **ML** \u2014 single/mixed-script confusables that apply only after lowercasing (relevant for case-insensitive matching; ICU's `ANY_CASE` option controls this) [7]\\n\\n**intentional.txt** [3] lists characters whose confusable mapping is deliberate (fullwidth forms, mathematical alphanumerics, compatibility characters). Data is glyph-shape based, reviewed across fonts, and revised annually \u2014 it is deliberately conservative and does **not** cover everything (e.g., `rn` \u2248 `m`, font-specific shapes).\\n\\n## 2. Identifier Restriction Levels [1][4][5]\\n\\nImplemented via `Identifier_Status` (Allowed/Restricted) and `Identifier_Type` (Inclusion, Recommended, Technical, Exclusion, Not_XID, etc.) in DerivedCoreProperties.txt [4], plus the Recommended-script list in UAX #31 [5]:\\n\\n| Level | Rule |\\n|---|---|\\n| **Inclusion** | Only the small curated \\\"Inclusion\\\" set (essentially ASCII plus a few safe characters, e.g., middle dot) \u2014 highest security |\\n| **Highly Restrictive** | Recommended scripts, but multiple scripts only within a *script group* ({Han, Bopomofo}, {Han, Hiragana, Katakana}, {Hangul}); each other script alone |\\n| **Moderately Restrictive** | Latin may mix with other Recommended scripts **except Cyrillic, Greek, and Cherokee** (Cherokee's syllabary borrows Latin glyph shapes) |\\n| **Minimally Restrictive** | Arbitrary mixing of Recommended scripts (e.g., Latin + Cyrillic allowed) |\\n| **Unrestricted** | Any characters |\\n\\nScript mixing is evaluated using *augmented script sets* (ScriptExtensions.txt) [6] so Common characters (digits, punctuation) don't falsely trigger \\\"mixed-script.\\\"\\n\\n## 3. Detecting Mixed-Script Spoofing (Greek/Cyrillic Homoglyphs) [1][7]\\n\\n**Key homoglyph sets** (all mapped in confusables.txt):\\n- Cyrillic \u2192 Latin: \u0430\u2192a, \u0441\u2192c, \u0435\u2192e, \u043e\u2192o, \u0440\u2192p, \u0445\u2192x, \u0443\u2192y, \u0456\u2192i, \u0455\u2192s, \u0458\u2192j, \u043a\u2192k, \u043c\u2192m, \u0442\u2192t\\n- Greek \u2192 Latin: \u03bf\u2192o, \u03b1\u2192a, \u03b5\u2192e, \u03b9\u2192i, \u03ba\u2192k, \u03bd\u2192v, \u03c1\u2192p, \u03c4\u2192t, \u03c5\u2192u, \u03c7\u2192x\\n- Cyrillic \u2194 Greek directly: \u0456 (U+0456) \u2248 \u03b9 (U+03B9), etc.\\n\\n**Detection procedure:**\\n1. Normalize (NFKC_Casefold per UAX #31 identifier profile) [5][9]\\n2. Resolve per-character script sets; if their intersection is empty \u2192 **mixed-script identifier**\\n3. Compute the skeleton\\n4. Classify:\\n   - **Mixed-script confusable**: identifier is mixed-script but its skeleton is single-script (e.g., `\u0430dmin` with Cyrillic \u0430 + Latin dmin \u2192 skeleton `admin` is pure Latin)\\n   - **Whole-script confusable**: identifier is single-script but its skeleton is single-script in a *different* script (e.g., all-Cyrillic `\u0441\u043e\u0440\u0435` \u2248 Latin `cope`)\\n5. Pairwise: `skeleton(candidate) == skeleton(existing)` \u2192 block or verify\\n\\nICU's SpoofChecker exposes these as `MIXED_SCRIPT_CONFUSABLE`, `WHOLE_SCRIPT_CONFUSABLE`, `SINGLE_SCRIPT_CONFUSABLE`, plus `INVISIBLE` (zero-width/combining-mark tricks), `RESTRICTION_LEVEL`, and `CHAR_LIMIT` checks; `getSkeleton()` and `areConfusable()` are the core APIs [7]. ICU requires normalized input.\\n\\n**Worked example:** candidate `\u03c1aypal` (Greek rho U+03C1). Scripts = {Greek, Latin} \u2192 mixed-script; skeleton = `paypal` \u2192 single-script \u2192 mixed-script confusable \u2192 reject, or flag because skeleton matches the existing account.\\n\\n## 4. Best Practices for User Identifiers [1][5][7][8]\\n\\n1. **Normalize canonically**: store and compare NFKC_Casefold forms; use full CaseFolding, not ASCII lowercase.\\n2. **Enforce a restriction level** at creation: Highly Restrictive is a good default for global user-facing services; Inclusion/ASCII for high-assurance or monolingual services; Moderate for programming-language identifiers.\\n3. **Skeleton-index check at registration**: persist `skeleton(username)` in an indexed column; on signup, compare candidate skeleton against all existing ones; on match, reject or require out-of-band verification (email/phone).\\n4. **Reject invisible/spoofing characters**: default-ignorables, zero-width joiners, suspicious mark sequences (ICU `INVISIBLE`).\\n5. **Keep data current**: confusables.txt changes every Unicode release; re-validate stored identifiers after upgrades.\\n6. **Scope scripts to your audience**: allow Cyrillic/Greek for relevant locales, but never mixed with Latin under Highly/Moderately Restrictive.\\n7. **UX mitigations**: warn users about mixed-script names at creation; annotate or inspect-render names in feeds; use fonts that differentiate homoglyphs; pair with verified badges and impersonation reporting.\\n8. **Don't silently mutate** \u2014 reject with a clear error instead.\\n9. **Know the limits**: skeleton matching is approximate and font-dependent; supplement with policy (UTR #36) [8], monitoring, and rate-limiting lookalike registrations.\\n\\n## Sources\\n[1] UTS #39, *Unicode Security Mechanisms* \u2014 unicode.org/reports/tr39/\\n[2] confusables.txt \u2014 unicode.org/Public/security/latest/confusables.txt\\n[3] intentional.txt \u2014 unicode.org/Public/security/latest/intentional.txt\\n[4] DerivedCoreProperties.txt (Identifier_Status/Identifier_Type) \u2014 unicode.org/Public/UCD/latest/ucd/DerivedCoreProperties.txt\\n[5] UAX #31, *Identifier and Pattern Syntax* \u2014 unicode.org/reports/tr31/\\n[6] UAX #24, *Script Property* / ScriptExtensions.txt \u2014 unicode.org/reports/tr24/\\n[7] ICU Spoof Checker (ICU4C `uspoof` / ICU4J `SpoofChecker`) \u2014 unicode.org/reports/tr39/#ICU (userguide: strings/spoofchecker)\\n[8] UTR #36, *Unicode Security Considerations* \u2014 unicode.org/reports/tr36/\\n[9] UAX #15, *Normalization Forms* \u2014 unicode.org/reports/tr15/\\n\\n---\\n\\n## Detecting and mitigating invisible character and bidi spoofing attacks in text pipelines: Trojan Source attack, RLO/LRO overrides, UTS #55 source code handling, compiler warnings, GitHub invisible character warnings, OpenAI/LLM prompt injection via hidden Unicode characters\\n# Invisible Character & Bidi Spoofing: Detection and Mitigation in Text Pipelines\\n\\n*Note: no live search tool was available for this query; the summary below reflects primary sources documented through early 2025, indexed at the end.*\\n\\n## 1. The Trojan Source attack\\n\\n- **Core mechanism:** Unicode bidirectional (bidi) control characters embedded in source code (comments, string literals, or identifiers) cause the *visual* rendering to differ from the *compiled/executed* semantics \u2014 e.g., an \\\"early return\\\" attack where malicious code appears to be inside a comment, or a string whose characters display in reversed order via RLO (U+202E) [1][2].\\n- **Publication:** Boucher & Anderson (Cambridge), *\\\"Trojan Source: Invisible Vulnerabilities,\\\"* arXiv:2111.00169 (Nov 2021), later USENIX Security 2023 [1]. Coordinated disclosure to ~20 vendors; most patched within weeks [2].\\n- **CVEs:** CVE-2021-42574 (bidi spoofing in source) and CVE-2021-42694 (homoglyph identifier spoofing, e.g., Cyrillic \u0430 U+0430 vs. Latin a) [3][4].\\n- **Blast radius:** Nearly all major compilers/interpreters accepted bidi characters in comments and strings, making the attack language-agnostic (C/C++, Rust, Go, Swift, JavaScript, Python, etc.); the authors found bidi characters already present in real code in numerous popular repositories [1][2].\\n\\n## 2. Relevant invisible-character classes\\n\\n- **Bidi formatting:** LRE/RLE/PDF/LRO/RLO (U+202A\u2013202E), isolates LRI/RLI/FSI/PDI (U+2066\u20132069), LRM/RLM (U+200E/F) [16].\\n- **Zero-width/invisible:** ZWSP (U+200B), ZWNJ (U+200C), ZWJ (U+200D), WORD JOINER (U+2060), BOM/ZWNBSP (U+FEFF), soft hyphen (U+00AD).\\n- **Tag characters:** U+E0000\u2013U+E007F (legitimately used in emoji flag sequences, e.g., \ud83c\udff4\udb40\udc67\udb40\udc62\udb40\udc73\udb40\udc63\udb40\udc74\udb40\udc7f) \u2014 invisible in almost all renderers but machine-readable; central to LLM smuggling (\u00a75).\\n- **Homoglyphs:** cross-script confusables; handled via UTS #39 `confusables.txt` / skeleton algorithm and UAX #31 identifier profiles [6].\\n\\n## 3. UTS #55 and standards response\\n\\n- The Unicode Consortium published **UTS #55, \\\"Unicode Source Code Handling\\\"** (v1.0, late 2021) in direct response to Trojan Source. Recommendations: flag unpaired bidi controls in source; render bidi/invisible characters as visible escapes in editors, diffs, and terminals; restrict identifiers via UAX #31; guidance for code review tools and compilers [5].\\n- Key nuance: outright banning bidi breaks legitimate RTL localization; UTS #55 favors *pairing* checks (stack-based, per UAX #9) and visibility over blanket prohibition [5][16].\\n\\n## 4. Compiler, editor, and hosting mitigations\\n\\n- **GCC 12:** new `-Wbidi-chars=[none|unpaired|any]` warning covering comments, string/char literals, and identifiers; enabled by default at the `unpaired` level [7].\\n- **Rust 1.56:** lint `text_direction_codepoint_in_literal` (warn by default) flags bidi controls in literals [8].\\n- **Clang/LLVM:** added diagnostics for unpaired bidi control characters in comments and literals (LLVM 14 era); LLVM also hardened its own review practices [9].\\n- **VS Code 1.63 (Nov 2021):** default-on Unicode highlighting (`editor.unicodeHighlight.*`) for invisible and ambiguous/homoglyph characters, explicitly as a Trojan Source mitigation [11].\\n- **GitHub:** yellow banner on files/commits containing bidirectional text \u2014 *\\\"This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below\\\"* \u2014 plus a \\\"show hidden characters\\\" view (Oct\u2013Nov 2021) [10]. GitLab rolled out comparable warnings [10].\\n- **Important limits:** NFKC normalization does **not** remove zero-width or tag characters (they have no compatibility decompositions), and it does **not** unify cross-script homoglyphs (Cyrillic \u0430 and Latin a remain distinct) \u2014 so normalization alone is insufficient; filtering by code point/block and confusable detection are required [5][6].\\n\\n## 5. LLM prompt injection via hidden Unicode\\n\\n- **ASCII smuggling:** Tag characters (U+E0000\u2013E007F) are invisible in chat UIs but are tokenized as ordinary byte-level tokens by LLM tokenizers. Demonstrated by Rehberger (\\\"ASCII smuggling,\\\" 2023\u20132024): hidden instructions ride into prompts, and model outputs can encode exfiltrated data invisibly \u2014 e.g., Microsoft Copilot appending tag-character-encoded data to URLs for exfiltration. Microsoft's MSRC addressed this in \\\"Securing AI Model Outputs\\\" (May 2024) [12].\\n- **Hidden instructions:** Goodside (2023) demonstrated invisible prompt injection using Unicode tags with GPT models [13]; the academic paper *\\\"Invisible Prompt Injection: A Hijacking Attack on Large Language Model Applications\\\"* (arXiv, 2024) systematized hijacking via tag characters [14].\\n- **Other vectors:** zero-width characters used to split keywords and evade safety filters; RLO/bidi characters in *model output* spoofing chat UIs, logs, and terminals; homoglyph substitution bypassing keyword filters. Indirect injection via hidden characters in RAG-retrieved documents/emails is a recognized subcase [12][14][15].\\n- **Status:** No complete model-level fix exists; OWASP's LLM Top 10 lists prompt injection (LLM01) as the top risk, with input/output filtering as the primary mitigation [15].\\n\\n## 6. Practical detection/mitigation checklist for text pipelines\\n\\n1. **Scan** for bidi controls, zero-width, tag, and variation-selector characters, e.g. Python: `[\\\\u200b-\\\\u200f\\\\u202a-\\\\u202e\\\\u2060-\\\\u2064\\\\u2066-\\\\u2069\\\\u00ad\\\\ufeff\\\\U000e0000-\\\\U000e007f]` (plus FE00\u2013FE0F/E0100\u2013E01EF where relevant).\\n2. **Escape or strip** rather than silently pass through; log raw code points. Preserve valid emoji flag tag sequences only if required.\\n3. **Enforce bidi pairing** (stack-based per UAX #9) instead of blanket bans where RTL content is legitimate [5][16].\\n4. **Homoglyph defense:** UTS #39 skeleton/confusables + single-script restriction for identifiers [6].\\n5. **CI/CD:** enable GCC `-Wbidi-chars`, Rust's lint, Clang diagnostics; run the trojan-source detector tooling released with the paper [1][7][8][9].\\n6. **LLM pipelines:** sanitize inputs *and* outputs in both directions; never render model output as raw HTML/Markdown; treat retrieved documents as untrusted [12][14][15].\\n\\n## References\\n\\n[1] Boucher & Anderson, \\\"Trojan Source: Invisible Vulnerabilities,\\\" arXiv:2111.00169 (2021); USENIX Security 2023.\\n[2] trojansource.codes (project site: attack demos, vendor responses, detector tooling).\\n[3] CVE-2021-42574 (bidi source-code spoofing).\\n[4] CVE-2021-42694 (homoglyph identifier spoofing).\\n[5] UTS #55, \\\"Unicode Source Code Handling,\\\" Unicode Consortium (2021\u2013).\\n[6] UAX #31 (Identifier and Pattern Syntax); UTS #39 (Unicode Security Mechanisms).\\n[7] GCC 12 release notes / `-Wbidi-chars` documentation.\\n[8] Rust 1.56.0 release notes (`text_direction_codepoint_in_literal`).\\n[9] Clang/LLVM diagnostics for unpaired bidi control characters (LLVM 14 era).\\n[10] GitHub Changelog (Oct\u2013Nov 2021) bidirectional Unicode warnings; GitLab equivalent.\\n[11] VS Code 1.63 release notes \u2014 Unicode highlighting.\\n[12] Rehberger, Embrace The Red, \\\"ASCII Smuggling\\\" (2023\u20132024); Microsoft MSRC, \\\"Securing AI Model Outputs\\\" (2024).\\n[13] Goodside, invisible prompt injection via Unicode tag characters (2023).\\n[14] \\\"Invisible Prompt Injection: A Hijacking Attack on Large Language Model Applications\\\" (arXiv, 2024).\\n[15] OWASP Top 10 for LLM Applications \u2014 LLM01: Prompt Injection.\\n[16] UAX #9, Unicode Bidirectional Algorithm.\",\n    \"t_start\": \"2026-09-26T00:54:20.947439+00:00\",\n    \"t_end\": \"2026-09-26T00:59:34.778230+00:00\",\n    \"index\": 14\n  }\n]", "creation_timestamp": "2026-09-29T05:37:32.000000Z"}]}