MS 2 UTF8 - Mass Save To Unicode

[Mind Rapist 897]

M2 Download Center

This is the hidden content, please

• Sign In
• or
• Sign Up

( Internal )

Spoiler

 

I made this script because I got tired of all the binary warnings about saving the file to Unicode format. So I am now releasing it to the community.

This is a Windows PowerShell script that converts all the files of the destination that have the postfix *.cpp and *.h to UTF-8 (without BOM) encoding.

 

I made this script because I got tired of all the binary warnings about saving the file to Unicode format. So I am now releasing it to the community.

This is a Windows PowerShell script that converts all the files of the destination that have the postfix *.cpp and *.h to UTF-8 (without BOM) encoding.

By default, the file will look inside the directory "C:\Users\user\Documents\" but there is a -Dir parameter you can use (besides editing the file of course). Specify your working directory and wait a few seconds for the process to start. After that, there is really nothing else to do!

 

EXTRA: There is one more parameter you can specify here. A boolean check by the name -Src (disabled by default). If you pass "1" in here the script will automatically check for file extensions *.c and *.hpp (the source files of the Extern folder).

 

Update

   Since this is a simple re-encoding process I am sharing an automated prompt for LLM agents that may be useful, since AI has become popular over devs, scripts like this one may be obsolete now. In any case someone needs the prompt, I posted it here: 

 

Edited August 13 by Mind Rapist

[Kori 543]

Very useful nice Thanks 

[DaCookie -Raven 48]

Thanks for the tool.
I ran into some errors using the locale_string.txt, because mine can not properly be converted into UTF-8.
This is because the korean texts are encoded in cp949 and my local translation has some special characters in ISO-8859-1.
Therefore I wrote a small tool converting the korean texts encoded as CP949 into UTF-8 and some european character set ISO-8859-1 into UTF-8.
Maybe it is useful for you and you can adjust the character sets for your needs.

NOTE: You need iconv by GNU to link this application

This is the hidden content, please

• Sign In
• or
• Sign Up

This is the hidden content, please

• Sign In
• or
• Sign Up

[Mind Rapist 897]

2 minutes ago, DaCookie -Raven said:

Thanks for the tool.
I ran into some errors using the locale_string.txt, because mine can not properly be converted into UTF-8.
This is because the korean texts are encoded in cp949 and my local translation has some special characters in ISO-8859-1.
Therefore I wrote a small tool converting the korean texts encoded as CP949 into UTF-8 and some european character set ISO-8859-1 into UTF-8.
Maybe it is useful for you and you can adjust the character sets for your needs.

NOTE: You need iconv by GNU to link this application

iconvpp.h

main.cpp

Sorry to break it to you but this tool is meant to be used in the Client source only. Because of the locale messages, server source should never be converted to UTF-8 unless it's original encoding was Unicode. If you need a UTF-8 source:

1. Use Vanilla
2. Make an exact copy of your source and convert it using the tool. Then start copying the Korean strings from the non-Unicode to Unicode encoded source, one by one.

Best regards

[DaCookie -Raven 48]

1 hour ago, Mind Rapist said:

Sorry to break it to you but this tool is meant to be used in the Client source only. Because of the locale messages, server source should never be converted to UTF-8 unless it's original encoding was Unicode. If you need a UTF-8 source:

1. Use Vanilla
2. Make an exact copy of your source and convert it using the tool. Then start copying the Korean strings from the non-Unicode to Unicode encoded source, one by one.

Best regards

Well I used my IDEs option to convert all server source files to UTF-8.
That's why I ran into problems with locale_string.txt
Because I`m lazy and don't want to copy the korean strings one by one I wrote this tool.
Also it is a nice thing to work with a standard encoding

[Mind Rapist 897]

Bumping on an old post but I see recent reactions so this may come in handy for some of yall:

Since every single dev nowadays has access to some sort of LLM assistance on their code, this tool is now legacy, as your AI agent can convert your encodings automatically, and since this is a very simple process it does not require any manual checks on your part no matter how large your batch is.

ALWAYS KEEP BACKUPS!

Here is a prompt that can help you automate this process:

Spoiler

Scan my [input dir] and create a list of all non-UTF8-content files in it. Resave all files as UTF-8 (no BOM) in [output dir] in the exact same folder structure as [input dir]. **Do not change anything in the contents of the files**, except converting non-ASCII characters from the old legacy encoding to readable UTF8 format.

Always verify the source encoding of each file before processing it, but you will usually come across these patterns:
 - Source files (cpp, h, hpp, c, py, txt, lua) will have a legacy EUC-KR or cp949 and almost every single comment will be in Korean
 - Translation files (locale_XXX(?_XX).txt, translate(?_XX).lua, item_names(?_XX).txt, mob_names(?_XX).txt) and similarly named files will most probably not have Korean, but you will most definitely encounter the following encodings:
	- 1252: en, fr, de, dk, es, it, nl, pt
	- 1256: ae
	- 1253: gr
	- 1254: tr
	- 1251: ru
	- 1250: hu, pl, ro, cz

**NEVER** change a filename or directory name, even if it contains Korean/Mojibake characters!!!
**ALWAYS** create backups of my existing files before any processing starts in [backup dir]

If you encounter unreadable Mojibake strings/comments, I am providing you a reference here: [reference dir], where I have all my files converted in EUC-KR/cp494 encoding, so everything will be in Korean, not Mojibake.

[input dir]: "C:\path\to\target\files\root"
[output dir]: "C:\path\to\export\root" <== can be the same as your [input dir]
[backup dir]: "C:\path\to\my\backups <== new empty folder is recommended
[reference dir]: "C:\path\to\my\korean\source\files" <== here it is recommended that you copy your files before processing and open them all with notepad++ as it will auto apply the legacy encoding where there would be Mojibake otherwise

 

Hope this is helpful to some people.