Locale File Cleaner

[Takuma 181]

Hi devs ! I was getting bored during my vacation, so i had to keep myself occupied...

At the request of @ Gurgarath :3, I created a small Python script that allows you to clean up unused lines from your locale_game.txt and locale_interface.txt. It is coded with Python 3.11, compatibility with earlier versions is not guaranteed. It also uses the chardet module, which is specified in the requirements.txt. For any additional information (such as the How To), please refer to the README.md.

 This script currently does not support imports like import LocaleInfo as li. It also does not support dynamically constructing variable names. It only detects raw constants like localeInfo.MY_CONSTANT.

 

Downloads :

This is the hidden content, please

• Sign In
• or
• Sign Up

Find this tool on my Github or

This is the hidden content, please

• Sign In
• or
• Sign Up

 

Exemple of output :

Quote

> python main.py

Reading locale/locale_game.txt...
Reading locale/locale_interface.txt...

Reading source/uicommon.py
    -> Found 4 constants for localeInfo
    -> Found 1 constants for uiScriptLocale

Writing new files...
Writing locale/new_locale_game.txt...
Writing locale/new_locale_interface.txt...

Ended ! Enjoy you new cleaned files !

 

Takuma.

 

[PetePeter 36]

That's cool thanks ! What if we use Python's `ast` (Abstract Syntax Tree) module to handle the substitution of dynamic variable names in the code? I mean, we could technically parse the Python source file and replace all possible variables. Check this out, I've added a function that handles dynamic names in simple cases, specifically looking for Call nodes and Attribute nodes that could resemble a dynamic variable name access. Though it's not a perfect solution, it can cover many simple scenarios:

 

from ast import parse, Import, ImportFrom
...


class SourceFile:
    ...

    def read(self):
        ...
        if file_name.lower() == module.lower():
            self.regex_pattern = self.create_regex_pattern(content, file_name)
            self.modules_constants[module] = findall(
              self.regex_pattern, content, IGNORECASE
            )
        ...

    @staticmethod
    def create_regex_pattern(content: str, module: str) -> str:
        """Create regex patterns for different types of imports"""
        module_mapping = SourceFile.get_import_aliases(content)

        if module in module_mapping:
            return r"\b" + escape(module_mapping[module]) + r"\.([a-zA-Z_]\w*)(?![\[\(])\b"
        else:
            return r"\b" + escape(module) + r"\.([a-zA-Z_]\w*)(?![\[\(])\b"

    @staticmethod
    def get_import_aliases(content: str) -> dict[str, str]:
        """Parse the file and get the aliases for imports"""
        root = parse(content)
        import_aliases = {}

        for node in ast.walk(root): # Use the python built-in library AST to parse the python file
            if isinstance(node, Import): # check for import statements
                for name in node.names:
                    module = name.name
                    alias = name.asname if name.asname else name.name
                    
                    import_aliases[module] = alias

            elif isinstance(node, ImportFrom): # check for from import statements
                for name in node.names:
                    module = node.module
                    alias = name.asname if name.asname else name.name

                    import_aliases[module] = alias
        return import_aliases

import ast

class SourceFile:
    ...

    @staticmethod
    def handle_dynamic_names(content: str, module: str):
        root = ast.parse(content)

        dynamic_names = []

        for node in ast.walk(root):
            if isinstance(node, ast.Call) and getattr(node.func, "id", "") == "getattr":
                module_obj = node.args[0]
                attr_obj = node.args[1]
                
                if isinstance(module_obj, ast.Subscript) and isinstance(attr_obj, ast.Str):
                    if module_obj.value.id == "globals" and attr_obj.s == module:
                        dynamic_names.append(node.func.attr)

        return dynamic_names

    def read(self):

        ...

        dynamic_names = self.handle_dynamic_names(content, module)
        for dynamic_name in dynamic_names:
            self.modules_constants[module].append(dynamic_name)

        ...

Now, this might not always work perfectly, especially for more complex cases. For example, situations where attribute names are generated through loops, or via mathematical operations could throw a spanner in the works. Basically, when the attribute name ain't directly visible in the script, this approach could miss it. But all in all, this seems like a step in the right direction. Curious to hear your thoughts on this.

[Takuma 181]

My opinion is that I didn't even know about the ast module; I've never really had the opportunity to use it. I'll be back to work on Monday and will be less available. However, if you find that this improvement is beneficial and necessary, I invite you to make a pull request on GitHub so that you can be directly listed as a contributor.

As soon as I have some time, I'll study the matter. Thank you!