The error "CommandInvokationFailure: Unable to list keys in the keystore" is thrown when setting a password or doing anything related to the Keystore

1. What happened

According to the customer the error “CommandInvokationFailure: Unable to list keys in the keystore” is thrown in the Console Window when setting a password or doing anything related to Keystore.

The individual attempted to investigate the source code metadata and reflection and found the issue within AndroidSDKTools.RunCommandInternal. It is observed that this method endeavours to execute the tool command using StreamWriter. In Unity 2023, the cmd or Process class subsequently appends the \uFEFF character, rendering the command invalid.

Further investigation of the source code:

In their machine, The real difference between Unity 2023 and 2022 is GetConsoleCP function of Kernel32.dll is not an error

And because the function is not an error, it then causes the encoding of StandardInput of the sdktool process to be UTF8 with BOM instead of UTF8 without BOM. This wrong encoding makes the console prepend the BOM to input string

Reading the doc it says GetConsoleCP returns 0 means it was an error. The error code was 6 on their machine. But the weird thing is the error is correct behaviour. If it is a success then it makes UTF8 with BOM.

Also in their test code, They investigated the internal source code of the Unity android process. And I could then set

psi.StandardInputEncoding = new UTF8Encoding(false);

This makes the process of starting this java sdktool process seem like a success as it should.

Please take a look at this thread for more details:
https://forum.unity.com/threads/unity-2023-commandinvokationfailure-unable-to-list-keys-in-the-keystore.1519348/

Not Reproducible with:
Cpu: AMD Ryzen 7 5800, GPU: Radeon RX 6800 XT, Os: Windows 10
CPU: AMD Ryzen 7 5800 8-Core Processor 3.40 GHz, Windows 10 Pro 64-bit, AMD Radeon RX 6800 XT

Notes:

• CQA could not reproduce this issue as we don’t have the specific user machine(AMD Ryzen 5). We tried with Ryzen 7 but still not reproducible.
• This doesn’t occur in Unity 2022
• (user) This seems to happen only in some machines so cannot be reproduced from the project. Every project in the same machine will cause the same issue.