This was something I thought about when remembering that my settings aren't quite the same as a lot of other players and remembering it irked me when I experimented with different voice language settings.
If you choose to play with English voices everyone else in-game will have English voices as well as other player characters. However, I found that some of my characters don't have the vocal tone I had in mind at all because I played with the Japanese voices and it was jarring for me to hear my characters come off with a completely different vocal tone and delivery than I expected. The same goes the other way where if you play with a male character using the English voices you tend to have more masculine-sounding voice options and that a large variety of the voice options for the males tend to be on the younger or more effeminate side of things.
My suggestion is to expand the voice catalog that players can use so that at the bare minimum players can select a voice per language so I can set an English voice option for my characters and the Japanese voice options so that players who have either setting can use what I would prefer for my characters to sound like in either language. The other suggestion I would have, due to the potential possibility of using Voice Tickets (and assuming that they may not be dubbed due to their nature outside of specific voice tickets that we have long since missed) is to combine the voice options for all languages so that regardless of language settings players can have access to both the English and Japanese voices. The problem then I see is that of when characters use dialogue that simply no longer matches a player's chosen language settings but games like Blazblue have dealt with this by simply letting players keep their language preferences even in the presence of others (for example a mirror match where a player can use the English voices and the opponent can be using the Japanese voices so both languages are present and are tied to the character/players).