Feature Request/thought: Ability to add different OCRs

[michleefits]

I really like this program but it would be nice if you allowed us to add different OCRs (maybe you already do and I missed it somehow) but for example EasyOCR is amazing for default plain non stylized Japanese text but when it comes to even SLIGHTLY stylized text it struggles immensely but for Googles Cloud Vision is great at stylized text so I wish there was a way for me to add it in the list of OCRs in the program because trying to create a python script with the help of AI is just not working for me 😭

[SethRobinson]

Yeah, I see what you mean.

UGTLive's design has it doing the OCR as fast as possible (several times a second or so) to detect changes - doing this via cloud might be a bit costly in terms of API usage and bandwidth. (currently all OCR methods are fully local, using GPU)

So couple thoughts:

1. Could add it, I mean, why not. It can be done in C#, no need to use the python backend.

2. Could add a "Snapshot" button, so instead of "Start" and "Stop", you could click snapshot to just OCR/translate once until hit again, so it doesn't constantly hit the cloud service, maybe we don't need "Live" all the time. (btw, check out https://github.com/SethRobinson/UGT , it can do that, although UGTLive's interface is better I think)

3. Add a new local OCR method that just works better for non-manga Japanese. If you attach a test image I could maybe try some stuff.. I have some ideas.

Maybe doing all three would be nice.

[SethRobinson]

Update: I went ahead and did #1 for now, let me know how Cloud Vision works for ya. Note that it has "horizontal glue" and "vertical glue" settings in its settings menu

[michleefits]

So sorry I didnt reply sooner Ive been busy but first I wanna say thanks for actually replying I didnt expect that 😂. Second Im not good at wording and stuff so this is gonna be all over the place, so anyway thank you for putting in the effort for #1 putting in the cloud vision but unfortunately for me idk if I used it wrong but it just really wouldnt work live and when it did work it was when I pressed stop on the server😂but maybe I did something wrong. (EDIT: IS WHAT I WOULD SAY but after writing the 3rd paragraph I looked at the settings and saw the "broken" leave translation on screen or whatever and turned that off and the translations were indeed much faster (still pretty slow tho and the text was not lined up sometimes but that may be needed to be experimented with the glue settings you added)

But on that note I do have a question about how your program works so there is the screen monitor and the actual overlay I originally thought that the overlay would go over your screen then the program would just put the english translated text over the japanese text but it seems like I actually need to have the monitor on the side because thats the screen that will show my translations and not my regular screen? idk like I said maybe I just got it wrong but I just wanted to make sure I wasnt using it wrong.

And if it isnt supposed to show you the translations with the overlay on top of your normal screen without the use of the monitor I recently discovered Gaminik which pretty much does that, it has the overlay like you do but you can choose to do screenshot (quick capture and translate) or auto-translate (live translate), and this kinda speaks to your 3rd point about the local OCR cuz they do have that but it does technically do a not so good job and they have an online (translation engine they call it)( I dont get how they do things because they dont label an OCR and/or translation service type thing in the settings you usually see) one that does a pretty great job with translation but they have a problem with refreshing in their auto-translate, once it translates the text on screen and lets say its a game that has a lot of those passive texts that fly by when someones talking that you dont click to move the dialogue forward ( or just ya know timed text that goes away fast) they wont capture that new text and translate it cuz the previous translation is in the way and stuff.

That brings me to another point is it just one of those hardware limitation type things or what, because if you look at Googles mobile translation app with the camera, THAT THING WORKS PERFECTLY, like sure it does sitll have the refresh problem where it doesnt re-translate whats on the screen till you move your camera away then point it back at the screen but still even if you cover a huge part of the screen with a lot of jap text on it translates it perfect and scales it with the jap text pretty perfectly as well idk does google really just have unknown tech that no one can replicate or come close to it 😂cuz I looked it up and they said that app uses the same cloud vision plus google translate setup so im just lost anyways sorry for talkin to much most of this is probably pointless 😂

[SethRobinson]

OCR engines like UGTLive (that have to capture from the screen 'live') generally have a limitation that they can't capture from the same place they show the translation at the same time.

I saw a video of Gaminik and it looks like they have the same limitation - in "auto" mode, you have to manually close the translation before it can capture again from that spot it looks like. (Try my older app UGT, that's how it works)

And when you use "live" mode, they show the translation in a different place, not over the same place. (they show the text near the bottom of the screen)

UGTLive can do that: (see here) , but if you want to translate the entire screen on top of the screen at the correct places, yeah, you basically have a second screen.

There are some bugs in UGTLive making it slow and laggy in some cases, I'll try to fix 'em

Anyway, no worries, thanks for helping me test it and giving feedback

[michleefits]

Ya true Gaminik does have that limitation and ya sorry I meant to say in my essay that I did try UGT and personally I just didnt like it because it kinda takes me out of the game immersion wise as well as me just being lazy.

"And when you use "live" mode, they show the translation in a different place" I'm confused what you mean by this too because when I do a manual translation (for me I just press shift F) and in like a second the english translated text pops up over the OG Jap text in the same spot 🤔

[michleefits]

Thank you, thanhkeke97! I can't believe that worked! michleefits: I'll try to add a better "live" overlay mode soonish

awesome thank you cant wait to see how it turns out 😎

[michleefits]

@SethRobinson Based on your wonderful software, I have created another version with overlay functionality, you can refer to and add to it: https://github.com/thanhkeke97/RSTGameTranslation

oh I'll have to try this out😃

[michleefits]

@SethRobinson Based on your wonderful software, I have created another version with overlay functionality, you can refer to and add to it: https://github.com/thanhkeke97/RSTGameTranslation

unfortunately for me at least, the program does not work well. I've tried all the OCRs and both the google translates but I cannot get a solid result. One problem is that the overlay doesnt put the english translated text over the Japanese text properly but I also think that the thing contributing to this is the OCRs themselves, the games I'm trying them on have Jap text that are more stylized than not which a lot of OCRs struggle with.

what I meant by properly is that sometimes it will translate some of the words on screen but completely put them in the wrong place and/or bunched up together. But like I said maybe thats the OCRs fault or something idk. The other issue is the same thing me and Seth were discussing, the refreshing of the translation it doesnt refresh cuz the old translation is there (is what I would say but I obviously cant confirm this isnt because of the OCR issues already present)

[SethRobinson]

@SethRobinson Based on your wonderful software, I have created another version with overlay functionality, you can refer to and add to it: https://github.com/thanhkeke97/RSTGameTranslation

Thanks, the more translation options the better!

[michleefits]

Another thing I was gonna say that you guys may or may not care about is that I discovered a potential solution to this translating Japanese game text problem. I found this program called XUnity.AutoTranslator, basically it scans the games files for the text of the game and then it translates it in real-time while you're playing and it works pretty phenomenally. The only problem is, as the name implies, its only for games made with the Unity engine ☹️but so many Japanese games are made with their own engines or just different ones entirely that store their game text files differently so I'm definitely gonna ask their github for more advice or something.