- cross-posted to:
- opensource@programming.dev
- technology@lemmy.zip
- cross-posted to:
- opensource@programming.dev
- technology@lemmy.zip
cross-posted from: https://lemmy.ca/post/37011397
The popular open-source VLC video player was demonstrated on the floor of CES 2025 with automatic AI subtitling and translation, generated locally and offline in real time. Parent organization VideoLAN shared a video on Tuesday in which president Jean-Baptiste Kempf shows off the new feature, which uses open-source AI models to generate subtitles for videos in several languages.
You must log in or register to comment.
I will be impressed only when it can get through a single episode of Still Game without making a dozen mistakes
No such comment yet? I’ll be the first then.
Oh no, AI bad, next thing they add is cryptocurrency mining!
ai for accessibility is nowhere near the same thing as crypto mining
But it’s burning amazon forests for capitalist greed!
/s
Lol, that is his opinion as well, this was sarcasm, i’m 99% sure Yours does not seem so sarcastic…
Finally, some good fucking AI
Finally some good AI fucking 🤭
I was just thinking, this is exactly what AI should be used for. Pattern recognition, full stop.
Yup, and if it isn’t perfect that is ok as long as it is close enough.
Like getting name spellings wrong or mixing homophones is fine because it isn’t trying to be factually accurate.
Problem ist that now people will say that they don’t get to create accurate subtitles because VLC is doing the job for them.
Accessibility might suffer from that, because all subtitles are now just “good enough”
I imagine it would be not-exactly-simple-but-not- complicated to add a “threshold” feature. If Ai is less than X% certain, it can request human clarification.
Edit: Derp. I forgot about the “real time” part. Still, as others have said, even a single botched word would still work well enough with context.
That defeats the purpose of doing it in real time as it would introduce a delay.
Regular old live broadcast closed captioning is pretty much ‘good enough’ and that is the standard I’m comparing to.
Actual subtitles created ahead of time should be perfect because they have the time to double check.
I have a feeling that if you care enough about subtitles you’re going to look for good ones, instead of using “ok” ai subs.
Or they can get OK ones with this tool, and fix the errors. Might save a lot of time
I’d like to see this fix the most annoying part about subtitles, timing. find transcript/any subs on the Internet and have the AI align it with the audio properly.
YES! I can’t stand when subtitles are misaligned to the video. If this AI tool could help with that, it would be super useful.
When does this get released? I really want to try it
This might be one of the few times I’ve seen AI being useful and not just slapped on something for marketing purposes.
And not to do evil shit
But the toppings contains potassium benzoate.
The nice thing is, now at least this can be used with live tv from other countries and languages.
Think you want to watch Japanese tv or Korean channels with out bothering about downloading, searching and syncing subtitles
I prefer watching Mexican football announcers, and it would be nice to know what they’re saying. Though that might actually detract from the experience.
GOOOOOOAAAAAAAAALLLLLLLLLL
Just fill up the whole screen with this.
The opposing team has scored.
As vlc is open source, can we expect this technology to also be available for, say, jellyfin, so that I can for once and for all have subtitles.done right?
Edit: I think it’s great that vlc has this, but this sounds like something many other apps could benefit from
In the *arr suite, bazarr has a plugin called Subgen which you can add and you can set it to generate subtitles on your entire library if you want, or only missing subtitles. The sync is spot on compared to 90% of what Opensubtitles delivers. I sometimes re-gen them with this plugin just because opensubtitles is so constantly out of sync (e.g. highly rated subtitles 4 lines will be at breakneck pace and the next 10 will be super slow and then everything is 3 seconds off)
It isn’t in-player but it works. The downside is it is a larger model and takes ~20 minutes to generate a movie length of subtitles.
crunchyroll is currently using AI subtitles. it’s obvious because when someone says “mothra. Funky…” it captions “mother fucker”
That explains why their subtitles have seemed worse to me lately. Every now and then I see something obviously wrong and wonder how it got by anyone who looked at it. Now I know why. No one looked at it.
my wife and I love laughing at the dumbass mistakes it makes.
some characters name is Asura Halls?
instead of “That’s Asura Halls!” you get “That asshole!”
but if I was actually hearing impaired I’d be really pissed that I’m being treated as second class even though Sony still took my money like everyone else.
Malevolent Kitchen Intensifies
I hope it’s available for Stash App. I wanna know what this JAV girls are saying.
( ͡° ͜ʖ ͡°)
It’s already available for anyone to use. https://github.com/openai/whisper
They’re using OpenAI’s Whisper model for this: https://code.videolan.org/videolan/vlc/-/merge_requests/5155
Has there been any estimated minimal system requirements for this yet, since it runs locally?
It’s actually using whisper.cpp
From the README:
Memory usage Model Disk Mem tiny 75 MiB ~273 MB base 142 MiB ~388 MB small 466 MiB ~852 MB medium 1.5 GiB ~2.1 GB large 2.9 GiB ~3.9 GiB
Those are the model sizes
Oh wow those pretty tiny memory requirements for a decent modern system! That’s actually very impressive! :D
Many people can probably even run this on older media servers or even just a plain NAS! That’s awesome! :D
Note that openai’s original whisper models are pretty slow; in my experience the distil-whisper project (via a tool like whisperx) is more than 10x faster.
Ooooh I like this
Haven’t watched the video yet, but it makes a lot of sense that you could train an AI using already subtitled movies and their audio. There are times when official subtitles paraphrase the speech to make it easier to read quickly, so I wonder how that would work. There’s also just a lot of voice recognition everywhere nowadays, so maybe that’s all they need?
What’s important is that this is running on your machine locally, offline, without any cloud services. It runs directly inside the executable
YES, thank you JB
Justin Bieber?
❤️
JB stands for Jean-Baptiste, who is the main maintainer of vlc
Good ol’ Zorg
Jack Black?
James Brown?
Amazing. I can finally find out exactly what that nurse is yelling about while she gets railed by the local basketball team.
Something about a full-court press?
As long as the models are OpenSource I have no complains
And the data stays local.
When we getting amd’s fsr upscaling and frame-gen? Also would subtitles make more sense to use the jellyfin approach.
I have an AMD card, add VLC as a game in the drivers, and you can turn on AMFM (frame gen).
If it doesn’t work you could just turn it on system wide in display settings of the Adrenaline Software (gear upper right corner, display/gaming).
I think it requires at least a 6000 series GPU however.
If you have a Samsung TV or other modern smart TV connected to a laptop, you can also turn on frame-gen using Auto Motion Plus, set to Custom.
Judder Reduction 10 is double frames, so 24 FPS -> 48.
I’m using Linux. There probably is a way to have a similar outcome but an integrated solution as vlc and amd once mentioned would be better.
I hope Mozilla can benefit of a good local translation engine that could come out of it as well.
They technically already do with Project Bergamot.
I know they do, but it’s lacking so many languages.
And it takes forever. I’m using the TWP plugin for Firefox (which uses external resources, configurable to google, bing and yandex translate respectively), and it’s near instantaneous. The local one from Mozilla often takes 30 seconds, and sometimes hangs until I refresh the page.
Weird, I can translate with FF even on my phone and it’s not a strong one.
Will it be possible to export these AI subs?
Imagine the possibilities!
Now I want some AR glasses that display subtitles above someone’s head when they talk à la Cyberpunk that also auto-translates. Of course, it has to be done entirely locally.
I guess we have most of the ingredients to make this happen. Software-wise we’re there, hardware wise I’m still waiting for AR glasses I can replace my normal glasses with (that I wear 24/7 except for sleep). I’d accept having to carry a spare in a charging case so I swap them out once a day or something but other than that I want them to be close enough in terms of weight and comfort to my regular glasses and just give me AR like overlaid GPS, notifications, etc, and indeed instant translation with subtitles would be a function that I could see having a massive impact on civilization tbh.
soon
Breaking news: “WW3 starts over an insult due to a mistranslated phrase at the G7 summit. We will be nuked in 37 seconds. Fuck like rabbits, it’s all we can do. Now over to Robert with traffic.”
I believe you can put prescription lenses in most AR glasses out there, but I suppose the battery is a concern…
I’m in the same boat, I gotta wear my glasses 24/7.
I think we’re closer with hardware than software. the xreal/rokid category of hmds are comfortable enough to wear all day, and I don’t mind a cable running from behind my ear under a clothes layer to a phone or mini PC in my pocket. Unfortunately you still need to byo cameras to get the overlays appearing in the correct points in space, but cameras are cheap, I suspect these glasses will grow some cameras in the next couple of iterations.
It’d be incredible for deaf people being able to read captions for spoken conversations and to have the other person’s glasses translate from ASL to English.
Honestly I’d be a bit shocked if the AI ASL -> English doesn’t exist already, there’s so much training data available, the Deaf community loves video for obvious reasons.
