Could we use AI to update 4:3 media to 16:9?

Submitted ⁨⁨6⁩ ⁨months⁩ ago⁩ by ⁨snake_case@feddit.uk⁩ to ⁨[deleted]⁩

I’ve been re-watching star trek voyager recently, and I’ve heard when filming, they didn’t clear the wide angle of filming equipment, so it’s not as simple as just going back to the original film. With the advancement of AI, is it only a matter of time until older programs like this are released with more updated formats?

And if yes, do you think AI could also upgrade to 4K. So theoretically you could change a SD 4:3 program and make it 4k 16:9.

I’d imagine it would be easier for the early episodes of Futurama for example due to it being a cartoon and therefore less detailed.

source

Comments

Sort:hotnew top

CaptainBlagbird@lemmy.world ⁨6⁩ ⁨months⁩ ago
I think it would be possible. But adding previously unseen stuff would be changing/redirecting the movie/show.

Each scene is set up and framed deliberately by the director, should AI just change that? It’s a similar problem like with pan-and-scan, where content was removed to fit 4:3.

You wouldn’t want to add content to the left and right of the Mona Lisa, would you? And if so what? Continuing the landscape, which adds just more uninteresting parts? Other people, which removes the focus from her? And this is just a one frame example, there are more problems with moving pictures.

It would be an interesting experiment, but imo it wouldn’t improve the quality of the medium, in contrary.

source
- JohnDClay@sh.itjust.works ⁨6⁩ ⁨months⁩ ago
  I hate to brake it to you…
  
  Mona Lisa landscape
  
  source
  - catsup@lemmy.one ⁨6⁩ ⁨months⁩ ago
    
    I hate to brake it to you…
    
    You just proved his point lol
    
    source
  - CeruleanRuin@lemmings.world ⁨6⁩ ⁨months⁩ ago
    Yeah that sucks.
    
    source
  - Honytawk@lemmy.zip ⁨6⁩ ⁨months⁩ ago
    I think both look great, better than the original because of the added content.
    
    You still get the same detail of the original, nothing about it is changed, but with a more wide view.
    
    source
- Pechente@feddit.de ⁨6⁩ ⁨months⁩ ago
  
  But adding previously unseen stuff would be changing/redirecting the movie/show.
  
  You could see this with The Wire 16:9 remake. They rescanned the original negatives that were shot in 16:9 but framed and cropped to 4:3. As a result the framing felt a bit off and the whole thing felt a bit awkward / amateurish.
  
  source
  - sizzler@lemmy.world ⁨6⁩ ⁨months⁩ ago
    There’s a what??!
    
    source
- ColeSloth@discuss.tchncs.de ⁨6⁩ ⁨months⁩ ago
  I think you’re looking at it from the wrong direction. Instead of adding new stuff in to get the width, you could get AI to stretch the image to fit 16:9 and then redraw everything there to no longer look stretched out. Slim the people and words back down. Things like bottles on a table would be slimmed down to look like normal bottles but have the horizontal table be drawn a bit longer to fill in the space etc.
  
  If it were done this way there would be a minimal amount of things that the AI would have to artificially create that weren’t there in the original 4:3. It would just mostly be fixing things looking wider than they should look.
  
  source
  - Kissaki@feddit.de ⁨6⁩ ⁨months⁩ ago
    Stretching while preserving proportions is still stretching. You change the spacing and relative sizing between objects.
    
    Framing is not only about the border of the frame.
    
    source
    -> View More Comments
  - jungle@lemmy.world ⁨6⁩ ⁨months⁩ ago
    What about person A putting an arm over person B’s shoulder? That’d have to be a pretty long arm.
    
    source
    -> View More Comments
- RightHandOfIkaros@lemmy.world ⁨6⁩ ⁨months⁩ ago
  Sometimes thats true, but not all things in a shot are important. There may be buildings or plamts or people whose placement in the shot is not important. They only exist in the shot to communicate that the film is happening in a real living world.
  
  Ai improving a shot would be debatable, but it is definitely possible. 4:3 media on a 16:9 display is pretty annoying to most people seeing the black bars on the sides. Even if the AI only adds backgrounds or landscapes, simply removing the black bars would be an improvement enough for most viewers.
  
  source
  - setsneedtofeed@lemmy.world ⁨6⁩ ⁨months⁩ ago
    If the AI is only drawing in unimportant objects, I wonder what the value is?
    
    At the risk of ruining the original framing, the potential gain is stuff you aren’t supposed to focus on?
    
    Who is out there watching classic TV shows who isn’t adapted to the old framing?
    
    source
- Honytawk@lemmy.zip ⁨6⁩ ⁨months⁩ ago
  The only thing that would seem wrong is that the actors stand closer than they have to. But other than that, I doubt many would notice.
  
  source
Crul@lemm.ee ⁨6⁩ ⁨months⁩ ago
Do you mean something like this? (warning: reddit link)

using AI to fill the scenes vertically : StableDiffusion
source
- Nerd02@lemmy.basedcount.com ⁨6⁩ ⁨months⁩ ago
  Holy cow that is beyond impressive. Sure enough, sometimes it does hallucinate a bit, but it’s already quite wild. Can’t help but wonder where we’ll be in the next 5-10 years.
  
  source
  - Tar_alcaran@sh.itjust.works ⁨6⁩ ⁨months⁩ ago
    Eh, doing this on cherrypicked stationary scenes and then cherrypicking the results isn’t that impressive. I’ll be REALLY impressed when AI can extrapolate someone walking into frame.
    
    source
- Zeus@reddthat.com ⁨6⁩ ⁨months⁩ ago
  just fyi, your link is broken for me
  
  i wonder if it’s a new url scheme, as i’ve never seen duplicates in a reddit url before, and if i switch it out for comments it works fine
  
  source
  - Crul@lemm.ee ⁨6⁩ ⁨months⁩ ago
    Thanks! Fixed
    
    i wonder if it’s a new url scheme, as i’ve never seen duplicates in a reddit url before
    
    I think you’re right. It should work with the old frontend (which I have configured as the default when I’m logged in):
    
    old.reddit.com/…/using_ai_to_fill_the_scenes_vert…
    
    source
    -> View More Comments
drdiddlybadger@pawb.social ⁨6⁩ ⁨months⁩ ago
You should be able to but remember that aspect ratios and framing are done intentionally so what is generated won’t be at all true to what should be in scene once the frame is there. You’d be watching derivative media. Upscaling should be perfectly doable but eventually details will be generated that will not have originally existed in scenes as well.

Probably would be fun eventually to try the conversion and see what differences you get.

source
- Deestan@lemmy.world ⁨6⁩ ⁨months⁩ ago
  4:3 - Jumpscare, gremlin jumps in from off-camera.
  
  16:9 AI upsized - Gremlin hangs out awkwardly to the left of the characters for half a minute, then jumps in.
  
  source
  - setsneedtofeed@lemmy.world ⁨6⁩ ⁨months⁩ ago
    I was just thinking that. Or something like a comedy bit where the camera pans to a character who had just been out of frame.
    
    Overall it seems like impressive technology to be able to reform old media, but I’d rather put it to use in tastefully sharpening image quality rather than reframing images.
    
    source
    -> View More Comments
- Ghost33313@kbin.social ⁨6⁩ ⁨months⁩ ago
  Exactly, and to add to it, you can't know the director's vision or opinion on how the framing should be adjusted. AI can make images easily but it won't understand subtext and context that was intended. No time soon at least.
  
  source
- CaptainBlagbird@lemmy.world ⁨6⁩ ⁨months⁩ ago
  This.
  
  Surprise! If you want to go from 4:3 to wide screen, you still have exactly the same problem as when using pan&scan for going from wide to 4:3.
  
  source
- sanguinepar@lemmy.world ⁨6⁩ ⁨months⁩ ago
  Definitely. I remember hearing about The Wire being released in a 16:9 format, even though it was shot and broadcast in 4:3, and how that potentially messes up some of the shot framing.
  
  They did I by cropping from top and bottom rather than AI infilling, but the issue is the same.
  
  IIRC, David Simon wrote a really interesting piece about how they did it but did everything they could to try and stay true to Robert Colesberry’s carefully planned framing, as they were aware that had it been intended for 16:9 he’d have framed things differently.
  
  Personally I wish they had kept it at 4:3 and only released it in a higher resolution. Glad I still have my old 4:3 DVDs.
  
  source
- FelipeFelop@discuss.online ⁨6⁩ ⁨months⁩ ago
  Very true, I remember a few years ago someone converting old cartoons to a consistent 60 frames a second.
  
  If they’d asked an animator they’d have found out that animation purposely uses different rates of change to give a different feel to scenes. So the improvement actually ruined what they were trying to improve.
  
  source
TheInsane42@lemmy.world ⁨6⁩ ⁨months⁩ ago
Previous century I bought a TV that was 16:9 and had software to stretch 4:3 broadcasts to fit the screen. It chopped off a tad at the top and bottom and stretched the sides a tad to fill the screen… it was horrible. I’d rather have the dark borders at the sides over mutilated images. Somehow I doubt AI would be a bit more creative.

source
- OhmsLawn@lemmy.world ⁨6⁩ ⁨months⁩ ago
  Yeah it’s always better to watch the original aspect ratio. I remember renting VHS movies and being frustrated either way because the screen was so small. That isn’t a problem these days.
  
  source
setsneedtofeed@lemmy.world ⁨6⁩ ⁨months⁩ ago
Who is the person that enjoys old shows, but also can’t get past the old aspect ratio?

If the AI is just adding complimentary, unobtrusive parts to the shot, so as not to disrupt the original intent, I have to ask- is there really value being added? Why do this at all?

George Lucas thought CGI could make the original Star Wars movies better.

source
- Sethayy@sh.itjust.works ⁨6⁩ ⁨months⁩ ago
  A similar thought I’ve had is AI removal of laugh tracks (maybe introduce background based on non-laugh track scenes)
  
  This would make old Scooby doos actually watchable for me, so I can judge modifying the original a little if it’d what someone prefers - you can always just not watch it
  
  Thinking about it a bit more I 100% would be the type to use a 16:9 ratio, just cause I hate black bars
  
  source
  - setsneedtofeed@lemmy.world ⁨6⁩ ⁨months⁩ ago
    I’m not 100% against tweaking old media (so long as it’s an alternative to the original rather than a replacement), I just think the effort and outcome of widening shots is misguided. More for live action than cartoons. Like if an actor is entering a scene from a side are we going to trust the AI to perfectly add them and merge them without it looking obvious and weird?
    
    For removing laugh tracks, the Scooby cartoons might be a good case for this. They seem paced properly without them. A lot of sitcoms pause for laugher so removing it makes the pacing weird. I’m sure the Big Bang Theory video with laugher removed proves how nightmarish scenes can become (you know in addition to being Bug Bang Theory).
    
    source
- Honytawk@lemmy.zip ⁨6⁩ ⁨months⁩ ago
  I generally avoid old aspect ratio shows/movies, especially for animated stuff.
  
  They just look so extremely dated.
  
  source
  - asdfasdfasdf@lemmy.world ⁨6⁩ ⁨months⁩ ago
    That’s a mental thing. You’re missing out. Same with people who can’t enjoy the enormous amount of fantastic black and white movies out there.
    
    source
xia@lemmy.sdf.org ⁨6⁩ ⁨months⁩ ago
Ahhh… the much-fabled “uncrop” operation!

source
- echodot@feddit.uk ⁨6⁩ ⁨months⁩ ago
  Also it was good as Enhance
  
  source
CeruleanRuin@lemmings.world ⁨6⁩ ⁨months⁩ ago
Why would you want that? It’s always best to consume media in its purest form, and that means with it’s original aspect ratio. Resolution is something I’m flexible on, because I figure that filmmakers and tv directors in prior eras would have used HD if it was available, but aspect in general is tied to format, though it can be used to great effect to convey the space of a scene in different ways. Changing the ratio is akin to changing the color pallet. Might as well offer Instagram-style filters for older content while you’re at it.

source
- Globulart@lemmy.world ⁨6⁩ ⁨months⁩ ago
  Exactly, the filmmaker knew exactly what the aspect ratio was and framed shots specifically for it, why would anyone ever want this…?
  
  source
  - Squirrel@thelemmy.club ⁨6⁩ ⁨months⁩ ago
    Ooo, maybe we can get a nice blurred copy of the picture to fill the edges of the screen, just like TikTok!
    
    I feel sick even jokingly suggesting that…
    
    source
- Honytawk@lemmy.zip ⁨6⁩ ⁨months⁩ ago
  Because adding more detail to the sides doesn’t change the quality of the show? If anything, it improves it.
  
  I’m also not going to install some old CRT monitors to “consume media in its purest form”, because I want the best quality, not the quality the filmmaker wanted back in the day it was created.
  
  source
  - Globulart@lemmy.world ⁨6⁩ ⁨months⁩ ago
    The best quality IS the quality the filmmaker wanted when it was created.
    
    People buy CRT TVs even now btw, the absence of input lag makes it ideal for competitive smash bros.
    
    Most people are perfectly happy watching an old film/show on a new screen because it can near as dammit replicate exactly what the director wanted us to see. Adding actual stuff to the framing of shots is a nuts idea and it absolutely changes the quality.
    
    To give an example, having someone’s face fill the screen versus being centred with more background visible around it changes the feel of the shot a LOT. 12 angry men is a good example here because the camera gets almost imperceptibly closer to the person speaking throughout the film to ramp up tension and draw the viewer into the story. It creates a feeling of claustrophobia that would be dulled if you added content to the edges of the screen.
    
    source
Pyroglyph@lemmy.world ⁨6⁩ ⁨months⁩ ago
I remember watching a great video about why this isn’t a good idea.

Here it is.

source
- uniqueid198x@lemmy.dbzer0.com ⁨6⁩ ⁨months⁩ ago
  Clicked expecting Noodle. It was Noodle.
  
  source
- rufus@discuss.tchncs.de ⁨6⁩ ⁨months⁩ ago
  Nice video. But it can’t be the right one. There’s nothing about AI in it.
  
  source
  - Pyroglyph@lemmy.world ⁨6⁩ ⁨months⁩ ago
    I was more referring to how changing aspect ratios is a bad idea, not that using AI to do it would be a bad idea.
    
    source
SARGEx117@lemmy.world ⁨6⁩ ⁨months⁩ ago
[deleted]
source
- Couldbealeotard@lemmy.world ⁨6⁩ ⁨months⁩ ago
  The issues in Stargate were not because they were framing for 4:3, they were just clumsy mistakes.
  
  Stargate was always framed for widescreen.
  
  If you watch the HD versions of Atlantis you will also notice lots of focus issues. Not to mention across all 3 series there were plenty of things that ended up on screen by mistake: scripts laying around in a Goa’uld cargo hold, a Snickers wrapper in an alien space station, people holding up plants, and offensive remarks about the French.
  
  source
  - SARGEx117@lemmy.world ⁨6⁩ ⁨months⁩ ago
    Oh, for sure they made tons of mistakes, I just thought of that one as the most glaringly obvious “framing issue”. I’ve seen all the episodes as aired and as on DVD, with commentary and special features, because I’m a super nerd, and given your pfp and the fact you know how they framed it tells me you know a thing or two yourself!
    
    If I remember correctly, they framed it for wide-screen knowing that in the future it could be put out in other formats than TV, so wanted widescreen from the start. Bts footage shows the framing boxes for tv/wide on the monitors.
    
    I love catching mistakes and weird choices in my shows. For instance: in Firefly, Alan Tudyk is pretending to hold the controls of Serenity because they couldn’t have him up in the normal spot for framing reasons.
    
    But yeah, I’m actually rewatching SGA now and their weird focus issues (and let’s be honest, terrible backgrounds) are especially bad in season 1.
    
    source
    -> View More Comments
PopOfAfrica@lemmy.world ⁨6⁩ ⁨months⁩ ago
Yeah, but why would you want to? It would have to generate new imagery to fill out the gaps. That’s bound to not look right. It at the very least would not be fitting the artist’s intention.

source
IAmDotorg@lemmy.world ⁨6⁩ ⁨months⁩ ago
Voyager and DS9 were shot on video, not film.

That’s why there’s HD versions of TOS and TNG but not DS9 and Voyager.

source
- JTheDoc@lemmy.world ⁨6⁩ ⁨months⁩ ago
  They were shot on negative 35mm film, and edited on tape in that aspect ratio.
  
  Only some odd scenes and shots were captured on tape, but it was mostly film. They won’t need to upscale any live scenes, but they would have to work on rendering all the digital effects and blue screen shots for example. As it was only ever edited on tape, there’s unlikely want digital effects that could be rendered or upscaled, it would likely need to be entirely reworked from scratch.
  
  Very daunting task. Although, they did it for TNG. ¯_(ツ)_/¯
  
  source
  - echo64@lemmy.world ⁨6⁩ ⁨months⁩ ago
    It’s worth noting that edited on tape also means that to make hd versions, you have to re-scan all the camera negatives (if they even still exist), then re-edit all the scenes to be exactly the same editing as the tape editing. Plus, all the colour timing needs to be redone.
    
    It’s a huge amount of work, and the tng stuff wasn’t profitable because of it. It’s just too much.
    
    source
    -> View More Comments
  - Tar_alcaran@sh.itjust.works ⁨6⁩ ⁨months⁩ ago
    On the other hand, recreating Voyagers CGI is doable for any home enthusiast nowadays.
    
    source
    -> View More Comments
echodot@feddit.uk ⁨6⁩ ⁨months⁩ ago
There is no original film for voyager it was filmed on tape.

TNG used film so that can be rescanned but the original analogue broadcast is the literally the best quality we have of Voyager.

source
Potatos_are_not_friends@lemmy.world ⁨6⁩ ⁨months⁩ ago
I used adobe’s Generative Fill tool (where it uses AI to fill in the blanks like adding more sky/backgrounds/hiding people) and it’s pretty miss. 20% of the time, it would kinda work. And I say that loosely.

I think in a few years, we’ll get there. But not today.

source
- BallsInTheShredder@lemmy.world ⁨6⁩ ⁨months⁩ ago
  Is that for movies, images or what? Either way it’s interesting
  
  source
SHITPOSTING_ACCOUNT@feddit.de ⁨6⁩ ⁨months⁩ ago
Adding imagery that reliably looks good is currently beyond what AI can do, but it’s likely going to become possible eventually. It’s fiction, so the AI making stuff up isn’t a problem.

Upscaling is already something AI can do extremely well (again, if you’re ok with hallucinations).

source
- MeatsOfRage@lemmy.world ⁨6⁩ ⁨months⁩ ago
  I’m not sure it’s really beyond the scope of AI. Stuff like stable diffusion in-painting and some of the stuff Adobe was showing off at their recent keynote shows were already there.
  
  source
  - Tar_alcaran@sh.itjust.works ⁨6⁩ ⁨months⁩ ago
    Those are on a completely different level from having someone walk into frame though, and they still only work on small things that can be extrapolated from the image.
    
    source
thirstyhyena@lemmy.world ⁨6⁩ ⁨months⁩ ago
I was wondering the same last week, but for Buffy the Vampire Slayer tv series, which received a horrible HD release some years back.

source
- tankplanker@lemmy.world ⁨6⁩ ⁨months⁩ ago
  For Buffy they recut the shot often using the raw footage, and they did so very cheaply so film equipment was often visible. They also didn’t address how bad the make up looked in HD, but then soft focus face filters are also garbage.
  
  The Simpsons when they tried they cut the frame, which is just laughably bad as it removed information and the context for scenes.
  
  When the Wire was done they got David Simon back to work on the conversion, he considers it a completely different cut of the show. I think this is the only way to do it as it means re framing the shot, this is a decision for the director, editor, and DP IMO
  
  AI making shit up to add to the frame removes the context for the shot. Nothing wrong with black bars for me, I just want good colour balance and upscaling.
  
  source
zwaetschgeraeuber@lemmy.world ⁨6⁩ ⁨months⁩ ago
its already possible, but not really good atm. saw some scenes, where they changed the aspect ratio from 16:9 to tiktok. im waiting for that tool, google is working on. idk how it works, but it can guess, what should be outside of the image and the images i saw were pretty amazing compared to SD. 4k upscaling at 120hz also no problem, as i did this to the first season of spongebob and it really works fine with my hobby equipment in one night. id really love to see an upscaled 16:9 version of the it crowd and the first 6 seasons of scrubs.

source
Stamets@startrek.website ⁨6⁩ ⁨months⁩ ago

Ive been rewatching Star Trek Voyager recently

Good choice.

I’d imagine that AI probably will be able to in 10-15 years. We already have that photoshop stuff that can analyze the surroundings and then fill in gaps/erase stuff. It’s not perfect but it’s the ground floor. I can only imagine that in the not too distant future it’ll be able to fill in the gaps of video too. Especially with a consistent set like Voyagers main engineering, as an example.

Actually come to think of it, the same principle could be used to make VR environments too.

source
SHITPOSTING_ACCOUNT@feddit.de ⁨6⁩ ⁨months⁩ ago
It works well enough to be shown for a few seconds at a keynote for static pictures in some cases. It won’t yet work well enough to be permanently known as the official “remastered version” for moving video consistently.

Now, someone uploading a watchable version on YouTube? That will happen in the next years if it hasn’t already. But that version would be widely ridiculed if released officially because something, somewhere will be off and fans will notice.

source
- snake_case@feddit.uk ⁨6⁩ ⁨months⁩ ago
  Isn’t that the real beauty of AI? We can go over the parts that fans flag as distracting or obviously wrong, and release a new version. This would have to be done on some streaming service so no physical copies until a finalised version everyone is happy with.
  
  source
  - SHITPOSTING_ACCOUNT@feddit.de ⁨6⁩ ⁨months⁩ ago
    I guess it could be done as a Bandersnatch-style experiment on one streaming platform, but it’s so far from the regular way things are done that it seems unlikely.
    
    source
  - sanguinepar@lemmy.world ⁨6⁩ ⁨months⁩ ago
    That’s the nightmare of AI as far as I’m concerned. Endlessly revising movies and TV away from what was originally created by the filmmakers.
    
    source
taiyang@lemmy.world ⁨6⁩ ⁨months⁩ ago
I don’t understand why there are so many purists bashing the idea. To answer directly, it can be done and it can be done well enough, albeit you’d need a fair amount of GPU. There’s a reason it’s mostly used for still images. Also, don’t expect scene to scene consistency. New hallways, etc, unless we get models that can account for that.

Also yes, you can upres stuff, even before all this image stuff. It’s gotten pretty good, even for video-- usually benefits from some human touch ups though; it’s used quite well on some video games. That’s easier than out painting, but has trouble with text and finer designs that are harder to guess.

Plus, it’s arguably easier with live action rather than animation although I’m not certain. Redoing an art style isn’t always easier than just filling in scenes.

Honestly at the rate we’re going we could probably even have AI generate whole episodes of stuff, but that’s when we’re really going to be treading dangerous waters.

source
kuneho@lemmy.world ⁨6⁩ ⁨months⁩ ago
Voyager 💛 I’m on last episode of season 6.

I have an old Panasonic plasma and I was always a bit worried when I watch Star Trek (right from TOS) that those black bars on the side of the screen will damage the panel.

But I don’t care. It’s Star Trek.

source
- PolarisFx@lemmy.dbzer0.com ⁨6⁩ ⁨months⁩ ago
  I have 3 old Panasonic plasmas, they are the greatest tv’s ever. My wife wants an led smart tv, she will have to pry these tv’s from my cold dead hands. They weigh so much she can’t move them herself
  
  source
  - kuneho@lemmy.world ⁨6⁩ ⁨months⁩ ago
    I can totally understand that 😄
    
    Tho it’s a pretty low-end plasma, the panel still looks amazingly gorgeous.
    
    source