The real improvement will be when the software engineers get into the training loop. Then we can have MoE that use cache-friendly expert utilisation and maybe even learned prefetching for what the next experts will be.
> maybe even learned prefetching for what the next experts will be
Experts are predicted by layer and the individual layer reads are quite small, so this is not really feasible. There's just not enough information to guide a prefetch.
The structure of most residential construction in the US is standardized. Foundation (or slab), wood framing, etc. There are different levels of quality, but codes and standards mean that standardization is the norm.
It's fun, but testing has become more of a PITA. When I write code I test and understand each piece. With AI generated code I need to figure out how it works and why it isn't working.
Why bother transcoding on the fly? Storage is cheaper than CPU and the work it takes to determine what needs encoding is excessive.
It implies that you guys are generating the playlists on the fly, tracking the client requests, then feeding that over to your transcoder - which then needs to get the original, seek, and transcode. Why bother?
First, it does save money. A meaningful percentage of videos on the internet are never watched in the first place, and an even larger percentage are watched soon after upload and never watched again. We're able to prune unwatched renditions, and if they happen to be requested years later, they're still playable. Transcoding on the fly lets us save both CPU and storage.
Second, it is ridiculously fast. Our median time-to-publish for a 5-20 minute video is 9 seconds. We had a customer (God bless them) complaining a few months ago that it took us something like 40 seconds to transcode a 40 minute video, which actually was slower than normal for us. If you do an async transcode up front, you're looking at 20 minutes, not <1 minute.
> A meaningful percentage of videos on the internet are never watched in the first place [...] We're able to prune unwatched renditions, and if they happen to be requested years later, they're still playable.
I worked on something similar a while back, and the data that helped me make a call on whether or not we should transcode on the fly or store renditions was looking at analytics for how often the files are accessed.
I figured out that a large file being transcoded and stored would use more compute resources in ~15 minutes than it was likely to use over the span of _several years_ if it was transcoded on the fly. In a situation where you don't know if the company will exist in several years... You opt for the choice which allows you to stack on the storage later if it's necessary.
That's probably one of very few times I've ever applied YAGNI properly. That was ripe for over engineering
But can you still seek on a video that's being instantly transcoded? To be honest I don't know if anyone does that except YouTube, and it jumps to the time so theoretically you have about a second or two when the request comes into pull the file and start encoding. It sounds like the mezzanine file is chunked, so the time to pull it down is pretty fast.
Since it's your own player you can hint to the backend.
Do you dynamically generate the manifests too? Or do things get transcoded on request?
Oh, never mind - yeah, by using the access logs of segments you can effectively anticipate and pre-encode when you need to. And once the hls or cmaf stabilizes you can just encode one resolution. And the player will tell you it wants to move up or down, so you can trigger the encode it wants.
It's interesting your customers want the video immediately; ours don't care about that. But you guys can really build your manifest files and encode immediately, since you're making those mezzanine files.
Then the encoded files are basically a cache that you can evict whenever.
How long did it take you guys to prove that design out?
Yes and yes: you can immediately seek wherever you want. If it's the first time a rendition has been watched, and if we haven't chosen to pre-encode the later segments (which we do sometimes), you might see a slightly longer seek time as we create the first segment after seek, but the difference is marginal and goes away on second view.
> It's interesting your customers want the video immediately
Yep. Some do, some don't. User-uploaded workflows usually care about this; imagine uploading a video to post to a social network and then waiting 20 minutes for the post to go live. (News and sports care about fast publishing too.) Premium media usually doesn't; if you spend a few hours recording a lecture or a yoga class, you don't care if it takes 10 seconds or 10 minutes to publish.
> How long did it take you guys to prove that design out?
You don't want to know. It wasn't easy. The biggest challenge is the ongoing tax; other additions to our transcoding layer have an added degree of complexity. But it's been absolutely worth it for us.
They already hit 3 ships. They don't need to mine the strait, just making it impassable due to insurance cost because you can end up with a hole in a tanker is enough.
When I was in finance there was a question whether US debt would crowd out other debt instruments. The answer, obviously, is "no." There seems to be an unlimited appetite for zero-risk debt, which makes no sense.
reply