VideoChat3:Fully Open Video MLLM for Efficient and Generalist Video Understanding
VideoChat3 is a fully open-source video multimodal LLM (4B parameters) that tackles three concrete problems: generalization across short/long/streaming video, computational efficiency for video token explosion, and reproducibility through complete open-sourcing. The core innovation is I3D-ViT (Inflated 3D Vision Transformer) plus adaptive frame resolution, which compresses spatiotemporal redundancy early in the pipeline instead of treating each frame as an independent image. Three curated datasets (2M academic + 116K long-form + 617K streaming = 3M samples total) and multi-stage curriculum learning enable the model to handle diverse video scenarios. Jessica sees a shippable foundation for real-world video apps; Cathy pushes on whether the efficiency gains hold under production load and whether the data pipeline's scale claim is reproducible.
Transcript
Jessica Okay, so video MLLMs are still getting crushed by token count. You throw a five-minute video at an image tokenizer, sparse-sample it, and you've already lost the motion. VideoChat3 says: stop doing that.
Cathy Right. The standard move is frame-by-frame encoding — each frame is basically an independent image. You get no temporal compression, no redundancy reduction, and the LLM has to stitch the motion story together after the tokens are already huge.
Jessica Which is backwards.
Cathy Exactly. VideoChat3 inverts it: model the spatiotemporal structure early, in the encoder, before you ever hand tokens to the LLM. That's the I3D-ViT move — Inflated 3D Vision Transformer. You're processing video patches as 3D spatiotemporal blocks, not 2D image patches.
Jessica So the vision encoder is doing temporal modeling, not the language model.
Cathy Yeah. And because you're modeling spatiotemporal structure, you can compress redundancy — adjacent frames have massive overlap, right? If the encoder understands that, it can throw away the redundant stuff early. You end up with way fewer tokens before the LLM even sees them.
Jessica How much fewer?
Cathy The paper doesn't give a raw token count comparison, which is annoying. But the throughput numbers suggest significant compression. The real win is streaming: they add adaptive frame resolution, so if motion is low, you drop spatial resolution. High motion, you keep it. That's the kind of thing that makes streaming actually feasible.
Jessica Streaming is real-world. Autonomous driving, surveillance, live sports commentary — you can't wait for the full video. You need to respond as frames come in. And if you're doing adaptive resolution, you're not just being efficient, you're being smart about where efficiency matters.
Cathy Right. The architecture is sound. But here's where I want to push: they're claiming 4B parameters and they say it outperforms larger open models like Qwen3-VL and Molmo2. That's a bold claim. Better architecture and better data can beat parameter count, but I'd want to see those benchmarks independently reproduced.
Jessica Fair. But the data story is where I think they actually have a real move. They're not just throwing everything into one pile. VideoChat3-Academic2M for general video, VideoChat3-LV116K for long-form, VideoChat3-OL617K for streaming. Three datasets, three distinct scenarios.
Cathy And they use multi-stage curriculum learning — general first, then long-form, then streaming. That's deliberate. You don't want task interference.
Jessica Exactly. And the curriculum is the thing that actually matters for generalization. You can have a great architecture, but if your training signal is a mess, the model learns a mess.
Cathy Yeah. The curriculum is load-bearing.
Jessica But here's the thing that actually got me: they're releasing everything. Model weights, training code, training strategy, complete datasets, the data construction pipeline. That is not the move most labs make.
Cathy No. Most papers are either closed or partially open — you get weights maybe, but no training code, no data, no recipe. You can't reproduce it. You can't improve it.
Jessica Right. And VideoChat3 says: reproducibility is not a nice-to-have, it's the product. If you can't reproduce it, you can't build on it.
Cathy Okay, so that's the thesis. The question is: does it actually ship? Does anyone use this? Or is it research-grade forever?
Jessica That's the real test. A 4B video model that's efficient and open is genuinely useful for teams that don't have billions in compute budget. If this actually works at inference time, it could be the foundation for a bunch of stuff.
Cathy The efficiency claim is testable. You can run it, measure latency, measure tokens per second, see if it actually holds up under production load. That's the bet I'd make: the architecture is sound, the data strategy is smart, but whether it's efficient at scale is still an open question.
Jessica Yeah. And the streaming part especially — because streaming is not just about efficiency, it's about latency. If you're doing autonomous driving, you need sub-hundred-millisecond latency. Does VideoChat3 actually hit that?
Cathy Right. The benchmarks they report are standard video understanding tasks. Real-time latency under production constraints is a different test.
Jessica Okay, so the move is: go grab the code and the datasets from Hugging Face, spin up a model, and actually test it. The full pipeline is open, so you can do that.
Cathy Yeah. That's the whole point. You don't have to wait for a paper. You don't have to reverse-engineer it. You can just… build with it.
Jessica The model weights are on Hugging Face under MCG-NJU/VideoChat3, the code is on GitHub at MCG-NJU/VideoChat3, and the datasets are all there too. If you're building a video understanding system and you don't have a billion-dollar budget, this is your starting point.
Cathy And the data construction pipeline is open, which means you can actually understand how they built the datasets. That's not trivial. Most papers, the data is a black box. Here, you can see the process.
Jessica Which means the community can improve it. You can add your own video types, your own scenarios, your own curriculum. The foundation is reproducible, so iteration is possible.
Cathy That's the rare part. Most video MLLMs are not built for iteration. They're built to be published and then left alone.
Jessica VideoChat3 is built to be built on. That's the actual product.
Cathy Yeah. Okay. I'm in.