Domain
Multimodal
47 episodes
-
OpenAI updating ChatGPT desktop app with GPT Voice for talking through work 9to5Mac
Masonry and Eyre dig into OpenAI bringing GPT Voice to the ChatGPT desktop app, where it can now coordinate work across Chat, Work, and Codex by voice. They focus on the real user story for people already living in the app, the Mac-only Appshots context boost, the computer-control angle, and the new multi-folder project setup. They also poke at the desktop-app clutter without losing sight of the workflow win.
-
Think through hard problems in voice mode | Claude by Anthropic
Asteria and Draco dig into Anthropic’s update to Claude voice mode, where Opus and Sonnet now power spoken sessions, connected tools are usable from voice, and multilingual support expands. They focus on the real argument: voice mode becomes useful when it’s no longer just fast chatter, but a place to work through half-formed thinking and then hand off to action. They also question where the feature stops being a convenience and starts being a real workflow, especially given model switching, permission prompts, and the different value between free and paid tiers.
-
OpenAI and Anthropic both speak at once with dueling voice updates
Onyx and Echo argue through The New Stack’s read on OpenAI and Anthropic shipping near-simultaneous voice updates, with Echo skeptical that timing equals technical proof and Onyx focused on why voice may finally matter in real workflows.
-
EvolvingWorld: An Open Schema Framework for Co Evolving Role Play Agents and World Model in Interactive Literary World
Masonry and Eyre dig into EvolvingWorld, a new framework that lets fictional characters and their world co-evolve across long stories. Eyre walks through the open-schema architecture and seven supervised tasks; Masonry sizes up who would actually build with this and where product pain lives. The hosts end up excited about the open-schema premise but skeptical of the benchmark’s generality.
-
Alibabas Tongyi Lab Releases Qwen Audio 3 0 TTS a Hosted Text to Speech Model in Flash and Plus Tiers Across 16 Languages
Cooper and Miles examine Alibaba's Qwen-Audio-3.0-TTS, comparing its Flash and Plus tiers, multilingual support, voice controls, architecture, hosted-only trade-offs, pricing, and real production use cases.
-
VideoChat3:Fully Open Video MLLM for Efficient and Generalist Video Understanding
VideoChat3 is a fully open-source video multimodal LLM (4B parameters) that tackles three concrete problems: generalization across short/long/streaming video, computational efficiency for video token explosion, and reproducibility through complete open-sourcing. The core innovation is I3D-ViT (Inflated 3D Vision Transformer) plus adaptive frame resolution, which compresses spatiotemporal redundancy early in the pipeline instead of treating each frame as an independent image. Three curated datasets (2M academic + 116K long-form + 617K streaming = 3M samples total) and multi-stage curriculum learning enable the model to handle diverse video scenarios. Jessica sees a shippable foundation for real-world video apps; Cathy pushes on whether the efficiency gains hold under production load and whether the data pipeline's scale claim is reproducible.
-
Overview: Embeddings
We’re finally doing a full pass on embeddings, because they keep showing up under half the things we talk about. We get into what an embedding actually is, why it turns meaning into usable coordinates, and why that little geometric trick sits under so much of modern AI.
-
Concurrent Image Understanding and Generation: Self Correcting Coupled Markov Jump Processes
Fern and Lintel dig into a new paper on doing image understanding and image generation at the same time, inside one decoding loop. The hook is simple: most systems either describe first and draw later, or they run both sides in parallel without letting the latest text and image decisions correct each other mid-step. This paper tries to fix that with a coupled masked-diffusion sampler that can both coordinate and backtrack.
-
Thinking Machines open sources first multimodal language model, Inkling, focused on low cost and 'resistance to censorship'
Inkling, Thinking Machines' open-source multimodal MoE model (975B total / 41B active parameters), lands as a broad, balanced generalist with a standout feature: a controllable 'thinking effort' knob to dial cost vs. performance from 0.2 to 0.99. Enterprises get native text+image+audio fusion, Apache 2.0 weights, and a lighter Inkling-Small preview, but benchmarks show it trails specialized open and closed models on coding and pure reasoning, while remaining competitive on multimodality and agentic workflows. The episode debates whether the real win is the runtime control surface (Tinker platform) and a cautious, non-censoring epistemics posture — not the headline parameters.
-
The A.I. Design Aesthetic That’s Taking Over the Internet
Justy and Cody dig into the argument that Claude Design is creating a recognizable internet look almost overnight, and why that matters less as a style complaint than as a product and workflow signal. They talk through the article’s evidence, where the claim holds technically, and why the real issue may be default paths, shared component libraries, and how much labor people are actually willing to spend to get past the default.
-
The Millions of Songs Mashed Into AI Generated Music
The article discusses the use of millions of songs to train AI music generators, raising concerns about copyright infringement and the impact on artists.
-
JoyAI VL Interaction: Real Time Vision Language Interaction Intelligence
JoyAI-VL-Interaction is an 8B vision-language model that flips the script on AI assistants: instead of waiting to be asked, it continuously watches a video stream and decides moment-to-moment whether to respond, stay silent, or delegate complex tasks to a background model. The system is fully deployable, open-sourced (weights, training recipe, data, and complete stack), and beats Doubao and Gemini's in-app video assistants on quality and timing in real-world scenarios.
-
LLM Agents Can See Code Repositories
Justy and Cody dig into SeeRepo, a paper asking whether coding agents should literally see repository structure instead of flattening everything into text. The result is narrower and more useful than the headline: vision alone is worse, but a hybrid setup that adds rendered dependency graphs during fault localization cuts token use and often keeps accuracy flat or slightly better.
-
OmniVideo 100K: A Dataset for Audio Visual Reasoning through Structured Scripts and Evidence Chains
Justy and Cody dig into OmniVideo-100K, a new dataset and generation pipeline for audio-visual reasoning. They focus on why the old video-caption-to-QA setup keeps breaking sound-to-source links, what entity-anchored scripting and clue-guided QA actually do, and whether this looks like a research artifact or something teams can build on now.
-
PixelRAG beats text parsers, cuts agent costs 10x
Justy and Cody dissect PixelRAG, a new research system that skips text parsing entirely by feeding rendered webpage screenshots directly to vision-language models. They break down the three specific failure modes of traditional parsers (parser loss, rank loss, reader loss) and discuss whether the 10x cost reduction and accuracy gains hold up against the engineering reality of managing image indices.
-
Apple Foundation Models
Apple's Claude for Foundation Models is a Swift package that wraps Claude into Apple's Foundation Models framework, letting developers swap Claude in and out of the same LanguageModelSession API used for on-device models. Requests route directly to Anthropic's API (Apple doesn't see them), and developers pay standard Claude API rates. The package handles model capabilities, effort levels, structured output, client and server-side tools, vision, and error mapping — all with the same interface whether you're calling Claude or an on-device model.
-
Lip Forcing: Few Step Autoregressive Diffusion for Real time Lip Synchronization
Justy and Cody dig into Lip Forcing, a paper on making diffusion-based video-to-video lip sync actually fast enough for streaming. They unpack the core problem, the teacher-student distillation setup, the key mid-trajectory guidance insight, and what the reported speedups might mean for real products like live translation, avatars, and dubbing systems.
-
I Spent May Evaluating Different Engines for OCR | Towards Data Science
Justy and Cody react to a hands-on OCR engine shootout across 93 messy real-world documents. The author’s core claim: OCR is now a routing problem, not a single-engine race—specialist models excel in their niche but break on out-of-domain docs, while paid structured APIs may be overkill for many use cases. They debate the economics, practicality of ‘classify-then-route,’ and whether most teams should just test on their own data.
-
Google's new open source Gemma 4 12B analyzes audio, video — and runs entirely locally on a typical 16GB enterprise laptop
Justy and Cody debate whether Google's new Gemma 4 12B—an 11.95B-parameter model that runs locally on 16GB laptops with encoder-free multimodal processing—is a genuine breakthrough for edge AI or just a cleverly marketed niche tool. They clash on the practical trade-offs: Cody questions the real-world performance and fine-tuning complexity, while Justy highlights the enterprise use cases where offline, private inference is non-negotiable. They land on it being a specialized win for specific scenarios, not a universal replacement.
-
Task Focused Memorization for Multimodal Agents
Justy and Cody dig into TaskMem, a paper on teaching multimodal agents what to remember from endless streams of video. They unpack the core idea of turning memory creation into a learnable policy, why that matters for embodied agents and long-horizon systems, and how the two-phase reinforcement learning setup tries to balance faithful recall with task usefulness.
-
SwanVoice: Expressive Long Form Zero Shot Speech Synthesis for Both Monologue and Dialogue
Justy and Cody dig into SwanVoice, a zero-shot text-to-speech paper aimed at long monologues and multi-speaker dialogue. They focus on the real bottleneck the paper targets: keeping a whole conversation acoustically and emotionally coherent instead of generating each turn separately and stitching it together. Cody breaks down the pipeline, data construction, VAE compression, flow-matching DiT, speaker-turn conditioning, and the training curriculum. Justy keeps pulling it back to production reality for podcasts, dramas, and multi-voice tools, while both note the paper’s strongest caveat: content accuracy still looks like the main weak spot.
-
How we chose the voices of Coda | Rime
The hosts discuss an article about how the voice model Coda was developed, focusing on the selection of voices and categorization into styles like professional, formal, casual, and energetic.
-
GitHub Resemble ai/DramaBox: super expressive prompting model based on ltx2
Justy and Cody dig into DramaBox, Resemble AI's expressive TTS model that uses screenplay-style prompts to control delivery, emotion, laughs, and pauses — built as an IC-LoRA fine-tune on top of Lightricks' LTX-2.3 audio model.
-
Thinking Machines shows off preview of near realtime AI voice and video conversation with new 'interaction models'
Thinking Machines previews 'interaction models'—AI that processes voice and video in real-time, simultaneously listening and responding instead of waiting for user input to finish. Cody is skeptical about whether this solves a real problem or is architectural theater; Justy argues the latency gains and enterprise safety use cases (manufacturing oversight, customer service) are genuinely useful. They debate whether 'full-duplex' is a fundamental shift or incremental polish on existing models.
-
Gemini API File Search is now multimodal: build efficient, verifiable RAG
Justy and Cody dig into Gemini API File Search getting multimodal retrieval, metadata filters, and page-level citations, and why that matters for anyone tired of flaky RAG over PDFs and image folders.
-
Building with Gemini Embedding 2: Agentic multimodal RAG and beyond Google Developers Blog
Exploring Next, episode 357. Gemini Embedding 2 just made multimodal retrieval a lot more practical: text, images, video, audio, and PDFs can all land in one embedding space, which changes search, RAG, and agent workflows.
-
SketchVLM: Vision language models can annotate images to explain thoughts and guide users
In this episode, Justy and Cody dig into SketchVLM, a training-free framework that lets vision-language models explain answers by drawing editable SVG annotations on top of images. They talk through why text-only answers are hard to verify, how SketchVLM uses a draft-and-refine loop plus visual grounding to produce overlays, where it looks production-friendly, and where the trade-offs still show up.
-
Openmoss Releases Moss Audio an Open Source Foundation Model for Speech Sound Music and Time Aware Audio Reasoning
Exploring Next, episode 331, on MOSS-Audio from OpenMOSS, an open-source foundation model that tries to handle speech, sound, music, and time-aware audio reasoning in one stack.
-
Text Summarization with Scikit LLM MachineLearningMastery
Justy and Cody kick around a MachineLearningMastery post on using scikit-LLM for text summarization inside scikit-learn pipelines. Cody is skeptical about the real value of wrapping a summarizer as a transformer, while Justy argues it fits messy, text-heavy workflows where teams already live in sklearn. They land on a cautious verdict: useful for specific preprocessing jobs, but not a magic shortcut, especially once cost, latency, and summary quality enter the picture.
-
Mind’s Eye: A Benchmark of Visual Abstraction, Transformation and Composition for Multimodal LLMs
Justy and Cody dig into Mind’s Eye, a new benchmark for testing whether multimodal models can actually do visual thinking like rotation, folding, analogy, and composition instead of just describing images well. They unpack the paper’s A-R-T taxonomy, the gap between human and model scores, why prompting helps some tasks and hurts others, and what this means for anyone trying to ship multimodal features.
-
Anthropic just launched Claude Design, an AI tool that turns prompts into prototypes and challenges Figma
Anthropic’s Claude Design is a big deal because it aims to collapse the gap between idea, prototype, and stakeholder feedback. Justy and Cody dig into why that matters now, what Claude Design likely does under the hood, why pairing it with Opus 4.7 matters, and where it could genuinely pressure Figma versus where the old product realities still bite.
-
MiniAppBench: Evaluating the Shift from Text to Interactive HTML Responses in LLM Powered Assistants
Exploring Next digs into MiniAppBench, a new benchmark that evaluates how well LLMs can generate interactive HTML applications instead of just text responses. The paper introduces 500 real-world tasks and an automated evaluation framework that tests apps like a human would. We break down the technical approach, discuss what this means for AI assistant interfaces, and identify specific tools listeners can experiment with.
-
LLM2Vec Gen: Generative Embeddings from Large Language Models
Episode 221 explores LLM2Vec-Gen, a breakthrough approach that creates embeddings by learning to represent what a language model would generate, rather than encoding the input. Instead of traditional contrastive learning, this method adds special tokens that capture the model's potential response, achieving state-of-the-art results while maintaining safety alignment and reasoning capabilities.
-
Thinking in Frames: How Visual Context and Test Time Scaling Empower Video Reasoning
Today, we dive into a game-changing approach to visual reasoning in video generation. How does this solve real-world problems?
-
Reddit The heart of the internet
In today's episode, we dive deep into an exciting achievement in the world of game development using AI. One developer crafted a pixel-art open-world shooter in just 24 hours using Gemini 3.0 Pro for both coding and art. We explore what this means for developers, the implications of using AI in creative workflows, and the future of game design. Join us as we unpack the significance of this innovative approach and its potential impact on the gaming industry.
-
moonshotai/Kimi K2.5 · Congratulations on this release and on one important realization!
The release of Moonshot AI's Kimi-K2.5 model marks a significant advancement in multimodal AI capabilities, enabling seamless integration of text and image processing. This technology not only enhances conversational AI but also opens new avenues for local deployment, making powerful tools accessible to a broader audience.
-
Flashlabs Researchers Release Chroma 1 0 a 4b Real Time Speech Dialogue Model with Personalized Voice Cloning
This episode dives into the groundbreaking Chroma 1.0 model, which offers real-time speech dialogue capabilities with personalized voice cloning. We explore its implications for various sectors, including entertainment and education, and discuss potential use cases that could reshape how we interact with technology.
-
GitHub ByteVisionLab/NextFlow: NextFlow🚀: Unified Sequential Modeling Activates Multimodal Understanding and Generation
NextFlow is a major advancement in multimodal AI, integrating text and image generation in a single framework. It enables rapid, high-quality visual generation and editing, which has significant implications for various industries, from content creation to education. This episode breaks down how NextFlow works, its real-world applications, and why it represents a paradigm shift in the field.
-
Scientists Create a “Periodic Table” for Artificial Intelligence
Researchers have created a unifying framework for multimodal AI, akin to a periodic table, helping developers efficiently design AI systems. This model can improve accuracy, reduce data needs, and make AI more environmentally friendly, potentially revolutionizing various applications in technology and healthcare.
-
Differences That Matter: Auditing Models for Capability Gap Discovery and Rectification
The research introduces AuditDM, a novel framework to audit multimodal LLMs by identifying their capability gaps through reinforcement learning. This approach not only helps in discovering failure modes but also offers a pathway for model improvement without extensive annotation. The implications for developers are significant, as they can utilize these insights to enhance model performance and effectiveness in real-world applications.
-
Paper page Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forward
This dialogue explores the research on Unified Multimodal Models, focusing on the gap between understanding and generation in AI systems. It emphasizes the significance of addressing this gap for practical applications and future advancements in AI technologies.
-
Paper page Grasp Any Region: Towards Precise, Contextual Pixel Understanding for Multimodal LLMs
This episode dives into the innovative research on Grasp Any Region (GAR), which enhances multimodal language models' ability to understand complex visual scenes. We discuss its practical implications for developers and the real-world applications that can benefit from this advanced technology.
-
Paper page RAG Anything: All in One RAG Framework
The RAG-Anything framework transforms how multimodal data is processed by integrating diverse knowledge types, addressing the limitations of current models. This innovation has significant implications for developers, enhancing user experience and expanding application areas. The discussion delves into practical uses, the technology's potential impact, and the challenges it still faces.
-
How to Use Frontier Vision LLMs: Qwen3 VL | Towards Data Science
Large Language Models How to Use Frontier Vision LLMs: Qwen3-VL Learn how you can use vision language models to perform advanced document understanding tasks. Eivind Kjosbakken Oct 20, 2025 11 min read Share Learn how to use vision LLMs.
-
Qwen3 VL · Ollama Blog
Qwen3-VL October 14, 2025 Qwen3-VL , the most powerful vision language model in the Qwen series is now available on Ollama’s cloud. The models will be made available locally soon.
-
Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources | Towards Data Science
Large Language Models Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources Why do few chatbots return figures from source documents in their responses? Partha Sarkar Nov 3, 2025 11 min read Share Photo by Steve Johnson on Unsplash Retrieval-Augmented Generation (RAG) has been one of the earliest and most successful applications of Generative AI.
-
xSVVTj9qiY
In this episode, we dive into the latest developments from the r/singularity community, focusing on AI advancements and the implications of human enhancement technologies. We discuss Google's SIMA 2, an innovative agent that interacts and learns in 3D environments, and what this means for our future.