Topic
GitHub Copilot
7 episodes
-
Better tools made Copilot code review worse. Here's how we actually improved it.
Pippa and Tyler dig into GitHub’s post on why giving Copilot code review better tools actually regressed its performance—and how rewriting tool instructions for a reviewer’s workflow flipped the regression into a 20% cost win without losing review quality.
-
Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks
GitHub Copilot's agentic harness is a single cross-experience SDK component that orchestrates tools, context, and workflow across CLI, app, and code review. The team claims it delivers task-resolution parity with model-vendor harnesses while cutting token usage across several configurations, backed by public and internal benchmarks and real-world metrics. We debate technical validity, practical stakes for teams, and whether the harness should get most of the credit.
-
What 50,000 Runs of a 5 Line Eval Taught Us
The VS Code team ran a simple 5-line eval 50,000 times to test AI models' efficiency in completing a basic task: writing a string to a file. The goal was to see how reliably models could finish the work and what kinds of failures show up. The eval, called 'say_hello,' asks the model to add 'HELLO' to a file named 'HELLO.txt.'
-
How we built an internal data analytics agent
Qubot, GitHub's internal Copilot-powered analytics agent, lets any employee query the data warehouse in plain language. Cody digs into the architecture (federated context layer, MCP servers for Kusto and Trino, offline eval framework) and lands on the real bottleneck: curation—keeping documentation current so the agent doesn't hallucinate. Justy sees the workflow win (Slack iteration vs. Jira wait) but flags the caveat: this only works for exploratory questions, not high-stakes decisions. Both agree the eval loop is what separates a demo from a production system.
-
Validating agentic behavior when “correct” isn’t deterministic
GitHub's new validation framework for agentic systems moves beyond brittle, step-by-step testing toward outcome-focused validation. When autonomous agents (like Copilot Coding Agent) interact with real environments, correctness is no longer deterministic—loading screens may appear or vanish, timing shifts, and multiple valid action sequences can succeed. The framework uses dominator analysis and graph-based modeling (Prefix Tree Acceptors) to distinguish between essential outcomes and incidental noise, requiring only 2–10 successful traces to build a ground-truth model. Cody finds the approach clever but questions whether it scales beyond UI automation; Justy sees real market traction in CI/CD reliability and enterprise adoption.
-
Copilot Content Exclusion REST API in public preview GitHub Changelog
GitHub's new Content Exclusion REST API lets organizations programmatically manage what code Copilot can and can't learn from — a game-changer for enterprises juggling AI productivity with IP protection.
-
Configure MCP server access for your organization or enterprise GitHub Docs
GitHub Copilot / How-tos / Administer Copilot / Manage MCP usage / Configure MCP server access Configure MCP server access for your organization or enterprise You can configure an MCP registry URL and access control policy to determine which MCP servers developers can discover and use in supported IDEs with GitHub Copilot. Who can use this feature?