Stewart Alsop

We live in an alphabet soup of AI benchmarks - MMLU, GPQA Diamond, HLE, LiveCodeBench, SciCode, AIME 2025, IFBench, AA-LCR, Terminal-Bench Hard. Benchmarks are supposed to measure progress. Instead, they’ve become the AI equivalent of EBITDA: technically informative, practically an exercise in costume design. Two predictable failures: ▪️Saturation: Everyone hits 90% eventually. Once frontier models ace MMLU or GSM-8K, the test stops meaning anything. It’s like grading Olympic sprinters on whether they can jog a 10-minute mile. Impressive the first time, meaningless once everyone’s jogging across the finish line. The scores go up, the signal goes down. So researchers invent the next “harder” exam, and the cycle repeats. ▪️Optimization: Benchmarks get hacked. Models fine-tune for the test, not the world. You can wire a model to dominate HumanEval without making it a better programmer. You can goose GSM-8K with chain-of-thought tricks that would never help in an actual math class. In some cases (hello, Llama 4), vendors allegedly submitted “special” benchmark-only models. That’s not progress - it’s leaderboard cosplay. If you’re buying AI solutions, this is where things get tricky. On paper, everyone looks great. But none of that tells you whether Vendor X’s chatbot will resolve your customer tickets, or Vendor Y’s model will flag fraud in your transaction flow better than the rest. Benchmarks test the generic. Your business runs on the specific. Which is why you need a golden dataset: a private exam with your data, your edge cases, your definition of success. Fraud logs in finance. De-identified patient notes in healthcare. Customer chat transcripts with the slang and sarcasm only your users know. This is boring work, but it’s also the only thing that matters. Because: 1. Different vendors optimize differently. One for reasoning, one for retrieval, one for latency. All of them might be “top-ranked” broadly - but only your dataset will reveal which matches your needs. 2. The real world is messy. Benchmarks (especially static ones) sanitize the world. Your data has noise, ambiguity, system interactions, edge cases, domain jargon. A model might get 95 % on MMLU but tank on your rare edge case. 3. Contamination happens. Tests leak into training data, so models memorize. Private datasets cut through that. 4. The tails matter. In public benchmarks, everyone clusters near the top. The spread shows up in the long-tail failures. Benchmarks are like standardized tests. They’re useful for broad comparisons, but if you’re hiring, you don’t just want to know if a candidate can ace the SAT. You want to know if they can actually do the job you’re hiring for. AI is no different. So yes, clap politely when a model sets a new record. But when it comes to procurement, bring your own exam. The only benchmark that matters is the one no one else has - yours.

124

9 Comments

"OpenAI has signed about $1tn in deals this year for computing power to run its artificial intelligence models...the deals would give OpenAI access to more than 20 gigawatts of computing capacity, roughly equivalent to the power from 20 nuclear reactors, over the next decade." https://lnkd.in/eGirY8hc

146

28 Comments

This founder is solving a massive bottleneck in space technology - satellites collect 100 TB of data daily but take a week to send it back to Earth. His optical ground stations can do it in just 3 hours, making data transfer 100x faster. With a team that's built quantum computers and quantum satellite prototypes, plus an advisor who mentors: Astronaut Shubhanshu Shukla, they're already getting commitments from major satellite players. Shreyaans Jain QOSMIC

135

11 Comments

Europe is f*cking hype atm 🙌 In the past 2 days... 👉 Fyxer AI closes $30m Series B (Richard Hollingsworth, Archie Hollingsworth & Matthew Ffrench, well done my guys 😉 ) 👉 Born closes $15m Series A (Fabian Kamberi is a Project Europe Partner and 20VC OG) 👉 ElevenLabs hits $6.6bn valuation with $100m employee tender (Mati Staniszewski is a special special founder) 👉 Mistral AI raises $1.7bn led by ASML at €11.7bn post 👀 And they say great companies companies can't be built out of Europe 😉

761

46 Comments

Hear why I warn that regulators will be watching the NVIDIA-OpenAI deal very closely… On this TWiST highlight, I am eager to dig into the details on Nvidia and OpenAI’s blockbuster, reportedly $100 billion new datacenter partnership. Some big questions remain: How are these funds being paid out, and over what kind of a timeframe? Is this an example of “round-tripping,” a questionable financial practice in which companies essentially move cash around? What, if anything, will the SEC have to say about all this? Hear me and Lon Harris unpack all this and more in a This Week in Startups exclusive clip.

76

9 Comments

Is your AI startup's first $10M ARR really just a tech echo chamber? Our latest analysis reveals that for Bay Area founders, 60-65% of early revenue comes from fellow tech companies—not the broader U.S. market in states like Ohio or Florida—based on data from CB Insights, PitchBook, and McKinsey. We've unpacked global variations, counterexamples like Veracyte and Kabbage, and actionable strategies to break free—check out the full breakdown in my new post! What’s your take on diversifying beyond the Valley bubble? Share your experiences or tips in the comments! https://lnkd.in/gCKTVyT4 #AIStartups #TechRevenue #Entrepreneurship #VentureCapital

37