Themes in AI

Over the last few weeks, I’ve talked with investors at an assortment of PE firms. In every conversation, I’ve typically asked about how their respective firms viewed the conventional AI boom. This was especially because I knew that some PE firms were beginning to invest in model providers, etc. From these chats, I’ll outline a few themes that I find compelling.

AI Cost Management

People are spending enormously on tokens. I remember reading a recent tweet by Garry Tan. He said something along the lines of if his engineers aren’t spending $50k/month on tokens, then he would be concerned. That sums up this push to use AI tools like Claude, Chat, etc. Let’s think back to the mid-2000s when cloud became a thing (AWS launched in 2006). First, we saw an adoption surge. AWS grows fast, so big bills are accepted as the cost of innovation. Eventually, companies get surprise bills and waste becomes obvious (idle EC2, forgotten snapshots, etc.). In time, engineers may begin building internal dashboards to track their server usage. This excites entrepreneurs to build companies like CloudHealth, Cloudability, etc. to professionalize these rookie scripts. As these startups grow, big players begin an acquisition spree. VMware bought CloudHealth; Apptio acquired others and so forth. Now, AWS, Acure, and GCP build native cost tools, but third parties survive on multi-cloud and depth. If we proxy this history lesson to our current day, then I believe we are sitting before this full-scale emergence of AI cost management startups.

The frontier labs are not incentivized to provide cost management services – the same way AWS and other cloud providers were not incentivized to provide cloud cost management solutions. Why make your users more cost-aware? This is why I think the AI cost management innovation we see would initially come from startups rather than internal products. LLM eval/observability companies (LangChain, Braintrust, etc.) may have a right-to-win here. They already work on solutions for recommending what model to use over a select group. Suggestions on pricing are a half-step away; however, one can argue that these eval/observability companies may be slightly disincentivized to offer these solutions. This is because (i) their revenue is tied to usage calls and (ii) their core identity/go-to-market is built around the developers and ML engineers, not finance or procurement, meaning the cost optimization conversation requires selling to a different buyer they have yet to cultivate. Thus, I’m most bullish on separate startups who can rationalize their value/pricing based on money saved for startups – they would sell to teams with no conflicting revenue interest in keeping spend high.

Open-Source Models

Almost as a supplement to the above thesis on AI cost management, the emergence of open-source as a likely solution for mature firms reflects an inclination to consider cost more. If PE investors are thinking about it, then enterprises (many that may be PE-backed) are arguably close to thinking about it [if not already]. The economics seem to check. Companies running substantial inference volume against a hosted API are basically renting compute at a meaningful markup. An open-source model like Llama, Mistral, or another (on owned/reserved infrastructure) are 6–7x cheaper. I can certainly understand why PE firms are thinking optimistically about them.

There is some latent risk worth acknowledging, though. Will the openness of models persist? Licenses may tighten (as Meta’s Llama licenses already have added commercial restrictions over time) and maybe some open-source models intentionally will become proprietary after adoption sufficiently rises. Will the quality of open-source models remain competitive enough? Distillation is surely a process that has helped some of these models increasingly do better. I would need to read further on the durability of distillation. Will enterprises continue to worry about China (e.g., re: DeepSeek or Moonshot)? Maybe – especially if you marry the first concern with data privacy problems.

AI as an allocation problem

A friend recommended a recent podcast with Dylan Patel of SemiAnalysis. Deven describes that there is a massive spectrum between (i) incessant AI token usage being justified given the benefits and (ii) the economics not working out and AI tokens being far too expensive on a cost basis. Along that spectrum, AI users need to get very good at allocating a budget to the fraction of total tasks they have. If we have 100 tasks for Opus to tackle, what are the 40 most important based on some combined metric of efficiency/my learning? Though a tangent, this efficiency-learning tradeoff is something I plan to think deeper on. I appreciate this style of thinking. It’s kind of like the demand-side version of everything I’ve just hypothesized on. Cost management tooling, model routing, OS adoption all are infrastructure solving for this far-out allocation problem – assuming firms expect to be tactical about AI usage.

This has an intriguing parallel to the attention economy. Firms got very good at allocating ad spend across channels. What impressions are worth paying for? Which aren’t? There is a similar discipline for AI token budgets. Firms are probably developing some type of perspective on which workflows justify GPT-4-class spend vs. a smaller model vs. an open-source model vs. no AI at all. As I alluded to earlier, I don’t believe firms should just index on cost. If they did, then I surmise that the learning curve of employees (in some contexts) may flatten. I want to work at a firm that prioritizes my learning first. Maybe I’m misguided, but this likely is how many people think. As I become smarter, I can become more productive. The capability for human judgement on model task viability probably also matters significantly – if we more forcefully think about allocation, then that can even stem into the human/org design direction. This kind of AI ROI theme could become a practice that MBB+ consulting firms think more deeply about.

Definitely excited to dig into these topics more. If anyone reads this, I would love some thoughts.