BREAKING: AI costs a sh*tload.

  • June 6, 2024
  • Pete Whiting
  • 5 min read

Semi hot take incoming: investing in AI is a lot like buying a house

Sounds odd. We know. We’ll explain. 

On the surface both seem like wise investments. There’s a lot of potential upside. That one guy you knew in high school with "entrepreneur" and a plane emoji in his Twitter bio (we refuse to call it X) talks about both endlessly. And, if the market conditions are right, you know what you want, and are investing for the long term, then both will probably pay off. 

Oh, and one more thing, they both cost more than you think they will. That’s another big way they’re similar.

Houses have maintenance costs – about 1% of your home’s value per year. There are property taxes and buying a lawn mower and a ton of other things you may not be aware of until after you’ve signed on the dotted line. 

AI is no different. 

Four hidden costs of AI

As Mr. T always says, “I pity the fool who doesn’t diligently explore all potential costs before investing in new technologies.” 

Or something like that. 

If you’re thinking of investing in AI below are four hidden costs you need to consider.  

Paying for tokens 

Similar to maintenance costs with a house, AI has some ongoing costs. And one of the main ongoing costs is paying for tokens. What are tokens? We’re glad you asked. 

Almost all AI tools are powered by Large Languages Models (LLMs). Building your own LLM is generally cost prohibitive. So, most connect to existing LLMs developed by companies like Google and OpenAI (ChatGPT). And that connection costs money. 

Instead of paying a flat monthly, or yearly, rate, you pay by volume with tokens. Each request you make to the LLM has a certain token amount associated with it - so the more requests you make, the more it costs. Further complicating things is token definition. 

Defining a token and understanding a per-token cost is highly complex, although there are calculators out there that help. Generally speaking, the more instructions you provide to get your result, the higher the cost.

If you know that higher use means more dollars for you, then it’s not an issue. However, if there isn’t a correlation between the two you’ll start losing on your AI investment quickly. Even ChatGPT loses money right now despite its millions of paying users.

Building or training a LLM

Broadly speaking, a LLM is basically the “brain” of any AI product. Mainstream LLMs, due to their intended wide-ranging use, are trained on more generalized knowledge. So, let’s say you run an ecommerce business and want to use an AI Chatbot to help answer customer questions. 

Since the AI only has general knowledge, it will probably be able to tell a customer who the 13th president of the United States was (shout out Millard Fillmore) but it’s not going to be able to answer SKU specific questions or offer guidance on return policies. You’ll need additional training for that. 

There are a couple of ways to train a model, but they all require development time. That’s where the cost comes in - a financial cost to pay an external development partner to set things up or an opportunity cost to get an internal team member involved.

Updating your system

AI is still a very new, rapidly evolving, technology. Because of that, iterations and updates are also happening very quickly. Meaning, at worst, the tech you choose to invest in today could become obsolete fairly quickly. The best case scenario being regular updates and tweaks to stay aligned with market demands. 

A really classic example of this is the U-Matic vs VHS. Even though U-Matic was better on the surface, it didn’t catch on like VHS tapes did. The main lesson being it’s hard to know what technology is going to last when it first comes out. Given a couple years, winners become much clearer. But if you bet too soon you might end up with a really expensive paper weight.  

To keep update costs low, we recommend taking a measured, well-researched approach to your investments. It’s also a good idea to start small. That way if you need to make a change in the future you’re not left with a big bill or product that is quickly outdated. 

QA and general oversight

There are plenty of examples of AI tools making up information, or giving wrong answers to questions. Whether it’s Google Bard’s incorrect telescope reporting, or Midjourney making images of people with too many fingers and teeth, AI doesn’t always get everything right. 

Along with a propensity to make things up at times – ChatGPT “hallucinates” around 15-20% of the time – it’s also common for AI to give different answers to the same question. In some cases that isn’t too problematic, but with things like code it easily can be. 

Because of those realities, AI is not a “set it and forget it” technology. You need to be prepared - operationally and financially - to have resources checking AI’s work and making necessary changes to ensure mistakes don’t happen again.

Moving forward

Just like buying a house, AI can be a good investment. But only when the conditions are right. Make sure you’re taking the time to understand how AI plays into your larger goals, and ways it will contribute to the bottom line – whether by generating revenue, or through cost savings. And if it comes to a net positive even after taking into account all the associated costs, not just the upfront ones, AI might be right for you.  

And if you’re still on the fence, we wrote this article about five questions to ask before investing in AI. 

Also, if you liked this article please subscribe to our newsletter

Be assured that we will not sell, trade or otherwise share your information with any other party.

Interested in building with us?