A $2000/month LLM

I don’t want to get into the habit of reporting on rumours, so I’m not linking to an original story here. While there are no definitive sources, there’s bego seren talk of a $2000/month subscription for a new OpenAI LLM.

To justify a price like this, a service would need to do more than chat with you. It would need to be a tool that you can delegate tasks to, and trust that they would get done. It would be an employee, not a chatbot.

I don’t think we’re far from this now. I think three things are missing:

• A “run loop” that gives the LLM the ability to figure out what needs to be done next, and do it.
• The ability for an LLM to take actions on your behalf.
• An oversight framework, where a human can instruct, monitor and approve actions.

These don’t seem particularly difficult, but in practice the various frameworks we’ve been using (like langchain for example) have fallen short of delivering a truly useful agent. They get lost, go off on tangents, and waste a lot of time and tokens getting nothing useful done.

But it seems to me like we’re not that far from useful agents. Not AGI, but agents that can be productive in limited domains.

Larger context is a big part of this. To run a task to completion will require many steps and all the requests working on it need the output from all the previous steps. Gpt-4o and Llama 3.1 are only 128k, Claude 3 is 200k, while Gemini Pro comes with a 2m context window.

1m is sufficient for some pretty long tasks, but consider the cost of running jobs that are approaching these limits. At $1+ per million tokens, an agent looping on tasks will be expensive to run. Hence the $2000/month cost.

Will these agents be competent enough at the jobs we can assign them to justify that cost?