To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.
To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.