To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason is that each Model can architecturally have different tokenizers, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.
To expand on this, using tokens as a unit of billing is quite reasonable as it is also the unit of (inference) cost for LLMs.
The inference is essentially nothing but a loop of “Hey if the first X tokens are this, what would token X+1 most likely be?” until the LLM says its done. Every single loop is a ton of math which, on ridiculoudly expensive hardware, uses electricity and produces heat. So its a combination of those costs, and then we haven’t even taken into account the cost to train the model.
To add on, when using GitHub Copilot you can see a distinction between AI “Credits” and Tokens (in the context window for that conversation). One reason is that each Model can architecturally have different tokenizers, so one what one Model considers as one Token is different from another. To normalize the overall compute costs across different Models, the AI Credit unit is used as an abstraction.