Great for solo developers & teams alike.

Credits are simple. A response from any model costs 1 credit. Whether it's in a playground chat, or a response from a benchmark being executed - it all costs 1 credit. We make it clear before you run any action what the credit cost is.

We use OpenRouter to bring the 180+ models in ModelBench. Unless you're only using free models in the playground, you'll need to connect your OpenRouter account. This is very straightforward, and new OpenRouter accounts come with a small amount of free credits to get started. In our near term roadmap, we also have the ability to allow you to use provider keys (non-OpenRouter).

To "judge our judges", we actually use a panel of experienced LLM developers to evaluate and improve these - which we're going to share in an upcoming blog post.

Across 120 different domains and use cases, we found on average the pass or fail result to be 99.4% satisfactory. There are exceptions, which we are working on improving daily.

For more complex use cases, team plans allow you to invite guest judges to provide hybrid AI & human-based benchmarking (useful for more creative tasks).

They do not. We have a duty to ensure our own OpenRouter account is sufficiently rate limited, and in allowing credit rollover, we would put our application at risk of overloading the shared usage of certain application features.

In order to get more credits, you need to upgrade your plan or add more seats. If you're interested in enterprise pricing, get in touch.