GPT 5 vs GPT OSS 20B vs GPT OSS 120B

A real fraud detection use case

At Lightning AI, we use our own platform to train and deploy various agents/models that we use to detect various fraudulent activities. When GPT-5 came out, we wanted to understand whether we should upgrade one of our fraud systems to use GPT-5. In this article, I go into the methodology we use to decide to upgrade an agent or not to highlight the importance of having good evals. 

I won't spoil the article too much, but after testing GPT-5, GPT OSS 120B, GPT OSS 20B and 5 other models, we decided to stick to GPT-4o. Read on to find out why.

Fraud agent

While I can't get into the specifics of this use case because we actually use it in production, I can motivate it with a similar example that has similar characteristics. For all intent and purposes, let's imagine we want to detect fraudulent credit card activity. Each activity entry has associated metadata with it.

Data

This agent evaluates credit card transactions to determine if they are fraudulent or not... in reality, you'd also look at other data by the user (other transactions) to identify anomalies. But for this blog, we keep it simple. 

Here are synthetic credit card transactions (i'll explain in a different blog how to generated these):

Deterministic agent decisions

The agent itself is built with LitAI our new minimal framework for building AI agents and routing LLM calls. I personally hate non-deterministic, magical agents, so we build very deterministic agents that give us deterministic outcomes we can use to explain the model.... here's a snippet of an agent like this for this fake credit card transaction example.

The idea is simple... identify each thing you care about and check it one by one using an LLM as a yes/no classifier.

# [suspicious] check if it's suspicious first. 
is_suspicious_desc, reason_desc = self._is_sus_description_or_merchant(transaction)
if is_suspicious_desc:
  return reason_desc
  
# [high risk] if not suspicious, check if it's high risk
is_high_risk, reason_risk = self._is_high_risk_item(transaction)
if is_high_risk:
  return reason_risk

one of these decisions is a simple yes/no decision by an LLM

# [Yes/no decision] Force the method to return only True or False
def _is_suspicious_description_or_merchant(self, transaction_data):
    # [transaction] pull out relevant metadata for the transaction
    condition_string = (
        f"merchant name: {transaction_data.get('merchant_name', 'n/a')}\n"
        f"transaction description: {transaction_data.get('transaction_description', 'n/a')}"
    )
    # [binary prompt] describe what to look for in a suspicious description or merchant
    prompt = f"""
    <task>
    you are a fraud detection assistant. 
    analyze the following credit card transaction to determine if the transaction description
    or merchant name seem suspicious or indicative of fraud.

    consider factors like:
    - overly generic descriptions (e.g., "online purchase") for large amounts.
    - vague merchant names that don't match typical businesses.
    - unusual keyword combinations.
    </task>

    <transaction_details>
    merchant name: {transaction_data.get('merchant_name')}
    transaction description: {transaction_data.get('transaction_description')}
    </transaction_details>

    do the transaction description or merchant name seem suspicious or indicative of fraud?
    """
    # [LLM if] LitAI has this simple utility for an LLM if statement
    is_suspicious = self.llm.if_(condition_string, prompt)
    
    # [Deterministic reason] return a deterministic reason...
    reason = "sus description/merchant (llm)" if is_suspicious else "desc/merchant ok (llm)"
    return is_suspicious, reason

Benchmarking methodology

To benchmark which model is better, we loop through a few key models and run the transactions through the agent to give us the predictions. I'll get into all the benchmarking details in a separate article, but the important thing here is that we will measure a few summary statistics (mean, median, etc...) for the predictions and plot a confusion matrix to see why it's making mistakes. Another benefit of having deterministic answers is that we can frequency count and see where the model makes the wrong decision description/merchant seems ok (llm decision). 

Breaking up agent decisions into sub decisions gives us clarity into what it gets right or wrong so we can improve.

To loop through the models, we again, use LitAI and simply run the program with different models:

# pip install litai
# [Router] LitAI provides model routing as well (https://github.com/Lightning-AI/litAI)
from litai import LLM

# [Models] we ran through 8 models (only showing a few here)
models_to_benchmark = [
        'google/gemini-2.5-flash',
        'lightning-ai/gpt-oss-20b',
        'lightning-ai/gpt-oss-120b',
        'openai/gpt-5',
        'openai/gpt-4o',
        'google/gemini-2.5-flash', 
]
# [end]

all_benchmark_results = []

for model_name in models_to_benchmark:
    # [Detector] run the detector with the model
    llm = LLM(model=model_name, fallback_models=['openai/gpt-5'])
    detector = CreditCardFraudDetector(llm)
   
    # [Results] collect all results for plotting
    results = run_benchmark(detector, transactions_df, model_name)
    all_benchmark_results.append(results)

Results

The model we care about is the one who will give us the highest accuracy for the lowest latency.... cost is not so much of an issue, since all these models are fairly cheap.  Remember that each call needs to be fast because the agent system makes about a dozen calls (which can be parallelized of course). But nevertheless, latency adds up fast.

Here's a summary of the results:

The model, is the model name we use in LitAI. The accuracy is how often the model got the prediction right. Both Avg time and median times show the time to generate the prediction across 2,000 samples. Cost is the price of this model (input tokens/output tokens per million). (remember a token is a part of a word).

Let's go model by model

GPT 5

GPT 5 had the second highest accuracy after GPT 4o (76.37%). However, it is 5-6x slower than GPT-4o, so it makes it unusable for our use case, because the fraud agent cannot wait 8s per each decision - especially because the agent makes a few dozen of these decisions.... so it would take multiple minutes to make a classification decision.

Here's a histogram of how long it took to respond across 2,000 calls.

GPT-5-mini

The mini models are supposed to be much faster. Since GPT-5 was so slow, we decided to try the smaller versions to see if the accuracy could hold up but give us faster results.

The mini variant got 71.04% accuracy. The mini does speed up the prediction by 2x, but at a cost of 6 points of accuracy. In a fraud model, every single point of accuracy makes a massive difference. 

Overall, it was 2x faster than GPT 5 on average. However, this distribution is interesting because a large portion of results were actually super fast, but a non-negligible number were very slow. 

As a result of the unpredictability, we discard this as a viable candidate.

GPT-5-nano

Next up we have the nano. This one is consistently fast at 2.59s on average, or 3-4x faster than GPT 5. However the accuracy is 68.47% which is a full 9 points worse than GPT 5.

Although this model makes the latency more bearable, the agent degradation is too strong to be viable in production. 

GPT-5 OSS 120B

So far, the GPT 5 models aren't fast enough or accurate enough to be useful in our fraud detection use case. We decided to try the opensource variants to see how they perform. Our platform has these models available on a pay-by-token basis (this model is $0.10/0.40 for input/output tokens) (get access here).

The opensource 120B variant got 70.24% accuracy with a super fast prediction time of 1.72s per prediction. Now, the speed is more about the inference provider than anything else in this case. This model is not hosted by OpenAI, but optimized by our expert in-house PyTorch team and deployed on the Lightning platform. 

However, at 70% accuracy, this model was a good experiment but not really useful for our production system that needs to be closer to 80% to be usable.

GPT-5 OSS 20B

Next we tried the 20B opensource equivalent, again hosted on our platform. Trying this model is a bit counter-intuitive because if the 120B doesn't gives us the accuracy we want, then the 20B probably wouldn't either... right? 

But welcome to AI... not everything is what it seems. Shockingly, the GPT-5 OSS 20B variant got  73.42% accuracy 🤯🤯. Three full points higher than the 120B variant and not too far from the full GPT-5 model but at 1/25th the price (GPT 5 is $1.25 per million tokens and GPT OSS 20B is $0.05 per million tokens).... 

Speed-wise, this model is .4s faster on average than the 120B. These numbers are so close to the 120B that it probably just means that our PyTorch team can iterate on this model this week and drastically speed it up, since there should be a much bigger gap in performance given the parameter size difference.

At this price, latency and accuracy, this becomes a pretty viable model now for this production agent.

Gemini Flash 2.5

Our agents are not partial to any model provider... we want our fraud detection system to have the best performance on whatever model is needed. 

This Gemini model gets a 74.56% accuracy which is slightly worse than GPT-5 (at 76.37% accuracy). However it is 2-3x faster than GPT 5, but still slow at 3.68s.

GPT 5 OSS 20B is still the top contender given the accuracy for the value/speed.

Gemini Flash 2.5 lite

The Lite version of Flash 2.5 is supposed to be very fast. If we can get the same performance of Flash 2.5 Lite but at much faster speeds, then we might have a new winner.

This model gets 66.3% accuracy. This is by far the least accurate model (the top is still GPT 5 at 76.37% accuracy). 

Flash Lite does stay true to its promise of being the fastest at 1.28s... much faster than the GPT OSS models. But I think the GPT 5 OSS models provide a real alternative to this model given that they are opensource AND a lot more accurate.

GPT 5 OSS 20B is still winning.

GPT 4o

Finally, we test our baseline model (the current model we use in this fraud agent).

This model gets 77.41% accuracy, slightly higher than GPT 5. However, GPT 4o is 5-6x faster than GPT 5 🤯🤯🤯... if you noticed this while typing into the ChatGPT window, then you aren't crazy.

Not only is GPT-4o 5-6x faster than GPT 5 but it's actually more accurate. However, the cost is $2.50 per million token, a full 50x more expensive than the GPT 5 OSS 20B hosted at Lightning AI for $0.05 cents and 2x the price of GPT 5.

Conclusion

Start with evals

Overall, I would strongly suggest everyone have a strong baseline before deciding to switch to a new model. In AI research we often create a static dataset that we benchmark against to see how certain model decisions affect the overall outcome.

In industry, you can do the same by creating a dataset for yourself that you can run new versions of your agent or model to understand the impact of this choice. 

If we had just jumped on the hype bandwagon without actually running our internal evals, we would have degraded the performance of a pretty critical production system by ~2% in accuracy and made it 5-6x slower. Instead, we ran rigorous evals to quantify the impact of these models.

GPT OSS and GPT-4 win

Based on these results, we will just keep GPT 4o as the model that powers this fraud detection agent. The added 3 percentage points of accuracy make a huge difference to the business outcomes, so we are happy to pay 50x for the tokens for this use case. However, if a the difference in accuracy shrank to 1-2% vs GPT 4o, we'll move the agent to use GPT-5 OSS 20B because it's 50x cheaper than GPT 4o.

Switching for us also takes 2 seconds because we just need to change the model name in our LitAI LLM calls

from litai import LLM

# [Using GPT 4] LitAI with GPT 4
llm = LLM('openai/gpt-4o')
llm.chat('hi')

# [GPT 5 oss] Switching to GPT 5 OSS 20B is trivial
llm = LLM('lightning-ai/gpt-oss-20b')
llm.chat('hi')

Tooling setup

This section breaks down all the tools we used here.

Agent framework

We used LitAI to write the agent (opensource). It's simple, minimal and allows us to avoid the magic of graphs and unnecessary clutter so we can have full transparency into decision making.

pip install litai

LitAI also handles all the retries and fallbacks to other models in case a model call fails (which it will)

from litai import LLM

llm = LLM(model="openai/gpt-5", fallback_models=['openai/gpt-4o'])
answer = llm.chat("who are you?")
print(answer)

# I'm an AI by OpenAI

Model Router

All LLM calls were routed through Lightning AI's model router so we could pay for everything with Lightning tokens and have a simple place to monitor all these calls.

If you use the OpenAI client directly, here is the URL

from openai import OpenAI

client = OpenAI(
  base_url="https://lightning.ai/api/v1",
  api_key="LIGHTNING_API_KEY",
)

completion = client.chat.completions.create(
  model="openai/gpt-5-mini",
  messages=[
    {
      "role": "user",
      "content": "What is a fun fact about space?"
    }
  ]
)

print(completion.choices[0].message.content)

Measuring costs

Overall, this benchmarking cost about $20 to run all 2,000 transactions to each of the 8 models. The costs are easy to see in the Lightning model router which centralizes all LLM calls across every project in your organization with the ability to filter by user, set limits, projects, etc...

Development Studio

While I can't give you the real data, I will publish a Studio with the credit card use case for you to run your own benchmarks and see for yourself.

All these benchmarks were ran on a Lightning AI Studio - a persistent cloud VM. You can start a Studio in a few seconds, ssh in with your favorite IDE or code right from the browser. The Studio allowed me to run these benchmarks overnight since it's all on the cloud. When the benchmarks were done, the Studio automatically went to sleep which means I didn't get charged when it was idle. Next morning, I just turned on the Studio, and pulled out the results since it's got built-in persistent storage.

Read more blogs and case studies:

• From Flask to vLLM: How Model Inference has evolved (2017-2025)
• OpenClaw Hit 100K GitHub Stars in Weeks: What It Is and Why Running It Locally Is Risky
• View all