Why replace a commercial LLM with a self-hosted model?

In this case, because the commercial model discarded information. Reading the same tender document, it extracted 0 to 3 qualification requirements where the self-hosted pipeline extracted 23 to 24. Cost savings were real but secondary — the deciding factor was completeness of extraction.

Can a 7B open model outperform a large commercial LLM?

On a narrow, well-engineered task, yes. The advantage does not come from the model being smarter — it comes from controlling the pipeline: reading every attachment in full rather than triaging, chunking with map-reduce so nothing is truncated, and taking numeric values from the official API instead of asking a model to read them.

Where does machine learning fit versus the language model?

They do different jobs. Multilingual sentence embeddings and a gradient-boosted classifier trained on the product's own labelled history handle ranking and scoring, evaluated with cross-validation, AUC-PR and Precision@K. The language model reads documents and acts as a judge with written justification. Exact figures come from the official API, never from a model.

← All work Engineering case study August 2026

We replaced a commercial LLM with our own engine — and found 23 documents where it found 1

LicitaSmart reads Brazilian public tenders and tells a company which ones it can actually win. Its first version ran on a commercial LLM API. We took that out and built a self-hosted engine instead — not to save money, but because the commercial model was quietly throwing information away. This is what changed, measured.

23–24Qualification documents extracted per tender, where the commercial model found 0–3

0 → dozensReal opportunities surfaced per week, from zero positives in 3,000 matches

100%Inference self-hosted — no third-party LLM anywhere in the pipeline

−86%Monthly inference and infrastructure cost

The problem nobody sees until they measure it

Brazil's government buys more than R$ 1 trillion a year, and it publishes every tender as a PDF — often with dozens of attachments, each holding requirements that decide whether a company is eligible. A mid-sized firm can't read that volume. That's the product: read everything, score the fit, tell the company where it has a real chance.

Version one used a commercial LLM API for reading and matching. It worked, in the sense that it returned answers. The failure was invisible until we tested it properly, and it came in two forms.

Commercial LLM

0–3

Qualification documents extracted from a tender that actually contained 23. It read the document partially and discarded the rest — silently, with confident output.

Self-hosted engine

23–24

The same tender, read end to end including every attachment. Nothing summarized away, because nothing was skipped in the first place.

The second failure was worse and harder to spot: across the 3,000 most recent matches in the database, the commercial pipeline had produced zero positives. Not a few. Zero. Every day, opportunities the customer could have won were passing through the system unnoticed — and nothing in the interface suggested anything was wrong. A system that finds nothing looks exactly like a market with nothing in it.

Why the model wasn't the problem

The instinct is to blame model quality and buy a bigger one. That reading is wrong, and it's the reason this case is worth publishing.

The commercial model was truncating because the pipeline around it was designed to triage — filter tenders by keyword, pick the "relevant" attachments, summarize early to save tokens. Every one of those decisions is defensible on cost, and every one of them throws away the exact information that decides eligibility. The requirement that disqualifies a company is rarely in the main document. It's in annex seven.

So we rebuilt around four principles, and those principles — not the model — produced the result:

Read the whole tender. No keyword triage before reading. Download everything, read it all, and let the match decide against the complete content.
Read every attachment. Not the relevant ones. All of them.
Never let a model produce a number. Values and line items come from the official procurement API. A language model that reads a price can hallucinate a price.
Store the tender decomposed, not summarized. The asset isn't a tidy summary — it's the full document, page by page, with tables and line items, queryable later.

The architecture

Everything below runs on our own infrastructure. No third-party LLM is called at any stage.

Ingestion — read everything, keep the structure

Every attachment is pulled and parsed with PyMuPDF, broken down page by page along with its tables, and written to a local database. This is the layer that decides the ceiling of everything downstream: what isn't ingested can never be matched.

Line items — from the source, with zero AI

Quantities and values are taken directly from the official PNCP /itens endpoint. Deterministic, auditable, and immune to hallucination. The rule is simple: language models read language, APIs provide figures.

Extraction — map-reduce over the full document

A locally hosted Qwen 2.5 7B reads the tender in chunks and extracts the summary, the qualification documents and the items — map-reduce rather than one truncated pass, which is precisely why nothing gets dropped at the tail of a long document.

Matching — three layers, not one

Keyword filtering, then multilingual sentence embeddings (paraphrase-multilingual-mpnet-base-v2) for semantic similarity, then a local model acting as judge that returns a score with written justification. Three layers because each one fails differently, and the combination cancels those failures out.

Supervised scoring — trained on our own labelled history

On top of the embeddings, a gradient-boosted classifier (XGBoost) trained on the product's accumulated compatibility history, plus a fine-tuned multilingual cross-encoder. Evaluated with cross-validation on AUC-PR, ROC-AUC and Precision@K — average precision because the classes are heavily imbalanced and plain accuracy would flatter a model that says "no" to everything.

What the semantic layer actually fixed

Two real failures from the old system show why generic keyword matching doesn't work on procurement language:

It stopped matching the wrong thing. A print shop selling "disposable" goods was being matched to tenders for disposable diapers. Same word, unrelated business.
It started matching the right thing. A company doing "vehicle rental with driver" now matches tenders for "school transportation" — no shared keyword, same service. That match is worth real money, and pure keyword search will never find it.

The decision that says the most about how we build

When we tied notification to summarization — a customer is only alerted once the tender has actually been read — someone proposed a safety valve: if a deadline is closing and the summary isn't ready, notify anyway.

That was rejected. The system does not alert without the analysis behind it, even at the cost of missing a deadline. Sending a customer an opportunity they can't evaluate isn't a feature; it's noise that erodes trust in every future alert. The choice was quality of information over speed of notification, and it's the same choice that drove the whole rebuild.

What this case proves for your project

LicitaSmart is a procurement product, and you almost certainly don't need one. What it demonstrates is the capability underneath, which transfers directly:

Ingesting messy, unstructured data at volume — thousands of PDFs with inconsistent structure, decomposed into something queryable.
Knowing where AI belongs and where it doesn't. Language models for language, official APIs for figures, deterministic parsing for structure. Most failed AI projects get this boundary wrong.
Supervised ML with honest evaluation. AUC-PR and Precision@K on imbalanced data, not a demo that looks good on a slide.
Running inference on our own infrastructure. If your data can't leave your environment — health, legal, financial — we've already built and operated that, in production, on a schedule.
Measuring instead of assuming. The zero-positives finding came from auditing our own output. That's the habit we bring to your data.

“The change we noticed wasn't technical — it was that opportunities started appearing at all. Before, the system would go through the day finding almost nothing, and we assumed that was the market. It wasn't. The tenders were there the whole time and we were blind to them. Now every match arrives with a score and the reasoning attached, so we know why it's there.”

Kamille Melo Partner, LicitaSmart

Disclosure: LicitaSmart is a Summitview product. Kamille Melo is a partner in LicitaSmart and is related to Summitview's founder. We're stating that plainly because the engineering figures on this page — 23–24 documents versus 0–3, zero positives in 3,000 matches — are the part that matters, and they're reproducible independently of anyone's opinion.

Have data that isn't telling you anything?

That's the work. We audit first and tell you honestly whether it's worth repairing or rebuilding — and you talk to the engineers who'd do it, not a sales layer.

Talk to us