ENIGLIO TECH PRESS

Eduardo Niglio · Eniglio Tech Press

Reflections on AI engineering — Eduardo Niglio

Here are three reflections on evaluating, building, and improving AI systems.

By Eduardo Niglio

A convincing answer still needs evaluation

When I think about quality in AI systems, I try to separate two things: how an answer is presented and how well it actually meets the task. Fluent writing can inspire confidence even when it contains an unsupported conclusion. That is why I do not consider good presentation a sufficient reason to approve a solution.

I prefer to start by defining an acceptable answer. What information must be correct? What evidence supports the conclusion? When should the system acknowledge that it does not know? These questions help me build an evaluation that goes beyond selecting the most impressive examples.

For me, errors are also part of learning as an engineer. I want to understand what failed, under which conditions, and with what consequences. Before asking whether an answer is impressive, I ask: would I have enough evidence to trust it for this task?

Complexity needs to earn its place

I am interested in the possibilities of agents, tools, and information retrieval, but I try not to begin a project by choosing its architecture. I prefer to start with the problem. What does the user need to solve? What decision should the system support? What requirements for quality, security, and response time must it meet?

An additional step can be useful, but it also creates new dependencies and potential points of failure. That is why I consider it important to compare an elaborate solution with a simpler alternative. I want to know which specific difficulty that complexity addresses and how I will recognize an improvement.

To me, simplicity requires judgment. Some scenarios need more components; in others, adding them only makes a solution harder to understand and maintain. My question is: if I remove this step, which meaningful capability will disappear?

The cost I care about is the cost of a useful outcome

When comparing AI configurations, I try to look beyond the price of a single call. An alternative can look economical while requiring more retries, more review, or more of the user's time. If I consider only the unit price, I risk optimizing one part while making the overall experience worse.

That is why I prefer to define success first. Was the task completed? Did the answer meet the quality criteria? How much additional effort was needed? From there, I can discuss cost, latency, and reliability in relation to the same expected outcome.

I do not expect to find one ideal configuration for every situation. I want to make the choices explicit and compare alternatives under similar conditions. The question guiding my analysis is: how much does it cost to deliver an outcome the user can actually use?

Explore the bookBack to the main experience