Human in the loop: oversight, quality, data, compliance

Monday, 9:40. A customer asks on chat where her parcel is. A draft reply is already waiting in the queue: order number, carrier status, expected delivery date. A human reads it, clicks approve, the reply goes out. Next to it, the machine has just finished an overnight batch of three hundred variant descriptions. That is the workflow from the previous chapter when it works.

The catch: that state does not hold on its own. The catalog changes, somebody adds an exception to a rule, a vendor swaps the model version. Six months later the same workflow returns output nobody would sign off on today. Oversight is what separates automation that removes work from automation that quietly ruins your catalog. Five things need setting up: approval gates, quality measurement on a sample, drift detection, accountability for the output, and the rules for handling data alongside your transparency duties.

Where to put an approval gate, and where to let go

A gate costs time. A process where a human signs off on every step is not automation, it is retyping work from one window into another. The place for a gate is set by a simple product: the risk of a single error times the scale it spreads across. Ask four questions at each step of the workflow:

• Who sees the error first: the customer, the sales channel or someone on the team?
• Reach: how many items does one bad rule touch?
• Reversibility: how fast can the change be undone?
• Cost: what does a single error cost once you count the complaint, the return and the labour of handling it?

Scale can invert the whole judgment. A typo in one description is a rounding error. The same typo introduced by a rule across twelve thousand items and pushed to three channels is a week of cleanup. That is why the tightest gate does not stand where the machine generates a single output. It stands where somebody changes a rule, a dictionary or a mapping, because one such change has more reach than a thousand individual results.

The gate is set by risk times scale, not by how good the tool feels.

The reverse holds just as strongly. Draft classification, a ticket summary for a support agent, a first pass at flagging duplicates: low risk, instant rollback, so a gate on every item wastes somebody's morning. A periodic sample is enough there. What that boundary looks like in customer service, where the machine answers repetitive questions about order status, is covered in the piece on automating order status replies.

The control sample: one hour a month instead of a hunch

A sample answers one question: is quality still where you last measured it. For the answer to mean anything, you cannot just grab whatever is easiest to reach. The first thirty items on a list are usually your bestsellers, the best-described and most hand-corrected part of the catalog. Draw in strata:

• Across categories: a few items from each major category.
• Fresh stock: a few items from the latest delivery.
• The tail: a few items from the part of the catalog nobody visits.

Keep the deliberately hard cases in a separate pool and score them separately, because folding them into one number drags the result down and ruins comparability between cycles.

Criteria written down before you score

Score pass or fail across a few independent dimensions, and write the criteria down before you look at anything. Four usually suffice:

• Consistency with the source data.
• Completeness of required elements.
• Correct format.
• Compliance with brand rules and any requirements specific to the category.

Whoever scores the sample should not know which items were already touched by a human. That knowledge shifts the judgment, even in honest people.

The log: single measurements become a curve

Record the result in the same sheet every time: date, workflow name, rule and prompt version, name and version of the model used, sampling method and sample size, score per dimension, error types, decision. The log is the only way to see a trend instead of a single point.

And on a well-run workflow the trend has a characteristic shape: the share of drafts approved without edits grows batch by batch, because reviewer corrections flow back into the rules and the next batch is better. If the curve is flat, corrections are going nowhere. If it drops, you have drift.

Quality drift: why March worked and June did not

A workflow rarely breaks overnight. It slides. There are three sources and each calls for a different response:

• The catalog: a new supplier sends a file where colours are in German and dimensions in inches, and the rules have never seen anything like it.
• The rules: somebody added an exception in April, somebody tweaked the prompt in May. Each change sensible on its own, and nobody has checked the whole.
• The tool: the vendor swaps the model version and the same prompt returns something different from last quarter.

Three mechanisms are enough to catch it:

• A recurring sample: on a fixed rhythm, for example monthly and on every larger upload to a channel.
• A control set: a dozen or so items with a known, approved result that you push through the workflow after every rule change and every notice of a new model version. A divergence on the control set is an earlier signal than a drop in your sample.
• External signals: channel rejections, returns marked "not as described", repeated service questions about something that should have been in the description. These arrive on their own; you do not have to go looking.

Decide in advance what happens when a threshold is crossed. A good rule reads: on systematic errors the workflow moves back one level, under tighter oversight, and stays there until two consecutive samples come back clean. Without a written response, the argument about whether things are bad enough starts over every time, and usually ends with the topic being postponed.

Who is accountable when the machine gets it wrong

Every workflow has one owner by name. Not a team, not a department. That person keeps the sampling rhythm, maintains the log, and holds the right to stop the automation.

Changes to rules, dictionaries and prompts get a separate approver, ideally somebody other than the author of the change, plus a short changelog: what changed, when, why, and what the sample showed afterwards. It is the same habit you already apply to pricing changes, and for the same reason: the reach of the change is wider than the reach of the person making it.

Separate the roles at review time as well. The author of a rule knows its intent and reads the output through that intent, so they consistently see more sense in it than a customer will. That is not about distrust, it is a matter of vantage point.

Data: what you may send to an external model

There is only one rule: as much data as the task requires, and not one field more. To write a product description a model needs product attributes, not an export of your customer database. To propose a category mapping it needs category names and examples, not the full table of purchase prices and margins.

As a rule, customer personal data does not leave for external models. Where a process genuinely requires it, that happens on a basis agreed beforehand and with the consequences understood, not as a by-product of a quick test. Treat sensitive counterparty data the same way: commercial terms, purchase prices, margins, contract text.

In practice this comes down to four habits:

• Copies and samples: work on those rather than the production database.
• Minimum identifiers: strip the ones the task does not need.
• No pasted attachments: supplier files do not go into tools outside the approved list.
• A tool list: the tools in use across the company, with an answer to three questions: where the vendor processes the data, how long it keeps it, and whether it trains models on it.

Building the list is often the hardest part of the whole clean-up, because it usually turns out there are more tools in play than anyone assumed.

Compliance: two disclosure duties, not a revolution

For an eCommerce store, the part of the AI Act (Regulation (EU) 2024/1689) that matters most is transparency. It comes down to two duties:

• Labelling content: AI-generated content has to be labelled appropriately.
• Disclosing the machine: a user has to know when they are interacting with an AI system rather than a person.

In practice that touches chat and automated replies in customer service, plus material published in your own channels. Separately, customer personal data falls under the GDPR, so every new tool in a service process is simultaneously a decision about data processing. This is informational material, not legal advice; confirm the scope of your own obligations with a lawyer.

Where to start

Let's be honest: in a typical store this is mostly a disclosure duty and process hygiene, not a revolution in how you work. Three things usually suffice to begin with:

• An unambiguous notice in the channels where a machine answers.
• A written labelling policy for generated content.
• The tool list from the previous section, with an answer to what data reaches each one.

Neglect this area, though, and it can stall a project at the worst possible moment: during an audit by a large partner, or on entry into a new market, when nobody in the company can say where service data ends up. The points to check are collected in our compliance checklist.

All of this oversight is far easier to sustain when it is built into the platform rather than bolted alongside it in spreadsheets and calendar reminders. How you build and maintain a store where gates, logs and permissions are part of the system is the subject of the next chapter.