---
language: en
title: "Data cleaning"
description: "Documented handling of errors, formats, and duplicates before analysis, while preserving original data, exclusion rules, and the basis for metrics."
canonical: "https://tropk.ai/en/resources/glossary/data-cleaning"
last_modified: "2026-10-12T16:12:02.702Z"
---

# Data cleaning

Reviewed on 2026-10-06.

## What it means

Data cleaning is the process of handling errors, inconsistent formats, and duplicates to prepare a dataset for analysis. It can involve standardizing dates, correcting values, and separating invalid records. In AI monitoring, this includes distinguishing collection failures from valid responses. The process should preserve the original data and record what changed, using verifiable criteria.

## Why it matters for marketing

A rate can change without the responses changing, simply because records entered or left the dataset. The team needs to trust both the final number and the decisions that produced it. Removing valid responses that are unfavorable to the brand artificially improves the presentation. Correcting an import duplicate is a different situation, requiring evidence of the error and an explanation of the denominator.

## Example in practice

Imagine an analyst receiving 120 rows of responses: ten repeat the same execution ID due to an export error, and five record a timeout with no response. The analyst preserves the file as received, marks the reasons, and calculates brand presence based on 105 valid responses. The report retains the ten duplicates and five failures as distinct states, without converting them into absences.

## How to apply or verify

Define the identification key and rules before calculating metrics. Keep the raw file, processed version, operation history, and a list of exclusions with reasons. Check samples of the corrections and reconcile input and output totals. Tools such as OpenRefine offer previews of transformations and a reversible history; it is still up to the person in charge to decide whether similar values represent the same object.

## Important distinctions

Identical responses from different runs may be legitimate observations, not duplicates. Textual similarity also does not prove that brands or URLs are identical. Cleaning corrects problems defined in the protocol; selecting a sample determines which observations will be studied. Neither authorizes discarding results for convenience. Null values, collection failures, and genuine absences must remain distinguishable in the dataset and in reporting results.

## Sources and references

- [Cell editing — Transform and Cluster and edit](<https://openrefine.org/docs/manual/cellediting>) — OpenRefine
- [Running OpenRefine — History (Undo/Redo)](<https://openrefine.org/docs/manual/running#history-undoredo>) — OpenRefine

## Continue a leitura

- [Prompt set](</en/resources/glossary/prompt-set>)
- [Non-determinism in LLMs](</en/resources/glossary/non-determinism-in-llms>)
- [AI presence rate](</en/resources/glossary/ai-presence-rate>)

## About the author

[Vinicius Pacheco](</en/team/vinicius-pacheco>)

Responsible for Tropk’s technical content, focusing on crawling, indexing, structured data, and agent access to content.

---

[All pages in Markdown](<https://tropk.ai/sitemap.md>)