Skip to content
Michael Lizzio

Legacy Data Cleanup

Tools for finding and fixing decades of messy records, with a person approving the fixes

  • JavaScript
  • Node.js
  • Python
  • Flask
  • PostgreSQL
  • Local language models
  • Data profiling

July 2026Internship project

AI-generated
What it is
Tools that find and fix messy records left over from a county's old DOS system, with a person approving each fix.
Why I made it
When the old data moved to a new web app, notes got split across rows, words got stuck together and values landed in the wrong place.
What I did
At my 2026 internship at GPI, I designed the steps and checked the work, and AI wrote most of the code: scripts that flag odd values, an AI pass in small batches, a review app and a strict spacing fixer.
How it turned out
A dry run on all the data found 21,616 spacing fixes with zero rule breaks, and every other change goes through a person first.

The details

Why the data was messy

In the old system, people typed long notes into narrow fixed-width fields that wrapped onto the next row. When it was imported, those line breaks disappeared, so words got glued together and comments got chopped into pieces. The tricky part is that some weird-looking values are actually correct, so you can't just fix everything that looks odd.

Finding the problems

First I wrote scripts that scan every column and flag what looks wrong, without changing anything. A rare value in a column full of repeats is probably a typo, while a rare value in a column of IDs is normal.

Using AI carefully

The flagged records went to an AI in small batches, with instructions for how to read the table and exactly what format to answer in. Anything that came back incomplete or didn't match its batch got rejected. Then a review app showed each suggestion next to the original cells, and a person approved, edited or dismissed it.

    • Old recordsImported from the DOS system
    • Profile and detectFlags suspicious cells and pieces of notes
    • AI in small batchesSuggests how to put them back together
    • Human reviewApprove, edit, flag or dismiss
    • Staged changesVerify, back up, then apply

Fixing the easy ones automatically

One kind of problem was purely mechanical: a missing space where a line wrapped. For those I made a separate fixer that's only allowed to add or remove spaces, never change a letter or a number. If even one change broke that rule, the whole run stopped. A dry run on all the data found 21,616 fixes and zero rule breaks.

How I built it

Most of this code was written by AI. My part was explaining where the data came from, giving it real examples, deciding what the steps should be, and checking its work. Since the AI wrote so much of it, the strict checks and the human review are what made me trust the results.