How to Build a Self‑Healing Webhook Pipeline

The Problem

If you’ve ever worked with data pipelines, you already know the headache: garbage in, garbage out. I’ve been working with various MarTech stacks lately, and I realized that brittle data is a universal bottleneck.

How to build your own?

Well, LeadMend can be used as a reference project if you wish to build your own. This is something I commonly do when I’m deep in LLM-assisted development. I’ll literally clone a repo into my project folder and ask the LLM to analyze the architecture and the logic. It’s the fastest way to replicate and learn. However, even without an AI holding your hand, building your own pipeline is actually quite straightforward once you understand the flow.

We always start with the “dirty payload.” In my demo, I use prebuilt scenarios to simulate the chaos, but in reality, this would be genuine form entries from a website, a messy database export, or some ancient third-party integration throwing unhinged JSON at you.

Now that you have your source and your dirty webhook, we need to try to understand what actual problems this payload causes before we touch a database.

So, before this payload can be safely routed to your primary database or a Slack channel, we need to run it through a few crucial steps:

Step 1: The Pydantic First Responder (Normalizer)

First, we need to figure out if there are any missing fields. Let’s say you receive a webhook with a missing email field. A standard API drops the lead. But what if we actually do have the email, and the chaotic user just typed it inside the free-text message block? If it’s a situation that can be simply fixed, our Pydantic normalizer steps in, runs some regex, extracts the email from the message body, and maps it where it belongs. We are actively self-healing malformed strings on the fly.

Step 2: The Garbage Chute (Validator)

Just because we can format a lead doesn’t mean we should keep it. This step checks the lead itself for pure, unadulterated garbage data. We’re talking “Mickey Mouse” test entries or keys mashing on a keyboard. If it’s garbage, what’s the point of having it, right? Instead of wasting compute, the Validator short-circuits the process and dumps the nonsensical payload straight into an #ops-invalid channel.

Step 3: The Detective (Enrichment)

Okay, the lead isn’t garbage, and the formatting is fixed. But it might still be pretty sparse—maybe we just have a company name and a person’s name, no more. We take what we have and pass it to the Enrichment Service to fetch deep firmographics like company size and industry. (Note: It is incredibly important that this happens after the garbage filter, otherwise you are literally paying third-party API credits to enrich a fake lead!)

Step 4: The Deja Vu Catcher (Deduplication)

Now we have a beautifully enriched, fully formatted JSON object. Are we done? Nope. There is nothing a sales team hates more than accidentally double-dialing a prospect. We run a fuzzy SQL-based deduplication check against our database to identify if this is a returning lead or an exact match of an existing one.

Step 5: The Judge (Algorithmic Scoring)

Not all leads are created equal. Before routing, we pass the data to our Lead Scorer. This runs a multi-factor weighted algorithm based on the company size, the industry we just fetched, and the overall data quality to spit out a score from 0 to 100.

Step 6: The Traffic Cop (Tiered Routing)

Finally, we have a perfectly clean, enriched, unique, and scored lead. Now, the Intelligent Router decides its fate. Based on that 0-100 score, we map the lead to specific destinations. A high-value enterprise lead? Fire off an urgent webhook to n8n to put it in the CRM and ping the VIP Slack channel. A low-value lead? Just quietly persist the record in PostgreSQL and move on.

Why even build a demo like this?

So, why go through the trouble of building a multi-layered, fully dockerized pipeline just for a demo? Honestly, because it’s a genuinely fun project to do. It’s a great way to play around with a specific architectural concept: building systems that actually expect things to go wrong.

Now, let me be clear, this demo isn’t some omniscient magic bullet. You can’t just throw literally any random, unhinged string into this pipeline and expect it to magically come out perfectly clean. In this project, I use predetermined scenarios. I built specific types of “dirty” payloads to prove out the logic of catching, healing, and routing data. It’s a sandbox for demonstrating a reliable architectural pattern, not an attempt to build an impossible data-cleaning AI.

But the real reason I built this? It is the absolute perfect playground for practicing vibecoding.

If you are getting into LLM-assisted development, a scoped project like LeadMend is a dream scenario. You can lean on an LLM to hammer out the tedious boilerplate, spinning up the docker-compose.yml, laying out the basic FastAPI routes, or setting up the database connections.

By offloading the busywork, you get to stay in the flow state. You get to just… architect. You spend your time purely on the interesting stuff: tweaking the Pydantic regex, playing with the scoring algorithms, and visually mapping out the routing rules in n8n. It’s fast, it’s creative, and it’s honestly just a really enjoyable way to build software.