API & webhook integration

The bug that taught me the most looked like success. A POST to create a contact timed out, my code retried it, and the retry went through. Two contacts, one real person, and the CRM reported no error either time. Retries without idempotency create duplicates. That one pattern, more than any diagram of push versus pull, is what separates an integration that runs unattended from one you babysit.

Most GTM integrations are written for the happy path and then patched every time reality intrudes. The network is not reliable, the sender does not deliver exactly once, and the rate limit is a hard wall, not a polite suggestion. You either design for those three facts up front or you discover them one incident at a time, usually at quarter-end when the deal that should have provisioned did not. The good news is that the patterns are old, boring, and borrowed from payments infrastructure, where getting a duplicate charge wrong costs real money. Copy them wholesale.

Idempotency and retries are one pattern

Networks drop responses. Not requests, responses. Your write may have landed even though you got a timeout, so a blind retry creates a second record. Stripe solved this years ago and the pattern is worth copying wholesale: send an idempotency key on every POST, a stable unique string per logical operation. If the server already processed that key, it returns the original result instead of doing the work twice. Now a retry is safe by construction, not by luck.

Pair it with exponential backoff, and add jitter. Fixed-interval retries from many clients synchronize into a thundering herd that hammers a recovering service at the same instant, which turns a brief outage into a long one. AWS found that adding randomized jitter to backoff cut retry volume by more than half. So retry at roughly 1s, 2s, 4s, 8s, but scatter each by a random fraction. In the CRM world you often get idempotency for free by keying writes on a stable external ID: an upsert on external_id instead of a blind insert means a duplicate write updates the existing row instead of creating a twin.

# Idempotent write with jittered exponential backoff.
# The key is stable per logical operation, so every retry is the same write.
import random, time, requests

def idempotent_post(url, payload, idem_key, max_attempts=5):
    for attempt in range(max_attempts):
        resp = requests.post(
            url, json=payload,
            headers={"Idempotency-Key": idem_key},
            timeout=10,
        )
        if resp.status_code == 429:                      # rate limited
            wait = int(resp.headers.get("Retry-After", 2 ** attempt))
        elif resp.status_code >= 500:                    # server-side, safe to retry
            wait = 2 ** attempt
        else:
            return resp                                  # 2xx or 4xx: stop
        time.sleep(wait + random.uniform(0, wait))       # jitter: 0..wait extra
    raise RuntimeError(f"exhausted {max_attempts} attempts for {idem_key}")

The two failure classes matter here. A 429 is the server telling you to slow down, so honor its Retry-After header rather than guessing. A 5xx is the server falling over, safe to retry because the idempotency key protects you. A 4xx that is not 429 is your bug, so stop retrying and surface it; hammering a malformed request four more times only delays the alert.

1. 1

   1. Attach an idempotency key

   A stable unique string per operation on every POST. Reuse the same key across retries of the same write so the server can recognize it.

2. 2

   2. Back off exponentially on failure

   1s, 2s, 4s, 8s, capped. Never a tight retry loop; a tight loop is a denial-of-service attack on the service you depend on.

3. 3

   3. Add jitter to every wait

   Randomize each interval so many clients do not retry in lockstep. AWS measured this cutting retry volume by more than half.

4. 4

   4. Prefer upsert on a stable ID

   Key on external_id so a duplicate write updates instead of inserts. This is idempotency you get for free from the CRM.

5. 5

   5. Give up to a dead-letter queue

   After N attempts, park the payload and alert a human. Retrying forever hides the failure and fills the log with noise.

Webhooks first, polling as fallback

Reach for webhooks before polling. The source tells you the instant something happens instead of you asking on a schedule and burning quota on empty answers. Polling is the fallback for when no webhook exists, for bulk reconciliation, or for event volumes high enough that push would flood you. The cost difference is not subtle. A five-minute poll is 288 calls a day whether or not anything changed. Stack ten such jobs and you are spending real quota to mostly hear “nothing new.”

View as table

Point	Value
60 min	24
30 min	48
15 min	96
5 min	288
1 min	1,440

A webhook receiver has one hard truth baked in: delivery is at-least-once and unordered. The same event will arrive twice, and event B can beat event A. Design for it or it will find you in production, usually as a provisioning job that ran twice or a status that flapped backward because a stale event landed last.

The HMAC verification is the step people get almost right and then break. You must compute the signature over the exact raw bytes you received, before any JSON parsing. Parse, re-serialize, and the bytes change (key order, whitespace, number formatting), so a valid signature fails and you either reject good events or, worse, disable verification to make it work. Here is the receiver in the shape that holds:

import hashlib, hmac, time

def verify_and_handle(raw_body: bytes, sig_header: str, ts_header: str, secret: bytes):
    # 1. Verify HMAC on the RAW bytes, not a re-serialized dict.
    expected = hmac.new(secret, raw_body, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(expected, sig_header):
        return 401                                   # forged or corrupted

    # 2. Reject stale timestamps to block replay of a captured payload.
    if abs(time.time() - int(ts_header)) > 300:      # 5-minute window
        return 401

    # 3. Parse only after trust is established.
    event = json.loads(raw_body)

    # 4. Dedupe on the event ID before doing any work (at-least-once delivery).
    if already_seen(event["id"]):
        return 200                                   # ack, do nothing
    mark_seen(event["id"])

    enqueue(event)                                   # 5. real work off the request path
    return 200                                       # 6. ack fast, always

Salesforce fires event

Verify HMAC on raw body

Return 200 fast

Dedupe on event ID

Provision async

Slack alert

Rate limits are a hard budget

Rate limits are not a suggestion you route around; they are the ceiling you design under. The numbers are knowable, so budget against them before you ship, not after the sync starts dropping calls.

Platform	Limit	Practical meaning
Salesforce daily API	100,000 + 1,000 per licensed user	A 50-license org gets 150,000/day, shared across every integration
HubSpot CRM Search	4 requests/second	Bursty search-driven syncs need a token bucket, not a loop
HubSpot other APIs	100-190 requests / 10 sec (tier-dependent)	Verify against your subscription tier before sizing
Clay HTTP column	~1-2 credits per call	Volume runs cost real money; gate behind an ICP filter first

Read the 429 and honor its Retry-After header instead of guessing an interval. Use the Bulk API above a few thousand records rather than looping single calls, because 5,000 single inserts is 5,000 calls against a budget the Bulk API spends in a handful. And loop pagination to exhaustion on the next_cursor. A sync that silently reads only page one is the most common data-completeness bug I find in GTM stacks, because it throws no error, it quietly misses rows.

# Exhaust the cursor. Stopping at page one is silent data loss.
def fetch_all(client, endpoint):
    rows, cursor = [], None
    while True:
        page = client.get(endpoint, params={"after": cursor, "limit": 100})
        rows.extend(page["results"])
        cursor = page.get("paging", {}).get("next", {}).get("after")
        if not cursor:            # no next cursor means you actually reached the end
            break
    return rows

Webhook versus polling, side by side

	Webhook (push)	Polling (pull)
Latency	Seconds; source pushes on the event	Up to your interval
Best for	Real-time events: form submit, closed-won, reply received	No webhook exists, bulk reconciliation, very high volume
Rate-limit cost	Near zero	Empty calls burn your daily budget
Failure mode	Dropped events if the receiver is down	Stale data between polls
Ordering	At-least-once, unordered; dedupe required	You control order and completeness

The two are not rivals; the durable pattern runs both. Webhooks carry the real-time path so a closed-won provisions in seconds. A nightly or hourly poll runs reconciliation to catch anything the webhook dropped while your receiver was down, because “dropped events if the receiver is down” is the one failure webhooks own. Push for latency, pull for completeness. This is the same reliability logic behind idempotent automations and the reconciliation thinking in the reconciliation layer: assume the fast path will occasionally lie, and keep a slow path that tells the whole truth.

When to stop wiring point to point

When you cross two to four connected tools, point-to-point integration turns into an N-squared mess of pairwise syncs, each with its own auth, retry logic, and failure mode. Five tools wired directly is up to ten connections to maintain. That is the moment to route through a warehouse or an iPaaS hub instead, so every tool talks to one place and you maintain N connections, not N-squared.

Hub option	Billing model	Cheapest when
Zapier	Per task	Low volume, fastest to stand up, priciest at scale
Make	Per credit (operation)	Mid-volume; cheaper than Zapier for multi-step flows
n8n	Per workflow execution	High volume; free self-host makes it cheapest at scale
Warehouse hub	Compute + reverse ETL	Past 2-4 tools, when data volume dwarfs event volume

The whole discipline reduces to three assumptions you bake in on day one: the network will drop a response, so make every write idempotent; the sender will deliver twice and out of order, so verify then dedupe then work; and the rate limit is a wall, so budget against it and exhaust your cursors. Wire those in and the integration stops being something you watch and becomes something you forget about, which is the only real measure that it works. The model underneath it matters as much, so pair this with CRM data modeling before you point a single webhook at production.