Analytics Firewall: Architecture and Algorithm

1. Product Goal and Solved Problems

Goal: To provide safe, controlled, and stable data collection (Server-Side Tracking). It works perfectly even with strict browser limits (like ITP and AdBlock) and privacy laws (GDPR, CCPA).

Business problems it solves:

  • Stopping PII (Personally Identifiable Information) Leaks: It removes or hashes personal data (like emails, phone numbers, and credit card numbers) before sending it to third-party servers. This protects the business from legal risks and huge fines.
  • Bypassing Tracker Blocks: It moves data sending from the weak client browser to a secure server. This fixes conversion tracking.
  • Data Quality and Standardization: It checks if the incoming data matches strict rules. It stops bad, duplicate, and wrong events before they get into the database.
  • Data Enrichment: It adds server business logic to events (like real product profit, fraud score, or customer LTV). It is not safe to show or calculate this data on the frontend website.

2. Platform and Deployment (using Google Cloud Run)

To host the Analytics Firewall, you need a strong infrastructure. It must handle sudden high traffic (for example, during Black Friday) without paying for idle time. The best choice is Serverless.

  • Runtime Environment (Google Cloud Run): The app (written in Python/FastAPI, Go, or Node.js) is packed into a Docker container. Cloud Run scales from 0 to thousands of instances in seconds. You pay only for the exact time of request processing (down to milliseconds).
  • Network Layer and DNS: A custom first-level domain (like metrics.company.com) is connected to Cloud Run. The traffic looks like first-party requests. Because of this, browser algorithms do not see it as a third-party tracker.
  • Buffering (Google Cloud Pub/Sub): When Cloud Run receives a request, it quickly sends it to a message broker (Pub/Sub). This allows the server to return a 200 OK status to the user’s browser in milliseconds (so the website does not slow down). The heavy checking and cleaning work is done later in the background.
  • Log Storage (Google BigQuery): All events that fail validation are not deleted. They are sent to a special table (Dead Letter Queue) for engineers to check.

3. Workflow Algorithm (Pipeline)

  1. Ingestion and Buffering: The endpoint receives a POST request with JSON data from the client. It takes the headers and IP address. The server immediately returns a successful HTTP response to the client, and the event goes to a queue for processing.
  2. Consent Enforcement: The system checks user consent flags. If the user said “no” (denied for analytics_storage or ad_storage), the algorithm removes ad IDs (like GCLID, FBCLID) and hides the IP address. It only leaves the anonymous event fact.
  3. Schema Validation: It compares the JSON structure with the correct data dictionary. If required keys are missing (for example, no value for a purchase event) or the data type is wrong, the event is blocked and sent to the Dead Letter Queue.
  4. PII Detection and Redaction: It scans key values and URL parameters for sensitive data. It hashes the found PII (using the SHA-256 algorithm with a secret salt). This allows vendors to connect sessions without getting real contact information.
  5. Data Enrichment: It uses a secure user ID to ask internal databases for hidden business information (like customer segment or loyalty status). It adds this data directly into the JSON body.
  6. Routing and Dispatch: It changes the clean and enriched JSON to match the rules of specific API vendors (like Meta Conversions API or Google Analytics Measurement Protocol). Then it sends the requests. If the receiving API is down, it uses a retry algorithm with a delay (Exponential Backoff).

Similar Posts