Sign in Subscribe

What is Core Webhook Module?

Introduction

Core Webhook Module is a production-ready, enterprise-grade webhook receiver and processor built with FastAPI. It serves as a secure gateway for receiving webhooks from external services and routing them to your internal systems with comprehensive security, validation, and routing capabilities.

What is a Webhook?

Before diving into the tool itself, let's understand what webhooks are. A webhook is an HTTP callback mechanism that allows external services to notify your application when specific events occur. Instead of your application continuously polling external APIs for updates, webhooks push data to your application in real-time.

Common examples include:

  • Payment processors (Stripe, PayPal) notifying you of payment events
  • Version control systems (GitHub, GitLab) sending repository event notifications
  • IoT devices sending telemetry data
  • SaaS platforms triggering workflows based on user actions

The Problem Core Webhook Module Solves

Building a robust webhook receiver from scratch involves many challenges:

  1. Security: Validating webhook authenticity, preventing unauthorized access, and protecting against common attacks
  2. Routing: Sending webhook data to multiple destinations (databases, message queues, cloud storage)
  3. Reliability: Handling failures, retries, and ensuring data isn't lost
  4. Scalability: Processing high volumes of webhooks efficiently
  5. Observability: Tracking webhook usage, monitoring performance, and debugging issues
  6. Configuration: Managing multiple webhooks with different authentication and routing requirements

Core Webhook Module solves all of these challenges out of the box.

What Does Core Webhook Module Do?

Core Webhook Module acts as a centralized webhook processing service that:

1. Receives Webhooks Securely

  • Provides RESTful API endpoints (POST /webhook/{webhook_id})
  • Supports multiple authentication methods (11 total!)
  • Validates webhook signatures and authenticity
  • Protects against common security vulnerabilities

2. Validates and Processes Webhooks

  • Validates payload structure and content
  • Enforces JSON schema validation
  • Checks payload size, depth, and format
  • Applies rate limiting per webhook
  • Sanitizes sensitive data before logging

3. Routes to Multiple Destinations

  • Message Queues: RabbitMQ, Redis RQ, Apache Kafka, AWS SQS, GCP Pub/Sub, ActiveMQ, ZeroMQ
  • Databases: PostgreSQL, MySQL/MariaDB, ClickHouse
  • Cloud Storage: AWS S3
  • Real-time Protocols: WebSocket, MQTT
  • Local Storage: File system, stdout logging
  • HTTP Forwarding: Forward to other HTTP endpoints

4. Provides Enterprise Features

  • Live Configuration Reload: Update webhook configs without restarting
  • Statistics & Analytics: Track webhook usage with Redis and ClickHouse
  • Retry Mechanism: Automatic retries with exponential backoff
  • Task Management: Concurrent processing with resource limits
  • Dynamic OpenAPI Docs: Auto-generated API documentation

Key Features

🔐 Enterprise Security (11 Authentication Methods)

  1. Authorization Header - Bearer token authentication
  2. Basic Authentication - HTTP Basic Auth (RFC 7617)
  3. JWT Authentication - Full JWT token validation
  4. HMAC Signature Validation - GitHub/Stripe-compatible signatures
  5. IP Whitelisting - Restrict access by IP address
  6. Google reCAPTCHA - Bot prevention (v2 and v3)
  7. HTTP Digest Authentication - Challenge-response authentication
  8. OAuth 1.0 - Legacy API support (Twitter-style)
  9. OAuth 2.0 - Token introspection and JWT validation
  10. Query Parameter Authentication - API keys in query strings
  11. Header-Based Authentication - Custom header API keys

🚀 17 Integration Modules

Route webhooks to:

  • Message Queues: RabbitMQ, Redis RQ, Kafka, AWS SQS, GCP Pub/Sub, ActiveMQ, ZeroMQ
  • Databases: PostgreSQL, MySQL/MariaDB, ClickHouse
  • Cloud Storage: AWS S3
  • Real-time: WebSocket, MQTT
  • Local: File system, stdout
  • HTTP: Forward to other endpoints

📊 Production-Ready Features

  • Live Configuration Reload: Update configs without restart
  • Statistics & Analytics: Redis-based stats + ClickHouse logging
  • Retry Mechanism: Automatic retries with exponential backoff
  • Rate Limiting: Per-webhook rate limits with sliding window
  • Task Management: Concurrent processing with resource limits
  • Dynamic OpenAPI Docs: Auto-generated from configuration
  • Credential Cleanup: Automatic masking of sensitive data
  • Webhook Chaining: Send to multiple destinations sequentially or in parallel

Architecture Overview

┌─────────────────────────────────────────┐
│      External Services (Webhooks)       │
│  (GitHub, Stripe, Payment Processors)   │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│      Core Webhook Module                │
│  ┌──────────────────────────────────┐  │
│  │  Security & Validation Layer     │  │
│  │  • 11 Authentication Methods     │  │
│  │  • Rate Limiting                │  │
│  │  • Input Validation              │  │
│  └──────────────────────────────────┘  │
│  ┌──────────────────────────────────┐  │
│  │  Processing Layer                │  │
│  │  • Task Management               │  │
│  │  • Retry Logic                   │  │
│  │  • Credential Cleanup            │  │
│  └──────────────────────────────────┘  │
│  ┌──────────────────────────────────┐  │
│  │  Routing Layer (17 Modules)      │  │
│  │  • Message Queues                │  │
│  │  • Databases                     │  │
│  │  • Cloud Storage                 │  │
│  │  • Real-time Protocols           │  │
│  └──────────────────────────────────┘  │
└──────────────┬──────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────┐
│      Internal Systems                   │
│  (Databases, Queues, Storage, APIs)     │
└─────────────────────────────────────────┘

Use Cases

1. Payment Processing

Receive payment notifications from Stripe, PayPal, or other payment processors. Validate HMAC signatures, route to your payment processing system, and store in a database for audit trails.

2. Version Control Integration

Receive repository events from GitHub or GitLab. Validate webhook signatures, trigger CI/CD pipelines, and update issue trackers.

3. IoT Device Integration

Receive device telemetry via MQTT. Validate device authentication, store in time-series databases, and trigger alerts.

4. Third-Party Service Integration

Receive webhooks from SaaS services. Validate OAuth 2.0 tokens, route to internal systems, and transform data as needed.

5. Event-Driven Architecture

Receive events from multiple sources, validate and route to message queues, enabling event-driven microservices.

6. Analytics & Monitoring

Collect webhook events, store in ClickHouse for analytics, generate real-time statistics, and monitor usage patterns.

Why Use Core Webhook Module?

✅ Production-Ready

  • 2,700+ tests covering all functionality
  • 60 security audits completed
  • 90%+ code coverage
  • Battle-tested in production environments

✅ Easy Configuration

  • Simple JSON-based configuration
  • Environment variable support
  • Live configuration reload
  • No code changes needed for new webhooks

✅ Comprehensive Security

  • 11 authentication methods
  • Multi-layer validation
  • Security headers and CORS protection
  • Credential cleanup and sanitization
  • Protection against OWASP Top 10 vulnerabilities

✅ Flexible & Extensible

  • Plugin-based architecture
  • Easy to add new modules
  • Support for 17 different destinations
  • Webhook chaining (multiple destinations)

✅ Developer Friendly

  • Dynamic OpenAPI documentation
  • Comprehensive documentation
  • Type-safe codebase
  • Easy to test and extend

Quick Example

Here's a simple example of configuring a webhook:

webhooks.json:

{
  "stripe_payments": {
    "data_type": "json",
    "module": "postgresql",
    "connection": "postgres_prod",
    "module-config": {
      "table": "payment_events"
    },
    "authorization": "Bearer {$STRIPE_WEBHOOK_SECRET}",
    "hmac": {
      "secret": "{$STRIPE_HMAC_SECRET}",
      "header": "X-Stripe-Signature",
      "algorithm": "sha256"
    },
    "rate_limit": {
      "max_requests": 100,
      "window_seconds": 60
    }
  }
}

connections.json:

{
  "postgres_prod": {
    "type": "postgresql",
    "host": "{$POSTGRES_HOST}",
    "port": 5432,
    "database": "webhook_db",
    "user": "{$POSTGRES_USER}",
    "password": "{$POSTGRES_PASSWORD}"
  }
}

That's it! The webhook is now configured to:

  • Accept POST requests at /webhook/stripe_payments
  • Validate Bearer token authentication
  • Validate HMAC signature from Stripe
  • Apply rate limiting (100 requests per minute)
  • Store payloads in PostgreSQL database
  • All without writing any code!

Getting Started

  1. Install dependencies:

    pip install -r requirements.txt

  2. Configure webhooks in webhooks.json

  3. Configure connections in connections.json

  4. Run the server:

    uvicorn src.main:app --reload

  5. Access API docs:

    • Swagger UI: http://localhost:8000/docs
    • ReDoc: http://localhost:8000/redoc

Conclusion

Core Webhook Module is a comprehensive solution for webhook processing that handles security, routing, reliability, and observability out of the box. Whether you're building a payment processing system, integrating with third-party services, or creating an event-driven architecture, Core Webhook Module provides the foundation you need.

With 11 authentication methods, 17 integration modules, comprehensive security features, and production-ready reliability, it's the perfect tool for mission-critical webhook processing in modern distributed systems.

Next Steps:

  • Check out the README for detailed setup instructions
  • Explore the Architecture Documentation to understand the internals
  • Review the Product Overview for comprehensive feature documentation

This is the first in a series of blog posts about Core Webhook Module. Stay tuned for more articles on specific features, use cases, and best practices!

Subscribe to Free Webhook Tool

Sign up now to get access to the library of members-only issues.
Jamie Larson
Subscribe