Skip to main content

S3 Module - Webhooks to Amazon S3

The S3 Module archives every incoming webhook payload as an object in an Amazon S3 bucket. Use it to build a durable, queryable archive of webhook events: payment notifications from Stripe, push events from GitHub, IoT telemetry, or any other webhook source you need an audit trail for.

How It Works​

  1. A webhook arrives at POST /webhook/{webhook_id} and passes authentication (HMAC, Bearer, IP whitelist, or any of the other supported methods).
  2. The payload is serialized (JSON payloads are pretty-printed) and uploaded with put_object.
  3. The object key is date-partitioned automatically: {prefix}/YYYY/MM/DD/HH/{filename}, so downstream tools like Amazon Athena, AWS Glue, or S3 lifecycle rules can work with the archive efficiently.

Example object key:

webhooks/archive/2026/08/28/14/webhook_2026-08-28T14-05-33.123456_1c9f9c7e-8f4b-4a6e-9d2f-3f8a1b2c3d4e.json

Quick Start​

1. Webhook Configuration (webhooks.json)​

{
    "stripe_archive": {
        "data_type": "json",
        "module": "s3",
        "connection": "s3_storage",
        "module-config": {
            "bucket": "my-webhook-archive",
            "prefix": "webhooks/stripe",
            "filename_pattern": "event_{timestamp}_{uuid}.json",
            "include_headers": true
        },
        "hmac": {
            "secret": "{$STRIPE_WEBHOOK_SECRET}",
            "header": "Stripe-Signature",
            "algorithm": "sha256"
        }
    }
}

2. Connection Configuration (connections.json)​

{
    "s3_storage": {
        "type": "s3",
        "aws_access_key_id": "{$AWS_ACCESS_KEY_ID}",
        "aws_secret_access_key": "{$AWS_SECRET_ACCESS_KEY}",
        "region": "us-east-1"
    }
}

If aws_access_key_id and aws_secret_access_key are omitted, the module falls back to the standard AWS credential chain: IAM roles for EC2/ECS/EKS, environment variables, or ~/.aws/credentials. On AWS infrastructure, prefer an IAM role and leave the keys out entirely.

3. IAM Policy​

The credentials only need s3:PutObject on the target bucket:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::my-webhook-archive/webhooks/*"
        }
    ]
}

4. Test​

curl -X POST http://localhost:8000/webhook/stripe_archive \
  -H "Content-Type: application/json" \
  -H "Stripe-Signature: <valid signature>" \
  -d '{"type": "payment_intent.succeeded", "id": "evt_123"}'

Then verify the object landed:

aws s3 ls s3://my-webhook-archive/webhooks/stripe/ --recursive | tail -1

Module Configuration Options​

OptionRequiredDefaultDescription
bucketyes-Target S3 bucket name
prefixnowebhooksKey prefix; validated against path traversal
filename_patternnowebhook_{uuid}.jsonObject filename; supports {timestamp} and {uuid} placeholders
include_headersnofalseStore incoming HTTP headers as S3 object metadata
content_typenoapplication/jsonContent-Type of the stored object

Notes:

  • {timestamp} expands to a filesystem-safe ISO timestamp; {uuid} to a random UUID4, so concurrent webhooks never collide.
  • With include_headers: true, headers are stored as S3 object metadata (lowercased, - replaced by _, values truncated to the S3 limit of 1024 bytes), not inside the object body. The body always contains only the payload.
  • Prefix, filename pattern, and the final object key are validated against path traversal (..), control characters, and invalid characters before upload.

Connection Configuration Options​

OptionRequiredDefaultDescription
typeyes-Must be "s3"
aws_access_key_idnocredential chainAccess key; supports {$ENV_VAR} substitution
aws_secret_access_keynocredential chainSecret key; supports {$ENV_VAR} substitution
regionnous-east-1AWS region of the bucket

Secrets can also be pulled from HashiCorp Vault with the {$vault:...} syntax.

Common Use Cases​

  • Payment webhook audit trail: archive every Stripe or PayPal event with its verified signature headers for compliance and dispute resolution.
  • Event replay: keep raw payloads so failed downstream processing can be replayed later.
  • Analytics on webhook data: the YYYY/MM/DD/HH partitioning maps directly to Athena partition projection.
  • Fan-out with chaining: combine with webhook chaining to archive to S3 and forward to Kafka or RabbitMQ in one webhook.

Troubleshooting​

SymptomCause and fix
S3 bucket not specified in module-configAdd bucket under module-config
AccessDenied in server logsThe credentials lack s3:PutObject on the bucket/prefix; check the IAM policy and bucket policy
Generated filename contains invalid charactersfilename_pattern produced characters outside a-zA-Z0-9_-.; only use the supported placeholders
Objects missing header metadataSet include_headers: true; note headers land in object metadata, not the body
Wrong region errorsSet region in the connection to the bucket's region

Client-facing error responses are sanitized; full S3 error codes appear only in server logs.