Logs and audit trail

This is where you answer three questions.

1. What did we send, and what came back?
2. Has that webhook reached us? Which attempt failed, and why?
3. Who on the team pressed what in the back office?

1. Request / response history

GET /v1/bo/logs

Every response we return carries a request_id, on success and on error alike.

{ "error": {
    "code": "signature_invalid",
    "type": "authentication_error",
    "message": "Signature does not match the canonical string.",
    "message_th": "ลายเซ็นไม่ตรงกับ canonical string",
    "retryable": false,
    "retry_after_seconds": null,
    "request_id": "req_01JQA5T8V0X2Z4B6D8F0H2J4L6",
    "details": {} } }

Store request_id on every call, not only on errors

When you send us a case, a single request_id saves a day of work. A ticket that says "we could not create an order yesterday afternoon", with no request_id and no deposit_id, is close to untraceable.

What must never appear in your logs

• qr_payload — it contains the withdrawer's bank details. The terms of use state it must never be logged, never stored in a database, never forwarded. Render it and discard it.
• Full destination account numbers.
• sk_live_* and the value of PG-Signature.

qr_payload lives exactly as long as expires_at, which on some routes is as short as ~4 minutes ⇒ logging it buys you nothing and carries all of the risk.

2. Delivery status of every webhook

curl -sS "https://api.example-wl.com/v1/bo/webhooks/whep_01K2A9CDEF3G/deliveries?limit=50&status=failed" \
  -H "Authorization: Bearer $BO_OIDC_ACCESS_TOKEN"

{
  "data": [
    {
      "id": "whd_01K3M8QX4C7RZ2XT5NBVKD9H0P",
      "object": "webhook_delivery",
      "endpoint_id": "whep_01K2A9CDEF3G",
      "event_id": "evt_01K3M8QW7Z4N2VYB6H0RSTA9CD",
      "event_type": "deposit.settled",
      "status": "failed",
      "attempt": 11,
      "resend": false,
      "created_at": "2026-08-21T04:12:33.940Z",
      "last_attempt_at": "2026-08-25T00:49:07.552Z",
      "next_retry_at": null,
      "response_status": 504,
      "response_body_excerpt": "upstream request timeout",
      "duration_ms": 15002,
      "error": "response_timeout"
    },
    {
      "id": "whd_01K3M9F3AB8SY4WT6PCXLE1J2Q",
      "object": "webhook_delivery",
      "endpoint_id": "whep_01K2A9CDEF3G",
      "event_id": "evt_01K3M9F2XN8VQKR4T0BDHY7WZE",
      "event_type": "deposit.under_review",
      "status": "delivered",
      "attempt": 2,
      "resend": false,
      "created_at": "2026-08-21T04:19:08.810Z",
      "last_attempt_at": "2026-08-21T04:19:19.043Z",
      "next_retry_at": null,
      "response_status": 200,
      "response_body_excerpt": "ok",
      "duration_ms": 214,
      "error": null
    }
  ],
  "has_more": false,
  "next_cursor": null
}

A delivery has four statuses, and no others

status	Meaning
pending	still in the retry cycle · the next attempt is at next_retry_at
delivered	we got a 2xx
failed	11 attempts, still no 2xx · the event is kept 30 days and can be resent
canceled	the endpoint was deleted or disabled while the delivery was still queued

What counts as success

Criterion	Value
Success	200 201 202 204, and nothing else
Connection timeout	5 seconds
Time to the first response byte	10 seconds
Total request timeout	15 seconds (aim to answer within 3 seconds)
Response body we keep	up to 8 KB · the content has no effect on success
Size of the body we send	at most 256 KB (guaranteed)
3xx	🔴 we do not follow redirects = failure
410 Gone	🔴 permanent failure, retries stop immediately and the endpoint is flagged as gone
429	failure → retried on the schedule (we do not read your Retry-After; a fixed schedule keeps it predictable)
Any other 4xx/5xx · TLS/DNS/connection error	failure → retried on the schedule

Answer 2xx first, then do the work

Heavy work inside the handler ⇒ you answer later than 15 seconds ⇒ we record a failure ⇒ retry ⇒ you process it twice ⇒ double credit. The correct shape: verify the signature → write to a queue → answer 200 → do the real work asynchronously.

The retry schedule

attempt	Delay from the previous one	Cumulative from event creation
1	immediately	00:00:00
2	10 seconds	00:00:10
3	1 minute	00:01:10
4	5 minutes	00:06:10
5	30 minutes	00:36:10
6	2 hours	02:36:10
7	6 hours	08:36:10
8	12 hours	20:36:10
9	24 hours	44:36:10
10	24 hours	68:36:10
11	24 hours	92:36:10

11 attempts across a window of roughly 92 hours 36 minutes (3 days 20 hours), with ±10% jitter per round ⇒ the real time is in next_retry_at.

The retry window is deliberately shorter than the Idempotency-Key lifetime

The last retry (~92 hours) must still be able to collide with an idempotency record on your side, which lives 7 days. ⇒ 🔴 Your idempotency TTL must not be shorter than 7 days, or duplicates will appear out of the late retries.

When retries stop

• We got a 2xx
• 11 attempts are used up → failed (the event is kept 30 days and can be resent)
• We got a 410
• You deleted or disabled the endpoint (queued deliveries become canceled)
• The system disabled the endpoint automatically

3. Resending

Resending one delivery

curl -sS -X POST "https://api.example-wl.com/v1/bo/webhooks/whep_01K2A9CDEF3G/deliveries/whd_01K3M8QX4C7RZ2XT5NBVKD9H0P/resend" \
  -H "Authorization: Bearer $BO_OIDC_ACCESS_TOKEN" \
  -H "X-PG-Step-Up-Token: $BO_STEP_UP_TOKEN" \
  -H "Idempotency-Key: resend-whd01K3M8QX4C7RZ2XT5NBVKD9H0P-01" \
  -H "Content-Type: application/json"

{
  "delivery_id": "whd_01K3V2R8T5WYZ7XB1KDMNQH4PS",
  "event_id": "evt_01K3M8QW7Z4N2VYB6H0RSTA9CD",
  "status": "pending",
  "attempt": 1,
  "resend": true,
  "created_at": "2026-08-23T05:00:00.412Z"
}

Thing	What happens
event.id	🔴 exactly the same ⇒ your deduplication still works, so resending is always safe
The bytes of the body	byte for byte identical — the payload was stored at creation and is not re-rendered ⇒ no API-version drift mid-flight
The signature	recomputed (because t must be fresh), signed with the currently active key
Delivery metadata	Headers only: a new X-PG-Delivery-Id · X-PG-Attempt: 1 · X-PG-Resend: manual
The retry cycle	a fresh set of 11 attempts
Age limit	resendable within 30 days of the event's created_at

Does resending double-credit? Not if you have both of these

1. A transactional inbox with UNIQUE(event_id), retained at least 31 days · on a duplicate, answer 200, never an error.
2. A version guard written as UPDATE ... WHERE version < $new, then check the affected row count · zero rows = an older event ⇒ answer 200 and do nothing.

The system guarantees that no two events for the same object share a version ⇒ this pattern is genuinely safe.

Bulk resend

curl -sS -X POST "https://api.example-wl.com/v1/bo/webhooks/whep_01K2A9CDEF3G/replay" \
  -H "Authorization: Bearer $BO_OIDC_ACCESS_TOKEN" \
  -H "X-PG-Step-Up-Token: $BO_STEP_UP_TOKEN" \
  -H "Idempotency-Key: replay-whep01K2A9CDEF3G-20260823-01" \
  -H "Content-Type: application/json" \
  -d '{
    "event_types": ["deposit.settled", "deposit.reversed"],
    "created_after": "2026-08-18T00:00:00.000Z",
    "created_before": "2026-08-21T00:00:00.000Z",
    "only_failed": true
  }'

Every /v1/bo/* route uses the operator's OIDC access token, not sk_live_* or PG HMAC. Reading deliveries needs the access token only; resend/replay are privileged actions and also require an unexpired step-up token.

• Up to 10,000 events per command, released at 50 deliveries/second.
• Every command is written to the audit trail.
• 🔴 A command like this does not count against the webhook_resend quota (that quota counts "commands" at 10/minute per endpoint).

Before you press replay, prove your handler really is idempotent

Replaying 10,000 events into a handler with no UNIQUE(event_id) means crediting 10,000 times. Test by resending one delivery and confirming your balance does not move, then run the batch.

Prolonged endpoint outages

6 hours down in a row     → warning email + a banner (retries continue as normal)
12 hours down in a row    → second warning email + the provider above is notified
24 hours down in a row    → 🔴 the system disables the endpoint automatically
    Condition: 0 successes in the last 24 hours **and** at least 50 failed deliveries
    Effect: no new deliveries are created (events are still kept for 30 days)
        · queued deliveries become canceled
        · webhook.endpoint.disabled fires to your other endpoints + email + an audit record

• Re-enabling is manual, from the back office — it offers a button to resend the deliveries that failed in the last 72 hours.
• An endpoint with at least one success in 24 hours will not be disabled ⇒ an endpoint failing 90% of the time survives, but gets warned.
• 🔴 If you have only one endpoint and it gets disabled, the events that cannot be unsubscribed are emailed to your technical contact.

This is why the technical contact must be a mailbox somebody reads

Not noreply@, and not an inbox nobody opens. The day we email you that your endpoint was disabled is the day you are about to stop hearing about reversals against your own wallet.

Headers on every delivery

POST /webhooks/pg HTTP/1.1
Content-Type: application/json; charset=utf-8
User-Agent: PG-Webhooks/1.0
X-PG-Signature: v1,kid=whk_2026a,t=1755749553,s=3rPYqVQ
X-PG-Event-Id: evt_01K3M8QW7Z4N2VYB6H0RSTA9CD
X-PG-Event-Type: deposit.settled
X-PG-Delivery-Id: whd_01K3M8QX4C7RZ2XT5NBVKD9H0P
X-PG-Attempt: 3
X-PG-Api-Version: 2026-08-01
X-PG-Livemode: true

Delivery headers are for observability only

Verify the signature over the raw bytes before you touch the body. The business event is in the body; delivery headers are not part of the canonical signature and must never drive money, authorization, or state decisions. X-PG-Signature must be parsed as key=value pairs. Never parse it by position, because we may add fields later.

Log these on every delivery you receive: event.id · X-PG-Delivery-Id · X-PG-Attempt · the result of the version guard (how many rows changed) · the time spent inside the handler.

Chasing a missing event

You found a sequence gap using the reconciliation script. Work through it in this order.

1. A gap from the last 4 days is normal — the event is still in the retry cycle.
2. Fetch the event directly with GET /v1/pay/events?since_sequence= — if it comes back, nothing is lost, it just has not been delivered.
3. Check GET /v1/bo/webhooks/{endpoint_id}/deliveries for that delivery's status and the response_status it failed on.
4. Fix the root cause first (endpoint answering 3xx · expired TLS · handler timeout) and only then resend.
5. A gap older than 4 days that /v1/pay/events cannot return is a real loss ⇒ raise it with us, quoting the sequence range.

4. Who pressed what in the back office

GET /v1/bo/reports/action-logs

Every action that affects money or configuration is recorded — approving and rejecting withdrawals · extending timeouts · opening claims · resending webhooks · changing settings.

What our audit trail does not give you

The internal notes staff write when deciding a review-queue case, the names of those staff, and internal binding deadlines are never emitted through any channel. What you see about the review queue is three things: review.escalated · review.four_eyes_required · review.estimated_resolution_bucket.

Logging checklist

• [ ] Log event.id + X-PG-Delivery-Id + X-PG-Attempt on every webhook received
• [ ] Transactional inbox has UNIQUE(event_id), retained at least 31 days · a duplicate answers 200
• [ ] Store request_id for every outbound request, not only errors
• [ ] Never log qr_payload, never log full destination account numbers
• [ ] Alert when non-2xx handler responses exceed 1% over 5 minutes
• [ ] Alert on a single signature verification failure
• [ ] Test a single resend and confirm your balance does not move, before using bulk replay
• [ ] The technical contact is a mailbox somebody reads

Next

• Retries and endpoint outages — the full webhook-side detail
• Ordering is not guaranteed — the version guard — why resending is safe
• Verifying Ed25519 signatures — verify over raw bytes
• Reports and reconciliation — where sequence gaps come from
• Settings — endpoint requirements and the events you cannot unsubscribe from