Idempotency of a PUT Request

07/02/2020 posted in  Storage System

1. How To Avoid Duplicate Order

E-commerce Scenario - Common Flow of Placing Order

  • Go to Cart
  • Cart Page + Checkout button
  • Checkout Page + Place order button
  • Payment:
    US E-commerce has no such step - Credit card or PayPal are filled in Checkout Page.
  • Success Page

Cause:

  • User clicks PlaceOrder multiple times
  • Backend retries the call to Order service

Solution 1: One-time token

  • Checkout page requests one-time token
  • Store token in Redis as Key.
  • Place order request comes with the token
  • Order system verifies token by
    • DEL token
    • Return 0 means the key is already removed.

Assessment:

  • Scalability: High.
    • Redis Sharding.
  • Availability: High.
    • Service is down after DEL token: retries will fail. Place order will fail.
    • Redis node is down. Token not available, place order will fail.

Solution 2: Leverage DB Primary Key uniqueness

  • Checkout page requests orderId
  • Place order request comes with the orderId
  • DB creates order using orderId as primary key.
  • Insert failure means PK already exists, this is a duplicate request.

Assessment:

  • Scalability: High
  • Availability: Depends on proxy. If using consistent hashing which can remove the failure node, then it's high.
  • Security: Attacker can use invalid key, we have to verify it.

2. In general how to guarantee Idempotence of an API put request.

We can have a primary key in an insert directive. The primary key is unique id of the data, so retried writes won't succeed.

Take SQL for example -

INSERT INTO users (uid, age, gender, createTime)
VALUES (1234567, 20, "male", 1593684708)

Can we assume all INSERTS are idempotent when there is a Primary Key in request?

Yes when the Primary Key can come from a ID generation service, or even client side auto generation.

DDB and MongoDB generate UUID or ObjectId at client side:

This means we should try to avoid using the AUTO_INCREMENT feature on the DB server side

Conclusion:
We can guarantee idempotency to DB when the write has a primary key generated on client side - from an ID Generation Service or client side random UUID.

However the actual idempotency is determined by upstream services. Will they retry the calls to the DAO service?

The ideal way is the top level service provide an one time token. The DAO layer checks the token using Solution1 of Section1.