Note: DLQ for failed lambda invokation stores stream information, not the record itself
You need to call get-shard-iterator, and then call getRecords to get the actual impacted records.
Two Reasons why this is not given by AWS out-of-the box:
Reason1:
SQS message size has a limit, your DDB record update size may exceed the message limit. Hence by delivering the shard information which is of fixed size, AWS internally didn’t had to worry about actual message size limitations.
Reason2: – needs reconciliator to decide whether updates should be applied
Above is the approach to get the impacted records. You might ask Why can’t AWS DDB or lambda deliver the impacted records to DLQ?, to answer this, we need to understand what DDBTrigger and shard is ?
Since, DDB trigger is providing you the realtime updates to table data using kinesis behind the scenes(shard mechanism), and if particular read from shard failed, then it’s delivered to dlq and continue with next data in the shard.
So within a shard, let’s say your record Id=ABC was updated multiple times(v0,v1,v2,v3 be the versions), and if some udpate(v1) from DDB failed processing in your lambda, they are sidelined to DLQ. It’s possible that next updated to same record Id=ABC, were succesffully processes later on by the lambda. Let’s assume your lambda is writing to S3 (eventually overriding older versions). In this scenarios,
- v0 was written to S3, as lambda was able to process succesffuly
- v1’s updates got delivered to dlq since lambda failed
- v2 was written to S3, as lambda was able to process succesffuly
- v3 was written to S3, as lambda was able to process succesffuly
Now, you need a processor over the dlq (shard reader + getRecords) to read and identify whether you should write to S3 or not. In this case you should ignore and delete the dlq message, since v3 has been written by lambda later on.