Research into s3 read object

AWS list object with pagination

AWS CLI examples

AWS selectObjectContent with nodejs

GitHub example code for selectObjectContent

read json with nodejs

 SelectParameters: {
      Expression: 'STRING_VALUE', /* required */
      ExpressionType: SQL, /* required */
      InputSerialization: { /* required */
        CSV: {
          AllowQuotedRecordDelimiter: true || false,
          Comments: 'STRING_VALUE',
          FieldDelimiter: 'STRING_VALUE',
          FileHeaderInfo: USE | IGNORE | NONE,
          QuoteCharacter: 'STRING_VALUE',
          QuoteEscapeCharacter: 'STRING_VALUE',
          RecordDelimiter: 'STRING_VALUE'
        },
        CompressionType: NONE | GZIP | BZIP2,
        JSON: {
          Type: DOCUMENT | LINES
        },
        Parquet: {
        }
      },
      OutputSerialization: { /* required */
        CSV: {
          FieldDelimiter: 'STRING_VALUE',
          QuoteCharacter: 'STRING_VALUE',
          QuoteEscapeCharacter: 'STRING_VALUE',
          QuoteFields: ALWAYS | ASNEEDED,
          RecordDelimiter: 'STRING_VALUE'
        },
        JSON: {
          RecordDelimiter: 'STRING_VALUE'
        }
      }
    },

Pagination with AWS-SDK V2

GetObject read stream:

var readableStream = await s3.getObject({ Bucket: bucket, Key: filename }).createReadStream();

selectObjectContent response

Limitations of s3 selectObjectContent from blog with performance comparison with getObjects

Limitations posted by AWS

Read large object in one line at a time – S3.getObject(s3Config).createReadStream();