Aws Series Decoupling Workflows

Aws Series Decoupling Workflows

Post Date : 2023-12-30T23:48:50+07:00

Modified Date : 2023-12-30T23:48:50+07:00

Category: systemdesign aws

Tags: aws

The issue with tight coupling


The solution : loose coupling



Simple Queue Service(SQS)


  • SQS is fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
  • A message queue that allows asynchronous processing of work. One resource write a message to an SQS queue, and then another resource retrieve that messages from SQS

Some important settings

  • Delivery delay: default is 0, max value is 15 minutes
  • Message size: up to 256KB of text in any format
  • Encryption: messages are encrypted in transit by default, but you can add at-rest
  • Message retention: default is 4 days, can be set between 1 minutes and 14 days.
  • Long vs Short: Long polling isn’t the default, but it should be.
  • Queue Depth: this can be a trigger for autoscaling -> if two many messages in queue, add more instances to solve it.



Visibility Timeout


Simple Notification Service(SNS)

  • SNS is a fully managed messaging service for both application-to-application (A2A) and application-to-person(A2P) communication

API Gateway

  • API Gateway is a fully managed service that makes easy for developers to create,publish,maintain, monitor, and secure APIs at any scale.

Sidelining Message Queue with Dead-Letter Queues

  • DLQ are the targets for messages that can not be processed successfully
  • Works with SQS and SNS!
  • Useful for debugging applications and messaging systems
  • Ability to isolate unconsumed messages to troubleshoot
  • Redrive capacity allows you to move the message back into the source queue
  • These are technically just other SQS queues
  • DLQs used FIFO SQS queues must ALSO be FIFO queues


  • Configure alarms based on message availability counts
  • Quickly identify which logs to investigate for exceptions
  • Analyze the SQS message contents for any errors
  • Troubleshoot consumer permissions


Order Messages using SQS FIFO

  • Guaranteed ordering
  • No message duplication
  • 300 transactions per second <–> Batching can achieve up to 3,000 messages per second, per API call


FIFO High throughput

  • Process up to 9,000 transactions per second, per API without batching
  • Up to 90,000 transactions per second by using batching APIs


Delivering Messages with SNS


SNS: Simple Notification Service

  • Push-based messaging service, proactively delivers messages to the endpoints that are subscribed to it.
  • This can be used to alert a system or a person
  • One message can be sent to many

SNS Settings and Quotas


  • Kinesis Data Firehose, SQS, Lambda, email, HTTP(s), SMS and platform application endpoint.

Message Size

  • Up to 256KB of text in any format

DLQ support

  • Messages that failed to delivered can be stored in SQS DLQ

FIFO or Standard

  • FIFO only supports SQS FIFO queues as subcribers


  • Messages are encrypted in transit by default, and you can add at-rest via AWS KMS

Access Policies

  • A resource policy can be added to a topic, similar to S3. Useful for cross-account access.

Large Message Payloads

  • The SNS Extended Library allows sending messages up to 2GB in size
  • The payload is stored in Amazon S3, then SNS published a reference to the object

SNS Fanout

  • Messages published in SNS topics are replicated in multiple endpoint subcriptions
  • Allow for fully decoupled parallel asynchronous processing

SNS Architecture

image image image

Message Filtering

  • By default, every message published to a topic is sent to all subcribers
  • Filter policies use JSON to define which messages get sent to specific subscribers


API Gateway

  • Fully managed service that allow you to easy publish, create, maintain, monitor and secure your API.
  • It allow you to put a safe “frontdoor” on your application

Notable Features

  • Protect endpoints by attaching WAF
  • Easily implement DDos protection and rare limiting
  • Easy to use

API Options

  • REST API: API Keys, per-client throttles, validation of requests, WAF integration -> Restful API
  • HTTP API: Simpler option than REST API, cheaper, minimum features -> Restful API
  • Websocket API: Collection of WebSocket routes integrated with Lambda functions, HTTP Endpoints and other AWS services

Endpoint Types

  • Edge-Optimized: Default option. API requests get sent through a CloudFront edge. Best for global users
  • Regional: Perfect for clients that reside in the same, specific region. Ability to leverage with CloudFront if required.
  • Private: only accessible via VPCs using interface VPC Endpoints.

Securing your APIs

  • User authentication can be accomplished to control access to your APIs
  • Authentication methods include IAM roles, Amazon Coginito, or even your own custom authorizer(Lambda functions)
  • DNS: Edge optimized endpoints require ACM(AWS Certificate Manager) certs in the us-east-1
  • SSL: Regional endpoints require ACM certs in the same region.
  • WAF: you can place WAF in front of your API to prevent DDos

Example usecase


AWS Batch

Batch Workloads

  • You can use AWS Batch to run batch computing workloads withi AWS(run on EC2 or EC2/Fargate)
  • Scale based on your configuration
  • Automatically Provision and Scale
  • No install required

Important Components


  • Units of work that are submitted to AWS Batch(shell scripts, executeables, and Docker Images)

Job Definitions

  • Specify how your jobs are to be run(blueprint for resources in job)


  • Jobs get submitted to specific queues and reside until scheduled to run in a compute environment.

Compute Environment

  • Set of managed or unmanaged compute resources used to run your jobs

How do you choose between Fargate and EC2 compute environment



  • Recommend approach for MOST workloads
  • Require fast start times(< 30 seconds)
  • Require 16 vCPU or less
  • Require no GPUs
  • Require 120 Gib of memory or less


  • Need more control over instance selection
  • Require GPUs
  • Require Elastic Fabric Adapter
  • Require custom AMIs
  • High levels of concurrency
  • Require access to Linux parameters

AWS Batch or Lambda

Time limits

  • Lambda : 15 mins execution time limit
  • Batch: does not have this

Disk Space

  • Lambda: has limited disk space, and if you wanna leverage it with EFS requires functions live within a VPC

Runtime limitations

  • Lambda is serverless but it has limited runtimes

Batch Runtimes

  • Batch uses Docker, so any runtime can be used

How to leverage AWS Batch in your application



Summary for AWS Batch


Amazon MQ

image image

  • Message broker service allowing easier migrating message broker system to AWS Clouds
  • Allows you to leverage both Apache ActiveMQ and RabbitMQ engine types
  • New applications should try and leverage SNS with SQS
  • Amazon MQ restricts access to private networks. So require VPC connectivity
  • Amazon MQ offer HA architectures: cluster deployments for Amazon RabbitMQ across multi AZ behind NLB

Cordinating Distributed Apps with AWS Step Functions

  • A serverless orchetration service to manage and run event-driven task executions



Ingesting Data from SASS Applications to AWS with Amazon AppFlow

  • Integration: Fully managed integration service for exchanging data between SaaS apps and AWS Services

  • Ingest Data: Pull data records from third-party SaaS vendors and store them in S3

  • Bi-directional data transfer with limited combinations

  • Flow : flows transfer data between sources and destinations

  • Map: determines how your source data is stored within your destination

  • Filters: criteria to control which data is transfered

  • Trigger: how flow start(run on demand, run on event, run on schedule)



  • Transfering Salesforce records to Amazon Redshift (datawarehouse but cheaper)
  • Ingesting and analyzing Slack conversations in S3
  • Migrate Zenddesk and other help desk support tickets to Snowflake(datawarehouse but more expesive)
  • Transfering aggregate data on a schedule basis to S3*