Aws Series Decoupling Workflows

Post Date : 2023-12-30T23:48:50+07:00

Modified Date : 2023-12-30T23:48:50+07:00

Category: systemdesign aws

Tags: aws

The issue with tight coupling

The solution : loose coupling

Simple Queue Service(SQS)

• SQS is fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
• A message queue that allows asynchronous processing of work. One resource write a message to an SQS queue, and then another resource retrieve that messages from SQS

Some important settings

• Delivery delay: default is 0, max value is 15 minutes
• Message size: up to 256KB of text in any format
• Encryption: messages are encrypted in transit by default, but you can add at-rest
• Message retention: default is 4 days, can be set between 1 minutes and 14 days.
• Long vs Short: Long polling isn’t the default, but it should be.
• Queue Depth: this can be a trigger for autoscaling -> if two many messages in queue, add more instances to solve it.

SSE-SQS

Visibility Timeout

Simple Notification Service(SNS)

• SNS is a fully managed messaging service for both application-to-application (A2A) and application-to-person(A2P) communication

API Gateway

• API Gateway is a fully managed service that makes easy for developers to create,publish,maintain, monitor, and secure APIs at any scale.

Sidelining Message Queue with Dead-Letter Queues

• DLQ are the targets for messages that can not be processed successfully
• Works with SQS and SNS!
• Useful for debugging applications and messaging systems
• Ability to isolate unconsumed messages to troubleshoot
• Redrive capacity allows you to move the message back into the source queue
• These are technically just other SQS queues
• DLQs used FIFO SQS queues must ALSO be FIFO queues

Benefits:

• Configure alarms based on message availability counts
• Quickly identify which logs to investigate for exceptions
• Analyze the SQS message contents for any errors
• Troubleshoot consumer permissions

Order Messages using SQS FIFO

• Guaranteed ordering
• No message duplication
• 300 transactions per second <–> Batching can achieve up to 3,000 messages per second, per API call

FIFO High throughput

• Process up to 9,000 transactions per second, per API without batching
• Up to 90,000 transactions per second by using batching APIs

Delivering Messages with SNS

SNS: Simple Notification Service

• Push-based messaging service, proactively delivers messages to the endpoints that are subscribed to it.
• This can be used to alert a system or a person
• One message can be sent to many

SNS Settings and Quotas

Subcribers

• Kinesis Data Firehose, SQS, Lambda, email, HTTP(s), SMS and platform application endpoint.

Message Size

• Up to 256KB of text in any format

DLQ support

• Messages that failed to delivered can be stored in SQS DLQ

FIFO or Standard

• FIFO only supports SQS FIFO queues as subcribers

Encryption

• Messages are encrypted in transit by default, and you can add at-rest via AWS KMS

Access Policies

• A resource policy can be added to a topic, similar to S3. Useful for cross-account access.

Large Message Payloads

• The SNS Extended Library allows sending messages up to 2GB in size
• The payload is stored in Amazon S3, then SNS published a reference to the object

SNS Fanout

• Messages published in SNS topics are replicated in multiple endpoint subcriptions
• Allow for fully decoupled parallel asynchronous processing

SNS Architecture

Message Filtering

• By default, every message published to a topic is sent to all subcribers
• Filter policies use JSON to define which messages get sent to specific subscribers

API Gateway

• Fully managed service that allow you to easy publish, create, maintain, monitor and secure your API.
• It allow you to put a safe “frontdoor” on your application

Notable Features

• Protect endpoints by attaching WAF
• Easily implement DDos protection and rare limiting
• Easy to use

API Options

• REST API: API Keys, per-client throttles, validation of requests, WAF integration -> Restful API
• HTTP API: Simpler option than REST API, cheaper, minimum features -> Restful API
• Websocket API: Collection of WebSocket routes integrated with Lambda functions, HTTP Endpoints and other AWS services

Endpoint Types

• Edge-Optimized: Default option. API requests get sent through a CloudFront edge. Best for global users
• Regional: Perfect for clients that reside in the same, specific region. Ability to leverage with CloudFront if required.
• Private: only accessible via VPCs using interface VPC Endpoints.

Securing your APIs

• User authentication can be accomplished to control access to your APIs
• Authentication methods include IAM roles, Amazon Coginito, or even your own custom authorizer(Lambda functions)
• DNS: Edge optimized endpoints require ACM(AWS Certificate Manager) certs in the us-east-1
• SSL: Regional endpoints require ACM certs in the same region.
• WAF: you can place WAF in front of your API to prevent DDos

Example usecase

AWS Batch

Batch Workloads

• You can use AWS Batch to run batch computing workloads withi AWS(run on EC2 or EC2/Fargate)
• Scale based on your configuration
• Automatically Provision and Scale
• No install required

Important Components

Jobs

• Units of work that are submitted to AWS Batch(shell scripts, executeables, and Docker Images)

Job Definitions

• Specify how your jobs are to be run(blueprint for resources in job)

JobQueues

• Jobs get submitted to specific queues and reside until scheduled to run in a compute environment.

Compute Environment

• Set of managed or unmanaged compute resources used to run your jobs

How do you choose between Fargate and EC2 compute environment

Fargate

• Recommend approach for MOST workloads
• Require fast start times(< 30 seconds)
• Require 16 vCPU or less
• Require no GPUs
• Require 120 Gib of memory or less

EC2

• Need more control over instance selection
• Require GPUs
• Require Elastic Fabric Adapter
• Require custom AMIs
• High levels of concurrency
• Require access to Linux parameters

AWS Batch or Lambda

Time limits

• Lambda : 15 mins execution time limit
• Batch: does not have this

Disk Space

• Lambda: has limited disk space, and if you wanna leverage it with EFS requires functions live within a VPC

Runtime limitations

• Lambda is serverless but it has limited runtimes

Batch Runtimes

• Batch uses Docker, so any runtime can be used

How to leverage AWS Batch in your application

Summary for AWS Batch

Amazon MQ

• Message broker service allowing easier migrating message broker system to AWS Clouds
• Allows you to leverage both Apache ActiveMQ and RabbitMQ engine types
• New applications should try and leverage SNS with SQS
• Amazon MQ restricts access to private networks. So require VPC connectivity
• Amazon MQ offer HA architectures: cluster deployments for Amazon RabbitMQ across multi AZ behind NLB

Cordinating Distributed Apps with AWS Step Functions

• A serverless orchetration service to manage and run event-driven task executions

Ingesting Data from SASS Applications to AWS with Amazon AppFlow

• Integration: Fully managed integration service for exchanging data between SaaS apps and AWS Services

• Ingest Data: Pull data records from third-party SaaS vendors and store them in S3

• Bi-directional data transfer with limited combinations

• Flow : flows transfer data between sources and destinations

• Map: determines how your source data is stored within your destination

• Filters: criteria to control which data is transfered

• Trigger: how flow start(run on demand, run on event, run on schedule)

Usecases

• Transfering Salesforce records to Amazon Redshift (datawarehouse but cheaper)
• Ingesting and analyzing Slack conversations in S3
• Migrate Zenddesk and other help desk support tickets to Snowflake(datawarehouse but more expesive)
• Transfering aggregate data on a schedule basis to S3*