Determine high-performing data ingestion and transformation solutions

Haverly Telecom runs a nightly batch of Apache Spark and Hive jobs on an on-premises Hadoop cluster to process 50 TB of call detail records. The jobs rely on custom native libraries installed through cluster bootstrap scripts and on specific Hadoop configuration overrides. Haverly wants to move the processing to AWS with the data in S3, keep the jobs unchanged, run the cluster only for the few hours the batch takes, and use Spot capacity to lower cost.

Which solution will meet these requirements?

1. ARun the jobs on a transient Amazon EMR cluster that reads from and writes to S3, uses bootstrap actions for the libraries, and uses Spot Instances for task nodesEMR runs Spark and Hive natively with cluster-level control: bootstrap actions install custom software, configuration classifications apply Hadoop overrides, the cluster can terminate after the steps complete, and instance fleets can draw task nodes from Spot capacity.
2. BRewrite the jobs as AWS Lambda functions triggered on a scheduleLambda has a 15-minute limit and no Spark or Hive runtime; the jobs would have to be rewritten from scratch and cannot process 50 TB in one invocation.
3. CRun the jobs as AWS Glue ETL jobsGlue runs serverless Spark and is attractive for new ETL, but it does not run Hive jobs, does not expose cluster bootstrap actions or Hadoop configuration overrides, and does not use Spot Instances.
4. DLoad the records into Amazon Redshift and rewrite the jobs as SQLA data warehouse changes the processing model entirely and would be charged continuously rather than for a few hours per night.

Ashgrove Foods stores sales data in Amazon Redshift and web analytics data in S3 queried through Amazon Athena. Two hundred business users across sales and operations need interactive dashboards with filters and drill-downs, delivered through a browser and a mobile app, with row-level restrictions so that regional managers see only their own regions. Ashgrove does not want to run or patch any servers for the dashboards.

Which service should a solutions architect recommend?

1. AGrafana installed on an Auto Scaling group of EC2 instancesSelf-hosting Grafana means operating and patching servers, which the company wants to avoid, and Grafana is oriented to operational metrics rather than business intelligence.
2. BAmazon Quick (formerly Amazon QuickSight)Amazon Quick is a serverless business intelligence service that connects directly to Redshift and Athena, serves interactive dashboards in the browser and on mobile, scales to thousands of users, and enforces row-level security per user or group.
3. CAmazon CloudWatch dashboardsCloudWatch dashboards visualize metrics and logs from AWS resources; they do not connect to Redshift or Athena as business data sources or offer row-level security for end users.
4. DAmazon OpenSearch Service with OpenSearch DashboardsOpenSearch Dashboards visualizes data indexed in OpenSearch; the company's data lives in Redshift and S3, and it would need a managed OpenSearch domain to be sized and maintained.

Wrenfield Insurance receives 500 GB of claims data each day as gzip-compressed CSV files in an S3 bucket named wrenfield-claims-raw. Analysts query the data with Amazon Athena, but queries that touch only a few columns are slow and expensive because Athena scans every byte of every file. Wrenfield wants faster, cheaper queries with a managed, scheduled transformation and no servers to maintain.

Which solution will meet these requirements?

1. AIncrease the number of Athena workgroups and assign each analyst team its own workgroupWorkgroups separate query history, limits, and cost tracking; they do not change how much data each query scans.
2. BRecompress the CSV files with a higher compression levelStronger compression shrinks files slightly, but Athena must still decompress and read every row and column because CSV is row-oriented and not splittable when gzipped.
3. CLoad the CSV files into an Amazon RDS for PostgreSQL instance and query it from Athena with a federated connectorA relational instance adds a server-sized database to manage and pay for continuously, and federated queries add latency rather than reduce scan cost.
4. DCreate a scheduled AWS Glue ETL job that converts the CSV files to Apache Parquet partitioned by claim date in a curated bucket, and point the Athena table at the curated dataParquet is columnar and compressed, so Athena reads only the referenced columns, and date partitions let it skip entire directories; Glue runs the conversion serverlessly on a schedule.

Lockridge Logistics ingests GPS pings from its truck fleet into an Amazon Kinesis Data Streams stream named fleet-positions with 20 shards. Five consumer applications read the stream using standard shared-throughput consumers. As consumers were added, end-to-end latency rose above 2 seconds and the applications began logging ReadProvisionedThroughputExceeded errors even though write throughput is well within limits. Lockridge needs each consumer to receive records within about 70 milliseconds and wants to avoid resharding solely to add read capacity.

What should a solutions architect recommend?

1. ARegister each consumer application as an enhanced fan-out consumer of fleet-positionsEnhanced fan-out gives every registered consumer its own dedicated 2 MB per second per shard of read throughput pushed over HTTP/2, so consumers no longer compete for the shared 2 MB per second and latency drops to about 70 milliseconds.
2. BDouble the number of shards to 40More shards raise aggregate read capacity but each shard still offers only 2 MB per second shared across all five consumers, and it doubles cost for the write side that is not constrained.
3. CUse the Kinesis Producer Library to aggregate records before writing themProducer-side aggregation improves write efficiency; the problem is on the read side, where consumers exceed the shared per-shard read limit.
4. DReplace the stream with Amazon Data Firehose delivering to S3 and have the consumers read the filesFirehose batches records and delivers them to S3 on a buffer size or interval; even with zero buffering, delivery takes seconds, and reading files from S3 offers no real-time consumer API, so it cannot meet a 70-millisecond latency target.