laserdisc-io/fs2-aws

This system is a Scala library that provides functional, stream-based interfaces for interacting with major AWS services, including Kinesis, S3, DynamoDB, SQS, and SNS. It leverages the fs2 library to handle data processing pipelines and utilizes the AWS SDK v2 for asynchronous, non-blocking operations. The library supports both direct stream manipulation and algebraic, tagless-final patterns for service-agnostic code, facilitating robust integration with AWS infrastructure.

10 changes.

The project underwent a major infrastructure overhaul, migrating from AWS SDK v1 to v2 and upgrading the primary language version to Scala 3. This period involved restructuring the repository, removing legacy S3 file reading capabilities, and introducing new testing utilities and stream processing features for Kinesis and SQS.

17 changes.

This period focused on migrating the codebase to the AWS SDK v2, introducing new tagless-final interpreters for S3, SNS, SQS, Kinesis, and DynamoDB. It also added new fs2 modules for SNS and DynamoDB streaming, modernized the Kinesis consumer API with a builder pattern, and expanded test coverage and benchmarking infrastructure.

Introduces a new \fs2-aws-sns\ module that provides an \SNS\ trait and \SnsSettings\ for publishing messages to Amazon SNS topics via an fs2 \Pipe\. The implementation uses the \SnsAsyncClientOp\ from \io.laserdisc.pure\ to handle concurrent publishing, and includes a test suite (\SnsSpec\) that verifies message delivery to an SQS queue using localstack.

fs2-aws-sns · high confidence

The fs2-aws-benchmarks module now includes JMH benchmarks for measuring the performance of Kinesis stream processing and S3 multipart uploads. This adds a new logback configuration to control log verbosity during benchmark runs, a KinesisFlowBenchmark that tests reading and checkpointing records from a Kinesis stream using a test context, and an S3Benchmark that measures multipart upload throughput against a local S3 endpoint using a stubbed S3 client implementation.

fs2-aws-benchmarks · high confidence

The S3 upload API now supports optional ETag validation for multipart uploads to ensure data integrity, allowing users to verify that the server-side checksum matches the expected value. Additionally, new \AwsRequestModifier\ traits enable users to customize AWS request builders for both single-part and multipart uploads, providing greater control over upload parameters.

fs2-aws-s3/src/main/scala · high confidence

A new shell script, scripts/localstack/create-stream.sh, has been added to facilitate local development and testing with LocalStack. This script executes the awslocal CLI command to create a Kinesis stream named 'test' with a single shard, enabling developers to quickly provision the necessary stream infrastructure locally.

scripts · high confidence

Added example applications and configuration helpers for DynamoDB, Kinesis, S3, SQS, and SNS. The new files include DynamoParallelScan for parallel table copying, KinesisExample and KinesisExampleNew for single-stream consumption, KinesisMultistreamExample for multi-stream processing, PureAWSKleisli for SQS/SNS integration using Kleisli and tagless finalism, and S3/TracedS3 examples for object storage. Supporting configuration classes (KinesisAppConfig, KinesisMultistreamAppConfig) provide localstack setup and SDK builder extensions.

fs2-aws-examples/src/main/scala · high confidence

A new \StreamScan\ component has been added to the fs2-aws-dynamodb module, providing a \scanDynamoDB\ method that streams raw DynamoDB items via an FS2 Stream. This implementation uses a bounded queue to enforce back-pressure against the AWS SDK paginator, ensuring that memory usage remains controlled during large scans, and includes error handling via Deferred to propagate failures from the reactive stream subscription.

fs2-aws-dynamodb/src/main · high confidence

A new \groupBy\ helper has been added to the \fs2.aws.core\ package, providing a functional pipe to partition a stream into multiple sub-streams based on a dynamic selector function. This utility manages the lifecycle of internal queues for each unique key, allowing users to process distinct partitions of data concurrently within a single stream pipeline.

fs2-aws-core/src/main · high confidence

Added \SnsAsyncClientOp\ and \SnsInterpreter\ in the \pure-sns-tagless\ module, providing a new tagless-final interface for the AWS SNS service. This introduces effectful equivalents for all SNS operations (such as publishing, subscribing, and topic management) backed by the AWS SDK v2, allowing users to write service-agnostic code that can be interpreted against the async client.

pure-aws/pure-sns-tagless · high confidence

This location introduces the core algebraic interface (\KinesisAsyncClientOp\) and its Kleisli-based interpreter (\KinesisInterpreter\) for AWS Kinesis. The algebra defines effectful equivalents for all Kinesis operations (such as \putRecord\, \getRecords\, \describeStream\, and pagination support), while the interpreter provides the concrete implementation that bridges these effects to the AWS SDK. The interpreter also includes deprecated aliases for the previous \Interpreter\ name to maintain backward compatibility with version 6.x code.

pure-aws/pure-kinesis-tagless · high confidence

The testkit now includes \SchedulerFactoryTestContext\ and \TestKinesisProducerClient\ to facilitate testing of Kinesis consumer and producer logic. \SchedulerFactoryTestContext\ provides a mockable scheduler that tracks shard record processors and allows tests to control execution flow via latches. \TestKinesisProducerClient\ implements the producer client interface using an in-memory \Ref\ to capture emitted records, enabling verification of data sent to Kinesis streams without requiring real AWS connections.

fs2-aws-testkit · high confidence

The S3 file reading functionality previously provided in the \fs2.aws.s3\ package object has been removed. This eliminates the ability to stream files from S3 using the \readFile\ method defined in this module.

src/main · high confidence

The \CloudWatchInterpreter\ and \DynamoDbInterpreter\ objects and traits have been renamed to \CloudWatchInterpreter\ and \DynamoDbInterpreter\ respectively (previously just \Interpreter\). The old \Interpreter\ names are retained as deprecated aliases for version 7.0.0 to ensure backward compatibility with existing 6.x code, but users should migrate to the new names. This change accompanies the broader migration to the AWS SDK v2, as evidenced by the imports of \software.amazon.awssdk.services.\*\ classes in the new \\*AsyncClientOp\ and \\*Interpreter\ files.

pure-aws/pure-dynamodb-tagless · high confidence

The S3 module now uses refined types to enforce that multipart upload parts are at least 5 MB, preventing invalid uploads that would fail AWS validation. Additionally, bucket names and file keys are now strictly validated as non-empty strings, ensuring that empty or null values cannot be passed to S3 operations.

fs2-aws-s3/src/main/scala-3.0+ · high confidence

The library introduces a new, phase-based \KinesisStreamBuilder\ API for configuring Kinesis consumers, replacing the previous settings-based approach. This builder guides users through providing required AWS clients (Kinesis, DynamoDB, CloudWatch) and configuration options like app name and scheduler ID. Alongside the builder, the library adds a \CommittableRecord\ type that wraps raw Kinesis records with checkpointing capabilities, allowing users to explicitly commit offsets via a \checkpointRecords\ pipe. The change also includes a new \ChunkedRecordProcessor\ to handle record batching and a \KinesisProducerClient\ for publishing data, effectively modernizing the consumer/producer interfaces while maintaining compatibility with the underlying Kinesis Client Library (KCL).

fs2-aws-kinesis/src/main · high confidence

The project has been restructured for the 7.x release, migrating from AWS SDK v1 to v2 and from Scala 2 to Scala 3. This update introduces breaking changes for users upgrading from the 6.x series, requiring them to follow the new migration notes. The repository now includes configuration for the new toolchain (scalafmt 3.11.5, Scala 3 dialect), automated release workflows via Mergify, and local development support through Docker Compose (LocalStack, MinIO).

(repo-wide) · high confidence

The S3 tagless interpreter in \pure-s3-tagless\ has been updated to wrap the AWS SDK v2 \S3AsyncClient\ instead of the previous synchronous client. This change introduces effectful equivalents for all S3 operations (such as bucket management, object uploads/downloads, and pagination) that return \F\[Response\]\ types, enabling non-blocking, asynchronous interactions with Amazon S3 within the Cats Effect ecosystem.

pure-aws/pure-s3-tagless · high confidence

The fs2-aws-sqs module introduces a new \changeMessageVisibilityPipe\ that allows users to extend the visibility timeout of messages during processing, preventing them from becoming visible to other consumers prematurely. Additionally, the \SqsConfig\ now exposes a \fetchMessageCount\ parameter (defaulting to 10) to control how many messages are retrieved per poll, and the legacy \sqs.SqsConfig\ is deprecated in favor of \fs2.aws.sqs.SqsConfig\.

fs2-aws-sqs/src/main · high confidence

The \pure-sqs-tagless\ module now provides a new \SqsAsyncClientOp\ trait and \SqsInterpreter\ object to define and implement effectful SQS operations using the AWS SDK v2 \SqsAsyncClient\. This change introduces a modern, tagless-finalizer-based API for SQS interactions. As part of this transition, the previous \Interpreter\ trait and object are deprecated in favor of \SqsInterpreter\ for version 7.0.0, ensuring existing code continues to compile while encouraging migration to the new API.

pure-aws/pure-sqs-tagless · high confidence

Added integration tests for SQS message lifecycle; Added test resources and test logging configuration; Added tests for DynamoDB stream scanning; Added tests for multipart ETag validation and S3 upload flows; Added tests for the groupBy stream operator; Added tests for the new Kinesis producer and consumer implementation; Removed S3 integration tests.

The project has migrated from the legacy AWS SDK v1 to AWS SDK v2 (version 2.40.5) and upgraded the primary Scala version to 3.3.8, while maintaining cross-compilation support for Scala 2.13.18. This change replaces the old \com.amazonaws\ SDK dependencies with the new \software.amazon.awssdk\ modules (S3, SQS, SNS, Kinesis, DynamoDB, CloudWatch) and updates the build infrastructure to reflect these modern standards.

(dependencies) · high confidence

The build system has been upgraded from sbt 0.13.16 to 1.13.0, and the project now depends on AWS SDK v2 (version 2.54.12) instead of the legacy v1. This change updates the core library dependencies, including the Amazon Kinesis Client to version 3.5.1, and refreshes the build plugins (such as sbt-ci-release and sbt-scalafmt) to their latest versions.

project · high confidence

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

• First survey — no prior run to compare against. CAI 59.

• Code Health 92
• Architecture 100
• Maturity 48
• Readiness 53
• Security 75

• 300 commits — 283 feature/other, 17 fixes

• project/Dependencies.scala — 159 commits
• (root) — 52 commits
• project/plugins.sbt — 31 commits
• project/build.properties — 21 commits
• fs2-aws-kinesis/src — 8 commits
• fs2-aws-dynamodb/src — 4 commits
• fs2-aws-s3/src — 4 commits
• (repo) — 3 commits
• .github/workflows — 3 commits
• docs/modules — 3 commits
• fs2-aws-examples/src — 3 commits
• fs2-aws-sqs/src — 2 commits
• project/TaglessGen.scala — 2 commits
• docs/site.css — 1 commit
• project/build.sbt — 1 commit
• pure-aws/pure-cloudwatch-tagless — 1 commit
• pure-aws/pure-dynamodb-tagless — 1 commit
• pure-aws/pure-s3-tagless — 1 commit

• fix: Fix deduplication in TaglessGen
• fix: Fix incorrect CompletableFuture integration (#1088)
• fix: Fix the link to S3Example in readme (#1112)
• fix: Potential fix for pull request finding
• fix: code review fixes
• fix: docs fix
• fix: fix docs
• fix: fix scaladoc
• fix: fix scalafmt for generated files
• fix: fix up docs a bit
• fix: fix up readme
• fix: fix versioning scheme
• fix: fixes
• fix: improved tagless gen, fix up docs
• fix: many upgrades, fix build (#1343)
• fix: revert temporary test state
• fix: validate code examples, fix S3 docs, some styling
• change: Adds controls for SQS message visibility. (#1341)
• change: Avoid allocating a vector if you can use the underlying chunks (#1136)
• change: Avoid shutdown race conditions
• …and 280 more

Written by watchdog.canine.dev from the codebase's own history, inside the signed delivery this page is composed from.

laserdisc-io/fs2-aws was measured the same way every project in this corpus was: the same rubric, at a pinned commit, with the result published in full. Point a surveyor at a repository you know and see whether you agree with it.