feat: add aws and azure support with localstack and azurite integration#8
Open
erenaslandev wants to merge 6 commits into
Open
feat: add aws and azure support with localstack and azurite integration#8erenaslandev wants to merge 6 commits into
erenaslandev wants to merge 6 commits into
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (2)
WalkthroughAdds LocalStack and Azurite emulator support: case schema, orchestrator wiring, generator S3/Azure Blob upload modes, receivers for S3/SQS/Kinesis/CloudWatch/Azure Blob, ~20+ new benchmark cases and subject configs, CI emulator-smoke jobs, and docs updates. ChangesCloud Emulator Integration for Benchmark Testing
Estimated code review effort: Possibly related PRs
Suggested reviewers
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
README.md (1)
32-63:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winFix the stale total-case count in the project tree snippet.
Lines 32 and 62 update the matrix to 25 performance and 17 correctness, but Line 130 still says
31 test cases (17 performance + 14 correctness).Suggested patch
- cases/ 31 test cases (17 performance + 14 correctness), each with per-subject configs + cases/ 42 test cases (25 performance + 17 correctness), each with per-subject configs🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@README.md` around lines 32 - 63, Update the stale summary sentence that currently reads "31 test cases (17 performance + 14 correctness)" to reflect the new totals after you expanded the matrices: replace it with "42 test cases (25 performance + 17 correctness)"; locate the string in README.md (the summary line referencing total test cases) and change the numbers accordingly so the breakdown matches the updated performance and correctness sections.
🧹 Nitpick comments (1)
containers/receiver/azureblob.go (1)
115-116: Use typedazqueueerror detection for “queue already exists” instead ofstrings.Contains
Incontainers/receiver/azureblob.go(lines 115-116), replacestrings.Contains(err.Error(), "QueueAlreadyExists")withqueueerror.HasCode(err, <queue-already-exists code>)to match the blob-side typed pattern (bloberror.HasCode) and avoid brittle error-message matching.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@containers/receiver/azureblob.go` around lines 115 - 116, Replace the brittle string-match in the retry branch for queue creation: instead of checking strings.Contains(err.Error(), "QueueAlreadyExists") in azureblob.go, use the azqueue typed helper queueerror.HasCode(err, <queue-already-exists code>) (e.g., queueerror.HasCode(err, queueerror.QueueAlreadyExists)) so the condition mirrors the bloberror.HasCode pattern and reliably detects the "queue already exists" error.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Around line 96-103: In the emulator-smoke job replace mutable action tags and
disable checkout credential persistence: change the uses entries "uses:
actions/checkout@v5", "uses: actions/setup-go@v6", and "uses:
docker/setup-buildx-action@v3" to pinned commit SHAs (replace the `@vX` tags with
the corresponding full commit SHA for each action) and add persist-credentials:
false to the actions/checkout step; ensure you edit the emulator-smoke job's
checkout step to include persist-credentials: false and swap the three uses
lines to their respective commit SHA pins.
In `@cases/s3_to_tcp_performance/case.yaml`:
- Around line 30-40: The case declares a TCP receiver (receiver.mode and
receiver.listen = ":9001") but the Filebeat subject is configured to send to
:9002, so update the Filebeat subject configuration to target the case receiver
port: change the Filebeat output/receiver target in the filebeat.yml used by the
"filebeat" subject to point to the case receiver (listen) at :9001, or
alternatively change receiver.listen to ":9002" so both match; ensure the symbol
names to adjust are receiver.listen and the Filebeat subject's output/receiver
setting.
In `@cases/tcp_to_s3_correctness/configs/vector.toml`:
- Around line 1-2: Update the header comment in the Vector config to reference
the correct case name: replace the string "tcp_to_s3_performance" with
"tcp_to_s3_correctness" so the comment matches the file's case directory; ensure
the rest of the comment (e.g., mention of LocalStack and batch timeout) remains
unchanged.
In `@containers/receiver/go.mod`:
- Line 34: The indirect dependency github.com/aws/aws-sdk-go-v2/service/signin
appears unused; run go mod why github.com/aws/aws-sdk-go-v2/service/signin and
then go mod tidy to confirm and remove it if not required, then commit the
tidy-updated module files; after removal, rebuild and run tests (go build / go
test) to ensure nothing breaks and re-add only if go mod why shows a valid
consumer.
In `@internal/config/subject.go`:
- Around line 168-183: The comment next to the vmetric entry is misleading: the
struct sets Version: "2.0.2" while the comment states cloud capabilities require
a director build “newer than 2.0.2”; update the source of truth by either (A)
changing the Version field in the vmetric map to the actual minimum version that
supports emulator-specific fields (e.g., bump Version to the correct semver) or
(B) adjust the comment to explicitly state that although Capabilities (the
Capabilities slice) enables cloud flags, emulator endpoint fields (awss3
endpoint/use_path_style, azblob connection_string) require a newer build and
list the exact minimum version; modify the block containing the vmetric entry
(inspect the Version and Capabilities symbols) accordingly so the comment and
Version value are consistent.
---
Outside diff comments:
In `@README.md`:
- Around line 32-63: Update the stale summary sentence that currently reads "31
test cases (17 performance + 14 correctness)" to reflect the new totals after
you expanded the matrices: replace it with "42 test cases (25 performance + 17
correctness)"; locate the string in README.md (the summary line referencing
total test cases) and change the numbers accordingly so the breakdown matches
the updated performance and correctness sections.
---
Nitpick comments:
In `@containers/receiver/azureblob.go`:
- Around line 115-116: Replace the brittle string-match in the retry branch for
queue creation: instead of checking strings.Contains(err.Error(),
"QueueAlreadyExists") in azureblob.go, use the azqueue typed helper
queueerror.HasCode(err, <queue-already-exists code>) (e.g.,
queueerror.HasCode(err, queueerror.QueueAlreadyExists)) so the condition mirrors
the bloberror.HasCode pattern and reliably detects the "queue already exists"
error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 478b1e8c-c2bc-4d4d-9064-a294c2319479
⛔ Files ignored due to path filters (2)
containers/generator/go.sumis excluded by!**/*.sumcontainers/receiver/go.sumis excluded by!**/*.sum
📒 Files selected for processing (68)
.github/workflows/ci.ymlREADME.mdcases/azure_blob_to_tcp_performance/case.yamlcases/azure_blob_to_tcp_performance/configs/vmetric.ymlcases/s3_to_tcp_correctness/case.yamlcases/s3_to_tcp_correctness/configs/logstash.confcases/s3_to_tcp_correctness/configs/vector.tomlcases/s3_to_tcp_correctness/configs/vmetric.ymlcases/s3_to_tcp_performance/case.yamlcases/s3_to_tcp_performance/configs/filebeat.ymlcases/s3_to_tcp_performance/configs/logstash.confcases/s3_to_tcp_performance/configs/vector.tomlcases/s3_to_tcp_performance/configs/vmetric.ymlcases/tcp_to_azure_blob_correctness/case.yamlcases/tcp_to_azure_blob_correctness/configs/fluent-bit.confcases/tcp_to_azure_blob_correctness/configs/vector.tomlcases/tcp_to_azure_blob_correctness/configs/vmetric.ymlcases/tcp_to_azure_blob_performance/case.yamlcases/tcp_to_azure_blob_performance/configs/fluent-bit.confcases/tcp_to_azure_blob_performance/configs/vector.tomlcases/tcp_to_azure_blob_performance/configs/vmetric.ymlcases/tcp_to_cloudwatch_performance/case.yamlcases/tcp_to_cloudwatch_performance/configs/fluent-bit.confcases/tcp_to_cloudwatch_performance/configs/vector.tomlcases/tcp_to_cloudwatch_performance/configs/vmetric.ymlcases/tcp_to_kinesis_performance/case.yamlcases/tcp_to_kinesis_performance/configs/fluent-bit.confcases/tcp_to_kinesis_performance/configs/vector.tomlcases/tcp_to_kinesis_performance/configs/vmetric.ymlcases/tcp_to_s3_correctness/case.yamlcases/tcp_to_s3_correctness/configs/fluent-bit.confcases/tcp_to_s3_correctness/configs/logstash.confcases/tcp_to_s3_correctness/configs/vector.tomlcases/tcp_to_s3_correctness/configs/vmetric.ymlcases/tcp_to_s3_performance/case.yamlcases/tcp_to_s3_performance/configs/fluent-bit.confcases/tcp_to_s3_performance/configs/logstash.confcases/tcp_to_s3_performance/configs/vector.tomlcases/tcp_to_s3_performance/configs/vmetric.ymlcases/tcp_to_sns_performance/case.yamlcases/tcp_to_sns_performance/configs/logstash.confcases/tcp_to_sns_performance/configs/vector.tomlcases/tcp_to_sns_performance/configs/vmetric.ymlcases/tcp_to_sqs_performance/case.yamlcases/tcp_to_sqs_performance/configs/logstash.confcases/tcp_to_sqs_performance/configs/vector.tomlcases/tcp_to_sqs_performance/configs/vmetric.ymlcontainers/generator/azureblob.gocontainers/generator/go.modcontainers/generator/main.gocontainers/generator/s3.gocontainers/receiver/azureblob.gocontainers/receiver/cloudpoll.gocontainers/receiver/cloudpoll_test.gocontainers/receiver/cloudwatch.gocontainers/receiver/go.modcontainers/receiver/kinesis.gocontainers/receiver/main.gocontainers/receiver/s3.gocontainers/receiver/sqs.gointernal/config/case.gointernal/config/cloud.gointernal/config/cloud_test.gointernal/config/subject.gointernal/orchestrator/awsinit.gointernal/orchestrator/compose_render_test.gointernal/orchestrator/docker.gointernal/runner/runner.go
Deploying pipebench with
|
| Latest commit: |
5303814
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://9bfadd42.pipebench.pages.dev |
| Branch Preview URL: | https://feature-aws-azure-support.pipebench.pages.dev |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary by CodeRabbit
New Features
Tests
Documentation