aws.securityhub_findings: Improve support for CDR #11158
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Sep 17, 2024
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
s3access |
4651.16 | 3831.42 | -819.74 (-17.62%) | 💔 |
apigateway_logs |
10989.01 | 4464.29 | -6524.72 (-59.37%) | 💔 |
ec2_metrics |
25000 | 17857.14 | -7142.86 (-28.57%) | 💔 |
firewall_logs |
3300.33 | 2645.5 | -654.83 (-19.84%) | 💔 |
To see the full report comment with /test benchmark fullreport
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
efd6
reviewed
Sep 24, 2024
Contributor
efd6
left a comment
There was a problem hiding this comment.
The issue refers to a document upload, but I cannot find it. So I cannot see whether this follows what has been designed. Is there a reason this is not a public document in the issue?
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Show resolved
Hide resolved
…spm-aws_securityhub
andrewkroh
reviewed
Oct 28, 2024
packages/aws/kibana/dashboard/aws-c9f103d0-5f63-11ed-bd69-473ce047ef30.json
Outdated
Show resolved
Hide resolved
Comment on lines
36
to
43
| - set: | ||
| field: observer.vendor | ||
| value: AWS Security Hub | ||
| tag: set_observer_vendor | ||
| - set: | ||
| field: cloud.provider | ||
| value: aws | ||
| tag: set_cloud_provider |
Member
There was a problem hiding this comment.
The three fields being converted to constant_keyword would all benefit from removal from _source.
I recommend setting the static values in the ecs.yml file where they fields are declared instead of the ingest pipeline, and then exchange the three set processors with a single remove processor that has a description field explaining that the fields are defined as constant_keyword and we are removing the fields from _source to gain storage efficiency.
Member
|
Also, please update the commit message (in the PR description) to specify why the minimum kibana version was changed. |
Contributor
Author
|
@andrewkroh the comments are addressed in 0e44091 and PR commit message is also updated. |
CohenIdo
approved these changes
Oct 30, 2024
andrewkroh
approved these changes
Oct 30, 2024
Member
andrewkroh
left a comment
There was a problem hiding this comment.
I left a few more minor comments.
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/aws/data_stream/securityhub_findings/elasticsearch/ingest_pipeline/default.yml
Outdated
Show resolved
Hide resolved
packages/aws/elasticsearch/transform/latest_cdr_misconfigurations/fields/ecs.yml
Outdated
Show resolved
Hide resolved
packages/aws/elasticsearch/transform/latest_cdr_misconfigurations/fields/base-fields.yml
Show resolved
Hide resolved
shmsr
approved these changes
Oct 30, 2024
Member
shmsr
left a comment
There was a problem hiding this comment.
Based on other approvals, approving as a CODEOWNER from @elastic/obs-infraobs-integrations
zmoog
approved these changes
Oct 30, 2024
|
💚 Build Succeeded
History
cc @kcreddy |
|
Package aws - 2.31.0 containing this change is available at https://epr.elastic.co/search?package=aws |
5 tasks
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 4, 2025
* Add CSPM fields - 1 * reformat * reformat * Add more ECS fields * Consider multiple resources * Split single and multiple resource logic. Add multiple resources test. * Add tags and update comments * Add visualization to findings dashboard * update typeMigrationVersion on kibana searches * Address PR comments. * Address PR comments-1 * Add PR comment-2 - Add host.ip and host.name * Address PR comments-3. Use constant_keyword * Address PR comments-4. Separate res.Details != null condition block and field separation. * fix HEAD * ecs fields sorted. * Address Pr comments-5. Remove unused fields from mapping. * Add misconfiguration_latest transform * Address PR comment. Update transform retention to 90d. * Address PR comments. Updated rule fields to keyword. * update readme * Remove references from pipeline tests * update fields to ecs * address pr comments. * fix static test * update/fix readme * address pr comments * address pr comments. Remove unnecessary `ignore_empty_value` option
harnish-crest-data
pushed a commit
to chavdaharnish/integrations
that referenced
this pull request
Feb 5, 2025
* Add CSPM fields - 1 * reformat * reformat * Add more ECS fields * Consider multiple resources * Split single and multiple resource logic. Add multiple resources test. * Add tags and update comments * Add visualization to findings dashboard * update typeMigrationVersion on kibana searches * Address PR comments. * Address PR comments-1 * Add PR comment-2 - Add host.ip and host.name * Address PR comments-3. Use constant_keyword * Address PR comments-4. Separate res.Details != null condition block and field separation. * fix HEAD * ecs fields sorted. * Address Pr comments-5. Remove unused fields from mapping. * Add misconfiguration_latest transform * Address PR comment. Update transform retention to 90d. * Address PR comments. Updated rule fields to keyword. * update readme * Remove references from pipeline tests * update fields to ecs * address pr comments. * fix static test * update/fix readme * address pr comments * address pr comments. Remove unnecessary `ignore_empty_value` option
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Improve support for CDR.
securityhub_findingsdata stream's ingest pipeline to support CDR.securityhub_findingsdata stream's mappings according to the new fields.8.16.0as the transform privileges are added in [Cloud Security] Add privileges required for AWS SecurityHub related to CDR misconfiguration features elasticsearch#112574 merged into8.16Stack release.Fixes: #11040
Note to reviewers: Please DM me for access to the document(s) linked in the issue, it might help in the review.
Checklist
changelog.ymlfile.How to test this PR locally
Related issues
Screenshots