[AWS] [API Gateway] Add custom access logging fields #7151

lucian-ioan · 2023-07-26T12:36:01Z

Urgency

Normal

Activity Type

Enhancement

What does this PR do?

Adds apiId, domainName and stage fields which need Custom Access Logging enabled in the AWS API Gateway settings and to be added to the default log format via context variables. Also enhances current tests.

These fields are common between all API types supported by AWS and provide value in filtering data/enhancing dashboards.

apiId -> unique identifier for the API
domainName -> full URL path to the API, easier to identify with than apiId
stage -> environment typically used as a step in the lifecycle of the API (ex. dev/beta/prod/)

Documentation for context variables based on API type:

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Set up an API using the AWS API Gateway service and enable Custom Access Logging
Add the following mapping to collect the additional fields in AWS: {"apiId": "$context.apiId", "domainName": "$context.domainName", "stage": "$context.stage"}
Run the AWS API Gateway integration locally using elastic-package and validate the fields are properly ingested

Related issues

Closes [AWS] [API Gateway] Custom Access Logging support for logs integration #7150

elasticmachine · 2023-07-26T12:47:08Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-11T12:11:58.034+0000
Duration: 53 min 44 sec

Test stats 🧪

Test	Results
Failed	0
Passed	208
Skipped	4
Total	212

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

elasticmachine · 2023-07-26T15:23:19Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`17/17`)	💚
Files	94.444% (`17/18`)	👎 -5.556
Classes	94.444% (`17/18`)	👎 -5.556
Methods	85.953% (`257/299`)	👍 12.62
Lines	86.024% (`7509/8729`)	👎 -13.976
Conditionals	100.0% (`0/0`)	💚

packages/aws/_dev/build/docs/apigateway.md

zmoog · 2023-07-31T23:27:28Z

packages/aws/_dev/build/docs/apigateway.md

We should expand this section.

Here's the information we should convey to the users:

the API Gateway offers a wide array of additional fields you can pick from

you can leverage the custom pipeline to process the field names or values

you can leverage the custom mapping to store data efficiently

zmoog

This PR is looking good; I have a few minor suggested changes.

lucian-ioan · 2023-08-02T15:47:33Z

It seems changing my github username finally caught up to me (resigned the CLA).

What's your opinion on the new docs @zmoog ?

cla-checker-service · 2023-08-02T15:52:43Z

💚 CLA has been signed

zmoog · 2023-08-03T08:50:20Z

@lucian-ioan, do you still have problems with CLA?

lucian-ioan · 2023-08-03T08:52:55Z

/test

yes @zmoog, apparently so, I believe there need to be some signatures on the other end as well for it to be in effect.

Can the PR not be merged without the CLA passing?

packages/aws/_dev/build/docs/apigateway.md

zmoog · 2023-08-03T08:57:36Z

Can the PR not be merged without the CLA passing?

No.

We need to sort this out.

annamikhailovskaia · 2023-08-09T16:03:23Z

cla/check

umatiw · 2023-08-10T12:32:04Z

cla/check

umatiw · 2023-08-10T13:38:44Z

cla/check

packages/aws/_dev/build/docs/apigateway.md

zmoog

nit: tiny suggestion on the wording.

packages/aws/_dev/build/docs/apigateway.md

Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

bhapas · 2023-08-11T12:00:23Z

packages/aws/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.52.2"


May be a minor version bump here as we are adding the fields and it is an enhancement

bhapas

LGTM

elasticmachine · 2023-08-11T13:31:36Z

Package aws - 1.53.0 containing this change is available at https://epr.elastic.co/search?package=aws

lucian-ioan self-assigned this Jul 26, 2023

lucian-ioan added enhancement New feature or request Integration:aws AWS labels Jul 26, 2023

lucian-ioan requested review from gpop63 and zmoog July 26, 2023 12:43

lucian-ioan marked this pull request as ready for review July 31, 2023 06:23

lucian-ioan requested a review from a team as a code owner July 31, 2023 06:23

zmoog reviewed Jul 31, 2023

View reviewed changes

packages/aws/_dev/build/docs/apigateway.md Outdated Show resolved Hide resolved

zmoog reviewed Jul 31, 2023

View reviewed changes

zmoog requested changes Jul 31, 2023

View reviewed changes

elastic deleted a comment from cla-checker-service bot Aug 2, 2023

lucian-ioan requested a review from zmoog August 2, 2023 15:31

zmoog reviewed Aug 3, 2023

View reviewed changes

packages/aws/_dev/build/docs/apigateway.md Outdated Show resolved Hide resolved

lucian-ioan force-pushed the aws_apigateway_additional_fields branch 2 times, most recently from 0da8058 to 144af0f Compare August 9, 2023 14:21

lucian-ioan requested a review from a team as a code owner August 10, 2023 12:37

Ioan-Lucian Deaconescu added 4 commits August 11, 2023 12:34

remove previous tests

d24c0ee

add new tests by API type

e27e577

add new fields

2fb5ea9

add docs

d8d62a7

add changelog entry

5e32eba

lucian-ioan force-pushed the aws_apigateway_additional_fields branch from 134917a to 5e32eba Compare August 11, 2023 09:53

lucian-ioan requested a review from zmoog August 11, 2023 10:12

zmoog requested changes Aug 11, 2023

View reviewed changes

packages/aws/_dev/build/docs/apigateway.md Outdated Show resolved Hide resolved

Ioan-Lucian Deaconescu added 3 commits August 11, 2023 14:08

more explicit docs

751e0e3

slight rewording

ceaa2fb

fix link

1fa6019

zmoog approved these changes Aug 11, 2023

View reviewed changes

packages/aws/_dev/build/docs/apigateway.md Outdated Show resolved Hide resolved

lucian-ioan and others added 2 commits August 11, 2023 14:25

Update packages/aws/_dev/build/docs/apigateway.md

ffee5fa

Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

package build

5a1af9b

bhapas reviewed Aug 11, 2023

View reviewed changes

minor version bump

d5809c9

lucian-ioan requested a review from bhapas August 11, 2023 12:12

bhapas approved these changes Aug 11, 2023

View reviewed changes

gpop63 approved these changes Aug 11, 2023

View reviewed changes

lucian-ioan merged commit 03533c0 into elastic:main Aug 11, 2023

This was referenced Aug 24, 2023

[AWS] [API Gateway] Enhance logs dashboard with filtering #7526

Closed

[AWS] [API Gateway] Enhance logs dashboard with filtering #7541

Merged

lucian-ioan deleted the aws_apigateway_additional_fields branch September 6, 2023 11:36

lucian-ioan mentioned this pull request Sep 12, 2023

[awsfirehose] Add support for routing api gateway logs #7701

Merged

4 tasks

		@@ -1,4 +1,9 @@
		# newer versions go on top
		- version: "1.52.2"

[AWS] [API Gateway] Add custom access logging fields #7151

[AWS] [API Gateway] Add custom access logging fields #7151

Uh oh!

Conversation

lucian-ioan commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Urgency

Activity Type

What does this PR do?

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

elasticmachine commented Jul 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

Uh oh!

zmoog Jul 31, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

lucian-ioan commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cla-checker-service bot commented Aug 2, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zmoog commented Aug 3, 2023

Uh oh!

lucian-ioan commented Aug 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

zmoog commented Aug 3, 2023

Uh oh!

annamikhailovskaia commented Aug 9, 2023

Uh oh!

umatiw commented Aug 10, 2023

Uh oh!

umatiw commented Aug 10, 2023

Uh oh!

Uh oh!

zmoog left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bhapas Aug 11, 2023

Choose a reason for hiding this comment

Uh oh!

bhapas left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Aug 11, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

lucian-ioan commented Jul 26, 2023 •

edited

Loading

elasticmachine commented Jul 26, 2023 •

edited

Loading

elasticmachine commented Jul 26, 2023 •

edited

Loading

lucian-ioan commented Aug 2, 2023 •

edited

Loading

cla-checker-service bot commented Aug 2, 2023 •

edited

Loading

lucian-ioan commented Aug 3, 2023 •

edited

Loading