Skip to content

Run soak tests from EC2 #1908

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Run soak tests from EC2 #1908

wants to merge 10 commits into from
+135 −429

Conversation

anchitj
Copy link
Contributor

@anchitj anchitj commented Jan 23, 2025 •
edited
Loading

What

Adds OTEL config which would be used by OTEL agent to export the metrics and logs of Soak tests to Datadog.

Instructions to install OTEL agent on EC2:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"
export OTEL_METRICS_EXPORTER=otlp
export OTEL_LOGS_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES=service.name=njc-py-soak-tests-normal-2-2-1,service.version=v2.2.1-RC1-dev_soaktest_k8s_on_2.2.1
export OTEL_TRACES_EXPORTER=none

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.117.0/otelcol-contrib_0.117.0_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.117.0_linux_amd64.deb
otelcol-contrib --config otel_config.yaml

This would start OTEL agent and tests would start exporting metrics and logs to it.

Other changes to run with EC2

export HOSTNAME=$(hostname)
PYTHONPATH=/home/ubuntu/confluent-kafka-python/tests opentelemetry-instrument confluent-kafka-python/tests/soak/soakclient.py -t $topic -r 80 -f confluent-kafka-python/ccloud.config -i t1
pip install psutil

Checklist

  • Contains customer facing changes? Including API/behavior changes
  • Did you add sufficient unit test and/or integration test coverage for this PR?
    • If not, please explain why it is not required

References

JIRA:

Test & Review

Open questions / Follow-ups

@confluent-cla-assistant
Copy link

confluent-cla-assistant bot commented Jan 23, 2025 •
edited
Loading

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ anchitj
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

@sonarqube-confluent

This comment has been minimized.

Sign in to view
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@airlock-confluentinc airlock-confluentinc bot force-pushed the dev_otel_agent_soak_test branch 2 times, most recently from 7ff3316 to 6b72e5e Compare February 28, 2025 11:11
@anchitj anchitj marked this pull request as ready for review February 28, 2025 11:12
@Copilot Copilot AI review requested due to automatic review settings February 28, 2025 11:12
@anchitj anchitj requested review from a team as code owners February 28, 2025 11:12
@anchitj anchitj changed the title Add OTEL config Run soak tests from EC2 Feb 28, 2025
Copilot
Copilot AI reviewed Feb 28, 2025
View reviewed changes
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Overview

This PR adds a configuration file for the OTEL agent to export metrics and logs from soak tests to Datadog.

  • Adds a new YAML file for OTEL configuration.
  • Configures receivers, processors, exporters, and service pipelines for both metrics and logs.

Reviewed Changes

File Description
tests/soak/otel-config.yaml Added OTEL configuration with receivers, processors, exporters, and service pipelines for Datadog integration

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

@sonarqube-confluent

This comment has been minimized.

Sign in to view
2 similar comments
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@airlock-confluentinc airlock-confluentinc bot force-pushed the dev_otel_agent_soak_test branch from 6b72e5e to ff78cfb Compare July 21, 2025 13:48
@airlock-confluentinc airlock-confluentinc bot requested a review from MSeal as a code owner July 21, 2025 13:48
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@airlock-confluentinc airlock-confluentinc bot force-pushed the dev_otel_agent_soak_test branch 2 times, most recently from 69967b4 to 1ca46af Compare July 22, 2025 13:42
@sonarqube-confluent

This comment has been minimized.

Sign in to view
1 similar comment
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@airlock-confluentinc airlock-confluentinc bot force-pushed the dev_otel_agent_soak_test branch from 1ca46af to 58c32c2 Compare July 22, 2025 14:07
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@emasab
Use a local librdkafka installation different for every started test
bd252ad 
Use test code from the checked out branch instead of the selected CKPy version
Bootstrap the same way as when changing version: with `build.sh`
@airlock-confluentinc airlock-confluentinc bot force-pushed the dev_otel_agent_soak_test branch from 58c32c2 to bd252ad Compare July 22, 2025 14:17
@sonarqube-confluent

This comment has been minimized.

Sign in to view
3 similar comments
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@sonarqube-confluent

This comment has been minimized.

Sign in to view
@sonarqube-confluent
Copy link

Passed

Analysis Details

1 Issue

  • Bug 0 Bugs
  • Vulnerability 0 Vulnerabilities
  • Code Smell 1 Code Smell

Coverage and Duplications

  • Coverage No coverage information (66.10% Estimated after merge)
  • Duplications No duplication information (5.50% Estimated after merge)

Project ID: confluent-kafka-python

View in SonarQube

@airlock-confluentinc airlock-confluentinc bot force-pushed the dev_otel_agent_soak_test branch from 5e79cd4 to bf3a2da Compare July 30, 2025 07:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@MSeal MSeal Awaiting requested review from MSeal MSeal is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@anchitj @emasab