Skip to content

Designs for Custom Agent (2 Scenarios) #11876

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
jpshackelford wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Designs for Custom Agent (2 Scenarios) #11876

jpshackelford wants to merge 4 commits into from
+1,770 −0

Conversation

@jpshackelford
Copy link
Contributor

@jpshackelford jpshackelford commented Dec 2, 2025 •
edited by github-actions bot
Loading

Problem Statement

OpenHands Enterprise (OHE) currently lacks a supported mechanism for users to deploy custom agents created with software-agent-sdk. While the software-agent-sdk provides the foundation for building custom agents, there is no clear path for users to deploy them in OHE.

This creates barriers for users who want to:

  • Customize agent behavior with specialized prompts and reasoning logic
  • Add domain-specific tools and capabilities
  • Integrate with custom APIs and external services
  • Deploy agents with different LLM configurations or workflows
  • Use agents that require specialized system dependencies or runtime environments

Proposed Solutions

This PR introduces comprehensive design documents for two complementary scenarios that address different custom agent deployment needs:

Scenario 1: Custom Agent Packages with Custom Runtime Images

File: doc/design-doc/custom-agent-s1.md

For users who need custom system dependencies, specialized tools, or non-Python runtime environments:

  • Approach: Custom Docker images extending the base agent server image
  • Use Cases: ML frameworks, specialized compilers, databases, non-Python tools
  • Benefits: Maximum flexibility and isolation
  • Trade-offs: Requires Docker image building and maintenance

Scenario 2: Dynamic Custom Agent Package Loading

File: doc/design-doc/custom-agent-s2.md

For users who need pure Python customization without additional system dependencies:

  • Approach: Dynamic package loading within existing agent server container
  • Use Cases: Custom Python tools, API integrations, specialized prompts
  • Benefits: Simple deployment, no Docker image required
  • Trade-offs: Limited to Python packages compatible with base image

Technical Foundation

Both designs are grounded in detailed analysis of:

  • V1 Architecture: Main server → HTTP API → Agent Server communication flow
  • Software Agent SDK: AgentBase interface and tool system from software-agent-sdk repository
  • Existing Integration Points: Sandbox service, container orchestration, environment configuration
  • Security Considerations: Package validation, isolation, and fallback mechanisms

Implementation Approach

Each design includes:

  • ✅ User Interface: Clear examples of how users create and deploy custom agents
  • ✅ Technical Design: Detailed architecture with code examples and integration points
  • ✅ Implementation Plan: 4-milestone approach with specific file paths and acceptance criteria
  • ✅ Security & Error Handling: Comprehensive validation and fallback mechanisms

Compatibility

Both scenarios maintain full compatibility with:

  • Existing V1 HTTP API (/ask_agent endpoint)
  • Current sandbox service and container orchestration
  • OpenHands tooling ecosystem and conversation management

Next Steps

These design documents provide the foundation for implementing custom agent support in OpenHands. The modular approach allows for independent implementation of either scenario based on user demand and development priorities.

Note: This PR contains design documents only. Implementation will follow in separate PRs based on the approved designs.

@jpshackelford can click here to continue refining the PR

To run this PR locally, use the following command:

GUI with Docker:

docker run -it --rm   -p 3000:3000   -v /var/run/docker.sock:/var/run/docker.sock   --add-host host.docker.internal:host-gateway   -e SANDBOX_RUNTIME_CONTAINER_IMAGE=docker.openhands.dev/openhands/runtime:635a29a-nikolaik   --name openhands-app-635a29a   docker.openhands.dev/openhands/openhands:635a29a

@openhands-agent
Add design document for Custom Agent Packages with Custom Runtime Ima…
39c2405 
…ges (Scenario 1)

- Comprehensive design for supporting custom agent packages that require custom system dependencies
- Extends V1 architecture to support custom Docker images with specialized tools and runtimes
- Maintains compatibility with existing HTTP API and tooling ecosystem
- Includes detailed technical design, implementation plan, and user interface specifications

Co-authored-by: openhands <openhands@all-hands.dev>
@openhands-agent
Add design document for Dynamic Custom Agent Package Loading (Scenari…
d9b6901 
…o 2)

- Comprehensive design for dynamic loading of custom agent packages within existing agent server container
- Supports Git repositories, PyPI packages, and ZIP archives without requiring custom Docker images
- Maintains full compatibility with existing V1 HTTP API and infrastructure
- Includes security validation, error handling, and fallback mechanisms
- Covers 4-milestone implementation plan with detailed technical specifications

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025 •
edited
Loading

Coverage report

This PR does not seem to contain any modification to coverable code.

@openhands-agent
Enhance Scenario 2 design doc with detailed V1 architecture explanation
9f04705 
- Add comprehensive explanation of current V1 agent instantiation and execution flow
- Detail how agents are currently created, managed, and executed in agent server containers
- Explain the distributed service model and communication patterns
- Clarify integration points and how dynamic loading would modify existing processes
- Provide better context for understanding the technical changes required

Co-authored-by: openhands <openhands@all-hands.dev>
@openhands-agent
Revise Scenario 1 design to align with engineer's sandbox spec approach
635a29a 
Major changes to align with existing OpenHands infrastructure:
- Use enhanced sandbox specifications instead of parallel CustomAgentSpec system
- Implement permissions as core M1 requirement (not M4 afterthought)
- Support both pre-built image registration and secure user upload workflows
- Extend existing SandboxSpecService rather than creating separate infrastructure
- Integrate with existing conversation creation API via sandbox_spec_id parameter
- Address V0 security issues with comprehensive validation and approval workflows
- Maintain backward compatibility with existing sandbox system

This approach is more pragmatic and leverages proven OpenHands infrastructure
while addressing the engineer's specific requirements for permissions and security.

Co-authored-by: openhands <openhands@all-hands.dev>
@jpshackelford jpshackelford mentioned this pull request Dec 2, 2025
Open
@openhands-ai
Copy link

openhands-ai bot commented Dec 2, 2025

Looks like there are a few issues preventing this PR from being merged!

  • GitHub Actions are failing:
    • Lint

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #11876 at branch `jps/custom-agent`

Feel free to include any additional details that might help me get this PR into a better state.

You can manage your notification settings

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jpshackelford @openhands-agent