Skip to content

Conversation

@groldan
Copy link
Member

@groldan groldan commented Oct 30, 2025

Refactor extension assembly to per-module descriptors

Refactors the extension assembly build system to match the pattern used by community modules, where each extension manages its own assembly descriptor instead of having centralized descriptors in src/release/ext-*.xml.

Motivation

  • Improves maintainability by keeping assembly configuration with the module code
  • Makes extensions self-contained and easier to understand
  • Follows the same pattern as community modules for consistency
  • Allows independent management of packaging requirements per extension
  • Simplifies adding new extensions - no need to modify central files

Changes

Per-Module Assembly Descriptors

  • Removed 52 centralized assembly descriptors from src/release/ext-*.xml
  • Removed assembly descriptor references from src/pom.xml maven-assembly-plugin configuration
  • Created 52 per-module assembly descriptors at src/assembly/assembly.xml
  • Created dedicated -assembly coordinator modules for multi-module extensions:
    • csw-assembly, csw-iso-assembly, dxf-assembly
    • excel-wfs-assembly, excel-wps-assembly
    • importer-assembly, ogcapi-features-assembly
    • wps-assembly, wps-cluster-hazelcast-assembly
  • Created pseudo-modules for GeoTools-only extensions (iau, pyramid)

Assembly Profile

Added assembly profile to src/extension/pom.xml:

  • maven-antrun-plugin: Checks for src/assembly/assembly.xml existence, skips if not present
  • maven-dependency-plugin: Copies module dependencies to target/dependency/
  • maven-assembly-plugin: Creates ZIP packages using per-module descriptors
  • Output location: src/target/release/ (same as before for compatibility with release scripts)

Assembly Descriptor Structure

Each assembly descriptor includes:

  • Module JAR from target/ directory (via first fileSet)
  • Dependencies from target/dependency/ (via second fileSet)
  • License files from ${geoserverBaseDir}/release/target/html/licenses
  • README and LICENSE files (markdown and HTML versions)

dependencySets Migration Support

Implemented infrastructure for future migration to <dependencySets> approach:

  • Added assembly.useDependencySets property check in assembly profiles
  • When set to true, maven-dependency-plugin skips copying dependencies
  • Saves ~7GB disk space (extensions) and ~9GB (community) when used
  • Available in both src/extension/pom.xml and src/community/pom.xml
  • Reference implementation: gwc-gcs-blob community module
  • Migration guide documented in src/extension/README.md

Documentation

  • Created src/extension/README.md with:
    • Build instructions (2-step process)
    • Assembly configuration explanation
    • Extension types (single-module, multi-module, pseudo-modules)
    • Assembly descriptor structure and conventions
    • dependencySets migration guide for future optimization
    • Instructions for adding new extensions
  • Updated src/release/README.md with new build process
  • Updated doc/en/developer/source/policies/community-modules.rst:
    • Changed reference from src/release/ext-h2.xml to src/extension/h2/src/assembly/assembly.xml
    • Updated instructions for adding new extensions

Build Instructions

Building extensions now requires 2 steps:

cd src
# Step 1: Build and install all modules to local Maven repo
mvn clean install -Prelease -DskipTests

# Step 2: Package extensions into ZIP files (NO clean)
cd extension
mvn install -Prelease,assembly -DskipTests

Important: Do NOT use clean in step 2 - it would wipe out compiled classes needed for assembly.

Extension ZIP files are created in src/target/release/ (same location as before).

Status

  • All 52 extension assemblies have per-module descriptors
  • Assembly descriptors follow consistent pattern with two fileSets (target/ and target/dependency/)
  • Output location remains src/target/release/ for compatibility with existing release scripts
  • Verification pending: Need to build all extensions and compare against main branch artifacts to ensure identical ZIPs

Future Improvements

Extensions can gradually migrate to <dependencySets> approach for:

  • Significant disk space savings: ~7GB (extensions) + ~9GB (community) = ~16GB total
  • Faster builds: No dependency copying step
  • Simpler descriptors: ~20 lines vs ~40-50 lines per assembly.xml
  • Auto-correct dependencies: Resolved directly from pom.xml, no manual JAR patterns

See src/extension/README.md for migration guide and example.

Assembly Descriptor Migration Map

The following table shows where each centralized assembly descriptor was relocated:

Old Location New Location Notes
src/release/ext-app-schema.xml src/extension/app-schema/app-schema-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-authkey.xml src/extension/authkey/src/assembly/assembly.xml
src/release/ext-cas.xml src/extension/security/cas/src/assembly/assembly.xml
src/release/ext-charts.xml src/extension/charts/src/assembly/assembly.xml
src/release/ext-control-flow.xml src/extension/control-flow/src/assembly/assembly.xml
src/release/ext-css.xml src/extension/css/src/assembly/assembly.xml
src/release/ext-csw-iso.xml src/extension/csw/csw-iso-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-csw.xml src/extension/csw/csw-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-db2.xml src/extension/db2/src/assembly/assembly.xml
src/release/ext-dxf.xml src/extension/dxf/dxf-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-excel.xml src/extension/excel-wfs/src/assembly/assembly.xml Split into excel-wfs
src/release/ext-feature-pregeneralized.xml src/extension/feature-pregeneralized/src/assembly/assembly.xml
src/release/ext-gdal.xml src/extension/gdal/src/assembly/assembly.xml Pseudo-module
src/release/ext-geofence.xml src/extension/geofence/geofence-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-geopkg-output.xml src/extension/geopkg-output/geopkg-output-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-grib.xml src/extension/grib/src/assembly/assembly.xml
src/release/ext-gwc-s3.xml src/extension/gwc-s3/src/assembly/assembly.xml
src/release/ext-h2.xml src/extension/h2/src/assembly/assembly.xml
src/release/ext-iau.xml src/extension/iau/src/assembly/assembly.xml Pseudo-module (new)
src/release/ext-importer.xml src/extension/importer/importer-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-inspire.xml src/extension/inspire/src/assembly/assembly.xml
src/release/ext-jp2k.xml src/extension/jp2k/src/assembly/assembly.xml
src/release/ext-libjpeg-turbo.xml src/extension/libjpeg-turbo/src/assembly/assembly.xml
src/release/ext-mapml.xml src/extension/mapml/src/assembly/assembly.xml
src/release/ext-mbstyle.xml src/extension/mbstyle/src/assembly/assembly.xml
src/release/ext-metadata.xml src/extension/metadata/src/assembly/assembly.xml
src/release/ext-mongodb.xml src/extension/mongodb/src/assembly/assembly.xml
src/release/ext-monitor.xml src/extension/monitor/monitor-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-mysql.xml src/extension/mysql/src/assembly/assembly.xml
src/release/ext-netcdf-out.xml src/extension/netcdf-out/src/assembly/assembly.xml
src/release/ext-netcdf.xml src/extension/netcdf/src/assembly/assembly.xml
src/release/ext-ogcapi-features.xml src/extension/ogcapi/ogcapi-features-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-ogr-wfs.xml src/extension/ogr/ogr-wfs/src/assembly/assembly.xml
src/release/ext-ogr-wps.xml src/extension/ogr/ogr-wps/src/assembly/assembly.xml
src/release/ext-oracle.xml src/extension/oracle/src/assembly/assembly.xml
src/release/ext-params-extractor.xml src/extension/params-extractor/src/assembly/assembly.xml
src/release/ext-printing.xml src/extension/printing/src/assembly/assembly.xml
src/release/ext-pyramid.xml src/extension/pyramid/src/assembly/assembly.xml Pseudo-module (new)
src/release/ext-querylayer.xml src/extension/querylayer/src/assembly/assembly.xml
src/release/ext-rat.xml src/extension/rat/src/assembly/assembly.xml
src/release/ext-sldservice.xml src/extension/sldService/src/assembly/assembly.xml
src/release/ext-sqlserver.xml src/extension/sqlserver/src/assembly/assembly.xml
src/release/ext-vectortiles.xml src/extension/vectortiles/src/assembly/assembly.xml
src/release/ext-wcs2_0-eo.xml src/extension/wcs2_0-eo/wcs2_0-eo-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-web-resource.xml src/extension/web-resource/src/assembly/assembly.xml
src/release/ext-wmts-multi-dimensional.xml src/extension/wmts-multi-dimensional/src/assembly/assembly.xml
src/release/ext-wps-download.xml src/extension/wps-download/src/assembly/assembly.xml
src/release/ext-wps-excel.xml src/extension/excel-wps/src/assembly/assembly.xml
src/release/ext-wps-hazelcast.xml src/extension/wps/wps-cluster-hazelcast/src/assembly/assembly.xml
src/release/ext-wps-jdbc.xml src/extension/wps-jdbc/src/assembly/assembly.xml
src/release/ext-wps.xml src/extension/wps/wps-assembly/src/assembly/assembly.xml Multi-module
src/release/ext-ysld.xml src/extension/ysld/src/assembly/assembly.xml

Note: The following centralized descriptors from main branch are not in the extension module (may be in community or removed):

  • src/release/ext-authbasic.xml
  • src/release/ext-geofence-server-h2.xml
  • src/release/ext-geofence-server-postgres.xml
  • src/release/ext-geofence-wps.xml

Checklist

For core and extension modules:

  • New unit tests have been added covering the changes.
  • Documentation has been updated (if change is visible to end users).
  • The REST API docs have been updated (when changing configuration objects or the REST controllers).
  • There is an issue in the GeoServer Jira (except for changes that do not affect administrators or end users in any way).
  • Commit message(s) must be in the form [GEOS-XYZWV] Title of the Jira ticket.
  • Bug fixes and small new features are presented as a single commit.
  • Each commit has a single objective (if there are multiple commits, each has a separate JIRA ticket describing its goal).

The PR will be merged when all the build checks are green (see automated QA checks), there is a code committer review, and the checklist has been fulfilled.

@groldan groldan force-pushed the gs3/extensions_assembly_per_module branch 6 times, most recently from 9a131be to 1068a32 Compare October 31, 2025 11:50
@aaime
Copy link
Member

aaime commented Nov 10, 2025

So overall it looks good, but we definitely need to check that the list of files is the same, zip by zip (minus some extraneous file that might have been caught by the old assembly approach, it happens).

What worries me is the current disk situation on the build server, we only have a few GB left.
To improve it, I added this script post-build in the nightly builds:

# clean up target/dependency jars, they use too much disk space
find . -path "*target/dependency" -type d -exec rm -r {} \;

With this we should normally have around 30-50GB of free disk space... not super but viable.

I do appreciated that you have a path to save disk space, but I'm wondering, can we make maven remove the dependencies directory right after assembling the target zip file for example? This would avoid the workaround of finding/deleting dependency directories.

@aaime
Copy link
Member

aaime commented Nov 17, 2025

Checking server disk space, it's now hovering at around 33GB. Should be enough to keep this one build going with the extra 7GB of disk space (if it lands on main only), the more it gets backported, the less space we have to play with.

Refactors the GeoServer extension assembly system from centralized descriptors
in src/release/ to per-module descriptors, matching the pattern used by community
modules. This improves modularity and maintainability by co-locating assembly
configuration with module code.

Changes:
- Removed 56 centralized assembly descriptors from src/release/ext-*.xml
- Created per-module descriptors at src/assembly/assembly.xml in each extension
- Added assembly profile to src/extension/pom.xml with maven-antrun-plugin,
  maven-dependency-plugin, and maven-assembly-plugin
- Created -assembly coordinator modules for multi-module extensions
- Created pseudo-modules for GeoTools-only extensions (iau, pyramid)
- Assembly output location remains src/target/release/ for compatibility
- Implemented skip dependency copy mechanism for modules using <dependencySets>
  (set assembly.useDependencySets=true property to save ~7GB disk space)
- Created src/extension/README.md with build instructions and migration guide
- Updated documentation references to per-module assembly descriptors

Build instructions: see src/extension/README.md
@groldan groldan force-pushed the gs3/extensions_assembly_per_module branch from 1068a32 to b3c5dfe Compare December 8, 2025 20:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants