Skip to content

feat(lifecycle): Integrate cold tiering support #1050

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
tennisleng wants to merge 4 commits into
base: main
Choose a base branch
from
Open

feat(lifecycle): Integrate cold tiering support #1050

tennisleng wants to merge 4 commits into from

Conversation

@tennisleng
Copy link
Contributor

@tennisleng tennisleng commented Dec 8, 2025

Splitting #1042: Integration of cold tiering.

@tennisleng tennisleng mentioned this pull request Dec 8, 2025
Closed
@loverustfs
Copy link
Contributor

loverustfs commented Dec 8, 2025

Hey @tennisleng ,

Thank you for your contribution. Could you please provide a detailed design plan?

@tennisleng
Copy link
Contributor Author

tennisleng commented Dec 8, 2025

📋 Detailed Design Plan: Cold Tiering Integration

Thank you for the review! Here's the detailed design plan for this PR.

Overview

This PR implements cold tiering support for RustFS, enabling automatic migration of infrequently accessed objects to cheaper storage tiers based on configurable policies. This addresses issue #1022.

Architecture

┌────────────────────────────────────────────────────────────────┐
│                    Configuration Layer                          │
│  cold_tier_config.rs                                            │
│  ├─ ColdTierConfig (per-bucket settings)                        │
│  └─ GlobalColdTierConfig (system-wide defaults from env vars)   │
├────────────────────────────────────────────────────────────────┤
│                    Metadata Layer                                │
│  store_api.rs     │  fileinfo.rs    │  filemeta.rs              │
│  └─ ObjectInfo    │  └─ FileInfo    │  └─ FileMeta              │
│     +last_access  │     +last_access│     +last_access          │
├────────────────────────────────────────────────────────────────┤
│                    Lifecycle Layer                               │
│  lifecycle.rs                                                    │
│  ├─ ObjectOpts.access_time                                       │
│  ├─ ObjectOpts.is_cold() - determines if object should migrate  │
│  └─ effective_access_time() - fallback to mod_time if no access │
└────────────────────────────────────────────────────────────────┘

Key Components

1. Configuration (cold_tier_config.rs)

Setting Env Variable Default Description
enabled COLD_TIER_ENABLED false Enable/disable cold tiering
threshold_days COLD_TIER_THRESHOLD_DAYS 30 Days since last access before object is "cold"
target_tier COLD_TIER_TARGET GLACIER Target storage class
min_size_bytes COLD_TIER_MIN_SIZE 1MB Skip small objects
sample_rate COLD_TIER_SAMPLE_RATE 1.0 Fraction of objects to evaluate
concurrency COLD_TIER_CONCURRENCY 4 Parallel migration workers

2. Access Time Tracking

  • ObjectInfo.last_access_time: New field tracking when object was last accessed
  • FileInfo.last_access_time: Persisted in file metadata
  • Fallback: If last_access_time is None, uses mod_time instead

3. Lifecycle Integration

  • ObjectOpts.access_time: Exposes access time to lifecycle rules
  • ObjectOpts.is_cold(threshold): Returns true if now - access_time > threshold
  • effective_access_time(): Returns access_time or falls back to mod_time

Data Flow

GET Object Request
       │
       ▼
Update last_access_time in metadata
       │
       ▼
Lifecycle Scanner (periodic)
       │
       ▼
For each object: is_cold(threshold_days)?
       │
       ├─ No  → Skip
       └─ Yes → Transition to target_tier

Related PRs

This is Part 2 of splitting #1042. Part 1 (#1049) adds the core cold tier config module.

Testing Plan

  1. Unit tests for is_cold() with various thresholds
  2. Integration tests for access time updates on GET
  3. End-to-end test for lifecycle transition to cold tier

@tennisleng tennisleng closed this Dec 8, 2025
@tennisleng tennisleng reopened this Dec 8, 2025
@loverustfs
Copy link
Contributor

loverustfs commented Dec 9, 2025

Hey @tennisleng ,

Our lifecycle management still has bugs. Cold tier related issues need to be reviewed and merged after lifecycle management is complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tennisleng @loverustfs