-
Notifications
You must be signed in to change notification settings - Fork 1.1k
docs: add data retention documentation #21038
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+198
−2
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
209c70a
docs: add data retention documentation
mafredri d158959
fix: format markdown tables
mafredri 1b10d7e
docs: clarify heading - 'Keeping Data Indefinitely' instead of 'Disab…
mafredri f8cefc6
chore: remove global retention from documentation
mafredri File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| # Data Retention | ||
|
|
||
| Coder supports configurable retention policies that automatically purge old | ||
| Audit Logs, Connection Logs, and API keys. These policies help manage database | ||
| growth by removing records older than a specified duration. | ||
|
|
||
| ## Overview | ||
|
|
||
| Large deployments can accumulate significant amounts of data over time. | ||
| Retention policies help you: | ||
|
|
||
| - **Reduce database size**: Automatically remove old records to free disk space. | ||
| - **Improve performance**: Smaller tables mean faster queries and backups. | ||
| - **Meet compliance requirements**: Configure retention periods that align with | ||
| your organization's data retention policies. | ||
|
|
||
| > [!NOTE] | ||
| > Retention policies are disabled by default (set to `0`) to preserve existing | ||
| > behavior. The only exception is API keys, which defaults to 7 days. | ||
|
|
||
| ## Configuration | ||
|
|
||
| You can configure retention policies using CLI flags, environment variables, or | ||
| a YAML configuration file. | ||
|
|
||
| ### Settings | ||
|
|
||
| | Setting | CLI Flag | Environment Variable | Default | Description | | ||
| |-----------------|-------------------------------|-----------------------------------|----------------|--------------------------------------| | ||
| | Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (disabled) | How long to retain Audit Log entries | | ||
| | Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (disabled) | How long to retain Connection Logs | | ||
| | API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys | | ||
|
|
||
| ### Duration Format | ||
|
|
||
| Retention durations support days (`d`) and weeks (`w`) in addition to standard | ||
| Go duration units (`h`, `m`, `s`): | ||
|
|
||
| - `7d` - 7 days | ||
| - `2w` - 2 weeks | ||
| - `30d` - 30 days | ||
| - `90d` - 90 days | ||
| - `365d` - 1 year | ||
|
|
||
| ### CLI Example | ||
|
|
||
| ```bash | ||
| coder server \ | ||
| --audit-logs-retention=365d \ | ||
| --connection-logs-retention=90d \ | ||
| --api-keys-retention=7d | ||
| ``` | ||
|
|
||
| ### Environment Variables Example | ||
|
|
||
| ```bash | ||
| export CODER_AUDIT_LOGS_RETENTION=365d | ||
| export CODER_CONNECTION_LOGS_RETENTION=90d | ||
| export CODER_API_KEYS_RETENTION=7d | ||
| ``` | ||
|
|
||
| ### YAML Configuration Example | ||
|
|
||
| ```yaml | ||
| retention: | ||
| audit_logs: 365d | ||
| connection_logs: 90d | ||
| api_keys: 7d | ||
| ``` | ||
|
|
||
| ## How Retention Works | ||
|
|
||
| ### Background Purge Process | ||
|
|
||
| Coder runs a background process that periodically deletes old records. The | ||
| purge process: | ||
|
|
||
| 1. Runs approximately every 10 minutes. | ||
| 2. Processes records in batches to avoid database lock contention. | ||
| 3. Deletes records older than the configured retention period. | ||
| 4. Logs the number of deleted records for monitoring. | ||
|
|
||
| ### Effective Retention | ||
|
|
||
| Each retention setting controls its data type independently: | ||
|
|
||
| - When set to a non-zero duration, records older than that duration are deleted. | ||
| - When set to `0`, retention is disabled and data is kept indefinitely. | ||
|
|
||
| ### API Keys Special Behavior | ||
|
|
||
| API key retention only affects **expired** keys. A key is deleted only when: | ||
|
|
||
| 1. The key has expired (past its `expires_at` timestamp). | ||
| 2. The key has been expired for longer than the retention period. | ||
|
|
||
| Setting `--api-keys-retention=7d` deletes keys that expired more than 7 days | ||
| ago. Active keys are never deleted by the retention policy. | ||
|
|
||
| Keeping expired keys for a short period allows Coder to return a more helpful | ||
| error message when users attempt to use an expired key. | ||
|
|
||
| ## Best Practices | ||
|
|
||
| ### Recommended Starting Configuration | ||
|
|
||
| For most deployments, we recommend: | ||
|
|
||
| ```yaml | ||
| retention: | ||
| audit_logs: 365d | ||
| connection_logs: 90d | ||
| api_keys: 7d | ||
| ``` | ||
|
|
||
| ### Compliance Considerations | ||
|
|
||
| > [!WARNING] | ||
| > Audit Logs provide critical security and compliance information. Purging | ||
| > Audit Logs may impact your organization's ability to investigate security | ||
| > incidents or meet compliance requirements. Consult your security and | ||
| > compliance teams before configuring Audit Log retention. | ||
|
|
||
| Common compliance frameworks have varying retention requirements: | ||
|
|
||
| - **SOC 2**: Typically requires 1 year of audit logs. | ||
| - **HIPAA**: Requires 6 years for certain records. | ||
| - **PCI DSS**: Requires 1 year of audit logs, with 3 months immediately | ||
| available. | ||
| - **GDPR**: Requires data minimization but does not specify maximum retention. | ||
mafredri marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ### External Log Aggregation | ||
|
|
||
| If you use an external log aggregation system (Splunk, Datadog, etc.), you can | ||
| configure shorter retention periods in Coder since logs are preserved | ||
| externally. See | ||
| [Capturing/Exporting Audit Logs](../security/audit-logs.md#capturingexporting-audit-logs) | ||
| for details on exporting logs. | ||
|
|
||
| ### Database Maintenance | ||
|
|
||
| After enabling retention policies, you may want to run a `VACUUM` operation on | ||
| your PostgreSQL database to reclaim disk space. See | ||
| [Maintenance Procedures](../security/audit-logs.md#maintenance-procedures-for-the-audit-logs-table) | ||
| for guidance. | ||
|
|
||
| ## Keeping Data Indefinitely | ||
|
|
||
| To keep data indefinitely for any data type, set its retention value to `0`: | ||
|
|
||
| ```yaml | ||
| retention: | ||
| audit_logs: 0s # Keep audit logs forever | ||
| connection_logs: 0s # Keep connection logs forever | ||
| api_keys: 0s # Keep expired API keys forever | ||
| ``` | ||
|
|
||
| ## Monitoring | ||
|
|
||
| The purge process logs deletion counts at the `DEBUG` level. To monitor | ||
| retention activity, enable debug logging or search your logs for entries | ||
| containing the table name (e.g., `audit_logs`, `connection_logs`, `api_keys`). | ||
|
|
||
| ## Related Documentation | ||
|
|
||
| - [Audit Logs](../security/audit-logs.md): Learn about Audit Logs and manual | ||
| purge procedures. | ||
| - [Connection Logs](../monitoring/connection-logs.md): Learn about Connection | ||
| Logs and monitoring. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.