From 209c70ad5321f41929bd1c82b0a0417c0d8ef07e Mon Sep 17 00:00:00 2001 From: Mathias Fredriksson Date: Mon, 1 Dec 2025 15:40:00 +0000 Subject: [PATCH 1/4] docs: add data retention documentation Document configurable retention policies for Audit Logs, Connection Logs, and API keys. Add new data-retention.md page and update existing docs to reference it. Depends on #21021 Updates #20743 --- docs/admin/monitoring/connection-logs.md | 8 + docs/admin/monitoring/index.md | 1 + docs/admin/security/audit-logs.md | 17 ++- docs/admin/setup/data-retention.md | 180 +++++++++++++++++++++++ docs/manifest.json | 5 + 5 files changed, 209 insertions(+), 2 deletions(-) create mode 100644 docs/admin/setup/data-retention.md diff --git a/docs/admin/monitoring/connection-logs.md b/docs/admin/monitoring/connection-logs.md index b69bb2db186a8..210ca76d740cf 100644 --- a/docs/admin/monitoring/connection-logs.md +++ b/docs/admin/monitoring/connection-logs.md @@ -106,6 +106,14 @@ connection log entry, when `code-server` is opened: [API] 2025-07-03 06:57:16.157 [info] coderd: connection_log request_id=de3f6004-6cc1-4880-a296-d7c6ca1abf75 ID=f0249951-d454-48f6-9504-e73340fa07b7 Time="2025-07-03T06:57:16.144719Z" OrganizationID=0665a54f-0b77-4a58-94aa-59646fa38a74 WorkspaceOwnerID=6dea5f8c-ecec-4cf0-a5bd-bc2c63af2efa WorkspaceID=3c0b37c8-e58c-4980-b9a1-2732410480a5 WorkspaceName=dev AgentName=main Type=workspace_app Code=200 Ip=127.0.0.1 UserAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" UserID=6dea5f8c-ecec-4cf0-a5bd-bc2c63af2efa SlugOrPort=code-server ConnectionID= DisconnectReason="" ConnectionStatus=connected ``` +## Data Retention + +Coder supports configurable retention policies that automatically purge old +Connection Logs. To enable automated purging, configure the +`--connection-logs-retention` flag or `CODER_CONNECTION_LOGS_RETENTION` +environment variable. For comprehensive configuration options, see +[Data Retention](../setup/data-retention.md). + ## How to Enable Connection Logs This feature is only available with a [Premium license](../licensing/index.md). diff --git a/docs/admin/monitoring/index.md b/docs/admin/monitoring/index.md index 996d8040b0129..61e27e7930607 100644 --- a/docs/admin/monitoring/index.md +++ b/docs/admin/monitoring/index.md @@ -22,3 +22,4 @@ Learn how to install & read the docs on the Coder deployment, regardless of your monitoring stack. - [Health Check](./health-check.md): Learn about the periodic health check and error codes that run on Coder deployments. +- [Connection Logs](./connection-logs.md): Monitor connections to workspaces. diff --git a/docs/admin/security/audit-logs.md b/docs/admin/security/audit-logs.md index 29e9fe7b6f6c8..0d43a05e3ee1f 100644 --- a/docs/admin/security/audit-logs.md +++ b/docs/admin/security/audit-logs.md @@ -132,8 +132,21 @@ log entry: > Audit Logs provide critical security and compliance information. Purging Audit Logs may impact your organization's ability > to investigate security incidents or meet compliance requirements. Consult your security and compliance teams before purging any audit data. -Audit Logs are not automatically purged from the database, though they can account for a large amount of disk usage. -Use the following query to determine the amount of disk space used by the `audit_logs` table. +### Automated Retention + +Coder supports configurable retention policies that automatically purge old +Audit Logs. To enable automated purging, configure the +`--audit-logs-retention` flag or `CODER_AUDIT_LOGS_RETENTION` environment +variable. For comprehensive configuration options, see +[Data Retention](../setup/data-retention.md). + +### Manual Purging + +Alternatively, you can purge Audit Logs manually by running SQL queries +directly against the database. + +Audit Logs can account for a large amount of disk usage. Use the following +query to determine the amount of disk space used by the `audit_logs` table. ```sql SELECT diff --git a/docs/admin/setup/data-retention.md b/docs/admin/setup/data-retention.md new file mode 100644 index 0000000000000..7e1f40d675438 --- /dev/null +++ b/docs/admin/setup/data-retention.md @@ -0,0 +1,180 @@ +# Data Retention + +Coder supports configurable retention policies that automatically purge old +Audit Logs, Connection Logs, and API keys. These policies help manage database +growth by removing records older than a specified duration. + +## Overview + +Large deployments can accumulate significant amounts of data over time. +Retention policies help you: + +- **Reduce database size**: Automatically remove old records to free disk space. +- **Improve performance**: Smaller tables mean faster queries and backups. +- **Meet compliance requirements**: Configure retention periods that align with + your organization's data retention policies. + +> [!NOTE] +> Retention policies are disabled by default (set to `0`) to preserve existing +> behavior. The only exception is API keys, which defaults to 7 days. + +## Configuration + +You can configure retention policies using CLI flags, environment variables, or +a YAML configuration file. + +### Settings + +| Setting | CLI Flag | Environment Variable | Default | Description | +| --------------- | ----------------------------- | --------------------------------- | ---------------- | ------------------------------------------------------------------------- | +| Global | `--global-retention` | `CODER_GLOBAL_RETENTION` | `0` (disabled) | Default retention for all data types. Individual settings override this. | +| Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (use global) | How long to retain Audit Log entries. | +| Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (use global) | How long to retain Connection Log entries. | +| API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys. | + +### Duration Format + +Retention durations support days (`d`) and weeks (`w`) in addition to standard +Go duration units (`h`, `m`, `s`): + +- `7d` - 7 days +- `2w` - 2 weeks +- `30d` - 30 days +- `90d` - 90 days +- `365d` - 1 year + +### CLI Example + +```bash +coder server \ + --global-retention=90d \ + --audit-logs-retention=365d \ + --api-keys-retention=7d +``` + +### Environment Variables Example + +```bash +export CODER_GLOBAL_RETENTION=90d +export CODER_AUDIT_LOGS_RETENTION=365d +export CODER_API_KEYS_RETENTION=7d +``` + +### YAML Configuration Example + +```yaml +retention: + global: 90d + audit_logs: 365d + connection_logs: 0s + api_keys: 7d +``` + +## How Retention Works + +### Background Purge Process + +Coder runs a background process that periodically deletes old records. The +purge process: + +1. Runs approximately every 10 minutes. +2. Processes records in batches to avoid database lock contention. +3. Deletes records older than the configured retention period. +4. Logs the number of deleted records for monitoring. + +### Effective Retention + +For each data type, the effective retention is determined as follows: + +1. If the individual setting is non-zero, use that value. +2. If the individual setting is zero, use the global retention value. +3. If both are zero, retention is disabled (data is kept indefinitely). + +### API Keys Special Behavior + +API key retention only affects **expired** keys. A key is deleted only when: + +1. The key has expired (past its `expires_at` timestamp). +2. The key has been expired for longer than the retention period. + +Setting `--api-keys-retention=7d` deletes keys that expired more than 7 days +ago. Active keys are never deleted by the retention policy. + +Keeping expired keys for a short period allows Coder to return a more helpful +error message when users attempt to use an expired key. + +## Best Practices + +### Recommended Starting Configuration + +For most deployments, we recommend: + +```yaml +retention: + global: 90d + audit_logs: 365d + connection_logs: 0s # Use global + api_keys: 7d +``` + +### Compliance Considerations + +> [!WARNING] +> Audit Logs provide critical security and compliance information. Purging +> Audit Logs may impact your organization's ability to investigate security +> incidents or meet compliance requirements. Consult your security and +> compliance teams before configuring Audit Log retention. + +Common compliance frameworks have varying retention requirements: + +- **SOC 2**: Typically requires 1 year of audit logs. +- **HIPAA**: Requires 6 years for certain records. +- **PCI DSS**: Requires 1 year of audit logs, with 3 months immediately + available. +- **GDPR**: Requires data minimization but does not specify maximum retention. + +### External Log Aggregation + +If you use an external log aggregation system (Splunk, Datadog, etc.), you can +configure shorter retention periods in Coder since logs are preserved +externally. See +[Capturing/Exporting Audit Logs](../security/audit-logs.md#capturingexporting-audit-logs) +for details on exporting logs. + +### Database Maintenance + +After enabling retention policies, you may want to run a `VACUUM` operation on +your PostgreSQL database to reclaim disk space. See +[Maintenance Procedures](../security/audit-logs.md#maintenance-procedures-for-the-audit-logs-table) +for guidance. + +## Disabling Retention + +Setting a retention value to `0` means "use global retention", not "disable". +To disable all automatic purging, set global to `0` and leave individual +settings at `0`: + +```yaml +retention: + global: 0s + audit_logs: 0s + connection_logs: 0s + api_keys: 0s +``` + +There is no way to disable retention for a specific data type while global +retention is enabled. If you need to retain one data type longer than others, +set its individual retention to a larger value. + +## Monitoring + +The purge process logs deletion counts at the `DEBUG` level. To monitor +retention activity, enable debug logging or search your logs for entries +containing the table name (e.g., `audit_logs`, `connection_logs`, `api_keys`). + +## Related Documentation + +- [Audit Logs](../security/audit-logs.md): Learn about Audit Logs and manual + purge procedures. +- [Connection Logs](../monitoring/connection-logs.md): Learn about Connection + Logs and monitoring. diff --git a/docs/manifest.json b/docs/manifest.json index 8abfe48cf1c41..0ac633576f9c9 100644 --- a/docs/manifest.json +++ b/docs/manifest.json @@ -363,6 +363,11 @@ "title": "Telemetry", "description": "Learn what usage telemetry Coder collects", "path": "./admin/setup/telemetry.md" + }, + { + "title": "Data Retention", + "description": "Configure data retention policies for database tables", + "path": "./admin/setup/data-retention.md" } ] }, From d1589598c60ee1c77e4c32e1fbcd8754629f5caa Mon Sep 17 00:00:00 2001 From: Mathias Fredriksson Date: Mon, 1 Dec 2025 15:57:39 +0000 Subject: [PATCH 2/4] fix: format markdown tables --- docs/admin/setup/data-retention.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/admin/setup/data-retention.md b/docs/admin/setup/data-retention.md index 7e1f40d675438..94cb739429b3e 100644 --- a/docs/admin/setup/data-retention.md +++ b/docs/admin/setup/data-retention.md @@ -25,12 +25,12 @@ a YAML configuration file. ### Settings -| Setting | CLI Flag | Environment Variable | Default | Description | -| --------------- | ----------------------------- | --------------------------------- | ---------------- | ------------------------------------------------------------------------- | -| Global | `--global-retention` | `CODER_GLOBAL_RETENTION` | `0` (disabled) | Default retention for all data types. Individual settings override this. | -| Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (use global) | How long to retain Audit Log entries. | -| Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (use global) | How long to retain Connection Log entries. | -| API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys. | +| Setting | CLI Flag | Environment Variable | Default | Description | +|-----------------|-------------------------------|-----------------------------------|------------------|--------------------------------------------------------------------------| +| Global | `--global-retention` | `CODER_GLOBAL_RETENTION` | `0` (disabled) | Default retention for all data types. Individual settings override this. | +| Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (use global) | How long to retain Audit Log entries. | +| Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (use global) | How long to retain Connection Log entries. | +| API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys. | ### Duration Format From 1b10d7e9d4decd549839cfedd7cd9b286d105ce5 Mon Sep 17 00:00:00 2001 From: Mathias Fredriksson Date: Mon, 1 Dec 2025 16:23:44 +0000 Subject: [PATCH 3/4] docs: clarify heading - 'Keeping Data Indefinitely' instead of 'Disabling Retention' --- docs/admin/setup/data-retention.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/admin/setup/data-retention.md b/docs/admin/setup/data-retention.md index 94cb739429b3e..d892d048e9da1 100644 --- a/docs/admin/setup/data-retention.md +++ b/docs/admin/setup/data-retention.md @@ -148,7 +148,7 @@ your PostgreSQL database to reclaim disk space. See [Maintenance Procedures](../security/audit-logs.md#maintenance-procedures-for-the-audit-logs-table) for guidance. -## Disabling Retention +## Keeping Data Indefinitely Setting a retention value to `0` means "use global retention", not "disable". To disable all automatic purging, set global to `0` and leave individual From f8cefc6e9db7bdc647546676fa346eb397dbd918 Mon Sep 17 00:00:00 2001 From: Mathias Fredriksson Date: Tue, 2 Dec 2025 10:08:17 +0000 Subject: [PATCH 4/4] chore: remove global retention from documentation Rewrite documentation to reflect that each retention setting is independent. Set to a non-zero duration to enable, or 0 to keep data indefinitely. --- docs/admin/security/audit-logs.md | 2 +- docs/admin/setup/data-retention.md | 43 +++++++++++------------------- 2 files changed, 17 insertions(+), 28 deletions(-) diff --git a/docs/admin/security/audit-logs.md b/docs/admin/security/audit-logs.md index 0d43a05e3ee1f..913611af283df 100644 --- a/docs/admin/security/audit-logs.md +++ b/docs/admin/security/audit-logs.md @@ -132,7 +132,7 @@ log entry: > Audit Logs provide critical security and compliance information. Purging Audit Logs may impact your organization's ability > to investigate security incidents or meet compliance requirements. Consult your security and compliance teams before purging any audit data. -### Automated Retention +### Data Retention Coder supports configurable retention policies that automatically purge old Audit Logs. To enable automated purging, configure the diff --git a/docs/admin/setup/data-retention.md b/docs/admin/setup/data-retention.md index d892d048e9da1..68409ad93f866 100644 --- a/docs/admin/setup/data-retention.md +++ b/docs/admin/setup/data-retention.md @@ -25,12 +25,11 @@ a YAML configuration file. ### Settings -| Setting | CLI Flag | Environment Variable | Default | Description | -|-----------------|-------------------------------|-----------------------------------|------------------|--------------------------------------------------------------------------| -| Global | `--global-retention` | `CODER_GLOBAL_RETENTION` | `0` (disabled) | Default retention for all data types. Individual settings override this. | -| Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (use global) | How long to retain Audit Log entries. | -| Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (use global) | How long to retain Connection Log entries. | -| API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys. | +| Setting | CLI Flag | Environment Variable | Default | Description | +|-----------------|-------------------------------|-----------------------------------|----------------|--------------------------------------| +| Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (disabled) | How long to retain Audit Log entries | +| Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (disabled) | How long to retain Connection Logs | +| API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys | ### Duration Format @@ -47,16 +46,16 @@ Go duration units (`h`, `m`, `s`): ```bash coder server \ - --global-retention=90d \ --audit-logs-retention=365d \ + --connection-logs-retention=90d \ --api-keys-retention=7d ``` ### Environment Variables Example ```bash -export CODER_GLOBAL_RETENTION=90d export CODER_AUDIT_LOGS_RETENTION=365d +export CODER_CONNECTION_LOGS_RETENTION=90d export CODER_API_KEYS_RETENTION=7d ``` @@ -64,9 +63,8 @@ export CODER_API_KEYS_RETENTION=7d ```yaml retention: - global: 90d audit_logs: 365d - connection_logs: 0s + connection_logs: 90d api_keys: 7d ``` @@ -84,11 +82,10 @@ purge process: ### Effective Retention -For each data type, the effective retention is determined as follows: +Each retention setting controls its data type independently: -1. If the individual setting is non-zero, use that value. -2. If the individual setting is zero, use the global retention value. -3. If both are zero, retention is disabled (data is kept indefinitely). +- When set to a non-zero duration, records older than that duration are deleted. +- When set to `0`, retention is disabled and data is kept indefinitely. ### API Keys Special Behavior @@ -111,9 +108,8 @@ For most deployments, we recommend: ```yaml retention: - global: 90d audit_logs: 365d - connection_logs: 0s # Use global + connection_logs: 90d api_keys: 7d ``` @@ -150,22 +146,15 @@ for guidance. ## Keeping Data Indefinitely -Setting a retention value to `0` means "use global retention", not "disable". -To disable all automatic purging, set global to `0` and leave individual -settings at `0`: +To keep data indefinitely for any data type, set its retention value to `0`: ```yaml retention: - global: 0s - audit_logs: 0s - connection_logs: 0s - api_keys: 0s + audit_logs: 0s # Keep audit logs forever + connection_logs: 0s # Keep connection logs forever + api_keys: 0s # Keep expired API keys forever ``` -There is no way to disable retention for a specific data type while global -retention is enabled. If you need to retain one data type longer than others, -set its individual retention to a larger value. - ## Monitoring The purge process logs deletion counts at the `DEBUG` level. To monitor