Skip to content

Commit d9888ce

Browse files
mafredrimafredri
authored
docs: add data retention documentation (#21038)
Document configurable retention policies for Audit Logs, Connection Logs, and API keys. Add new data-retention.md page and update existing docs to reference it. Depends on #21021 Updates #20743
1 parent 9ec90cf commit d9888ce

File tree

5 files changed

+198
-2
lines changed

5 files changed

+198
-2
lines changed

‎docs/admin/monitoring/connection-logs.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,14 @@ connection log entry, when `code-server` is opened:
106106
[API] 2025-07-03 06:57:16.157 [info] coderd: connection_log request_id=de3f6004-6cc1-4880-a296-d7c6ca1abf75 ID=f0249951-d454-48f6-9504-e73340fa07b7 Time="2025-07-03T06:57:16.144719Z" OrganizationID=0665a54f-0b77-4a58-94aa-59646fa38a74 WorkspaceOwnerID=6dea5f8c-ecec-4cf0-a5bd-bc2c63af2efa WorkspaceID=3c0b37c8-e58c-4980-b9a1-2732410480a5 WorkspaceName=dev AgentName=main Type=workspace_app Code=200 Ip=127.0.0.1 UserAgent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36" UserID=6dea5f8c-ecec-4cf0-a5bd-bc2c63af2efa SlugOrPort=code-server ConnectionID=<nil> DisconnectReason="" ConnectionStatus=connected
107107
```
108108

109+
## Data Retention
110+
111+
Coder supports configurable retention policies that automatically purge old
112+
Connection Logs. To enable automated purging, configure the
113+
`--connection-logs-retention` flag or `CODER_CONNECTION_LOGS_RETENTION`
114+
environment variable. For comprehensive configuration options, see
115+
[Data Retention](../setup/data-retention.md).
116+
109117
## How to Enable Connection Logs
110118

111119
This feature is only available with a [Premium license](../licensing/index.md).

‎docs/admin/monitoring/index.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,3 +22,4 @@ Learn how to install & read the docs on the
2222
Coder deployment, regardless of your monitoring stack.
2323
- [Health Check](./health-check.md): Learn about the periodic health check and
2424
error codes that run on Coder deployments.
25+
- [Connection Logs](./connection-logs.md): Monitor connections to workspaces.

‎docs/admin/security/audit-logs.md‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -132,8 +132,21 @@ log entry:
132132
> Audit Logs provide critical security and compliance information. Purging Audit Logs may impact your organization's ability
133133
> to investigate security incidents or meet compliance requirements. Consult your security and compliance teams before purging any audit data.
134134
135-
Audit Logs are not automatically purged from the database, though they can account for a large amount of disk usage.
136-
Use the following query to determine the amount of disk space used by the `audit_logs` table.
135+
### Data Retention
136+
137+
Coder supports configurable retention policies that automatically purge old
138+
Audit Logs. To enable automated purging, configure the
139+
`--audit-logs-retention` flag or `CODER_AUDIT_LOGS_RETENTION` environment
140+
variable. For comprehensive configuration options, see
141+
[Data Retention](../setup/data-retention.md).
142+
143+
### Manual Purging
144+
145+
Alternatively, you can purge Audit Logs manually by running SQL queries
146+
directly against the database.
147+
148+
Audit Logs can account for a large amount of disk usage. Use the following
149+
query to determine the amount of disk space used by the `audit_logs` table.
137150

138151
```sql
139152
SELECT

‎docs/admin/setup/data-retention.md‎

Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Data Retention
2+
3+
Coder supports configurable retention policies that automatically purge old
4+
Audit Logs, Connection Logs, and API keys. These policies help manage database
5+
growth by removing records older than a specified duration.
6+
7+
## Overview
8+
9+
Large deployments can accumulate significant amounts of data over time.
10+
Retention policies help you:
11+
12+
- **Reduce database size**: Automatically remove old records to free disk space.
13+
- **Improve performance**: Smaller tables mean faster queries and backups.
14+
- **Meet compliance requirements**: Configure retention periods that align with
15+
your organization's data retention policies.
16+
17+
> [!NOTE]
18+
> Retention policies are disabled by default (set to `0`) to preserve existing
19+
> behavior. The only exception is API keys, which defaults to 7 days.
20+
21+
## Configuration
22+
23+
You can configure retention policies using CLI flags, environment variables, or
24+
a YAML configuration file.
25+
26+
### Settings
27+
28+
| Setting | CLI Flag | Environment Variable | Default | Description |
29+
|-----------------|-------------------------------|-----------------------------------|----------------|--------------------------------------|
30+
| Audit Logs | `--audit-logs-retention` | `CODER_AUDIT_LOGS_RETENTION` | `0` (disabled) | How long to retain Audit Log entries |
31+
| Connection Logs | `--connection-logs-retention` | `CODER_CONNECTION_LOGS_RETENTION` | `0` (disabled) | How long to retain Connection Logs |
32+
| API Keys | `--api-keys-retention` | `CODER_API_KEYS_RETENTION` | `7d` | How long to retain expired API keys |
33+
34+
### Duration Format
35+
36+
Retention durations support days (`d`) and weeks (`w`) in addition to standard
37+
Go duration units (`h`, `m`, `s`):
38+
39+
- `7d` - 7 days
40+
- `2w` - 2 weeks
41+
- `30d` - 30 days
42+
- `90d` - 90 days
43+
- `365d` - 1 year
44+
45+
### CLI Example
46+
47+
```bash
48+
coder server \
49+
--audit-logs-retention=365d \
50+
--connection-logs-retention=90d \
51+
--api-keys-retention=7d
52+
```
53+
54+
### Environment Variables Example
55+
56+
```bash
57+
export CODER_AUDIT_LOGS_RETENTION=365d
58+
export CODER_CONNECTION_LOGS_RETENTION=90d
59+
export CODER_API_KEYS_RETENTION=7d
60+
```
61+
62+
### YAML Configuration Example
63+
64+
```yaml
65+
retention:
66+
audit_logs: 365d
67+
connection_logs: 90d
68+
api_keys: 7d
69+
```
70+
71+
## How Retention Works
72+
73+
### Background Purge Process
74+
75+
Coder runs a background process that periodically deletes old records. The
76+
purge process:
77+
78+
1. Runs approximately every 10 minutes.
79+
2. Processes records in batches to avoid database lock contention.
80+
3. Deletes records older than the configured retention period.
81+
4. Logs the number of deleted records for monitoring.
82+
83+
### Effective Retention
84+
85+
Each retention setting controls its data type independently:
86+
87+
- When set to a non-zero duration, records older than that duration are deleted.
88+
- When set to `0`, retention is disabled and data is kept indefinitely.
89+
90+
### API Keys Special Behavior
91+
92+
API key retention only affects **expired** keys. A key is deleted only when:
93+
94+
1. The key has expired (past its `expires_at` timestamp).
95+
2. The key has been expired for longer than the retention period.
96+
97+
Setting `--api-keys-retention=7d` deletes keys that expired more than 7 days
98+
ago. Active keys are never deleted by the retention policy.
99+
100+
Keeping expired keys for a short period allows Coder to return a more helpful
101+
error message when users attempt to use an expired key.
102+
103+
## Best Practices
104+
105+
### Recommended Starting Configuration
106+
107+
For most deployments, we recommend:
108+
109+
```yaml
110+
retention:
111+
audit_logs: 365d
112+
connection_logs: 90d
113+
api_keys: 7d
114+
```
115+
116+
### Compliance Considerations
117+
118+
> [!WARNING]
119+
> Audit Logs provide critical security and compliance information. Purging
120+
> Audit Logs may impact your organization's ability to investigate security
121+
> incidents or meet compliance requirements. Consult your security and
122+
> compliance teams before configuring Audit Log retention.
123+
124+
Common compliance frameworks have varying retention requirements:
125+
126+
- **SOC 2**: Typically requires 1 year of audit logs.
127+
- **HIPAA**: Requires 6 years for certain records.
128+
- **PCI DSS**: Requires 1 year of audit logs, with 3 months immediately
129+
available.
130+
- **GDPR**: Requires data minimization but does not specify maximum retention.
131+
132+
### External Log Aggregation
133+
134+
If you use an external log aggregation system (Splunk, Datadog, etc.), you can
135+
configure shorter retention periods in Coder since logs are preserved
136+
externally. See
137+
[Capturing/Exporting Audit Logs](../security/audit-logs.md#capturingexporting-audit-logs)
138+
for details on exporting logs.
139+
140+
### Database Maintenance
141+
142+
After enabling retention policies, you may want to run a `VACUUM` operation on
143+
your PostgreSQL database to reclaim disk space. See
144+
[Maintenance Procedures](../security/audit-logs.md#maintenance-procedures-for-the-audit-logs-table)
145+
for guidance.
146+
147+
## Keeping Data Indefinitely
148+
149+
To keep data indefinitely for any data type, set its retention value to `0`:
150+
151+
```yaml
152+
retention:
153+
audit_logs: 0s # Keep audit logs forever
154+
connection_logs: 0s # Keep connection logs forever
155+
api_keys: 0s # Keep expired API keys forever
156+
```
157+
158+
## Monitoring
159+
160+
The purge process logs deletion counts at the `DEBUG` level. To monitor
161+
retention activity, enable debug logging or search your logs for entries
162+
containing the table name (e.g., `audit_logs`, `connection_logs`, `api_keys`).
163+
164+
## Related Documentation
165+
166+
- [Audit Logs](../security/audit-logs.md): Learn about Audit Logs and manual
167+
purge procedures.
168+
- [Connection Logs](../monitoring/connection-logs.md): Learn about Connection
169+
Logs and monitoring.

‎docs/manifest.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -363,6 +363,11 @@
363363
"title": "Telemetry",
364364
"description": "Learn what usage telemetry Coder collects",
365365
"path": "./admin/setup/telemetry.md"
366+
},
367+
{
368+
"title": "Data Retention",
369+
"description": "Configure data retention policies for database tables",
370+
"path": "./admin/setup/data-retention.md"
366371
}
367372
]
368373
},

0 commit comments

Comments
 (0)