Skip to content

Conversation

@cstyan
Copy link
Contributor

@cstyan cstyan commented Nov 4, 2025

In this PR we're optimizing the GetTemplateAppInsightsByTemplate query by pre-filtering out apps which do not have an active session during the start/end time window.
Note: as of Nov 4 this is our most expensive query internally in terms of DB load though it only is called ~900 times in a 24h period it has a 1s average execution time and minimum of ~400ms.

This query currently looks at all entries in workspace_app_stats (IIUC each row is an app session), then splits them into per-minute buckets before filtering out buckets that do not fall within our start/end time window. This leads to expensive query processing time to do sequential scans and joins for various data that will eventually just be thrown away due to being outside the time range.

Instead, we can pre-filter out the majority of the buckets that need to be thrown away by only retrieving entries from workspace_app_stats for sessions where at least some portion of the sessions active time range is within our start/end time range. We keep the existing filter of buckets to ensure we still filter out buckets that fall outside the time range.

The default time window for the query is 5 minutes, so start = time.Now() - 5m and end = time.Now(). Using EXPLAIN and a specific 5 minute time window from earlier today (2025-11-03 18:00–18:05 UTC) the difference is pretty obvious:

  • Current: 444.8 ms, shared buffers 6,385, Parallel Seq Scan on workspace_app_stats, generate_series called ~67,380 times.
  • Optimized: 16.09 ms, shared buffers 1,782, only 4 generate_series calls.

I compared/verified the output of the queries, the filtered rows and distinct app names/template IDs, and that terminal is still not considered an app.

@github-actions github-actions bot added the stale This issue is like stale bread. label Nov 12, 2025
@github-actions github-actions bot closed this Nov 16, 2025
@cstyan cstyan reopened this Nov 17, 2025
@github-actions github-actions bot removed the stale This issue is like stale bread. label Nov 18, 2025
@cstyan cstyan changed the title perf: optimize GetTemplateAppInsightsByTemplate by pre-filtering apps based on start/end times perf: optimize GetTemplateAppInsightsByTemplate by pre-filtering on start/end times Nov 18, 2025

-- name: GetTemplateAppInsightsByTemplate :many
-- GetTemplateAppInsightsByTemplate is used for Prometheus metrics. Keep
-- in sync with GetTemplateAppInsights and UpsertTemplateUsageStats.
Copy link
Contributor

@zedkipp zedkipp Nov 19, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Double checking: is there anything to update in GetTemplateAppInsights or UpsertTemplateUsageStats?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh nice catch, I hadn't seen that part of the comment. Will double check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zedkipp UpsertTemplateUsageStats uses the same workspace_app_stats table to we can apply the same change there. GetTemplateAppInsights already uses a table with pre-aggregated data, there's no expensive calculation post-join so I don't believe there's any benefit to doing the same optimization for the join there. We can reevaluate later if we see that query show up in query insights as being expensive.

@github-actions github-actions bot added the stale This issue is like stale bread. label Nov 27, 2025
@github-actions github-actions bot closed this Nov 30, 2025
@cstyan cstyan reopened this Dec 2, 2025
start/end times

Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
Signed-off-by: Callum Styan <callumstyan@gmail.com>
@cstyan cstyan force-pushed the callum/app-insights-optimize branch from 391b435 to 2074695 Compare December 2, 2025 22:09
@github-actions github-actions bot removed the stale This issue is like stale bread. label Dec 3, 2025
@cstyan cstyan merged commit a59a84b into main Dec 9, 2025
31 checks passed
@cstyan cstyan deleted the callum/app-insights-optimize branch December 9, 2025 23:21
@github-actions github-actions bot locked and limited conversation to collaborators Dec 9, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants