Optimize Index Permission Automatons for Has Privileges #136625

jfreden · 2025-10-15T13:58:17Z

This PR optimizes how index permission automatons are created.

In #110633 a feature was added to allow for doing regular expression "has privilege" style checks against index permission groups. The reason this was added was to support regular expressions for the manage_roles privilege (so unrelated to the has privileges API). The implementation was known to have performance issues and the change was therefore made optional (and disabled in the has privileges API). Unfortunately, a change to simplify the logic by doing Automatons.unionAndMinimize on all index permissions in a group was left outside the conditional and therefore caused a performance regression.

See this comment for details on the original change.

jfreden · 2025-10-16T06:21:13Z

The reported CI failure in serverless is valid I think:

REPRODUCE WITH: ./gradlew ":modules:serverless-security:javaRestTest" --tests "co.elastic.elasticsearch.serverless.security.crossproject.CrossProjectSearchRestIT.testCrossProjectResolveIndexWithUniversalApiKey" -Dtests.seed=286311FBFFFE4865 -Dtests.locale=ar-DZ -Dtests.timezone=America/Rainy_River -Druntime.java=25
  |  
  | CrossProjectSearchRestIT > testCrossProjectResolveIndexWithUniversalApiKey FAILED
  | org.elasticsearch.client.ResponseException: method [GET], host [http://[::1]:33125], URI [/_resolve/index/_all], status line [HTTP/1.1 500 Internal Server Error]
  | {"error":{"root_cause":[{"type":"null_pointer_exception","reason":"Cannot invoke \"org.elasticsearch.action.ResolvedIndexExpressions.expressions()\" because \"localResolvedExpressions\" is null"}],"type":"null_pointer_exception","reason":"Cannot invoke \"org.elasticsearch.action.ResolvedIndexExpressions.expressions()\" because \"localResolvedExpressions\" is null"},"status":500}
  | at __randomizedtesting.SeedInfo.seed([286311FBFFFE4865:AD3657883BFBAD09]:0)
  | at app//org.elasticsearch.client.RestClient.convertResponse(RestClient.java:351)
  | at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:317)
  | at app//org.elasticsearch.client.RestClient.performRequest(RestClient.java:292)
  | at app//co.elastic.elasticsearch.serverless.security.cloud.AbstractUniversalIamIT.performRequestWithCloudApiKey(AbstractUniversalIamIT.java:208)
  | at app//co.elastic.elasticsearch.serverless.security.cloud.AbstractUniversalIamIT.performRequestWithCloudApiKey(AbstractUniversalIamIT.java:198)
  | at app//co.elastic.elasticsearch.serverless.security.crossproject.CrossProjectSearchRestIT.resolveIndicesAndAssert(CrossProjectSearchRestIT.java:792)
  | at app//co.elastic.elasticsearch.serverless.security.crossproject.CrossProjectSearchRestIT.testCrossProjectResolveIndexWithUniversalApiKey(CrossProjectSearchRestIT.java:381)
  |

Working on reproducing locally now.

elasticsearchmachine · 2025-10-16T07:26:19Z

Hi @jfreden, I've created a changelog YAML for you.

elasticsearchmachine · 2025-10-16T07:27:24Z

Pinging @elastic/es-security (Team:Security)

jfreden · 2025-10-16T07:30:15Z

docs/changelog/136625.yaml

@@ -0,0 +1,5 @@
+pr: 136625
+summary: Optimize index permissions


Will update this as soon as CI finishes

…issue

slobodanadamovic

LGTM 🚀

Thanks for fixing it!

elasticsearchmachine · 2025-10-16T09:37:28Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts
✅	9.1
✅	9.2

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 136625

This PR optimizes how index permission automatons are created. In elastic#110633 a feature was added to allow for doing regular expression "has privilege" style checks against index permission groups. The reason this was added was to support regular expressions for the `manage_roles` privilege (so unrelated to the has privileges API). The implementation was known to have performance issues and the change was therefore made optional (and disabled in the has privileges API). Unfortunately, a change to simplify the logic by doing `Automatons.unionAndMinimize` on all index permissions in a group was left outside the conditional and therefore caused a performance regression. See [this](https://github.com/elastic/elasticsearch/blob/2a9c07eb6fe4e8bbb7fc9fbab6fa59ccc006e10b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java#L840-L860) comment for details on the original change.

jfreden · 2025-10-16T10:01:21Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

This PR optimizes how index permission automatons are created. In elastic#110633 a feature was added to allow for doing regular expression "has privilege" style checks against index permission groups. The reason this was added was to support regular expressions for the `manage_roles` privilege (so unrelated to the has privileges API). The implementation was known to have performance issues and the change was therefore made optional (and disabled in the has privileges API). Unfortunately, a change to simplify the logic by doing `Automatons.unionAndMinimize` on all index permissions in a group was left outside the conditional and therefore caused a performance regression. See [this](https://github.com/elastic/elasticsearch/blob/2a9c07eb6fe4e8bbb7fc9fbab6fa59ccc006e10b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java#L840-L860) comment for details on the original change. (cherry picked from commit 3df0abb) # Conflicts: # x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java

…36674) This PR optimizes how index permission automatons are created. In #110633 a feature was added to allow for doing regular expression "has privilege" style checks against index permission groups. The reason this was added was to support regular expressions for the `manage_roles` privilege (so unrelated to the has privileges API). The implementation was known to have performance issues and the change was therefore made optional (and disabled in the has privileges API). Unfortunately, a change to simplify the logic by doing `Automatons.unionAndMinimize` on all index permissions in a group was left outside the conditional and therefore caused a performance regression. See [this](https://github.com/elastic/elasticsearch/blob/2a9c07eb6fe4e8bbb7fc9fbab6fa59ccc006e10b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java#L840-L860) comment for details on the original change.

…36678) This PR optimizes how index permission automatons are created. In #110633 a feature was added to allow for doing regular expression "has privilege" style checks against index permission groups. The reason this was added was to support regular expressions for the `manage_roles` privilege (so unrelated to the has privileges API). The implementation was known to have performance issues and the change was therefore made optional (and disabled in the has privileges API). Unfortunately, a change to simplify the logic by doing `Automatons.unionAndMinimize` on all index permissions in a group was left outside the conditional and therefore caused a performance regression. See [this](https://github.com/elastic/elasticsearch/blob/2a9c07eb6fe4e8bbb7fc9fbab6fa59ccc006e10b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java#L840-L860) comment for details on the original change. (cherry picked from commit 3df0abb) # Conflicts: # x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java

…36673) This PR optimizes how index permission automatons are created. In #110633 a feature was added to allow for doing regular expression "has privilege" style checks against index permission groups. The reason this was added was to support regular expressions for the `manage_roles` privilege (so unrelated to the has privileges API). The implementation was known to have performance issues and the change was therefore made optional (and disabled in the has privileges API). Unfortunately, a change to simplify the logic by doing `Automatons.unionAndMinimize` on all index permissions in a group was left outside the conditional and therefore caused a performance regression. See [this](https://github.com/elastic/elasticsearch/blob/2a9c07eb6fe4e8bbb7fc9fbab6fa59ccc006e10b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java#L840-L860) comment for details on the original change.

This PR optimizes how index permission automatons are created. In elastic#110633 a feature was added to allow for doing regular expression "has privilege" style checks against index permission groups. The reason this was added was to support regular expressions for the `manage_roles` privilege (so unrelated to the has privileges API). The implementation was known to have performance issues and the change was therefore made optional (and disabled in the has privileges API). Unfortunately, a change to simplify the logic by doing `Automatons.unionAndMinimize` on all index permissions in a group was left outside the conditional and therefore caused a performance regression. See [this](https://github.com/elastic/elasticsearch/blob/2a9c07eb6fe4e8bbb7fc9fbab6fa59ccc006e10b/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/permission/IndicesPermission.java#L840-L860) comment for details on the original change.

Optimize index permissions

9181291

elasticsearchmachine added the v9.3.0 label Oct 15, 2025

fixup!

02e6617

slobodanadamovic added a commit to slobodanadamovic/elasticsearch that referenced this pull request Oct 15, 2025

apply fix from elastic#136625

c19227e

jfreden and others added 2 commits October 16, 2025 08:30

Merge branch 'main' into fix_combine_groups_issue

6dac86e

Add chached supplier optimization

b71e7c7

jfreden added >bug :Security/Security Security issues without another label labels Oct 16, 2025

Update docs/changelog/136625.yaml

5cec1c2

jfreden added v8.19.6 v9.1.6 v9.2.1 auto-backport Automatically create backport pull requests when merged labels Oct 16, 2025

jfreden marked this pull request as ready for review October 16, 2025 07:27

elasticsearchmachine added the Team:Security Meta label for security team label Oct 16, 2025

jfreden changed the title ~~Optimize index permissions~~ Optimizes Index Permission Automatons for Has Privileges Oct 16, 2025

jfreden changed the title ~~Optimizes Index Permission Automatons for Has Privileges~~ Optimize Index Permission Automatons for Has Privileges Oct 16, 2025

jfreden commented Oct 16, 2025

View reviewed changes

jfreden requested a review from slobodanadamovic October 16, 2025 08:12

jfreden added 2 commits October 16, 2025 10:27

fixup! name changes + comment changes

53d6bc9

Merge remote-tracking branch 'upstream/main' into fix_combine_groups_…

821df6b

…issue

slobodanadamovic approved these changes Oct 16, 2025

View reviewed changes

jfreden added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Oct 16, 2025

elasticsearchmachine merged commit 3df0abb into elastic:main Oct 16, 2025
40 checks passed

jfreden deleted the fix_combine_groups_issue branch October 16, 2025 09:36

This was referenced Oct 16, 2025

[9.1] Optimize Index Permission Automatons for Has Privileges (#136625) #136673

Merged

[9.2] Optimize Index Permission Automatons for Has Privileges (#136625) #136674

Merged

elasticsearchmachine added the backport pending label Oct 16, 2025

jfreden mentioned this pull request Oct 16, 2025

[8.19] Optimize Index Permission Automatons for Has Privileges (#136625) #136678

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Optimize Index Permission Automatons for Has Privileges #136625

Optimize Index Permission Automatons for Has Privileges #136625

Uh oh!

jfreden commented Oct 15, 2025 •

edited

Loading

Uh oh!

jfreden commented Oct 16, 2025

Uh oh!

elasticsearchmachine commented Oct 16, 2025

Uh oh!

elasticsearchmachine commented Oct 16, 2025

Uh oh!

jfreden Oct 16, 2025

Uh oh!

sarayut852709 Nov 15, 2025

Uh oh!

slobodanadamovic left a comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 16, 2025

Uh oh!

jfreden commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -0,0 +1,5 @@
		pr: 136625
		summary: Optimize index permissions

Optimize Index Permission Automatons for Has Privileges #136625

Optimize Index Permission Automatons for Has Privileges #136625

Uh oh!

Conversation

jfreden commented Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jfreden commented Oct 16, 2025

Uh oh!

elasticsearchmachine commented Oct 16, 2025

Uh oh!

elasticsearchmachine commented Oct 16, 2025

Uh oh!

jfreden Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

sarayut852709 Nov 15, 2025

Choose a reason for hiding this comment

Uh oh!

slobodanadamovic left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Oct 16, 2025

💔 Backport failed

Uh oh!

jfreden commented Oct 16, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jfreden commented Oct 15, 2025 •

edited

Loading