Support for source only dev/ci builds #400

mthalman · 2025-05-06T17:10:40Z

Adds support for source-only dev/ci (non-official) builds which allows source-only jobs to run in PR validation within the UB pipeline.

The changes can be classified in the following groups:

Roslyn analyzer dependencies on Microsoft.CodeAnalysis: These versions needed to be updated because otherwise it would have caused a package downgrade error as described in vmr main fails to build with NU1109 errors while building roslyn source-build#5135 (comment). In this case Microsoft.CodeAnalysis gets lifted to the live version, which has a version of 10.0.0-ci. But due to semantic versioning rules, that value is interpreted as a lower version than 10.0.0-preview.4.25225.6. But we really don't want it to be lifted at all. Instead, we can compile against an SBRP version. This works for most of the projects but can't be done for the tools (GenerateDocumentation) which are relied upon for execution in the build. So a ref assembly won't do there. Instead, these projects are explicitly configured to not use CPM to lift the version for the offending packages.
There are cases in aspnetcore and sdk where CS9057 had to be suppressed. Search for CS9057 in the changed files for a description of why that was needed.
Test updates: Some tests rely on having access to Microsoft-built packages, such as portable packages, which obviously aren't produced by a source-only build. Given that this scenario is for PR validation and not all the build legs are running in that context, the tests won't have these Microsoft-built packages available. Instead, these tests are being disabled in this context.

am11 · 2025-05-09T10:55:46Z

Suggestions:

add --id, --build-id <value> argument in top-level build scripts and set OfficialBuildId accordingly
when it's not present, DisableDevBuildAsDefaultForSourceOnly=true is implied

Dev builds will look like:

-  ./build.sh --clean-while-building -sb --os linux --arch riscv64 -p:DisableDevBuildAsDefaultForSourceOnly=true
+  ./build.sh --clean-while-building -sb --os linux --arch riscv64

mthalman · 2025-05-09T13:27:34Z

add --id, --build-id argument in top-level build scripts and set OfficialBuildId accordingly

For official builds, we will either not require a build ID as input at all or address the UX with dotnet/source-build#5109.

mmitche · 2025-05-13T23:55:41Z

...oslynAnalyzers/Microsoft.CodeAnalysis.Analyzers/Core/Microsoft.CodeAnalysis.Analyzers.csproj

@@ -8,7 +8,7 @@
      Restore would conclude that there is a cyclic dependency between Microsoft.CodeAnalysis and Microsoft.CodeAnalysis.Analyzers.
    -->
    <PackageId>*$(MSBuildProjectFile)*</PackageId>
-    <MicrosoftCodeAnalysisVersion Condition="'$(MicrosoftCodeAnalysisVersion)' == ''">$(MicrosoftCodeAnalysisVersionForCodeAnalysisAnalyzers)</MicrosoftCodeAnalysisVersion>
+    <MicrosoftCodeAnalysisVersion>$(MicrosoftCodeAnalysisVersionForCodeAnalysisAnalyzers)</MicrosoftCodeAnalysisVersion>


mmitche · 2025-05-13T23:57:46Z

...ionAndConfigFilesForBrokenRuntime/GenerateDocumentationAndConfigFilesForBrokenRuntime.csproj


-    <PackageReference Include="System.Composition" />
+  <ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true' and '$(OfficialBuild)' == 'false'">


Have the roslyn repo-SB legs been removed? If not, has this been tested in that context?

Testing these changes here: dotnet/roslyn#78608

mmitche · 2025-05-13T23:59:17Z

src/sdk/src/BuiltInTools/dotnet-watch/dotnet-watch.csproj

+      analyzer assembly version is not the same as the compiler assembly version being used by the sdk that the project
+      is being built with. But this is fine because the analyzer is just being used here to distribute it with the tool.
+    -->
+    <NoWarn Condition="'$(DotNetBuildSourceOnly)' == 'true' and '$(OfficialBuild)' != 'true'">$(NoWarn);CS9057</NoWarn>


Really appreciate the in-depth comments here.

ViktorHofer · 2025-05-14T13:21:44Z

In this case Microsoft.CodeAnalysis gets lifted to the live version, which has a version of 10.0.0-ci. But due to semantic versioning rules, that value is interpreted as a lower version than 10.0.0-preview.4.25225.6.

Is that a fundamental problem that needs attention regardless of the changes in this PR?

But due to semantic versioning rules, that value is interpreted as a lower version than 10.0.0-preview.4.25225.6. But we really don't want it to be lifted at all. Instead, we can compile against an SBRP version.

Why don't we want it to be lifted and prefer SBRP? Because of the above semver issue or because of a design decision?

Instead, these projects are explicitly configured to not use CPM to lift the version for the offending packages.

Is that a sustainable pattern and if not, do we have ideas how to counter future issues around that?

MichaelSimons · 2025-05-14T13:27:40Z

repo-projects/aspnetcore.proj

+      a source generator which must be built targeting a version of Roslyn contained in the executing SDK. Otherwise, because of how
+      Arcade works for non-official builds, M.CA will get an assembly version of 42.42.42.42 and the source generator would target that
+      version leading to an error like the following: 
+        CSC error CS9057: Analyzer assembly '/repos/dotnet/src/aspnetcore/artifacts/bin/Microsoft.AspNetCore.Http.RequestDelegateGenerator/Release/netstandard2.0/Microsoft.AspNetCore.Http.RequestDelegateGenerator.dll' cannot be used because it references version '42.42.42.42' of the compiler, which is newer than the currently running version '5.0.0.0'.


Suggested change

CSC error CS9057: Analyzer assembly '/repos/dotnet/src/aspnetcore/artifacts/bin/Microsoft.AspNetCore.Http.RequestDelegateGenerator/Release/netstandard2.0/Microsoft.AspNetCore.Http.RequestDelegateGenerator.dll' cannot be used because it references version '42.42.42.42' of the compiler, which is newer than the currently running version '5.0.0.0'.

CSC error CS9057: Analyzer assembly '/repos/dotnet/src/aspnetcore/artifacts/bin/Microsoft.AspNetCore.Http.RequestDelegateGenerator/Release/netstandard2.0/Microsoft.AspNetCore.Http.RequestDelegateGenerator.dll'

cannot be used because it references version '42.42.42.42' of the compiler, which is newer than the currently running version '5.0.0.0'.

mthalman · 2025-05-16T12:55:34Z

Hey folks, I'm going to get this prioritized to be merged and will address feedback in a follow-up. Dev builds need to be enabled to unblock things (see dotnet/source-build#5135 (comment)).

mthalman · 2025-05-16T14:38:14Z

Getting a prebuilt for Microsoft.CodeAnalysis.Common.5.0.0-1.25230.108 coming from aspnetcore. This is due to now being dependent on the PSB and the PSB has some incorrect dependencies. The PSB contains Microsoft.CodeAnalysis.AnalyzerUtilities.5.0.0-1.25265.106 which has a dependency on Microsoft.CodeAnalysis.Common.5.0.0-1.25230.108. That Microsoft.CodeAnalysis.Common version isn't right, it should have the same version as AnalyzerUtilities.

mthalman · 2025-05-16T14:51:45Z

Getting a prebuilt for Microsoft.CodeAnalysis.Common.5.0.0-1.25230.108 coming from aspnetcore. This is due to now being dependent on the PSB and the PSB has some incorrect dependencies. The PSB contains Microsoft.CodeAnalysis.AnalyzerUtilities.5.0.0-1.25265.106 which has a dependency on Microsoft.CodeAnalysis.Common.5.0.0-1.25230.108. That Microsoft.CodeAnalysis.Common version isn't right, it should have the same version as AnalyzerUtilities.

It's because Microsoft.CodeAnalysis.AnalyzerUtilities has a package reference to Microsoft.CodeAnalysis.Common instead of a project reference:

dotnet/src/roslyn/src/RoslynAnalyzers/Microsoft.CodeAnalysis.AnalyzerUtilities/Microsoft.CodeAnalysis.AnalyzerUtilities.csproj

Line 21 in c7f1d4f

    
           <PackageReference Include="Microsoft.CodeAnalysis.Common" VersionOverride="$(MicrosoftCodeAnalysisVersion)" />

This causes it to have a dependency on the PSB version of Microsoft.CodeAnalysis.Common (5.0.0-1.25230.108) instead of the live version.

mthalman · 2025-05-17T00:40:08Z

Getting a prebuilt for Microsoft.CodeAnalysis.Common.5.0.0-1.25230.108 coming from aspnetcore. This is due to now being dependent on the PSB and the PSB has some incorrect dependencies. The PSB contains Microsoft.CodeAnalysis.AnalyzerUtilities.5.0.0-1.25265.106 which has a dependency on Microsoft.CodeAnalysis.Common.5.0.0-1.25230.108. That Microsoft.CodeAnalysis.Common version isn't right, it should have the same version as AnalyzerUtilities.

It's because Microsoft.CodeAnalysis.AnalyzerUtilities has a package reference to Microsoft.CodeAnalysis.Common instead of a project reference:

dotnet/src/roslyn/src/RoslynAnalyzers/Microsoft.CodeAnalysis.AnalyzerUtilities/Microsoft.CodeAnalysis.AnalyzerUtilities.csproj

Line 21 in c7f1d4f

<PackageReference Include="Microsoft.CodeAnalysis.Common" VersionOverride="$(MicrosoftCodeAnalysisVersion)" />

This causes it to have a dependency on the PSB version of Microsoft.CodeAnalysis.Common (5.0.0-1.25230.108) instead of the live version.

This seems to be temporarily resolved by using the output of the stage 2 build as rebootstrap source: #598. This makes sense because the issue is that it's using the wrong version from the PSB artifacts. But in stage 2 the correct version is being output from stage 1 which is then selected as the PSB artifact in the stage 2 build.

mthalman · 2025-05-17T01:53:21Z

Merging this now. I'll address the comments in a follow-up.

mthalman force-pushed the dev/mthalman/sb5061-dev-builds branch 6 times, most recently from b427802 to 8401a4f Compare May 7, 2025 16:28

mthalman mentioned this pull request May 7, 2025

IL-related diagnostic warnings not reported as errors dotnet/aspnetcore#61827

Open

mthalman force-pushed the dev/mthalman/sb5061-dev-builds branch 8 times, most recently from bb9a5a2 to 8a37d68 Compare May 8, 2025 21:13

mthalman force-pushed the dev/mthalman/sb5061-dev-builds branch 2 times, most recently from bc58e04 to 1de58cb Compare May 9, 2025 15:41

am11 mentioned this pull request May 9, 2025

Evaluate distro maintainer UX for building with OfficialBuildId dotnet/source-build#5109

Open

mthalman force-pushed the dev/mthalman/sb5061-dev-builds branch from 1de58cb to 37f23f5 Compare May 9, 2025 18:16

akoeplinger mentioned this pull request May 13, 2025

Remove packageVersion from the source-manifest.json dotnet/arcade-services#2428

Closed

mthalman added 3 commits May 13, 2025 13:56

Support for source only dev/ci builds

d003611

arcade fix

057c639

Use N-1 version for M.CA

1911617

mthalman force-pushed the dev/mthalman/sb5061-dev-builds branch from 37f23f5 to 1911617 Compare May 13, 2025 18:56

get rid of CS9057 nowarn

6d1770b

mthalman changed the title ~~[DRAFT] Support for source only dev/ci builds~~ Support for source only dev/ci builds May 13, 2025

mthalman marked this pull request as ready for review May 13, 2025 22:49

mthalman requested a review from a team as a code owner May 13, 2025 22:49

mmitche reviewed May 13, 2025

View reviewed changes

am11 mentioned this pull request May 14, 2025

Remove SkipUsingCrossgen filipnavara/dotnet-riscv#7

Merged

mthalman mentioned this pull request May 14, 2025

Re-Bootstrap Source Build to .NET 10.0.100-preview.5.25230.108 and use Unified-Build's SB legs for PR validation #515

Merged

MichaelSimons reviewed May 14, 2025

View reviewed changes

Merge branch 'main' into dev/mthalman/sb5061-dev-builds

209dec1

jkoritzinsky mentioned this pull request May 14, 2025

[main] Source code updates from dotnet/runtime #544

Merged

mthalman added 2 commits May 15, 2025 09:33

re-enable tests

53eff7d

Remove references to DisableDevBuildAsDefaultForSourceOnly property

03fbfaa

mthalman mentioned this pull request May 16, 2025

vmr main fails to build with NU1109 errors while building roslyn dotnet/source-build#5135

Closed

mthalman added 2 commits May 16, 2025 08:20

Merge branch 'main' into dev/mthalman/sb5061-dev-builds

655786f

Remove fake OfficialBuildId for PR builds

3a09d50

mthalman mentioned this pull request May 16, 2025

Change AnalyzerUtilities to reference Microsoft.CodeAnalysis.Common by project instead of package #596

Closed

Merge branch 'main' into dev/mthalman/sb5061-dev-builds

1fec9ab

mthalman merged commit 21afdab into dotnet:main May 17, 2025
11 checks passed

mthalman deleted the dev/mthalman/sb5061-dev-builds branch May 17, 2025 01:53

This was referenced May 19, 2025

In RoslynAnalyzers, change Microsoft.CodeAnalysis package references to project references dotnet/roslyn#78643

Closed

Misc updates from source-only dev build support #622

Merged

akoeplinger mentioned this pull request May 21, 2025

[main] Source code updates from dotnet/dotnet dotnet/aspnetcore#62019

Merged

This was referenced May 21, 2025

Use SBRP version of M.CA for analyzers in the context of source build dotnet/roslyn#78668

Merged

dev/ci build bundled frameworks are not considered newer than previews #1055

Open

eerhardt mentioned this pull request May 28, 2025

[main] Source code updates from dotnet/dotnet dotnet/aspnetcore#62072

Merged

mthalman mentioned this pull request Jun 23, 2025

Re-Bootstrap Source Build to .NET 10.0.100-preview.7.25322.101 #1289

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support for source only dev/ci builds #400

Support for source only dev/ci builds #400

Uh oh!

mthalman commented May 6, 2025 •

edited

Loading

Uh oh!

am11 commented May 9, 2025

Uh oh!

mthalman commented May 9, 2025

Uh oh!

mmitche May 13, 2025

Uh oh!

mmitche May 13, 2025

Uh oh!

mthalman May 16, 2025

Uh oh!

mmitche May 13, 2025

Uh oh!

ViktorHofer commented May 14, 2025 •

edited

Loading

Uh oh!

MichaelSimons May 14, 2025

Uh oh!

mthalman commented May 16, 2025

Uh oh!

mthalman commented May 16, 2025

Uh oh!

mthalman commented May 16, 2025

Uh oh!

mthalman commented May 17, 2025

Uh oh!

mthalman commented May 17, 2025

Uh oh!

Uh oh!

Uh oh!


		<PackageReference Include="System.Composition" />
		<ItemGroup Condition="'$(DotNetBuildSourceOnly)' == 'true' and '$(OfficialBuild)' == 'false'">

	CSC error CS9057: Analyzer assembly '/repos/dotnet/src/aspnetcore/artifacts/bin/Microsoft.AspNetCore.Http.RequestDelegateGenerator/Release/netstandard2.0/Microsoft.AspNetCore.Http.RequestDelegateGenerator.dll' cannot be used because it references version '42.42.42.42' of the compiler, which is newer than the currently running version '5.0.0.0'.
	CSC error CS9057: Analyzer assembly '/repos/dotnet/src/aspnetcore/artifacts/bin/Microsoft.AspNetCore.Http.RequestDelegateGenerator/Release/netstandard2.0/Microsoft.AspNetCore.Http.RequestDelegateGenerator.dll'
	cannot be used because it references version '42.42.42.42' of the compiler, which is newer than the currently running version '5.0.0.0'.

Support for source only dev/ci builds #400

Support for source only dev/ci builds #400

Uh oh!

Conversation

mthalman commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

am11 commented May 9, 2025

Uh oh!

mthalman commented May 9, 2025

Uh oh!

mmitche May 13, 2025

Choose a reason for hiding this comment

Uh oh!

mmitche May 13, 2025

Choose a reason for hiding this comment

Uh oh!

mthalman May 16, 2025

Choose a reason for hiding this comment

Uh oh!

mmitche May 13, 2025

Choose a reason for hiding this comment

Uh oh!

ViktorHofer commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelSimons May 14, 2025

Choose a reason for hiding this comment

Uh oh!

mthalman commented May 16, 2025

Uh oh!

mthalman commented May 16, 2025

Uh oh!

mthalman commented May 16, 2025

Uh oh!

mthalman commented May 17, 2025

Uh oh!

mthalman commented May 17, 2025

Uh oh!

Uh oh!

Uh oh!

mthalman commented May 6, 2025 •

edited

Loading

ViktorHofer commented May 14, 2025 •

edited

Loading