Missing Source Files from Database for XNU

[qu0dy]

Hi,

Most of the .c files are missing after compilation of XNU in DB.

Log entry:

/opt/homebrew/Caskroom/codeql/2.11.4/codeql/cpp/tools/osx64/extractor --trapfolder /private/tmp/xnu/build-xnu-xnu-8792.41.9/xnu-xnu-8792.41.9/xnu-database/trap/cpp --src_archive /private/tmp/xnu/build-xnu-xnu-8792.41.9/xnu-xnu-8792.41.9/xnu-database/src --mimic_config /private/tmp/xnu/build-xnu-xnu-8792.41.9/xnu-xnu-8792.41.9/xnu-database/working/compiler_mimic_cache/0bf1e9bd38ad --executable_filename /private/tmp/xnu/build-xnu-xnu-8792.41.9/xnu-xnu-8792.41.9/BUILD/obj/SETUP/installfile/installfile.o -w --error_limit 1000 --disable_system_macros --variadic_macros --gcc --gnu_version 40000 --clang --isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.0.sdk -I/tmp/xnu/build-xnu-xnu-8792.41.9/xnu-xnu-8792.41.9/SETUP/installfile/ -I. --blocks -D__GCC_HAVE_DWARF2_CFI_ASM=1 -- installfile.c /tmp/xnu/build-xnu-xnu-8792.41.9/xnu-xnu-8792.41.9/SETUP/installfile/installfile.c

[E 02:03:00 36601] Warning[extractor-c++]: In construct_message: Catastrophic error: cannot open source file "installfile.c"

Env:

• Builder script: https://gist.github.com/qu0dy/57145e3ca850536169566c1560fd23cd
• XNU: https://github.com/apple-oss-distributions/xnu/releases/tag/xnu-8792.41.9
• OS: MacOS Ventura 13.1 Beta 4
• Clang: 14.0.0
• CodeQL: 2.11.4
• CPU: M1 Pro

[hvitved]

Thanks for your report. @github/codeql-c is this something you can look into, please?

[jketema]

Hi @qu0dy,

Thanks for the report.

Just to confirm: does the build script you linked to allow you to successfully build XNU when invoked by itself? If so, could you provide us with the complete build-tracer.log? The parts of the log you included do not quite provide sufficient information.

I also see you're running a beta version of MacOS Ventura, does the same problem occur with the latest released version of Ventura?

[qu0dy]

Hi @jketema,

Just to confirm: does the build script you linked to allow you to successfully build XNU when invoked by itself?

Yes, it's.

If so, could you provide us with the complete build-tracer.log?

Here is the log, it's quite large (~70mb).

I also see you're running a beta version of MacOS Ventura, does the same problem occur with the latest released version of Ventura?

Unfortunately I can't test it on stable MacOS version right now.

[qu0dy]

@jketema the issue resolved when I used:
codeql database create xnu-database --language=cpp --command="./build.sh"

Instead of:
codeql database create xnu-database --language=cpp --command="make ARCH_CONFIGS="x86_64" KERNEL_CONFIGS=RELEASE BUILD_WERROR=0".

I thought that I have to use it only with the latest step of XNU building.

[jketema]

@jketema the issue resolved when I used: codeql database create xnu-database --language=cpp --command="./build.sh"

Just to confirm: Did you check - using a CodeQL query - that the database actually contains the missing source files?

Instead of: codeql database create xnu-database --language=cpp --command="make ARCH_CONFIGS="x86_64" KERNEL_CONFIGS=RELEASE BUILD_WERROR=0".

I thought that I have to use it only with the latest step of XNU building.

Indeed it should. From the logs I can see that the clang compiler is not properly identified in this case. What is the output of

/applications/xcode.app/contents/developer/toolchains/xcodedefault.xctoolchain/usr/bin/clang --version

when you run it instead of

codeql database create xnu-database --language=cpp --command="make ARCH_CONFIGS="x86_64" KERNEL_CONFIGS=RELEASE BUILD_WERROR=0"

?

[qu0dy]

Just to confirm: Did you check - using a CodeQL query - that the database actually contains the missing source files?

Yes, it does.

Indeed it should. From the logs I can see that the clang compiler is not properly identified in this case. What is the output of when you run it instead of

Here is a log, which produces correct db.

[jketema]

Hi @qu0dy,

Thanks for the confirmation and the logs. Since you have a workaround, there doesn't seem to be an immediate urgency to get this resolved. So, we'll track this in an internal ticket and revisit this at some later date.