Bruker M4 Tornado bcf-files spectrum_image does not match pixel spectra

[paprika113]

testBCF_M4_Tornado.zip

Hello,
I am new to this community. Firstly, I would like to thank the community and especially @sem-geologist for creating the bruker library and making it available in rsciio!!
I am working with M4 Tornado µXRF from Bruker and I know that the library is initially designed for Bruker EDX which I do not have access to.
I observed inconsistencies comparing the spectrum maps from bruker.file_reader of the bcf files and the single pixel spectra .spx files, which I can generate using the M4 Tornado Software.

I used
bcf = bruker.file_reader(bcf_file, select_type='spectrum_image')[0]
to get a spectrum image of the bcf file "minBcf.bcf" (a 3x3-pixel bcf added in the attachment).
I used the M4 Tornado software to generate the "P1.spx" file of the first pixel (added in attachment)
spx = bruker.file_reader(spx_file)[0]['data'] gives me the spectrum of the P1.spx-file
I compared the spectra that I got within the file_reader
bcf['data'][0][0]
I plotted the spectra (channels 80 to 120) in 'P1_compare_spx_bcf.png' for quick visualization.
Obviously they don`t match, although they should.

I observed this kind of behaviour in all bcf files I looked at.

I had a closer look at the py_parse_hypermap function and tried to investigate the bitstream of the first pixel.
(BTW: again big respect to @sem-geologist for figuring out all that!)
In the attached file 'p1_bitstream.pkl' I dumped the bitstream of the first Pixel as a binary file (although I am not 100% sure that it is only the first pixel). But it does start with the pixel header (x_pix, chan1, chan2, ...) encoded in <IHHIHHHI. Then comes the instructively packed data (flag == 2). The first 88 channels of the example file have counts = 0. So size_p, channels (the first two bytes) encoded in <BB give me (0, 88). Fine so far! Up to channel 88 everything matches with P1.spx.
Then things get messy. Next two bites give me size_p, channels = (2, 6) meaning the next seven bytes must be gain = 0 in <B and then six bytes actually giving me the counts for channels 89 to 94 encoded in <BBBBBB.
These six number are (2, 5, 11, 22, 24, 47). In the P1-spx file the counts of channel 89 to 94 are (2, 8, 15, 34, 60, 90).

So for now I am rather sure that M4 Tornado uses a slightly different encoding than Bruker EDX Software.
The question is which one?

I hope the explanation and the example files can help to fix that issue.

Python environment:

• RosettaSciIO version: 0.13.0
• Python version: 3.12.8

[sem-geologist]

I think this is not a bug. (kind of) just peculiarity of your BCF and sign of beautifully working Tornado M4. I looked to your spx. and I see your system has 2 EDS, which in that spx is summed up into signle EDS spectra.
by calling bcf = bruker.file_reader(bcf_file, select_type='spectrum_image')[0] you selected only data from one of EDS cubes (from 2). that is why it is just half of intensity. Try summing data from [0] and [1].
Your BCF has following internal file structure (sub-files):
HeaderData, SpectrumData0, SpectrumPositions0, SpectrumData1, SpectrumPositions1;
unfortunately to find that out you need to dwell in terribly hidden api accessible with:
from rsciio.bruker import _api as bruker_api
and then you can use:
bruker_api.SFS_reader("yourbcf_file.bcf")
to look around the internals of file.

Actually your BCF is a missing piece in my doubts of reverse engineering. It initially looked to me like bcf is designed with possibility to preserve hypermaps per spectrometer, but so far I could not come along any of such BCF files, thus I was going into doubt if someone had not over-engineered the format, and such features got forgotten by team applying it to specific detectors? And I was already keen just oversimplifying things (i.e. on SEM with two EDS detectors all signals are summed into single hypermap). Your data is very valuable as gives direct proof that indeed these formats can hold the counts for every EDS separately. That is nice, especially for morphologically uneven samples.

This actually gives some new frontiers for developing further (nad refactoring) bruker reader. As Esprit by default sums these spectras up, It is probably that the same should be done here.
This is not so simple (and I understand why it is saved separately, and there is a value of doing it like that). The spectrum position calibration is per EDS detector, not per whole system. In case of slight miscallibration of one detector, it is then possible to recalibrate only hypermap of that particular detector post-acquisition, (and then sum). If they would be summed, post-acquisition re-calibration of data is practically not possible as by recalibrating one - the other gets miscallibrated.

There is additional things I will need to address: the HeaderData holds in such multi-EDS system many sum EDS, not just a single EDS. I think reader currently miss that. Also your HeaderData is pecular as it takes majority of space (20MB) where raw data cubes just few kbytes.

[paprika113]

@sem-geologist ,

Thank´s a lot!! That makes perfect sense and just didn´t occur to me. It also explains why the difference is kind of almost a factor two in each channel but not exactly.
Actually file_reader has to be called with index='all' as:
bcf = bruker.file_reader(bcf_file, select_type='spectrum_image', index='all'),
which I somehow never did. So I only got the data of the first detector.
If I sum up both it all fits with the spx data :).

I am happy the data is useful for further development.

Regarding the Header: The M4 Tornado µXRF makes a camera based optical overview image before the actual XRF acquisition is happening (The software calls that 'mosaic image'). This image is (in case of my file) a rather larger RGB image (Around 9 MB for the example if I export it to .bmp).
This image must also be stored somewhere in the bcf file. I imagine this probably works differently in the Esprit Software and bcf files with few pixels are generally much smaller.
In the example the mosaic image is actually unnecessarily large and I didn´t really take care about that when uploading.
Let me know if it would help to provide a bcf with a smaller mosaic image.
For a different example, I could also upload the mosaic image in bmp format.

[ericpre]

Let me know if it would help to provide a bcf with a smaller mosaic image.

Smaller test file are always useful, if you can provide one, this would be appreciated. Please read https://hyperspy.org/rosettasciio/contributing.html#making-test-data-files for more explanation and guidance.