[BUG] mt8195 ADSP Fatal Exception on debug build of Zephyrized SOF firmware

[andrew-mtk]

Describe the bug
Zephyrized mt8195 SOF fimware cause fatal exception in debug only build with the following output:

root@genio-1200-evk:/lib/firmware/mediatek/sof# python3 mtk_adsp_load.py load zephyr.img
[2235939792] <inf> init: FW ABI 0x301d001 DBG ABI 0x5003000 tags SOF:v2.12-rc1-321-gf3e34039db4b-dirty zephyr:v4.1.0-588-g218de8dc676c src hash 0x4fb565c8 (ref hash 0x4fb565c8)
[2235940029] <err> os:  ** FATAL EXCEPTION
[2235940124] <err> os:  ** CPU 0 EXCCAUSE 28 (load prohibited)
[2235940143] <err> os:  **  PC 0x60018a31 VADDR (nil)
[2235940158] <err> os:  **  PS 0x60120
[2235940180] <err> os:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:1 CALLIN                       C:2)
[2235940214] <err> os:  **  A0 0xa001853a  SP 0x6005eb70  A2 0x40000480  A3 0x30
[2235940230] <err> os:  **  A4 (nil)  A5 0xc0  A6 0x60720  A7 (nil)
[2235940254] <err> os:  **  A8 0xa0018a2f  A9 0x6005eb50 A10 (nil) A11 (nil)
[2235940270] <err> os:  ** A12 (nil) A13 0x60920 A14 0x40000300 A15 0x7fff
[2235940291] <err> os:  ** LBEG 0x60027ab0 LEND 0x60027ab4 LCOUNT (nil)
[2235940307] <err> os:  ** SAR 0x12
[2235940322] <err> os:  **  THREADPTR 0xa002568c
[2235940382] <err> os: >>> ZEPHYR FATAL ERROR 0: CPU exception on CPU 0
[2235940423] <err> os: Current thread: 0x60057ba8 (unknown)
[2235946302] <err> zephyr: Halting system

PC points to an address inside zephyr_ll_domain() in zephyr_ll.c

 .text.zephyr_ll_domain
                0x0000000060018a20       0x3c modules/sof/libmodules_sof.a(zephyr_ll.c.obj)
                                         0x48 (size before relaxing)
                0x0000000060018a20                zephyr_ll_domain
 .text.set_cpuclk
                0x0000000060018a5c       0x15 modules/sof/libmodules_sof.a(platform.c.obj)

To Reproduce
Build FW with the following command:
./sof/scripts/xtensa-build-zephyr.py -d -p -u mt8195

Load FW on Genio 1200 EVK with:
# python3 mtk_adsp_load.py load zephyr.img

Reproduction Rate
100%

[lgirdwood]

@andrew-mtk not seeing this yet on other platforms, but it looks like a NULL pointer deref (maybe due to a race since not seen elsewhere??) . Are you able to add more debug and root cause.

[2235940124] <err> os:  ** CPU 0 EXCCAUSE 28 (load prohibited)
[2235940143] <err> os:  **  PC 0x60018a31 VADDR (nil)

[andrew-mtk]

It's caused by enabling CONFIG_COLD_STORE_EXECUTE_DEBUG=y, which in turn enables assert_can_be_cold(). I have added check for a NULL pointer in a few places in Zephyr scheduler source code. That fixed the problem:
PR# 9947

[lgirdwood]

It's caused by enabling CONFIG_COLD_STORE_EXECUTE_DEBUG=y, which in turn enables assert_can_be_cold(). I have added check for a NULL pointer in a few places in Zephyr scheduler source code. That fixed the problem: PR# 9947

@lyakh can you check this - platforms with no cold storage should not crash here. Thanks !

[andrew-mtk]

@lyakh, I run more tests and found that platform_init() for MTK platforms call ipc_init() before scheduler_init_ll(). If I reverse the order, there is no crash.
I've pushed an updated patch.

[lgirdwood]

@andrew-mtk can we close ?