Kernel panic in WLAN driver

[sshane]

My comma 3X just froze and reboot after parking. I first noticed it by my car throwing a fault on the dash since I guess the screen froze too (didn't show any alerts, went to black after a few seconds).

https://connect.comma.ai/d9b97c1d3b8c39b2/00000164--129ea9d3d4

After reboot: https://connect.comma.ai/d9b97c1d3b8c39b2/00000165--1cfbe9210b

few kernel oops/bugs in the bootlog: 000000d1--254ca472f6.zst

[adeebshihadeh]

Confirmed it's not a regression so removing from the milestone

[sunnyhaibin]

@adeebshihadeh Some users using sunnypilot on the comma 4 is seeing a similar issue from time to time, but mostly experienced it while engaged. Could the root cause be related?

• https://connect.comma.ai/208f95b6156cdcad/00000013--05ed2d2046
• https://connect.comma.ai/208f95b6156cdcad/00000015--be3ce37614

Forum topic: https://community.sunnypilot.ai/t/c4-crash-and-reboot/2616?u=sunnyhaibin

[jyoung8607]

Two more incidents here: #37044 (comment)

[adeebshihadeh]

FYI, we're working on this as part of some larger AGNOS work (e.g. commaai/agnos-builder#520). Currently, the Qualcomm WiFi stack is massive and sprawls all over the place.

With Void and mainline kernel, we should have something much cleaner and easier to reason about. I suspect this is a bug in either the Qualcomm WiFi firmware or kernel driver that gets exacerbated by both environmental and userspace changes (e.g. NetworkManager changing how it roams).

[michael-was-taken]

Two more incidents here: #37044 (comment)

Thank you for looking into this one @jyoung8607
I was connected to WiFi via mobile hotspot from my cell phone. After the second instance I got kind of scared to run nightly again so I switched over to cd210-model for a few days to see if it would happen again and everything was fine. After cd210-model was gone I switched over to nightly-dev and didn’t experience issues. Now I’m on nightly again and I have not experienced the issue again, the crazy low speed ping pongs or the comma 4 crashing.

I did notice my WiFi was acting weird a couple of nights ago, I was trying to connect to my hotspot and it was connect then immediately disconnect. Not sure if that’s also related to the driver.

[jyoung8607]

Adding pstore contents to aid in GitHub issue searching, and to help people understand if they're affected.

The issue results in a kernel panic. While onroad, that'll usually manifest as the display image freezing for a few seconds while lots of alerts/faults appear on the car. The device will generally reboot on its own, but the car may need to be restarted to clear the car-side faults.

If you see what appears to be a kernel panic, it's important to let the device finish rebooting and going onroad, and then restart the car if you need to. This allows openpilot to save the ramoops buffer to flash storage, for later automatic upload with the rest of your driving logs.

Not all kernel panics are this bug, but if you experience this bug, expect to see this in the uploaded bootlog for the route after the route with the actual panic, the one the device rebooted into. You can see the bootlog identifier in the route initData.

[ 2457.804477] arm-smmu 15000000.apps-smmu: Unhandled context fault: iova=0xa3aec0d8, fsr=0x40000402, fsynr=0x340003, cb=5
[ 2457.804512] arm-smmu 15000000.apps-smmu: FAR    = 00000000a3aec0d8
[ 2457.804533] arm-smmu 15000000.apps-smmu: FSR    = 40000402 [TF SS ]
[ 2457.804554] arm-smmu 15000000.apps-smmu: soft iova-to-phys=0x0000000000000000
[ 2457.804573] arm-smmu 15000000.apps-smmu: SOFTWARE TABLE WALK FAILED! Looks like 15000000.apps-smmu accessed an unmapped address!
[ 2457.804594] arm-smmu 15000000.apps-smmu: hard iova-to-phys (ATOS) failed
[ 2457.804613] arm-smmu 15000000.apps-smmu: SID=0x40
[ 2457.804632] arm-smmu 15000000.apps-smmu: Unhandled arm-smmu context fault!
[ 2457.804675] Kernel BUG at ffffff92f967fc68 [verbose debug info unavailable]
[ 2457.804701] ------------[ cut here ]------------
[ 2457.804714] Kernel BUG at ffffff92f967fc68 [verbose debug info unavailable]
[ 2457.804728] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP
[ 2457.804742] Modules linked in:
[ 2457.804769] CPU: 1 PID: 201 Comm: irq/170-arm-smm Not tainted 4.9.103 #1
[ 2457.804781] Hardware name: comma mici (DT)
[ 2457.804795] task: fffffff4f2719c00 task.stack: fffffff4f1bbc000
[ 2457.804816] PC is at arm_smmu_context_fault+0x5a0/0x5a8
[ 2457.804831] LR is at arm_smmu_context_fault+0x5a0/0x5a8
[ 2457.804844] pc : [<ffffff92f967fc68>] lr : [<ffffff92f967fc68>] pstate: 40c00145
[ 2457.804856] sp : fffffff4f1bbfcb0
[ 2457.804869] x29: fffffff4f1bbfcd0 x28: ffffff80100c5058 
[ 2457.804899] x27: fffffff4f2143400 x26: ffffff8000000040 
[ 2457.804928] x25: 0000000000000000 x24: 00000000a3aec0d8 
[ 2457.804957] x23: ffffff80100c5000 x22: 0000000040000402 
[ 2457.804987] x21: fffffff4f2143530 x20: ffffff92fb908788 
[ 2457.805016] x19: fffffff4f36f4a18 x18: 0000000000000010 
[ 2457.805045] x17: 0000007f98451d20 x16: ffffff92f924a9e8 
[ 2457.805074] x15: ffffffffffffffff x14: 6620747865746e6f 
[ 2457.805104] x13: 6320756d6d732d6d x12: 72612064656c646e 
[ 2457.805133] x11: 00000000000008e0 x10: 6d732d737070612e 
[ 2457.805162] x9 : 0000000000003ff0 x8 : 0000000000003fff 
[ 2457.805191] x7 : fffffff4fdc53090 x6 : 0000000004fdd16e 
[ 2457.805220] x5 : 00ffffffffffffff x4 : ffffff92fb909000 
[ 2457.805249] x3 : ffffff92fb909628 x2 : f2a309ebcca47000 
[ 2457.805278] x1 : f2a309ebcca47000 x0 : 0000000000000000 
[ 2457.805308] 
[ 2457.805308] PC: 0xffffff92f967fc28:
[ 2457.805321] fc28  94060ca6 17ffff6e b000a141 91270021 94060ca2 17fffff7 97e8b4ea f9400260
[ 2457.805416] fc48  b000a141 911fa021 94060c9c d4210000 f9400260 b000a141 9127e021 94060c97
[ 2457.805509] fc68  d4210000 d503201f a9bd7bfd 910003fd a90153f3 f90013f5 aa0103f4 9101c033
[ 2457.805601] fc88  aa0003f5 aa1e03e0 d503201f aa1303e1 52800040 97ecc897 14000001 d503201f
[ 2457.805694] 
[ 2457.805694] LR: 0xffffff92f967fc28:
[ 2457.805708] fc28  94060ca6 17ffff6e b000a141 91270021 94060ca2 17fffff7 97e8b4ea f9400260
[ 2457.805801] fc48  b000a141 911fa021 94060c9c d4210000 f9400260 b000a141 9127e021 94060c97
[ 2457.805898] fc68  d4210000 d503201f a9bd7bfd 910003fd a90153f3 f90013f5 aa0103f4 9101c033
[ 2457.805993] fc88  aa0003f5 aa1e03e0 d503201f aa1303e1 52800040 97ecc897 14000001 d503201f
[ 2457.806087] 
[ 2457.806087] SP: 0xfffffff4f1bbfc70:
[ 2457.806100] fc70  f967fc68 ffffff92 f1bbfcb0 fffffff4 f967fc68 ffffff92 40c00145 00000000
[ 2457.806192] fc90  fb909000 ffffff92 ffffffff 00ffffff ffffffff ffffffff fdc53090 fffffff4
[ 2457.806285] fcb0  faaf1208 ffffff92 faaf1208 ffffff92 faaa87d8 ffffff92 faaf1208 ffffff92
[ 2457.806377] fcd0  f1bbfd70 fffffff4 f9125810 ffffff92 f217de00 fffffff4 f3447200 fffffff4
[ 2457.806470] Process irq/170-arm-smm (pid: 201, stack limit = 0xfffffff4f1bbc000)
[ 2457.806484] Call trace:
[ 2457.806498] Exception stack(0xfffffff4f1bbfab0 to 0xfffffff4f1bbfbe0)
[ 2457.806512] faa0:                                   fffffff4f36f4a18 0000007fffffffff
[ 2457.806527] fac0: fffffff4f1bbfcb0 ffffff92f967fc68 0000000040c00145 000000000000003d
[ 2457.806541] fae0: 0000000000000000 ffffff92f914c3f4 fffffff4f1bbfb30 ffffff92f919d624
[ 2457.806555] fb00: ffffff92fb2ea700 ffffff92fb2ea708 fffffff4f1bbfb40 ffffff92f908ebbc
[ 2457.806570] fb20: ffffff92fb908788 f2a309ebcca47000 fffffff4f1bbfbe0 ffffff92f9802c1c
[ 2457.806584] fb40: fffffff4f34a6010 ffffff92faa34cc8 fffffff4f1bbfc48 0000000040000402
[ 2457.806598] fb60: fffffff4f1bbfbe0 fffffff4f1bbfbe0 fffffff4f1bbfbb0 f2a309ebcca47000
[ 2457.806612] fb80: 0000000000000000 f2a309ebcca47000 f2a309ebcca47000 ffffff92fb909628
[ 2457.806617] icnss: Received force error fatal request from FW
[ 2457.806632] fba0: ffffff92fb909000 00ffffffffffffff 0000000004fdd16e fffffff4fdc53090
[ 2457.806647] fbc0: 0000000000003fff 0000000000003ff0 6d732d737070612e 00000000000008e0
[ 2457.806664] [<ffffff92f967fc68>] arm_smmu_context_fault+0x5a0/0x5a8
[ 2457.806684] [<ffffff92f9125810>] irq_thread_fn+0x30/0x70
[ 2457.806698] [<ffffff92f9125b48>] irq_thread+0x150/0x1e0
[ 2457.806717] [<ffffff92f90d2a90>] kthread+0x100/0x108
[ 2457.806737] [<ffffff92f9083f00>] ret_from_fork+0x10/0x50
[ 2457.806753] Code: f9400260 b000a141 9127e021 94060c97 (d4210000) 
[ 2457.806775] ---[ end trace a5decdb2ccdd65ed ]---
[ 2457.825356] Unable to handle kernel paging request at virtual address ffffffffffffffd8
[ 2457.825379] pgd = fffffff4cfc87000
[ 2457.825393] [ffffffffffffffd8] *pgd=00000001640c4003, *pud=00000001640c4003, *pmd=0000000000000000
[ 2457.825464] ------------[ cut here ]------------
[ 2457.825477] Kernel BUG at ffffff92f90d374c [verbose debug info unavailable]
[ 2457.825491] Internal error: Oops: 96000005 [#2] PREEMPT SMP
[ 2457.825506] Modules linked in:
[ 2457.825532] CPU: 1 PID: 201 Comm: irq/170-arm-smm Tainted: G      D         4.9.103 #1
[ 2457.825546] Hardware name: comma mici (DT)
[ 2457.825559] task: fffffff4f2719c00 task.stack: fffffff4f1bbc000
[ 2457.825581] PC is at kthread_data+0x24/0x30
[ 2457.825598] LR is at irq_thread_dtor+0x28/0xc0
[ 2457.825612] pc : [<ffffff92f90d374c>] lr : [<ffffff92f9125960>] pstate: 604003c5
[ 2457.825625] sp : fffffff4f1bbf910
[ 2457.825638] x29: fffffff4f1bbf910 x28: fffffff4f2719c00 
[ 2457.825668] x27: fffffff4f2143400 x26: ffffff8000000040 
[ 2457.825698] x25: ffffff92fb91e000 x24: ffffff92fbbf5108 
[ 2457.825727] x23: 00000000000003c0 x22: ffffff92fbbf6ad0 
[ 2457.825757] x21: fffffff4f2719c00 x20: fffffff4f271a358 
[ 2457.825787] x19: fffffff4f2719c00 x18: 00000000000781f5 
[ 2457.825817] x17: 00000000000002c6 x16: 0000000000000009 
[ 2457.825847] x15: 00000000002ac5ff x14: fffffff4fdc5b318 
[ 2457.825876] x13: 00000000002ac5ff x12: 0000000000000000 
[ 2457.825906] x11: fffffff4f6c10800 x10: 0000000000000018 
[ 2457.825936] x9 : 0000000000003ff0 x8 : 0000000000003fff 
[ 2457.825966] x7 : fffffff4fdc53090 x6 : fffffff4fdc53090 
[ 2457.825996] x5 : fffffff4f1bbf910 x4 : fffffff4f271a38c 
[ 2457.826025] x3 : 0000000000000000 x2 : ffffff92f9125938 
[ 2457.826055] x1 : fffffff4f1bbfe08 x0 : 0000000000000000 
[ 2457.826085] 
[ 2457.826085] PC: 0xffffff92f90d370c:
[ 2457.826099] 370c  a8c27bfd d65f03c0 d4210000 17fffffc d4210000 17fffff8 d503201f a9be7bfd
[ 2457.826196] 372c  910003fd f9000bf3 aa0003f3 aa1e03e0 d503201f f942e660 f9400bf3 a8c27bfd
[ 2457.826291] 374c  f85d8000 d65f03c0 d503201f a9bd7bfd 910003fd a90153f3 aa0003f4 aa1e03e0
[ 2457.826385] 376c  d503201f b00141b3 911e2273 f942e681 f9400260 f90017e0 d2800000 d2800102
[ 2457.826480] 
[ 2457.826480] LR: 0xffffff92f9125920:
[ 2457.826494] 5920  52800061 97ff8e81 f9400bf3 a8c27bfd d65f03c0 d503201f a9be7bfd 910003fd
[ 2457.826589] 5940  a90153f3 aa1e03e0 d503201f d5384113 b9402e60 36100420 aa1303e0 97feb773
[ 2457.826684] 5960  aa0003f4 b944fa62 911b0261 9000c800 911a6000 b9403a83 9402b97c b9403a80
[ 2457.826778] 5980  97fff634 aa0003f3 91010281 52800000 940ed2c8 350001c0 b9407660 36280120
[ 2457.826875] 
[ 2457.826875] SP: 0xfffffff4f1bbf8d0:
[ 2457.826888] f8d0  f9125960 ffffff92 f1bbf910 fffffff4 f90d374c ffffff92 604003c5 00000000
[ 2457.826985] f8f0  f1bbf910 fffffff4 f9125960 ffffff92 ffffffff 0000007f fa4cce94 ffffff92
[ 2457.827082] f910  f1bbf930 fffffff4 f90d0614 ffffff92 00000000 00000000 f271a358 fffffff4
[ 2457.827177] f930  f1bbf960 fffffff4 f90b300c ffffff92 f2719c00 fffffff4 f1bbf9c8 fffffff4
[ 2457.827273] Process irq/170-arm-smm (pid: 201, stack limit = 0xfffffff4f1bbc000)
[ 2457.827287] Call trace:
[ 2457.827302] Exception stack(0xfffffff4f1bbf720 to 0xfffffff4f1bbf850)
[ 2457.827317] f720: fffffff4f2719c00 0000007fffffffff fffffff4f1bbf910 ffffff92f90d374c
[ 2457.827332] f740: 00000000604003c5 ffffff92f90edc58 fffffff4f1bbf7b0 ffffff92fa4cd044
[ 2457.827347] f760: 00000000000003c0 fffffff4fdc53db8 0000000000000001 ffffff92fb2e6dc0
[ 2457.827361] f780: fffffff4f1bbf7b0 ffffff92f908ebbc ffffff92fb908788 ffffff92fa4ccc24
[ 2457.827376] f7a0: fffffff4f1bbf7d0 ffffff92f908ebbc ffffff92fb908788 ffffff92fa4cd044
[ 2457.827391] f7c0: ffffff92fa4cd044 00000000000003dd fffffff4f1bbf830 f2a309ebcca47000
[ 2457.827405] f7e0: 0000000000000000 fffffff4f1bbfe08 ffffff92f9125938 0000000000000000
[ 2457.827420] f800: fffffff4f271a38c fffffff4f1bbf910 fffffff4fdc53090 fffffff4fdc53090
[ 2457.827434] f820: 0000000000003fff 0000000000003ff0 0000000000000018 fffffff4f6c10800
[ 2457.827449] f840: 0000000000000000 00000000002ac5ff
[ 2457.827465] [<ffffff92f90d374c>] kthread_data+0x24/0x30
[ 2457.827479] [<ffffff92f90d0614>] task_work_run+0xc4/0xe0
[ 2457.827498] [<ffffff92f90b300c>] do_exit+0x34c/0x9b8
[ 2457.827515] [<ffffff92f908bc88>] die+0x240/0x258
[ 2457.827530] [<ffffff92f908bcec>] bug_handler.part.2+0x4c/0x78
[ 2457.827545] [<ffffff92f908bd54>] bug_handler+0x3c/0x48
[ 2457.827560] [<ffffff92f9085204>] brk_handler+0x9c/0xd0
[ 2457.827575] [<ffffff92f9081874>] do_debug_exception+0xa4/0x170
[ 2457.827588] Exception stack(0xfffffff4f1bbfab0 to 0xfffffff4f1bbfbe0)
[ 2457.827602] faa0:                                   fffffff4f36f4a18 0000007fffffffff
[ 2457.827617] fac0: fffffff4f1bbfcb0 ffffff92f967fc68 0000000040c00145 000000000000003d
[ 2457.827631] fae0: 0000000000000000 ffffff92f914c3f4 fffffff4f1bbfb30 ffffff92f919d624
[ 2457.827646] fb00: ffffff92fb2ea700 ffffff92fb2ea708 fffffff4f1bbfb40 ffffff92f908ebbc
[ 2457.827661] fb20: ffffff92fb908788 f2a309ebcca47000 fffffff4f1bbfbe0 ffffff92f9802c1c
[ 2457.827676] fb40: fffffff4f34a6010 ffffff92faa34cc8 fffffff4f1bbfc48 0000000040000402
[ 2457.827691] fb60: fffffff4f1bbfbe0 fffffff4f1bbfbe0 fffffff4f1bbfbb0 f2a309ebcca47000
[ 2457.827705] fb80: 0000000000000000 f2a309ebcca47000 f2a309ebcca47000 ffffff92fb909628
[ 2457.827720] fba0: ffffff92fb909000 00ffffffffffffff 0000000004fdd16e fffffff4fdc53090
[ 2457.827734] fbc0: 0000000000003fff 0000000000003ff0 6d732d737070612e 00000000000008e0
[ 2457.827749] [<ffffff92f9083660>] el1_dbg+0x18/0x74
[ 2457.827764] [<ffffff92f9125810>] irq_thread_fn+0x30/0x70
[ 2457.827778] [<ffffff92f9125b48>] irq_thread+0x150/0x1e0
[ 2457.827793] [<ffffff92f90d2a90>] kthread+0x100/0x108
[ 2457.827807] [<ffffff92f9083f00>] ret_from_fork+0x10/0x50
[ 2457.827824] Code: d503201f f942e660 f9400bf3 a8c27bfd (f85d8000) 
[ 2457.827845] ---[ end trace a5decdb2ccdd65ee ]---
[ 2457.847121] Fixing recursive fault but reboot is needed!
[ 2457.947788] Fatal error on the modem.
[ 2457.947836] modem subsystem failure reason: err_qdi.c:456:EF:wlan_process:1:cmnos_thread.c:3900:Asserted in whal_interrupt_wifi_ip02.c:whalGetPendingInterrupts:499.
[ 2457.947844] subsys-restart: subsystem_restart_dev(): Restart sequence requested for modem, restart_level = RELATED.
[ 2457.949817] R0: [kworker/0:1][47276354523] [01:53:53.228879]  wlan: [54991:I :HDD] wlan_hdd_pld_uevent: 1402: pld event 1
[ 2457.950593] ipa-wan ipa3_ssr_notifier_cb:2692 IPA received MPSS BEFORE_SHUTDOWN
[ 2457.955777] ipa client_lock_unlock_cb:692 Invalid EP
[ 2457.955814] ipa client_lock_unlock_cb:692 Invalid EP
[ 2457.955914] rmnet_ipa started deinitialization
[ 2457.955934] ipa ipa3_teardown_sys_pipe:1113 bad parm.
[ 2457.955957] ipa-wan ipa3_wwan_remove:2516 Failed to teardown IPA->APPS pipe
[ 2457.956056] ipa ipa3_teardown_sys_pipe:1113 bad parm.
[ 2457.956164] ipa-wan ipa3_wwan_remove:2521 Failed to teardown APPS->IPA pipe
[ 2457.963276] [RMNET:HI] rmnet_config_notify_cb(): Kernel is trying to unregister rmnet_ipa0
[ 2458.050879] spi_geni 880000.spi: Xfer[len 1275 tx 0000000000000000 rx 0000000000000000 n 8] timed out.
[ 2458.052154] [RMNET:HI] rmnet_config_notify_cb(): Kernel is trying to unregister rmnet_ipa0
[ 2458.053125] ipa-wan ipa3_del_dflt_wan_rt_tables:413 Deleting Route hdl:(0x7) with ip type: 0
[ 2458.057583] ipa-wan ipa3_del_dflt_wan_rt_tables:422 Deleting Route hdl:(0x9) with ip type: 1
[ 2458.059468] rmnet_ipa completed deinitialization
[ 2458.059625] ipa-wan ipa3_ssr_notifier_cb:2708 IPA BEFORE_SHUTDOWN handling is complete
[ 2458.059663] icnss: PD service down, pd_state: 2, state: 0x4d8f: cause: Root PD crashed
[ 2458.059774] icnss: ASSERT at line 2460
[ 2458.059811] ------------[ cut here ]------------
[ 2458.059825] Kernel BUG at ffffff92f95d1e3c [verbose debug info unavailable]
[ 2458.059849] ------------[ cut here ]------------
[ 2458.059862] Kernel BUG at ffffff92f95d1e3c [verbose debug info unavailable]
[ 2458.059875] Internal error: Oops - BUG: 0 [#3] PREEMPT SMP
[ 2458.059890] Modules linked in:
[ 2458.059917] CPU: 0 PID: 84 Comm: kworker/u16:1 Tainted: G      D         4.9.103 #1
[ 2458.059930] Hardware name: comma mici (DT)
[ 2458.059955] Workqueue: icnss_driver_event icnss_driver_event_work
[ 2458.059977] task: fffffff4f4664600 task.stack: fffffff4f3858000
[ 2458.059993] PC is at icnss_driver_event_work+0x131c/0x18d0
[ 2458.060007] LR is at icnss_driver_event_work+0x16c4/0x18d0
[ 2458.060020] pc : [<ffffff92f95d1e3c>] lr : [<ffffff92f95d21e4>] pstate: 40c00145
[ 2458.060033] sp : fffffff4f385bc60
[ 2458.060046] x29: fffffff4f385bc60 x28: ffffff92fc45f000 
[ 2458.060076] x27: 0000000000000000 x26: ffffff92fb908000 
[ 2458.060106] x25: fffffff4e43d2b80 x24: dead000000000100 
[ 2458.060135] x23: ffffff92fa566108 x22: fffffff4f2731818 
[ 2458.060165] x21: ffffff92fbb71eb8 x20: ffffff92fc45f7b0 
[ 2458.060194] x19: ffffff92faa899d8 x18: 0000000000000001 
[ 2458.060224] x17: 0000007fa27b5e70 x16: ffffff92fbbf9a48 
[ 2458.060253] x15: ffffffffffffffff x14: ffffff92fb908788 
[ 2458.060282] x13: fffffff4f385bc21 x12: fffffff4f385bb50 
[ 2458.060312] x11: 0000000000000989 x10: fffffff4f385b990 
[ 2458.060342] x9 : 0000000000003ff0 x8 : 0000000000003fff 
[ 2458.060373] x7 : fffffff4fdc3c090 x6 : fffffff4fdc3c090 
[ 2458.060403] x5 : fffffff4f385ba40 x4 : ffffff92fb909000 
[ 2458.060432] x3 : ffffff92fb909628 x2 : f2a309ebcca47000 
[ 2458.060462] x1 : 0000000000000000 x0 : 0000000000000000 
[ 2458.060492] 
[ 2458.060492] PC: 0xffffff92f95d1dfc:
[ 2458.060506] 1dfc  11000421 b904f101 35000682 f94033e1 9000a5d3 91276273 5280aaa2 aa1303e0
[ 2458.060604] 1e1c  97f00853 f9400680 b40000c0 9000a902 aa1303e1 5280aaa3 91082042 97ef84bc
[ 2458.060698] 1e3c  d4210000 b94097e3 d000a5d6 9120c2d6 f000a301 aa1603e0 91332021 f90033e1
[ 2458.060793] 1e5c  97f00843 f9400680 b40000c0 295213e3 9000a902 aa1603e1 91082042 97ef84ac
[ 2458.060890] 
[ 2458.060890] LR: 0xffffff92f95d21a4:
[ 2458.060903] 21a4  f943db88 17fffe8a f000a5b3 91276273 aa1303e0 d000a301 52813382 91332021
[ 2458.061012] 21c4  97f00769 f9400680 b4ffe380 aa1303e1 f000a8e2 52813383 91082042 97ef83d2
[ 2458.061115] 21e4  17ffff16 b000a5d3 91152273 d000a301 91332021 f90033e1 aa1303e0 2a1603e2
[ 2458.061226] 2204  97f00759 f9400680 b4ffce60 2a1603e3 aa1303e1 f000a8e2 91082042 97ef83c2
[ 2458.061322] 
[ 2458.061322] SP: 0xfffffff4f385bc20:
[ 2458.061335] bc20  f95d21e4 ffffff92 f385bc60 fffffff4 f95d1e3c ffffff92 40c00145 00000000
[ 2458.061435] bc40  fb909000 ffffff92 ffffffff 00ffffff ffffffff ffffffff fdc3c090 fffffff4
[ 2458.061531] bc60  f385bd70 fffffff4 f90cb7cc ffffff92 f2731a50 fffffff4 f47f1d00 fffffff4
[ 2458.061626] bc80  f1bc0100 fffffff4 00000000 00000000 f6c17400 fffffff4 fbb5ee08 ffffff92
[ 2458.061722] Process kworker/u16:1 (pid: 84, stack limit = 0xfffffff4f3858000)
[ 2458.061736] Call trace:
[ 2458.061750] Exception stack(0xfffffff4f385ba60 to 0xfffffff4f385bb90)
[ 2458.061765] ba60: ffffff92faa899d8 0000007fffffffff fffffff4f385bc60 ffffff92f95d1e3c
[ 2458.061779] ba80: 0000000040c00145 000000000000003d fffffff4e43d2b80 fffffff4f385bc60
[ 2458.061794] baa0: fffffff4f385bc30 f2a309ebcca47000 fffffff4f385bc60 ffffff92f95d21e4
[ 2458.061809] bac0: ffffff92faa899d8 ffffff92fc45f7b0 ffffff92fbb71eb8 fffffff4f2731818
[ 2458.061823] bae0: fffffff4f385bc60 fffffff4f385bc60 fffffff4f385bc30 00000000ffffffd0
[ 2458.061838] bb00: fffffff4f385bc60 fffffff4f385bc60 fffffff4f385bc30 00000000ffffffd0
[ 2458.061852] bb20: daf6080131023102 f2a309ebcca47000 0000000000000000 0000000000000000
[ 2458.061867] bb40: f2a309ebcca47000 ffffff92fb909628 ffffff92fb909000 fffffff4f385ba40
[ 2458.061881] bb60: fffffff4fdc3c090 fffffff4fdc3c090 0000000000003fff 0000000000003ff0
[ 2458.061895] bb80: fffffff4f385b990 0000000000000989
[ 2458.061910] [<ffffff92f95d1e3c>] icnss_driver_event_work+0x131c/0x18d0
[ 2458.061928] [<ffffff92f90cb7cc>] process_one_work+0x20c/0x4d8
[ 2458.061943] [<ffffff92f90cbae8>] worker_thread+0x50/0x4d0
[ 2458.061958] [<ffffff92f90d2a90>] kthread+0x100/0x108
[ 2458.061975] [<ffffff92f9083f00>] ret_from_fork+0x10/0x50
[ 2458.061990] Code: aa1303e1 5280aaa3 91082042 97ef84bc (d4210000) 
[ 2458.062011] ---[ end trace a5decdb2ccdd65ef ]---
[ 2458.092886] Unable to handle kernel paging request at virtual address ffffffffffffffd8
[ 2458.092903] pgd = fffffff4d2a15000
[ 2458.092908] [ffffffffffffffd8] *pgd=000000014ff88003, *pud=000000014ff88003, *pmd=0000000000000000