Apr 14 07:31:24 14333 kernel: [843556.475850] INFO: rcu_sched detected stalls on CPUs/tasks: Apr 14 07:31:24 14333 kernel: [843556.475854] 0-...!: (1 GPs behind) idle=ffa/1/0 softirq=7326149/7326150 fqs=0 Apr 14 07:31:24 14333 kernel: [843556.475855] (detected by 1, t=34565 jiffies, g=5573741, c=5573740, q=262) Apr 14 07:31:24 14333 kernel: [843556.475859] Sending NMI from CPU 1 to CPUs 0: Apr 14 07:31:24 14333 kernel: [843556.476867] NMI backtrace for cpu 0 Apr 14 07:31:24 14333 kernel: [843556.476868] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O L 4.15.0-47-generic #50-Ubuntu Apr 14 07:31:24 14333 kernel: [843556.476869] Hardware name: BHYVE, BIOS 1.00 03/14/2014 Apr 14 07:31:24 14333 kernel: [843556.476869] RIP: 0010:io_serial_in+0x18/0x20 Apr 14 07:31:24 14333 kernel: [843556.476869] RSP: 0018:ffff9b1a3fc03c10 EFLAGS: 00000002 Apr 14 07:31:24 14333 kernel: [843556.476870] RAX: 0000000084a02960 RBX: ffffffff85efee20 RCX: 0000000000000000 Apr 14 07:31:24 14333 kernel: [843556.476871] RDX: 00000000000003fd RSI: 0000000000000005 RDI: ffffffff85efee20 Apr 14 07:31:24 14333 kernel: [843556.476872] RBP: ffff9b1a3fc03c10 R08: 0000000000000000 R09: 0000000000001d7d Apr 14 07:31:24 14333 kernel: [843556.476872] R10: 000000000000003e R11: ffffffff85d5380e R12: 0000000000002710 Apr 14 07:31:24 14333 kernel: [843556.476873] R13: 0000000000000020 R14: ffffffff85d53818 R15: ffffffff85efee20 Apr 14 07:31:24 14333 kernel: [843556.476873] FS: 0000000000000000(0000) GS:ffff9b1a3fc00000(0000) knlGS:0000000000000000 Apr 14 07:31:24 14333 kernel: [843556.476874] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Apr 14 07:31:24 14333 kernel: [843556.476874] CR2: 000000c4202c5000 CR3: 000000002140a003 CR4: 00000000000606f0 Apr 14 07:31:24 14333 kernel: [843556.476874] Call Trace: Apr 14 07:31:24 14333 kernel: [843556.476875] Apr 14 07:31:24 14333 kernel: [843556.476875] wait_for_xmitr+0x47/0xb0 Apr 14 07:31:24 14333 kernel: [843556.476875] serial8250_console_putchar+0x1c/0x40 Apr 14 07:31:24 14333 kernel: [843556.476876] ? wait_for_xmitr+0xb0/0xb0 Apr 14 07:31:24 14333 kernel: [843556.476876] uart_console_write+0x4c/0x60 Apr 14 07:31:24 14333 kernel: [843556.476876] serial8250_console_write+0x10a/0x2a0 Apr 14 07:31:25 14333 kernel: [843556.476877] ? msg_print_text+0x80/0x100 Apr 14 07:31:25 14333 kernel: [843556.476877] univ8250_console_write+0x25/0x30 Apr 14 07:31:25 14333 kernel: [843556.476877] console_unlock+0x34b/0x4e0 Apr 14 07:31:25 14333 kernel: [843556.476878] vprintk_emit+0x333/0x3a0 Apr 14 07:31:25 14333 kernel: [843556.476878] vprintk_default+0x29/0x50 Apr 14 07:31:25 14333 kernel: [843556.476878] vprintk_func+0x27/0x60 Apr 14 07:31:25 14333 kernel: [843556.476879] printk+0x52/0x6e Apr 14 07:31:25 14333 kernel: [843556.476879] rcu_check_callbacks+0x6b6/0x900 Apr 14 07:31:25 14333 kernel: [843556.476879] ? update_wall_time+0x474/0x6f0 Apr 14 07:31:25 14333 kernel: [843556.476880] ? tick_sched_do_timer+0x50/0x50 Apr 14 07:31:25 14333 kernel: [843556.476880] update_process_times+0x2f/0x60 Apr 14 07:31:25 14333 kernel: [843556.476880] tick_sched_handle+0x26/0x70 Apr 14 07:31:25 14333 kernel: [843556.476881] ? tick_sched_do_timer+0x3f/0x50 Apr 14 07:31:25 14333 kernel: [843556.476881] tick_sched_timer+0x39/0x80 Apr 14 07:31:25 14333 kernel: [843556.476881] __hrtimer_run_queues+0xdf/0x230 Apr 14 07:31:25 14333 kernel: [843556.476882] hrtimer_interrupt+0xa0/0x1d0 Apr 14 07:31:25 14333 kernel: [843556.476882] smp_apic_timer_interrupt+0x6f/0x130 Apr 14 07:31:25 14333 kernel: [843556.476882] apic_timer_interrupt+0x84/0x90 Apr 14 07:31:25 14333 kernel: [843556.476883] Apr 14 07:31:25 14333 kernel: [843556.476883] RIP: 0010:native_safe_halt+0x6/0x10 Apr 14 07:31:25 14333 kernel: [843556.476883] RSP: 0018:ffffffff85803e28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11 Apr 14 07:31:25 14333 kernel: [843556.476884] RAX: ffffffff84da0060 RBX: 0000000000000000 RCX: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.476885] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.476885] RBP: ffffffff85803e28 R08: 0000000000000002 R09: 0000000000000001 Apr 14 07:31:25 14333 kernel: [843556.476885] R10: ffffffff85803da0 R11: 0000000000000000 R12: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.476886] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.476886] ? __cpuidle_text_start+0x8/0x8 Apr 14 07:31:25 14333 kernel: [843556.476886] default_idle+0x20/0x100 Apr 14 07:31:25 14333 kernel: [843556.476887] arch_cpu_idle+0x15/0x20 Apr 14 07:31:25 14333 kernel: [843556.476887] default_idle_call+0x23/0x30 Apr 14 07:31:25 14333 kernel: [843556.476887] do_idle+0x172/0x1f0 Apr 14 07:31:25 14333 kernel: [843556.476888] cpu_startup_entry+0x73/0x80 Apr 14 07:31:25 14333 kernel: [843556.476888] rest_init+0xae/0xb0 Apr 14 07:31:25 14333 kernel: [843556.476888] start_kernel+0x4dc/0x4fd Apr 14 07:31:25 14333 kernel: [843556.476889] x86_64_start_reservations+0x24/0x26 Apr 14 07:31:25 14333 kernel: [843556.476889] x86_64_start_kernel+0x74/0x77 Apr 14 07:31:25 14333 kernel: [843556.476889] secondary_startup_64+0xa5/0xb0 Apr 14 07:31:25 14333 kernel: [843556.476890] Code: 48 89 e5 d3 e6 48 63 f6 48 03 77 10 8b 06 5d c3 0f 1f 00 0f 1f 44 00 00 0f b6 8f a1 00 00 00 8b 57 08 55 48 89 e5 d3 e6 01 f2 ec <0f> b6 c0 5d c3 0f 1f 00 0f 1f 44 00 00 0f b6 8f a1 00 00 00 89 Apr 14 07:31:25 14333 kernel: [843556.476904] rcu_sched kthread starved for 34565 jiffies! g5573741 c5573740 f0x0 RCU_GP_WAIT_FQS(3) ->state=0x402 ->cpu=1 Apr 14 07:31:25 14333 kernel: [843556.476905] rcu_sched I 0 8 2 0x80000000 Apr 14 07:31:25 14333 kernel: [843556.476908] Call Trace: Apr 14 07:31:25 14333 kernel: [843556.476917] __schedule+0x291/0x8a0 Apr 14 07:31:25 14333 kernel: [843556.476921] ? __switch_to_asm+0x34/0x70 Apr 14 07:31:25 14333 kernel: [843556.476923] ? __switch_to_asm+0x40/0x70 Apr 14 07:31:25 14333 kernel: [843556.476924] schedule+0x2c/0x80 Apr 14 07:31:25 14333 kernel: [843556.476926] schedule_timeout+0x15d/0x350 Apr 14 07:31:25 14333 kernel: [843556.476932] ? __next_timer_interrupt+0xe0/0xe0 Apr 14 07:31:25 14333 kernel: [843556.476934] rcu_gp_kthread+0x53a/0x960 Apr 14 07:31:25 14333 kernel: [843556.476938] kthread+0x121/0x140 Apr 14 07:31:25 14333 kernel: [843556.476940] ? rcu_note_context_switch+0x150/0x150 Apr 14 07:31:25 14333 kernel: [843556.476942] ? kthread_create_worker_on_cpu+0x70/0x70 Apr 14 07:31:25 14333 kernel: [843556.476944] ret_from_fork+0x35/0x40 Apr 14 07:31:25 14333 kernel: [843556.477839] NMI backtrace for cpu 0 Apr 14 07:31:25 14333 kernel: [843556.477839] CPU: 0 PID: 0 Comm: swapper/0 Tainted: P O L 4.15.0-47-generic #50-Ubuntu Apr 14 07:31:25 14333 kernel: [843556.477839] Hardware name: BHYVE, BIOS 1.00 03/14/2014 Apr 14 07:31:25 14333 kernel: [843556.477839] Call Trace: Apr 14 07:31:25 14333 kernel: [843556.477839] Apr 14 07:31:25 14333 kernel: [843556.477839] dump_stack+0x63/0x8b Apr 14 07:31:25 14333 kernel: [843556.477839] nmi_cpu_backtrace+0xca/0xd0 Apr 14 07:31:25 14333 kernel: [843556.477839] ? lapic_can_unplug_cpu+0xb0/0xb0 Apr 14 07:31:25 14333 kernel: [843556.477839] nmi_trigger_cpumask_backtrace+0xe6/0x130 Apr 14 07:31:25 14333 kernel: [843556.477839] arch_trigger_cpumask_backtrace+0x19/0x20 Apr 14 07:31:25 14333 kernel: [843556.477839] rcu_dump_cpu_stacks+0xa3/0xdb Apr 14 07:31:25 14333 kernel: [843556.477839] rcu_check_callbacks+0x6c6/0x900 Apr 14 07:31:25 14333 kernel: [843556.477839] ? update_wall_time+0x474/0x6f0 Apr 14 07:31:25 14333 kernel: [843556.477839] ? tick_sched_do_timer+0x50/0x50 Apr 14 07:31:25 14333 kernel: [843556.477839] update_process_times+0x2f/0x60 Apr 14 07:31:25 14333 kernel: [843556.477839] tick_sched_handle+0x26/0x70 Apr 14 07:31:25 14333 kernel: [843556.477839] ? tick_sched_do_timer+0x3f/0x50 Apr 14 07:31:25 14333 kernel: [843556.477839] tick_sched_timer+0x39/0x80 Apr 14 07:31:25 14333 kernel: [843556.477839] __hrtimer_run_queues+0xdf/0x230 Apr 14 07:31:25 14333 kernel: [843556.477839] hrtimer_interrupt+0xa0/0x1d0 Apr 14 07:31:25 14333 kernel: [843556.477839] smp_apic_timer_interrupt+0x6f/0x130 Apr 14 07:31:25 14333 kernel: [843556.477839] apic_timer_interrupt+0x84/0x90 Apr 14 07:31:25 14333 kernel: [843556.477839] Apr 14 07:31:25 14333 kernel: [843556.477839] RIP: 0010:native_safe_halt+0x6/0x10 Apr 14 07:31:25 14333 kernel: [843556.477839] RSP: 0018:ffffffff85803e28 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11 Apr 14 07:31:25 14333 kernel: [843556.477839] RAX: ffffffff84da0060 RBX: 0000000000000000 RCX: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.477839] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.477839] RBP: ffffffff85803e28 R08: 0000000000000002 R09: 0000000000000001 Apr 14 07:31:25 14333 kernel: [843556.477839] R10: ffffffff85803da0 R11: 0000000000000000 R12: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.477839] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 Apr 14 07:31:25 14333 kernel: [843556.477839] ? __cpuidle_text_start+0x8/0x8 Apr 14 07:31:25 14333 kernel: [843556.477839] default_idle+0x20/0x100 Apr 14 07:31:25 14333 kernel: [843556.477839] arch_cpu_idle+0x15/0x20 Apr 14 07:31:25 14333 kernel: [843556.477839] default_idle_call+0x23/0x30 Apr 14 07:31:25 14333 kernel: [843556.477839] do_idle+0x172/0x1f0 Apr 14 07:31:25 14333 kernel: [843556.477839] cpu_startup_entry+0x73/0x80 Apr 14 07:31:25 14333 kernel: [843556.477839] rest_init+0xae/0xb0 Apr 14 07:31:25 14333 kernel: [843556.477839] start_kernel+0x4dc/0x4fd Apr 14 07:31:25 14333 kernel: [843556.477839] x86_64_start_reservations+0x24/0x26 Apr 14 07:31:25 14333 kernel: [843556.477839] x86_64_start_kernel+0x74/0x77 Apr 14 07:31:25 14333 kernel: [843556.477839] secondary_startup_64+0xa5/0xb0