MI3 TD Kernel Source #3
Comments
|
May code has not been perfect. |
|
so we can help fix it :) |
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
Oct 10, 2014
commit 84cc8fd upstream. The current code makes the assumption that a cpu_base lock won't be held if the CPU corresponding to that cpu_base is offline, which isn't always true. If a hrtimer is not queued, then it will not be migrated by migrate_hrtimers() when a CPU is offlined. Therefore, the hrtimer's cpu_base may still point to a CPU which has subsequently gone offline if the timer wasn't enqueued at the time the CPU went down. Normally this wouldn't be a problem, but a cpu_base's lock is blindly reinitialized each time a CPU is brought up. If a CPU is brought online during the period that another thread is performing a hrtimer operation on a stale hrtimer, then the lock will be reinitialized under its feet, and a SPIN_BUG() like the following will be observed: <0>[ 28.082085] BUG: spinlock already unlocked on CPU#0, swapper/0/0 <0>[ 28.087078] lock: 0xc4780b40, value 0x0 .magic: dead4ead, .owner: <none>/-1, .owner_cpu: -1 <4>[ 42.451150] [<c0014398>] (unwind_backtrace+0x0/0x120) from [<c0269220>] (do_raw_spin_unlock+0x44/0xdc) <4>[ 42.460430] [<c0269220>] (do_raw_spin_unlock+0x44/0xdc) from [<c071b5bc>] (_raw_spin_unlock+0x8/0x30) <4>[ 42.469632] [<c071b5bc>] (_raw_spin_unlock+0x8/0x30) from [<c00a9ce0>] (__hrtimer_start_range_ns+0x1e4/0x4f8) <4>[ 42.479521] [<c00a9ce0>] (__hrtimer_start_range_ns+0x1e4/0x4f8) from [<c00aa014>] (hrtimer_start+0x20/0x28) <4>[ 42.489247] [<c00aa014>] (hrtimer_start+0x20/0x28) from [<c00e6190>] (rcu_idle_enter_common+0x1ac/0x320) <4>[ 42.498709] [<c00e6190>] (rcu_idle_enter_common+0x1ac/0x320) from [<c00e6440>] (rcu_idle_enter+0xa0/0xb8) <4>[ 42.508259] [<c00e6440>] (rcu_idle_enter+0xa0/0xb8) from [<c000f268>] (cpu_idle+0x24/0xf0) <4>[ 42.516503] [<c000f268>] (cpu_idle+0x24/0xf0) from [<c06ed3c0>] (rest_init+0x88/0xa0) <4>[ 42.524319] [<c06ed3c0>] (rest_init+0x88/0xa0) from [<c0c00978>] (start_kernel+0x3d0/0x434) As an example, this particular crash occurred when hrtimer_start() was executed on CPU #0. The code locked the hrtimer's current cpu_base corresponding to CPU #1. CPU #0 then tried to switch the hrtimer's cpu_base to an optimal CPU which was online. In this case, it selected the cpu_base corresponding to CPU MiCode#3. Before it could proceed, CPU #1 came online and reinitialized the spinlock corresponding to its cpu_base. Thus now CPU #0 held a lock which was reinitialized. When CPU #0 finally ended up unlocking the old cpu_base corresponding to CPU #1 so that it could switch to CPU MiCode#3, we hit this SPIN_BUG() above while in switch_hrtimer_base(). CPU #0 CPU #1 ---- ---- ... <offline> hrtimer_start() lock_hrtimer_base(base #1) ... init_hrtimers_cpu() switch_hrtimer_base() ... ... raw_spin_lock_init(&cpu_base->lock) raw_spin_unlock(&cpu_base->lock) ... <spin_bug> Solve this by statically initializing the lock. Signed-off-by: Michael Bohan <[email protected]> Link: http://lkml.kernel.org/r/[email protected] Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
Oct 10, 2014
commit 42a5cf4 upstream. An inactive timer's base can refer to a offline cpu's base. In the current code, cpu_base's lock is blindly reinitialized each time a CPU is brought up. If a CPU is brought online during the period that another thread is trying to modify an inactive timer on that CPU with holding its timer base lock, then the lock will be reinitialized under its feet. This leads to following SPIN_BUG(). <0> BUG: spinlock already unlocked on CPU#3, kworker/u:3/1466 <0> lock: 0xe3ebe000, .magic: dead4ead, .owner: kworker/u:3/1466, .owner_cpu: 1 <4> [<c0013dc4>] (unwind_backtrace+0x0/0x11c) from [<c026e794>] (do_raw_spin_unlock+0x40/0xcc) <4> [<c026e794>] (do_raw_spin_unlock+0x40/0xcc) from [<c076c160>] (_raw_spin_unlock+0x8/0x30) <4> [<c076c160>] (_raw_spin_unlock+0x8/0x30) from [<c009b858>] (mod_timer+0x294/0x310) <4> [<c009b858>] (mod_timer+0x294/0x310) from [<c00a5e04>] (queue_delayed_work_on+0x104/0x120) <4> [<c00a5e04>] (queue_delayed_work_on+0x104/0x120) from [<c04eae00>] (sdhci_msm_bus_voting+0x88/0x9c) <4> [<c04eae00>] (sdhci_msm_bus_voting+0x88/0x9c) from [<c04d8780>] (sdhci_disable+0x40/0x48) <4> [<c04d8780>] (sdhci_disable+0x40/0x48) from [<c04bf300>] (mmc_release_host+0x4c/0xb0) <4> [<c04bf300>] (mmc_release_host+0x4c/0xb0) from [<c04c7aac>] (mmc_sd_detect+0x90/0xfc) <4> [<c04c7aac>] (mmc_sd_detect+0x90/0xfc) from [<c04c2504>] (mmc_rescan+0x7c/0x2c4) <4> [<c04c2504>] (mmc_rescan+0x7c/0x2c4) from [<c00a6a7c>] (process_one_work+0x27c/0x484) <4> [<c00a6a7c>] (process_one_work+0x27c/0x484) from [<c00a6e94>] (worker_thread+0x210/0x3b0) <4> [<c00a6e94>] (worker_thread+0x210/0x3b0) from [<c00aad9c>] (kthread+0x80/0x8c) <4> [<c00aad9c>] (kthread+0x80/0x8c) from [<c000ea80>] (kernel_thread_exit+0x0/0x8) As an example, this particular crash occurred when CPU MiCode#3 is executing mod_timer() on an inactive timer whose base is refered to offlined CPU mitwo-dev#2. The code locked the timer_base corresponding to CPU mitwo-dev#2. Before it could proceed, CPU mitwo-dev#2 came online and reinitialized the spinlock corresponding to its base. Thus now CPU MiCode#3 held a lock which was reinitialized. When CPU MiCode#3 finally ended up unlocking the old cpu_base corresponding to CPU mitwo-dev#2, we hit the above SPIN_BUG(). CPU #0 CPU MiCode#3 CPU mitwo-dev#2 ------ ------- ------- ..... ...... <Offline> mod_timer() lock_timer_base spin_lock_irqsave(&base->lock) cpu_up(2) ..... ...... init_timers_cpu() .... ..... spin_lock_init(&base->lock) ..... spin_unlock_irqrestore(&base->lock) ...... <spin_bug> Allocation of per_cpu timer vector bases is done only once under "tvec_base_done[]" check. In the current code, spinlock_initialization of base->lock isn't under this check. When a CPU is up each time the base lock is reinitialized. Move base spinlock initialization under the check. Signed-off-by: Tirupathi Reddy <[email protected]> Link: http://lkml.kernel.org/r/[email protected] Signed-off-by: Thomas Gleixner <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
Oct 10, 2014
commit 058ebd0eba3aff16b144eabf4510ed9510e1416e upstream.
Jiri managed to trigger this warning:
[] ======================================================
[] [ INFO: possible circular locking dependency detected ]
[] 3.10.0+ #228 Tainted: G W
[] -------------------------------------------------------
[] p/6613 is trying to acquire lock:
[] (rcu_node_0){..-...}, at: [<ffffffff810ca797>] rcu_read_unlock_special+0xa7/0x250
[]
[] but task is already holding lock:
[] (&ctx->lock){-.-...}, at: [<ffffffff810f2879>] perf_lock_task_context+0xd9/0x2c0
[]
[] which lock already depends on the new lock.
[]
[] the existing dependency chain (in reverse order) is:
[]
[] -> MiCode#4 (&ctx->lock){-.-...}:
[] -> MiCode#3 (&rq->lock){-.-.-.}:
[] -> mitwo-dev#2 (&p->pi_lock){-.-.-.}:
[] -> #1 (&rnp->nocb_gp_wq[1]){......}:
[] -> #0 (rcu_node_0){..-...}:
Paul was quick to explain that due to preemptible RCU we cannot call
rcu_read_unlock() while holding scheduler (or nested) locks when part
of the read side critical section was preemptible.
Therefore solve it by making the entire RCU read side non-preemptible.
Also pull out the retry from under the non-preempt to play nice with RT.
Reported-by: Jiri Olsa <[email protected]>
Helped-out-by: Paul E. McKenney <[email protected]>
Signed-off-by: Peter Zijlstra <[email protected]>
Signed-off-by: Ingo Molnar <[email protected]>
Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
Oct 10, 2014
commit ea3768b4386a8d1790f4cc9a35de4f55b92d6442 upstream. We used to keep the port's char device structs and the /sys entries around till the last reference to the port was dropped. This is actually unnecessary, and resulted in buggy behaviour: 1. Open port in guest 2. Hot-unplug port 3. Hot-plug a port with the same 'name' property as the unplugged one This resulted in hot-plug being unsuccessful, as a port with the same name already exists (even though it was unplugged). This behaviour resulted in a warning message like this one: -------------------8<--------------------------------------- WARNING: at fs/sysfs/dir.c:512 sysfs_add_one+0xc9/0x130() (Not tainted) Hardware name: KVM sysfs: cannot create duplicate filename '/devices/pci0000:00/0000:00:04.0/virtio0/virtio-ports/vport0p1' Call Trace: [<ffffffff8106b607>] ? warn_slowpath_common+0x87/0xc0 [<ffffffff8106b6f6>] ? warn_slowpath_fmt+0x46/0x50 [<ffffffff811f2319>] ? sysfs_add_one+0xc9/0x130 [<ffffffff811f23e8>] ? create_dir+0x68/0xb0 [<ffffffff811f2469>] ? sysfs_create_dir+0x39/0x50 [<ffffffff81273129>] ? kobject_add_internal+0xb9/0x260 [<ffffffff812733d8>] ? kobject_add_varg+0x38/0x60 [<ffffffff812734b4>] ? kobject_add+0x44/0x70 [<ffffffff81349de4>] ? get_device_parent+0xf4/0x1d0 [<ffffffff8134b389>] ? device_add+0xc9/0x650 -------------------8<--------------------------------------- Instead of relying on guest applications to release all references to the ports, we should go ahead and unregister the port from all the core layers. Any open/read calls on the port will then just return errors, and an unplug/plug operation on the host will succeed as expected. This also caused buggy behaviour in case of the device removal (not just a port): when the device was removed (which means all ports on that device are removed automatically as well), the ports with active users would clean up only when the last references were dropped -- and it would be too late then to be referencing char device pointers, resulting in oopses: -------------------8<--------------------------------------- PID: 6162 TASK: ffff8801147ad500 CPU: 0 COMMAND: "cat" #0 [ffff88011b9d5a90] machine_kexec at ffffffff8103232b #1 [ffff88011b9d5af0] crash_kexec at ffffffff810b9322 mitwo-dev#2 [ffff88011b9d5bc0] oops_end at ffffffff814f4a50 MiCode#3 [ffff88011b9d5bf0] die at ffffffff8100f26b MiCode#4 [ffff88011b9d5c20] do_general_protection at ffffffff814f45e2 MiCode#5 [ffff88011b9d5c50] general_protection at ffffffff814f3db5 [exception RIP: strlen+2] RIP: ffffffff81272ae2 RSP: ffff88011b9d5d00 RFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880118901c18 RCX: 0000000000000000 RDX: ffff88011799982c RSI: 00000000000000d0 RDI: 3a303030302f3030 RBP: ffff88011b9d5d38 R8: 0000000000000006 R9: ffffffffa0134500 R10: 0000000000001000 R11: 0000000000001000 R12: ffff880117a1cc10 R13: 00000000000000d0 R14: 0000000000000017 R15: ffffffff81aff700 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 MiCode#6 [ffff88011b9d5d00] kobject_get_path at ffffffff8126dc5d MiCode#7 [ffff88011b9d5d40] kobject_uevent_env at ffffffff8126e551 MiCode#8 [ffff88011b9d5dd0] kobject_uevent at ffffffff8126e9eb MiCode#9 [ffff88011b9d5de0] device_del at ffffffff813440c7 -------------------8<--------------------------------------- So clean up when we have all the context, and all that's left to do when the references to the port have dropped is to free up the port struct itself. Reported-by: chayang <[email protected]> Reported-by: YOGANANTH SUBRAMANIAN <[email protected]> Reported-by: FuXiangChun <[email protected]> Reported-by: Qunfang Zhang <[email protected]> Reported-by: Sibiao Luo <[email protected]> Signed-off-by: Amit Shah <[email protected]> Signed-off-by: Rusty Russell <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
Nov 15, 2014
An inactive timer's base can refer to a offline cpu's base. In the current code, cpu_base's lock is blindly reinitialized each time a CPU is brought up. If a CPU is brought online during the period that another thread is trying to modify an inactive timer on that CPU with holding its timer base lock, then the lock will be reinitialized under its feet. This leads to following SPIN_BUG(). <0> BUG: spinlock already unlocked on CPU#3, kworker/u:3/1466 <0> lock: 0xe3ebe000, .magic: dead4ead, .owner: kworker/u:3/1466, .owner_cpu: 1 <4> [<c0013dc4>] (unwind_backtrace+0x0/0x11c) from [<c026e794>] (do_raw_spin_unlock+0x40/0xcc) <4> [<c026e794>] (do_raw_spin_unlock+0x40/0xcc) from [<c076c160>] (_raw_spin_unlock+0x8/0x30) <4> [<c076c160>] (_raw_spin_unlock+0x8/0x30) from [<c009b858>] (mod_timer+0x294/0x310) <4> [<c009b858>] (mod_timer+0x294/0x310) from [<c00a5e04>] (queue_delayed_work_on+0x104/0x120) <4> [<c00a5e04>] (queue_delayed_work_on+0x104/0x120) from [<c04eae00>] (sdhci_msm_bus_voting+0x88/0x9c) <4> [<c04eae00>] (sdhci_msm_bus_voting+0x88/0x9c) from [<c04d8780>] (sdhci_disable+0x40/0x48) <4> [<c04d8780>] (sdhci_disable+0x40/0x48) from [<c04bf300>] (mmc_release_host+0x4c/0xb0) <4> [<c04bf300>] (mmc_release_host+0x4c/0xb0) from [<c04c7aac>] (mmc_sd_detect+0x90/0xfc) <4> [<c04c7aac>] (mmc_sd_detect+0x90/0xfc) from [<c04c2504>] (mmc_rescan+0x7c/0x2c4) <4> [<c04c2504>] (mmc_rescan+0x7c/0x2c4) from [<c00a6a7c>] (process_one_work+0x27c/0x484) <4> [<c00a6a7c>] (process_one_work+0x27c/0x484) from [<c00a6e94>] (worker_thread+0x210/0x3b0) <4> [<c00a6e94>] (worker_thread+0x210/0x3b0) from [<c00aad9c>] (kthread+0x80/0x8c) <4> [<c00aad9c>] (kthread+0x80/0x8c) from [<c000ea80>] (kernel_thread_exit+0x0/0x8) As an example, this particular crash occurred when CPU MiCode#3 is executing mod_timer() on an inactive timer whose base is refered to offlined CPU mitwo-dev#2. The code locked the timer_base corresponding to CPU mitwo-dev#2. Before it could proceed, CPU mitwo-dev#2 came online and reinitialized the spinlock corresponding to its base. Thus now CPU MiCode#3 held a lock which was reinitialized. When CPU MiCode#3 finally ended up unlocking the old cpu_base corresponding to CPU mitwo-dev#2, we hit the above SPIN_BUG(). CPU #0 CPU MiCode#3 CPU mitwo-dev#2 ------ ------- ------- ..... ...... <Offline> mod_timer() lock_timer_base spin_lock_irqsave(&base->lock) cpu_up(2) ..... ...... init_timers_cpu() ..... spin_unlock_irqrestore(&base->lock) ...... <spin_bug> Allocation of per_cpu timer vector bases is done only once under "tvec_base_done[]" check. In the current code, spinlock_initialization of base->lock isn't under this check. When a CPU is up each time the base lock is reinitialized. Move base spinlock initialization under the check. CRs-Fixed: 471127 Change-Id: I73b48440fffb227a60af9180e318c851048530dd Signed-off-by: Tirupathi Reddy <[email protected]> Signed-off-by: Ed Tam <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
May 25, 2015
commit e5851da upstream. Remove spinlock as atomic_t can be used instead. Note we use only 16 lower bits, upper bits are changed but we impilcilty cast to u16. This fix possible deadlock on IBSS mode reproted by lockdep: ================================= [ INFO: inconsistent lock state ] 3.4.0-wl+ MiCode#4 Not tainted --------------------------------- inconsistent {IN-SOFTIRQ-W} -> {SOFTIRQ-ON-W} usage. kworker/u:2/30374 [HC0[0]:SC0[0]:HE1:SE1] takes: (&(&intf->seqlock)->rlock){+.?...}, at: [<f9979a20>] rt2x00queue_create_tx_descriptor+0x380/0x490 [rt2x00lib] {IN-SOFTIRQ-W} state was registered at: [<c04978ab>] __lock_acquire+0x47b/0x1050 [<c0498504>] lock_acquire+0x84/0xf0 [<c0835733>] _raw_spin_lock+0x33/0x40 [<f9979a20>] rt2x00queue_create_tx_descriptor+0x380/0x490 [rt2x00lib] [<f9979f2a>] rt2x00queue_write_tx_frame+0x1a/0x300 [rt2x00lib] [<f997834f>] rt2x00mac_tx+0x7f/0x380 [rt2x00lib] [<f98fe363>] __ieee80211_tx+0x1b3/0x300 [mac80211] [<f98ffdf5>] ieee80211_tx+0x105/0x130 [mac80211] [<f99000dd>] ieee80211_xmit+0xad/0x100 [mac80211] [<f9900519>] ieee80211_subif_start_xmit+0x2d9/0x930 [mac80211] [<c0782e87>] dev_hard_start_xmit+0x307/0x660 [<c079bb71>] sch_direct_xmit+0xa1/0x1e0 [<c0784bb3>] dev_queue_xmit+0x183/0x730 [<c078c27a>] neigh_resolve_output+0xfa/0x1e0 [<c07b436a>] ip_finish_output+0x24a/0x460 [<c07b4897>] ip_output+0xb7/0x100 [<c07b2d60>] ip_local_out+0x20/0x60 [<c07e01ff>] igmpv3_sendpack+0x4f/0x60 [<c07e108f>] igmp_ifc_timer_expire+0x29f/0x330 [<c04520fc>] run_timer_softirq+0x15c/0x2f0 [<c0449e3e>] __do_softirq+0xae/0x1e0 irq event stamp: 18380437 hardirqs last enabled at (18380437): [<c0526027>] __slab_alloc.clone.3+0x67/0x5f0 hardirqs last disabled at (18380436): [<c0525ff3>] __slab_alloc.clone.3+0x33/0x5f0 softirqs last enabled at (18377616): [<c0449eb3>] __do_softirq+0x123/0x1e0 softirqs last disabled at (18377611): [<c041278d>] do_softirq+0x9d/0xe0 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&(&intf->seqlock)->rlock); <Interrupt> lock(&(&intf->seqlock)->rlock); *** DEADLOCK *** 4 locks held by kworker/u:2/30374: #0: (wiphy_name(local->hw.wiphy)){++++.+}, at: [<c045cf99>] process_one_work+0x109/0x3f0 #1: ((&sdata->work)){+.+.+.}, at: [<c045cf99>] process_one_work+0x109/0x3f0 mitwo-dev#2: (&ifibss->mtx){+.+.+.}, at: [<f98f005b>] ieee80211_ibss_work+0x1b/0x470 [mac80211] MiCode#3: (&intf->beacon_skb_mutex){+.+...}, at: [<f997a644>] rt2x00queue_update_beacon+0x24/0x50 [rt2x00lib] stack backtrace: Pid: 30374, comm: kworker/u:2 Not tainted 3.4.0-wl+ MiCode#4 Call Trace: [<c04962a6>] print_usage_bug+0x1f6/0x220 [<c0496a12>] mark_lock+0x2c2/0x300 [<c0495ff0>] ? check_usage_forwards+0xc0/0xc0 [<c04978ec>] __lock_acquire+0x4bc/0x1050 [<c0527890>] ? __kmalloc_track_caller+0x1c0/0x1d0 [<c0777fb6>] ? copy_skb_header+0x26/0x90 [<c0498504>] lock_acquire+0x84/0xf0 [<f9979a20>] ? rt2x00queue_create_tx_descriptor+0x380/0x490 [rt2x00lib] [<c0835733>] _raw_spin_lock+0x33/0x40 [<f9979a20>] ? rt2x00queue_create_tx_descriptor+0x380/0x490 [rt2x00lib] [<f9979a20>] rt2x00queue_create_tx_descriptor+0x380/0x490 [rt2x00lib] [<f997a5cf>] rt2x00queue_update_beacon_locked+0x5f/0xb0 [rt2x00lib] [<f997a64d>] rt2x00queue_update_beacon+0x2d/0x50 [rt2x00lib] [<f9977e3a>] rt2x00mac_bss_info_changed+0x1ca/0x200 [rt2x00lib] [<f9977c70>] ? rt2x00mac_remove_interface+0x70/0x70 [rt2x00lib] [<f98e4dd0>] ieee80211_bss_info_change_notify+0xe0/0x1d0 [mac80211] [<f98ef7b8>] __ieee80211_sta_join_ibss+0x3b8/0x610 [mac80211] [<c0496ab4>] ? mark_held_locks+0x64/0xc0 [<c0440012>] ? virt_efi_query_capsule_caps+0x12/0x50 [<f98efb09>] ieee80211_sta_join_ibss+0xf9/0x140 [mac80211] [<f98f0456>] ieee80211_ibss_work+0x416/0x470 [mac80211] [<c0496d8b>] ? trace_hardirqs_on+0xb/0x10 [<c077683b>] ? skb_dequeue+0x4b/0x70 [<f98f207f>] ieee80211_iface_work+0x13f/0x230 [mac80211] [<c045cf99>] ? process_one_work+0x109/0x3f0 [<c045d015>] process_one_work+0x185/0x3f0 [<c045cf99>] ? process_one_work+0x109/0x3f0 [<f98f1f40>] ? ieee80211_teardown_sdata+0xa0/0xa0 [mac80211] [<c045ed86>] worker_thread+0x116/0x270 [<c045ec70>] ? manage_workers+0x1e0/0x1e0 [<c0462f64>] kthread+0x84/0x90 [<c0462ee0>] ? __init_kthread_worker+0x60/0x60 [<c083d382>] kernel_thread_helper+0x6/0x10 Signed-off-by: Stanislaw Gruszka <[email protected]> Acked-by: Helmut Schaa <[email protected]> Acked-by: Gertjan van Wingerde <[email protected]> Signed-off-by: John W. Linville <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
May 25, 2015
…condition commit 26c1917 upstream. When holding the mmap_sem for reading, pmd_offset_map_lock should only run on a pmd_t that has been read atomically from the pmdp pointer, otherwise we may read only half of it leading to this crash. PID: 11679 TASK: f06e8000 CPU: 3 COMMAND: "do_race_2_panic" #0 [f06a9dd8] crash_kexec at c049b5ec #1 [f06a9e2c] oops_end at c083d1c2 mitwo-dev#2 [f06a9e40] no_context at c0433ded MiCode#3 [f06a9e64] bad_area_nosemaphore at c043401a MiCode#4 [f06a9e6c] __do_page_fault at c0434493 MiCode#5 [f06a9eec] do_page_fault at c083eb45 MiCode#6 [f06a9f04] error_code (via page_fault) at c083c5d5 EAX: 01fb470c EBX: fff35000 ECX: 00000003 EDX: 00000100 EBP: 00000000 DS: 007b ESI: 9e201000 ES: 007b EDI: 01fb4700 GS: 00e0 CS: 0060 EIP: c083bc14 ERR: ffffffff EFLAGS: 00010246 MiCode#7 [f06a9f38] _spin_lock at c083bc14 MiCode#8 [f06a9f44] sys_mincore at c0507b7d MiCode#9 [f06a9fb0] system_call at c083becd start len EAX: ffffffda EBX: 9e200000 ECX: 00001000 EDX: 6228537f DS: 007b ESI: 00000000 ES: 007b EDI: 003d0f00 SS: 007b ESP: 62285354 EBP: 62285388 GS: 0033 CS: 0073 EIP: 00291416 ERR: 000000da EFLAGS: 00000286 This should be a longstanding bug affecting x86 32bit PAE without THP. Only archs with 64bit large pmd_t and 32bit unsigned long should be affected. With THP enabled the barrier() in pmd_none_or_trans_huge_or_clear_bad() would partly hide the bug when the pmd transition from none to stable, by forcing a re-read of the *pmd in pmd_offset_map_lock, but when THP is enabled a new set of problem arises by the fact could then transition freely in any of the none, pmd_trans_huge or pmd_trans_stable states. So making the barrier in pmd_none_or_trans_huge_or_clear_bad() unconditional isn't good idea and it would be a flakey solution. This should be fully fixed by introducing a pmd_read_atomic that reads the pmd in order with THP disabled, or by reading the pmd atomically with cmpxchg8b with THP enabled. Luckily this new race condition only triggers in the places that must already be covered by pmd_none_or_trans_huge_or_clear_bad() so the fix is localized there but this bug is not related to THP. NOTE: this can trigger on x86 32bit systems with PAE enabled with more than 4G of ram, otherwise the high part of the pmd will never risk to be truncated because it would be zero at all times, in turn so hiding the SMP race. This bug was discovered and fully debugged by Ulrich, quote: ---- [..] pmd_none_or_trans_huge_or_clear_bad() loads the content of edx and eax. 496 static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) 497 { 498 /* depend on compiler for an atomic pmd read */ 499 pmd_t pmdval = *pmd; // edi = pmd pointer 0xc0507a74 <sys_mincore+548>: mov 0x8(%esp),%edi ... // edx = PTE page table high address 0xc0507a84 <sys_mincore+564>: mov 0x4(%edi),%edx ... // eax = PTE page table low address 0xc0507a8e <sys_mincore+574>: mov (%edi),%eax [..] Please note that the PMD is not read atomically. These are two "mov" instructions where the high order bits of the PMD entry are fetched first. Hence, the above machine code is prone to the following race. - The PMD entry {high|low} is 0x0000000000000000. The "mov" at 0xc0507a84 loads 0x00000000 into edx. - A page fault (on another CPU) sneaks in between the two "mov" instructions and instantiates the PMD. - The PMD entry {high|low} is now 0x00000003fda38067. The "mov" at 0xc0507a8e loads 0xfda38067 into eax. ---- Reported-by: Ulrich Obergfell <[email protected]> Signed-off-by: Andrea Arcangeli <[email protected]> Cc: Mel Gorman <[email protected]> Cc: Hugh Dickins <[email protected]> Cc: Larry Woodman <[email protected]> Cc: Petr Matousek <[email protected]> Cc: Rik van Riel <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
May 25, 2015
commit 3cf003c upstream. [The async read code was broadened to include uncached reads in 3.5, so the mainline patch did not apply directly. This patch is just a backport to account for that change.] Jian found that when he ran fsx on a 32 bit arch with a large wsize the process and one of the bdi writeback kthreads would sometimes deadlock with a stack trace like this: crash> bt PID: 2789 TASK: f02edaa0 CPU: 3 COMMAND: "fsx" #0 [eed63cbc] schedule at c083c5b3 #1 [eed63d80] kmap_high at c0500ec8 mitwo-dev#2 [eed63db0] cifs_async_writev at f7fabcd7 [cifs] MiCode#3 [eed63df0] cifs_writepages at f7fb7f5c [cifs] MiCode#4 [eed63e50] do_writepages at c04f3e32 MiCode#5 [eed63e54] __filemap_fdatawrite_range at c04e152a MiCode#6 [eed63ea4] filemap_fdatawrite at c04e1b3e MiCode#7 [eed63eb4] cifs_file_aio_write at f7fa111a [cifs] MiCode#8 [eed63ecc] do_sync_write at c052d202 MiCode#9 [eed63f74] vfs_write at c052d4ee MiCode#10 [eed63f94] sys_write at c052df4c MiCode#11 [eed63fb0] ia32_sysenter_target at c0409a98 EAX: 00000004 EBX: 00000003 ECX: abd73b73 EDX: 012a65c6 DS: 007b ESI: 012a65c6 ES: 007b EDI: 00000000 SS: 007b ESP: bf8db178 EBP: bf8db1f8 GS: 0033 CS: 0073 EIP: 40000424 ERR: 00000004 EFLAGS: 00000246 Each task would kmap part of its address array before getting stuck, but not enough to actually issue the write. This patch fixes this by serializing the marshal_iov operations for async reads and writes. The idea here is to ensure that cifs aggressively tries to populate a request before attempting to fulfill another one. As soon as all of the pages are kmapped for a request, then we can unlock and allow another one to proceed. There's no need to do this serialization on non-CONFIG_HIGHMEM arches however, so optimize all of this out when CONFIG_HIGHMEM isn't set. Reported-by: Jian Li <[email protected]> Signed-off-by: Jeff Layton <[email protected]> Signed-off-by: Steve French <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
May 25, 2015
…d reasons commit 5cf02d0 upstream. We've had some reports of a deadlock where rpciod ends up with a stack trace like this: PID: 2507 TASK: ffff88103691ab40 CPU: 14 COMMAND: "rpciod/14" #0 [ffff8810343bf2f0] schedule at ffffffff814dabd9 #1 [ffff8810343bf3b8] nfs_wait_bit_killable at ffffffffa038fc04 [nfs] mitwo-dev#2 [ffff8810343bf3c8] __wait_on_bit at ffffffff814dbc2f MiCode#3 [ffff8810343bf418] out_of_line_wait_on_bit at ffffffff814dbcd8 MiCode#4 [ffff8810343bf488] nfs_commit_inode at ffffffffa039e0c1 [nfs] MiCode#5 [ffff8810343bf4f8] nfs_release_page at ffffffffa038bef6 [nfs] MiCode#6 [ffff8810343bf528] try_to_release_page at ffffffff8110c670 MiCode#7 [ffff8810343bf538] shrink_page_list.clone.0 at ffffffff81126271 MiCode#8 [ffff8810343bf668] shrink_inactive_list at ffffffff81126638 MiCode#9 [ffff8810343bf818] shrink_zone at ffffffff8112788f MiCode#10 [ffff8810343bf8c8] do_try_to_free_pages at ffffffff81127b1e MiCode#11 [ffff8810343bf958] try_to_free_pages at ffffffff8112812f MiCode#12 [ffff8810343bfa08] __alloc_pages_nodemask at ffffffff8111fdad MiCode#13 [ffff8810343bfb28] kmem_getpages at ffffffff81159942 #14 [ffff8810343bfb58] fallback_alloc at ffffffff8115a55a #15 [ffff8810343bfbd8] ____cache_alloc_node at ffffffff8115a2d9 #16 [ffff8810343bfc38] kmem_cache_alloc at ffffffff8115b09b #17 [ffff8810343bfc78] sk_prot_alloc at ffffffff81411808 #18 [ffff8810343bfcb8] sk_alloc at ffffffff8141197c #19 [ffff8810343bfce8] inet_create at ffffffff81483ba6 #20 [ffff8810343bfd38] __sock_create at ffffffff8140b4a7 #21 [ffff8810343bfd98] xs_create_sock at ffffffffa01f649b [sunrpc] #22 [ffff8810343bfdd8] xs_tcp_setup_socket at ffffffffa01f6965 [sunrpc] #23 [ffff8810343bfe38] worker_thread at ffffffff810887d0 #24 [ffff8810343bfee8] kthread at ffffffff8108dd96 #25 [ffff8810343bff48] kernel_thread at ffffffff8100c1ca rpciod is trying to allocate memory for a new socket to talk to the server. The VM ends up calling ->releasepage to get more memory, and it tries to do a blocking commit. That commit can't succeed however without a connected socket, so we deadlock. Fix this by setting PF_FSTRANS on the workqueue task prior to doing the socket allocation, and having nfs_release_page check for that flag when deciding whether to do a commit call. Also, set PF_FSTRANS unconditionally in rpc_async_schedule since that function can also do allocations sometimes. Signed-off-by: Jeff Layton <[email protected]> Signed-off-by: Trond Myklebust <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
May 25, 2015
[ Upstream commit 89d7ae3 ] As reported by Alan Cox, and verified by Lin Ming, when a user attempts to add a CIPSO option to a socket using the CIPSO_V4_TAG_LOCAL tag the kernel dies a terrible death when it attempts to follow a NULL pointer (the skb argument to cipso_v4_validate() is NULL when called via the setsockopt() syscall). This patch fixes this by first checking to ensure that the skb is non-NULL before using it to find the incoming network interface. In the unlikely case where the skb is NULL and the user attempts to add a CIPSO option with the _TAG_LOCAL tag we return an error as this is not something we want to allow. A simple reproducer, kindly supplied by Lin Ming, although you must have the CIPSO DOI MiCode#3 configure on the system first or you will be caught early in cipso_v4_validate(): #include <sys/types.h> #include <sys/socket.h> #include <linux/ip.h> #include <linux/in.h> #include <string.h> struct local_tag { char type; char length; char info[4]; }; struct cipso { char type; char length; char doi[4]; struct local_tag local; }; int main(int argc, char **argv) { int sockfd; struct cipso cipso = { .type = IPOPT_CIPSO, .length = sizeof(struct cipso), .local = { .type = 128, .length = sizeof(struct local_tag), }, }; memset(cipso.doi, 0, 4); cipso.doi[3] = 3; sockfd = socket(AF_INET, SOCK_DGRAM, 0); #define SOL_IP 0 setsockopt(sockfd, SOL_IP, IP_OPTIONS, &cipso, sizeof(struct cipso)); return 0; } CC: Lin Ming <[email protected]> Reported-by: Alan Cox <[email protected]> Signed-off-by: Paul Moore <[email protected]> Signed-off-by: David S. Miller <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
maxime-poulain
pushed a commit
to maxime-poulain/mi2_kernel
that referenced
this issue
May 25, 2015
commit f1b5c99 upstream. The ZTE (Vodafone) K5006-Z use the following interface layout: 00 DIAG 01 secondary 02 modem 03 networkcard 04 storage Ignoring interface MiCode#3 which is handled by the qmi_wwan driver. Signed-off-by: Bjørn Mork <[email protected]> Cc: Thomas Schäfer <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
ivan19871002
pushed a commit
to mitwo-dev/android_kernel_xiaomi_msm8960
that referenced
this issue
Nov 26, 2015
commit ce7514526742c0898b837d4395f515b79dfb5a12 upstream. It is possible for ata_sff_flush_pio_task() to set ap->hsm_task_state to HSM_ST_IDLE in between the time __ata_sff_port_intr() checks for HSM_ST_IDLE and before it calls ata_sff_hsm_move() causing ata_sff_hsm_move() to BUG(). This problem is hard to reproduce making this patch hard to verify, but this fix will prevent the race. I have not been able to reproduce the problem, but here is a crash dump from a 2.6.32 kernel. On examining the ata port's state, its hsm_task_state field has a value of HSM_ST_IDLE: crash> struct ata_port.hsm_task_state ffff881c1121c000 hsm_task_state = 0 Normally, this should not be possible as ata_sff_hsm_move() was called from ata_sff_host_intr(), which checks hsm_task_state and won't call ata_sff_hsm_move() if it has a HSM_ST_IDLE value. PID: 11053 TASK: ffff8816e846cae0 CPU: 0 COMMAND: "sshd" #0 [ffff88008ba03960] machine_kexec at ffffffff81038f3b #1 [ffff88008ba039c0] crash_kexec at ffffffff810c5d92 #2 [ffff88008ba03a90] oops_end at ffffffff8152b510 MiCode#3 [ffff88008ba03ac0] die at ffffffff81010e0b MiCode#4 [ffff88008ba03af0] do_trap at ffffffff8152ad74 MiCode#5 [ffff88008ba03b50] do_invalid_op at ffffffff8100cf95 MiCode#6 [ffff88008ba03bf0] invalid_op at ffffffff8100bf9b [exception RIP: ata_sff_hsm_move+317] RIP: ffffffff813a77ad RSP: ffff88008ba03ca0 RFLAGS: 00010097 RAX: 0000000000000000 RBX: ffff881c1121dc60 RCX: 0000000000000000 RDX: ffff881c1121dd10 RSI: ffff881c1121dc60 RDI: ffff881c1121c000 RBP: ffff88008ba03d00 R8: 0000000000000000 R9: 000000000000002e R10: 000000000001003f R11: 000000000000009b R12: ffff881c1121c000 R13: 0000000000000000 R14: 0000000000000050 R15: ffff881c1121dd78 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018 MiCode#7 [ffff88008ba03d08] ata_sff_host_intr at ffffffff813a7fbd MiCode#8 [ffff88008ba03d38] ata_sff_interrupt at ffffffff813a821e MiCode#9 [ffff88008ba03d78] handle_IRQ_event at ffffffff810e6ec0
ivan19871002
pushed a commit
to mitwo-dev/android_kernel_xiaomi_msm8960
that referenced
this issue
Nov 26, 2015
An inactive timer's base can refer to a offline cpu's base. In the current code, cpu_base's lock is blindly reinitialized each time a CPU is brought up. If a CPU is brought online during the period that another thread is trying to modify an inactive timer on that CPU with holding its timer base lock, then the lock will be reinitialized under its feet. This leads to following SPIN_BUG(). <0> BUG: spinlock already unlocked on CPU#3, kworker/u:3/1466 <0> lock: 0xe3ebe000, .magic: dead4ead, .owner: kworker/u:3/1466, .owner_cpu: 1 <4> [<c0013dc4>] (unwind_backtrace+0x0/0x11c) from [<c026e794>] (do_raw_spin_unlock+0x40/0xcc) <4> [<c026e794>] (do_raw_spin_unlock+0x40/0xcc) from [<c076c160>] (_raw_spin_unlock+0x8/0x30) <4> [<c076c160>] (_raw_spin_unlock+0x8/0x30) from [<c009b858>] (mod_timer+0x294/0x310) <4> [<c009b858>] (mod_timer+0x294/0x310) from [<c00a5e04>] (queue_delayed_work_on+0x104/0x120) <4> [<c00a5e04>] (queue_delayed_work_on+0x104/0x120) from [<c04eae00>] (sdhci_msm_bus_voting+0x88/0x9c) <4> [<c04eae00>] (sdhci_msm_bus_voting+0x88/0x9c) from [<c04d8780>] (sdhci_disable+0x40/0x48) <4> [<c04d8780>] (sdhci_disable+0x40/0x48) from [<c04bf300>] (mmc_release_host+0x4c/0xb0) <4> [<c04bf300>] (mmc_release_host+0x4c/0xb0) from [<c04c7aac>] (mmc_sd_detect+0x90/0xfc) <4> [<c04c7aac>] (mmc_sd_detect+0x90/0xfc) from [<c04c2504>] (mmc_rescan+0x7c/0x2c4) <4> [<c04c2504>] (mmc_rescan+0x7c/0x2c4) from [<c00a6a7c>] (process_one_work+0x27c/0x484) <4> [<c00a6a7c>] (process_one_work+0x27c/0x484) from [<c00a6e94>] (worker_thread+0x210/0x3b0) <4> [<c00a6e94>] (worker_thread+0x210/0x3b0) from [<c00aad9c>] (kthread+0x80/0x8c) <4> [<c00aad9c>] (kthread+0x80/0x8c) from [<c000ea80>] (kernel_thread_exit+0x0/0x8) As an example, this particular crash occurred when CPU MiCode#3 is executing mod_timer() on an inactive timer whose base is refered to offlined CPU #2. The code locked the timer_base corresponding to CPU #2. Before it could proceed, CPU #2 came online and reinitialized the spinlock corresponding to its base. Thus now CPU MiCode#3 held a lock which was reinitialized. When CPU MiCode#3 finally ended up unlocking the old cpu_base corresponding to CPU #2, we hit the above SPIN_BUG(). CPU #0 CPU MiCode#3 CPU #2 ------ ------- ------- ..... ...... <Offline> mod_timer() lock_timer_base spin_lock_irqsave(&base->lock) cpu_up(2) ..... ...... init_timers_cpu() ..... spin_unlock_irqrestore(&base->lock) ...... <spin_bug> Allocation of per_cpu timer vector bases is done only once under "tvec_base_done[]" check. In the current code, spinlock_initialization of base->lock isn't under this check. When a CPU is up each time the base lock is reinitialized. Move base spinlock initialization under the check. CRs-Fixed: 471127 Change-Id: I73b48440fffb227a60af9180e318c851048530dd Signed-off-by: Tirupathi Reddy <[email protected]> Signed-off-by: Ed Tam <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: