Excessive TLB flush ranges

[Thomas Gleixner]

On Wed, May 17 2023 at 15:43, Mark Rutland wrote:
> On Wed, May 17, 2023 at 12:31:04PM +0200, Thomas Gleixner wrote:
>> The way how arm/arm64 implement that in software is:
>> 
>>     magic_barrier1();
>>     flush_range_with_magic_opcodes();
>>     magic_barrier2();
>
> FWIW, on arm64 that sequence (for leaf entries only) is:
>
> 	/*
> 	 * Make sure prior writes to the page table entries are visible to all
> 	 * CPUs, so that *subsequent* page table walks will see the latest
> 	 * values.
> 	 *
> 	 * This is roughly __smp_wmb().
> 	 */
> 	dsb(ishst)		// AKA magic_barrier1()
>
> 	/*
> 	 * The "TLBI *IS, <addr>" instructions send a message to all other
> 	 * CPUs, essentially saying "please start invalidating entries for
> 	 * <addr>"
> 	 *
> 	 * The "TLBI *ALL*IS" instructions send a message to all other CPUs,
> 	 * essentially saying "please start invalidating all entries".
> 	 *
> 	 * In theory, this could be for discontiguous ranges.
> 	 */
> 	flush_range_with_magic_opcodes()
>
> 	/*
> 	 * Wait for acknowledgement that all prior TLBIs have completed. This
> 	 * also ensures that all accesses using those translations have also
> 	 * completed.
> 	 *
> 	 * This waits for all relevant CPUs to acknowledge completion of any
> 	 * prior TLBIs sent by this CPU.
> 	 */
> 	dsb(ish) 		// AKA magic_barrier2()
> 	isb()
>
> So you can batch a bunch of "TLBI *IS, <addr>" with a single barrier for
> completion, or you can use a single "TLBI *ALL*IS" to invalidate everything.
>
> It can still be worth using the latter, as arm64 has done since commit:
>
>   05ac65305437e8ef ("arm64: fix soft lockup due to large tlb flush range")
>
> ... as for a large range, issuing a bunch of "TLBI *IS, <addr>" can take a
> while, and can require the recipient CPUs to do more work than they might have
> to do for a single "TLBI *ALL*IS".

And looking at the changelog and backtrace:

       PC is at __cpu_flush_kern_tlb_range+0xc/0x40
       LR is at __purge_vmap_area_lazy+0x28c/0x3ac

I'm willing to bet that this is exactly the same scenario of a direct
map + module area flush. That's the only one we found so far which
creates insanely large ranges.

The other effects of coalescing can still result in seriously oversized
flushs for just a couple of pages. The worst I've seen aside of that BPF
muck was a 'flush 2 pages' with an resulting range of ~3.8MB.

> The point at which invalidating everything is better depends on a number of
> factors (e.g. the impact of all CPUs needing to make new page table walks), and
> currently we have an arbitrary boundary where we choose to invalidate
> everything (which has been tweaked a bit over time); there isn't really a
> one-size-fits-all best answer.

I'm well aware of that :)

Thanks,

        tglx