https://en.wikipedia.org/wiki/Cache_coherence
Theoretically, coherence can be performed at the load/store granularity. However, in practice it is generally performed at the granularity of cache blocks.[3]
https://www.geeksforgeeks.org/cache-coherence/
Cache coherence is the discipline that ensures that changes in the values of shared operands are propagated throughout the system in a timely fashion.
http://tutorials.jenkov.com/java-concurrency/cache-coherence-in-java-concurrency.html
Ticket spinlockis the spinlock implementation used in the Linux kernel prior to 4.2. A lock waiter gets a ticket number and spin on the lock cacheline until it sees its ticket number. By then, it becomes the lock owner and enters the critical section.
Queued spinlockis the new spinlock implementation used in 4.2 Linux kernel and beyond. A lock waiter goes into a queue and spins in its own cacheline until it becomes the queue head. By then, it can spin on the lock cacheline and attempt to get the lock.
https://easyperf.net/blog/2018/09/04/Performance-Analysis-Vocabulary
Majority of modern CPUs including Intel’s and AMD’s ones don’t have fixed frequency on which they operate. Instead, they have dynamic frequency scaling. In Intel’s CPUs this technology is called Turbo Boost, in AMD’s processors it’s called Turbo Core. There is nice explanation of the term “reference cycles” on this stackoverflow thread:
Having a snippet A to run in 100 core clocks and a snippet B in 200 core clocks means that B is slower in general (it takes double the work), but not necessarily that B took more time than A since the units are different. That’s where the reference clock comes into play - it is uniform. If snippet A runs in 100 ref clocks and snippet B runs in 200 ref clocks then B really took more time than A.
https://easyperf.net/blog/2018/09/04/Performance-Analysis-Vocabulary
https://software.intel.com/content/www/us/en/develop/documentation/vtune-help/top/analyze-performance/custom-analysis/custom-analysis-options/hardware-event-list/instructions-retired-event.html
The Instructions Retired is an important hardware performance event that shows how many instructions were completely executed.
Modern processors execute much more instructions that the program flow needs. This is called a speculative execution. Instructions that were “proven” as indeed needed by the program execution flow are “retired”.
In the Core Out Of Order pipeline leaving the Retirement Unit means that the instructions are finally executed and their results are correct and visible in the architectural state as if they execute in-order.
Let us assume a ‘classic RISC pipeline’, with the following five stages:
Each stage requires one clock cycle and an instruction passes through the stages sequentially. Without pipelining, in a multi-cycle processor, a new instruction is fetched in stage 1 only after the previous instruction finishes at stage 5, therefore the number of clock cycles it takes to execute an instruction is five (CPI = 5 > 1). In this case, the processor is said to be subscalar. With pipelining, a new instruction is fetched every clock cycle by exploiting instruction-level parallelism, therefore, since one could theoretically have five instructions in the five pipeline stages at once (one instruction per stage), a different instruction would complete stage 5 in every clock cycle and on average the number of clock cycles it takes to execute an instruction is 1 (CPI = 1). In this case, the processor is said to be scalar.
http://web.eece.maine.edu/~vweaver/projects/perf_counters/retired_instructions.html
Retired instruction counts on x86 in general also include at least one extra instruction each time a hardware interrupt happens, even if only user space code is being monitored. The one exception to this is the Pentium 4 counter.
Another special case are rep prefixed string instructions. Even if the instruction repeats many times, the instruction is only counted as one instruction.
https://easyperf.net/blog/2018/06/01/PMU-counters-and-profiling-basics
In a really simplified view our processor looks like this:
There is a clock generator that sends pulses to every piece of the system to make everything moving to the next stage. This is called a cycle. If we add just a little bit of silicon and connect it to the pulse generator we can count a number of cycles, yay!
If we don’t set the scaling governor policy to be performance kernel can decide that it’s better to save power and throttle. Setting scaling_governor to ‘performance’ helps to avoid sub-nominal clocking. Here is the documentation about Linux CPU frequency governors.
Here is how we can set it for all the cores:
|
|
https://easyperf.net/blog/2019/08/02/Perf-measurement-environment-on-Linux
https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt
pool1-n104-vpod1-wpool1-n23:~/pmu-tools # x86info -c
x86info vVERSION
Found 80 identical CPUsMP Configuration Table Header MISSING!
Extended Family: 0 Extended Model: 5 Family: 6 Model: 85 Stepping: 7
Type: 0 (Original OEM)
CPU Model (x86info's best guess): Unknown model.
Processor name string (BIOS programmed): Intel(R) Xeon(R) Gold 6230N CPU @ 2.30GHz
Cache info
L1 Data Cache: 32KB, 8-way associative, 64 byte line size
L1 Instruction Cache: 32KB, 8-way associative, 64 byte line size
L2 Unified Cache: 1024KB, 16-way associative, 64 byte line size
L3 Unified Cache: 28160KB, 11-way associative, 64 byte line size
TLB info
Instruction TLB: 2M/4M pages, fully associative, 8 entries
Instruction TLB: 4K pages, 8-way associative, 64 entries
Data TLB: 1GB pages, 4-way set associative, 4 entries
Data TLB: 4KB pages, 4-way associative, 64 entries
Shared L2 TLB: 4KB/2MB pages, 6-way associative, 1536 entries
64 byte prefetching.
Total processor threads: 80
This system has 2 20-core processors with hyper-threading (2 threads per core) running at an estimated 2.30GHz
Huge pages part 5: A deeper look at TLBs and costs
TLB and Java
A translation lookaside buffer (TLB) is a memory cache that is used to reduce the time taken to access a user memory location. It is a part of the chip’s memory-management unit (MMU). The TLB stores the recent translations of virtual memory to physical memory and can be called an address-translation cache. A TLB may reside between the CPU and the CPU cache, between CPU cache and the main memory or between the different levels of the multi-level cache
Intel Turbo Boost is a feature that automatically raises CPU operating frequency when demanding tasks are running. It can be permanently disabled in BIOS. Check FAQ for more information. To disable turbo in Linux do:
# Intel
echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo
# AMD
echo 0 > /sys/devices/system/cpu/cpufreq/boost
Also you might want to take a look at how it’s done in uarch-bench.
Example (single-threaded workload running on Intel® Core™ i5-8259U):
# TurboBoost enabled
$ cat /sys/devices/system/cpu/intel_pstate/no_turbo
0
$ perf stat -e task-clock,cycles -- ./a.out
11984.691958 task-clock (msec) # 1.000 CPUs utilized
32,427,294,227 cycles # 2.706 GHz
11.989164338 seconds time elapsed
# TurboBoost disabled
$ echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo
1
$ perf stat -e task-clock,cycles -- ./a.out
13055.200832 task-clock (msec) # 0.993 CPUs utilized
29,946,969,255 cycles # 2.294 GHz
13.142983989 seconds time elapsed
You can see the average frequency is much higher when TurboBoost is on.