Resizable Base Address Register. In short, rather than the 256MB of mapped memory available to the CPU for any PCI device, ReBAR capable devices can map larger memory to the CPU’s addressable memory space. Without ReBAR or sufficient tricks (that Intel GPUs lack) you have to shuffle 256MB chunks around between GPU and CPU.
Using an Intel discrete GPU on a system that lacks ReBAR support results in somewhat lower performance, not a complete inability to use more than 256MB of VRAM.
Its commercial name is AMD Smart Access Memory. This generated much fanfare in the early 2020s after AMD officially supported it and marketed it as a new technology to boost GPU/gaming performance. What AMD did was just rebranding an obscure feature in the PCIe specification [1] for a decade that nobody bothered to use, which allowed one to map more VRAM, which improved performance. Presumably, AMD did it after validating and fixing any driver compatibility problems.
[1] This is nothing new in the tech industry. Intel rebrands DVFS as SpeedStep, IOMMU as VT-d, AMD rebrands the NX bit as Advanced Virus Protection, etc.
I feel part of the "Smart Access Memory" name is actually the driver features and paths to actually make use of it.
It's not like ReBar is a single toggle "Make Things Faster", but a different option in how it can map gpu memory to the cpu. The driver still needs to use it - and decide where it's use vs the "staging buffer" approach would actually be be benefitial.
It's hard to explain without going into a few low-level details of PCI Express, but let me try.
Most PCI devices expose some memory and/or I/O ports to the CPU. That memory (or I/O ports) is mapped to somewhere in the address space visible to the CPU. Besides the memory and I/O ports, all PCI devices also expose a separate set of configuration registers; among these registers, there are the Base Address Registers (BARs), which configure where the memory or I/O ports is mapped.
Here's an example output from "lspci -vv" for a GPU:
Region 0: Memory at 7c00000000 (64-bit, prefetchable) [size=8G]
Region 2: Memory at 7e00000000 (64-bit, prefetchable) [size=256M]
Region 4: I/O ports at f000 [size=256]
Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fcb00000 [disabled] [size=128K]
Note that regions 0 and 2 are above the 4GB addressable by old 32-bit CPUs. To be compatible with these old CPUs, this card and many others like it allow the firmware (and/or the operating system) to choose not only where the memory is mapped, but also its size. We can see this in the same "lspci -vv" output for this GPU:
Capabilities: [200 v1] Physical Resizable BAR
BAR 0: current size: 8GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB
BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
Older systems which do not understand this extended capability will still treat these regions as fixed size, probably with the first size in this list (256MB for region 0, 2MB for region 2). Newer systems can tell the device to "resize" the BAR to a bigger size, which obviously needs the first region to be placed above the 4GB barrier since it's too big.
Why is this useful? This particular GPU has 8GB of VRAM; it's quite obvious that region 0 is a direct view into that VRAM. When using the maximum BAR size, the CPU can directly read and write anywhere into the VRAM; when using a smaller BAR, the CPU can only see a small window into the VRAM, and has to use less direct methods to access it.
(As an aside: go right now and do a "sudo lspci -vv" on your computer, if you see a Resizable BAR capability which isn't using the maximum size, you can probably gain a bit more speed for free by going into the BIOS and enabling "Resizable BAR" and/or "Above 4G decoding". If you can't find these options, well, AFAIU that's what this project is all about..)
> (As an aside: go right now and do a "sudo lspci -vv" on your computer, if you see a Resizable BAR capability which isn't using the maximum size, you can probably gain a bit more speed for free by going into the BIOS and enabling "Resizable BAR" and/or "Above 4G decoding". If you can't find these options, well, AFAIU that's what this project is all about..)
IIRC Linux doesn't need resizable bar enabled in the BIOS since the kernel will resize the bar if supported by the GPU, Windows however relies on the UEFI doing it which is where it being enabled in the BIOS is needed.
Since about 5 or 6 years ago, PCIe Video Cards advertise a capability known as Resizeable BAR. PCI Devices like GPUs require some memory to use for PCI MMIO, which is directly visible on the CPU Address Space. As a side note, this Address Space is shared with RAM, and anyone that was around when having 4 GiB RAM with a 32 Bit OS was common (Earlier than 2010 or so) knows that you only saw about 3.25 GiB RAM or so because of sharing the Address Space with PCI MMIO, and those fortunate enough to have used SLI usually saw even less than that, like 2.87 GiB RAM.
GPUs has been using a 256 MiB PCI MMIO window regardless of how much VRAM they actually have since... nearly forever? At least since PCIe is a thing, since I recall than AGP Aperture Size was seteable in era accurate BIOSes. PCIe 3.0 specification introduced a feature known as Resizeable BAR, where the PCI Device can tell a compatible Firmware how much MMIO it actually wants. GPUs uses that to tell a ReBAR capable UEFI Firmware that it wants more MMIO (Usually as big as the GPU VRAM), or uses legacy 256 MiB otherwise.
Just to make sure I understand: Is the resizable BAR/MMIO a RAM buffer for PCie packets? Or does it have some deeper integration with DMA or something? I assume it's not like memory mapped peripherals on an AXI bus which is why you need the buffer?
For historic reasons (e.g. 32 bit address spaces, plus the need to reserve the space for multiple pci peripherals) it has been a narrow, movable aperture.
Resizable BAR lets the size of the aperture be chosen (which is usually chosen to allow all of VRAM to fit in and be directly accessible).
No idea on the AXI Bus you're talking about, so can't make comparisons.
MMIO (Memory Mapped I/O) is essentially memory (Whenever RAM or ROM) from OTHER devices that is directly visible on the CPU Address Space. My understanding is that from the CPU side, MMIO is mostly transparent (Except for the massive increase in latency) because it gets used like if it was interacting with its own workspace with regular instructions like MOV.
What PCIe ReBAR changes is that before, you could only see a 256 MiB window onto the GPU VRAM, so there was an added overhead since the GPU may need to relocate things from inside that window somewhere else on its total VRAM (So yes, it may be interpreted as if what you see from the CPU side is just some kind of exchange buffer).
I believe the best way to describe how it operates is comparing it to EMS (Expanded Memory) from the DOS days since it also worked with a similar, if not the same idea. You could only see a portion of the total memory from what was installed on the EMS card (A 128 KiB window located on the upper part of the 1 MiB address space from the 8086 CPU), so you had to switch which Page (Region) of the memory was visible, adding a lot of overhead and most likely requiring an additional buffer in main RAM to move data from one Page to another. However, since I have no knowledge if the GPUs really work like that I can't confirm. I never knew whenever the 256 MiB is "fixed" (You always see the same Region) or if you can decide which section of the VRAM to make visible.
GPUs have been using a 256MB window since they started coming with 256MB of VRAM, as it would otherwise be a pointless waste of address space to have a window larger than the actual amount of VRAM. Previously, they would have a window exactly equal to the size of VRAM (likely rounded up to the next power of 2.)
There would also be a block of memory mapped hardware registers also needing address space that you have to poke to make the GPU actually do GPU things, instead of just being an expensive way to add extra memory to a system, no?
And if my experience from embedded development is in any way transferable, they're probably fairly spread out and probably takes a fairly big chunk of address space too.
Yes, typically the memory mapped memory is in one (64-bit) bar, memory mapped registers in another (64-bit) bar, plus for compatability with vga, probably a 32-bit memory bar and an i/o bar. 64-bit bars take up two bar slots, so that fills all six slots in the PCI config.
this fixes a real annoyance on older workstation boards where above 4g decoding is a requirement but the option isn't even in the bios menu. saw a build recently on an x79 board where it had to be exposed with a hand patched bios before anything like this or a modern gpu would work at all.
thanks for the throwback, I remember having to hand patch the bios on my x79 system to use a gpu with 12GB vram. Fun times. My motherboard also mysteriously stopped working shortly after…
I patched my old Haswell system to add an NVMe DXE so I could boot from a modern SSD. I currently have a Lets Note laptop that definitely has UEFI, but pretty much pretends to be BIOS - apparently it's possible to modify to expose UEFI features but I have yet to try...
These kind of BIOS modding shenannigans is why I got interesed in Coreboot, so that enhancing a Firmware capabilities becomes possible without touching propietary tools/code.
Other similar mods included adding NVMe Boot capabilities to older Firmwares that didn't supported them so that you could boot from modern NVMe drives if you used some kind of adapter (As none of those Motherboards have M.2 Slots, but you can use them with a PCIe-to-M.2 adapter card). Also adding required stuff to get certain CPUs booting in Motherboard that were never officially supported (Depending on generation, could be as simple as adding new CPU Microcode).
These days you might be able to just point an AI at the original BIOS and tell it what you need done, instead of attempting to install a complete rewrite like Coreboot (which comes with its own limitations, the biggest one being that only a rather small subset of mobos are actually supported.)
Regarding Coreboot, it is far worse than that. They literally do not care anymore; focus is not on corebooting all the boards, but on serving some sort of commercial "customer".
e.g. they do seldom if ever take patches for adding support or fixing issues in older boards. They just sit there ignored.
A fork or entire new project with different focus is sorely needed.
There is also the Libreboot which may be of your interest. What I see is that there is a lot of development on Coreboot but it is indeed mostly driven by companies working on their latest platforms. There are not many contributors working on older platforms. Though in my experience the response to patches is relatively quick.
I had the exact opposite impression, there are a lot of user contributions working on old platforms, and even specific platform communities. For example, you have 15h.org which is a community dedicated to AMD Bulldozer / Piledriver generation, and they managed to fix a few things that were left broken in Coreboot. I also saw an influx of users working in Sandy Bridge / Ivy Bridge / Haswell era Intel platforms with an occasional Skylake. The problem is, precisely, that most of the user contributions are for old platforms that nobody that does NOT currently owns those will want to touch if you're purchasing new. I would have been very enthusiastic about that work if it was done about 10 years ago or so, nowadays they seem to be hobby projects that don't advance anything forward.
There do exists individual users that work on newer stuff, after the MSI Z690-A / Z790-P there were 2 people that did ports for H610 since at that point the platform was supported and they only had to port the Motherboard. However, they are rather rare, since the kind of user that Coreboot seems to attract are from the privacy and security anti-ME/PSP crowd that don't even want to use newer platforms, which is the exact opposite of my view.
A lot of the work companies does may be just for their internal systems or reference platform and these may not get to the hands of end users, ever. I still recall Phoronix articles covering Coreboot releases always taking a jab at the new supported Motherboards until the MSI Z690-A DDR4 port from 2022.
Tempted to see if this will work on the work dell laptop, which hardware supports rebar, but Dell just can't be assed to enable it - use an external ARC GPU to properly support 3 screens (again, Dells awful drivers / firmware - the same hardware supports 3 screens just fine).
FYI: Resizable BAR (Base Address Register) is a PCI Express feature that removes the traditional 256MB memory limit, allowing your CPU to access your graphics card's entire VRAM at once for a potential 5% to 15% boost in gaming performance
Only on specific devices, but generally every GPU released in last few generations, and every dedicated GPU from intel[1], supports "large BAR" this way to map entire GPU memory into MMIO space. This has to be supported on both target device and AFAIK the PCI Root Port[2], and of course needs to be properly handled in firmware - which this tool fixes.
[1] Intel GPUs actually can't work without it, because they do not have legacy "windowing" mode where portions of 256MB BAR window are mapped to different areas in GPU memory
[2] This tool implies that it's possible to handle root port side entirely in firmware, but it's new to me at least, so I can't vouch for that :)
You most likely have this enabled and unless you run some ancient board.
This UEFI mod is for motherboards from circa 2018.
For workstation machines that would be 1st and 2nd generation Threadripper. So, if your CPU's model name starts with 3 or higher (and you have an AMD workstation), you likely have this already enabled by default.
"Do I need to flash a new UEFI image on the motherboard, to enable ReBAR on the GPU ? Yes, this is how it works for Turing GPUs (GTX 1600 / RTX 2000)."
Sounds scary. Like there is a good chance to brick the motherboard permanently.
I think i'm going to skip this for my ye olde pc with crappy GTX 1660 Ti card. Can't afford a new system.
just did it, worked perfectly from the guides on my MSI motherboard. Only scary moment was the actual bios flashing, because it took a small while and you shouldn't touch anything around you that risks blowing a fuse.
Can anyone verify if this works on a Lenovo Thinkpad? I checked the List of Working Motherboards [1]. They often have Thunderbolt but no way to enable ReBAR.
This made me finally try to migrate an old system of mine. Migrate what you ask? Well, ReBAR driver stuff requires GPT-based disks, and on that machine it's still running on MBR...
Since I don't want to tinker with my HP Firmware, a brief investigation led me to OpenCore's ability to enable Resizable BAR in the bootloader before Windows boots. Not sure how reliable it is.
https://github.com/xCuri0/ReBarUEFI/discussions/118
Real question, why go through the hassle to install chrome on a linux machine?
I moved away from chrome a while ago, but last time I installed it on linux it was a pain, and firefox and its derivatives are essentially at feature and speed parity or better than chrome across the board.
51 comments
[ 0.25 ms ] story [ 8.5 ms ] threadIsn't it just being able to shift the window of GPU memory visible to the CPU?
[1] This is nothing new in the tech industry. Intel rebrands DVFS as SpeedStep, IOMMU as VT-d, AMD rebrands the NX bit as Advanced Virus Protection, etc.
It's not like ReBar is a single toggle "Make Things Faster", but a different option in how it can map gpu memory to the cpu. The driver still needs to use it - and decide where it's use vs the "staging buffer" approach would actually be be benefitial.
Most PCI devices expose some memory and/or I/O ports to the CPU. That memory (or I/O ports) is mapped to somewhere in the address space visible to the CPU. Besides the memory and I/O ports, all PCI devices also expose a separate set of configuration registers; among these registers, there are the Base Address Registers (BARs), which configure where the memory or I/O ports is mapped.
Here's an example output from "lspci -vv" for a GPU:
Note that regions 0 and 2 are above the 4GB addressable by old 32-bit CPUs. To be compatible with these old CPUs, this card and many others like it allow the firmware (and/or the operating system) to choose not only where the memory is mapped, but also its size. We can see this in the same "lspci -vv" output for this GPU: Older systems which do not understand this extended capability will still treat these regions as fixed size, probably with the first size in this list (256MB for region 0, 2MB for region 2). Newer systems can tell the device to "resize" the BAR to a bigger size, which obviously needs the first region to be placed above the 4GB barrier since it's too big.Why is this useful? This particular GPU has 8GB of VRAM; it's quite obvious that region 0 is a direct view into that VRAM. When using the maximum BAR size, the CPU can directly read and write anywhere into the VRAM; when using a smaller BAR, the CPU can only see a small window into the VRAM, and has to use less direct methods to access it.
(As an aside: go right now and do a "sudo lspci -vv" on your computer, if you see a Resizable BAR capability which isn't using the maximum size, you can probably gain a bit more speed for free by going into the BIOS and enabling "Resizable BAR" and/or "Above 4G decoding". If you can't find these options, well, AFAIU that's what this project is all about..)
IIRC Linux doesn't need resizable bar enabled in the BIOS since the kernel will resize the bar if supported by the GPU, Windows however relies on the UEFI doing it which is where it being enabled in the BIOS is needed.
GPUs has been using a 256 MiB PCI MMIO window regardless of how much VRAM they actually have since... nearly forever? At least since PCIe is a thing, since I recall than AGP Aperture Size was seteable in era accurate BIOSes. PCIe 3.0 specification introduced a feature known as Resizeable BAR, where the PCI Device can tell a compatible Firmware how much MMIO it actually wants. GPUs uses that to tell a ReBAR capable UEFI Firmware that it wants more MMIO (Usually as big as the GPU VRAM), or uses legacy 256 MiB otherwise.
For historic reasons (e.g. 32 bit address spaces, plus the need to reserve the space for multiple pci peripherals) it has been a narrow, movable aperture.
Resizable BAR lets the size of the aperture be chosen (which is usually chosen to allow all of VRAM to fit in and be directly accessible).
MMIO (Memory Mapped I/O) is essentially memory (Whenever RAM or ROM) from OTHER devices that is directly visible on the CPU Address Space. My understanding is that from the CPU side, MMIO is mostly transparent (Except for the massive increase in latency) because it gets used like if it was interacting with its own workspace with regular instructions like MOV.
What PCIe ReBAR changes is that before, you could only see a 256 MiB window onto the GPU VRAM, so there was an added overhead since the GPU may need to relocate things from inside that window somewhere else on its total VRAM (So yes, it may be interpreted as if what you see from the CPU side is just some kind of exchange buffer). I believe the best way to describe how it operates is comparing it to EMS (Expanded Memory) from the DOS days since it also worked with a similar, if not the same idea. You could only see a portion of the total memory from what was installed on the EMS card (A 128 KiB window located on the upper part of the 1 MiB address space from the 8086 CPU), so you had to switch which Page (Region) of the memory was visible, adding a lot of overhead and most likely requiring an additional buffer in main RAM to move data from one Page to another. However, since I have no knowledge if the GPUs really work like that I can't confirm. I never knew whenever the 256 MiB is "fixed" (You always see the same Region) or if you can decide which section of the VRAM to make visible.
[1] https://en.wikipedia.org/wiki/Advanced_eXtensible_Interface
And if my experience from embedded development is in any way transferable, they're probably fairly spread out and probably takes a fairly big chunk of address space too.
Other similar mods included adding NVMe Boot capabilities to older Firmwares that didn't supported them so that you could boot from modern NVMe drives if you used some kind of adapter (As none of those Motherboards have M.2 Slots, but you can use them with a PCIe-to-M.2 adapter card). Also adding required stuff to get certain CPUs booting in Motherboard that were never officially supported (Depending on generation, could be as simple as adding new CPU Microcode).
e.g. they do seldom if ever take patches for adding support or fixing issues in older boards. They just sit there ignored.
A fork or entire new project with different focus is sorely needed.
There do exists individual users that work on newer stuff, after the MSI Z690-A / Z790-P there were 2 people that did ports for H610 since at that point the platform was supported and they only had to port the Motherboard. However, they are rather rare, since the kind of user that Coreboot seems to attract are from the privacy and security anti-ME/PSP crowd that don't even want to use newer platforms, which is the exact opposite of my view.
A lot of the work companies does may be just for their internal systems or reference platform and these may not get to the hands of end users, ever. I still recall Phoronix articles covering Coreboot releases always taking a jab at the new supported Motherboards until the MSI Z690-A DDR4 port from 2022.
Claude tells me the BAR is outside the encrypted part of the card firmware so theoretically should be possible?
[1] Intel GPUs actually can't work without it, because they do not have legacy "windowing" mode where portions of 256MB BAR window are mapped to different areas in GPU memory
[2] This tool implies that it's possible to handle root port side entirely in firmware, but it's new to me at least, so I can't vouch for that :)
Sounds scary. Like there is a good chance to brick the motherboard permanently. I think i'm going to skip this for my ye olde pc with crappy GTX 1660 Ti card. Can't afford a new system.
[1] https://github.com/xCuri0/ReBarUEFI/issues/11
I moved away from chrome a while ago, but last time I installed it on linux it was a pain, and firefox and its derivatives are essentially at feature and speed parity or better than chrome across the board.