How would you rebuild the computer from the ground up?

I've only scratched the surface of low-level systems (mostly x86 and Unix), but their complexity is mind-boggling.

We've evolved directly from a system designed in the 70s, when we had no idea what we'd want, need, or be able to do in 2017.

Forget all the inertia for a second, and imagine you've got infinite resources at hand.

How might you do it differently? Wipe the slate clean and remake everything – simpler, faster, easier, or to open up avenues that just aren't feasible to go down today.

Do you work in a problem domain where we're especially held back, by the hardware itself, or its complexity?

A couple of related links:

Gary Bernhardt talks about speedups from ditching virtual memory (at least partially)

https://news.ycombinator.com/item?id=7605687

Can programming be liberated from the Von Neumann bottleneck?

https://news.ycombinator.com/item?id=12159792

Tyranny of the clock

https://news.ycombinator.com/item?id=11995966

Rex

https://news.ycombinator.com/item?id=12155561

Mill

https://millcomputing.com/