How your morning coffee is changing the structure of your brain (newatlas.com)

Frustratring lack of conclusiveness but alas, such is science.

Hard to imagine how a reduction in neural cell/synapse volume, albeit acute, wouldn't have any lasting effects.

  "The effect of caffeine on the brain was noted as particularly relevant in the right medial temporal lobe. This area of the brain includes the hippocampus and is responsible for processes such as memory formation and spatial cognition."

If you become temporarily worse at memory formation, that stands to reason as being ipso facto bad for overall memory accumulation.

Small sample size 20 participants 10 days

https://academic.oup.com/cercor/article/31/6/3096/6135013

Completely anecdotal; the amount of people that do not consume caffeine seems overrepresented among very very smart people (i.e. math profs, researchers etc)