As an internet startup, at what point does PHP start to bottleneck you?

It depends on what you're doing obviously, but for the majority of businesses the answer is "Never". A well written PHP/MySQL app will cope with tens of thousands of users easily. There are good reasons to use other languages (security, ease of use, strong typing, maturity, lower developer costs, etc) but choosing PHP certainly isn't going to make your startup any more or less likely to work.

At one of my previous jobs we scaled a large travel website you have heard of on PHP pretty successfully. The only thing that hurt us badly was using Symfony, it was a HUGE memory hog and exposed really bad issues with PHP APC. We basically need 1gb of ram per CPU core, an insane figure compared to modern PHP.

This basically caused us to not use the framework in specific areas to maximize performance, and use a network layer cache to prevent booting up the framework entirely.

As with most web tech, the main issues revolved around our database scale, we had to switch to multi-master around 2012 and it was a huge pain in the ass, also database migrations causing multi hour replica lag...

Honestly with modern cloud infra, auto scaling groups or serverless, I would have no concerns scaling PHP.

The way I always put it to anyone asking this kind of question: Build an app that scales that large and then hire someone else to deal with it :)