Ask HN: Physicists of HN, what are you working on these days?
Of late, except for few headline-friendly fields (colliders, quantum computing, gravitational waves and astrophysics in general), I don't get to see/relate with a lot of activities in Physics. Also I have noticed a growing trend of physicists becoming data scientists post phD. Although I understand the money factor, are there any other reasons for this as well?
332 comments
[ 3.0 ms ] story [ 263 ms ] threadIt was 2003 when the company I was working for acquired InterBiz from Computer Associates. By the time we were integrating teams from both companies I met this guy who were the guru, the #1 product manager from one of the most profitable lines of business that the company had. I remember the shock in people's faces when he, a Portuguese guy, said he's got a PhD in Physics, in Germany, in a second language to his own, but then ended up developing Warehouse Management Systems (WMS).
Yeah. I keep kicking myself for not taking the Wall Street guys who called me more seriously. That would be one important message I'd like to send backward in time.
I remember some of the Physics departments I knew well commenting that they were getting 250+ applications for each open tenure track position (early/mid 90s). It skyrocketed to 1000+ at one point for a few places.
Based on some back of the envelope math, and a realization that a fresh Ph.D. with ~6 publications (2 PRL, 4 others) would not be a good competitor to a senior FSU physicist with 50+ publications to their name, a solid reputation, and an interest in moving out of the FSU quickly.
I determined that market forces had flooded the supply side of physics with, well, extremely good candidates for the same positions I was applying for. And my ability to compete with them was low based upon the main components that hiring committees cared about.
So I looked elsewhere. 20+ years later, I've made a good career working in computing, but I really do miss physics and research in general.
This was certainly a lot fun, but you won't be getting any thing particularly revolutionary out of it. I remember my advisor at the time running a few dozen CPUs of our poor excuse for a cluster (built mostly by yours truly with off the shelf components) for the entire summer to add a couple of digits to the ground level energy estimate of a completely artificial magnetic model :)
On the bright side, it taught me a lot about programming, numerical simulation and optimization
> So why don't we have a good theory of brains? People have been working on it for 100 years. Let's first take a look at what normal science looks like. This is normal science. Normal science is a nice balance between theory and experimentalists. The theorist guy says, "I think this is what's going on," the experimentalist says, "You're wrong." It goes back and forth, this works in physics, this in geology.
> But if this is normal science, what does neuroscience look like? This is what neuroscience looks like. We have this mountain of data, which is anatomy, physiology and behavior. You can't imagine how much detail we know about brains. There were 28,000 people who went to the neuroscience conference this year, and every one of them is doing research in brains. A lot of data, but no theory. There's a little wimpy box on top there.
Not that physics has no theories, but I dropped out of studying physics myself over a decade ago, and at that time it felt a lot like the balance in physics has shifted towards having to measure and process disproportionate amounts of data with so much precision that it has to be automated, or like you said do a ton of really complicated modelling. It feels a bit "stuck" that way.
[0] https://www.ted.com/talks/jeff_hawkins_on_how_brain_science_...
Most parts of the Standard Model are verified to 5σ at all energy levels that are accessible to us in everyday life and in contemporary particle accelerators, so it's increasingly unlikely that we've missed any obvious mechanisms.
If we're going to find some completely new physics (in the same way that quantum theory and relativity theory were completely new at the turn of the 20th century), it's probably going to be at energy levels that contemporary particle accelerators cannot yet explore.
EDIT: That's not to say that we know everything that is to know about the accessible energy levels. There's so much in our theory that's yet unexplained, e.g. for all I know, we still don't have any idea why time works the way it does.
The reason for is just how damn good all the Physics Theory is. People keep having to look closer and closer to try to find a place, any place, where the theory "fails" (in a sufficiently spectacular fashion) hoping that that might point the way to new phenomena and new theory. There's a giant multibillion dollar hole on the ground in Switzerland who's main purpose was to find a chink in the armor of HEP or help point the way to new theories. It failed spectacularly, but they're already talking about building a bigger one :)
> (majewski's comment below) I think you're seeing this in physics because, in terms of experimental accessibility, the low-hanging fruit are mostly picked.
I mean, yes and no. The Standard Model is amazing. But I think we're also kind of focused too much on the Big Questions, and in the arena of the not-so-big questions there are still a ton of gaps and missing insights.
Perhaps in the not-so-big questions, most of the low-hanging fruit may appear to be picked as well, but I suspect that is more a kind of myopia that can be solved by collaborating outside of our field.
Whenever I see an article these days about physics that really excites me, it tends to be a story where one or more physicists got excited about a seemingly small question, or ate a bit of humble pie and joined forces with chemists, biologists or some other field, and or both, and that working it out turned out to bring far-reaching consequences and novel insights. Same with maths, really.
I'm aware this is completely personal of course, and that there is a likely bias in that this kind of research tends to be more accessible and thus more fun to read for me, but still.
Awww it isn't that bad. They found the Higgs and at least verified we need to look at higher energy levels to find anything new.
Now I develop software for a medium-sized ISP and IT outsourcing company.
A surprising amount of people whom I studied with ended up as programmers themselves, after finishing the physics degree.
Then again, I guess I value stability and predictability in my day job a bit more than most Silicon Valley startup employees.
But I enjoyed my time as a physics student. Physics is an interesting field to study even if you don't end up working in it, in my opinion. :)
I saw a postdoc who is now rather well known struggling with anxiety over his career even though he had written half a book and done a lot of great work. When we were both at Cornell I'd come to the conclusion that many papers involving "power law" distributions were bogus because nobody knew how to test for them with any rigor. It was years later, after he had tenure, that he published something about it in a statistics journal.
Seeing that made me run for the exit after my first year as a postdoc.
> after he had tenure, that he published something about it in a statistics journal
If I'm reading you right, you're saying he struggled for awhile doing bogus things only to question those things outside the relevant field?
(Also, can you say more about power law papers..?)
So there is a theory based on the renormalization group that can explain many (but not all) "fractal" phenomena that are observed.
I was part of the false paradigm of "bin up a probability distribution", "plot it up on log-log paper", and "draw a straight line."
Sometimes when you do that you will get an answer that has something to do with reality but it is not a valid answer to the problems of: "is a power law distribution a good model of this phenomenon", "what is a good estimate of the exponent", "are these two power law charts drawn from the same distribution" not to mention how to handle the problems that turn up at the highly frequent and highly rare ends of the distribution.
The root cause is this attitude
http://thinkexist.com/quotation/if_your_experiment_needs_sta...
and an academic system where power is too concentrated, where people who write review articles do the most good but get the least career advancement, etc.
I've been out for a few years, so I don't know what the current state of affairs is.
Is there any published criticism against his work? His papers get cited everywhere, I know citations are more metric of popularity than scientific quality but I find it puzzling that his work continues to get attention if the results don't hold up.
My perspective is that there is just not many academic positions available. I realized I could do a couple of post docs for a few years and hope that something opens up, or not delay the most probable outcome and start an industry career sooner rather than later.
Embedded graphics drivers for real-time systems.
I keep the physics part of my brain alive by developing physics based Unity assets (nbodyphysics.com) and supporting a package for GR on github (grtensor).
I still buy WAY too many physics books. Current aspiration is to work through "Modern Classical Physics" Thorne/Blandford.
If you're thinking of getting it but want to check it out, there's a 2012 draft version that the authors have previously taught from here: http://www.pmaweb.caltech.edu/Courses/ph136/yr2012/. It's not the same as the book, of course, but from a skim it seems quite similar.
A good choice for less math might be "Gravity from the ground up" by Schutz.
My grad work had no connection to data/stats and I have not bumped into any low hanging fruit where Riemannian geometry might be the answer!
The central issue is understanding how changes in control parameters (for instance concentrations of catalysts in a chemical system, or local fields in a spin system) affect the evolution of the probability distribution over states. Some work has been done in close to steady state (for instance [1,2,3]) but it's far from resolved.
This has some nice applications - designing efficient protocols for microscale devices, for instance.
[1] https://arxiv.org/abs/1603.07758 [2] https://arxiv.org/abs/1507.06269v1 [3] https://arxiv.org/abs/1201.4166
None of this pays the bills, right now I help build clouds, and I used to build supercomputers, and high performance storage systems.
Yeah, I'm keen to give that a go. Interestingly it is not a problem that is very easy to parallelize, since every mass will affect every other mass. There may still be some cool ways to use ECS though.
Somehow I doubt it! In any case, you are like me - doing engineering and software, while doing my physics as a hobby and loving it. I'm working on Physics from Symmetry. Definitely a 21st century book - not at all like the traditional development of the subject.
Now I'm doing aerospace vehicle modeling and simulation in MATLAB and C++ and primarily work with other physicists. This is by far the most enjoyable work I've done and the pay is excellent - my salary is 3-4x the average household income for the city I live in.
While my PhD did start out as supposedly being on Spin Glasses it quickly diverged to Complex Networks and what people would now call Data Science. Since then, I've worked on Social Science, Epidemiology, Human Behavior, etc... For the last year or so, I've been doing Data Science and Finance in one of the Big Banks.
Working on my thesis analysis (collider based HEP), squeezing in time for OSS development and prepping for a jump to software dev.
I love physics, but academia is in a rough place right now. Almost everyone I know pursuing the academic career path has nightmare stories.
The physics job market has been, and will remain in a "rough place". As with all things, it is who you know.
If you want to stay in the field, network like mad. Get people to know your name. Make sure you have work known to them. Get a great Postdoc with someone who will make calls for you when you are done with your project, to help your search.
Otherwise, computing is nice :D
Though theory groups in general tend to use computational simulations as a tool to complete calculations, groups that develop novel computational methods and techniques tend to be headed by younger, more junior professors. These groups are typically well-funded and do very exciting (trendy? cutting edge?) work with distributed computation, machine learning, neural networks, etc, so they tend to pull quite a few students.
While these computational groups tend to bring in funding and are well-staffed by excited grad students, the junior professors leading them tend to be marginalized by the more traditional, seniority-focused establishment. Which is to say, a new PhD might have a lot of trouble landing a prestigious postdoc because a) their adviser might have been too young to have high name recognition outside their field and b) departments might place limits the amount of staff for these more junior professors/young groups doing exciting computational work. This is, of course, on top of the overall scarcity of jobs in academia.
But there's no such job scarcity in industry-- especially not for stats-smart programmers with years of experience a) wrangling data in python, b) writing fortran that runs on distributed clusters, or c) designing algorithms to solve /approximate hilariously expensive problems. Advisers know this and point some of their students who might thrive more in industry than academia towards that route.
(Anecdote: And of course, as a physicist who builds models/simulations in industry, I can speak a personally a little re: thriving. If you're someone in love with solving disparate problems, you're unlikely to find that in academia. Some of us learn in graduate school that we can't spend our whole lives-- or in my case, more than a few months-- solving one problem. Academia just... didn't seem like something that would be worth fighting for.)
I assume that this will gradually change as there's turnover within physics departments and we get more computational-first professors with seniority (or even in leadership). There are a few departments with better-known professors you can see it happening now. Universities are spinning up incubators and institutes for computational research. Physics departments are just slower to adapt to new developments, and the hierarchy of theorists can have more to do with seniority and internal politics than it does with technology.
The fundamental issue is the field is not growing (very much). Each professor will graduate 20-50 students over their career, but only one will get their job when they retire (on average).
> Some of us learn in graduate school that we can't spend our whole lives solving one problem. Would you please expand on this. I am not sure if you meant that problems are hard enough or what.
Sounds like engineering and SE, not physics
The reason was I sucked in physics, did not find my PhD position motivating and loved computer graphics.
13 years later I'm a valued technical contributor in a team in an ISV creating a valuable global software packages in the CAD field.
Middle class income, could probably make lot more in US with my skillset but family situation really is not awesome for expatriation.
I'm doing OK.
With physics skillset you can pretty much make your career what you want it to be. You just need a proactive attitude to read on other fields. You need to understand the other guys mindset so you understand the overall game going on. It's usually not nefarious (although it can be), but most of the time the rules that are used are not the vocalized ones. In this empirical physics is a wonderfull philosophical background. Organizations have certain dynamic rules which half of the people are not aware of. You don't need to "play the game", but you need to understand the rules so that when the wind blows into the direction you want to go into you can grab the opportunity.
I had a pretty good idea where I wanted to be 13 years ago (R&D in a position that values quality over quantity and speed with a great team) and that's pretty much where I am now.
You need to know where you want to go, and go there. No one will guide your path. Physicists are an outlier but that math and mindset is really an asset. Just don't get stuck in fixing bugs in some legacy monstrosity, that is absolutely soul crushing. But such a position can function as a stepping stone if you are operating in the industry you want to work in.
What's an ISV?
ISV - independent software vendor. An entity having an ownership of a software product, usually developing the software as well.
The other entity this is often compared against is the ESP - external service provider. I.e. consultants.
I spent years with legacy monstrosity and it does develop important skills. If the organization is otherwise ok it might not be that bad - but generally production software is absolutely horrible. The thing is legacy maintenance is important, and there's quite nothing like it that will teach you about the lifecycle requirements of software development. But I find it much more fullfilling to maintain and develop software that is alive and well and actively kept out of the 'legacy' label.
Not all old software is 'legacy'. One definition is that does it have tests. I've found an equally good definition of 'you are not afraid to modify it' and 'you can understand what changes in one place do elsewhere'.
So to be clear, I was not saying "anyhing but greenfield development sucks".
I have begun looking about at jobs outside of academia in order to decide whether to stay or go. Data science is one of the easiest transitions a physicist with data-analysis can make, which I think explains the prevalence of physicists in that role. We have the training in both the techniques and a sober assessment of uncertainties, which makes us desirable.
So far, in my search for outside employment, I haven't found anything that draws me as much as my present work, but if you're in the Seattle area and looking for an experimental physicist with a broad range of experience, please get in touch.
Imagine taking Newton down in the Replication Crisis Wars!
"The Replication Crisis" has basically involved rather different fields - psychology and human biology/medicine most often. The crisis could be called "human and animal testing crisis" imo.
https://en.wikipedia.org/wiki/Replication_crisis
What we do is search for deviations from Newton's inverse square law. Depending upon the distance over which an experiment is performed, the ISL can be correct to a precision as high as ~10^-10 [1]. In particular, our group works to test whether the inverse-square law (or indeed gravity at all) happens as predicted at distances shorter than 100 microns [2]. It still amazes me that at distances shorter than half the diameter of a human hair, nobody has any idea whether gravity even happens.
Newton's early observations (and Kepler's) have been replicated many times -- there is no risk of a major replication-crisis upheaval there.
[1] https://arxiv.org/abs/hep-ph/0307284 (Figure 4)
[2] https://arxiv.org/abs/hep-ph/0611184
I graduated and did a few strtuos, now working on autonomous self-delivering ebikes in SODO.
Any room for scientific software developer at LIGO? Gravity is a beast... I will leave it at that. But I can signal process and I am not afraid to ask stupid questions. Looking for growth industries, I've had my eye on quantum computing and am just putting resumes together now. I'm really into differential geometry and topology but my understanding is amateurish and what I've learned of QFT, geometric physics, topology, etc., is uneven at best.
If you really are interested in software dev / data science I can ask around my company. They have a Seattle office* and a broad range of needs for different developer backgrounds --- at least when all offices are considered. Typically there is wiggle room to work from one office with a group based elsewhere. I can't guarantee anything though -- it's been hard to get anyone hired lately!
Back to gravity and such: I doubt you find anything, without physics in it, that draws you in as much as your present work, but it can't hurt to look around! How about a switch to focus on quantum computing if data science doesn't draw you? Bar that, what about numerical simulation?
*I'm not in Seattle.
That said, if one wanted to move toward LIGO from a software background, the chink in the armor is that the signal-processing is hard and computationally bound. Helping signal-analysis groups to find efficiencies there would have an impact.
I continue to keep an eye on quantum-computing infrastructure hires in the Seattle area -- so far, I've proven to be too physics-focused to want to work solely on provisioning the (awesome) cryostats that house the quantum-computing devices under test. That may prove to be a sub-optimal life-decision -- time will tell :).
"For now, however, in hard-core physical science at least, there is little evidence of any major BD-driven breakthroughs, at least not in fields where insight and understanding rather than zerosales resistance is the prime target: physics and chemistry do not succumb readily to the seduction of BD/ML/AI. It is extremely rare for specialists in these domains to simply go out and collect vast quantities of data, bereft of any guiding theory as to why it should be done. There are some exceptions, perhaps the most intriguing of which is astronomy, where sky scanning telescopes scrape up vast quantities of data for which machine learning has proved to be a powerful way of both processing it and suggesting interpretations of recorded measurements. In subjects where the level of theoretical understanding is deep, it is deemed aberrant to ignore it all and resort to collecting data in a blind manner. Yet, this is precisely what is advocated in the less theoretically grounded disciplines of biology and medicine, let alone social sciences and economics. The oft-repeated mantra of the life sciences, as the pursuit of ‘hypothesis driven research’, has been cast aside in favour of large data collection activities [7]. And, if the best minds are employed in large corporations to work out how to persuade people to click on online advertisements instead of cracking hard-core science problems, not much can be expected to change in the years to come. An even more delicate story goes for social sciences and certainly for business, where the burgeoning growth of BD, more often than not fuelled by bombastic claims, is a compelling fact, with job offers towering over the job market to anastonishing extent. But, as we hope we have made clear in this essay, BD is by no means the panacea its extreme aficionados want to portray to us and, most importantly, to funding agencies. It is neither Archimedes’ fulcrum, nor the end of insight."
https://royalsocietypublishing.org/doi/full/10.1098/rsta.201...
> Yet, this is precisely what is advocated in the less theoretically grounded disciplines of biology and medicine, let alone social sciences and economics. The oft-repeated mantra of the life sciences, as the pursuit of ‘hypothesis driven research’, has been cast aside in favour of large data collection activities
The thing is, I've just spent two years working for molecular neurobiologists in the field of Single Cell RNA Sequencing, and large data collection has definitely lead through tons of breakthroughs there.
We can now classify cell types based on gene activation, on top of the previously existing morphology and location the cells are found. That can then be used to discover new subtypes, the origins of cells during embryonic development, and even predict which cells will evolve into others[0][1][2][3]. All of this requires vast amounts of data to ensure there is enough statistical power. In fact, the insistence on using unbiased samples before applying clustering algorithms is a big part of overcoming biases based on pre-existing expectations.
(Also, may I request that you edit your comment and break up that block of text into sub-paragraphs, for the sake of readability?)
[0] http://mousebrain.org/
[1] https://linnarssonlab.org/osmFISH/
[2] http://gioelelamanno.com/post/velocitynature/
[3] https://www.nature.com/articles/d41586-018-05882-8
So if we take some of the thought processes behind Stat Mech (or S&M as we used to call it in grad school), and you kinda squint your eyes hard to blur the less robust discussions you read about, you get the sense that ML is more about the "thermodynamics of information" than anything else.
I find this intriguing and definitely want to spend more time on this stuff.
This is so sad
You can merge ML and theory in at least one way. I attended a talk by Prof. Karen Willcox of the University of Texas at Austin (I'm a PhD student in mechanical engineering there) where she argued that in fluid dynamics and combustion at least, it's better to use "model order reduction" instead of machine learning. The problem with many models (e.g., Navier-Stokes equations) in these fields is that they are computationally expensive. Model order reduction looks for ways to reduce the computational cost of the model while maintaining accuracy, and it uses many of the same techniques as machine learning. Based on the examples she gave it seemed to be the closest thing I've seen to merge the two.
https://en.wikipedia.org/wiki/Model_order_reduction
http://kiwi.ices.utexas.edu/
This makes me so incredibly depressed.
I'll also be joining the data science world.
There's some debate about whether it's the best or worse time ever to be in particle physics. Either way I see a field that is overstaffed. Add to that the fact that CERN accelerators are shut down for two years and we're in the middle of the european strategy review.
Seems like a good moment to take some time out
One thing I noticed is that because physics has multiple decades more experience with dealing with big data compared to just about every other scientific field, a lot of physicists who jump ship tend to end up in a position where they can apply that expertise.
I worked as a programmer for a molecular neurobiology research group for two years. Biology is going through a kind of Cambrian explosion of new data (especially when it comes to anything that involves genetics). So it's probably not surprising that a number of people at work told me that it is extremely common to see physicists switch to biology because that's where all the exciting new research is happening, with new theories and discoveries, and lots of people who are very happy to steal whatever the physicists have already figured out about how to process and interpret mountains of data.
Finished my PhD in quantum optics in 2014, but immediately moved to data science.
Why not physics? Full version: http://p.migdal.pl/2015/12/14/sci-to-data-sci.html
tl:dr: I wanted a fast-pasted field with more freedom. Physics is now stale (no fundamental changes in the last decades, compared to each year in deep learning; cf. physics in 1900-1920), and academia offers a rigid framework of grants and feudal dependence. In data science, as a freelancer/consultant, I get much more freedom. Even in companies, one is able to migrate in a matter of weeks, not years.
Money was a nice perk, not anywhere near to the main motivation.
For myself at least I would add job security as another benefit. I struggled in vain for 6-7 months during my lectureship applying to any and all postdocs I could find, so the prospect of cycling through 2-3 year postdocs in various locations eventually overwhelmed me. I loved the research but realised I didn't love it enough to potentially live apart from my partner and regularly go through months-long application processes.
On the flip side, for me freedom is actually something I miss from academia. Admittedly that's partly because I'm a regular employee rather than a freelancer. Outside of teaching, my time was basically my own and I was free to work from home or anywhere else. I'm somewhat resentful of having to justify what I'm spending my time on day by day. Not that my manager would particularly mind if I spent a day researching something slightly blue-sky, but the fact that I might have to respond to questions about it creates this background tinge of anxiety that I find prevents deep creative thought.
I'm happy with the change, my life is more relaxed now. I've time for my family, for me. I've also better and stable earnings.