Ask HN: Has anyone worked at the US National Labs before?

Expect a very different work culture. If academia is cozy to you, you'll fit in fine.

Things don't move fast, as another commenter said. In my area of work, projects tended to last 1-3 years and you'd be on several projects at any given time. In general, it is ICs rather than managers who run the projects. Your manager might say "Bob over in 9876 has a neat project that could use somebody like you, send him an email if you're interested".

You have to acknowledge that the core mission of the DOE National Labs is nuclear weapons. You might not ever come in contact with the mission, but it is there. They have strong HPC programs--because HPC as we know it is basically driven by the need to simulate nuclear weapons. Some people have moral objections to this, and that's fine!

I thought it was a good place to work, all in all.

Edit: I'd like to stress that probably the biggest advantage of the labs is the opportunity for self-directed work. If you can convince somebody (external sponsors, internal R&D funding committees) to give you money, you can work on just about anything. If you can't get funding of your own, you are still more or less able to choose what you work on.

Your work environment will depend highly on which group you're in. Some groups look like a university department without the students: you work in the unclassified area, you publish papers, you can even open-source software (with some effort). Other people spend their whole day in a windowless SCIF working on very sensitive stuff which they can never, ever discuss outside of a SCIF -- but while their public visibility is nil, their impact is arguably greater.

I interned in high school at Brookhaven National Lab working on a team that analyzed STAR (Solenoidal Tracker at RHIC) data from RHIC (Relativistic Heavy Ion Collider). I didn't contribute all that much as a high school intern but the program director said at the end that he liked the high school program because he wanted to help funnel and bring people back to help build up the labs.

My experience was that everyone there was extremely smart, but all post-doc and top scientists in their field (the team I worked on was looking for Anti-Alpha particles from gold-gold particle collisions that also helped create Quark Gluon Plasma). So I'm not sure their relative need for regular software engineers.

In terms of bureaucracy, you're still working for the government. The scientists all complained about the layers of government bureaucracy but were mostly okay with it. High-tier science moves at a pretty slow pace; coming from a tech background you might not be used to the slow pace around the actual physical construction of some of these devices, let alone the fund-seeking, approvals, testing, runs, and data collection. and 33-50% is a hopeful estimate. Let's say one is a 500k a year senior/staff SWE at FAANG. at a similar level of experience, one's pay would be very lucky to break 150k.

So fascinating science, layers of bureaucracy, slow moving stuff, PhD's in their fields, and reduced pay. Again I was only a high school intern, but I spoke with the scientists about their experiences so take my recollection with massive salt. I walked away from the summer fascinated by the work and I had a love of physics at the time; but I also left (this was 2010 IIRC?) watching the world of tech explode at a massive pace and thought that I didn't like physics enough ( I had spent my junior/senior year of high school doing a capstone project on theoretical physics and having taken a lot of physics classes). When I went to college the next year, I tried a few engineering courses, and switched to CS. I'm glad I made the switch.

I did workflow management systems, environmental controls in labs, and lightning prediction software.

https://carcc.org/

Sometimes prior clearance can be negotiated and a well qualified candidate who is likely to clear might be accepted and placed in a holding pen until they clear but I'm not sure what the backlog is now or even if they do that anymore. At the very least you will almost certainly need to be a US citizen proper.

There's also annoyances coming from political things, such as the budget not being done on time so no one gets paid or there are furlough days.

The people I worked with were super smart in their fields, but were pretty bad at writing code / handling data outside of Excel so they usually hired interns to help with code-related stuff. Some of the divisions had full-time software developer teams, but I was the only software developer in mine.

The pace was extremely relaxed, deadlines were not tight at all.

I worked remotely, but came in to the office a few times a month. The campus is beautiful, as it is right inside a nature preserve. Everyone there is doing scientific work, so it feels like a real scientific think tank atmosphere and I loved it.

https://usds.gov/

You will find a research group within a division to work for. For example, mine was the Computational Earth Science group (since renamed) within the Earth & Environmental Sciences division.

You will be working with a handful (3+) of research scientists as their supporting engineer. On some projects, you may be doing machine learning work in Julia. On others, you may be coding a fluid dynamics simulation in FORTRAN or C++. On others, you might be doing data analytics in Python. It's highly, highly variable, depending strongly on the PIs you're working with, and can change as frequently or infrequently as you wish (within reason).

Ultimately, I did the reverse: went from a DOE lab into a FAANG company. My reasons are particular to me, but if you're at all interested in a slower paced, more varied and collaborative environment, you can't do much better than working for the labs.

For context, at LANL, I was making ~$100,000 / yr with 3 YOE (circa 2019). This is in northern New Mexico, with such a low cost of living that this amount of money goes about as far as $150k+ up in the Boston area (where I am now).

[0]: https://www.lanl.gov/

I liked the work and really enjoyed getting to be a consultant on many projects. Turnover is massive among the researchers because there are few permanent positions, and most groups are heavy on postdocs since graduate students tended to be primarily on campus (UC Berkeley).

If pay is a concern, look closely for the open databases of salaries. At LBNL there is the "book of tears" at the library under the cafeteria, listing every employee and their salary. The exact amount you get varies wildly with the department: prior to unionization in 2016, the range was from 20k to 125k annual salary for postdocs. I hear they raised the floor to NIH levels at least, but I assume they did not make NERSC take a paycut.

Staff levels probably pretty good if your motivation isn't to move upwards and focus on whatever your interests are.

These are my speculations.

The cafeteria is way less fancy than tech company cafeterias. :-)

As a few posts have pointed out, there are national labs that do only unclassified work (LBNL is one). So you don't have to get a clearance or be prohibited from accessing lots of places or conversations, and you don't have to work on weapons. You do still have to sign a loyalty oath as a state government employee (the lab being managed under contract by the University of California), something that became highly objectionable to me in retrospect.

https://www.techstars.com/accelerators/industries-of-the-fut...

LQCD is kind of funny because it doesn't YET have anything practical to say about nuclear physics, which is what the lab cares about, but it will someday. So I was pretty insulated from all the weapons+complex integration stuff; my work was 'pure research', which is not that common (though it is more common at the postdoc level, which the lab views as a way of recruiting talent). But unless you can find your own funding (usually from a DOE grant), you're working on something that advances the lab's programs. I can't bring myself to work on nuclear weapons, which is why I didn't stay [there's a lot more to LLNL than that, of course, but it's what my field funnels into, broadly speaking].

The computational expertise for HPC is really unparalleled, especially at Livermore (and Oak Ridge, which I've only visited). They're consistently pushing the envelope in terms of high-performance machines which can address scientific questions that require extremely tight coupling between computing resources, rather than a cluster, and they have a lot of experimental architectures and things like that. LLNL publishes a lot of open-source software; if you've used a cluster in a scientific setting you might be most familiar with SLURM or spack.

The day-to-day can be a bit surreal. At the defense labs people with enormous machine guns thoroughly check your badge on the way in. On your walk to the cafeteria you might pass a beach volleyball court that's inside the superblock [an extra-high-security area where they've got plutonium etc.], next to a machine gun turret. Very few employers have teams that regularly win SWAT competitions.

The food was fine. No luxuries like free snacks or anything else I'd seen my tech-company friends enjoy. No dogs allowed. LLNL has a lot of employee organizations for sports, charities, exercise, etc. Transportation around the LLNL site is via sporadic shuttles but more practically there's a bike share, which is just a bunch of bikes you can leave anywhere (on the sidewalk / by a building).

My recollection is that the whole experience depended on which group you were in, and mine was fortunately very chill. Smart, friendly people who arrived and left more or less at the same time every day. Lots of matrixing and loaning of people from different orgs -- I had the feeling that if I were making a career there I would wind up slowly drifting around between projects.

The biggest surprise to naive me-in-my-early-20s was that "Department of Energy" is a euphemism for "Department of Nukes." Nuclear stockpile stewardship was a large portion of the activity there, and so a lot of your colleagues will be people who are at least vaguely comfortable with that.

There was a ton of "basic research" too -- some high-energy group had a daily experiment that would deliver a "whomp" of a shockwave around 3:15pm most afternoons, there was a room temp. fusion group, lots of interest in assisted driving cars and unmanned aerial vehicles... you just had to appreciate that all the first applications of all this tech was going to be military.

Also the security clearances.... the joke was that the "L" clearance stood for "Lavatory pass" because in our building until you got one, you needed a line-of-sight escort at all times, even in the bathroom. Even for the "L" the process was quite onerous, and I understood that the 'Q' clearance held by nearly all full-time staff was even more burdensome. I heard stories of people waiting for their clearance getting stuck in rooms with nothing to work on. One person in my group basically got sent offsite to some "think tank" or something for several months while he waited for his clearance - I only met him once the whole summer, at a conference.

You might also want to check out the US Digital Service, which might be more aligned with traditional SWE skills.

The campus there was also different than many other national lab campuses in that it's an open campus and doesn't have the military entrances that others have. It felt like the culture was much more laid back than the FAANG and other corporate cultures that OP mentioned, but perhaps more bureaucratic as well. Again, I was an intern, so didn't have much visibility into that aspect. Overall, definitely a positive experience and I could see myself there if things lined up right.