Surprised that this is missing a recommendation to run 'npm shrinkwrap' and include npm-shrinkwrap.json in your deployment package so that the dependencies you have been testing against (and more importantly your dependencies' dependencies) don't shift unintentionally under your feet. Although maybe this isn't a problem with npm v3?
I'd also recommend including your dependencies in your deployment package anyway to avoid a deployment being held up by npmjs.org downtime.
Author here! You're totally right, I've just added a shrinkwrap section.
PS. npm v3 still needs it, in fact it makes it better - `install --save` updates the shrinkwrap file.
Re: checking in modules, I used to do this during 2013 when npm was up and down every few days, but stopped a couple of years ago after npm Inc stabilised everything. It's been solid so far and the smaller repo sizes (and faster deploys) have been worth it.
As someone who never deployed a node app I was a bit surprised to see it doesn't require a server like Apache or Nginx as a reverse proxy ? Is it a commmon practice to deploy node apps like this ?
For most cases you want load balancing, hence HAProxy. But if you don't, node has event driven IO like nginx does, so it's quite capable of things like static file serving and https (like nginx, node uses openssl for all the hard work).
Don't you think nginx is better for serving static files? I first served directly croon node with express and did a test with chrome with a simulated modem connection speed. A simultaneous fast client was blocked because the number of connection nodejs/express was exhausted by the slow modem connection getting a lot of js files...
I think I also read many other blogs suggesting serving static files with nginx in front of nodejs. Are you sure that this does not make sense?
It's not so much a matter of 'better' vs 'worse' as a balance between complexity and speed. node has very fast IO - that's what it's known for - and unlike Python or Ruby it can handle static files at speed: http://i.stack.imgur.com/amngX.png (from the second post at http://stackoverflow.com/questions/9967887/node-js-itself-or... which provides some excellent discussion). nginx is faster, but small projects that don't need load balancing may benefit from having less software to install.
As the article mentions, you'll probably want a load balancer for high availability. Whether you use haproxy or nginx for that is a whole different discussion.
Faster than Ruby or Python doesn't mean anything if it's objectively slow. And it is. I've only been able to serve a small handful of concurrent users at a time with Express. I'm talking 5 or 6 before things get out of hand.
You should aim to have the bulk of your payload be static, and all of the static payload be served by apache or nginx. It's the difference between getting good feedback from a Show HN post or completely missing out and looking like a fool.
> I've only been able to serve a small handful of concurrent users at a time with Express.
Something's massively wrong with your node setup. node won't be as fast, but it should be on the same order of magnitude as nginx: https://github.com/observing/balancerbattle (or any other benchmark)
After a few years of deploying apps in a reasonably similiar way to this, I switched to using dokku for deploying node applications recently, the experience so far has been extremely nice.
I get `git push dokku master` deployment for free, for any application, I dont have to worry about conflicting node versions between applications and it took a lot less setup for all of my applications than this process describes for one.
Sure - as discussed in the article, the steps of deploying a server should only be performed once, and customised to your needs.
From then on you can - and should - deploy everything with Ansible playbooks, AWS AMIs / Digital Ocean images, Dockerfiles, or whatever else. You've already been doing this for years and have the experience - this is written for developers who haven't done a lot of Linux before and want to take control of the process.
Dokku is great if you want to host all apps on a single server. I had used it for a while.
Switched to using github to push to, which fires off a build on a ci server, which fires off a push to docker registry and my server pulls that and rolls the versions.
This setup allows me to have everything on a single server until an app starts needing it's own, then I deploy a new CoreOS server and transfer the fire system to that one.
For hassle-free node apps deployment, I use and recommend PM2 : a [free and open source tool](https://github.com/Unitech/pm2)
As its doc states: PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.
It also provides a very nice integration with keymetrics.io, which is the paid service which finance PM2 development (they do provide a free tier).
PS: Apart from using it and loving it, I am not affiliated with this product's team.
Please note that this describes something you would do for a toy project. If you do this for work you probably want to:
* Install NPM in a way that you can upgrade easily. Your package manager was designed to do this, so use it. Never ever put untracked files in the system directories.
* Don't git clone into production. You need to know which version was deployed when and where. At the very least set a tag. Better yet, roll a package from that tag (see above) which you then can sign and store. It's very easy and there are tools to help.
* Schedule when and how you upgrade your operating system, NPM, and your application. To do this you need a way to take an application out of production, which brings us to...
* You generally want a load balancer or some sort of web server between the world and your application. This could be as simple as an Apache or nginx. Don't muck about with port forwards!
* And most importantly, document this down to every command in the internal wiki! Even better, write a Puppet/Salt/Ansible file and put it under version control.
In short: Tools exist for a reason. Use them. Don't hack files manually until you master the tools and know why they exist.
> * Install NPM in a way that you can upgrade easily. Your package manager was designed to do this, so use it. Never ever put untracked files in the system directories.
The articles uses the package manager - specifically, the article directs people to install node using nodesource's RPM and dpkg node packages (npm is included with node), and only mentions tarballs if these don't exist for your OS.
> * Schedule when and how you upgrade your operating system, NPM, and your application. To do this you need a way to take an application out of production, which brings us to...
> You generally want a load balancer or some sort of web server between the world and your application.
Hence covering that and load balancing with HAProxy in the article. This is mentioned in the introduction.
> * And most importantly, document this down to every command in the internal wiki! Even better, write a Puppet/Salt/Ansible file and put it under version control.
Hence mentioning exactly that in the opening few paragraphs.
It's very clear you didn't read the article before commenting.
>> * Install NPM in a way that you can upgrade easily. Your package manager was designed to do this, so use it. Never ever put untracked files in the system directories.
> The articles uses the package manager - specifically, the article directs people to install node using nodesource's RPM and dpkg node packages (npm is included with node), and only mentions tarballs if these don't exist for your OS.
Sure, but it also advises users that it's generally OK to extract tarballs into /usr/local. While this clearly is OK in some circumstances, the target audience for this article clearly aren't in a position to make that decision, or understand the consequences of doing this. The article could have suggested "/opt/node" for example and avoided the issue at the expense of explaining how PATH variables work.
> "The /usr/local hierarchy is for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable amongst a group of hosts, but not found in /usr."
> So you'd prefer, if no packages are available, unpackaged software be installed to /opt rather than /usr/local? OK.
Ah, I did miss the subfolder. This is a very unusual convention for installation into /usr/local. Typically, this is what modern day usage of /opt looks like.
> The FHS forbids distros from installing software to either location.
"A package to be installed in /opt must locate its static files in a separate /opt/<package> directory tree, where <package> is a name that describes the software package."
I see your point - I used to be reticent to use /opt since it was a bit of a Solaris-ism (it only got added to the FHS because the proprietary Unixes kept asking for it).
I'm thinking about changing it to /opt instead (and will credit you if I do it), alas /opt/bin isn't in $PATH on most distros.
They can't just add /opt/bin to $PATH
Read the link he wrote down:
> Programs to be invoked by users must be located in the
> directory /opt/<package>/bin. If the package includes UNIX
> manual pages, they must be located in /opt/<package>/man
> and the same substructure as /usr/share/man must be used.
Static packages should be kept together as /opt/myprogramm
and mostly follow the same structure as /usr than so like:
/opt/myprogramm/bin
/opt/myprogramm/include
/opt/myprogramm/lib
...
Mostly I put my programs there, too.
/opt/app1
/opt/app2
nod that's the thing - you'll break the interpreter for every node app - '#!/usr/bin/env node' - if you don't have node in $PATH. And as you say. /opt apps normally aren't in $PATH.
It won't. Why should I install node in /opt?
Mostly I install server-side or desktop apps there (which most other people do aswell (Chrome, Atlasssian))
They don't need to be in path since they are mostly called by initd or sytemd or .desktop file.
Also they don't screw the system as others do that won't get installed with the package manager.
Don't touch any directory without the package manager, if you still do use /opt.
I think most people would install desktop apps via their package manager. Unpackaged apps are generally the exception rather than the rule. /usr/local is also outside the package manager as previously discussed.
FHS tries to stay relevant, but I'm not convinced it is. However the exact location is not the important point here.
If no repository is available you should not proceed with anything until you have a plan in place how to keep track of updates and security issues. How often do you update? Do you backport security issues or upgrade? There will be backwards incompatible changes upstream which fixes security problems, and you need to have a plan in advance when in panic mode.
And if no package is available you must first know why. Is it not supported? Is there a better way? What does that mean for your deployment? With this knowledge you create a package. Install it and keep a copy for future troubleshooting. Always use packages in Linux, unless you have a very good and specific reason not to.
Under no circumstances should you drop untracked files in your environment, unless it is a one man show.
Yep, this is why the article pushes packages as the preferable install method. Thankfully the unpackaged binaries are officially maintained and also frequently updated.
I'm actually really surprised you showed up again. You made a post loudly proclaiming the lack of packaging, high availability, and Ansible in an article that had all those things, in the opening paragraphs.
I'm trying to think of a polite way to say this, but: putting down someone else's work based on it missing things it actually includes (since you haven't bothered to read the article) is incredibly rude. Perhaps you forgot something in your most recent reply?
Tools exist for a reason. Use them. Don't hack files manually until you master the tools and know why they exist.
I feel it's often the other way around: don't hack around with big powerful tools until you know what the underlying problem is and what steps are actually needed to solve it.
I think the GP post addressed this in their opening line, if you don't know what the underlying problem is, you're dealing with a toy project (Be it for pleasure, prototype, R&D) and his advice isn't aimed at these projects.
- The article uses systemd for process monitoring rather than PM2 since it's built into the OS.
- The article uses HAProxy as a load balancer.
- Ansible playbooks, Dockerfiles, and the AWS API are all mentioned in the opening section of the article. Serverless sounds fine too. Personally I use http://mikemaccana.com/images/work/screenshots/firework-0.pn... which I made using the AWS API. Either way, you will need some logic to capture inside these tools. Rather than write an article on Dockerfiles, Ansible playbooks, or the AWS or DO APIs, the article covers the devops basics one needs to know before using these tools.
'Use ${TOOL}' doesn't help you create or understand the contents of ${TOOL}s Dockerfile.
> 'Use ${TOOL}' doesn't help you create or understand the contents of ${TOOL}s Dockerfile.
Many tools are popular because they provide powerful and convenient abstractions. For many folks, it's all about saving time (and therefore money). There's value in leaving the underlying working internals to specialists (that's why we don't need to understand the linux In-kernel API when we use Ubuntu Server's networking)
Totally agreed, there's certainly a place for tools that don't need DevOps skills, and many places - typically PaaS - that take care of this.
However many developers wish to have more control over their environment than what a PaaS provides, and a lot of the tools they'd use for that - Ansible, Docker, etc - require basic DevOps skills.
As far as I know, using nginx in front helps with serving static files, which is a moot point on a REST API.
Well, that depends on the API; for example, a product inventory API might need to serve many product images, a document management API might have to serve PDFs and such, etc.
If you are deploying to a single server, consider using `pm2 deploy` which makes life much easier for rolling new releases and deploying them on server.
Mainly because I didn't want it to be a Docker/kubernetes tutorial. At some point I'll publish an Ansible playbook you can modify for your environment and call from a Dockerfile.
Can I shamelessly plug[0] and say maybe do all this and stick it in a Debian package if you're running Debian, or like most of the cloud world, Ubuntu.
If the node app uses express js and has multiple processes, then a redis server may be required to handle non-stikcy session. It would be nice if the article describes the redis server setup.
47 comments
[ 3.1 ms ] story [ 104 ms ] threadI'd also recommend including your dependencies in your deployment package anyway to avoid a deployment being held up by npmjs.org downtime.
PS. npm v3 still needs it, in fact it makes it better - `install --save` updates the shrinkwrap file.
Re: checking in modules, I used to do this during 2013 when npm was up and down every few days, but stopped a couple of years ago after npm Inc stabilised everything. It's been solid so far and the smaller repo sizes (and faster deploys) have been worth it.
I think I also read many other blogs suggesting serving static files with nginx in front of nodejs. Are you sure that this does not make sense?
As the article mentions, you'll probably want a load balancer for high availability. Whether you use haproxy or nginx for that is a whole different discussion.
You should aim to have the bulk of your payload be static, and all of the static payload be served by apache or nginx. It's the difference between getting good feedback from a Show HN post or completely missing out and looking like a fool.
Something's massively wrong with your node setup. node won't be as fast, but it should be on the same order of magnitude as nginx: https://github.com/observing/balancerbattle (or any other benchmark)
I get `git push dokku master` deployment for free, for any application, I dont have to worry about conflicting node versions between applications and it took a lot less setup for all of my applications than this process describes for one.
From then on you can - and should - deploy everything with Ansible playbooks, AWS AMIs / Digital Ocean images, Dockerfiles, or whatever else. You've already been doing this for years and have the experience - this is written for developers who haven't done a lot of Linux before and want to take control of the process.
Switched to using github to push to, which fires off a build on a ci server, which fires off a push to docker registry and my server pulls that and rolls the versions.
This setup allows me to have everything on a single server until an app starts needing it's own, then I deploy a new CoreOS server and transfer the fire system to that one.
As its doc states: PM2 is a production process manager for Node.js applications with a built-in load balancer. It allows you to keep applications alive forever, to reload them without downtime and to facilitate common system admin tasks.
It also provides a very nice integration with keymetrics.io, which is the paid service which finance PM2 development (they do provide a free tier).
PS: Apart from using it and loving it, I am not affiliated with this product's team.
[1] http://pm2.keymetrics.io/docs/usage/deployment/
[2] https://keymetrics.io/2014/06/25/ecosystem-json-deploy-and-i...
* Install NPM in a way that you can upgrade easily. Your package manager was designed to do this, so use it. Never ever put untracked files in the system directories.
* Don't git clone into production. You need to know which version was deployed when and where. At the very least set a tag. Better yet, roll a package from that tag (see above) which you then can sign and store. It's very easy and there are tools to help.
* Schedule when and how you upgrade your operating system, NPM, and your application. To do this you need a way to take an application out of production, which brings us to...
* You generally want a load balancer or some sort of web server between the world and your application. This could be as simple as an Apache or nginx. Don't muck about with port forwards!
* And most importantly, document this down to every command in the internal wiki! Even better, write a Puppet/Salt/Ansible file and put it under version control.
In short: Tools exist for a reason. Use them. Don't hack files manually until you master the tools and know why they exist.
The articles uses the package manager - specifically, the article directs people to install node using nodesource's RPM and dpkg node packages (npm is included with node), and only mentions tarballs if these don't exist for your OS.
> * Schedule when and how you upgrade your operating system, NPM, and your application. To do this you need a way to take an application out of production, which brings us to... > You generally want a load balancer or some sort of web server between the world and your application.
Hence covering that and load balancing with HAProxy in the article. This is mentioned in the introduction.
> * And most importantly, document this down to every command in the internal wiki! Even better, write a Puppet/Salt/Ansible file and put it under version control.
Hence mentioning exactly that in the opening few paragraphs.
It's very clear you didn't read the article before commenting.
> The articles uses the package manager - specifically, the article directs people to install node using nodesource's RPM and dpkg node packages (npm is included with node), and only mentions tarballs if these don't exist for your OS.
Sure, but it also advises users that it's generally OK to extract tarballs into /usr/local. While this clearly is OK in some circumstances, the target audience for this article clearly aren't in a position to make that decision, or understand the consequences of doing this. The article could have suggested "/opt/node" for example and avoided the issue at the expense of explaining how PATH variables work.
The FHS forbids distros from installing software to either location.
See http://www.pathname.com/fhs/2.2/fhs-4.9.html:
> "The /usr/local hierarchy is for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable amongst a group of hosts, but not found in /usr."
Ah, I did miss the subfolder. This is a very unusual convention for installation into /usr/local. Typically, this is what modern day usage of /opt looks like.
> The FHS forbids distros from installing software to either location.
> See http://www.pathname.com/fhs/2.2/fhs-4.9.html:
Correct, both options are valid per the FHS. Yet, the description for /opt describes the pattern used in your example.
See http://www.pathname.com/fhs/2.2/fhs-3.12.html:
"A package to be installed in /opt must locate its static files in a separate /opt/<package> directory tree, where <package> is a name that describes the software package."
I'm thinking about changing it to /opt instead (and will credit you if I do it), alas /opt/bin isn't in $PATH on most distros.
> Programs to be invoked by users must be located in the > directory /opt/<package>/bin. If the package includes UNIX > manual pages, they must be located in /opt/<package>/man > and the same substructure as /usr/share/man must be used.
Static packages should be kept together as /opt/myprogramm and mostly follow the same structure as /usr than so like: /opt/myprogramm/bin /opt/myprogramm/include /opt/myprogramm/lib ...
Mostly I put my programs there, too. /opt/app1 /opt/app2
Looks like /usr/local wins.
Also they don't screw the system as others do that won't get installed with the package manager.
Don't touch any directory without the package manager, if you still do use /opt.
If no repository is available you should not proceed with anything until you have a plan in place how to keep track of updates and security issues. How often do you update? Do you backport security issues or upgrade? There will be backwards incompatible changes upstream which fixes security problems, and you need to have a plan in advance when in panic mode.
And if no package is available you must first know why. Is it not supported? Is there a better way? What does that mean for your deployment? With this knowledge you create a package. Install it and keep a copy for future troubleshooting. Always use packages in Linux, unless you have a very good and specific reason not to.
Under no circumstances should you drop untracked files in your environment, unless it is a one man show.
I'm actually really surprised you showed up again. You made a post loudly proclaiming the lack of packaging, high availability, and Ansible in an article that had all those things, in the opening paragraphs.
I'm trying to think of a polite way to say this, but: putting down someone else's work based on it missing things it actually includes (since you haven't bothered to read the article) is incredibly rude. Perhaps you forgot something in your most recent reply?
I feel it's often the other way around: don't hack around with big powerful tools until you know what the underlying problem is and what steps are actually needed to solve it.
Exactly my thoughts... this is for toy projects.
It's 2016 and we have proper tools like PM2 [0], StrongPM [1], or even better, Serverless [2] (this is only for AWS but sooo cool) ...
[0] https://github.com/Unitech/pm2
[1] http://strong-pm.io/
[2] https://github.com/serverless/serverless
- The article uses systemd for process monitoring rather than PM2 since it's built into the OS.
- The article uses HAProxy as a load balancer.
- Ansible playbooks, Dockerfiles, and the AWS API are all mentioned in the opening section of the article. Serverless sounds fine too. Personally I use http://mikemaccana.com/images/work/screenshots/firework-0.pn... which I made using the AWS API. Either way, you will need some logic to capture inside these tools. Rather than write an article on Dockerfiles, Ansible playbooks, or the AWS or DO APIs, the article covers the devops basics one needs to know before using these tools.
'Use ${TOOL}' doesn't help you create or understand the contents of ${TOOL}s Dockerfile.
Many tools are popular because they provide powerful and convenient abstractions. For many folks, it's all about saving time (and therefore money). There's value in leaving the underlying working internals to specialists (that's why we don't need to understand the linux In-kernel API when we use Ubuntu Server's networking)
However many developers wish to have more control over their environment than what a PaaS provides, and a lot of the tools they'd use for that - Ansible, Docker, etc - require basic DevOps skills.
[replying from old account due to rate limit]
As far as I know, using nginx in front helps with serving static files, which is a moot point on a REST API.
Well, that depends on the API; for example, a product inventory API might need to serve many product images, a document management API might have to serve PDFs and such, etc.
[0] https://github.com/ehartsuyker/node-deb