HN Explorer

Ask YC: How do you upload to your web server?

29 points by sangguine ↗ HN
Hi. I was wondering how you upload your code from localhost to the web server. Right now, for my project, I upload the entire directory from localhost to my web hosting server and replace the existing files with the newly uploaded files. It was ok in the beginning because the files were small. Now that I have a bunch of libraries, it takes quiet a long time to upload all 600 files every time I upload.

I recently learned about rsync to upload only changes. I was wondering if you use this or other tools to upload.

47 comments

[ 3.3 ms ] story [ 92.3 ms ] thread
epi0Bauqu
Are you using a repository to store your code? If so, you can check it out from the server and then copy it over.
sangguine
Yes. I am using SVN to collaborate with my friends. How do you copy easily?
Hates_
Commit your changes on your local machine then SSH to your server and do an svn update.
jaf656s
one problem with this method, is that your web directory will have all of the .svn folders sprinkled throughout your subfolders. svn export avoids this issue.
tlrobinson
You can also set your webserver to ignore the .svn folders.
chrj
... but export can waste a lot of time and bandwidth if your project is large ;-)
jaf656s
I'd recommend that you look into SVN export command
hs
i use mercurial, assuming your base directory is /www/pages (lighttpd), initially:

server: $ mkdir /www/pages/project $ cd /www/pages/project $ hg init

localhost: $ cd /path/to/your/project $ hg init $ hg add * $ hg commit -m "message" $ hg push ssh://usr:pwd@123.123.123.123//www/pages/project

afterward it's just the localhost (ignore "$ hg init")

spydez
I agree that you need git or hg or svn or /something/ of the sort for large sites, like sangguine has, but for small/tiny sites, other options are available.

Heck, I use sshfs to mount my blog as a directory & edit files directly. rsync has also worked nicely for me in the past. But then, these are all sites that I either don't care about source control, or don't have git/hg available on the shared server.

witten
Here's the bash script I use with Mercurial to deploy:

hg qpop -a ; hg pull ; hg update $1 ; hg qpush -a

The $1 is a branch or tag argument given to the Bash script. That way you can easily specify a particular version to update to (or rollback to!) on the command line.

The qpop and qpush will rebase any MQ patches you've made, e.g. local configuration changes, hacks, etc.

More on rebasing applied patches with Mercurial: http://hgbook.red-bean.com/hgbookch12.html#x16-28900012.8

3KWA
We use BZR for version control when we have a rep we arehappy to deploy

bzr push to a REP on the server login over ssh on the server bzr export REP run the test suite on the server (last check) cp relevant files to PROD restart the server (FastCGI)

everything automated and sequential one failure -> process stops completely (never happened so far)

maxklein
Use SVN and setup a postcommit thingy that copies the new files to your public directory whenever you do an update.
Bluem00
You'll have a problem if updates involve more than code changes. Any database scripts or configuration changes will be left undone, potentially breaking the site.

You may choose to use a tool to run your update scripts automatically and create configuration files correctly, but you'll want to think it through in advance. A particular problem I have is that certain sensitive passwords should not exist anywhere but in the locations they are required, so even pre-created configuration files still require me to manually enter the password once installed on the server.

subwindow
Git or SVN using Capistrano.

If you're deploying code, you really should be using Capistrano. Even if it is not a Rails project. It is a world of difference.

yawl
I use Capistrano too. It is written in ruby but can be used by any non-ruby project.
[deleted]
(comment deleted)
rufo
I use Capistrano and am quite happy with it, but even if you don't go that way, seriously look into some kind of deployment management software. Versioned deployments, simple and automatic rollbacks if something fails, and the ability to easy scale up deployments if you wind up breaking out into multiple servers for app/db/web.

It turns updating your server from a mild headache to one line that you actually enjoy typing in.

rsa
I use capistrano for PHP, Python and Ruby applications. Its simple and works.
turkishrevenge
I was in the same boat as you not too long ago. As the others have said, Capistrano really is the way to go. Out of the box, it's meant to work with rails projects, but if you tweak it a bit, it'll work fine with really any language/environment. You can try www.capify.org to get started. Beware though, the majority of documentation floating around is for the 1.x release. By v2, a lot of the configuration specific details were changed though and unfortunately if you are starting now, the 2.x docs are really lacking. There is an effort underway to bring them up to speed, but so far I haven't seen anything. Read the docs from the 1.x release, as enough of it still applies to 2.x.
gcv
I took a page from Rails and Capistrano deployment, and like the idea of versioning releases and using symbolic links. In essence, you update the latest code out of your source control system into /myapp/source, then you run a build and installation script which installs the latest copy of your code to /myapp/release-2008-05-22-114500 (timestamp), then makes a symbolic link from /myapp/current to the right version of the code. That way, if you release something broken, you can trivially roll back to a previous version. (It obviously doesn't always work for things like database schema changes.)
a-priori
My webserver runs Puppet and periodically pulls both the code and its own configuration from my Git repository (which I'm currently hosting on Github, but I may someday self-host).

Yes, I'm going against the grain by using Puppet instead of Capistrano. All the Rails people are probably going to crucify me, but I personally think this setup works quite well.

strick
This is one way to do it on Windows.

Use WinKey to map win-P to project.bat.

project.bat: rsync -Crltvz -e ssh /cygdrive/c/personal/project/ me@mydomain.com:/bigfiles/project

then I can just hit win-P to force the rsync. Runs in a couple seconds even with hundreds of files.

But my favorite if I don't need version control is to use <a href="http://www.sftpdrive.com/">sftpdrive</a> to mount the remote server as a local disk. It is $39, but is probably my favorite windows software. I've done the Samba thing and like this much better.

orib
I actually have a commit hook in git that will update a copy of my website when I commit something with the right message. It's kind of crude, but it manages the busywork for me quite nicely.
[deleted]
(comment deleted)
iamnirav
Perhaps a bit noobish, but I use the web development software Coda (panic.com/coda) and it tracks changes for me; when I publish, it only uploads changed files since my last publishing.
breck
After I'm done editing my pages, I save them to a 5.25" floppy and mail it via USPS to GoDaddy in Arizona with the instructions to place on my webserver.
ssanders82
Seriously man, it's 2008. Time to move to 3.5" floppies already.
andresvi
Dude. He was being ironic.
hschenker
Upvoted for the obviously intentional meta-irony.
marrone
This is what I have set up. A Git repository on the server. I push all my code to the repo via ssh.

On the server there are Dev, Staging, and Production environments. Dev and Staging are bare checkouts from the git repo. I have deploy shell scripts that automate deploying to both (checking out from source control and running database migrations). The process should be that code is deployed to Dev, then Staging and finally to Production.

The Production environment is NOT a checkout from source control. It is an exact mirror of the Staging environment. And the deploy script to production does the following:

- tags source control with the new release number

- creates a tar.gz backup of the production directories

- does a database backup dump

- then uses rsync to copy files from the staging web directories to the production directories.

- lastly runs migration scripts on the prod database

bigtoga
Man - I'm low class apparently as I'm apparently the only one who uses Filezilla or just a plain FTP client. I support about 30 domains and this route helps me. I do have some internally-written apps that will upload files to a lot of different domains at once.
tlrobinson
You should at least be using SFTP, since FTP isn't encrypted.
jobeirne
I use sshfs (sudo apt-get install sshfs) and mount my web server to a local directory. From there, you can do a variety of things.
mark-t
For my personal site, I use rsync, and I keep the command in a makefile so I don't have to remember it (I also use the makefile to generate the static pages from templates). It looks like this:

  sync:
  	rsync -az --progress ./ user@host:public_html/

  .PHONY: sync
But for anything significant, I use a VCS, currently svn or git.
attack
You should really use ssh with that. Not much more complex:

rsync -azP local_dir -e ssh user@host.com:remote_dir

jsn
rsync is usually configured to use ssh by default. no need for -e.
attack
Never knew that, but you're right!
dualogy
tar, scp, ssh, (un)tar (whaddya mean it's not a process? it's got all of four steps in it! ^^)
radu_floricica
A separate SVN checkout (with Tortoise), then WinSCP. Surprisingly, you can use WinSCP very well with scripting. Could also make a backup in the same script but didn't yet. The same setup for getting down the database from the production to the development machines.

It's not very professional, but for our needs it's perfect.

wehriam
I use Fabric - http://pypi.python.org/pypi/Fabric/ - with a series of build scripts. I can create and update several different types of EC2 servers (database, app, etc) with a single commands.

The update command does something like this:

* Check out all pertinent application files from the repository.

* Add any production specific files, for example build number or database setups

* Combine, compress, and number all Javascript and CSS files (combined.384.css can be cached forever, on update it becomes combined.385.css)

* Tar and gzip the production files

* Find all EC2 servers that need to be updated

* Upload the gzipped files

* Stop apache

* Remove old files on the server(s)

* Unzip the new files

* Start apache

brooksbp
Been using SCP... it's a great tool for copying files between machines... but it's a shitty tool for prod deployment... capistrano ftw!
jeremiah
Upload? localhost? pfff. I hack right on the production server!
lunatech
I am hugely surprised no one suggested creating packages (rpm, deb etc.) and then deploying it on prod boxes