Sorting my mails with imapfilter. I have a yaml file where I write down which mails go into which folder depending on sender or recipient or another header field. Runs on a raspberry pi every ten minutes between 8 and 8.
Well it especially helps with work emails and sorts mails from our Confluence and Jira into respective folders. These are around 20~ mails per day. I look at them once or twice per week, grouped by subject.
Another thing is newsletters. Every two to four weeks I take the time reading them. Invoices are put away (amazon, ebay, electrictity, etc.).
Well I think the biggest win for me is, that mails in my inbox are typically your category 1 and I can react right away. I do not have to spare another thought on category 2 or 3. I can check my mails twice a day and it's enough. Really helps focusing on my other work.
Back when I used GMail I had some of those rules as filters, but the big advantage of imapfilter on the Pi is: I can add rules pretty easy with a git commit and add new Inboxes if needed.
EDIT: As I mentioned my email filter runs only between 8 and 8, which is really nice, because I do not see any new mails in the evening. This really frees my mind. If the house is burning down, don't write me a mail and give me a call.
I take enormous pleasure in automating every part of my research pipelines (comp sci).
As in, I like to get my experiment setup (usually distributed and many different components interacting with each other) to a point where one command resets all components, starts them in screen processes on all of the machines with the appropriate timing and setup commands, runs the experiment(s), moves the results between machines, generates intermediate results and exports publication ready plots to the right folder.
Upside: once it's ready, iterating on the research part of the experiment is great. No need to focus on anything else any more, just the actual research problem, not a single unnecessary click to start something (even 2 clicks become irritating when you do them hundreds of times).
Need another ablation study/explore another parameter/idea? Just change a flag/line/function, kick off once, and have the plots the next day. No fiddling around.
Downside: full orchestration takes very long initially, but a bit into my research career I now have tons of utilities for all of this. It also has made me much better at command line and general setup nonsense.
I've made $64,000 so far this year working online and I'm a full time student. Im using an online business opportunity I heard about and I've made such great money. It's really user friendly and I'm just so happy that I found out about it. Heres what I do,
•••••••••>> http://www.joinmate2.com
A few minutes - killing processes on one machine, restarting a database on another machine/cluster, reloading a schema, importing data (for some experiments), deploying a new service, loading tensorflow models, warming up benchmark clients. I found that the key of good research pipelines is having a really consistent way of passing around configurations between components.
My components arent strictly microservices (a mixture of open source components and handwritten tools) and they interact in all sorts o fprotocols with each other (importing jsons, csv, GRPC, HTTP), but I essentially treat the configuration flags as their API, so there are no implicit configurations that I could forget about. The rest is just naming things well, e.g. descriptive names for experiment result files etc.
Initially I thought everyone is doing that, but from talking to PhDs in other domains I noticed that there is a strong bias towards people working in any kind of complex distributed setting having these pipelines.
My friends who devise ML models and just test them on datasets on their laptop never had a real need to get a pipeline in place because they never felt the pain points of setting up large distributed experiments.
Another nice thing about setups like yours is reproducibility. So long as you've got your setup in git and you've stored the flags/lines/functions, you can instantly redo the same experiment.
I agree and I have been working to do this with some of my pipelines as well. One challenge I have been facing is that my compute environment may be quite different than others'. This is mainly the case with respect to distributed computing that seems to be an essential part of the pipelines: I often wish to experiment with multiple hyperparameter settings which creates a lot of processes to run.
Do you or the parent or others take steps to abstract away the distributed computing steps so that others may run the pipelines in their distributed computing environments? More specifically, I use Condor (http://research.cs.wisc.edu/htcondor/) but other batch systems like PBS are also popular. Ideally my pipeline would support both systems (and many others).
Docker can distribute the software needed to run the job well, which is definitely part of the issue and something I should use more.
However, I also have in mind a pipeline of scripts where one script may be a prerequisite to the other. Condor has some nice abstractions for this by organizing the scripts/jobs as a directed acyclic graph according to their dependencies. I was thinking other batch systems might support this as well. But some of my challenge comes in learning how each batch system would run these DAGs. Each one will have some commands to launch jobs, to wait for a job to finish before running some other job, to check if jobs failed, to rerun failed jobs in the case of hardware failure, etc.
It seems like the DAG representation would contain enough detail for any batch system but there may be some nuances. For example, I tend to think of these jobs as a command to run, the arguments to give that command, and somewhere to put stdout and stderr. But Condor also will report information about the job execution in some other log files. Cases like this illustrate where my DAG representation (or at least the data tracked in nodes) might break down, but I haven't used these other systems like PBS enough to know for sure.
[Luigi](https://github.com/spotify/luigi) out of Spotify sounds exactly like what you're looking for. It allows you to specify dependent tasks, pipe input/output between them, and more.
I ended up writing a simple distributed build system (https://github.com/azag0/caf, undocumented) that abstracts this away. (I use it at a couple of clusters with different batch systems). You define tasks as directories with some input files and a command, which get hashed and define a unique identity. The build system then helps with distributing these tasks to other machines, execution, and collection of processed tasks.
Ultimately though, I rely on some common environment (such as a particular binary existing in PATH) that lives outside the build system and would need to be recreated by whoever who would like to reproduce the calculations. I never looked into abstracting that away with something like Docker.
(I plan to document the build system once it's at least roughly stable and I finish my Phd.)
I actually had to think about this for a few minutes. I suppose having written a number of custom orchestration tools both in local clusters and the cloud, debugging distributed services is still incredibly tedious.
I do this as well (research in computational chemistry.) Incredibly useful once set up, plus forcing oneself to think about a project in abstract terms to automate it can give valuable perspective.
It's great that you do that, thank you. Do you also publish your data, programs and setup to allow others to reproduce/build on your research?
When talking about reproducibiluty in science, it's usually about the availability of original data to verify that the original conclusions were correct, but one level higher there's also computational reproducibility with its own challenges (freezing the original environment of the experiment).
So-called orphan repositories (i.e. content doesn't fit into any other bucket) like Zenodo welcome your articles, datasets and code.
High correlation between the BTC price and others. Lots of chaos in Bitcoin-land (August 1/SegWit drama) means most of the others are now falling, large investors just mass-dump their crypto holdings even though logically the prices shouldn't move in lockstep.
I liked writing an internal command line utility for our Go codebase. It automates common dev commands like deployments (including installing dependencies, migrations, etc), sending test emails (eg to check formatting), and running smoke tests. Pretty minor, but it makes my life a lot easier. I plan on expanding it more for accessing prod and dev APIs.
I've written a script which helps me copy-paste files from their folders in Material Design image library to my android project. This saves me at least 4 copy paste, and then renaming operations.
Automation also gets rid of context-switching, which has a cost of its own. It may only take you half an hour to do -foo-, but to switch into the mental space for it and switch out again are costs that most people don't factor in.
OTOH automating is an acquired skill, so I learned to say "screw it". Unless you feel you're trying to spend a month to save 10 minutes total, go ahead and automate it - that's how you get better at automating more stuff faster.
I love XKCD but that chart is missing a few important dimensions.
- How likely is it that you will make a mistake when performing the task manually, and how easy is it to fix the various mistakes that can be made?
- How likely is it that you'll overlook the task when the time comes? (e.g. SSL cert expiration)
- What are the costs of mistakes or failure to complete the task?
- What skills can you learn while developing the automated solution?
- How can techniques developed for the automated solution be reused for other tasks?
Of course it's fair to ask questions on the other side of the equation, too, like: if the automated solution goes bad, or is run in the wrong environment, what are the risks?
All of my thesis project in immunology was automated, which involved several hours of blood processing repeated several thousand times (with some parallelization) by a team of a dozen robots. There are pics, schematics and vids here: http://www.zachbjornson.com/projects/robotics/.
I also like to say that the final analysis was automated. It was done entirely in Mathemtica notebooks that talk to a data-processing API, and can be re-ran whenever. The notebooks are getting released along with the journal article for the sake of transparency and reprodibility.
Awesome. You should write a paper on your paper writing / research / automation / publication process, to help advance the way scientific publishing is done.
This is awesome. I work in a research lab that does a lot of bench work and imaging under microscopy. I think that automating some of these processes would greatly speed up the research being done!
Basically use a web automation tool to simulate the login process and then grab the data you want, save it to .csv file or something. For saved HTML pages, web automation tools generally can work with them. For saved PDF I'm not sure what is good.
I did the same lol (via scraping), run every midnight to multiple bank accounts, then sync to google spreadsheet. I was writing web automation scripts the hard way for a while, until I made a tool to simplify the process of writing web automation scripts - https://github.com/tebelorg/TagUI (frankly I am surprised why my banks never banned or send me a letter)
In theory they shouldn't, though I believe many financial websites explicitly declare in their terms of use that automated access or web scraping data is not allowed.
Do a careful reading of your banks terms of service and you will likely find a clause either barring you from doing this completely, or voiding fraud protection.
I use Selenium to drive a firefox browser under the assumption that banks don't care enough to detect it: https://github.com/tmerr/bank_wrangler . I am also slightly worried about banks not liking it, but if they do care I guess I will receive a scary cease and desist in the mail and I will cease and desist.
My concern is about what will happen if fraudulent activity happens, such as my account getting hacked somehow. Then they look into my account, discover I've been doing this somehow, tell me I've broken their TOS and I'm on my own.
It can be eye-opening to track and categorize every dollar spent, even if just for a few months. Especially for younger people who are earning serious money for the first time.
I wrote a bot to reserve hotel rooms a year in advance for a national park in the US.
It was so difficult to book. After couple days of failed attempts to reserve my desired dates, and after staying up late into the night one day, I went ahead and wrote a bot to automate the task of checking for availability and then completing the checkout process once available.
How does the bot reserve hotel rooms? Through an API or does your bot interact with a website. How do you validate that your code is still able to reserve an hotel room lets say 3 years after writing it; I can imagine that changing systems will break you code.
A month before July 4th, i tried booking a camp site in Yosemite and got none. I should have known the camp sites gets completely booked 6-8 months ahead. When i called one of the camp sites, the automated machine replied -
Ha. For a top-ten most visited park in the world? Walk-up permits are your friend. No experience at Yosemite but I got some backcountry permits at Grand Canyon, it helps if it's not peak season, I think people line up at 4am for those. Worst case you are posting up in a random field or Walmart parking lot for the night.
Nice! I've never been able to get a date I want for a national park. So much easier when single and or in college. I see other days, or a room here and there...but not enough for my family and for the dates to coincide with everyone's schedule.
I also don't have the foresight to book that advance. I've settled with staying offsite and just driving. Not the same experience but good enough compromise I guess.
I'm the kind of nerd who greatly prefers writing automation code to doing anything remotely repetitive. (I'm afraid to work out the actual timings because I'm pretty sure that I often spend more time coming up with the automation than just doing the task would take).
I've got a script that automatically rips, converts and stitches together audiobooks from the library so that I can play them on my phone. It just beeps periodically to tell me to put the next CD in.
I also had a batch job that downloaded Doonesbury cartoons (including some delay logic so I wasn't hammering the server) and built a linked series of html pages by year and month. I've ported it to a couple of other webcomics so that I can binge read.
I also write a lot of LaTeX macros, doing things like automatically import and format code from a github gist into lecture notes (something like \includegist{C,<path/to/gist>), or autogenerate pretty PDF'd marks summaries for students from my home-rolled marks. database.
Another thing I like is building little toys to demonstrate things for students, like a Mathematica page that calculated the convergence rate and error for the trapezoidal rule (numerical integration) with some pretty diagrams.
I once wrote a bunch of lisp code to help with crypto puzzles (the ones that use a substitution code, and you try to figure out the original text). The code did things like identifying letter, digraph and trigraph frequencies, allowed you to test substitutions, etc.
As developers, we tend to focus on these big integrated projects. But one of the biggest advantages that people who can code have is the ability to quickly get a general purpose computer to assist with individual tasks. I write an awful lot of code that only gets run a handful of times, yet some of those projects were the most pleasure I've ever had writing code.
I spend many months doing repetitive tasks. And than I realize I should automate them, and proceed to spend hours coming up with scripts/tools to automate them.
I believe there is also merit in spending a lot of time repeatedly doing something, before proceeding to automate that workflow. Because only through that, you gain a deep understanding of 'edge/exception' cases which you can directly code into your script later on to manage.
There are some really clever people here, but as a general rule, you can't truly automate something until you can do it manually to the point where you're fully aware of all the snags and exceptions that may occur.
Once you reach that point, it then becomes a matter of trading-off how much time/money/effort it will take to automate the task against what benefit you get in return.
Agreed, but it's important to include one criterion in the trade-off calculation: I'd much rather be writing automation code than doing most automateable tasks (i.e. repetitive, simple decision tree). Even if I don't save any time, or even if it actually costs me a little time, I count it as a win. Especially since I often discover useful tools and techniques (holy smokes! Someone already wrote a parser for this weird thing I'm playing with!) that end up being valuable later in a completely unrelated project. True story: some colleagues wanted to integrate a departmental Moodle server with some bespoke scheduling software we were running. Turned I already had most of what we needed, because a year earlier I'd gotten irritated at hand-loading class lists into Moodle and hacked together a bunch of code to directly translate entities from one database to the other. I'd even generalized it into a bunch of types and tables that I didn't really need because OCD. All that 'hobby' code ended up being really valuable later.
[...] The ski rental problem is the name given to a class of problems in which there is a choice between continuing to pay a repeating cost or paying a one-time cost which eliminates or reduces the repeating cost. [...]
Cool! Maybe you'll enjoy something such as Sikuli. It basically lets you automate the actions to take on your desktop or laptop visually. It's based on OpenCV and has a simple interface to use.
Yes, and I use it quite a lot. My local library system, however, has a huge collection of CD audiobooks, many of which aren't available (at least in Canada) on Overdrive or Hoopla. I run my script in a window when I'm working on other things so it's trivial to just feed the disk monster. The audiobook then just shows ip in my Dripbox, ready to play.
>(I'm afraid to work out the actual timings because I'm pretty sure that I often spend more time coming up with the automation than just doing the task would take).
You can use this calculator to estimate how much time is worth investing in automating a particular task =)
I'd be really interested in that audiobook script. I have some that are ripped but not stitched together. I want to get to the point where all my audiobooks are one file.
I usually do one file per CD. You can usually just cat .mp3 files together, although some apps that expect non-standard metadata can get confused (mp3wrap is a more robust solution). 'cat *.mp3 > disk.mp3' (assuming that ls order is the order you want. Or 'cat first.mp3 second.mp3 last.mp3 > all.mp3') Other formats (like m4a) can be stitched together using utilities like ffmpeg.
Just a PSA for anyone who happens across this: There are some apps for Android that understand and play multi-file audiobooks just fine. I use Smart Audiobook Player and it works great.
Support tickets integrated with service monitoring.
Around 3 years ago, we started to get a lot of customers for our VoIP tunneling solution, mostly from UAE. Most of these were unfriendly customers abusing our support, so I started to implement a CRM to track "support points". I spend a half year to develop this solution (with lots of other functionality such as service monitoring) and when I finished, there was no any demand for the VoIP tunneling solution anymore :)
This is how I wasted half year instead to focus to solutions relevant for our business.
Thanks good, we started to have new customers again since last year and actually my CRM/support point tracking software is very useful now, but I still don't think that it worth’s 6 months’ time investment.
Conclusion: focus on your main business and don’t spend too much time with automation and other helper software (or hire somebody to do it if your business is big enough)
Yes! Everything else in this thread is interesting but without automated backups it's all just temporary.
I would add that it's a good idea to automate backups using more than one method as well. For example, use a popular software package but also copy important stuff to an extra drive. A cheap setup with a Raspberry Pi + USB drive plugged in to your router will do just fine.
I do this. I have a 24/7 on RPI at my parents' house for blocking ads and tracking. I just added a 2TB USB hard drive and set it up to sync everything at night when the line is idle anyway.
I was kinda like a backup freak for my PC, but once I got starting using cloud drives like google drive, it kinda takes care of the problems of hard-disk failures (which are definitely bound to happen every couple/few years).
Find yourself a configuration management server such as Puppet, Chef, CFEngine etc, and learn to automate system deployment and management with it. I use Puppet CE as my main automation tool.
Use Python/Shell for tasks that are not well suited for a configuration management server. Usually, this is when procedural code makes more sense than the declarative style of Puppet manifests. Interactive "wizards" (i.e. add domain users accounts to a samba server, and create home directories for them) and database/file backups are my usual uses for these types of scripts.
Fabric is a useful tool to use with python. It allows you to send SSH commands that you put into functions to groups of servers in bulk.
I also use python for troubleshooting network issues. It has libraries to interact with all manner of network services/protocols, as well as crafting packets and creating raw sockets.
Look into PowerShell if you work in a Windows environment. Everything from Microsoft is hooked into PowerShell in their newer versions.
I evaluated Puppet, Chef, and Ansible and Ansible was by far the easiest for me to learn. My favorite thing I use it for is my personal laptop setup. I can go from clean Ubuntu install to having all the software I use along with system settings all perfect within an hour, without me being involved.
It's like choosing programming language by selecting the one that's easiest
for you to learn. You haven't learned anything beside new punctuation and
keywords.
Ansible is basically a glorified shell script executed semi-interactively
through SSH. It doesn't exhibit a new paradigm in server management by any
stretch.
CFEngine, on the other hand, shows a whole another way to approach managing
systems, and thus it's quite hard to learn (akin to learning Prolog or Coq
after writing in Java). Puppet stems from that, but then slaps on top of that
many weird and possibly unnecessary concepts, without much of an apparent
plan or strategy, so it ended up a mess. Chef is a Ruby framework and some
tooling around it, so no wonder it's more difficult to deploy than shell
scripts through SSH.
> It's like choosing programming language by selecting the one that's easiest for you to learn.
It is. Is there anything wrong with that if the most important feature to you based on your requirements is that it be the quickest to learn?
I needed something that was very quick to learn and I had most of my setup automated in Ansible before I had even finished trying to learn the other options. Ansible won hands down on the "easy to learn" factor. So far it meets all my needs, and thanks to Ansible Galaxy, I've had to do very little config myself. 95% of the time someone has already written what I need.
> thus it's quite hard to learn
Then it's definitely the wrong product based on my requirements at the time.
>> It's like choosing programming language by selecting the one that's easiest for you to learn.
> It is. Is there anything wrong with that if the most important feature to you based on your requirements is that it be the quickest to learn?
On its own it's not a wrong thing, but as I said, you didn't learn anything
substantially different, so you're stuck with the approach that doesn't scale.
You only added shell script SSH executor to your toolbelt, while you still
haven't seen a proper configuration management system.
If I had spent the time to learn a "proper" configuration management system, it would have been a loss. I had a very limited amount of time to make it worth automating what amounts to a few hours of manual work.
> you didn't learn anything substantially different
So what? Learning something substantially different wasn't one of my requirements. My goal was to automate a few simple deployments, including my personal laptop setup, which previously took at least a day. Ansible got the job done and I haven't looked back.
> You only added shell script SSH executor to your toolbelt
You've said this enough that I'll push back. That's definitely wrong. I can write shell scripts. I would have never attempted this in a shell script. Ansible was far easier than writing a shell script, and it does a lot more than just executing scripts via SSH. For one, it checks the current state of the system it is configuring to determine whether or not software needs to be installed.
> you're stuck with the approach that doesn't scale.
That wasn't one of my requirements either. Although I'm not even sure your claim is true considering that there are some pretty big companies using Ansible, and it was recommended to me by someone who is one of the better programmers I have met in my long career.
The whole point of my original post was that I actually tried products instead of reading about pros and cons. Based on my needs, Ansible was far superior to the other products I tried. Because my requirements aren't your requirements. I'm not so sure why it seems that it's so important to convince me and / or others that Ansible isn't a real tool. People should spend some time trying a few of the most popular tools and decide for themselves which tool best meets their needs. Which may include constraints like how much time they can budget towards learning a tool. "Proper" tool or not.
>> You only added shell script SSH executor to your toolbelt
> You've said this enough that I'll push back. That's definitely wrong. [...] For one, [Ansible] checks the current state of the system it is configuring to determine whether or not software needs to be installed.
So you claim that Ansible learns the state of the system through some magic
that is not possible to be called from shell? Because I could do pretty much
the same in a shell script just fine.
I've seen a different paradigm for managing servers, I thought about it for
a long time figuring out the differences from a shell script, and this is not
where they are.
> I'm not even sure your claim [the approach doesn't scale] is true considering that there are some pretty big companies using Ansible
There are many big companies that throw money at dumb processes only to make
them running, so this argument is pretty weak.
> [Ansible] was recommended to me by someone who is one of the better programmers I have met in my long career.
You see, precisely this is the issue here. It was recommended to you by
a programmer. You should have asked a sysadmin, because this is where
these tools come from and whose tasks they do. For some strange reason,
programmers tend to avoid tools available to sysadmins, which is a quite big
blind spot (even more so, since it's usually unrealized blind spot).
> So you claim that Ansible learns the state of the system through some magic that is not possible to be called from shell?
I never claimed that.
> Because I could do pretty much the same in a shell script just fine.
So could I. But I didn't have to. Because Ansible does it for me. So right there, just by comparing my declarative configuration with the current system state, it's clearly adding more value than "shell script executer".
> There are many big companies that throw money at dumb processes only to make them running
You said it can't scale. You're now moving the goalposts to "well, it can scale, but it takes more money".
> You should have asked a sysadmin
Fortunately I didn't ask this sysadmin because you keep confusing your own requirements with mine. I was never looking for a product that is best of class but takes weeks or months to learn. I've already made it dead clear several times that my most important requirement was "easy and quick to learn". Ansible is the best product of those I tried when that's the primary requirement.
> You see, precisely this is the issue here.
The only issue here is that you can't accept that Ansible was the best fit for my requirements.
> For some strange reason, programmers tend to avoid tools available to sysadmins, which is a quite big blind spot
Some programmers have enough common sense to not spend months learning a tool to automate a few hours of work that happens a few times a year. I asked my friend which would be the quickest to learn, he said Ansible, I wasn't hot on it because Chef and Puppet were more popular, so I tried learning Chef and Puppet and when I realized how long that was going to take I tried Ansible. Within a few hours I had everything automated with Ansible. It was hands-down the best solution based on my requirements. So I do agree there is quite a big blind spot here. But I wouldn't say it's on my side, or the friend who's advice turned out to be spot on. ;-)
> So right there, just by comparing my declarative configuration with the current system state, it's clearly adding more value than "shell script executer".
Very little value in this regard. As I said, I could just as easily write
declarative configuration for machine deployment in a shell script.
>> There are many big companies that throw money at dumb processes only to make them running
> You said it can't scale. You're now moving the goalposts to "well, it can scale, but it takes more money".
I'm not moving any goalposts. If a process takes unproportionately more money
or effort for more input, it doesn't scale. Heck, in this case it doesn't
scale even if it takes the amount of resources proportional to the input!
> [...] you keep confusing your own requirements with mine.
I don't. I'm saying that you didn't learn anything significant, and as such,
you would be just as good if you put some thought into how to do exactly the
same with the tools you had (shell scripts). But now you're probably worse
off, because you most probably got misled by Ansible's marketing into
believing it is also a configuration management solution.
While you were over there thinking about it, I had already completed the job. Which was my #1 requirement. It doesn't seem like you'll ever understand that different requirements often result in different tool choices, but that's ok.
> But now you're probably worse off
Probably not. I now have several things automated that I once had to do manually. I may not be the smartest person in tech, but I'm smart enough that if I need a better solution than Ansible at some point in my career then at that point I'll simply learn the better solution. Different requirements = different tool choices.
I started with Puppet and moved to Ansible. Ansible is much easier to debug, because everything happens in sequence and it doesn't freak out and break if you have the same identical command in two modules. Puppet builds this rats-nest dependency graph before it runs, and that's a pain in the arse to debug. Ansible's limitation is 'ssh bandwidth' - something like Puppet seems to me better for more heavyweight environments where you have three figures or more of servercount.
There are many kinds of "difficult" - sometimes the difficulties may be just conceptual, other times it's because the tool is not good for the job you have in mind.
For instance, I once took a serious look at Angular and then decided it's too difficult. I'm sure there is time and place for Enterprise JavaScript, but none of the things I'm doing right now would benefit from it (and frankly, neither do most of the smaller sites I've seen using it).
The real world tends to have a diversity of hosts and an automation system that only partially applies is in many ways worse than none at all.
I ended up with Ansible because it seems to require nothing on the far side but python2 and ssh. Everything else needed a specific version of a specific client software. CFEngine was particularly awful in a diverse setup, fighting SSL to make puppet work was never fun. I haven't found anything Ansible won't talk to, or can't be made to talk to eventually. I used Puppet for a bit less than a decade before switching to Ansible maybe two years ago, and I don't miss fighting its SSL at all. I miss when the puppetmaster would randomly crash and I'd find out when a deployment failed or worse failed partially, or when the puppetmaster process would take 5 minutes to start because, well, I donno why, but it sure was annoying.
One interesting anti-pattern I saw over the years was people using puppet (or ansible or otherwise) as the worlds weirdest version control system. You can git pull a release branch, or you can write 10K custom lines (no kidding) of automation software to replace one line of 'git pull'. Depends if you're paid by hour/LoC or not. The borderline between "this is managed with ansible/puppet/wtf" and "this is managed by living in a git repo" can be annoyingly fuzzy and going to either extreme doesn't turn out well...
The funny thing is that with Docker builds, that anti-pattern is cool again. Why bother telling the remote to pull from Git when you can just push it an entire disk image that knows nothing about source control? Of course, those 10K lines are still there, they're just hidden by Docker abstractions :)
A PBX that only let's you record voicemail greeting by dialing in and listening to the whole greeting before it can be saved. So... recording their greeting would take a good 15 minutes if they mess up and have to start over.
I wrote a simple lua script for freeswitch that dials the line, follows the prompts, and plays the person's greeting to the PBX. Of course, one day, the damn PBX will be replaced by freeswitch.
Designing and developing UIs. I want to develop web UIs like you develop UIs with Visual Studio or Xcode. I cannot believe how much efforts we need to build and modify web experiences.
- Downloading a song of youtube, adding meta data via beets and moving to my music lib
- Adding tasks to my todolist client from every app I use(including my bookmarking service when I bookmark with specific tags)
- Changing terminal colours based on time of the day(lower brightness in the evenings and hence dark colours, too much sunlight in the mornings and hence solarized themes)
- Automatically message people who message me based on
priority(parents immediately/girlfriend a longer buffer).
- Filters on said messages incase a few require my intervention
- Phone alerts on specific emails
- Waiting for a server which you were working with to recover from a 503(happens often in dev environments) and you are tired of checking every 5 seconds: Ping scripts which message my phone while I go play in the rec area.
- Disable my phone charging when it nears 95% (I'm an android dev and hate that my phone is always charging)
- Scraping websites for specific information and making my laptop ping when the scenario succeeds(I dont like continuously refreshing a page)
I dont think several of these count as automation as opposed to just some script work. But I prefer reducing keystrokes as much as possible for things which are fixed.
Relevant to this discussion:Excerpt from the github page
>OK, so, our build engineer has left for another company. The dude was literally living inside the terminal. You know, that type of a guy who loves Vim, creates diagrams in Dot and writes wiki-posts in Markdown... If something - anything - requires more than 90 seconds of his time, he writes a script to automate that.
> - Automatically message people who message me based on priority(parents immediately/girlfriend a longer buffer).
I'm curious what these automatic messages say. Are you talking about something like an answering machine message? "I'm at home but my phone hasn't moved in 20 minutes so I'm probably in the shower"?
I wonder if charging phone charging at 95% actually gives better battery lifetime. It leads to your battery cycling between 95% and whatever state of charge you start charging at again. If you charge up to 100% your phone will stop charging, but use power from the charger bypassing the battery therefore keeping the battery in better shape.
Charging to 80% and keeping it there by just letting the charger supply the operating current would probably be best if phones supported it.
I'm curious as to how you did the "Phone alerts on specific emails". I'm currently setting my raspberry pi up as a motion detector (we were burgled 2 weeks ago)... and while I can do the motion & send out the email, I really want my phone to go nuts when it gets the alert email (mostly I have notifications turned off).
Any hints appreciated! I'm also experimenting with Telegram, which I think will also solve my issue.
> - Downloading a song of youtube, adding meta data via beets and moving to my music lib
I don't want to be that guy, but a music collection made of probably repeatedly lossy encoded Youtube rips sounds as if it doesn't deserve the name. Not to mention the whole piracy angle - I'm not trying to claim moral authority, but music is DRM-free and reasonably priced these days. There are hardly any arguments for piracy left.
This is a great question.
The PC has been around for a long time now.
For the most part, users/developers have been sitting around, twiddling their thumbs and waiting for the tool and app gods to rain their blessings.
This question begs the need to be proactively involved in the process of designing how you use your PC
Same sentiments. In the beginning, the user and developer of the PC is the same person. I just feel that there are so much untapped computing power waiting for people to tell the computers to go do intelligent things.
Right. Early PC apps were developed by developers mostly for the developer community and PCs were used mostly by developers themselves as a cheap replacement for workstations.
When they started proliferating in the end-user space and experienced exponential growth in volumes, innovation took a back seat and commercialisation drove the chariot. Now that everyone's become rich, we need to get serious again about making it a lot more useful. Automation would probably be the first area I would address in that endeavor since the capabilities of the PC are now quite robust.
It definitely is an automation of tasks - it automates away you having to be concerned with multiple platforms, at the small cost of wasting computer power of all of your users.
Most of my side projects have been about automating the little things that end up taking me a lot of time.
At my first job, part of my work (next to junior dev) was to deploy EARs on Websphere. I automated it so that people just had to drop it on a shared folder and I'd just take a look if it failed to install automatically.
I made a browser plugin to compare the price of the European Amazon and a few other websites (it grew to more countries and websites) http://www.shoptimate.com
And now I'm working on a tool that regularly checks if some of my content is getting adblocked because it's something I periodically do by hand http://www.blockedby.com
In the end, automating things can take more time than actually doing it. But if it's used by others and saves them time as well, it's gratifying.
It is a file format for java applications designed to run on J2EE servers like JBOSS, Websphere, Weblogic, etc. It is a zip file with conventions on certain files and folders in the package.
348 comments
[ 2.8 ms ] story [ 313 ms ] thread1. Need action right now. I answer them and archive.
2. Need action later. I open a TODO and archive. If I need them later, I can search for them.
3. Do not need action. These are already archived. I can search through them, if I need anything.
Another thing is newsletters. Every two to four weeks I take the time reading them. Invoices are put away (amazon, ebay, electrictity, etc.).
Well I think the biggest win for me is, that mails in my inbox are typically your category 1 and I can react right away. I do not have to spare another thought on category 2 or 3. I can check my mails twice a day and it's enough. Really helps focusing on my other work.
Back when I used GMail I had some of those rules as filters, but the big advantage of imapfilter on the Pi is: I can add rules pretty easy with a git commit and add new Inboxes if needed.
EDIT: As I mentioned my email filter runs only between 8 and 8, which is really nice, because I do not see any new mails in the evening. This really frees my mind. If the house is burning down, don't write me a mail and give me a call.
As in, I like to get my experiment setup (usually distributed and many different components interacting with each other) to a point where one command resets all components, starts them in screen processes on all of the machines with the appropriate timing and setup commands, runs the experiment(s), moves the results between machines, generates intermediate results and exports publication ready plots to the right folder.
Upside: once it's ready, iterating on the research part of the experiment is great. No need to focus on anything else any more, just the actual research problem, not a single unnecessary click to start something (even 2 clicks become irritating when you do them hundreds of times). Need another ablation study/explore another parameter/idea? Just change a flag/line/function, kick off once, and have the plots the next day. No fiddling around.
Downside: full orchestration takes very long initially, but a bit into my research career I now have tons of utilities for all of this. It also has made me much better at command line and general setup nonsense.
My components arent strictly microservices (a mixture of open source components and handwritten tools) and they interact in all sorts o fprotocols with each other (importing jsons, csv, GRPC, HTTP), but I essentially treat the configuration flags as their API, so there are no implicit configurations that I could forget about. The rest is just naming things well, e.g. descriptive names for experiment result files etc.
Initially I thought everyone is doing that, but from talking to PhDs in other domains I noticed that there is a strong bias towards people working in any kind of complex distributed setting having these pipelines.
My friends who devise ML models and just test them on datasets on their laptop never had a real need to get a pipeline in place because they never felt the pain points of setting up large distributed experiments.
Do you or the parent or others take steps to abstract away the distributed computing steps so that others may run the pipelines in their distributed computing environments? More specifically, I use Condor (http://research.cs.wisc.edu/htcondor/) but other batch systems like PBS are also popular. Ideally my pipeline would support both systems (and many others).
However, I also have in mind a pipeline of scripts where one script may be a prerequisite to the other. Condor has some nice abstractions for this by organizing the scripts/jobs as a directed acyclic graph according to their dependencies. I was thinking other batch systems might support this as well. But some of my challenge comes in learning how each batch system would run these DAGs. Each one will have some commands to launch jobs, to wait for a job to finish before running some other job, to check if jobs failed, to rerun failed jobs in the case of hardware failure, etc.
It seems like the DAG representation would contain enough detail for any batch system but there may be some nuances. For example, I tend to think of these jobs as a command to run, the arguments to give that command, and somewhere to put stdout and stderr. But Condor also will report information about the job execution in some other log files. Cases like this illustrate where my DAG representation (or at least the data tracked in nodes) might break down, but I haven't used these other systems like PBS enough to know for sure.
[1]: http://www.pachyderm.io
Ultimately though, I rely on some common environment (such as a particular binary existing in PATH) that lives outside the build system and would need to be recreated by whoever who would like to reproduce the calculations. I never looked into abstracting that away with something like Docker.
(I plan to document the build system once it's at least roughly stable and I finish my Phd.)
Classic HN followup (at least I hope): What's currently getting in your way or annoying? What problem would you currently like to just disappear?
When talking about reproducibiluty in science, it's usually about the availability of original data to verify that the original conclusions were correct, but one level higher there's also computational reproducibility with its own challenges (freezing the original environment of the experiment).
So-called orphan repositories (i.e. content doesn't fit into any other bucket) like Zenodo welcome your articles, datasets and code.
--
This post has been automatically generated and may not reflect the opinion of the poster.
I just ... don't get it. I fear I may be a bear of small brain.
As someone else mentioned, there are other valuable considerations the chart ignores, such as likelihood and impact of mistakes.
- How likely is it that you will make a mistake when performing the task manually, and how easy is it to fix the various mistakes that can be made?
- How likely is it that you'll overlook the task when the time comes? (e.g. SSL cert expiration)
- What are the costs of mistakes or failure to complete the task?
- What skills can you learn while developing the automated solution?
- How can techniques developed for the automated solution be reused for other tasks?
Of course it's fair to ask questions on the other side of the equation, too, like: if the automated solution goes bad, or is run in the wrong environment, what are the risks?
I also like to say that the final analysis was automated. It was done entirely in Mathemtica notebooks that talk to a data-processing API, and can be re-ran whenever. The notebooks are getting released along with the journal article for the sake of transparency and reprodibility.
(Also, I automated my SSL cert renewal ;))
I wrote a bot to reserve hotel rooms a year in advance for a national park in the US.
It was so difficult to book. After couple days of failed attempts to reserve my desired dates, and after staying up late into the night one day, I went ahead and wrote a bot to automate the task of checking for availability and then completing the checkout process once available.
And... it worked.
Not trying to be a smart ass, just interested :)
press 1 to book for this year
press 2 to book for next year.
Ha. For a top-ten most visited park in the world? Walk-up permits are your friend. No experience at Yosemite but I got some backcountry permits at Grand Canyon, it helps if it's not peak season, I think people line up at 4am for those. Worst case you are posting up in a random field or Walmart parking lot for the night.
I also don't have the foresight to book that advance. I've settled with staying offsite and just driving. Not the same experience but good enough compromise I guess.
I've got a script that automatically rips, converts and stitches together audiobooks from the library so that I can play them on my phone. It just beeps periodically to tell me to put the next CD in.
I also had a batch job that downloaded Doonesbury cartoons (including some delay logic so I wasn't hammering the server) and built a linked series of html pages by year and month. I've ported it to a couple of other webcomics so that I can binge read.
I also write a lot of LaTeX macros, doing things like automatically import and format code from a github gist into lecture notes (something like \includegist{C,<path/to/gist>), or autogenerate pretty PDF'd marks summaries for students from my home-rolled marks. database.
Another thing I like is building little toys to demonstrate things for students, like a Mathematica page that calculated the convergence rate and error for the trapezoidal rule (numerical integration) with some pretty diagrams.
I once wrote a bunch of lisp code to help with crypto puzzles (the ones that use a substitution code, and you try to figure out the original text). The code did things like identifying letter, digraph and trigraph frequencies, allowed you to test substitutions, etc.
As developers, we tend to focus on these big integrated projects. But one of the biggest advantages that people who can code have is the ability to quickly get a general purpose computer to assist with individual tasks. I write an awful lot of code that only gets run a handful of times, yet some of those projects were the most pleasure I've ever had writing code.
Maybe this can save you some trouble:
https://www.comic-rocket.com/help/archive-binge/
I spend many months doing repetitive tasks. And than I realize I should automate them, and proceed to spend hours coming up with scripts/tools to automate them.
Happens way too often...
Once you reach that point, it then becomes a matter of trading-off how much time/money/effort it will take to automate the task against what benefit you get in return.
https://en.wikipedia.org/wiki/Ski_rental_problem
[...] The ski rental problem is the name given to a class of problems in which there is a choice between continuing to pay a repeating cost or paying a one-time cost which eliminates or reduces the repeating cost. [...]
You can use this calculator to estimate how much time is worth investing in automating a particular task =)
https://c.albert-thompson.com/xkcd/
Around 3 years ago, we started to get a lot of customers for our VoIP tunneling solution, mostly from UAE. Most of these were unfriendly customers abusing our support, so I started to implement a CRM to track "support points". I spend a half year to develop this solution (with lots of other functionality such as service monitoring) and when I finished, there was no any demand for the VoIP tunneling solution anymore :)
This is how I wasted half year instead to focus to solutions relevant for our business.
Thanks good, we started to have new customers again since last year and actually my CRM/support point tracking software is very useful now, but I still don't think that it worth’s 6 months’ time investment.
Conclusion: focus on your main business and don’t spend too much time with automation and other helper software (or hire somebody to do it if your business is big enough)
I would add that it's a good idea to automate backups using more than one method as well. For example, use a popular software package but also copy important stuff to an extra drive. A cheap setup with a Raspberry Pi + USB drive plugged in to your router will do just fine.
Now automating Radom file restores and checksumming is a great idea...
Use Python/Shell for tasks that are not well suited for a configuration management server. Usually, this is when procedural code makes more sense than the declarative style of Puppet manifests. Interactive "wizards" (i.e. add domain users accounts to a samba server, and create home directories for them) and database/file backups are my usual uses for these types of scripts.
Fabric is a useful tool to use with python. It allows you to send SSH commands that you put into functions to groups of servers in bulk.
I also use python for troubleshooting network issues. It has libraries to interact with all manner of network services/protocols, as well as crafting packets and creating raw sockets.
Look into PowerShell if you work in a Windows environment. Everything from Microsoft is hooked into PowerShell in their newer versions.
Ansible is basically a glorified shell script executed semi-interactively through SSH. It doesn't exhibit a new paradigm in server management by any stretch.
CFEngine, on the other hand, shows a whole another way to approach managing systems, and thus it's quite hard to learn (akin to learning Prolog or Coq after writing in Java). Puppet stems from that, but then slaps on top of that many weird and possibly unnecessary concepts, without much of an apparent plan or strategy, so it ended up a mess. Chef is a Ruby framework and some tooling around it, so no wonder it's more difficult to deploy than shell scripts through SSH.
It is. Is there anything wrong with that if the most important feature to you based on your requirements is that it be the quickest to learn?
I needed something that was very quick to learn and I had most of my setup automated in Ansible before I had even finished trying to learn the other options. Ansible won hands down on the "easy to learn" factor. So far it meets all my needs, and thanks to Ansible Galaxy, I've had to do very little config myself. 95% of the time someone has already written what I need.
> thus it's quite hard to learn
Then it's definitely the wrong product based on my requirements at the time.
> It is. Is there anything wrong with that if the most important feature to you based on your requirements is that it be the quickest to learn?
On its own it's not a wrong thing, but as I said, you didn't learn anything substantially different, so you're stuck with the approach that doesn't scale. You only added shell script SSH executor to your toolbelt, while you still haven't seen a proper configuration management system.
> you didn't learn anything substantially different
So what? Learning something substantially different wasn't one of my requirements. My goal was to automate a few simple deployments, including my personal laptop setup, which previously took at least a day. Ansible got the job done and I haven't looked back.
> You only added shell script SSH executor to your toolbelt
You've said this enough that I'll push back. That's definitely wrong. I can write shell scripts. I would have never attempted this in a shell script. Ansible was far easier than writing a shell script, and it does a lot more than just executing scripts via SSH. For one, it checks the current state of the system it is configuring to determine whether or not software needs to be installed.
> you're stuck with the approach that doesn't scale.
That wasn't one of my requirements either. Although I'm not even sure your claim is true considering that there are some pretty big companies using Ansible, and it was recommended to me by someone who is one of the better programmers I have met in my long career.
The whole point of my original post was that I actually tried products instead of reading about pros and cons. Based on my needs, Ansible was far superior to the other products I tried. Because my requirements aren't your requirements. I'm not so sure why it seems that it's so important to convince me and / or others that Ansible isn't a real tool. People should spend some time trying a few of the most popular tools and decide for themselves which tool best meets their needs. Which may include constraints like how much time they can budget towards learning a tool. "Proper" tool or not.
> You've said this enough that I'll push back. That's definitely wrong. [...] For one, [Ansible] checks the current state of the system it is configuring to determine whether or not software needs to be installed.
So you claim that Ansible learns the state of the system through some magic that is not possible to be called from shell? Because I could do pretty much the same in a shell script just fine.
I've seen a different paradigm for managing servers, I thought about it for a long time figuring out the differences from a shell script, and this is not where they are.
> I'm not even sure your claim [the approach doesn't scale] is true considering that there are some pretty big companies using Ansible
There are many big companies that throw money at dumb processes only to make them running, so this argument is pretty weak.
> [Ansible] was recommended to me by someone who is one of the better programmers I have met in my long career.
You see, precisely this is the issue here. It was recommended to you by a programmer. You should have asked a sysadmin, because this is where these tools come from and whose tasks they do. For some strange reason, programmers tend to avoid tools available to sysadmins, which is a quite big blind spot (even more so, since it's usually unrealized blind spot).
I never claimed that.
> Because I could do pretty much the same in a shell script just fine.
So could I. But I didn't have to. Because Ansible does it for me. So right there, just by comparing my declarative configuration with the current system state, it's clearly adding more value than "shell script executer".
> There are many big companies that throw money at dumb processes only to make them running
You said it can't scale. You're now moving the goalposts to "well, it can scale, but it takes more money".
> You should have asked a sysadmin
Fortunately I didn't ask this sysadmin because you keep confusing your own requirements with mine. I was never looking for a product that is best of class but takes weeks or months to learn. I've already made it dead clear several times that my most important requirement was "easy and quick to learn". Ansible is the best product of those I tried when that's the primary requirement.
> You see, precisely this is the issue here.
The only issue here is that you can't accept that Ansible was the best fit for my requirements.
> For some strange reason, programmers tend to avoid tools available to sysadmins, which is a quite big blind spot
Some programmers have enough common sense to not spend months learning a tool to automate a few hours of work that happens a few times a year. I asked my friend which would be the quickest to learn, he said Ansible, I wasn't hot on it because Chef and Puppet were more popular, so I tried learning Chef and Puppet and when I realized how long that was going to take I tried Ansible. Within a few hours I had everything automated with Ansible. It was hands-down the best solution based on my requirements. So I do agree there is quite a big blind spot here. But I wouldn't say it's on my side, or the friend who's advice turned out to be spot on. ;-)
Very little value in this regard. As I said, I could just as easily write declarative configuration for machine deployment in a shell script.
>> There are many big companies that throw money at dumb processes only to make them running
> You said it can't scale. You're now moving the goalposts to "well, it can scale, but it takes more money".
I'm not moving any goalposts. If a process takes unproportionately more money or effort for more input, it doesn't scale. Heck, in this case it doesn't scale even if it takes the amount of resources proportional to the input!
> [...] you keep confusing your own requirements with mine.
I don't. I'm saying that you didn't learn anything significant, and as such, you would be just as good if you put some thought into how to do exactly the same with the tools you had (shell scripts). But now you're probably worse off, because you most probably got misled by Ansible's marketing into believing it is also a configuration management solution.
While you were over there thinking about it, I had already completed the job. Which was my #1 requirement. It doesn't seem like you'll ever understand that different requirements often result in different tool choices, but that's ok.
> But now you're probably worse off
Probably not. I now have several things automated that I once had to do manually. I may not be the smartest person in tech, but I'm smart enough that if I need a better solution than Ansible at some point in my career then at that point I'll simply learn the better solution. Different requirements = different tool choices.
No true Scotsman...
For instance, I once took a serious look at Angular and then decided it's too difficult. I'm sure there is time and place for Enterprise JavaScript, but none of the things I'm doing right now would benefit from it (and frankly, neither do most of the smaller sites I've seen using it).
https://learnxinyminutes.com/docs/shutit/
use it all the time, eg:
https://zwischenzugs.wordpress.com/2017/03/04/a-complete-ope...
I ended up with Ansible because it seems to require nothing on the far side but python2 and ssh. Everything else needed a specific version of a specific client software. CFEngine was particularly awful in a diverse setup, fighting SSL to make puppet work was never fun. I haven't found anything Ansible won't talk to, or can't be made to talk to eventually. I used Puppet for a bit less than a decade before switching to Ansible maybe two years ago, and I don't miss fighting its SSL at all. I miss when the puppetmaster would randomly crash and I'd find out when a deployment failed or worse failed partially, or when the puppetmaster process would take 5 minutes to start because, well, I donno why, but it sure was annoying.
One interesting anti-pattern I saw over the years was people using puppet (or ansible or otherwise) as the worlds weirdest version control system. You can git pull a release branch, or you can write 10K custom lines (no kidding) of automation software to replace one line of 'git pull'. Depends if you're paid by hour/LoC or not. The borderline between "this is managed with ansible/puppet/wtf" and "this is managed by living in a git repo" can be annoyingly fuzzy and going to either extreme doesn't turn out well...
I wrote a simple lua script for freeswitch that dials the line, follows the prompts, and plays the person's greeting to the PBX. Of course, one day, the damn PBX will be replaced by freeswitch.
- Adding tasks to my todolist client from every app I use(including my bookmarking service when I bookmark with specific tags)
- Changing terminal colours based on time of the day(lower brightness in the evenings and hence dark colours, too much sunlight in the mornings and hence solarized themes)
- Automatically message people who message me based on priority(parents immediately/girlfriend a longer buffer).
- Filters on said messages incase a few require my intervention
- Phone alerts on specific emails
- Waiting for a server which you were working with to recover from a 503(happens often in dev environments) and you are tired of checking every 5 seconds: Ping scripts which message my phone while I go play in the rec area.
- Disable my phone charging when it nears 95% (I'm an android dev and hate that my phone is always charging)
- Scraping websites for specific information and making my laptop ping when the scenario succeeds(I dont like continuously refreshing a page)
I dont think several of these count as automation as opposed to just some script work. But I prefer reducing keystrokes as much as possible for things which are fixed.
Relevant to this discussion:Excerpt from the github page
>OK, so, our build engineer has left for another company. The dude was literally living inside the terminal. You know, that type of a guy who loves Vim, creates diagrams in Dot and writes wiki-posts in Markdown... If something - anything - requires more than 90 seconds of his time, he writes a script to automate that.
https://github.com/NARKOZ/hacker-scripts
I'm curious what these automatic messages say. Are you talking about something like an answering machine message? "I'm at home but my phone hasn't moved in 20 minutes so I'm probably in the shower"?
Charging to 80% and keeping it there by just letting the charger supply the operating current would probably be best if phones supported it.
Any hints appreciated! I'm also experimenting with Telegram, which I think will also solve my issue.
I don't want to be that guy, but a music collection made of probably repeatedly lossy encoded Youtube rips sounds as if it doesn't deserve the name. Not to mention the whole piracy angle - I'm not trying to claim moral authority, but music is DRM-free and reasonably priced these days. There are hardly any arguments for piracy left.
Definitely sounds like you want to be that guy.
Simple web interface where I have a list of packages I've ordered with the last status update from post service web tracking for.
I'm glad my national post office provides that data at all via their website.
At my first job, part of my work (next to junior dev) was to deploy EARs on Websphere. I automated it so that people just had to drop it on a shared folder and I'd just take a look if it failed to install automatically.
I wrote a command-line tool to search and download subtitles https://github.com/patrickdessalle/periscope
I made a browser plugin to compare the price of the European Amazon and a few other websites (it grew to more countries and websites) http://www.shoptimate.com
And now I'm working on a tool that regularly checks if some of my content is getting adblocked because it's something I periodically do by hand http://www.blockedby.com
In the end, automating things can take more time than actually doing it. But if it's used by others and saves them time as well, it's gratifying.
https://en.wikipedia.org/wiki/EAR_(file_format)