Ask HN: What automation tools have you used to replace mundane activities?
For my travel site side hustle (beatthatflight.com.au/), I find and publish cheap flights for Australians. Sounds simple, but even if say, someone gave me the flights each day, there's posting the, sometimes to deals sites, as well as then editing it and posting it on my site, and out to social media, and then to specific mailing lists on mailchimp, depending on the source city of the deal, or type of post.
A LOT of that becomes tedious. Mailchimp can auto post to FB/twitter/instagram etc, but even converting the blog post into mailchimp emails gets tiring, as you need to choose a feature photo, the specific mailing list etc.
I'm a selenium guy, so have built some scripts to minimise some of it, but the rush each time of finding a deal still pains me when I think of the tedium ahead of publishing it.
Ideally, my goal would be to submit [to][from][when] to a script and the whole process would be automatic. I can see that selenium could do it, but my goodness, it'd be slow and potentially flaky all via the web UI for those sites.
I'd love suggestions, examples of automation you've done, and tools used?
233 comments
[ 2.7 ms ] story [ 286 ms ] threadDisclaimer, it’s a project of mine, but if the websites you’re watching are compatible that will save you some hassle trying to extract data.
Use a lot of google sheets as well, it’s super helpful to make sense and use of the data you scrape.
You can check it out here [0], and feel free to ask if you need something.
[0]: https://monitoro.xyz
[1] https://en.wikipedia.org/wiki/Why_We_Sleep
[0] - https://community.home-assistant.io/t/circadian-lighting-cus...
https://lbnc.ca/luz-lightbulb-meet-sunlight
I used to have a little script running on my computer that changed one bulb’s color based on local weather forecasts and the time, so that I never had a screamingly bright light telling me it was about to rain in the middle of the night. That’s when it starts to get vaguely complicated.
I also have all the outside lights on wifi light switches that turn on automatically at dusk. There are a few lamps inside that turn on automatically too, so it looks like someone is home and so it automatically creates a nice ambiance when we are home.
The lamps also happen to use RGB Hue bulbs, so they're normally a white shade, but if one of the outside motion sensors (also Hue) is triggered, then they turn red, and if it's late enough that the outdoor lights are off, then it automatically triggers the wifi light switches to turn on all the outside lights.
Amazon echo show is great for checking cameras. Logitech harmony remote is great for starting up the tv/media box/amplifier and configuring them to their correct settings each time someone wants to watch tv. The automatic refill kit for the keurig keeps the water tank full. myQ lets me know if one of my garage doors was left open. A wemo plug turns on my hot water circulator during the hours when we're usually home and off when we're not. Nest camera lets me know when it spots a person on my driveway.
For Windows automation AutoIt is good. VBScript for MSOffice automation. Python/shell scripts for mostly everything.
Disclaimer: I work with the Playwright team at Microsoft.
Are you currently using Kantu? If so, I’d love to hear more about your specific use cases, and which capabilities you’d be interested to see. Thanks!
Out of curiosity: are there any specific extensions from puppeteer-extra that you’ve used and/or would be interested to see?
Cheers!
https://github.com/microsoft/WinAppDriver
As we build out Playwright, we hope to learn a lot about how to make automation more reliable, ergonomic, and enjoyable for developers. Ideally, we could then apply those principles/learnings to desktop, mobile, etc. automation too. Stay tuned!
Edit: Missed the Microsoft ref. Not so keen.
The team is working very hard to be open/transparent/available (https://github.com/microsoft/playwright), and we’re very excited to build a better automation stack with the help of the community. I only ask since I’d love to hear any feedback for how we could improve the way we run/position the project. Thanks!
> setTimeout-free automation
How ? you still need an upper limit for operations
To make the act of selecting elements more resilient to change, Playwright also supports a collection of “selector engines” that allow you to choose the best strategy for selecting elements (via CSS selector, xpath, text content). Additionally, you can author entirely custom selector engines, and compose them together with the built-in types. This is still somewhat experimental, but we think this can enable another level of robustness to test automation: https://github.com/microsoft/playwright/blob/master/docs/sel....
Finally, in addition to waiting for element visibility, Playwright allows you to wait for specific page events to occur (e.g. network request work made). This enables your code to leverage deterministic “signals”, not timeouts (which are the source of a lot of flakiness!). Overall, we want it to be possible to author automation that is entirely event-driven, and includes zero arbitrary timeouts.
https://github.com/microsoft/playwright
I’d love to hear a bit more about the scenario you have in mind, since that might help inform some improvements we can make. Thanks!
PS: I remember interacting with you from the early days of code-push I think :)
I’m glad to hear the examples worked out of the box! We’ve been iterating on the docs and API quite a bit (based on feedback), so don’t hesitate to let us know if/how they can be improved: https://github.com/microsoft/playwright.
Additionally you can bundle the interpreter for your scripts into a standalone .exe that can run on anybody's windows machine without needing ahk installed. It's a one-click operation done in a program that was also written in autohotkey (https://github.com/AutoHotkey/Ahk2Exe). I can't imagine anything as resilient as AHK when it comes to scripting automation programs on windows.
Others tell me if my server is getting swamped, forward important emails like renewing a cert, a deploy failed, bug in prod or something else that requires often immediate attention. Chat widgets from sites in production also go directly to Slack so I can reply quickly if needed. (Gmail, Sentry, Freshchat, Ploi etc, integrate them all and filter out what's critical).
Everything generally operates on a 15 min interval or so it's not noisy and I've pretty much tuned it to only receiving specific stuff I want to see. Having a central assistant like hub to go to is very useful and time saving.
So about every service I have connected to it is on a free tier except Ploi (free as well) but I need for actual work and is also too good not to pay for.
* A change in GBP -> NOK exchange rate, to indicate if it's a good time to buy ready for my next trip to Norway.
* Alerts on flight prices to help me purchase flights when they're at their cheapest.
* Daily digest email with content I'm interested in from HN and some web comics.
* A summary of new content coming into my RSS reader.
It works really well for this sort of stuff and I found it fairly easy to get up and running.
I have had numerous simpler projects automating my digital life but I made the (probable) mistake of always starting from scratch and writing stuff all on my own.
Since I set it up a few months ago I've been using it a bunch, it's event driven system is great for when you want to trigger off multiple streams half way through a flow. You can also go back through past events if you want which is really powerful.
I've semiautomated setting up a new laptop using a private repo with my dot files with some bash scripts. Started with linux, now works with Mac os x (but won't handle linux well and I haven't gone back enough to make it worth case by case stuff).
I still have to generate a public private key pair, and some things on macs aren't perfect, but I have a todo list for the non automated things in the repo. I get going less on new macs nowdays since the keyboards are such crap (fingers crossed this year will bring better 13 inch keyboards), but everytime I do it I add something.
Generally, I've had better luck using undocumented APIs for this kind of stuff. I was a heavy user of Selenium to automate many a task (I work in wholesale construction goods, tons of automations needed everywhere).
But then discovered that using internal APIs (which surprisingly don't change much) is far easier than trying to exception manage changing UIs.
Open up dev tools on your browser and watch the GETS/POSTS as you complete your daily tedium. I use Python (usually just requests and beautifulsoup is enough) to mimic the calls and I have yet to find a use case where I wasn't able to automate something online.
I'm currently developing an SDK for our 18th-century ERP system, which has no API. I've also automated getting shipping container tracking data from a various shipping lines and railroad company websites, many with complicated login processes.
Happy to chat, email in profile.
Man-in-the-middle Android apps to discover (sometimes different API backend).
I did the same with a shipping line which turned out to use a SOAP backend.
Were you using an android emulator on desktop or there's some way to easily do this on android?
I read your post.
Paused.
Opened up debugger on the site I'm currently selenium-ising.
reloaded.
boom. there it is. The request, and beautiful json response.
So...so..so much simpler.
Thank you, my evening is sorted ;)
I had the same experience a couple years ago after spending months selenium-ing our ERP software.
/starts looking at other APIs to use!
Thanks!
I’ve been working on a “mail to cloud storage” project on the side for fun, but also as an excuse to learn Rust.
As part of this effort, I decided to try out Ansible. Man, it is one great tool!
For development, I have all of my server “groups” pointing to the same host, so my Ansible playbooks install and setup everything on a single machine. Makes testing much easier tbh.
My next project will be creating Ansible Playbooks for my workstation, so I can easily re-setup my Arch workstations on Desktop, Laptop, private and at work.
One of our primary CAD suites was Rhino which is very nice and has a great python API. So I wrote a full-fledged 2.5D CAM processor for that. This allowed us to batch process hundreds of these parts with a single click.
I use it mainly for when I can't do things in IFTTT or have sensitive things I'd prefer to keep in a self-hosted system.
0. https://github.com/huginn/huginn
Next I use Tableau Public to read data from Google Sheets and publish a dashboard to Tableau Public Online. Tableau Public automatically refreshes data on the server side from Google Sheets every few hours.
The good part is that all of this can run automatically without any manual intervention whatsoever and I get a beautiful near real-time dashboard for my daily expenses that I can access from anywhere (dashboard is publicly accessible but hidden).
The strange thing is how many odd lxc container projects I’ve stood up just to play with. Proxmox is sweet.
To capture errors on my server I use PaperTrail's email alerts.
There's a bunch of stuff in there that saves you seconds each time you have to work with a user story or cut a branch or merge etc... Saved me a bunch of time over the years!
https://docs.pipedream.com/pricing/
Paid tiers are coming soon, and if you hit our daily compute limit before that, we're happy to raise your limit to make sure your workflows run without issue.
Let me know if you have any other questions! Feel free to reach out to me directly at dylan [at] pipedream [dot] com.
- FileJuggler on Windows, to automatically clean Desktop, Download folder and archive everything into a date-based folder structure in Drive.
- AutoHotKey to insert text templates into everything (e.g. current date, formatted Jira issues, annoying things to type)
The problem with automation is that one will want to break even quickly [time saved > (time spent finding/setting/maintaining)] otherwise automation is just a glorified delayed action. Time saving is not the only reason for automating but I assume it's n°1's when it comes to personal use.
This thread made me think about automating my invoicing. If all it did was mail me the completed PDF to then forward to clients, that would definitely make me do it more consistently on the 1st of the month even though it's not completely automated.
https://hackaday.com/2015/02/28/adam-savages-first-order-of-...
- Launching commonly used apps, like Notepad, Windows Explorer or Sublime Text
- Launching commonly used websites in a browser, e.g. gmail
- Paste from the clipboard into a language translation site (languageX to languageY via URL parameters)
- Open a to do list in Excel (hard coded file path)
- Text expansion for personal and work email addresses
- With one keypress, open the train schedule on the website, type in the 'from' and 'to' stations, search
- Text expand to symbols that I always forget, eg. euro symbol, hash sign etc
- Google/Wikipedia search clipboard text
Yes! That's a big time saver on my end to.
I was also using it to insert emojis, but the new `Windows+.` shortcut is actually useful for that.
https://lintalist.github.io/
which is a AHK based tool, very easy also for the "casual" user (like I am).
Another very useful tool that I recommend for automating PixelSearch and the like is Pulover's Macro Creator:
https://www.macrocreator.com/
It spits out AHK code after recording actions (e.g. a long mouse and keyboard sequence), it's a great time saver for more complex scripts.
1 - Notify me when Movie tickets are available to book. Used google scripts that run every minute and hit a URL to check if tickets are available now.
2 - Book a class when it's available, I have joined a Gym where you have to book a class before going, and the Zumba is very famous there the time it opens it fills up in 30seconds and there are limited seats, so if you are late it's gone. I wrote a script that checks and books the slot for me and then adds the same in my calendar. I extended the same using google forms when I want to book another class at some specific time which is currently not available if somebody cancels it will be available. The script keeps on checking until class start time, if available books and notify me.
3 - In my team, everyone plays Foosball and after lunch, it's everyday discussion who will play first and with whom. I wrote a script which will decide the matches and players in each team. I used the google app engine to deploy it, which is still running and just by hitting the API will sort the things for us.
There are a few other automation I did using IFTTT.
I remember reading a wild tale of how someone made a program for a used car salesman that would check a particular car auction site to make sure they get a good deal (I wish I could remember the details, or better yet, find the URL)
https://www.youtube.com/watch?v=sgz5dutPF8M
I wrote a blog post about how to monitor a competitor website using Python / Lambda and serverless framework for deployment: https://dzone.com/articles/monitor-your-competitors-with-aws...
For lots of workflows we also use No-code/Low-code tools like Zapier / Integromat.
The dashboard interface in TM is a lot nicer than GM too
Tampermonkey used to be open source, but unfortunately isn't any more. I still use it though, as it has better UX (e.g. nicer dashboard, prefills @match when creating new scripts, better editor, etc).
An alternative to Tampermonkey that is open source and still uses the old style of user scripts is Violentmonkey (which also lacks in the UX department, if i recall correctly).