One of the hardest challenges I have had in my career is convincing our company to move to Continuous Delivery. 90% of the challenges weren't technical, but emotional. Shipping software comes with lots of feelings, fear, politics, etc. I had to personally work with various leaders across the org to help them through these feelings and perceived blockers.
We aren't 100% there yet, but we are shipping numerous times per day across 20 or so services and quality has gone _up_, not down
That is interesting because I am still trying to figure out how people do CD. Of course I know CI, we have it all set up. But we still work in sprints with manual testing and release every 2 weeks.
I could spend time on marking features 'frontend only, low impact' which we could deploy pretty much the same day. Still there are quite some features that need bigger amount of work where they might be 'done' by dev but I am sure they are not actually done, because security, because error checking. It of course is usually that one dev has his not tested feature merged to develop and then also some other dev has production ready one, then if one feature is production ready, I would have to put also some time to make release and pick only changes for accepted change. I am not sure that additional work to check what we can release ' right now' will pay off vs just taking time for fixes on acceptance by people who were working on code and then release it (after 2 weeks o 1 depending how fast it is done in sprint).
So are you having people who work only on picking changes that are low impact, or working on making stuff production ready by picking from develop?
Maybe you pick changes by yourself, or you just defer manual testing to end users and rely on automated tests unit/integration?
p.s. Funny thing with automated tests is they are good at keeping old stuff working but not for testing newly developed features where actual tester can test new GUI/ new features. If you have a lot of GUI changes you cannot automate first tests...
A couple obvious things stand out from your situation. First, only allow merges of code that's been reviewed and has enough tests along with it. Second, have a pipeline that automatically runs all the tests whenever you merge, and if they pass, then goes on to automatically deploy. It's really that simple. It's not easy to get to that point, but it's simple.
Mostly it comes down to organizational changes and everybody getting used to what constitutes "enough" tests.
It was pretty important to work with the Project Managers and "scrum-masters" to decouple our release cycle from our sprint cycle. In our case, like yours, they were coupled together for no technical reason. It's hard to sum up all the changes we made to allow it to happen, but mostly it boiled down to a few technical decisions -
* Every PR is treated as "production" ready. This means if it isn't ready for user eyes, it gets feature flagged or dark shipped. Engineers have to assume their commit will go into prod right away. Feature flags become pretty important
* Product Owners and QA validate code is "done" in lower environments (acceptance or staging, or even locally during PR phase). This helped us decouple code being in prod and "definition of done" in our sprints.
* All API changes and migrations follow "expand and contract" models that makes the code shippable. e.g. even if we are building new features, we can ship our code at anytime cause the public api is only expanding.
* More automated quality checks at PR time. Unit tests, integration tests, danger rules, etc. These vary from codebase to codebase. A key part of this is trusting the owners of that code or service. To a degree, if they are happy with their coverage, then they can ship. (Within limits of course. Not having unit tests at least would be a red flag)
Also, we still ship numerous times a day without a full automation test, we just make sure each release is really small (1-3) commits. The smaller the release the easier it is to manually QA it. So nothing fancy is needed, just smaller releases.
So to answer your question, we don't pick and choose work that is "low" impact or "high" impact, all code gets shipped the same and with the same cadence. It is our responsibility to ensure that when it goes to prod, it won't break anything
I have come to be against continuous delivery - in 'whole product terms' there are vast hidden expenses.
In particular - documentation and support.
Using complicated interfaces is very challenging sometimes, to the point of obscene - Google searches for help turn up a variety of outdated answers - and it's impossible to know what's what.
I'm using Facebook Ads right now quite a lot - it's complex system that just shifts like quicksand under your feet.
Users make incredible efforts to learn the product, only to have it shift away from them like a ghost.
Documentation may or may not be up to date.
Locations of things change.
And who can you ask? Where do you search for support? Facebook has never really provided me answers to many questions in their documentation.
Tiny example - an advisor used to know how to list people that have commented on a post, so that you could 'invite them to like your page'. But it's changed and now he can't find it. One small thing now that he doesn't know how to do, a tool list from his tool-chest.
And who really gains from all of this? Seriously? I don't think anyone.
Is that 'new feature' really that important? It needs to be rolled out 'now' instead of in a major/minor delivery?
These changes are seldom well communicated either.
I suggest the total opposite might be better: release major iterations every year, minor iterations quarterly, and patches as needed.
Every time there is a release - provide users friendly release notes - something they can read at a quick glance and get up to speed. A little 10 second video for every change: "Oh, now you can do XYZ like this"
This way - you have predictability and consistency so users can know how to 'keep up'.
Also - the ability to use the OLD interface where possible for at least 1 year, or something like that - that we users are not forced onto the quicksand.
"Ok it's Jan 1 - FB Ads 2019 is released in 1 month - let's go over the changes - Mary, you can be responsible for highlighting the major changes and communicating them to the rest of the team, and highlighting any risks for us"
Otherwise, you're trolling along, these companies make changes that could feasibly have major impact on your business and you're out to lunch.
Maybe internally continuous delivery could be a useful thing ... but for the world at large the downsides are real and the upsides are limited.
I wrote an inliner once. It might seem like an easy problem, but a proper inliner is a rabid animal you can just barely restrain on a leash.
The simple part is making it work without clashing. The hard part is making inlining always be an optimisation with reasonable code growth with regards to compilation time and cache use.
The hardest problem I've ever solved is learning to sit down and collage. Like, where you cut out bits of magazine and paste them onto paper, etc..
My friends do it pretty often. I find it super relaxing, and know I'll be happy I did it, but the engineer / researcher in me wants to always have fingers on a keyboard.
Building a profiler for .NET. Learning the .NET profiling APIs is a massive learning curve and extremely complex. But, we figured it out! This is for Stackify, BTW.
Hey Leo, You may be interested in Dan Ariely’s short book on Motivation, which is based on his Ted talk here: https://www.ted.com/talks/dan_ariely_what_makes_us_feel_good... Part of it covers the motivational impact of having years of work cancelled. If you’ve struggled with this, I just want to wish you the best.
Just watched, great motivational content, thank you for sharing! Cake example was really weird and still making me think.
I dont struggle as I used to since I've learned it is about environment we are in (meaning as in video) and tomorrow is uncertain (because many people are involved)
Definitely building a design language that had a minimal learning curve, yet the expressiveness to build any website or shop imaginable. The MECE analysis of CSS, the development of a flexible JS framework for users to mix and mash without ever seeing the code. Took a few nights of thinking :)
Trying to get a Glowing Black Stone drop broke me on that game. The final straw was after camping the npc that drops it as a rare drop for days, with like 6-8 hours between spawns (Pyzjn), she finally spawned again and midway through casting a spell to kill her, she spawned because it was morning. :/
That’s the thing. I miss being broken by a game. I miss that struggle. I’ve tried to experience it again but maybe nostalgia is superior. I’m hoping for pantheon to bring that feeling back. I genuinely miss being addicted to the struggle and joy Everquest was. It helped mold me into who I am today and I love it.
I wrote a tool[0] that converts Erlang style Dialyzer messages to enough of Elixir to where the Elixir formatter can pretty print it. This isn't an intractable problem, but the source tool does really bizarre things so the output so it wound up being really annoying to deal with the quirks. Still not quite done, but it's in a workable state to where messages can be dealt with individually when the errors are poor.
Hey, I'm also working on something similar, namely automatic Elixir to Erlang code transformation at the AST level [0].
Despite it being really interesting, some features like scope analysis for variables detection and conversion, the possiblity of multiple modules in an Elixir file and the Elixir pipes which need to be translated to sequential function calls are honestly a pain in the ass.
My company is building a device which collects data from a car while you drive. We wanted to group those in "trips" on a very simple logic. If the device sends any data for the first time, start a trip and if there's none for more than 5 minutes, close the last trip.
Problem was that the source of the time is coming from the device which had a lot of bugs (e.g. time was in future, time is from year 1970, time is not in order...). Management also decided that it's too hard to fix them on the device. At the end it took me 3 full weeks +/- to make it work. At the end I was able to convince the management that some bugs were too serious and required fixing
A guy from the device developers and me talked with them about it. We argued that it's probably far easier and faster to fix the problems in the device code than to make some workarounds on server side.
I think the thing that it's faster and we were 2 convinced them since I tried it with the easier part already before.
On other parts we decided to not process the data.
ICOs and DAOs both present difficult legal problems vis-a-vis the SEC and securities laws.
Where others have failed we believe we have developed a legal framework for both a non security/unregistered ICO and legal DOA.
To legitimize our legal position we submitted a No Action Letter to the SEC. Uniquely this is not just another legal article/arguement, because the SEC will not respond to a hypothetical No Action Letter.
In parallel to the legal concept we have launched a Delaware LLC which will follow through with the ICO and execute itself pursuant to a decision tree depending on the SECs response to our letter. The decision tree even includes a possible branch with a vote of the project supporters (the “51% attack”).
Our SEC letter and our Delaware legal documents are open source if you are interested feel free to request and we will send you links.
Sent. Thanks we understand it’s extremely easy to blow off an idea that sounds like “legit ICO” especially when the SEC warns to look out for that language.
If you don’t find all the law to boring and get through it we would love feedback and/or questions.
We previously posted with the links, it was flagged before anyone read our work. Through email exchange Dang agreed we could repost but suggested not post the links to our work. Flagged again, which is strange because it is upvoted.
Abstract: The Green Bay Packers are the only community owned team in the NFL. Legally they are a very unique organization - a non profit with stock-stockholders - in fact the NFL even considers them at a competitive advantage to all other teams because of that and has control over Packers right to offer/issue new stock to raise capital. But when the NFL does permit the Green Bay Packers to raise money by issuing new stock, the Packers file a No Action Letter to the SEC setting out their legal argument that Packers Stock is not a security or does not need to register. We published their original documents, our redline versions and our final version of the SEC letter and Public Offering Document. Our Delaware Certificate of Formation and Operating Agreement are also worth a look because they memorialize the LLC units being maintained and issued on the Ethereum Blockchain and include the addresses for the “Blockchain Stock” (ERC20) and public offering smart contract.
The SEC does not accept hypothetical No Action Letters so behind the cloak of anonymity of The Satoshi Nakamoto Trust we launched a Delaware LLC and submitted our legal argument. Initially intended to be a safe/sandbox startup to legitimize our legal framework, we memorialized the duration/powers/duties of the LLC into a decision tree - which isn’t but could further be memorialized in smarts contracts in future iterations - which begins to look a lot like a Delaware DAO.
SEC No Action Letter- Delaware LLC & Operating Agreement
1. The Green Bay Packer’s Proposed Stock Offering “No Action” Letter dated June 16, 1997 [1]
2. A redline version of the Green Bay Packer’s Proposed Stock Offering “No Action” Letter dated June 16, 1997 [2]
3. The Blockchain Stock Company’s Proposed Blockchain Stock Offering “No Action” Letter dated 31 October 2018[3]
a. Exhibit A - The Blockchain Stock Company’s Certificate of Formation (Articles) [4]
b. Exhibit B - The Blockchain Stock Company’s Operating Agreement [5]
Public Offering Document:
1. The Green Bay Packer’s Public Offering Document [6]
2. A redline version of the Green Bay Packer’s Public Offering Document [7]
3. The Blockchain Stock Company’s Public Offering Document [8]
I started with the glimmer of a hope that perhaps network sync could be made stateless, went down a months-long rabbit-hole of research, and ended up writing a novella-length article about CRDTs: http://archagon.net/blog/2018/03/24/data-laced-with-history/
So many days spent thinking, sketching, trying to swallow a concept that seemed far to big for my jaws—only to suddenly find myself on the other side, with this arcane knowledge fully internalized.
Going from a few wayward thoughts to a working proof-of-concept was the most professionally satisfying thing I've done. It felt like alchemy.
That might be mine as well, https://medium.com/@raphlinus/towards-a-unified-theory-of-op... . It's very much unfinished work though. Certainly wrapping my head around the OT and CRDT literature took huge amounts of brainpower exertion, comparable to what I did for my PhD.
Also, I have your essay in my queue to read more thoroughly, it's high on my list of stuff to study as we possibly rethink the way this stuff works in xi-editor.
Hey Raph, thanks for the vote of confidence! I spent a lot of time looking through your research on OT and CRDTs (plus your Rope Science series) in the course of writing my article, especially as documented and implemented in xi. Really fascinating stuff. I love how many different ways there are to approach this problem.
I found it a few months ago, read it through the end, and shared it with my team. It's the best survey of synchronization techniques that I've come across.
It's just an hypothesis, but I think saying CF when you actually have an atypical form of it is a bit dishonest. I have no doubt that diet and lifestyle do wonders for many conditions including yours, but the CF everyone thinks about is the one discovered in infancy and the full set of symptoms, not the one with late onset (at an age which many CF patients do not even reach) and milder symptoms.
If you were willing to lift up that imprecision, I bet you wouldn't have such a hard welcome.
It's a case of damned if I do, damned if I don't. People with CF don't like me saying "mild" CF. They have a cow about that. I spent years on CF forums and got schooled on that detail quite thoroughly. There are more than 1600 alleles that can lead to a diagnosis of CF and the exact presentation of the condition varies from one person to the next in part because of that.
My condition is not late onset. I have had it my whole life. Being diagnosed late doesn't mean it "came on" late. I spent years being treated by people like I was some kind of hypochondriac and asking doctors "Can we test me for something? My body doesn't seem to work normally." and being blown off.
It's not imprecision. I'm as precise as I know how to be or you wouldn't know that my actual diagnosis is atypical cystic fibrosis. I in no way hide that.
So from where I sit, that just looks like yet another BS excuse or justification for people on the internet to be jerks to me.
I knew it because I looked up your bio for the first time You don't hide it, but I don't remember reading it in your posts here, not as much as the complaints about other people's behavior at least.
>My condition is not late onset.
And the only person I knew with CF had been diagnosed as a toddler and had a 20 y. life expectancy because of genetic bad luck. That you could live 30 years without heavy treatment sure says that your condition was not as bad as that person, and maybe there's a state of affliction that cannot be conservatively managed.
I've spent years trying to figure out how to explain it to other people. I actually joined HN in hopes of learning to program so I could write a simulation to more effectively convey the information, but that hasn't happened. Getting well and other life drama has taken all my time.
Trying to explain it with mere words seems to be something that needs like a million of them and still isn't really adequate.
But, in a nutshell (or an attempt at a nutshell explanation):
First, I did things I knew were helpful from before having a diagnosis and tried to then go figure out why it was helpful. Having gotten diagnosed late in life, I had my own mental models and ideas that didn't jibe with our current official understanding of the condition. I respected my innate knowledge rather than just agreeing with the experts.
Second, I tried to understand the pathology. We know the mechanism behind the condition: A malfunctioning (or sometimes missing) cell channel called the CFTR that handles trafficking of certain molecules into and out of the cell. So I tried to understand how we get from there to all the symptoms typically associated with CF, such as malabsorption and lung infections. That seems to be a different approach from what medicine currently does.
Some of my general conclusions:
Malabsorption leads to malnourishment. Many of my issues were rooted in malnourishment. Improving my nutritional status was a major cornerstone of my efforts to get healthier.
I found ways to compensate for the defective cell channel. One thing that is known is that CF involves misprocessing of fats. I got very picky about what fats I eat and tried to develop some understanding of why some fats are helpful and others are problematic. This seems to be rooted in chemistry and at least partly in how complex the molecule is. My body generally handles long chain triglycerides poorly and I favor medium chain triglcerides, though that is likely a wholly inadequate explanation of the phenomenon.
Most genetic disorders involve a misfolded protein. I learned what promotes misfolds. Chemical derangement in the cell promotes misfolds. I believe this is an important mechanism behind the progression of CF, where you steadily get worse. I think reversing the chemical derangement stopped the positive feedback loop (aka vicious cycle) and this was a huge, huge factor in being able to get healthier rather than just try to slow or halt the progression.
I'm currently trying to figure out what I want to say on my blog about my understanding of "the immune system." When we speak most systems, we can name specific organs. The digestive system involves the stomach, small intestine and large intestine. The circulatory system involves the heart and an extensive system of blood vessels. But I can't list for you the organs involved in the immune system.
So I think better understanding exactly what we mean by that -- what is going on when the body successfully fights off infection -- is a major thing and I am still trying to figure out what I want to say about that. I think we generally have a poor concept for how that happens.
That's probably several hundred thousand words short of conveying much of anything, but it's an attempt to reply to your question.
I was working on a calendaring system. We supported recurring events, and each event could also have a piece of equipment associated with it (one or more). Events could also be changed, either the current event or this and every event. Event start/stop times were stored in UTC, but the event was displayed in the users timezone, and could cross daylight savings time boundaries. This was done with a mix of javascript on the front end and ruby on the backend.
When first implemented, the equipment was reserved for the entire time of the event. A feature request came in to allow for partial reservations of equipment (we want to reserve space A for four hours, but equipment B for only the last hour and equipment C for the first two hours).
Solving it on initial event creation was pretty easy. The UX was difficult. But handling updates, especially across recurring events, in a way that was maintainable and correct was the hardest technical work I've done. I wrote a lot of tests.
The difficulty was compounded by the fact that this was a startup and I had no technical peers to discuss the issue with. The code implementing the work wasn't the cleanest either. I could have reached out to friends, but they wouldn't have had the understanding of the issue.
How did you handle the database storage for recurring events? Did you have just one entry that represents the recurring event as a abstract whole, or a database entry for each instance of the recurring event?
The former seems "better", but you also run into a whole lot of complications:
1) The user can delete specific instances of a recurring event. Eg: delete the event for thanksgiving Thursday but leave it intact for all other Thursdays
2) The user can make instance-specific edits. Eg: edit the event description with custom meeting-notes that's specific to that week's instance
3) The user can invite/dis-invite people for specific instances. Eg: invite Dave only for the event this Thursday, but not for the following weeks'
Creating a database entry for each instance avoids the above complications, but comes with its own drawbacks: if a user creates a recurring event with no end-date, you have to populate a large number of instances, all the way until some arbitrary MAX_DATE.
I was asked this question in an interview once and I recommended the latter option as the lesser evil, and the interviewer was visibly displeased with my recommendation. I'm wondering what the better solution would be.
Some very quick thoughts.. Have something to resemble the event series as a single instance - then have 1...n relation to actual occurrences for occasions when there has been a change, added notes or cancellation. These should be created when the change/added notes/etc. are made.
I have worked with recurring events, and the answer that I've seen is generally both. You have one record which is the "template" and then populate the database out with specific instances to some arbitrary date. Every so often then run a cron to populate further and further out.
That's pretty much what I did, but the original record wasn't stored in the DB, just the specific instances. Our use case was such that folks were moving around events periodically (it was a scheduling app), so we rarely had anyone "run off the end" of their repeated events.
I went with the latter. We limited the total number of events to something like 5 or 10 years to avoid it [edit: the max age problem]. The only real issues that came up from that approach were:
* UX was hard when people want to change individual events that are also part of a recurring event (who wins)
* daylight savings time is an issue when you store datetime in UTC (you need to shift the UTC value in Nov vs in June to have it be the same "8am every Friday" event)
* saving a change across all of a large number of events created a large, slow transaction that caused slow responses (in some cases triggering heroku timeouts)
I was lucky that we didn't need to implement inviting guests, as that wasn't needed.
There were times when I wish I'd implemented the RFC standard: https://www.ietf.org/rfc/rfc2445.txt but never got to that. (We also evaluated just building on top of Google Calendar, but I was worried about building on top of an API that we weren't paying for, and there was the issue of tying in child equipment events.)
I would say store a default event together with the pattern it repeats in. Then create exceptions that hold additional info or a cancellation when needed. When they finally add an end date in one way or the other, add the end date to the default.
On load/view time, generate events from the list of defaults and amend with the exceptions as needed. The user should not see the default/exceptions system, just events and the pattern.
I’d say working with exceptions in this way is more efficient than storing many copies of the same event.
Matterlist's recurring task descriptors ("Schedule Items") essentialy define infinite sequences of recurring events, which are not represented in the database, but are visible and editable via the same UI, just as the regular, non-recurring tasks.
Here's how we deal with the problems you outlined:
1) When a user deletes a specific instance, we create an "exclusion" record for that day, so that the function that generates an endless sequence of recurrences doesn't generate one for that day.
2) When a user edits a specific instance of a recurring task, we also create an "exclusion" record for that day (because otherwise we would have two tasks on that day, the edited instance, now with a database record, and the virtual instance), and we create a database record for the edited instance, so it just becomes a regular task.
3) We don't have shared tasks yet, but that would be handled via the same framework, as outlined in 1 and 2 above. We'd just de-virtualize an instance and add/remove users to it.
Creating a database entry for each instance of a recurring task was unacceptable for us, because we wanted infinite forward visibility, and because when a user edits the parameters of a recurring task (e.g. changes it from daily to weekly), we'd have to delete all the old records and create the new records.
BTW, we have finally released an Android app, so you can see our recurring tasks in action: https://matterlist.com/#apps
Outlook/exchange uses the former approach, a pattern with a list of instance-specific exceptions. One reservation system I worked on used the latter approach, with every instance saved to the database. I was tasked with building an outlook plugin to smoothly integrate the two, and it was probably the most difficult piece of software to get working reliably I ever wrote. The prototype took a few days, the fully reliable version compatible with all outlook versions took two years. Outlook’s internals are pure madness.
The hardest thing I ever worked on was a calendaring system for scheduling ads on/off. The UI was great, but I just told them "oh, send me a cron format string for scheduling". It was easy for them to do, but very hard to handle on the back end. Cron format strings aren't fun to reverse engineer. Why did I do that?
File under "things programmers assume about time". It kept doing "weird" things from the user perspective and I ended up writing more tests for that subsystem than anything else I've ever done.
Ya, I hate this question as an interview one. I’ve solved lots of hard problems, but only the ones I haven’t solved yet stick around in my head. It’s even worse if they aren’t interested in design problems, but rather technical ones.
Ha, this is actually why I thought to ask it. Sadly, "I read the docs and wrote sample code for days until it became obvious" isn't what interviewers are looking for.
I know, but I debugged a lot of use cases, tried to narrow the scope of the bug, thought really hard about the execution in my head, and then fixed it all using an explicit semicolon....
Asking an experienced dev this question is like asking someone what was the hardest food they had to eat and how did they manage to swallow it. What is the interviewer hoping to learn by that?
For my thesis project back in 2011, I wanted to create an interactive installation, the kind you see at museums. Since multitouch tables were insanely expensive to buy or rent, I decided to build one myself. Learned the basics of woodworking over a semester, and built a table after two failed attempts. Then I bought an LCD TV, took it apart, connected it to a playstation camera with a modified infrared lens and developed an Adobe Air application. The thing was so unresponsive to touches, I decided to ditch the whole multitouch table idea. Started from scratch again ... learned objective-c (iOS 5 or was it 6), and created an iPad app for my thesis. Don't think I've ever worked so hard in my life.
There's actually a quite nifty principle that makes it relatively easy to build your own multitouch table: Frustrated total internal reflection.
The idea is based on internal reflection: Whenever lights hits the boundary between two materials with different density it gets refracted. If it goes from dense to less dense and the angle is flat enough it will reflect back. This is the principle behind fibre optics. It's also the reason why the water's surface looks "silvery" (like a mirror) when you're submerged and look up at an angle (i.e. not straight up).
Now imagine you have a plane of glass and you put LEDs around the edges that shine into it from the side. The light will zig-zag trough the glass and come out at the other edge. However, if you touch the glass you inhibit that total internal reflection because your finger is alot denser than the air and so the light will "leak" out the glass where you touch it, illuminating your finger. If you look at the glass from behind you'll see a bright spot.
Use a camera to detect that spot and you basically have a touchpad. To make it a screen you can put a translucent sheet behind the glass and project an image onto it (and use infrared LEDs).
See e.g. http://wiki.nuigroup.com/FTIR for some helpful images. Just google "FTIR touch screen" or similar for build instructions and blob-detection software.
Multiplayer engine for an open-world RPG that doesn't require developer to run a dedicated server. Think Skyrim, GTA or Kingdom Come, but with a much smaller budget for content - aside from very fast movement and physics-based mechanics of GTA, this engine could've been used for any of these games. Previous developers did the single-player logic first, and then added very ad-hoc multiplayer sync, hoping that everything will be fine - but if you played long enough, you inadvertently ran into desyncs.
So, I had to write first the network engine itself (which was used both for networking and load/save), and then rewrite the whole game logic to take advantage of this mechanism. Now, if you want to write any game mechanic, you have to define it in proper MVC terms (although, due to amount of legacy code, I didn't separate the view and controller as well as I should have). Dozens and hundreds of small scripts for all the unique quest content and game mechanics working on top of a unified platform, relying on it to sync their data, one-off events (think "play sound" command that doesn't change any data but has to be replicated) and not even knowing on which one of connected clients is it run with authority, while all others run this particular object in "slave" mode.
I honestly tried to research if any other game developer have used a similar solution, but found nothing - it seems as if all open-world games that have multiplayer rely on dedicated authoritative servers, operated by developer, to run. Here's the project itself, although this update is not yet released: https://store.steampowered.com/app/526160/The_Wild_Eight/
Too bad that at the end of this project I got the worst case of burn out in my career, spiraled into the worst depression episode that I've ever experienced, and now quit the company while struggling to put my life back together. I have no idea if I'll be able to publish anything on this tech, but I still feel that it's been worth it.
Develop a tool to boost the productivity of a software engineering team and convince all members to adopt it. Software engineers (myself included) can be very resistant to changes (especially when it comes to their tooling) and an in-house tool with a single developer behind it didn't contribute to the ease of adoption, but it was adopted regardless. I guess that the tool was indeed very good, or maybe I'm a sweet talker :)
Note: the team was very specialized, so there really were no competing tools around for that specific task.
The team had to work with tens of different embedded devices with all kinds of weird compilers and flashing tools.
My tool would help with the context switching between projects, providing wrapper scripts and inline documentation. It was some sort of literate programming in bash :)
I built an olap cube for manipulating time series of financial models, and made an API to serve it through a query language that I created just for this.
I worked on an online test with different types of question with video viewing.
The hard part is the video viewing progress is tracked by system. I have to implement my own video server and player. Youtube was new and wanted to use it but I can't track their view progress.
Yes this is generally the idea. Didn't know about OpenCV though. Was a font-matching and trial and error by hand in MS Notepad + some scripts to help brute force the thing kind of solution... Turns out there are actually at least a couple dozen possible solutions based on what is visible so some small amount of information was lost through the redaction, but not tons.
Yes, it's a monospace font. The start, end and pattern of visible ascenders behind the redaction, along with the word list in electrum would allow you to reduce the search space by a lot, I suppose.
I'll give you a hint. Download the image. Because it is PNG that means it is lossless... Zoom in very closely on the top row of pixels above the redacted area and you may discover you have all that you need...
Thanks. I already did that, but there were quite a few letters that were completely covered.
Programming isnt' my day job - but I have an interest in it, so was curious to see what tools / approach you used to automate this and whether it was fully automated or whether your solution recovered the partially covered letters and your 'wetware' recovered the rest ;)
Cyph0n already provided an approach, which you confirmed - so thanks again.
Years ago I wrote an emulator for the Intel 8086 processor in C++. It's deceivingly difficult because the instruction encoding is complex and the emulation of each instruction has to have a very high fidelity. In a sense, software at the CPU instruction level is a chaotic system, as each instruction can influence the system state to a critical degree, so if there is a slight deviation from the spec/hardware, a snowball effect of deviations happens that leaves the system in a completely botched state where your emulator won't boot at all. Because the deviation can happen anywhere within an execution path of millions of instructions, it's very hard to debug, too.
Eventually I got it working and I could boot DOS and play games.
The aim of my project was to create a programmable emulator that could be used for the semi-automated analysis of malware, and sell it. Eventually this didn't really go anywhere as this was a too ambitious goal to do all by myself. See here [1] for a demonstration where I load tweets from Twitter and send them to the DOS text editor by triggering a keyboard interrupt via a Python API. Fun...
Later, the Unicorn CPU emulator framework implemented my idea much more effectively by creating Python bindings to an already mature emulator (QEMU).
I solved some interesting technical problems this year. One was optimizing a script that took over eight hours to run, to just twelve seconds (validation code brought that back up to several minutes, but still a lot better than eight hours). Another was taking several map pins whose coordinates were based on the same physical address (just different suites/floors, and having the same exact coordinates returned from the API) and staggering them. I ended up using a spiral pattern to do the staggering.
An ongoing soft-skill problem I am learning to solve is the effective customer development.
An interesting UI/UX problem I am currently thinking about is how to allow people to draw sun/shade patterns, wind current patterns, and essentially contour lines that show elevation on their property (the last is to ultimately learn how water flows on their property, and where there are drainage problems) in a fun, minimal-effort way. If anyone has any suggestions on this, please feel free to share.
The last two types of problems are for my side project, AutoMicroFarm.
IMO the most fun way to enter elevation data for your property would be to ride around on your farm bike with an altimeter lashed up to a smartphone, gathering data as you go.
Thanks for the idea! I found a variety of altimeter apps for the phone; I'll have to try just walking around my yard with one (or several), then see if it's possible to get elevation data on a map from such an app.
I'd suggest getting an app like Sensors Multitool for Android which gives you the raw output of your phone's sensors. I tried one of the altimeter apps a few years ago to see how tall a big hill was, and the value that the app reported turned out to be far off the actual value. There are various techniques for mixing and cleaning data from the GPS and barometric pressure sensors and a premade altimeter app (if it's not open source) obfuscates what's really going on with the values you'll actually end up seeing if you create an app for your microfarming project.
Also be aware that different phones have different levels of support for non-GPS location services. I was surprised to learn recently that my Pixel 3 can use data from the US, EU, and Russian satellite geolocation systems for better coverage/precision.
Thanks for the tips! I don't need accurate readings, just precise ones (so the map can be internally consistent; here's what I drew manually for my property: https://i.imgur.com/2ZGmB1M.png). I'll keep in mind what you suggest when I am ready to develop that part of the app.
Well, I tried both Accurate Altimeter and Sensors Multitool, and I must say I'm disappointed. The values varied by 20 meters! In reality, my property varies by 2 to 3 meters at most.
I'll keep thinking about how to solve the problem.
439 comments
[ 3.1 ms ] story [ 314 ms ] threadWe aren't 100% there yet, but we are shipping numerous times per day across 20 or so services and quality has gone _up_, not down
https://code.fb.com/web/rapid-release-at-massive-scale/
[1] commits were pushed to employees on commit, that's how we tested, but not to the public
I could spend time on marking features 'frontend only, low impact' which we could deploy pretty much the same day. Still there are quite some features that need bigger amount of work where they might be 'done' by dev but I am sure they are not actually done, because security, because error checking. It of course is usually that one dev has his not tested feature merged to develop and then also some other dev has production ready one, then if one feature is production ready, I would have to put also some time to make release and pick only changes for accepted change. I am not sure that additional work to check what we can release ' right now' will pay off vs just taking time for fixes on acceptance by people who were working on code and then release it (after 2 weeks o 1 depending how fast it is done in sprint).
So are you having people who work only on picking changes that are low impact, or working on making stuff production ready by picking from develop? Maybe you pick changes by yourself, or you just defer manual testing to end users and rely on automated tests unit/integration?
p.s. Funny thing with automated tests is they are good at keeping old stuff working but not for testing newly developed features where actual tester can test new GUI/ new features. If you have a lot of GUI changes you cannot automate first tests...
Mostly it comes down to organizational changes and everybody getting used to what constitutes "enough" tests.
* Every PR is treated as "production" ready. This means if it isn't ready for user eyes, it gets feature flagged or dark shipped. Engineers have to assume their commit will go into prod right away. Feature flags become pretty important * Product Owners and QA validate code is "done" in lower environments (acceptance or staging, or even locally during PR phase). This helped us decouple code being in prod and "definition of done" in our sprints. * All API changes and migrations follow "expand and contract" models that makes the code shippable. e.g. even if we are building new features, we can ship our code at anytime cause the public api is only expanding. * More automated quality checks at PR time. Unit tests, integration tests, danger rules, etc. These vary from codebase to codebase. A key part of this is trusting the owners of that code or service. To a degree, if they are happy with their coverage, then they can ship. (Within limits of course. Not having unit tests at least would be a red flag)
Also, we still ship numerous times a day without a full automation test, we just make sure each release is really small (1-3) commits. The smaller the release the easier it is to manually QA it. So nothing fancy is needed, just smaller releases.
So to answer your question, we don't pick and choose work that is "low" impact or "high" impact, all code gets shipped the same and with the same cadence. It is our responsibility to ensure that when it goes to prod, it won't break anything
In particular - documentation and support.
Using complicated interfaces is very challenging sometimes, to the point of obscene - Google searches for help turn up a variety of outdated answers - and it's impossible to know what's what.
I'm using Facebook Ads right now quite a lot - it's complex system that just shifts like quicksand under your feet.
Users make incredible efforts to learn the product, only to have it shift away from them like a ghost.
Documentation may or may not be up to date.
Locations of things change.
And who can you ask? Where do you search for support? Facebook has never really provided me answers to many questions in their documentation.
Tiny example - an advisor used to know how to list people that have commented on a post, so that you could 'invite them to like your page'. But it's changed and now he can't find it. One small thing now that he doesn't know how to do, a tool list from his tool-chest.
And who really gains from all of this? Seriously? I don't think anyone.
Is that 'new feature' really that important? It needs to be rolled out 'now' instead of in a major/minor delivery?
These changes are seldom well communicated either.
I suggest the total opposite might be better: release major iterations every year, minor iterations quarterly, and patches as needed.
Every time there is a release - provide users friendly release notes - something they can read at a quick glance and get up to speed. A little 10 second video for every change: "Oh, now you can do XYZ like this"
This way - you have predictability and consistency so users can know how to 'keep up'.
Also - the ability to use the OLD interface where possible for at least 1 year, or something like that - that we users are not forced onto the quicksand.
"Ok it's Jan 1 - FB Ads 2019 is released in 1 month - let's go over the changes - Mary, you can be responsible for highlighting the major changes and communicating them to the rest of the team, and highlighting any risks for us"
Otherwise, you're trolling along, these companies make changes that could feasibly have major impact on your business and you're out to lunch.
Maybe internally continuous delivery could be a useful thing ... but for the world at large the downsides are real and the upsides are limited.
The simple part is making it work without clashing. The hard part is making inlining always be an optimisation with reasonable code growth with regards to compilation time and cache use.
My friends do it pretty often. I find it super relaxing, and know I'll be happy I did it, but the engineer / researcher in me wants to always have fingers on a keyboard.
To understand what "controlling behaviour" means.
To empathise, and see oneself from the eyes of others.
I dont struggle as I used to since I've learned it is about environment we are in (meaning as in video) and tomorrow is uncertain (because many people are involved)
[0] https://github.com/asummers/erlex [1] https://github.com/jeremyjh/dialyxir
Despite it being really interesting, some features like scope analysis for variables detection and conversion, the possiblity of multiple modules in an Elixir file and the Elixir pipes which need to be translated to sequential function calls are honestly a pain in the ass.
Keep at it and I'm sure you'll crack it!
[0] https://github.com/jxub/doppelganger
This was the hard problem you solved, right? How did you go about it?
Where others have failed we believe we have developed a legal framework for both a non security/unregistered ICO and legal DOA.
To legitimize our legal position we submitted a No Action Letter to the SEC. Uniquely this is not just another legal article/arguement, because the SEC will not respond to a hypothetical No Action Letter.
In parallel to the legal concept we have launched a Delaware LLC which will follow through with the ICO and execute itself pursuant to a decision tree depending on the SECs response to our letter. The decision tree even includes a possible branch with a vote of the project supporters (the “51% attack”).
Our SEC letter and our Delaware legal documents are open source if you are interested feel free to request and we will send you links.
If you don’t find all the law to boring and get through it we would love feedback and/or questions.
The SEC does not accept hypothetical No Action Letters so behind the cloak of anonymity of The Satoshi Nakamoto Trust we launched a Delaware LLC and submitted our legal argument. Initially intended to be a safe/sandbox startup to legitimize our legal framework, we memorialized the duration/powers/duties of the LLC into a decision tree - which isn’t but could further be memorialized in smarts contracts in future iterations - which begins to look a lot like a Delaware DAO.
SEC No Action Letter- Delaware LLC & Operating Agreement
1. The Green Bay Packer’s Proposed Stock Offering “No Action” Letter dated June 16, 1997 [1]
2. A redline version of the Green Bay Packer’s Proposed Stock Offering “No Action” Letter dated June 16, 1997 [2]
3. The Blockchain Stock Company’s Proposed Blockchain Stock Offering “No Action” Letter dated 31 October 2018[3]
a. Exhibit A - The Blockchain Stock Company’s Certificate of Formation (Articles) [4]
b. Exhibit B - The Blockchain Stock Company’s Operating Agreement [5]
Public Offering Document:
1. The Green Bay Packer’s Public Offering Document [6]
2. A redline version of the Green Bay Packer’s Public Offering Document [7]
3. The Blockchain Stock Company’s Public Offering Document [8]
[1] https://drive.google.com/file/d/1hhv5dKkU2R9D81wZUYaH5wxWAE1... [2] https://drive.google.com/file/d/1ATCnecraourkjNx5FYPsXzPsoJe... [3] https://drive.google.com/file/d/1XbUuDO-R1JUMh_BnHnSbcq5rRG5... [4] https://drive.google.com/file/d/1mrzY9mTXAqgXq-0SzevvYOmGCvK... [5] https://drive.google.com/file/d/1noePTMojVq-sw3UVMxSffAss1zX... [6] https://drive.google.com/file/d/1yrfu1eTGiev8bOa_cnGiOPr3-YU... [7] https://drive.google.com/file/d/10J1-oGCKXmuK1WeUqdcNk0ntjy1... [8] https://drive.google.com/file/d/12I...
So many days spent thinking, sketching, trying to swallow a concept that seemed far to big for my jaws—only to suddenly find myself on the other side, with this arcane knowledge fully internalized.
Going from a few wayward thoughts to a working proof-of-concept was the most professionally satisfying thing I've done. It felt like alchemy.
Also, I have your essay in my queue to read more thoroughly, it's high on my list of stuff to study as we possibly rethink the way this stuff works in xi-editor.
I found it a few months ago, read it through the end, and shared it with my team. It's the best survey of synchronization techniques that I've come across.
Next up: Figuring out how to talk to the world about it effectively. I'm not sure that's solvable. "Curing CF" seems easier.
If you were willing to lift up that imprecision, I bet you wouldn't have such a hard welcome.
My condition is not late onset. I have had it my whole life. Being diagnosed late doesn't mean it "came on" late. I spent years being treated by people like I was some kind of hypochondriac and asking doctors "Can we test me for something? My body doesn't seem to work normally." and being blown off.
It's not imprecision. I'm as precise as I know how to be or you wouldn't know that my actual diagnosis is atypical cystic fibrosis. I in no way hide that.
So from where I sit, that just looks like yet another BS excuse or justification for people on the internet to be jerks to me.
>My condition is not late onset.
And the only person I knew with CF had been diagnosed as a toddler and had a 20 y. life expectancy because of genetic bad luck. That you could live 30 years without heavy treatment sure says that your condition was not as bad as that person, and maybe there's a state of affliction that cannot be conservatively managed.
>Next up: Figuring out how to talk to the world about it effectively.
My guess is that you will be unsuccessful. (from similar accounts that I have read )
I've spent years trying to figure out how to explain it to other people. I actually joined HN in hopes of learning to program so I could write a simulation to more effectively convey the information, but that hasn't happened. Getting well and other life drama has taken all my time.
Trying to explain it with mere words seems to be something that needs like a million of them and still isn't really adequate.
But, in a nutshell (or an attempt at a nutshell explanation):
First, I did things I knew were helpful from before having a diagnosis and tried to then go figure out why it was helpful. Having gotten diagnosed late in life, I had my own mental models and ideas that didn't jibe with our current official understanding of the condition. I respected my innate knowledge rather than just agreeing with the experts.
Second, I tried to understand the pathology. We know the mechanism behind the condition: A malfunctioning (or sometimes missing) cell channel called the CFTR that handles trafficking of certain molecules into and out of the cell. So I tried to understand how we get from there to all the symptoms typically associated with CF, such as malabsorption and lung infections. That seems to be a different approach from what medicine currently does.
Some of my general conclusions:
Malabsorption leads to malnourishment. Many of my issues were rooted in malnourishment. Improving my nutritional status was a major cornerstone of my efforts to get healthier.
I found ways to compensate for the defective cell channel. One thing that is known is that CF involves misprocessing of fats. I got very picky about what fats I eat and tried to develop some understanding of why some fats are helpful and others are problematic. This seems to be rooted in chemistry and at least partly in how complex the molecule is. My body generally handles long chain triglycerides poorly and I favor medium chain triglcerides, though that is likely a wholly inadequate explanation of the phenomenon.
Most genetic disorders involve a misfolded protein. I learned what promotes misfolds. Chemical derangement in the cell promotes misfolds. I believe this is an important mechanism behind the progression of CF, where you steadily get worse. I think reversing the chemical derangement stopped the positive feedback loop (aka vicious cycle) and this was a huge, huge factor in being able to get healthier rather than just try to slow or halt the progression.
I'm currently trying to figure out what I want to say on my blog about my understanding of "the immune system." When we speak most systems, we can name specific organs. The digestive system involves the stomach, small intestine and large intestine. The circulatory system involves the heart and an extensive system of blood vessels. But I can't list for you the organs involved in the immune system.
So I think better understanding exactly what we mean by that -- what is going on when the body successfully fights off infection -- is a major thing and I am still trying to figure out what I want to say about that. I think we generally have a poor concept for how that happens.
That's probably several hundred thousand words short of conveying much of anything, but it's an attempt to reply to your question.
When first implemented, the equipment was reserved for the entire time of the event. A feature request came in to allow for partial reservations of equipment (we want to reserve space A for four hours, but equipment B for only the last hour and equipment C for the first two hours).
Solving it on initial event creation was pretty easy. The UX was difficult. But handling updates, especially across recurring events, in a way that was maintainable and correct was the hardest technical work I've done. I wrote a lot of tests.
The difficulty was compounded by the fact that this was a startup and I had no technical peers to discuss the issue with. The code implementing the work wasn't the cleanest either. I could have reached out to friends, but they wouldn't have had the understanding of the issue.
It was one massive several-month nightmare. Totally relate...
The former seems "better", but you also run into a whole lot of complications:
1) The user can delete specific instances of a recurring event. Eg: delete the event for thanksgiving Thursday but leave it intact for all other Thursdays
2) The user can make instance-specific edits. Eg: edit the event description with custom meeting-notes that's specific to that week's instance
3) The user can invite/dis-invite people for specific instances. Eg: invite Dave only for the event this Thursday, but not for the following weeks'
Creating a database entry for each instance avoids the above complications, but comes with its own drawbacks: if a user creates a recurring event with no end-date, you have to populate a large number of instances, all the way until some arbitrary MAX_DATE.
I was asked this question in an interview once and I recommended the latter option as the lesser evil, and the interviewer was visibly displeased with my recommendation. I'm wondering what the better solution would be.
* UX was hard when people want to change individual events that are also part of a recurring event (who wins)
* daylight savings time is an issue when you store datetime in UTC (you need to shift the UTC value in Nov vs in June to have it be the same "8am every Friday" event)
* saving a change across all of a large number of events created a large, slow transaction that caused slow responses (in some cases triggering heroku timeouts)
I was lucky that we didn't need to implement inviting guests, as that wasn't needed.
There were times when I wish I'd implemented the RFC standard: https://www.ietf.org/rfc/rfc2445.txt but never got to that. (We also evaluated just building on top of Google Calendar, but I was worried about building on top of an API that we weren't paying for, and there was the issue of tying in child equipment events.)
On load/view time, generate events from the list of defaults and amend with the exceptions as needed. The user should not see the default/exceptions system, just events and the pattern.
I’d say working with exceptions in this way is more efficient than storing many copies of the same event.
Matterlist's recurring task descriptors ("Schedule Items") essentialy define infinite sequences of recurring events, which are not represented in the database, but are visible and editable via the same UI, just as the regular, non-recurring tasks.
Here's how we deal with the problems you outlined:
1) When a user deletes a specific instance, we create an "exclusion" record for that day, so that the function that generates an endless sequence of recurrences doesn't generate one for that day.
2) When a user edits a specific instance of a recurring task, we also create an "exclusion" record for that day (because otherwise we would have two tasks on that day, the edited instance, now with a database record, and the virtual instance), and we create a database record for the edited instance, so it just becomes a regular task.
3) We don't have shared tasks yet, but that would be handled via the same framework, as outlined in 1 and 2 above. We'd just de-virtualize an instance and add/remove users to it.
Creating a database entry for each instance of a recurring task was unacceptable for us, because we wanted infinite forward visibility, and because when a user edits the parameters of a recurring task (e.g. changes it from daily to weekly), we'd have to delete all the old records and create the new records.
BTW, we have finally released an Android app, so you can see our recurring tasks in action: https://matterlist.com/#apps
File under "things programmers assume about time". It kept doing "weird" things from the user perspective and I ended up writing more tests for that subsystem than anything else I've ever done.
https://www.amazon.com/Dreaming-Code-Programmers-Transcenden...
Great read, and shows how difficult it was and is to compete with office.
Asking an experienced dev this question is like asking someone what was the hardest food they had to eat and how did they manage to swallow it. What is the interviewer hoping to learn by that?
Tried many libraries and eve saas solutions like monkeylearn, etx but couldn't do it to what was required.
Had to switched to a mechanical turk like solution in the end :/
The idea is based on internal reflection: Whenever lights hits the boundary between two materials with different density it gets refracted. If it goes from dense to less dense and the angle is flat enough it will reflect back. This is the principle behind fibre optics. It's also the reason why the water's surface looks "silvery" (like a mirror) when you're submerged and look up at an angle (i.e. not straight up).
Now imagine you have a plane of glass and you put LEDs around the edges that shine into it from the side. The light will zig-zag trough the glass and come out at the other edge. However, if you touch the glass you inhibit that total internal reflection because your finger is alot denser than the air and so the light will "leak" out the glass where you touch it, illuminating your finger. If you look at the glass from behind you'll see a bright spot.
Use a camera to detect that spot and you basically have a touchpad. To make it a screen you can put a translucent sheet behind the glass and project an image onto it (and use infrared LEDs).
See e.g. http://wiki.nuigroup.com/FTIR for some helpful images. Just google "FTIR touch screen" or similar for build instructions and blob-detection software.
So, I had to write first the network engine itself (which was used both for networking and load/save), and then rewrite the whole game logic to take advantage of this mechanism. Now, if you want to write any game mechanic, you have to define it in proper MVC terms (although, due to amount of legacy code, I didn't separate the view and controller as well as I should have). Dozens and hundreds of small scripts for all the unique quest content and game mechanics working on top of a unified platform, relying on it to sync their data, one-off events (think "play sound" command that doesn't change any data but has to be replicated) and not even knowing on which one of connected clients is it run with authority, while all others run this particular object in "slave" mode.
I honestly tried to research if any other game developer have used a similar solution, but found nothing - it seems as if all open-world games that have multiplayer rely on dedicated authoritative servers, operated by developer, to run. Here's the project itself, although this update is not yet released: https://store.steampowered.com/app/526160/The_Wild_Eight/
Too bad that at the end of this project I got the worst case of burn out in my career, spiraled into the worst depression episode that I've ever experienced, and now quit the company while struggling to put my life back together. I have no idea if I'll be able to publish anything on this tech, but I still feel that it's been worth it.
Note: the team was very specialized, so there really were no competing tools around for that specific task.
My tool would help with the context switching between projects, providing wrapper scripts and inline documentation. It was some sort of literate programming in bash :)
Using a mock libc to intercept everything since there was no strace.
Porting Ruby to the IBM Blue Gene/L.
Updating a DAG stored in a SQL table.
The hard part is the video viewing progress is tracked by system. I have to implement my own video server and player. Youtube was new and wanted to use it but I can't track their view progress.
https://btcblockchain.files.wordpress.com/2015/02/e2.png
So what I would do is:
1. Estimate the possible locations of spaces (e.g., using OpenCV).
2. "Guess" certain letters based on what is visible above and below the black bar; these are your constraints.
3. Look up dictionary words that meet the constraints found in (2).
4. Loop through steps 1-3 to build a list of possible seeds.
5. Run these seed candidates through Electrum.
Programming isnt' my day job - but I have an interest in it, so was curious to see what tools / approach you used to automate this and whether it was fully automated or whether your solution recovered the partially covered letters and your 'wetware' recovered the rest ;)
Cyph0n already provided an approach, which you confirmed - so thanks again.
Eventually I got it working and I could boot DOS and play games.
The aim of my project was to create a programmable emulator that could be used for the semi-automated analysis of malware, and sell it. Eventually this didn't really go anywhere as this was a too ambitious goal to do all by myself. See here [1] for a demonstration where I load tweets from Twitter and send them to the DOS text editor by triggering a keyboard interrupt via a Python API. Fun...
Later, the Unicorn CPU emulator framework implemented my idea much more effectively by creating Python bindings to an already mature emulator (QEMU).
[1] https://www.youtube.com/watch?v=XwPZH8LAVIY
[2] https://www.unicorn-engine.org/
An ongoing soft-skill problem I am learning to solve is the effective customer development.
An interesting UI/UX problem I am currently thinking about is how to allow people to draw sun/shade patterns, wind current patterns, and essentially contour lines that show elevation on their property (the last is to ultimately learn how water flows on their property, and where there are drainage problems) in a fun, minimal-effort way. If anyone has any suggestions on this, please feel free to share.
The last two types of problems are for my side project, AutoMicroFarm.
Also be aware that different phones have different levels of support for non-GPS location services. I was surprised to learn recently that my Pixel 3 can use data from the US, EU, and Russian satellite geolocation systems for better coverage/precision.
I'll keep thinking about how to solve the problem.