Ask HN: How do you develop internal tools for your organization?
We are considering creating few internal tools to speed up the work of a few departments in our organization, but before starting development heads on, I would love to gather some wisdom from HN.
First, what tools are you developing?
How do you handle user authentication?
Do you develop one single big application for everything in your organization, or few distinct applications, each with its own specific goal?
Any specific thing to watch out for?
Thanks!
8 comments
[ 3.7 ms ] story [ 27.5 ms ] thread>We are considering creating few internal tools to speed up the work of a few departments in our organization, but before starting development heads on, I would love to gather some wisdom from HN. First, what tools are you developing?
TL;DR: a machine learning platform that lets you use your own Kubernetes clusters from your cloud provider (GCP, Azure, AWS) and get real-time collaborative notebooks to train, track, package, deploy, and monitor your machine learning models.
>How do you handle user authentication? TL;DR: manually added at first, and later added invite links and password-reset flows. Still invite based.
>Do you develop one single big application for everything in your organization, or few distinct applications, each with its own specific goal?
TL;DR: plugin system with a core that loads extensions at run-time. Each extension is its own separate repository. We could compose different products.
>Any specific thing to watch out for?
Yes, here we go...
Context: consultancy specialized in building end-to-end products for large enterprise. Around 17 people in 2018 working on about 8 projects simultaneously. CEO (co-founder) in one country and CTO (co-founder) in another country with the team. CTO quits a year and a half after I joined. Pretty much everyone quit after that except literally a handful. A colleague and I took over. We killed projects that were going nowhere, unstuck some, tightened the stack (it was Scala-Python-Spark-Kafka-Hadoop-Clojure-Vue-and what have you).
Problem: building end-to-end products for several sectors (telcos, energy, banking, transportation, health) meant that in addition to diving into different domains, acquiring data from peculiar data sources, and building the models, we also had to build bespoke "quick-no-time-for-reusability-one-off" applications that used these models and, in some cases, enabled domain experts at client organizations to train models themselves on new data. User authentication and administration, workspaces/groups, etc.
This also created a situation where I bumped into a university classmate founding a startup who described his problem, that was exactly the same as a problem we solved for a telco for which we built a product, but we were unable to sell him the product because it was custom-made. It sucked. He needed the exact same features but although they were contained, the containing application was too custom made.
The next project we took on, we wanted to change the way we develop product, and the way we worked. I remember the first few commits I made sure to set up a plugin architecture so we could have a minimal core that practically did nothing but load extensions dynamically, and all the functionality would be in these extensions with separate repositories so we could compose and add and remove to create different products.
Lesson: It should be as easy to remove things as it is to add things.
Next, in 2019, we started experimenting with remote work. It was painful to watch colleagues look at the window to check if their train arrived, or to arrive at the office at 11AM because commute sucked. They were wasting up to 6 hours per day in commute. We experimented with remote work one day a week and saw what broke and fixed it. Then two days a week. Then a whole week here and there. The rationale was the we may have to do it some day and we might as well learn the lessons right then when the stakes are low not to be taken by surprise when the stakes are higher.
In the meantime, my colleague (data scientist) had an obligation that put him away for one year. He could read emails but not reply, or open web pages due to the poor internet connection. This introduced a forcing function for asynchroneous communication, tracking, collaboration to be built into the internal platform. We were tired of not knowing which approaches generated the best models, and being unable to easily r...
Thanks a lot for your reply!
The product is quite rough right now, but I’ve been working with a first client who seem to enjoy it. Happy to demo it.
To answer your specific questions: 1) User authentication was a centralized service that managed roles and permissions. We integrated with LDAP so that we would only authenticate current employees or contractors, then the service would look up the appropriate permissions. We delegated management of roles/permissions to a business owner in the department and our IT team handled LDAP.
2) Each app was separate - each had 1 app, 1 database, 1 project board. I think the real benefit with this was a simpler mental model while working with the systems, but this came at the cost of managing shared services/packages.
We used frameworks that were more opinionated so that each app made similar assumptions and used the same tooling. I started the team using C# MVC, but we moved to NodeJS on the backend to reduce context switching between FE/BE. The big thing is consistency so that each app's codebase looks and feels the same. When you're jumping between apps, you don't want a day long ramp up period.
Similarly, I made sure all devops type tasks were streamlined. We did CI/CD before our product teams did and had full test suites. Preview environments were setup for developers and users. And everything was documented with guides. We could clone our project blueprint repo and deploy it in 5 minutes. A working CRUD app could be done in a few hours.
One unnecessary thing I choose to do was create SPA frontends. With our blueprint and shared services, it reduced the complexity, but it really gave the team something to learn/dive into. I think that little bit of "sexy tech" went a long way for morale and recruiting.
If I had to do this all over again: 1) I'd do a better job defining the team's value proposition. Those basic CRUD apps could be moved to a no-code platform. The team should be focused on apps where proprietary engineering will benefit the company most. Some of my apps had estimated value-add of $5mm+/year, but we would still get pressure from the execs to make a new app that would save the company 50k.
2) Similarly, I'd make sure my team was separate from rest of the org, as long as it makes sense. When we first got started, I was under a product director. As we grew, we never really found our own space in the org and got pulled in different directions. fwiw, I left the org because they wouldn't promote me.
3) I'd start with each app being separate then move to 1 app per department. We worked towards this as we grew and multiple apps needed to be integrated with each other. I think I moved too late though and our integrations reflected it. Especially as we could have justified a small team for each department, it would have made sense. As a management aside, I wanted to be more agile and form teams around projects - 1 app, 1 database, 1 project board, 1 team. It worked fine, but something something Conway's law.
Some things I think I did right: 1) My engineers did everything - project management, roadmaps, engineering, etc. Even when we had dedicated project managers, my engineers were always in the room because the business doesn't know what they want. We had a number of BIG wins because my engineers could come up with non-obvious solutions. I also gave my team the autonomy to solve the business problem whatever way they saw fit.
2) I hired people who cared more about solving problems than the tech. I encouraged doing more design work than engineering and spending time with our users. T...
You should put it into a blog post that I am sure a lot of people will love to read it!