Ask HN: How’d you go from a noodler to writing large well structured programs?

I suggest you hire a software developer, preferably a senior one if you can afford it. The difference between a bunch of impressive algorithms written by specialists in the field stitched together by brittle connections, and a bunch of impressive algorithms that are part of a well structured, tested and maintainable suite is huge. Reading a book or two is not going to solve the problem in the long term, which I assume is what you want.

Hiring a software developer who can take a lead role in organizing and structuring the codebase will also make the specialists better programmers, since they learn good programming techniques from someone who actually knows them.

Source: Senior software developer, where I initially started as a "algorithm/application developer" and saw the team grow and benefit by hiring some experienced developers.

Lots of mistakes, and working with smart people who were kind enough to point out the deficiencies in my early and crude approaches, reading many articles and blog posts on every sort of programming topic, and finally and most directly important was reading through codebases at places I worked or open-source projects in my early years and learning from what other people were doing more correctly.

https://bramcohen.livejournal.com/4563.html ... the founder of bittorent.

Ill add a personal note. Being a good professional programmer is also about understanding the context of your program. You will not always have the time to optimize for performance. You will not always have the time to refactor functionality to make it more modular for further variety. Sometimes you will need to skip test becuase deadline says it should be done yesterday. Learn to not be emotionally tied to the code. Get your job done, to the best quality you can for the time you have allotted. Dont be made if its not perfect. But during the train ride or in the shower think about how to make it better. This process of just grasping with the ideas of clean code are good training for when you have to approach a similar problem on the fly.

Look at some good open source projects, and look at the overall project/module structure as well as the OO graph.

Re: high-level project layout: - where are their build files? - how are the source packages broken up? - where are the tests stored?

Trace back from the entry point to the code and look at how things are encapsulated, where code gets re-used.

Diagramming out with something like UML is cumbersome but can be helpful to visualize the structure as you're getting started.

Looking at software design patterns (while often overkill) can give you a good sense for some common ways people try to re-use code. For example: - https://en.wikipedia.org/wiki/Design_Patterns#Patterns_by_Ty...

It wouldn't hurt to create some diagrams describing your current process. Then try to identify what parts are dependent on other parts. Also, try to encapsulate some of the pieces so you can separate concerns. This should start you down the path...

It seems like you have a grasp of what is wrong already. Stuff you can do right now without reading a book - Use source control - Enact a policy where all new code is to be well tested (which you can enforce with some build/checkin system) - Make a refactoring change to isolate sections of code that minimally touch other sections and separate them into projects

As for books, I would suggest looking at Head First Design Patterns by Freeman and Robson. This isn't explicitly about project design, but it is about designing reusable code modules which can really help in any language. It is aimed toward object-oriented languages, (the examples are in Java) but it has helped organize my code in other languages as well. If you're using the OO facilities of Matlab, VB and python this can be useful.

There are distinct schools of thought. "How to structure a program" is subjective but also divisive!

Here are the major ones (that I'm aware of), making an effort to be inclusive:

1. Functional programming. It's possible I'm doing a "retcon" on several models here, but these are a fair start:

1.1. http://www.erlang.se/doc/programming_rules.shtml

1.2. https://www.seas.upenn.edu/~cis341/current/programming_style...

2. C (the language) organization. Embedded, linux kernel, and many open source projects follow this pattern. O'Reilly books are a good start.

3. Java. There are so many competing strategies that the "no true scotsman" fallacy applies. Here's one that seems uncontroversial: https://www.udemy.com/java-design-patterns-tutorial/

Have a read of Clean Code and the Pragmatic Programmer.

I’ve seen this happen before in teams that code all day but don’t see themselves as developers. You need a instigate a cultural change at work where it becomes unacceptable to not apply modern development practices just because the team sees themselves as “quants” first.

I keep writing small programs and each time, I'm finding more efficient ways to write it or learning new things. As I develop new web apps, its like my mind has evolved to say, "I can do this more efficiently and with less code."

Definitely have come along way from the first web app I wrote. Can't say it was not without a lot of writing echo 'test'; to see exactly where the error was, though of course, most frameworks tell you exactly where the error is now, but it helps with testing and debugging.

I do a lot more brainstorming in the beginning, designing of the web app, and then actual coding. It's really just practice. Learning how to keep your code organized and understandable for yourself. Write as if someone is looking over your code and isn't as smart as you :P

SICP

I read it in my late teens, and it's what made things finally click re: structuring programs. (This was back in the 90s, when it was far more difficult to inspect other well-structured software that wasn't also low-level C code.)

https://en.wikipedia.org/wiki/Structure_and_Interpretation_o...

Familiarity with design patterns in general can help with structuring systems.

Look at structural and behavioral patterns on https://en.m.wikipedia.org/wiki/Software_design_pattern

Facade, strategy , adapter, etc.

Also think about "levelization" John lakos explains that clearly.

I've found that Matlab is good at prototyping but poor in structuring into large applications with complex physical structure. I'm sure things have changed over the years but the sort of folder oriented loading of modules from the working directory baked habits into people that are hard to break I imagine. Give Large Scale C++ Software Design by Lakos a read to learn about the issues involved in ramping up tall dependency chains. A lot of the book is naturally C++ oriented and thus focuses on impact to compile and link time, but the dependency concepts live on in other languages. Find a CASE tool that works for you https://en.wikipedia.org/wiki/Computer-aided_software_engine.... Visualizing codebases with auto generated diagrams from something like CppDepend or any of its cousins can enlighten a lot. But getting masterful at these subjects is really what Software Architecture is all about. There was a fella by the name of Juval Lowy peddling some architecture training/certification wizardry. Read up on Martin Fowler too, he has good subject matter in this area.

EDIT: But for your immediate Matlab concerns I remember some StackOverflow questions about structuring large matlab applications. I just learned from one that Matlab has this notion of packages I was previously unaware of: https://stackoverflow.com/questions/2748302/what-is-the-clos... https://stackoverflow.com/questions/27861304/managing-bigger... https://stackoverflow.com/questions/2326609/how-to-visualize...

you may search StackOverflow for "matlab packaging" "matlab organization" "matlab structure" etc to get likewise hits.

I suggest reading the book "Clean Code". I've worked with traders/business analysts who do some coding but it isn't their full time job and their code quality isn't great.

After having them read the book and then following up with code reviews from a senior developer they improved their code significantly in just a few months.

"Refactoring" is also a good book, but a lot of time it is too advanced for people who copy and paste code all over and don't use functions, etc.

IMO poor structure of a codebase is a sign that there is a culture problem. You acknowledge that there is an issue but have dedicated no resources to solve it. The way to fix sloppiness is to break your bad habits for new code, and then fix the issues with the old code over time.

My recommendation would be to begin adding tests to your code, keep files sizes manageable (under 120 lines), keep functions under 7 lines. In general just be a professional instead of running around trying to fight fires all the time. Luckily 15k lines of code is a pretty small project so you aren't completely screwed.

I work on a machine learning team that has gone through the exact same process. As others have noted, the biggest change for us was cultural.

If you send me a message I would be happy to chat about it.

Recommended resources to consider:

* Code Complete https://www.amazon.com/Code-Complete-Practical-Handbook-Cons...

* Programming Pearls https://www.amazon.com/Programming-Pearls-2nd-Jon-Bentley/dp...

* Pragmatic Programmer https://www.amazon.com/Pragmatic-Programmer-Journeyman-Maste...

I will reiterate some of what others have said:

- Use version control!

- Hire a professional software developer

- Good architecture and abstraction comes mostly from practice but if you really want to do some reading on software architecture check out "The Architecture of Open Source Applications" (vol 1 & vol 2)

http://www.aosabook.org/en/intro1.html

As a baseline recommendation try this: turn some of the apparently unfactorable pasta code into a large inline function and then start factoring from that point. Usually an opportunity will appear to realign the code in a new way that gets you something you didn't have before. Repeat a few times and you will end up with some original code abstractions that you never would have seen without a careful evolutionary process.

You learn it by writing out of hand code :) But more on the point, what has helped me immensively has been writing out my thought processes before starting to code. Write out the problem, and solve it in a journal or somewhere before just starting to write out code and go along.

There's nothing wrong with just writing code as you go and solving things while you find problems, but to get structured, documenting the behaviour of the program is a major point. Draw images how the code should work. What is connected to what, who owns what ?

By doing this process, you start to see the big picture. It's not always easy, and the tempation to just "go at it" is always there, but this easily leads to code you need to re-factor or re-think more than you would like.

But of course, sometimes you just have to write the crap version first, and then move on to a more advanced model. Actually, this is the expected way, as coding is a field where you just can't know what lies in that valley before venturing into it. So, accepting the fact that re-factoring is something you do constantly is also key. If you don't refactor, the old crappy solutions hinder and slow you down.

Experience of course helps you make better decisions in new projects.

Do searches that specifically address what you are trying to do and learn from the resulting blog posts....

https://www.google.com.au/search?q=how+to+structure+matlab+c...

Each time you encounter a new concept, search for that too.

Search for "beautiful matlab code"

Learn from others. Aggressively search for open source matlab code bases and read them.. look at their structure and try to understand why they are doing things the way they do... email the authors and ask them questions.

IMO, the two easy things:

- read other similar projects and understand how / why they work

- fix problems that you see.

If there are packages you're using from other systems that's often a very easy place to start with the first one... follow through all that code and see how it's organized.

With the second one, I'll say that it's really easy to put up with crappy workflows just because it is what you're used to. You have to not be willing to put up with the things you don't like in order to fix things. Use source control. Don't write functions that are 300 lines long. Think about the times that you have problems in your code and don't let those happen in the future.

As other commenters have said, I think you should consider hiring a software engineer. It's easy to get overwhelmed by tools and minor details.

It's difficult to give you advice, since you've asked such a general question. I've written some thoughts on the matter, but take it all with a grain of salt. At best, my comments are spherical cows [0].

There's nothing wrong with having everything in a single folder; it really depends on the project. If you find it difficult to navigate your way around the source, I'd say there's two high-level approaches for organizing application code: you either group things by feature or by functionality. For example, many traditional web frameworks group source by functionality, so you have a folder for all models, controllers, views, configs, etc.

When structuring application components, you'll typically want to separate things into pure modules and anything which produces side-effects. Pure modules are typically easier to test and debug. If a module depends on anything, consider passing in any parameters during startup or when calling the module, rather than having it look things up on its own. That's the basic idea behind dependency injection.

Git is pretty much the standard for version control. You can get pretty far by just memorizing a few commands [1] and throwing up your code on Bitbucket, GitHub, or GitLab. To make life easier on yourself and your teammates, I'd suggest using a GUI tool like SourceTree. It makes it much harder to shoot yourself in the foot, while providing a far more approachable and discoverable interface. I'm not the biggest fan of git, but since it's what everyone else uses, you pretty much have to suck it up.

Since I'm not familiarized with the ecosystem I can't give much advice with regards to testing. The closest thing to a standard testing interface that I'm aware of is TAP [2]. I'd highly suggest checking out "An introduction to property based testing" [3]; it's one of the most insightful testing-related talks I've seen.

[0] https://en.wikipedia.org/wiki/Spherical_cow

[1] https://xkcd.com/1597/

[2] https://en.wikipedia.org/wiki/Test_Anything_Protocol

[3] https://fsharpforfunandprofit.com/pbt/

> How do you learn to structure a large program?

You can try to organize you stuff by: - layers (time) - functional units (what it does) - subsystems (parts assigned to people)

Over that, you can write a state machine to sequence the jobs, add tracing and isolate parameters.

The idea is to group, organize and classify iteratively.

Your project looks interesting.

Feel free to reach me @gmail.

Practice, practice, practice, practice, practice, practice ... Reading a book to learn modularisation is like reading a book to learn how to play piano.