Ask HN: I am a data analyst and my code is a mess

The rule of thumb that most people stick with when doing OOP is duplicate code is bad.

The goal is to find data that needs to be grouped, and group it. Find functions that only use that grouped data, and stick them in classes.

For example a query can be an object. I.E.: A database connection (in java)

       public class DBconnect
       {
               private connection Con = null;

               public DBconnect(String Ip, int port)
               {
                        this.connection = mkConnection(ip, port);
               }

                public Object query(String query)
                {
                          return this.connection.ExecQuery(query);
                }
         }

Then you query specific pre-processing code can be added directly into the query.

                 public String query(String query, String regex)
                 {
                         return this.connection.ExecQuery(query).replaceAll(regex, "");
                 }

Which results in code like

                  DBConnect db = new DBConnect(127.0.0.1, 150);
                  String[] quereies = { "yada", "yada", yada"};
                  for(String str: queries)
                  {
                         String result = db.query(str, "\\s+");
                         doDataScience(result);
                  }

I don't know if this helps. But its a suggestion.

P.S.: I've been spending my free nights the past 2 weeks trying to throw together a javascript based data processing engine in java. It should be mostly workable by the weekend. I could throw it on a ShowHN if you'd be interested.

not a python dev, but:

- python probably has a lib like underscore (reduce map filter etc), could help

- check out the quake source code, any version, its huge and the entire thing is not only readable but possibley a work of art.

- have you tried lambdas? to some its more readable.. ex:

    nums = range(2,50)
    for i in range(2, 8):
        nums = filter(lambda x: x == i or x % i, nums)

personally when i have too complex process i like to go more functional, ex:

    main:
        prepare_data1()
        prepare_data2()
        do_long_stuff()
        nextstep()

that allows me to focus on only on building one step and still have readable code.

many game-devs prefer breaking their project into many tiny files with a specific purpose instead of spaghetti, ex:

    file.py
    parser.py
    display.py
    function1.py
    function2.py

its also a bit easier to nav around the project and make sense of it this way. you might want to check out rust or D or F or another lang also.

Has somebody reviewed your code and called it spaghetti, or is it your own opinion?

If it's your own opinion, then it's possible you're being unduly harsh on your own work. Perhaps you can publish it - or a suitable equivalent - on github and request people here for code reviews.

Hey Elliott It might be worth checking out Warewolf ESB - it's a visual programming platform with flow-based programming principles. It's primarily a service bus, but for your needs it will really help you move away from the "spaghetti code" and into a more modular, visual application. It's open source and free:

Compiled version: http://warewolf.io Source code from GitHub: https://github.com/Warewolf-ESB/Warewolf-ESB

We are building Sclera, an extensible SQL engine that enables you to push your analytics operations into a SQL query. The idea is to tame the code complexity through a declarative interface to analytics libraries. You can add your own libraries using the Sclera Extensions SDK. http://www.scleradb.com/doc/sdk/sdkintro

From the FAQ: http://www.scleradb.com/doc/info/faq#i-am-an-analytics-consu... why-do-i-need-sclera > Specifically, Sclera separates the analytics logic from the processing and data access. The analytics logic is specified declaratively as SQL queries with Sclera’s analytics extensions. This is just a few lines of code, which can be changed easily. The analytics libraries, database systems and external data sources form their own modules and are separated from the analytics logic. The analytics queries are compiled by Sclera into optimized workflows that dynamically tie everything together.