HN Explorer

Ask HN: Train ChatGPT on a large set of functional analysis documents

2 points by Pamar ↗ HN
I work as analyst on a legacy (~30 years old) app for a largish company.

The quality of documentation is exactly what you could expect from my first sentence.

We do not have much in terms of actual technical documents, because, hey, you can always open the source code in your editor and try to understand what the hell is going on.

What we do have, though, are a few thousands word doc documents that were used to by analysts to discuss features with the business users, and try to make it unambiguous enough to pass to the developers.

Quality is not that great here, either, but I just wonder if anyone had some experience with a similar situation.

What I hope to accomplish is being able to ask ChatGPT (or any other LLM system, I just mention ChatGPT as a shorthand here) questions like:

- What can you tell me about Feature X? There is a hardcoded limit of 3 X and I need to understand the implications if we remove it or make it a user-set parameter.

- What was purpose of Field XY on table WZ? It looks like it has been set to null for the last 5 years.

- Configuration parameter X is a bit obscure to me. What is it used for?

- Function X uses pessimistic locking. What should we change to make it use optimistic locking instead?

- We need to revamp feature Y due to new laws governing that part of the process. Specifically, we are now required to send an email describing the changes whenever FOO status becomes BAR. Can you suggest how to proceed with it, based on similar functionalities already implemented in the past?

As a sort of bonus question, I suppose we can also provide the source code for all the features described in the functional analysis (in the source code there are always comments defining which lines of code were added/removed/modified with the id of the analysis document requesting it) but unless it is absolutely vital I'd prefer to start by mining the human readable documents first.

[I am trying to create a business case for a pilot project on this space, so I would like to know if anyone has attempted this already, or if it is not even worth trying].

8 comments

[ 4.1 ms ] story [ 31.6 ms ] thread
jakubsobolewski
I’ve seen a few tools trying to tackle this kind of documentation challenge, but it’s definitely tricky with legacy systems and scattered docs. I wonder if anyone has found a solid way to handle these kinds of deep, context-heavy questions?
_mrmz_
I used LLMs to build a graph knowledge base about given system. This was done based on reverse-engineering the code and indexing existing documentation. The results were good enough that we are now working on onboarding a huuuuge monolith in largish company.
janushwalker
I saw a startup doing exactly that: https://modernaizeapp.com/
Pamar
Thanks, I have looked into this but it seems to be more like a way to address support tickets, i.e. "an expert system to support you in resolving tickets".

What I am aiming for is more like "an expert system to support you in working as an Analyst in charge of evolutive maintenance for an old and complicated app".

_mrmz_
isn't that the similar thing? When system supports you in incidents solving it has capabilities to provide support for analyst who maintain the system, help onboarding new team members to work with the legacy app or give support to the modernization team (architects, devs etc.)? If the assistant can learn from interaction with team members and store it in KB to distribute it - that would be nice feature. Unless you have smth else in mind when speaking about "in charge of evolutive maintenance for an old and complicated app".
Pamar
They seem to put most of the focus on analysis of logs and bug tickets.

Surely valuable but not what I am looking for.

It is still mostly a gedankexperiment at the moment but ... you can nowadays ask an AI to draw you something "in the style of Jack Kirby" or whoever, just by providing a textual description.

I would like to say "I need to add this feature to my system, this are the constraints/assumptions and these are the requirements. Write me an analysis document in the style of the 4000+ documents I trained you with".

I have the skill/experience to use its draft and put it in a usable form, and so do my colleagues, but I hope it will make us more efficient, discover old forgotten stuff that could be revamped, and help onboarding new people.