We have been working on that for the past few month. Right now the performance is pretty good for C++ projects up to around 2 million lines of code. For C projects the performance is even better! If you want to test it, just download the pre-indexed clang project (900k loc) from our website (https://www.sourcetrail.com/downloads#extra). Indexing the project took 12 minutes on a 5 year old laptop.
There is also a tool called Bear which will generate compile commands from Make-based builds, in case you were wondering. I've used it with some success.
Im well aware of bear. Works fine in some cases, not so well in others. Author is also doing scan-build replacement in python which imho is far superior to bear.
A Compile Command is a JSON file entry that specifies what the C/C++ compiler should do when compiling a file. So it defines a file to compile and all the required flags (like include paths or preprocessor defines). These Compile Commands are used for all kind of Clang based tools. You can read more here: https://clang.llvm.org/docs/JSONCompilationDatabase.html
Kotlin is definitely on the shortlist for the next language to be supported! Which language we'll pick up exactly depends on the landscape of available tools for that language. Kotlin is pretty new so I guess that there could be a need for Sourcetrail :)
If you ever go that route and you plan or using/creating some open-source components I would be very interested on hearing more (I am a JavaParser/JavaSymbolSolver contributor)
What languages are supported? At least on mobile, there is zero mention of language support outside of testimonials and screenshots. Is it just Java and C++?
Unfortunately it doesn't. Even though Scala and Java output the same kind of bytecode, these languages have different language specifications. We currently only support what's in the JLS.
Guess you could have some assertions together with the annotations to make sure things are up-to-date. Then integrate those tests with your CI of choice and it should be a bit easier to keep up-to-date.
As noted by others the annotations would be active checks. More like proofs in coq than static text. That way you'd be building up a logical understanding of the code base that was machine verifiable sidestepping the issue of stale comments.
If done properly this could even feed into code dynamics to capture information on the runtime information. So that at every point in the program you could start asking questions like how many times did this line execute, what was the type of this variable at this point, how much memory was allocated, and so on and so forth.
Seems like youve just abstracted things like static typing into a extension of the language thats checked at a less optimal time. At this point arent you better off just using a language that incorporates this functionality and compiles it? Sounds like youve just implemented static typing but in a really akward and unecessary way
To be honest: We have not decided what language we will pick up next. Our first priority is to improve on our Java coverage. In a few months we should be ready to choose our next language. We are thinking about either tackling Python or implement support for a more recent language like Kotlin, Swift or Go.
I spend a lot of time in clients' ios apps which were developed by other contractors long ago and then I'm asked to fix bugs and add new features. I think this would help me save a lot of time. Since it already supports C/C++ would objective-c support be relatively straightforward to add? Not sure if there are many others who would like to see it, but I know I would definitely appreciate it.
A Lisp might be an easy language to target, because of its consistent syntax and "code is data" philosophy. With Clojure, you could both reuse some of your work on Java, and support Clojure's Java interop. Attempting to support Clojure would drive improvements in the Java support.
Seems to me there is probably a huge amount of legacy Python code around that people need to explore. In fact, I've spent the entire day so far creating some so that we have enough legacy Python code around in a couple more years. Wouldn't want to run short.
Seems to me like the obvious choices would be languages that have large, long standing legacy codebases. Swift code is going to be relatively tight just because it's so recent, whereas something like C# isn't.
I might buy this for Go if it was available, this looks interesting!
Note that I'm mainly interested because my current editor of choice does not have the full suite of tools I'm used to for Go. I used to use Vim, and vim-go has such an excellent suite of tools I don't think I'd buy this. However, I've since switched to Kakoune, and the tool support is quite young and I've not had time to improve it myself. So this tool (at the $100 self employment pricing) is a nice fit for a tool I'm on the fence about. At $200 I don't think I would, but at $100 I think I would.
Anyway, just providing a bit of feedback. Looks really nice!
Code Maps requires Enterprise, I think. You can view generated Code Maps from other editions, but you need Enterprise in order to generate them. Kind of unfortunate.
I really want something like this for heterogenous codebases. Maybe along the lines of ollydbg and ida but for "reverse" engineering Ruby, python, and go.
Happy to hear that you like it! Full support of Python won't be easy but maybe we could tackle this one in stages. Implement highly demanded features in a first Python release and improve on that later on.
I used Sourcetrail a few months ago to figure out a ugly ball of embedded C code that I needed to reimpliment. Sourcetrail was super helpful for this. It paid for itself in an afternoon.
Yes, Sourcetrail also shows class internal calls and accessed fields. Code Maps in VS has a different approach because they try to keep the graph small by making the user add symbols. So if you are trying to see every access of a symbol it is easy to miss something. Sourcetrail on the other hand keeps the graph small and comprehensible by showing only one level of dependencies. You don't need to add anything yourself. So you cannot miss any relation ;)
Thanks. Sounds more promising than I took from you landing page. I will give it a go on my c/c++ code bases. See if it can help me.
I now notice the free trial cannot be run on your own project. I have to say I won't buy anything that I didn't try on a real sample before. What made you decide to go for this option instead of regular limited functionality or time bound (my preference).
CodeLens is great for finding references within the IDE. It also provides information about the history, which Sourcetrail doesn't do yet.
I would say we go a step further, by also providing a visual representation of the references to symbol, which quickly conveys how a symbol is embedded into the rest of the code base, without even reading a line of code. That makes exploring a lot faster.
Hm does anyone know of a web-based tool like this that uses Clang as a back end? I want to put open source code like the bash and Python interpreters in it and be able to navigate symbolically. And I want nice permalinks.
I just spent a few minutes clicking around but had a bit of trouble.
EDIT: nevermind, most of this is due to the fact that you have to keep "repo" in the search box, which I found confusing.
I quickly found the repo I wanted, but discovered that C/C++ isn't supported, so Woboq is more useful to me at the moment.
I tried to build something like this once myself, and made the mistake of going "wide" rather than "deep". That is, trying to support all languages. I think it does make sense to try to get the absolute best experience for one language. But yeah for now since C/C++ aren't supported, it's not useful for me.
------
I typed "cpython" in the search box on https://sourcegraph.com/ and got no results. (I also typed "python" and got no results, which seems pretty odd.)
I guess you have to import repositories yourself? There seem to be sample repositories, but they are in Go, and I'm interested in C/C++ (the Python interpreter, not Python source code).
I'm looking for something that uses Clang as a back end for intelligent navigation of C/C++ projects.
Also, I would like to browse the bash source code, but as far as I know it isn't available on Github. Can SourceGraph import from an arbitrary git repo?
We decided to leave out editing early in the process. Most developers already have their favorite editor and it's a lot of work to implement all features that users would expect from it.
We decided to focus on source exploration instead. Sourcetrail is made for the second screen. We implemented plugins for the most used source editors to enable easy switching. Please have a look at our documentation: https://www.sourcetrail.com/documentation/#CODEEDITORPLUGINS
Website feedback: Supported languages is the #1 most important piece of info about this software. It should be front and center and probably repeated a few times already on the landing page.
This may be a result of your feedback, but for me looking just now it it front and center: "Sourcetrail - A cross platform source explorer for C/C++ and Java"
Although physically front and center, the fact that that line is nearly the smallest thing on the page caused me to gloss over it entirely my first time through.
it should be in a bigger font, and it seems like it's the only place where language compatibility is clearly visible without going into the documentation.
Just a feature request if it isn't already in the works:
Runtime analysis of code that actually matters ie live code.
Often its hard to figure out what in the code base actually matters.
Debugging is still sort of difficult and dead code is really hard to find and get rid of. Particularly if reflection is used heavily which it almost always is these days.
Otherwise (IMO) as a Java developer IntelliJ and Eclipse are actually not that far off in their static analysis and already pretty good to discover the code. Especially IntelliJ as I can easily see them enhancing their code view/discovery as well as they already have static dead code analysis.
I guess I don't think discovering code bases is that much of a problem compared to understand what is actually running.
That is I think debugging particularly multithreaded applications is vastly more difficult problem. Thus runtime analysis would be very useful.
Thank you for your input! I wouldn't call it a feature request though, I'd say it's another tool. But, you are right, exploration of dynamicly analyzed code would also greatly benefit understanding.
This looks like an exceptionally useful tool! Thanks for making and sharing it. The video is excellent.
It reminds me of a feature I've always wanted, which would be a code unravel-er. Basically I'd like to be able to select a line anywhere in a codebase, and have said codebase expand to show me the entire execution path as if it were in one single scope, with the ability to re-collapse sections back into loops and functions while reading.
I think it would help to better understand the codepaths that can start to get fuzzy when jumping between abstractions. It would be great - not only for understanding unfamiliar code - but also for double-checking logic across longer code paths.
At any rate - great idea, great implementation, and beautiful website.
There's also Source Insight [1] which supports C/C++/Java/Obj-C - quite a fantastic tool, but unfortunately it's Windows only (Wine works with a few hiccups)
I am playing around with an idea like this for editing Clojure in Emacs with cider-mode. You have to manually build the 'thread' that such a code unravel-er would return, but then you can read and edit functions from many files in one buffer. You start with the highest-level function you are interested in, and then pull the definitions of functions it calls into view as needed.
Yes, but not the kind of visualization and calling stack shown by SourceTrial. Some of the code is 10 layers deep where the stuff actually happens with many complicated usage of inheritance, which I figure is where this kind of tool can shine.
118 comments
[ 2.6 ms ] story [ 173 ms ] threadNorth of 500k lines, few thousand cpp/h files, 40+mb
Thanks
https://github.com/rizsotto/Bear
If done properly this could even feed into code dynamics to capture information on the runtime information. So that at every point in the program you could start asking questions like how many times did this line execute, what was the type of this variable at this point, how much memory was allocated, and so on and so forth.
Note that I'm mainly interested because my current editor of choice does not have the full suite of tools I'm used to for Go. I used to use Vim, and vim-go has such an excellent suite of tools I don't think I'd buy this. However, I've since switched to Kakoune, and the tool support is quite young and I've not had time to improve it myself. So this tool (at the $100 self employment pricing) is a nice fit for a tool I'm on the fence about. At $200 I don't think I would, but at $100 I think I would.
Anyway, just providing a bit of feedback. Looks really nice!
https://www.exakat.io/up-and-running-with-exakat/
https://msdn.microsoft.com/en-us/library/jj739835.aspx
Seems they cover a similar thing.
I now notice the free trial cannot be run on your own project. I have to say I won't buy anything that I didn't try on a real sample before. What made you decide to go for this option instead of regular limited functionality or time bound (my preference).
CodeLens => https://msdn.microsoft.com/en-us/library/dn269218.aspx
I would say we go a step further, by also providing a visual representation of the references to symbol, which quickly conveys how a symbol is embedded into the rest of the code base, without even reading a line of code. That makes exploring a lot faster.
CodeMap => https://docs.microsoft.com/en-us/visualstudio/modeling/use-c...
EDIT: It looks like https://woboq.com/codebrowser.html is what I want. The code available but not under and open source license. https://github.com/woboq/woboq_codebrowser/
I will probably try this, but I'm interested in alternatives!
It gives you nice URLs (hit Alt+Y/Opt+Y to get the full commit ID in the URL to make it permanent).
Would love to hear your feedback.
EDIT: nevermind, most of this is due to the fact that you have to keep "repo" in the search box, which I found confusing.
I quickly found the repo I wanted, but discovered that C/C++ isn't supported, so Woboq is more useful to me at the moment.
I tried to build something like this once myself, and made the mistake of going "wide" rather than "deep". That is, trying to support all languages. I think it does make sense to try to get the absolute best experience for one language. But yeah for now since C/C++ aren't supported, it's not useful for me.
------
I typed "cpython" in the search box on https://sourcegraph.com/ and got no results. (I also typed "python" and got no results, which seems pretty odd.)
I would have liked to browse this repo:
https://github.com/python/cpython
I guess you have to import repositories yourself? There seem to be sample repositories, but they are in Go, and I'm interested in C/C++ (the Python interpreter, not Python source code).
I'm looking for something that uses Clang as a back end for intelligent navigation of C/C++ projects.
Also, I would like to browse the bash source code, but as far as I know it isn't available on Github. Can SourceGraph import from an arbitrary git repo?
https://savannah.gnu.org/git/?group=bash
That said I'm glad somebody is working on this problem! I like your mission statement.
That's an interesting link. It only works when you go to "Help -> Language features".
We decided to focus on source exploration instead. Sourcetrail is made for the second screen. We implemented plugins for the most used source editors to enable easy switching. Please have a look at our documentation: https://www.sourcetrail.com/documentation/#CODEEDITORPLUGINS
Nice work; seems like you hit a nice medium between trying out something new and keeping it connected to the real world and practical. :)
I left the site not really knowing what language/s are supported.
After digging through the entire page it looks like maybe C/C++/Java?
There should be a huge button, or something, front, and center. <<SUPPORTED LANGUAGES>>
Looks like a cool idea, but I don't know if I can even utilize this tool.
https://news.ycombinator.com/item?id=9500455
Runtime analysis of code that actually matters ie live code.
Often its hard to figure out what in the code base actually matters.
Debugging is still sort of difficult and dead code is really hard to find and get rid of. Particularly if reflection is used heavily which it almost always is these days.
Otherwise (IMO) as a Java developer IntelliJ and Eclipse are actually not that far off in their static analysis and already pretty good to discover the code. Especially IntelliJ as I can easily see them enhancing their code view/discovery as well as they already have static dead code analysis.
I guess I don't think discovering code bases is that much of a problem compared to understand what is actually running.
That is I think debugging particularly multithreaded applications is vastly more difficult problem. Thus runtime analysis would be very useful.
It reminds me of a feature I've always wanted, which would be a code unravel-er. Basically I'd like to be able to select a line anywhere in a codebase, and have said codebase expand to show me the entire execution path as if it were in one single scope, with the ability to re-collapse sections back into loops and functions while reading.
I think it would help to better understand the codepaths that can start to get fuzzy when jumping between abstractions. It would be great - not only for understanding unfamiliar code - but also for double-checking logic across longer code paths.
At any rate - great idea, great implementation, and beautiful website.
My previous company used Understand on a large unfamiliar codebase, it helped greatly, but was pricey.
[1] https://scitools.com/
[1] https://www.sourceinsight.com/
https://github.com/fazzone/multifiles.el/tree/cider-hacks
This would be particularly useful in open source.