Ask HN: Right technology for a high performance commodity trading platform

1 points by startupq 11y ago ↗ HN

Hello, I am tinkering with a start-up idea for developing a high performance ETRM (energy/commodity trading and risk) platform similar to Allegro (www.allegrodev.com) and Eka Plus (https://www.ekaplus.com/) based on open source technologies. The idea is to combine energy and commodity trading processes with high performance/in-memory cluster computing and speed up risk analysis considerably.

Not sure how trading savvy people on HN are, but I am interested in knowing what kind of tech you would recommend for such a platform? There are different approaches for such an architecture:

1. In-memory analytics based on Apache Spark 2. High-performance multi-core C/C++ GPGPU based on CUDA 3. FPGA systems as used for high frequence trading systems 4. Other in-memory database systems with full ACID compliance (e.g. VoltDB)

Keen on your thoughts, industry experience, and general viability of this idea.

1 comment

[ 3.3 ms ] story [ 15.2 ms ] thread

jalateras 11y ago ↗

I would break the problem up into capture, processing and visualization. For capture something I would look at using an Apache Kafka cluster. For processing either Spark or Storm or maybe even Samza (although i haven't used it) seem like goos options. Spark probably has the best community but Storm has been around for several years now. For visualization i'm leaning towards something like React with Flux and then D3/Chart libraries.

Deploy on EC2 because they seem to offer the best availability in town (i've used several providers and they have been the best by far)

Use Ansible to automate the infrastructure and away you go.

Not sure whether you need to store the time series data but you could look at ElasticSearch or Cassandra (columnar database), which have great performance. Maybe even Crate, which also builds on ES. The size and frequency of the data will dictate the best store.

Here is a great list of open source and commercial resources all in one place http://www.bigdata-careers.com/?page_id=99