Databot: High-performance Python data-driven programming framework (github.com)

I'm still looking for a perfect natural python ETL dsl, so I will follow that project.

So far I'm using https://github.com/petl-developers/petl and mostly happy with it.

https://www.coursera.org/lecture/software-architecture/3-2-7...

However, one downside is the massive RAM consumption. 1GB of RAM even if it does pretty much nothing is quite a bill to start off with.

1: https://nifi.apache.org/

So here's an implementation of the example from the readme using make, and bash, and jq, and a silly python script to implement a timer by modifying a file every X seconds:

  ~/projects/makething$ cat Makefile 

  default: d

  b:  a
      curl -o $@ "http://api.coindesk.com/v1/bpi/currentprice.json"

  c:  b
      jq '.bpi.USD.rate_float' $< > $@

  d:  c
      cat $<
      cp $< $@


  ~/projects/makething$ cat timer.py 

  import sys
  import time

  def main():
      delay = float(sys.argv[1])
      fn = sys.argv[2]
      while True:
          with open(fn, 'w') as f:
              f.write(str(time.time()))
          time.sleep(delay)

  if __name__ == '__main__':
      main()


  ~/projects/makething$ cat go.sh 

  #! /usr/bin/env bash

  python timer.py 2.0 a &
  TIMER_PID=$!

  function cleanup() {
      kill $TIMER_PID
  }
  trap cleanup EXIT
  
  while true
  do
      while make -j -q
      do
          sleep 0.1
      done
      make -j
  done