Show HN: VisionScript, abstract programming language for computer vision (github.com)

I have been thinking about this! What would you like to be able to do?

On a separate note, one idea I had was to have rock paper scissors as a quickstart. You could Use["rock paper scissors"] to use that as a model; there could perhaps be a registry of curated, fun models that people can play with out of the box (aka ones good for classroom use).

Shameless plug: take a look to our embedded computer vision library SOD: https://sod.pixlab.io. It's a lightweight OpenCV alternative targeting embedded devices with most of the modern image processing algorithms already implemented including an experimental Stable Diffusion implementation.

Great question! Development has been guided by my asking myself -- and discussing with friends -- what problems I/we would like to solve (or what would be fun!), then building for those use cases. This is evident in the WIP DetectDistinctScenes[] method.

Example script:

    In["file.mov"]
        Classify["building", "garden"]
    Endin

    DetectDistinctFrames[]

This method is more advanced than most. It uses CLIP behind the scenes to detect when classifications change for more than N frames in a video, and generates a list of timestamps you can use to determine scene changes.

I'm keen to build more specific functions for common CV tasks. My code most certainly needs cleaned (the main lang.py is > 2k LOCs right now) but I do have a contributing guide that you can follow to add features should you be interested: https://github.com/capjamesg/visionscript/blob/main/CONTRIBU...

a new, popular library for basic functionality (converting between annotation formats, evaluating models, doing object tracking) has been supervision: https://github.com/roboflow/supervision

Thank you so much! Comments like this have really made my day.

Scratch does have an "Extensions" feature, so perhaps it would be possible (https://github.com/scratchfoundation/scratch-vm/blob/develop...)!

Thank you so much!

I have been thinking about learning progressions with a language like VisionScript. I see a notebook-like environment being a great place to start; dragging and dropping is intuitive (and fun!).

But if that notebook can introduce someone to the same primitives they can use in a text-based programming language? That gives you the ability to help someone graduate to a full programming environment.

Notebook -> Text that uses the same syntax as the notebook (you could drag+drop in the notebook then write out the text as a starting point!) -> Writing your own programs.

A friend suggested that the implicit state approach of VisionScript could be used in other areas too like audio. I reasoned that we could actually use many of the same primitive names and syntax; just a different backend. I don't know enough about audio processing, but the idea is fascinating!

My pleasure! I sincerely appreciate your words! It was a pleasure to be on the front page of HN.

VisionScript goes beyond OpenCV in terms of utilities, providing models for use out of the box.

To classify images, you can use Classify["class 1", "class 2", ...]; to run object detection you can use Detect["object"]; to run segmentation, use Segment["class"]. More abstract methods include Search[], that lets you build a semantic search engine in a few lines of code:

    Load["./folder/"]
    Search["coffee cup"]
    Get[1] # get the top result
    Show[]

I am aiming to encode best practices from computer vision behind the scenes (I work in the industry) to enable one to build powerful applications intuitively, with few lines of code!

  vision = Vision()
  
  (vision
    .load("./folder/")
    .search("coffee cup")
    .get(1)
    .show()
  )