Geniusrise – inference APIs for text, vision, audio, multi-modal AI models (github.com)

Geniusrise is a modular, loosely-coupled AI-microservices framework.

It can be used to perform various tasks, including hosting inference endpoints, performing bulk inference, fine tune etc with open source models or closed source APIs.

- The framework provides structure for modules and operationalizes and orchestrates them. - The modular ecosystem provides a layer of abstraction over the myriad of models, libraries, tools, parameters and optimizations underlying the operationalization of modern AI models.

1. Install geniusrise and libs

    pip install torch
    pip install geniusrise
    pip install geniusrise-vision # vision multi-modal models
    pip install geniusrise-text # text models, LLMs
    pip install geniusrise-audio # audio models

2. Create YAML file

    version: '1'
    
    bolts:
        my_multimodal_api:
            name: VisualQAAPI
            state:
                type: none
            input:
                type: batch
                args:
                    input_folder: ./input
            output:
                type: batch
                args:
                    output_folder: ./output
            method: listen
            args:
                model_name: 'llava-hf/bakLlava-v1-hf'
                model_class: 'LlavaForConditionalGeneration'
                processor_class: 'AutoProcessor'
                device_map: 'cuda:0'
                use_cuda: True
                precision: 'bfloat16'
                quantization: 0
                max_memory: None
                torchscript: False
                compile: False
                flash_attention: False
                better_transformers: False
                endpoint: '*'
                port: 3000
                cors_domain: 'http://localhost:3000'
                username: 'user'
                password: 'password'

3. Launch API service

    genius rise


    MY_IMAGE=/path/to/test/image

    (base64 -w 0 $MY_IMAGE | awk '{print "{\"image_base64\": \""$0"\", \"question\": \"<image>\nUSER: Whats the content of the image?\nASSISTANT:\", \"do_sample\": false, \"max_new_tokens\": 128}"}' > /tmp/image_payload.json)
    curl -X POST http://localhost:3000/api/v1/answer_question \
        -H "Content-Type: application/json" \
        -u user:password \
        -d @/tmp/image_payload.json | jq

More: https://docs.geniusrise.ai/guides/usage/ and https://github.com/geniusrise/examples