Flowable is an open-source (Apache-2.0) BPMN, CMMN and DMN engine in Java — embed the process, case and rule engines in a JVM app or run them as a service.
Yes, Flowable is open source under the Apache-2.0 license.
Flowable is free to use.
Yes, Flowable can be self-hosted (the source is available under the Apache-2.0 license).
Flowable is a compact, open-source (Apache-2.0) BPM platform for developers: a fast, tried-and-tested BPMN process engine, CMMN case engine, and DMN rule engine written in Java. Runs embedded in a JVM app or as a server/cluster/cloud service. Apache-2.0 open source with a committed community edition; full documentation at flowable.org.