The Integra library

The Instance host is responsible for instantiating modules in a target environment, and maintaining their state in memory. Instantiation is handled via a target-specific bridge (see below). The instance host can operate in master or slave mode. A master host node acts as an OSC server, and receives module state notifications and other commands via the bridge. A master instance host must be linked to the serialization component at compile time, so it can load module definitions and save module state to persistent storage.

A slave host acts like a proxy, passing requests between its master node, and a module host. It can instantiate modules via function calls to the bridge, but it does not store module state.

Further details about communication between the instance host and the serialisation layer can be found here. Details about the OSC commands supported by the instance host see here

The instance host provides the following functions:

/* Add a node to the beginning of the instance list, and populate with def */
Instance *instance_new(Instance *list_node, Definition *definition);
 
/* Set the value of an attribute in an instance */
void instance_set_attribute_value(Attribute *list_node, value_t value);
 
/* Free an instance */
int instance_free(Instance *list_node);

The module host is any 'environment' capable of hosting Integra modules. This would typically be a modular DSP environment such as Max/MSP, Pure Data or SuperCollider

The purpose of the serialisation layer is to facilitate persistent storage of in-memory module state. If this functionality is required, the program wishing to make use of it must be linked (statically or dynamically) to the serialisation component at compile time. This serialization facility is used by the instance host and also on the database server. On the database server, the functionality of the serialisation component is provided to the python-based web interface using bindings generated by SWIG.

The serialization library and the instance host share a common header file, which defines the data structures for storing module definitions and module instance data. The serialisation component then fulfills the following tasks:

int ntg_xml_deserialize(ntg_definition *definition, ntg_xml_doc *xml_doc_struct)

• where definition is a pointer to the first element in a linked list of ntg_defintion data structs and
• xml_doc_struct is a pointer to the serialized (i.e. XML representation) version of the module definition.

int ntg_xml_collection_deserialize(ntg_collection *new_collection, ntg_xml_doc *ixd_collection)

• where new_collection is a pointer to a initialized ntg_collection data struct and
• ixd_collection is a pointer to the serialized (i.e. XML representation) version of the collection.

int ntg_xml_serialize(ntg_xml_doc *doc, ntg_definition *def)

• gives a IXD representation in doc of module defintion def.

int ntg_xml_document_save(ntg_xml_doc *doc, const char *path)

• writes the IXD representation doc to file path/<module name>.ixd.

int ntg_definition_free(ntg_definition *list_node)

• unlink a definition from the linked list.

The serialization layer can optionally provide the functionality to save to a remote host via XML-RPC, in which case it provides:

int xml_rpc_load_definitions(Definitions **definitions, char *url)

etc.

The the bridge is a binary shared object that gets loaded by the Instance host at runtime. Functionality provided by the bridge is accessed via a 'handle' returned by dlopen(), a subsequent call to dlsym() should return a data struct containing pointers to functions conforming to the following prototypes:

int (*module_load)(Instance *instance, char *path)

int (*module_remove)(int id)

int (*module_connect)(Port *target, Port *source)

int (*module_disconnect)(Port *target, Port *source)

int (*bridge_init)(void)

The return values indicate the success or failure of the function. The API will make no assumptions about the nature of the target environment. For example, graphical positioning of modules must be handled by the bridge.

The purpose of the bridge is to load modules in a target environment, and connect, disconnect and remove them as required.

The Integra Common API provides declarations for the data structures that are used by functions in both the serialization component and the instance host.

typedef struct Port_ {
 
    int moduleId;
    index_t portIndex;
 
} Port;
 
typedef struct Class_ {
 
    int id;
    void (*set_id)(void *self); /* argh! Instance not declared yet */
 
} Class;
 
typedef struct ClassDefinition_ {
 
    Class *parent;
    char *name;
    int parent_id;
    int *aggregatedClassIds;
    char *description;
    int documentationID;
    int *tagCodes;
    char **attributes;
    char **descriptions;
    int *attributeDocumentationIds;
    int *attributeTypeCodes;
 
} ClassDefinition;
 
typedef struct ModuleDefinition_ {
 
    ClassDefinition *parent;
    int *attributeUnitCodes;
    double *attributeMinima;
    double *attributeMaxima;
    value_t *attributeDefaults; /* array of unions */
    char **attributeExamples;
    int **subAttributeCodes;
    int isCore;
 
} ModuleDefinition;
 
/* Linked list node for definitions. */
typedef struct Definition_ {
 
    int type;
 
    union d_ {
        ModuleDefinition *module;
        ClassDefinition *Class;
    } d;
    struct Definition_ *next;
 
} Definition;
 
/* Linked list node for (instance) attributes */
typedef struct Attribute_ {
 
    char *name;
    void *next;
 
    index_t index;
    value_t value;
 
} Attribute;
 
/* Linked list node for instances */
typedef struct Instance_ {
 
    char *name;
    void *next;
 
    Class *parent;
    int type;
    char *className;
 
    Attribute *attributes;
 
    /* Array of arrays of destination ports.  Array index gives source port */
    Port **connections; 
 
} Instance;
 
/* type we cast to for searching lists */
 
typedef struct Finder_ {
 
    char *name;
    void *next;
 
} Finder;

Data sharing between the serialization component and the instance host is very simple. Two types of data are shared between the two components in memory: definition structs and instance structs.

Definitions are data structs of type Definition. They can only be written to by the serialization component. Typically the data for Definitions are provided by XML files. The serialization component can read definitions from these files, but it cannot write new definitions. This can only be done via the web interface to the Integra database.

Typically pointers to structs of type Definition will be stored in an linked list that is constructed when the serialization component reads the definitions from XML.

The procedure for reading definitions will begin with a call to xml_load_definitions() by the instance host. The instance host will pass in a pointer of type Definition, and a pointer to a string representing the absolute path of the Integra gzip containing the XML. The serialization component will proceed to unzip the file, locate the definitions, parse them, allocate memory for the required Definition structs, and then populate the Definition structs with data.

The linked list of structs of type Definition is then available to any application that links to the serialization component. For example the Definition struct maps to an in-memory Python object on the database server.

It is worth noting that the Definition struct can be used for any Integra class, not just Integra modules. It could for example be used for documentation and other class definitions.

Instances are data structs of type Instance. Data for Instances can be provided by XML files, or by the instance host. The serialization component can read instance data from XML files, and it can also write new data to the same files.

Typically pointers to structs of type Instance will be stored in a linked list that is constructed when the serialization component reads instance data from XML. This linked list is then available to the instance host, and the values of the members of each struct in the list are changed in real time to reflect module state as the program runs.

The procedure for reading instance data will begin with a call to xml_load_collection() by the instance host. The instance host will pass in a pointer: Instance *instance, to the root node of a linked list of structs of type Instance, and a pointer to a string representing the absolute path of the Integra gzip containing the XML. The serialization component will proceed to unzip the file locate the instances, parse them, allocate memory for the required instance structs, and populate the Definition structs with data.

The procedure for saving instance data will begin with a call to xml_save_collection() by the instance host. The instance host will pass in a pointer: Instance *instance, to the root node of a linked list of structs of type Instance, and a pointer to a string representing the absolute path of the Integra gzip containing the XML. The serialization component will proceed serialize the instance data, to unzip the file, locate the the insertion point in the XML, write the data as XML, and re-zip the archive.

The linked list of instances Instance *intances is then available to any application that links to the serialization component. For example this struct maps to an in-memory Python object on the database server.

It is worth noting that the Instance struct can be used for any Integra class, not just Integra modules. It could for example be used for documentation and other class instance data.

Because both the serialisation component and the instance host have write access to the instance list, a lock must be used to prevent unexpected behaviour.

In the Integra database it is possible to define data types and then assign Postgresql data values to these types. This is a rather arbitrary facility because the data ultimately gets stored using Postgresql built in data types. However, the purpose of allowing the user to assign arbitrary data types to values to is to facilitate conversion to target data types.

The Integra library provides facilities for storing values using these arbitrary data types, as well as conversion between types. However, it must be noted that values are currently stored in memory using builtin C storage types.

The Integra library uses the following enumeration to list available Integra data types:

enum integra_types {
 
    T_STRING,
    T_FLOAT,
    T_DOUBLE,
    T_INT,
    T_REF,
    UNSPECIFIED /* Deprecated: Use NTG_NONE = -1 */
 
};

These are mapped to C internal types using a data structure to hold the type: value_t.

typedef struct value_t_ {
 
    int type;
 
    union ctype_ {
        char *s;
        float f;
        double d;
        int i;
        void *r;
    } ctype;
 
} value_t;

All values passed between instances are stored as type value_t, so it is never necessary for the programmer to manually access members of this struct. The use of value_t facilitates dynamic type conversions when between source and target data types as data is passed between modules.