Bridging The Gap, Standard Library and Maya

Over the years I’ve found myself needing large enough, or better served, containers than what Maya provides out of the box.
After years of ad hoc hacks or local, thin wrappers I decided to start cooking a more extensive and reliable wrapper/bridge between Maya’s somewhat aging facilities and some of the Standard Library facilities1)C++’s Standard Library.

Having had the capital idea of leaving the Australian summer to spend time home when the weather is finger blackening cold I thought I might as well kill some time, and write about a few bits of interest as I go.
The following is just about equal parts ramblings and some notions and explanations of fundamentals of C++ and how they apply, with a handful of listings to exemplify things. Maybe they’ll be of use to someone, maybe not.

If you know enough Maya and enough C++ to cobble together a plugin, but not quite enough to risk tossing around raw pointers and pointer arithmetic, and intend to use the Standard Library containers, this could be for you.
If you are an intermediate programmer with enough knowledge of C++ and the Standard Library, then this might all be obvious and trivial.
If you are completely and utterly unfamiliar with C++ and/or Maya I’d have to ask how you even bumped into this and why you would have read this far :)

While I do simplify some concepts or operations, or only describe them in their standard paths without mentioning a lot of exceptions and optimizations that might or might not occur, I do so mostly for the sake of trying to make this less overwhelming. I don’t believe anything is overly or dangerously simplified that if you learn something from this you will find yourself having to re-learn it otherwise shortly afterwards if you decide to pursue more in-depth knowledge of the subjects.

 

  Some History.

These are just some musings on how things got to be the way they are. Literary warm up at best; feel free to skip ahead if you don’t care much for historical software speculations.

Back when the bricks of the Maya API were put together, second half of the nineties, C++ was a different beast than it is now.
Containers such as vector made an appearance in the standard only from the ’98 standard and on, and compilers would take a bit longer again to catch up; before then they were only available through libraries provided by SGI (STL from ’97), and HP before then (’94?).
STL, while considered a milestone in programming, was also regarded as a performance affecting choice over plain arrays by some (and to be fair it could be given the resource constraints of the time), so you had a gap between Arrays and more complex containers such as vectors.
Maya addressed this by offering its own type-specific wrappers for arrays that were very similar to Vectors, with many of the same design choices, but it did so taking some distance from templates and more generic containers.
Maya’s own thin wrappers offered comparable speed, but richer functionality and type specificity for Maya’s own types for a small and controllable overhead.

Fast forward sixteen or so years since V1 and Maya still has the same managed arrays you would have been getting by with on an SGI in ’97, while C++11 and up-to-standard compilers offer a large collection of containers both associative and serial, and performance comparable to raw C-Style arrays when possible, not to mention compiler response superior to any alternative that isn’t some extremely custom tailored, narrow solution.

If you think you can live with just Maya’s arrays, you probably didn’t have to do much with them yet.

 

  Some prerequisite notions

If you’re already familiar with Arrays in C++, Maya’s arrays, and pointer basics you can probably skip ahead.
Transmitting the entire amount of knowledge, even if most of it is relatively basic, would be more than this piece sets off to do. The following is mostly meant as a check point, if all of it sounds either familiar or within easy reach from where you stand then you should be cruising for the rest of the article; if not, then you might have to go back and forth between this and some other literature.

In C Arrays are still a fundamental building block, and therefore they receive some love and extensions, in C++, given you have alternatives, not so much; this makes STL containers even more important.
Extended array functionality might be introduced and supported by one compiler, but not another2)e.g. Variable Length Arrays as introduced by C-99 are supported in Gnu’s gcc compiler, but not in Microsoft’s VC. Platform divergence FTL. The C++ committee only sets C compatibility targets with C-89 AFAIK.
C-Style Arrays, simply Arrays from now on, are the simplest form of container you can have in C++. It doesn’t get more bare bones than that, and it just happens to be the only form of non proprietary container Maya supports to interface with its own arrays.
No run time sizing, no dynamic sizing, no coupled values to know the size of the contents and no fancy iterators. Arrays are, almost literally, a pointer to where the data begins and a summary promise that the data will be of constant type throughout and contiguous.

Enter Maya’s Arrays: Alias took generic arrays and built type specific, Maya specific objects around them to give them added functionality and ease of use. Looking at the methods, and fishing through the memory, it’s hard not to think they did take a fair bit of inspiration from STL’s Vectors of the day.
They are functional objects, they can be resized transparently, you can remove, insert or append elements, but they still have reduced overhead and array-like performance; for most you can even construct them and get them from and to C-Style Arrays.

For all their qualities, they are fairly limited by modern standards.
In first place they are equivalent to a subset of Vector; More function rich containers such as Map or Set are absent, and the lack of general utilities in Maya for operations like filtering and indexing makes it so you will eventually need to either dress them up a lot (honestly not worth it), or adopt something more capable and learn to deal with the translation efficiently.
They also aren’t template friendly, so forget meta-programming or polymorphism that doesn’t produce a lot of boilerplate, and relatively old and somewhat opaque even when it comes to functionality they clearly have internally but is, oddly, not exposed.

And lastly, you have the Standard Library and its containers, which these days are a metric ton.
All StdLib containers, within reason and when it applies, are lexically aligned and share as large as possible a set of functionality, which means the learning curve is practically flat once you wrap your head around some basic concepts.
They are all template based, which has the large pro of making them very polymorphic and malleable, at the small but present price of having to be familiar with basic template syntax, and depending on compiler some potentially brutally obscure error messages if they go awry.
Oh, and they are efficient, and very transparent.

Vector, the only one you need to fully supplant Maya’s arrays, is actually not dissimilar from them in its nature; data is guaranteed to be of one type and in a contiguous and discretely spaced sequence and it offers all the functionality you get in Maya’s arrays plus a good deal more 3)in all fairness Maya’s array growth size can be set while Vector’s is implementation dependent and therefore out of your control.

One thing that might be worth pointing out at this point that will be important later, and a similarity between Maya’s arrays and StdLib Vectors that is NOT shared by plain arrays, is that both of them have a preemptively flexible approach to memory.
Plain static Arrays have no overhead, the size in memory is the size of the sum of the contents. Vectors and Maya arrays allocate themselves more  space than just the contained data most of the time. That’s by design, and it’s done so that when you need to add to either, most of the time, unless the additions are considerable, you have good chances that the additional data can simply be plopped at the end of the current sequence and the size/length related members increased accordingly, without superfluous moving around.
This very often spares us from expensive copies when adding to the container, and it also makes removing from the tail end very cheap (except Maya for some reason has no pop() method, but maybe its remove() method on the tail end knows to do the smart thing, I haven’t tested).

If this is all news to you, it might become more obvious when we’ll get to some of the later recipes below.

  Some simple recipes

On to something more practical.
If you need richer containers there are several ways to take a Maya one and convert it to a StdLib one and viceversa.

We’ll start with an int array, by far the simplest, and offer various examples of both directions (from Maya Array to Vector and from Vector to Maya Array) and how they work and why they might or might not be a good idea. The most obvious corresponding container is a 1D Vector.

The first and most obvious, and also the single most horrifying one would look something like this:

// a Maya array of integers of size 10 containing all 0s
MIntArray mArr(10, 0);
// an empty vector
std::vector<int> vec();

// for the length of the array
for(unsigned int i = 0; i < mArr.length(); i++){
    // each elements is appended to the vector
    vec.push_back(mArr[i]);
}

// a new array to dump the vector into.
// we illustrate the opposite direction of the above.
MIntArray mArrCopy;

for(int i = 0; i < vec.size(); i++){
    mArrCopy.append(vec[i]);
}

The above will work, but it’s usually a bad idea in pretty much any language and any API.
When you append to a container its allocation could have to be grown to accommodate the new data 4)not all the time as it will usually allocate space to spare and then re allocate again only as needed, which is usually a geometric progression in most Vector implementations. Growing a contiguous container that has no allocation left to accommodate growth implies a full copy of the data, and for a large enough set can be a very expensive operation.
In short: It’s the bottom of the barrel both memory footprint wise and processing wise.

The only upside to such a method is that it’s easily readable for the uninitiated, and that it’s a valid extension in place (wanting to add the content of an array to a vector or viceversa). Barring maybe the noob friendliness aspect we can do better in every single regard.
Pros and cons are identical for both directions.

 

A much better option is, whenever possible, to pre-size your containers to the data they will contain.
Memory operations are expensive, and allocating only once and avoiding superfluous copies is often worth jumping through a few hoops.

MIntArray mArr(10, 0);
// this time we pre-size the array
std::vector<int> vec(10);

// for the length of the array
for(unsigned int i = 0; i < mArr.length(); i++){
    // each elements is written to the proper vector's address
    vec[i] = mArr[i];
}

// and the rather obvious reversal
MIntArray mArrCopy;
for(int i= 0; i < vec.size(); i++){
    mArrCopy[i] = vec[i];
}

The above will work to build a copy of the Maya Array into a Vector and viceversa.
Compilers know how to do a decent job of something like the code above, containers are only allocated once avoiding potential thrashing in memory, and the code makes the intent obvious enough.
All in all it’s not bad, but this will only work as a copy, if you want to extend in place you will have to do a bit more work resizing when needed and then appending from the tail.

 

We’ll do it the “Maya” way first for the sake of example.

void appendMArrToVector(const MIntArray& mArr, std::vector<int>& vec){
    int initialVecSize = vec.size();
    int newVecSize = initialVecSize + mArr.length();
    vec.resize(newVecSize);
    for(unsigned int i = 0; i < mArr.length(); i++){
        vec[initialVecSize + i] = mArr[i];
    }
}

std::vector<int> vec;
MIntArray mArr(10, 0);

appendMArrToVector(mArr, vec);

The above isn’t all that bad, performance should be OK, not too hard to read and since it doesn’t care about size it serves copying and extending in place in one go as you can pass it an empty.
There are a couple wasted lines in it not being 100% literal in what it does, but wrapping it in a function takes care of making it self-explanatory code.
The other direction (Vector to Maya Array) is pretty much identical, as you’ve seen in the previous examples it’s mostly knowing what method corresponds to what one way or another.

 

A better version again would be more explicit and compact in what it does. After all with all the work the C++ committee and compiler teams did it would be rude not to use the facilities.
If we stick to StdLib facilities it’s extremely compact and obvious.

std::vector<int> firstVec;
std::vector<int> secondVec;
/* ...
   do something with the vectors here to size them and fill them
   ... */
firstVec.insert(firstVec.end(), secondVec.begin(), secondVec.end();

That’s it, in one line and with rather literal code you are saying:
“take firstVec, and starting from the end of its data append the data in secondVec from beginning to end”

In an ideal world Maya’s API will be extended to provide some standard conforming methods so we could use begin() and end(), insert, and so on, and directly translate to STL/StdLib, but that’s not yet the case. The following would fail miserably.

std::vector<int> vec;
MIntArray mArr(10, 0);

vec.insert(vec.end(), mArr.begin(), mArr.end());

MIntArray simply doesn’t have those methods or anything equivalent, but since C++ supports Arrays extensively in the Standard Library, and Maya’s arrays are fairly well aligned to those, we won’t end up too far from it for the Maya -> Vector direction.

Methods that take begin and end as arguments don’t particularly care that end() and begin() exist, but that the edges of the memory the data lives in are provided as arguments; begin() and end() are just convenient methods to reach the memory addresses at the boundaries of contained data.

If you’ve got some C++ fundamentals down you should know memory location and pointer are practically interchangeable terms. A pointer is simply a qualifier for a memory address, and possibly but not necessarily the type of data expected in there.

Arrays, Maya arrays, and Vector all make the same promise5)or in the case of Maya arrays they seem to and I haven’t bumped into any indication to the contrary, though that promise isn’t explicitly stated anywhere I could find: The data contained is of one type, therefore the step size in memory is fixed, and the data will be contiguous.
This means a majority of the methods of interest for Standard Containers will work by design with pointers to array contents, and those work just fine coming from Maya Arrays or anywhere else respecting the above promise.
The following will work the same as providing begin() and end()

std::vector<int> vec;
MIntArray mArr(10, 0);

vec.insert(vec.end(), &mArr[0], &mArr[10]);

Note a couple of things:
First in both second and third arguments I’m referencing the array’s at-index to get a pointer6)The referencing operator & before an array entry returns the memory location of the entry and not a copy of what is contained there, if this sounds unfamiliar, I might lose you here and for a while longer.
Second, I have an index out of the array’s range.
Our Array is 0-indexed, which means being made of 10 elements, index 10 would be an 11th element, and therefore out of range. That is intentional and, if you tried the code, functional.

StdLib containers for methods like insert require a range from the start (included) to the end (excluded), therefore with plain arrays you want to go to an index one past the last initialized one7)If you want a slightly more detailed explanation: container<>.end() is aligned to the outer edge of the data’s memory boundaries, and given a pointer to an entry in an array is the beginning of the data in the cell we have to go one above the last to be on the outer edge of the last cell..
If this seems weird you might have to just accept it for now. Going one out of range with arrays is, in some cases, the prescribed way to do things8)if you feel like investigating further: it’s related to the symmetry in arrays and pointers and that typed pointers are by design incrementable.

Under the hood the above won’t be very different from what you would get dealing between StdLib containers, so it’s a pretty good way to append a Maya array to a standard container, and the general lesson of pointers being able to replace begin() and end() stands mostly everywhere.

 

Appending your vector to the end of a Maya Array is where things get a touch trickier since Maya doesn’t provide facilities to do it directly, but a fresh copy is actually quite easy since there’s a native method we can use that accepts arrays, and it should make for a continuation, in the opposite direction, of the above.

Maya’s MIntArray has a constructor that will take an int Array and its size and build a Maya Array from it, which allows creating a MIntArray type copy of a Vector source succinctly.

std::vector<int> vec(10, 0);
MIntArray mArrTest(&vec[0], vec.size());

The affinity between arrays and pointers does the work again here. While vec is a Vector, and the constructor we use takes arrays, the beginning of a plain array is pretty much a pointer, so passing a pointer to the first entry in the Vector’s data is equivalent, much like we did for the example before in the other direction9)passing an array as argument to a function is actually said  and meant to “decay to a pointer”, so there’s practically never a difference between offering the array or a pointer to the beginning of its data.
That’s that for a efficient copy, an efficient append though requires a slightly pushier approach.

 

Extending an (Maya) array is where they start being under-served. There is an append() method for single values but nothing  to do an at-once append of a sequence (say: another MIntArray, let alone a Vector or even a plain Array).
You could write the equivalent to the append function in the third listing, a simple walk and set by indices; computational price isn’t too steep, although not as good as what insert() does under the hood, and it still reads alright, but not as good as what you can do copying memory directly.

This is where you get a bit more adventurous and C like.
First we look at what an insert() would do in the worst case scenario for inserting a vector at the tail end of another:

  1. check target vector to see if its allocated size is enough to accommodate the addition
  2. if not, a vector of the appropriate size is created elsewhere in memory
  3. the contents of the vector we want to append to are copied in the new one that will have been allocated with room to spare at the end
  4. the contents we want to append to it are then copied in one stretch of bytecopy from a location in the source to the end of the initialized data from the step above
  5. the original vector is released, and the route to it is now pointed to the one we copied and extended

In the best case scenario the Vector would have enough memory allocated to simply accommodate the new stretch of data, and then only steps one and four would be performed.
While some data is opaque compared to Vector, Maya’s arrays operate similarly and we can rely on Maya for the management of elbow room. What we do not have though is the method to insert, so we need to take care of resizing ourselves, and we have to move the data in the new uninitialized values as efficiently as possible.

// the vector we'll want to insert at the end
std::vector<int>; vecToAdd(10, 0);
// setting some meaningful values
for(int i = 0; i < 10; i++){
    // squares of 1 to 10
    vecToAdd[i] = (i + 1) * (i + 1);
}

// the base array we want to extend
MIntArray mArr(10);
// some meaningful values in there too
for(int i = 0; i < 10; i++){
    // numbers 1 to 10
    mArr[i] = i + 1;
}

// we resize the base array
// this will contain uninitialized values at 10 to 19
mArr.setLength(mArr.length() + vecToAdd.size());

// first we ensure we have the memory size that we need
// to add the elements
// Note that we trust the implementation of MIntArray
// to be using standard ints.
// If you wanted to be safe you could compare sizeof() of one entry each
int memStretch = vecToAdd.size() * sizeof(int);

// now we go C-Style and explicitly copy the memory
memcpy(&mArr[10], &vecToAdd[0], memStretch);

// for good measure we print out the values to verify
for(unsigned int i = 0; i < mArr.length(); i++){
    std::cout << mArr[i] << (i < mArr.length() - 1 ? ", " : "\n");
}
// should look something like this:
// 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100

This might appear verbose and long, but it’s only because we have more frills than usual. If you look at the functional part it’s only two lines, a resize and then a memcpy; one line more than insert(), comparable performance, but… well, it uses memcpy. I have no issues handling memory directly, but you will find some people advising you not to (though that’s a subject for another rant some other time :p).

The short version with an added guard10)the guard is somewhat unnecessary or at least overly prudent in this case, and it certainly would be if it was run at every append, but giving we’re copying memory directly it never hurts to toss an example of something to look out for in there would look like this

std::vector<int>; vecToAdd(10, 0);
MIntArray mArr(10);

mArr.setLength(mArr.length() + vecToAdd.size());
if(sizeof(vecToAdd[0]) == sizeof(mArr[0]))
    memcpy(&mArr[10], &vecToAdd[0], sizeof(int) * vecToAdd.size());

Lastly, the above is pre C++11. If you have access to C++11 std::copy is recommended over memcpy.
Given this isn’t a C++11 tutorial that’s it for the note, I’ll stick to older standards for their wider availability.

 

Conclusions

This is probably a good place to stop for a part 1.
At this point it might not quite be clear why you would want to use something other than Maya’s arrays since pretty much all of the above can pretty much be done with Maya arrays, and what little extending we did with insert could still be done to simply work on MIntArray with other MIntArray, but having sorted through plenty of what is available online I thought there was a gap to fill before moving on to more interesting stuff.
There isn’t a lot around encouraging people to be efficient with memory, to dig a bit through options other than out of the box facilities, and certainly there is very little encouraging to have a look at pointer basics.

As soon as I’ll get bored again, which is likely to be soon, I’ll post again to offer something a bit more practical and exemplary of when you will care about the standard library, until then:

Thanks for reading,
Raff

 

Footnotes   [ + ]

1. C++’s Standard Library
2. e.g. Variable Length Arrays as introduced by C-99 are supported in Gnu’s gcc compiler, but not in Microsoft’s VC. Platform divergence FTL
3. in all fairness Maya’s array growth size can be set while Vector’s is implementation dependent and therefore out of your control
4. not all the time as it will usually allocate space to spare and then re allocate again only as needed, which is usually a geometric progression in most Vector implementations
5. or in the case of Maya arrays they seem to and I haven’t bumped into any indication to the contrary, though that promise isn’t explicitly stated anywhere I could find
6. The referencing operator & before an array entry returns the memory location of the entry and not a copy of what is contained there, if this sounds unfamiliar, I might lose you here and for a while longer
7. If you want a slightly more detailed explanation: container<>.end() is aligned to the outer edge of the data’s memory boundaries, and given a pointer to an entry in an array is the beginning of the data in the cell we have to go one above the last to be on the outer edge of the last cell.
8. if you feel like investigating further: it’s related to the symmetry in arrays and pointers and that typed pointers are by design incrementable
9. passing an array as argument to a function is actually said  and meant to “decay to a pointer”, so there’s practically never a difference between offering the array or a pointer to the beginning of its data
10. the guard is somewhat unnecessary or at least overly prudent in this case, and it certainly would be if it was run at every append, but giving we’re copying memory directly it never hurts to toss an example of something to look out for in there

1 Comment

  • Hey I came across your blog as I was checking if anyone else posted similar discussion about Maya arrays and STL. I’m glad someone else is discussing what is lacking in Maya’s arrays in preference to STL ones.

    I wanted to share some code I wrote that bridges this gap a little to make it easier to use iterators with Maya arrays and also using templates for generic programming. The goal is to replace Maya arrays as they are required for getting data in and out, but my approach wraps them into something more STL like and compatible. Sorry for the self promotion, but wanted to share.

    Code on my GitHub: https://github.com/scottenglert/MayaAPIUtils
    Post on my site about it: http://www.scottenglert.com/2016/05/02/making-mayas-arrays-more-like-stl/

    It’s great that you are posting and explaining these things! Thanks for sharing!

Leave a Comment