I've been playing around with Smarty more and more and I am finding it increasingly better than vBulletin's template engine. I'll go over some of the pros and cons of each system, and touch upon how they work.
vBulletin
vBulletin's template engine is very straight forward. It allows you to use variables, template conditions, and phrase construction. When you save the template, it compiles the template and stores it the database. It converts the entire template into raw PHP code, so it can be used within an eval() statement. Template conditionals (<if condition="$whatever">...</if>) are converted to inline ifs using the ternary operator, and variables are left as-is (except for a few major ones like $vbulletin, $bbuserinfo, $vboptions, etc, which are converted to $GLOBALS[\'variable_name\'] to fix any scope issues).
The biggest benefit with this template system is simplicity. Templates are not much harder than regular HTML, but contain some of PHP's benefits such as conditions and variables. It's weakness is that it it's not very powerful and is slow because it is all evaluated in a new PHP parser every time you use a template, not to mention you have to retrieve them from the database.
Smarty
Like vBulletin templates, Smarty templates are also compiled. Instead of being saved to the database, and evaluated, they are saved as PHP files. This makes it harder to edit as an administrator, but makes it much easier to edit as a developer. PHP can just include() these templates, which makes them MUCH faster than using eval(), and they don't need to be grabbed from the database first.
Smarty takes the template system much further with its built in functions. Instead of escaping a few things, and renaming a few things like vBulletin does when compiling, it actually changes pretty much everything. Smarty code is always wrapped in {}s (configurable) which makes it easy to pick out, but then again it makes editing it in a proper HTML editor a bit more difficult. Unlike vBulletin, it can process loops, do "elseif"s, and even modify variables with common functions.
So why is a loop useful in a template? Usually, if you want to display 5 items on a page, you would need two templates: one for the page, and one for the item (which is evaluated 5 times). With Smarty, you can just use one, and even have the loop code in your template! You can use this for generating HTML <select> options (so useful). One thing I noticed in vBulletin scripts was a large chunk of the code is doing these tasks. Cleaner files = simpler templates = better productivity.
Smarty's weakness is, because its all done with objects, you run into scope issues. If you have a variable $age in your script, you'll have to pass it to Smarty. Many would argue that this is a good thing (less global variables), and I would agree, but it can be extremely annoying if you are used to vBulletin.
I write a lot of vBulletin modifications. I have grown tired of the limitations of the template system vBulletin uses. I have even written scripts to make it easier and more convenient. I think I will try using Smarty with vBulletin on my next project, the only catch would be figuring out how to make it utilize vBulletin's style system.