A new version of CSOM was released today for SharePoint Online. You can find more information about that here: http://dev.office.com/blogs/new-sharepoint-csom-version-released-for-Office-365-may-2016
This version adds the ability to add/retrieve Managers and Contributors to a Term Group in the SharePoint Online Term Store.
Here is a quick code snippet I put together for achieving this:
Once executed, you can see that the Term Group in the Term Store is updated: