Sitecore Sidekick – Content Migrator

Something that has come up for me consistently is the ability to move content from one Sitecore instance to another, usually something like prod server to test server.  Now most people reading this will know that there are several ways to accomplish this such as Sitecore packages or database restores.  However those solutions are time consuming and difficult to manage.  The Content Migrator Sidekick app is my solution to this problem.

migrator.gif

What does it do?

At the core of this app is Kamsar‘s Rainbow serialization.  If that sounds familiar then you may have been using Unicorn to manage your develop level assets in sitecore (layout, template, etc..).  Yes it’s basically a way to Unicorn sync your content items.  However it is a completely stand alone product and takes no dependency on Unicorn whatsoever.

The Content Migrator will move content in a multi-threaded way.  The system is designed to be as fast as the Sitecore Item API can go.  When the operation starts it spins up two separate thread pools, one for transmission of serialized item data from the remote server and another to ingest that data into Sitecore.  Since the inhibiting factor here is the rate in which Sitecore can write item data to the database you can virtually eliminate the penalty of pulling items over the network in most cases (the obvious exceptions being transmission over a slow connection).

While the operation is running any event will be recorded in real time providing your with an immediate report of what is happening and a sense at the rate of ingestion.  A queue count is reported while the operation is running.  This is how many serialized items are in the queue to be written to Sitecore and can act as a time remaining notification

At any time during the life cycle of a migration event any user that has access to the content migrator can view a real time report of what’s happening or even cancel the operation.  There are four buckets that operations can fit into

  1. Currently Running Operations
  2. Completed Operations
  3. Preview Operations
  4. Cancelled Operations

cmoperationmaintenance

I’ve run a few time trials against a normal Sitecore package install.  Completely disregarding the time to actually create and generate the Sitecore package.  For a content migration that was large (24k items) but the majority of the content was unchanged between the source and the target.

Content migrator: 41 seconds

Sitecore Package Install: 70 minutes

yes that’s 102x faster.  Granted this is the use case this tool was designed for, but regardless, it should make synchronizing content between environments a snap!

Config

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
	<sitecore>
		<events>
		</events>
		<pipelines>
			<scsRegister>
				<processor type="ScsContentMigrator.ContentMigrationHandler, ScsContentMigrator" >
					<!-- leave blank for any role -->
					<param name="roles"></param>
					<!-- set to "true" to only allow admins-->
					<param name="isAdmin">true</param>
					<!-- leave blank for any users -->
					<param name="users"></param>
					<!-- number of threads that are going out to the remote server to queue up item data to be installed-->
					<param name="remotePullingThreads">20</param>
					<!-- number of threads taking queued up data and updating or installing the item data in the database-->
					<param name="databaseWriterThreads">10</param>
					<!-- if the root is from the core database, add an attribute database="core" -->
					<roots hint="raw:BuildRoot">
						<root>/sitecore/content</root>
						<root>/sitecore/media library</root>
						<root>/sitecore/system/Marketing Control Panel</root>
					</roots>
					<servers hint="raw:BuildServerList">
						<server>http://ddev/</server>
						<server>http://sc80</server>
					</servers>
				</processor>
			</scsRegister>
		</pipelines>
	</sitecore>
</configuration>

From the <roots> node you can control what item roots the current server will allow other servers to pull.  You’ll likely only want general content to be pull-able, however any Sitecore items are fair game.  You also need to maintain a white list of servers that your current server is allowed to pull from.

Additionally you can fine tune the number of threads used in the pools to match the robustness of your server.

Operation Configuration

ContentMigratorMainScren.png

Step 1:

Select a server.  A white list of servers is configured in the content migrator config file.  Keep in mind that the target server also needs to have the content migrator installed on it.

CMSelect Server.png

Step 2:

Select an item.  Select the main item to be pulled over from the remote server.

cmitemselect.png

Step 3:

Configure options for your operation.

  1. Migrate all children of selected item.
    1.  You would uncheck this option if you only wanted a single item.
  2. Overwrite all existing content with new content from the server.
    1. You would uncheck this option if you had content on the local server that you would rather not have deleted but you wanted to gather all new content items.
  3. If parent doesn’t exist locally add that too.
    1. You would uncheck this option if you only wanted the selected item or it’s children.
  4. Make the local content tree mirror the remote content tree.
    1. You would check this option if you wanted to get rid of all local content in favor for the content from the target, this would commonly be the case if you wanted to overwrite QA or Dev content with production content and dispose of the clutter.

cmoptions.png

Step 4:

Configure advanced sitecore options.

  1. Run using the event disabler.
    1. Unchecking this option will make all events run when installing, moving, deleting, etc.. Items.  This can slow down the operation but if you have lots of important custom on save events, it might be important.
    2. Checking this option will eliminate all events, this can greatly increase speed on installation.
  2. Run using the bulk update context.
    1. Unchecking this option will make the operation run without the bulk update context.  This makes it update the search indexes for each operation, as well as likely other side processes (it’s kind of a black box, if anyone can clarify further please comment).
    2. Checking this option will make the operation skip all index updates.  This is a good idea if you’re updating a lot of content at one time, just don’t forget to run an index rebuild when you’re done.

CMadvanced.png

Step 5:

Select your operation type

  1. Pull – Execute the operation immediately.
  2. Preview – Execute an estimation of the results where no actual changes are made.  Note that this isn’t 100% accurate as some insertion operations could result in unforeseen errors.

CMexecute.png

Now just imagine when someone is having a content problem in production you can now mirror down production in a matter of minutes and save yourself hours of debugging.