Source Synchronization and Reindexing

Every rule in the Rule Repository is written against a source object: the class, Source XML document or Adaptive Source that describes the data your rules look at, such as an Order or a Customer. The Rule Repository does not read your assemblies on its own. Your application tells it which source objects exist, and what each looks like today, by calling Context.SyncSources(..). The repository stores each one, recognizes it the next time, and keeps a version for every change that matters to the rules written against it.

You need synchronization because rules and sources change at different speeds. Developers add, rename and remove properties; business users write and publish rules. Source versions keep the two in step:

• A rule version always records the source version it was written against, so you can see later what a rule was written for.
• When the source object changes in a way that can change how a stored rule evaluates, the repository creates a new source version. The rules written against the old one are not changed, and users can see what changed and move rules to the new version deliberately.
• When only a label changes (a display name, a description), the stored version is updated in place. No new version is created and no rule is affected.

This page explains how to call the sync, when to call it, what each setting decides, what makes a new version, how to read the result, and how to keep the repository's member usage index current with Context.Reindex(..).

The Shortest Working Sync

Sync once at startup, after the database check, with every source object your application evaluates:

using CodeEffects.Rule.Connection;
using CodeEffects.Rule.Repository;

var dbProvider = new CodeEffects.Rule.Connection.SqlServer.DbProvider(
	builder.Configuration.GetConnectionString("Rules"));

var settings = new SourceSettings();
settings.SourceTypes.Add(typeof(Order));
settings.SourceTypes.Add(typeof(Customer));

SyncResult result = Context.SyncSources(dbProvider, settings);

That is all a first installation needs. The first call stores both source objects and makes their first versions live. Every later call with the same classes writes nothing. A call after you have changed Order creates a new version of Order, or updates its labels in place, and leaves Customer alone.

Calling the sync on every start is safe and cheap: a definition that matches what is stored is read and compared, and nothing is written.

When to Call It

The sync is a feature of its own, not a step the rest of the repository waits on. Nothing in the repository calls it, and the repository runs without it; there would simply be no source objects to write rules against. Call it whenever it suits your application:

When	Why
At application startup	The simplest choice. The source objects are synced every time a new build of your application starts.
From a deployment step	A console tool or a pipeline task that syncs right after the new build is deployed, before users reach it.
From an administration screen	A button an administrator presses after a deployment. Set SourceSettings.UserId to that administrator, so the activity log shows who did it.
On a schedule	For source objects that come from data rather than code, such as Adaptive Sources whose fields are configured by users.

Call it at least once before anyone writes a rule, and after that whenever the types or definitions your rules evaluate may have changed.

In a web farm, every server may run the same startup sync at the same time. That is supported: when two servers try to create the same new source object at once, the one that loses the race retries once and finds the source object the other one created. Each source object is synced in its own transaction, and a deadlock or a dropped connection re-runs that source's transaction once.

The Three Kinds of Input

SourceSettings takes source objects in three lists. Each definition, whatever its form, is converted to a Source XML document and then handled the same way: identified by its alias, compared, and versioned.

Property	Takes	Converted by
SourceTypes	.NET types decorated, or not, with the Code Effects attributes (Reflection source)	Control.GetSourceXml(Type, GetTypeDelegate), exactly as the Rule Editor converts a type
SourceXmls	Source XML documents, one string per source object	Used as they are
SourceProviders	Adaptive Sources: objects that implement ISourceProvider	Control.GetSourceXml(ISourceProvider): the provider's GetSource(), written as Source XML with every enumeration's members

var settings = new SourceSettings();

// A class
settings.SourceTypes.Add(typeof(Order));

// A Source XML document your application keeps as a file or a resource
settings.SourceXmls.Add(File.ReadAllText(Path.Combine(contentRoot, "Sources", "Invoice.xml")));

// An Adaptive Source
settings.SourceProviders.Add(new CustomerSourceProvider());

SyncResult result = Context.SyncSources(dbProvider, settings);

Two rules apply to all three lists:

• One definition per source object per call. Passing the same source object twice in one call, for example as a type and as a document, is not supported. Results are returned in the order of the lists: documents first, then types, then providers.
• An Adaptive Source must implement ISourceProvider. The Rule Repository never calls your provider at run time: its editors always run on the stored Source XML of a source version. The provider is read only when you sync.

How a Source Object Is Identified

A source object's identity is its alias: the type name of the first <source> node of its Source XML, with any assembly identity removed. For a class, that is its full name, such as Acme.Sales.Order, unless you give it an alias of your own with the TypeOrAliasName property of the [Source] attribute. For an Adaptive Source, it is the provider's SourceObjectTypeOrAliasName.

Because the assembly identity is removed, rebuilding your application, raising its version number or moving the class to another assembly does not change the alias. Everything else does:

Change in your code	Same source object?	What happens
Rebuild, new assembly version	Yes	Nothing
Move the class to another assembly, same namespace and name	Yes	Nothing
Rename a property, method or parameter	Yes	A new source version: the old name is a removal and the new name an addition
Rename the class or its namespace	No	A brand new source object with its first version. Rules, tests and history stay with the old alias
Add, change or remove an alias in [Source(TypeOrAliasName = ...)]	No	The same as a rename

Keep the class name, the namespace and any alias stable for the lifetime of a source object. The repository deliberately does not guess that a renamed class is the old one: matching definitions by similarity is unreliable and unsafe where rules are audited. If you must rename a class, give it the old name as its alias:

// The class was Acme.Sales.Order; the alias keeps every rule, test and version attached to it
[Source(TypeOrAliasName = "Acme.Sales.Order")]
public class SalesOrder
{
	// ...
}

An alias may be at most 450 characters (CE10051).

What Makes a New Source Version

The repository compares an incoming definition with a stored version and decides one of four things. The principle: a change counts only if a stored rule could evaluate differently because of it.

Outcome	When	What is written
SourceCreated	The alias is new	A source object and its first version, which is live
Unchanged	The definition matches the stored version exactly	Nothing
LabelsUpdated	It matches except for labels and settings that do not count	The stored version's document is replaced in place; one activity entry lists every change, with the document before and after
VersionCreated	It differs in something that counts	A new source version, live or not depending on NewVersionsGoLive

Changes that create a new version:

• A field, method, action, parameter or enumeration added, or removed while a rule may use it
• The declared type of a field, parameter or return value, and whether it can be null
• The name of the data source that supplies a member's values
• Min, Max, a string's maximum length, and StringComparison, of a field, a parameter or a return value
• A member's declared name (a rename is a removal and an addition; renames are not tracked)
• A method's or action's test method (TestMethodName, TestMethodType)
• A TimeSpan member's TimeType (time of day or duration)
• An enumeration member removed while it is used, or when the repository cannot tell whether it is used

Changes written in place (no new version):

• Display names and descriptions of anything
• AI aliases, AI descriptions and the correlation field
• Menu filters and groups, date and time display formats, a collection's singular display name
• Authoring settings the engine never reads: ValueInputType, Settable and Gettable, the calculation and concatenation flags, Prompt
• An enumeration member added or renamed (members are paired by number), or removed while nothing uses it
• Where a data source points (its items are never stored)

List<T> and ICollection<T> holding the same items are treated as the same type, so changing one to the other is not a new version.

Enumerations are a published contract. The repository pairs enumeration members by their number. Never renumber or reassign a member: a member given another member's number looks like a rename, and the rules that used the old meaning silently get the new one. Add new members with new numbers, and remove a member only when no rule uses it.

The Settings

Property	Type	Default	What it decides
SourceTypes	IList<Type>	empty	Classes to sync
SourceXmls	IList<string>	empty	Source XML documents to sync
SourceProviders	IList<ISourceProvider>	empty	Adaptive Sources to sync
ComparisonType	SourceVersionComparisonType	CompareToLastVersion	Which stored versions an incoming definition is compared with
TestsHandling	TestsHandlingType	CopyTestsFromProductionVersion	What happens to existing tests when a new version is created
NewVersionsGoLive	bool	false	Whether a new version of an existing source object becomes its live version at once
UserId, UserName, UserType	string, string, UserType	null, null, User	Who asked for the sync. Left empty, the work is shown as the system's
TypeResolver	GetTypeDelegate	null	Finds enumeration and data source types the definitions name but the runtime cannot load

ComparisonType

Value	Compares with	Use it when
CompareToLastVersion	The most recently stored version	The default. Your deployments move forward. A rollback to an older build creates a new version that duplicates an older one, which is harmless.
CompareToProductionVersion	The live version (or the latest, if none is live)	You keep several versions waiting and want each sync measured against what is live.
CompareToAllVersions	Every stored version, newest first; the first match wins	You deploy older builds back and forth (blue/green, rollbacks) and do not want a new version each time.

TestsHandling

When a new source version is created, saved tests can follow it, so users can run them against the new definition right away. Tests that are copied or moved are reset to Not Run, because their last results describe a run against the old version.

Value	Effect
CopyTestsFromProductionVersion	Copies the live version's tests; the originals stay. The default.
MoveTestsFromProductionVersion	Moves the live version's tests to the new version.
CopyTestsFromLastVersion	Copies the most recent version's tests.
MoveTestsFromLastVersion	Moves the most recent version's tests.
NoTests	The new version starts without tests.

Where a source object has no live version, the "production" choices take tests from its most recent version.

NewVersionsGoLive

The live version of a source object is the one the repository treats as current: new rules and tests are built on it, and "compare to production" and "copy tests from production" use it. It does not change what your application executes; your application runs whatever its own code is.

• The first version of a new source object is always live, whatever this setting says.
• Leave it false (the default) when rules run somewhere other than the application that syncs: another service, another environment, a later deployment that may still receive data shaped like the previous version. The new version waits until a person makes it live in the repository, when the place that evaluates is ready.
• Set it to true when the application that syncs is the application that evaluates. The newest definition is what runs the moment it is deployed, and the repository should say so. A version made live this way is recorded in the activity log as the work of UserId, or of the system.

var settings = new SourceSettings
{
	ComparisonType = SourceVersionComparisonType.CompareToAllVersions,
	TestsHandling = TestsHandlingType.CopyTestsFromProductionVersion,
	NewVersionsGoLive = true
};

TypeResolver

A Source XML document names the types of enumerations and data source classes it uses. The sync loads them to read their members. Normally they load by name. When they cannot, because the type was moved or renamed, or lives in an assembly your application does not load by name, the definition fails with CE10052 (no resolver given) or CE10067 (a resolver was given and returned nothing; the message names every type). Supply a resolver that returns the type, or null to let the runtime answer:

static Type ResolveType(string typeName)
{
	// The definitions name an enumeration that moved to another assembly
	if(typeName.StartsWith("Acme.Sales.OrderStatus", StringComparison.Ordinal))
		return typeof(Acme.Sales.Contracts.OrderStatus);

	return null;
}

var settings = new SourceSettings { TypeResolver = ResolveType };

The repository's per-request configuration has a TypeResolver of its own, for the same reason: the Rule Editor loads those types again when it builds itself from a stored definition. Usually both get the same delegate.

Who Synced

A sync that runs at startup or on a schedule was asked for by nobody, and its work is shown as the system's. When a person triggers it, say so:

var settings = new SourceSettings
{
	UserId = currentUser.Id,
	UserName = currentUser.DisplayName,
	UserType = UserType.User
};

The ID is whatever your application calls its users, exactly as for the request configuration. A deployment pipeline can identify itself as UserType.Service.

Reading the Result

SyncSources(..) returns a SyncResult with one SourceResult per definition, and two lists of warnings for the whole repository.

SyncResult result = Context.SyncSources(dbProvider, settings);

foreach(SourceResult source in result.Sources)
{
	logger.LogInformation("{Alias}: {Outcome} (version {VersionId}, {Differences} difference(s), " +
		"{Labels} label change(s), {Tests} test(s)) {Message}",
		source.AliasName, source.Outcome, source.SourceVersionId,
		source.DifferenceCount, source.LabelChangeCount, source.TestCount, source.Message);
}

SourceResult property	Meaning
Kind, Index	Which list the definition came from (Xml, Type, Provider) and its position in that list
AliasName	The source object's alias, or null when the definition could not be read far enough to find it
Outcome	Unchanged, LabelsUpdated, SourceCreated, VersionCreated or Failed
SourceRecordId, SourceVersionId	The source object and the version created, updated or matched
ComparedVersionId	The stored version the definition was compared with
DifferenceCount	How many differences made a new version
LabelChangeCount	How many label and setting changes were found
TestsSourceVersionId, TestCount	Where tests were copied or moved from, and how many
UsersToNotify	The users your application may want to tell: everyone for a new source object; the authors and last runners of the source object's tests for a new version
Message	What was done, or why it failed
ControlNumber, Error	For a failure: the Code Effects control number and the exception

UsersToNotify is only a list: the repository sends nothing itself.

Failures

Each definition is processed on its own. One that fails is rolled back and does not stop the others. When any failed, SyncSources(..) throws a RepositoryException with control number CE10059 after processing all of them, and the complete result rides on the exception:

try
{
	SyncResult result = Context.SyncSources(dbProvider, settings);
	Report(result);
}
catch(RepositoryException ex) when(ex.SyncResult != null)
{
	logger.LogError(ex, "Source sync failed: {ControlNumber}", ex.ControlNumber);

	// The definitions that succeeded are stored; report every one
	Report(ex.SyncResult);
}

A failed definition is a problem for the developers who wrote it: users never see these messages. The common causes:

Control number	Cause	Fix
CE10050	An empty definition	Remove it from the list
CE10051	An alias over 450 characters	Shorten the alias
CE10052	A type the definition names could not be loaded, and no TypeResolver was given	Supply a TypeResolver
CE10067	The TypeResolver returned nothing for one or more types	Return the type from the resolver
CE10170	A Source XML <references> section names the same property path twice	Correct the document
CE10001-CE10041	The definition could not be read or compared	Read the message: it names the node
CE19002, CE19003	The database failed or timed out	See Error Codes

The Member Usage Index and Reindexing

The repository keeps an index of every member each source version declares, and of every member and reusable rule each rule version uses. This member usage index answers questions that would otherwise mean reading every rule:

• Which rules use a field, method or action
• What a new source version would break, and which rules must move to it
• Which members no rule uses
• Whether a reusable rule can still be called by the rules that call it, before a version goes live

The sync and every rule save keep the index current as they write. Nothing needs doing in a new installation. The index needs rebuilding only for what was stored without it:

• A repository upgraded to a release that changes how the index is computed. The release notes say so, and the first sync after the upgrade lists every affected source object.
• Rules brought in from outside the repository, for example by a migration script that inserts rule versions directly into the database.
• A source object the sync reports as needing it.

Knowing When

After processing every definition, the sync looks at the whole repository, not only at the definitions you passed, and returns two warnings:

• SyncResult.AliasesToReindex: every source object whose index was never built, or was built by an earlier release that computed it differently. Its rules evaluate as they always did; only what the index answers about them is incomplete until it is reindexed.
• SyncResult.SourcesOutOfSync: every source object whose latest version embeds a type that is also a source object of its own, where the two declare different members (EmbeddedSourceMismatch). It happens when one of the two is synced and the other is not, or when they are synced from different definitions. Syncing both again from the same definitions clears it. Until then, a reusable rule written on the embedded type that uses a member the copy lacks cannot be called from the other source object's rules.

Neither is a failure, and both are read before the CE10059 check, so they are on the result an exception carries too.

Running the Reindex

The sync never reindexes by itself. Your application decides when, and over which source objects:

SyncResult result = Context.SyncSources(dbProvider, settings);

if(result.AliasesToReindex.Count > 0)
{
	// After the sync, never alongside it
	ReindexResult reindex = Context.Reindex(dbProvider, result.AliasesToReindex);

	foreach(ReindexSourceResult source in reindex.Sources)
	{
		logger.LogInformation("Reindexed {Alias}: {Versions} source version(s), {Rules} rule version(s), " +
			"{Refs} reference(s), {Unresolved} unresolved",
			source.AliasName, source.SourceVersionCount, source.RuleVersionCount,
			source.ReferenceCount, source.Unresolved.Count);
	}
}

Call	Reindexes
Context.Reindex(dbProvider)	Every source object in the repository
Context.Reindex(dbProvider, aliases)	Only the source objects named. An empty list reindexes nothing

What a reindex does for each source object, in one transaction:

1. Locks the source object, so rule saves on it wait until it is done.
2. Reconciles the member catalog of every one of its source versions with that version's stored definition.
3. Rewrites the references of every version of every rule written against it, as a save would.
4. Removes catalog entries for members no longer declared that no rule references.

A use a rule version makes that resolves to nothing, and a rule version whose definition cannot be read, do not fail the reindex: they are listed in ReindexSourceResult.Unresolved, each entry starting with the rule version's ID. A source object that fails is rolled back and the others go on; at the end, Reindex(..) throws a RepositoryException with CE10118 that carries the whole ReindexResult. An alias the repository does not have is CE10117.

Rerunning a reindex is safe: it writes the same result. A source object reindexed successfully drops out of AliasesToReindex until a later release changes the index again.

When and Where to Run It

• It can run while the repository is in use. A rule save and the reindex of the same source object take the same lock, so one waits for the other and neither misses what the other wrote.
• Not at the same moment as a sync of the same source object. The two can deadlock; each retries once. Run the reindex after the sync returns, as in the example above.
• A large repository is best reindexed at a quiet time, for example from a deployment step or a scheduled job, rather than on every application start.
• After a migration that writes rule versions directly into the database, reindex the source objects it touched.

Deciding What to Pass

Context.GetAliasNames(dbProvider) returns the alias of every source object already stored. Passing every source object on every sync is fine; use the list when you want to treat new ones differently, or to skip work you know is unnecessary:

var stored = new HashSet<string>(Context.GetAliasNames(dbProvider), StringComparer.Ordinal);

var settings = new SourceSettings();

foreach(var type in new[] { typeof(Order), typeof(Customer) })
{
	if(!stored.Contains(type.FullName)) logger.LogInformation("{Type} is new to the repository.", type.FullName);
	settings.SourceTypes.Add(type);
}

Context.SyncSources(dbProvider, settings);

Seeding is not syncing. A sync stores what your source objects look like now. Never call it with several older definitions of one source object to give an empty repository a history it did not have: that is a one-time seeding job, and it must check GetAliasNames(..) and do nothing once the alias exists, or each run would add more versions.

A Complete Startup Sync

The pattern a production host uses: check the database, sync, report, and reindex when told to.

using CodeEffects.Rule.Connection;
using CodeEffects.Rule.Repository;
using CodeEffects.Rule.Repository.Common;

public static class RuleSources
{
	public static void Sync(IDbProvider dbProvider, ILogger logger)
	{
		var schema = Context.CheckDatabase(dbProvider);
		if(!schema.IsCurrent)
		{
			logger.LogError("Rule Repository database: {Message}", schema.Message);
			return;
		}

		var settings = new SourceSettings
		{
			ComparisonType = SourceVersionComparisonType.CompareToLastVersion,
			TestsHandling = TestsHandlingType.CopyTestsFromProductionVersion,
			NewVersionsGoLive = false,
			UserType = UserType.Service
		};

		settings.SourceTypes.Add(typeof(Order));
		settings.SourceTypes.Add(typeof(Customer));

		SyncResult result;

		try
		{
			result = Context.SyncSources(dbProvider, settings);
		}
		catch(RepositoryException ex) when(ex.SyncResult != null)
		{
			logger.LogError(ex, "Rule Repository: source sync failed ({ControlNumber}).", ex.ControlNumber);
			result = ex.SyncResult;
		}

		foreach(var source in result.Sources)
			logger.LogInformation("Rule Repository: {Alias} {Outcome}. {Message}",
				source.AliasName ?? "(unread)", source.Outcome, source.Message);

		foreach(var mismatch in result.SourcesOutOfSync)
			logger.LogWarning("Rule Repository: {Alias} embeds {Embedded}, which differs from its own source object.",
				mismatch.AliasName, mismatch.EmbeddedAliasName);

		if(result.AliasesToReindex.Count > 0)
		{
			try
			{
				Context.Reindex(dbProvider, result.AliasesToReindex);
			}
			catch(RepositoryException ex) when(ex.ReindexResult != null)
			{
				logger.LogError(ex, "Rule Repository: reindex failed ({ControlNumber}).", ex.ControlNumber);
			}
		}
	}
}

// Program.cs, after the app is built
RuleSources.Sync(dbProvider, app.Logger);

In a .NET Framework application, call the same method from Application_Start in Global.asax; the library and the calls are the same.

Troubleshooting

What you see	Why	What to do
A new source object appeared after a refactoring, and the old one's rules are "missing"	The class or namespace was renamed, so the alias changed	Give the class its old name as [Source(TypeOrAliasName = ...)] and sync again. The accidental new source object can be deleted in the repository
Every deployment creates a new version	The definition really differs each time, or you roll back and forth with CompareToLastVersion	Compare two versions in the repository to see what differs; use CompareToAllVersions for rollbacks
A label you changed does not show	It is written in place on the version that matched, which may not be the live one	Check which version the result names in SourceVersionId
CE10052 or CE10067	A type the definition names cannot be loaded	See TypeResolver
AliasesToReindex lists every source object after an upgrade	The release changed how the index is computed	Run Context.Reindex(..) once, at a quiet time
SourcesOutOfSync lists a pair	An embedded type and its own source object were synced from different definitions	Sync both in the same call, from the same build

Related

• Source Objects
• Live Versions, Approvals and Loading Rules
• Error Codes
• Synchronizing Sources, the plain-language summary in the Rule Repository guide
• Source Versions, what users see
• API reference: Context, SourceSettings, SyncResult, SourceResult, ReindexResult, EmbeddedSourceMismatch, SourceVersionComparisonType, TestsHandlingType, SourceOutcome