Problems and Monitoring

Audit is built never to get in the way of your application's decisions: it does not throw into an evaluation, and its background work never stops your process or keeps it alive. The price of that is that a problem does not surface by itself. This page shows how Audit tells you something is wrong, what each kind of problem means and what to do about it, how to watch the recorder in production, and how to move a journal from another process.

Problems: OnProblem

AuditSettings.OnProblem is the only way Audit reports a problem. If you do not set it, problems are not reported anywhere. Set it when you create the recorder and send problems to your logging and alerting:

var settings = new AuditSettings
{
    OnProblem = problem =>
    {
        switch(problem.Kind)
        {
            case AuditProblemKind.DrainFailed:
            case AuditProblemKind.JournalFull:
            case AuditProblemKind.QueueFull:
                logger.LogWarning(problem.Exception, "Audit is falling behind: {Message}", problem.Message);
                break;
            case AuditProblemKind.RecordsDropped:
                logger.LogInformation("Audit: {Message}", problem.Message);
                break;
            default:
                logger.LogError(problem.Exception, "Audit {Kind}: {Message}", problem.Kind, problem.Message);
                break;
        }
    }
};

An AuditProblem has a Kind, a Message in plain words and, when there is one, the Exception behind it.
The handler may be called on an evaluating thread or on the recorder's background thread, so keep it short and thread-safe.
An exception your handler throws is ignored: recording does not fail because the handler did.
OnProblem is read each time a problem is reported, so a handler set on the recorder's Settings after it was created is the one called.

The Kinds of Problems

Kind	What happened	What to do
`DrainFailed`	Records could not be moved into the database. They stay in the journal or the queue and are tried again, with a growing delay.	Check that the database is reachable and that the account can write (see Database Rights). If the message says the database is behind, run the update scripts (see Updating the Database).
`JournalFull`	The journal reached `JournalCapMegabytes`. Records are dropped, or evaluations wait, as `WhenFull` says. Reported once until the journal is back under 90%.	The database has been out of reach for a long time. Restore it; consider a larger cap.
`QueueFull`	The in-memory queue reached `QueueCapacity`. Reported once until it is back under half.	As above, for the `Drain` and `Volatile` options.
`JournalFailed`	A journal file could not be written: that evaluation was not recorded.	Check the journal folder's free space and that the application's account can write to it.
`JournalDamaged`	A journal file ends in bytes that are not a whole record: a process stopped while writing, or the disk damaged the file. The damaged bytes were skipped; the rest was moved.	Usually nothing, after a crash. Repeated reports point at the disk.
`RecordFailed`	Something went wrong while an evaluation was being recorded: that evaluation was not recorded. The evaluation itself was not affected.	Read the exception. Report it to Code Effects support if it repeats.
`StoreFailed`	In synchronous mode, a record could not be written to the database. With `ThrowOnStoreFailure`, the evaluation failed with an `AuditException`.	As for `DrainFailed`.
`RecordsLost`	With `Drain` or `Volatile`, records held in memory were lost: the database could not take them before `DrainTimeoutSeconds` ran out as the recorder stopped.	Make sure the database is reachable at shutdown, raise `DrainTimeoutSeconds`, or use the journal.
`RecordsDropped`	Records were dropped because the version they name was deleted meanwhile, together with its whole source object.	Nothing: deleting the source object removes its evaluations anyway.

Watching the Recorder: Statistics

AuditRecorder.Statistics counts what the recorder has done since it was created. Every counter only grows; Pending is worked out from the others.

Counter	What it counts
`Captured`	Evaluations recorded and handed on to be written
`Stored`	Records the database took, including those recovered from the journal and those discarded
`Recovered`	Stored records that an earlier process, or another process sharing the journal folder, had left in the journal
`Discarded`	Stored records dropped because the version they name was deleted meanwhile
`Dropped`	Records lost: turned away by a full queue or journal, not written to the journal, lost at shutdown, or not written in synchronous mode
`StoreFailures`	Failed writes to the database. A failed batch is tried again, so this counts attempts, not lost records
`JournalFailures`	Failed journal writes and reads
`Pending`	Records this recorder captured that are neither stored nor dropped yet

Expose them to your monitoring, for example on an endpoint only your operations team can reach:

app.MapGet("/health/audit", (AuditRecorder audit) =>
{
    AuditStatistics s = audit.Statistics;
    return Results.Ok(new
    {
        s.Captured, s.Stored, s.Pending, s.Dropped, s.Recovered, s.Discarded, s.StoreFailures, s.JournalFailures
    });
}).RequireAuthorization();

What to alert on: Pending that keeps growing (the database is not keeping up or is down), any increase in Dropped, and StoreFailures growing over a period.

The Database Check

Call AuditRecorder.CheckDatabase(dbProvider) once at startup (see Recording Evaluations). It returns a SchemaCheckResult:

Property	Meaning
`IsCurrent`	True when the database is at the version this release of Audit needs, or later
`CurrentVersion`	The database's schema version, or null when it has none (not a Rule Repository database, or older than any version)
`RequiredVersion`	The version this release of Audit needs
`Message`	What to do, in words: which update scripts to run

Your application decides what to do with the answer: stop, or log it and run without recording. If you do not check and the database is behind, GetRules(..) throws CE19004 with the same message, and moving records into the database reports DrainFailed.

Moving a Journal from Another Process

Normally the recorder moves its own journal, and anything that processes that are gone left in the same folder. Use AuditJournal when you want something else to do the moving:

No database traffic from your application process. A separate service on the same machine moves the journals; the application's recorders keep journaling and leave the moving to it.
A maintenance step that moves what a decommissioned server or a crashed container left behind, from a copy of its journal folder.

AuditJournal.Replay(..) moves every record that processes no longer running left in the folder, then returns the number of records the database took. It throws CE20009 when another process is moving the folder.

long moved = AuditJournal.Replay("/var/lib/myapp/audit", dbProvider);

AuditJournal.Run(..) keeps moving the folder until it is cancelled: what processes that are gone left, and what running ones write, as they write it. It waits while another process moves the folder. A worker service that does it for every application on the machine:

public sealed class JournalMover : BackgroundService
{
    private readonly IDbProvider db;
    private readonly ILogger<JournalMover> logger;
    private readonly string folder;

    public JournalMover(IDbProvider db, IConfiguration configuration, ILogger<JournalMover> logger)
    {
        this.db = db;
        this.logger = logger;
        this.folder = configuration["Audit:JournalFolder"]!;
    }

    protected override Task ExecuteAsync(CancellationToken stoppingToken) => Task.Run(() =>
    {
        var settings = new AuditSettings { OnProblem = p => this.logger.LogWarning("Audit journal: {Message}", p.Message) };
        long moved = AuditJournal.Run(this.folder, this.db, settings, stoppingToken);
        this.logger.LogInformation("Moved {Count} evaluation records.", moved);
    }, stoppingToken);
}

One process at a time moves a folder. Records are never stored twice: each carries its own ID and the database takes an ID only once, so moving a folder again after a crash writes nothing twice.

Error Codes

Code	Thrown by	Meaning
CE19004	`GetRules(..)`	The database is behind the schema version this release needs. The message names the scripts to run.
CE20001	The recorder's constructor	`JournalFolder` is empty while the journal is on.
CE20002	`GetRules(..)`	No rule the query names has a live version.
CE20003	`GetRules(..)`	A rule calls a reusable or inline rule with no live version.
CE20004	`GetRules(..)`	A live version holds no rule, or not a rule's definition.
CE20005	`GetRules(..)`	A rule asked for by ID or name does not exist or has no live version.
CE20006	The recorder's constructor	An analytics field's name is empty, longer than 128 characters, or has spaces around it.
CE20007	`GetRules(..)`	With `AllFields`, a source version's definition cannot be read for its fields.
CE20008	The recorder's constructor, `AuditJournal`	A number in the settings that delivery cannot work with, such as a batch size below 1.
CE20009	`AuditJournal.Replay(..)`	Another process is moving the journal folder.

These are thrown as RuleException, with the code in the message; CE19004 as a ConnectionException. An AuditException is thrown only in synchronous mode with ThrowOnStoreFailure.

When Evaluations Do Not Appear in the Rule Repository

Work through these in order:

Is recording on? AuditSettings.Enabled is true by default; check nothing turned it off.
Were the rules loaded through the recorder? Rules loaded any other way are never recorded.
Were the evaluator's parameters attached? Attach(..) must be called on the same EvaluationParameters the Evaluator was created with.
Is anything pending? Check Statistics.Pending and OnProblem. DrainFailed problems mean the records are waiting: the database is unreachable, behind, or the account lacks rights.
Is the journal folder writable and persistent? JournalFailed problems say it is not. In a container without a mounted volume, records not yet moved are lost with the container.
Did a short-lived process exit before its records were moved? Call Flush() before it ends.
Are you looking at the right database? Each Rule Repository database needs its own journal folder; a folder is moved into the database of the recorder that moves it.
Is it an evaluator over a FlexSource? Those are not recorded.

Ask on Stack Overflow - Yes, it's still active
Ask in GitHub Discussions

p102

l107

l097 --