§learn

Wiring a service

Every object a service needs, built from one config file and closed in the right order, with a separate connection pool for each kind of work.

A service is wired by hand, in one function. There is no container, no scanning and no attribute to register things with. You list the parts, in the order they are needed, and you can read that list to know what the program contains.

Every snippet on this page comes from an example that passes its own tests: axum-wallet-rest-pg (async, src/app.rs) and job-worker-pg (blocking, src/app.rs).

The shape

One function that builds everything and returns one value, and one function that stops it. App and Worker are the examples’ own types, not ekbatan’s: you write yours to hold what your service needs.

/// The running worker.
pub struct Worker {
    pub registry: Arc<DatabaseRegistry>,
    pub executor: Arc<ActionExecutor>,
    pub wallets: Arc<WalletRepository>,
    pub notifications: Arc<NotificationRepository>,
    pub jobs: JobRegistry,
}

impl Worker {
    pub fn start(config: &EkbatanConfig) -> Result<Worker, Error> { /* ... */ }

    /// Stops taking new slots, lets running jobs finish, then closes every pool.
    pub fn stop(&self) {
        block_on(self.jobs.stop());
        self.registry.close();
    }
}

start does five things, in this order:

  1. run the migrations, on every shard’s database;
  2. build a DatabaseRegistry, with one pool per shard;
  3. build the repositories and the executor, with the actions;
  4. build a keyed lock provider and a job registry, each on a pool of its own;
  5. add the local event handler’s jobs, when the config turns them on, and start the jobs.

One config file

Everything starts from ekbatan.json. It holds an ekbatan section, with the same keys as the ekbatan: section of a Java service’s application.yml:

{
  "ekbatan": {
    "namespace": "example.wallet",
    "sharding": {
      "defaultShard": { "group": 0, "member": 0 },
      "groups": [
        {
          "group": 0,
          "name": "global",
          "members": [
            {
              "member": 0,
              "name": "global",
              "configs": {
                "primaryConfig": {
                  "jdbcUrl": "${DATABASE_URL:jdbc:postgresql://localhost:5432/wallet}",
                  "username": "${DATABASE_USER:wallet}",
                  "password": "${DATABASE_PASSWORD:wallet}",
                  "dataSourceProperties": { "application_name": "wallet-service" },
                  "maximumPoolSize": 20
                },
                "jobsConfig": {
                  "jdbcUrl": "${DATABASE_URL:jdbc:postgresql://localhost:5432/wallet}",
                  "username": "${DATABASE_USER:wallet}",
                  "password": "${DATABASE_PASSWORD:wallet}",
                  "maximumPoolSize": 5
                },
                "lockConfig": {
                  "jdbcUrl": "${DATABASE_URL:jdbc:postgresql://localhost:5432/wallet}",
                  "username": "${DATABASE_USER:wallet}",
                  "password": "${DATABASE_PASSWORD:wallet}",
                  "maximumPoolSize": 30
                }
              }
            }
          ]
        }
      ]
    },
    "local-event-handler": {
      "handling": { "enabled": true }
    }
  }
}

Reading it is one call:

let path = std::env::var("EKBATAN_CONFIG").unwrap_or_else(|_| "ekbatan.json".to_owned());
let config = EkbatanConfig::load_json(&path)?;

The full list of keys is in Config files, and the connection settings in Connecting.

Migrate before anything opens a pool

let sharding = config
    .sharding
    .as_ref()
    .ok_or_else(|| Error::Config("ekbatan.sharding is required".to_owned()))?;
// Migrations have no blocking twin: block_on runs the async call to its end.
block_on(
    Migrator::builder()
        .with_sharding_config(sharding)
        .locations([concat!(env!("CARGO_MANIFEST_DIR"), "/migrations")])
        .migrate(),
)?;

let registry = Arc::new(DatabaseRegistry::from_sharding_config(sharding)?);

The migrations are plain SQL files with Flyway’s names (V0001__eventlog.sql), applied by sqlx’s migrator from inside your program. There is no separate tool to install.

DatabaseRegistry::from_sharding_config opens no connection yet: each pool connects on first use. So the order is a rule you keep, not one ekbatan checks: migrate first, then serve. See Migrations.

Register what exists

let wallets = Arc::new(WalletRepository::new(registry.clone()));
let notifications = Arc::new(NotificationRepository::new(registry.clone()));
let executor = Arc::new(
    ActionExecutor::builder()
        .namespace(config.namespace.clone().unwrap_or_default())
        .database_registry(registry.clone())
        .action_registry(
            ActionRegistry::builder()
                .with_action(WalletCreateAction::new(Arc::new(SystemClock)))
                .with_action(WalletDepositMoneyAction::new(wallets.clone()))
                .with_action(CreateNotificationAction)
                .build(),
        )
        .repository_registry(
            RepositoryRegistry::builder()
                .with_repository(wallets.clone())
                .with_repository(notifications.clone())
                .build(),
        )
        .build()?,
);

The framework knows exactly what you put in a registry, and nothing else. That is a real trade:

namespace is written on every event row: it is how a reader tells one service’s events from another’s. It must be dot-separated names of letters, digits and underscores, such as example.wallet. A hyphen is refused when the executor is built, because the namespace is used downstream as a package name where a hyphen is not allowed. Write my_service, not my-service. An empty namespace is refused too.

Three pools, and why

One database gets three pools. Each is built from one of the member’s named configs:

// Locks and jobs live on the default shard's database, each on its own pool.
let home = sharding
    .member(sharding.default_shard)
    .ok_or_else(|| Error::Config("the default shard has no member".to_owned()))?;
let pool_for = |name: &str| {
    let config = home.config_for(name).unwrap_or(home.primary_config());
    ConnectionProvider::from_config(config)
};
let locks = Arc::new(SqlKeyedLockProvider::new(pool_for("lockConfig")?));
poolused bysized forwhy it is separate
primaryConfigrequests: the executor and the repositoriesrequests waiting on the database at oncethe ordinary one
jobsConfigthe job scheduler: finding due jobs, claiming them, stamping heartbeatsa handful; the examples use 5if requests take every connection, the heartbeat cannot be written. A job whose heartbeat stops for 60 seconds is taken for dead and run again on another instance
lockConfigkeyed locksthe most locks held at the same time; the examples use 30a held lock keeps its connection for as long as it is held: the lock belongs to that connection’s session (pg_advisory_lock, or GET_LOCK on MySQL and MariaDB)

Two things are easy to miss:

The lock provider and the job registry live on the default shard’s database, even when there are several shards. See Keyed locks and Distributed jobs.

The jobs and the event handler

let jobs_config = match &config.jobs {
    Some(section) => JobsConfig::from_value(section)?,
    None => JobsConfig::default(),
};
let handler_config = match &config.local_event_handler {
    Some(section) => LocalEventHandlerConfig::from_value(section)?,
    None => LocalEventHandlerConfig::default(),
};
let mut jobs = JobRegistry::builder()
    .connection_provider(pool_for("jobsConfig")?)
    .jobs_config(&jobs_config)
    .with_job(WalletReportJob::new(wallets.clone()))
    .with_job(WalletStipendJob::new(executor.clone(), locks, wallets.clone()));
if handler_config.handling_enabled {
    let handlers = || {
        EventHandlerRegistry::builder()
            .with_handler(WalletMoneyDepositedEventHandler::new(executor.clone()))
            .build()
    };
    jobs = jobs
        .with_job(handler_config.fanout_job(EventFanoutJob::new(registry.clone(), handlers()?)))
        .with_job(handler_config.handling_job(EventHandlingJob::new(registry.clone(), handlers()?)));
}
let jobs = jobs.build()?;
block_on(jobs.start())?;

ekbatan.jobs and ekbatan.local-event-handler are read here, by the app, not by ekbatan on its own. The event handler stays off until local-event-handler.handling.enabled is true.

jobs.start() writes a row in scheduled_tasks for each job that has none, and fails if it cannot - a missing table, or a pool without write access - rather than leaving jobs that never run. Then it starts polling. Consuming events, next in this path, is about the handler itself.

Start and stop

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let path = std::env::var("EKBATAN_CONFIG").unwrap_or_else(|_| "ekbatan.json".to_owned());
    let config = EkbatanConfig::load_json(&path)?;
    let worker = Worker::start(&config)?;

    let (stop, stopped) = mpsc::channel();
    ctrlc_on(stop);
    let _ = stopped.recv();
    worker.stop();
    Ok(())
}

/// Sends on `stop` at Ctrl-C (SIGINT) or SIGTERM.
fn ctrlc_on(stop: mpsc::Sender<()>) {
    ekbatan::blocking::runtime().spawn(async move {
        let mut term =
            tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate()).unwrap();
        tokio::select! {
            _ = tokio::signal::ctrl_c() => {}
            _ = term.recv() => {}
        }
        let _ = stop.send(());
    });
}

The order matters: stop taking work before closing what the work runs on.

  1. stop serving: axum’s graceful shutdown lets requests in flight finish;
  2. jobs.stop(): no new job starts, and running ones get up to shutdownMaxWait to finish;
  3. close the DatabaseRegistry, which closes every shard’s pools.

Three things to know before you copy this:

Under axum

The executor, the lock provider and the repositories go in the router’s state, behind Arc:

#[derive(Clone)]
pub struct AppState {
    pub executor: Arc<AsyncActionExecutor>,
    pub locks: Arc<dyn AsyncKeyedLockProvider>,
    pub wallets: Arc<WalletRepository>,
}

An ekbatan::Error becomes an HTTP answer in one place:

impl IntoResponse for ApiError {
    fn into_response(self) -> Response {
        let status = match &self.0 {
            Error::NotFound { .. } => StatusCode::NOT_FOUND,
            Error::Invalid(_) | Error::Action(_) => StatusCode::UNPROCESSABLE_ENTITY,
            Error::StaleRecord { .. } | Error::StaleRecords { .. } => StatusCode::CONFLICT,
            _ => StatusCode::INTERNAL_SERVER_ERROR,
        };
        // ...
    }
}

And the deposit takes the wallet’s keyed lock around the whole action, so two deposits to one wallet run one after the other instead of one failing on the optimistic lock:

let lease = state
    .locks
    .acquire(&format!("wallet:{id}"), DEPOSIT_LOCK_MAX_HOLD)
    .await?;
let deposited = state
    .executor
    .execute::<WalletDepositMoneyAction>(Some(PRINCIPAL), params)
    .await;
lease.release().await;
Ok(Json(WalletResponse::from(&deposited?)))

The result is kept, the lease released, and only then is the error returned. A lease that is dropped without release() is still released, in the background; max_hold is the safety net for a holder that hangs. acquire waits as long as the key is held; try_acquire takes a max_wait and gives None when the key stayed taken, which suits an endpoint that would rather answer “busy, retry”.

Never call ekbatan::blocking from an axum handler. See Blocking or async?.

See also