Deploy KinetiShift into your Azure subscription
For customers who subscribed to KinetiShift on Azure Marketplace. KinetiShift runs in your own environment as a single Azure Container App; your data stays in it. Typical time: about ten minutes.
Before you start
- An Azure subscription where you can create a resource group, and a region that offers Azure Container Apps.
- Your first pipeline: source, target and tables. The pipeline is a short
key = valuefile; we are glad to draft it with you. - Oracle sources only: an Oracle Database with the standard LogMiner preconditions in place, and your own Oracle Instant Client under your Oracle entitlement. KinetiShift ships no Oracle binaries.
Deploy
- Open the deployment link from the e-mail, signed in to the Azure subscription where KinetiShift should run.
- Basics. Choose the subscription, a new resource group (for example
rg-kinetishift), a region and a short lowercase name prefix. - Image & registry. Keep the pre-filled image and login server. Enter the pull token name and password from the e-mail.
- KinetiShift configuration.
- Licence token: paste the whole licence line.
- Pipeline: one
key = valueper line. Keepmetrics-addr = "0.0.0.0:9100"so health probes and the status dashboard work. - Connection environment: one
KEY=VALUEper line for each connection string the pipeline needs. These become a Container Apps secret and are never shown in outputs. - vCPU / memory: 1 vCPU and 2 GiB is a sensible start.
- Keep files across restarts?: keep Yes. It creates a small Azure Files share in the resource group, mounted at
/var/lib/kinetishift, for files that must survive a restart, such as the dead-letter file (dead-letter-file = "/var/lib/kinetishift/dead-letter.jsonl").
- Oracle Instant Client (Oracle sources only). Upload your Instant Client to an Azure Files share with
libclntsh.soat the share root, choose Yes, and enter the storage account name, share name and key. It is mounted read-only. For other sources choose No. - Review + create. The deployment takes two to four minutes and creates a Log Analytics workspace, a Container Apps environment, the KinetiShift Container App and, with Keep files across restarts?, a storage account with one file share.
Confirm it is running
Open the Container App, then Log stream. A healthy start looks like:
KinetiShift license verified customer=<you> edition=<edition> pipeline starting source="…" sink="…" metrics + lag dashboard listening (GET /metrics, GET /status) addr=0.0.0.0:9100
The metrics and status dashboard are on an internal ingress, reachable from inside the Container Apps environment or a peered network, not from the public internet.
Writing to Azure Databricks
The Container App has its own managed identity, so it can sign in to Azure Databricks without a secret. The Databricks guide covers every option; the short version:
- Copy
managedIdentityPrincipalIdfrom the deployment's Outputs, and look up its application id:az ad sp show --id <principal id> --query appId -o tsv(signed in to your organisation's tenant), or search for that object id under Microsoft Entra ID → Enterprise applications with the application type filter set to Managed Identities. - In the Databricks workspace, add that application id as a service principal (Settings → Identity and access → Service principals → Add → Microsoft Entra ID managed), give it the Databricks SQL access entitlement and CAN USE on the SQL warehouse, and grant it
USE CATALOGplusUSE SCHEMA, CREATE TABLE, SELECT, MODIFYon the target schema (which must exist). - Use a pipeline like this one, then redeploy:
source = "logminer" tables = "SALES.ORDERS,SALES.ORDER_LINES" poll-interval-secs = 2 initial-load = true sink = "databricks" databricks-host = "adb-1234567890123456.7.azuredatabricks.net" databricks-warehouse-id = "abcdef0123456789" databricks-azure-managed-identity = true databricks-catalog = "main" databricks-schema = "sales" databricks-checkpoint-table = "main.sales._kinetishift_checkpoint" metrics-addr = "0.0.0.0:9100"
with the Oracle connection in the connection environment:
KINETISHIFT_ORACLE_CONNECT=//dbhost.example.com:1521/ORCLCDB KINETISHIFT_LOB_FETCH_CONNECT=//dbhost.example.com:1521/ORCLPDB1 KINETISHIFT_ORACLE_USER=C##KINETISHIFT KINETISHIFT_ORACLE_PASSWORD=…
The first connect string is where LogMiner runs (the container database root of a multitenant database); the second is the pluggable database that holds the tables, used for the initial load. For a non-multitenant database, set both to the same value. Keep the checkpoint table: it lets KinetiShift resume exactly where it stopped after a restart. For tables with more than a few tens of thousands of rows, add a Unity Catalog volume for the first copy (databricks-bulk-load-volume). The Databricks guide also covers native column types, dropped and renamed columns, the change-log mode for Lakeflow AUTO CDC with ready-made pipeline templates, and signing in with a service principal instead.
If it does not start
| In the log | Meaning | Fix |
|---|---|---|
license … expired or invalid | Licence line incomplete or expired | Paste the exact line from the e-mail. Renewals send a new one automatically. |
Image pull failure (unauthorized) | Pull token name or password mismatch | Re-enter both from the latest e-mail. |
Connection or authentication error after pipeline starting | A connection string is wrong or unreachable from the app | Fix the value and redeploy; check firewalls and private endpoints. |
HTTP 401 or 403 from Databricks | The identity is not in the workspace, or lacks the SQL access entitlement or a grant | Repeat the Databricks steps above for the application id of managedIdentityPrincipalId. More in the Databricks guide. |
| Revision shows Container crashing | The process exits and restarts | Read the last log lines; usually one of the rows above. |
Change, renew or remove
- Change configuration: redeploy into the same resource group with the new values; the app updates in place.
- Renewal: a new licence line and pull password arrive by e-mail. Update the licence by redeploying with the new line.
- Remove: delete the resource group. Nothing else was created.
- One KinetiShift instance per pipeline: the app runs as a single replica by design. The pull token is for your organisation only.