AWS Secrets Manager lookup

Vegy — Mon, 16 Feb 2026 11:09:05 GMT

Hi,

I'm looking to do a Proof of Concept for using the external credentials addon feature mentioned here https://help.qlik.com/en-US/replicate/November2025/Content/Replicate/Main/Security/external_credentials.htm

Whilst I have a new addon written and loading successfully I am not quite able to get it working, with the error being "Unable to decrypt password" when I use lookup::key_name in a password field.

Our source is DB2LUW and DB2zOS. I believe these are both supported.

I understand I could reach out to professional services but I wondered if any users had got this working with AWS and if they might be comfortable sharing their .so code?

Cheers

Veg

Re: AWS Secrets Manager lookup

Vegy — Mon, 16 Feb 2026 14:25:13 GMT

I can see that there might be a bug in the version I'm testing with 2023.11 so trying a newer version.

Re: AWS Secrets Manager lookup

john_wang — Tue, 17 Feb 2026 00:32:21 GMT

Hello @Vegy ,

Thanks for the update.

Replicate 2023.11 has reached its end-of-support (EOS). We recommend using the latest 2025.11 build for the PoC to ensure full support and the best results.

Good luck!

John.

Re: AWS Secrets Manager lookup

Vegy — Tue, 17 Feb 2026 10:02:10 GMT

Hi @john_wang,

I've passed this on to our product manager.

I got this working last night any wanted to share the code to the community, however, I wanted to confirm it is fine to do so first?

Cheers

Veg

Re: AWS Secrets Manager lookup

john_wang — Tue, 17 Feb 2026 12:41:49 GMT

Hello @Vegy ,

Thank you very much for the update. This resolution will be valuable to all our users. Please feel free to share your code, as long as it doesn’t contain any sensitive or credential-related information.

Regards,

John.

Re: AWS Secrets Manager lookup

Vegy — Tue, 17 Feb 2026 13:00:16 GMT

No worries! So just for information I was able to get this working on 2023.11 but as @john_wang says this is EOS. This did also work in 2025.11.

This function accepts both ARN and just secret name, it also handles both string and JSON secrets as long as the secret has a key called password with the correct value in it, i.e. { "password" : "mypassword" }

AWSPasswordProvider.c

// AWSPasswordProvider.c // Qlik Replicate external-credentials provider that fetches endpoint passwords // from AWS Secrets Manager using the AWS CLI. // // Key points: // - Init follows Qlik's sample: AR_AO_INIT -> GET_AR_AO_PASSWORD_PROVIDER_DEF() // -> set get_secret_func -> AR_AO_REGISRATION->register_password_provider(...) // - get_secret signature matches your header (char *name, ...) // - Supports plaintext secrets and JSON {"password":"..."} // - Enforces 4KB max and output buffer size // - Adds breadcrumbs to /opt/attunity/replicate/tmp/awspp_breadcrumb.txt // // Build example: // gcc -fPIC -O2 -shared \ // -I/opt/attunity/replicate/addons/include \ // -o /opt/attunity/replicate/addons/AWSPasswordProvider.so \ // /opt/attunity/replicate/addons/AWSPasswordProvider.c // // Deploy the .so + addons_def.json to BOTH: // /opt/attunity/replicate/addons/ // /opt/attunity/replicate/endpoint_srv/addons/ // // In endpoint Password field: lookup::<secret-id> // e.g., lookup::vegtest // lookup::arn:aws:secretsmanager:eu-west-2:123456789012:secret:mysecret-ABC #include <stdio.h> #include <stdlib.h> #include <string.h> #include <ctype.h> #include <time.h> #include <unistd.h> #include "ar_addon.h" #include "ar_addon_transformation.h" #include "ar_addon_password_provider.h" #include <fcntl.h> #include <sys/stat.h> #define MAX_QLIK_SECRET_BYTES (4 * 1024) // Qlik documented limit // -------------------- optional breadcrumb for debugging -------------------- #ifndef BREADCRUMB_PATH #define BREADCRUMB_PATH "/opt/attunity/replicate/tmp/awspp_breadcrumb.txt" #endif static void debug_dump_secret(const char* lookup_key, const char* pw) { // Debugging of the AWS Secrets Manager can be enabled by setting environment variable DEBUG_AWS_ADDON=1 in the replicate process. // This will create/append to /opt/attunity/replicate/tmp/awspp_secret_dbg.txt with lines like: // 2026-02-16T19:00:00Z pid=1234 proc=replicate_task lookup::vegtest PLAIN="mysecret" HEX=6d79736563726574" // Therefore use should be used with care and only in non-production environments, ensuring the tmp file is securely handled and deleted after use. // The hex dump can help confirm if there are any hidden characters or encoding issues in the retrieved secret. const char* dbg = getenv("DEBUG_AWS_ADDON"); if (!dbg || strcmp(dbg, "1") != 0) return; // Ensure the directory exists with permissive mode for both processes mkdir("/opt/attunity/replicate/tmp", 0777); int fd = open("/opt/attunity/replicate/tmp/awspp_secret_dbg.txt", O_WRONLY | O_CREAT | O_APPEND, 0600); if (fd < 0) return; // Timestamp (UTC) time_t now = time(NULL); struct tm tmv; gmtime_r(&now, &tmv); char ts[32]; strftime(ts, sizeof(ts), "%Y-%m-%dT%H:%M:%SZ", &tmv); // Process name (best effort) char comm[64] = {0}; FILE *fc = fopen("/proc/self/comm", "r"); if (fc) { fgets(comm, sizeof(comm), fc); fclose(fc); size_t n = strlen(comm); if (n && (comm[n-1]=='\n' || comm[n-1]=='\r')) comm[n-1] = '\0'; } else { strncpy(comm, "unknown", sizeof(comm)-1); } // Plaintext line dprintf(fd, "%s pid=%d proc=%s key=%s PLAIN=\"%s\"\n", ts, (int)getpid(), comm, (lookup_key ? lookup_key : "(null)"), (pw ? pw : "(null)")); // Hex dump line if (pw) { dprintf(fd, "%s pid=%d proc=%s key=%s HEX=", ts, (int)getpid(), comm, (lookup_key ? lookup_key : "(null)")); for (const unsigned char *p=(const unsigned char*)pw; *p; ++p) dprintf(fd, "%02x", *p); dprintf(fd, "\n"); } close(fd); } static void write_breadcrumb(const char* tag) { // Writes "YYYY-MM-DDTHH:MM:SSZ pid=... proc=... :: <tag>" char ts[32]; time_t now = time(NULL); struct tm tmv; gmtime_r(&now, &tmv); strftime(ts, sizeof(ts), "%Y-%m-%dT%H:%M:%SZ", &tmv); char comm[64] = {0}; FILE *fc = fopen("/proc/self/comm", "r"); if (fc) { fgets(comm, sizeof(comm), fc); fclose(fc); size_t n = strlen(comm); if (n && (comm[n-1] == '\n' || comm[n-1] == '\r')) comm[n-1] = '\0'; } else { strncpy(comm, "unknown", sizeof(comm)-1); } FILE *f = fopen(BREADCRUMB_PATH, "a"); if (!f) return; // silently ignore if unwritable fprintf(f, "%s pid=%d proc=%s :: %s\n", ts, (int)getpid(), comm, tag); fclose(f); } // -------------------- helpers -------------------- static void trim_ws(char *s) { if (!s) return; char *p = s; while (*p && isspace((unsigned char)*p)) p++; size_t len = strlen(p); memmove(s, p, len + 1); while (len > 0 && isspace((unsigned char)s[len - 1])) { s[--len] = '\0'; } } static void unquote_if_wrapped(char *s) { size_t n = strlen(s); if (n >= 2 && s[0] == '"' && s[n-1] == '"') { memmove(s, s+1, n-2); s[n-2] = '\0'; } n = strlen(s); if (n >= 2 && s[0] == '\'' && s[n-1] == '\'') { memmove(s, s+1, n-2); s[n-2] = '\0'; } } static int starts_with_json_obj(const char *s) { if (!s) return 0; while (*s && isspace((unsigned char)*s)) s++; return (*s == '{'); } // naive extractor: finds "key":"value" static int extract_json_field(const char* json, const char* key, char* out, int out_len) { char pattern[128]; snprintf(pattern, sizeof(pattern), "\"%s\"", key); const char* p = strstr(json, pattern); if (!p) return 0; p = strchr(p, ':'); if (!p) return 0; p++; while (*p == ' ') p++; if (*p != '\"') return 0; p++; int i = 0; while (*p && *p != '\"' && i < out_len - 1) { out[i++] = *p++; } out[i] = '\0'; return (i > 0); } // escape single quotes for safe single-quoted shell args static void shell_escape_single_quotes(const char* in, char* out, size_t out_len) { size_t o = 0; for (size_t i = 0; in[i] && o + 4 < out_len; i++) { if (in[i] == '\'') { const char *esc = "'\\''"; // end, escaped quote, reopen size_t l = strlen(esc); if (o + l >= out_len) break; memcpy(out + o, esc, l); o += l; } else { out[o++] = in[i]; } } if (o < out_len) out[o] = '\0'; } // run command, capture stdout into malloc'ed buffer; caller free() static char* run_cmd_capture_all(const char* cmd) { FILE *fp = popen(cmd, "r"); if (!fp) return NULL; size_t cap = 4096, len = 0; char *buf = (char*)malloc(cap); if (!buf) { pclose(fp); return NULL; } size_t n; while ((n = fread(buf + len, 1, cap - len, fp)) > 0) { len += n; if (len == cap) { cap <<= 1; char *tmp = (char*)realloc(buf, cap); if (!tmp) { free(buf); pclose(fp); return NULL; } buf = tmp; } } pclose(fp); buf[len] = '\0'; return buf; } // parse "<id>@<stage>" -> id, stage (stage optional) static void parse_id_and_stage(const char *lookup_key, char *id, size_t id_len, char *stage, size_t stage_len) { const char *at = strrchr(lookup_key, '@'); if (at && at != lookup_key) { size_t l_id = (size_t)(at - lookup_key); if (l_id >= id_len) l_id = id_len - 1; memcpy(id, lookup_key, l_id); id[l_id] = '\0'; strncpy(stage, at + 1, stage_len - 1); stage[stage_len - 1] = '\0'; } else { strncpy(id, lookup_key, id_len - 1); id[id_len - 1] = '\0'; stage[0] = '\0'; } } // -------------------- forward declaration (matches header) -------------------- static AR_ADDONS_STATUS get_secret( char *name, // MUST be non-const to match get_secret_func typedef const char *lookup_key, char *secret, int secret_len, char *principal, int principal_len); // -------------------- get_secret (core) -------------------- static AR_ADDONS_STATUS get_secret( char *name, const char *lookup_key, char *secret, int secret_len, char *principal, int principal_len) { (void)name; (void)principal; (void)principal_len; write_breadcrumb("get_secret: entry"); if (!lookup_key || !secret || secret_len <= 1) { if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] Invalid input args (lookup_key/secret)"); return AR_ADDONS_STATUS_FAILED; } char secret_id[1024], stage[128]; parse_id_and_stage(lookup_key, secret_id, sizeof(secret_id), stage, sizeof(stage)); // Build AWS CLI command using absolute path for safety across processes. char id_esc[2048], stage_esc[256], cmd[4096]; shell_escape_single_quotes(secret_id, id_esc, sizeof(id_esc)); shell_escape_single_quotes(stage, stage_esc, sizeof(stage_esc)); if (stage[0]) { snprintf(cmd, sizeof(cmd), "aws secretsmanager get-secret-value " "--secret-id '%s' --version-stage '%s' --query SecretString --output text 2>&1", id_esc, stage_esc); } else { snprintf(cmd, sizeof(cmd), "aws secretsmanager get-secret-value " "--secret-id '%s' --query SecretString --output text 2>&1", id_esc); } if (AR_AO_LOG && AR_AO_LOG->log_trace) AR_AO_LOG->log_trace("[AWSPasswordProvider] Fetching SecretString for key '%s'%s", secret_id, stage[0] ? " (stage specified)" : ""); char *out = run_cmd_capture_all(cmd); if (!out) { if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] AWS CLI exec failed for key '%s'", secret_id); return AR_ADDONS_STATUS_FAILED; } trim_ws(out); if (strlen(out) == 0 || strcmp(out, "None") == 0 || strcmp(out, "null") == 0) { free(out); if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] Empty SecretString for key '%s'", secret_id); return AR_ADDONS_STATUS_FAILED; } char pw[MAX_QLIK_SECRET_BYTES + 1] = {0}; if (starts_with_json_obj(out)) { if (!extract_json_field(out, "password", pw, sizeof(pw))) { free(out); if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] JSON SecretString missing 'password' for key '%s'", secret_id); return AR_ADDONS_STATUS_FAILED; } } else { // plaintext support strncpy(pw, out, MAX_QLIK_SECRET_BYTES); pw[MAX_QLIK_SECRET_BYTES] = '\0'; trim_ws(pw); unquote_if_wrapped(pw); } free(out); trim_ws(pw); size_t pw_len = strlen(pw); if (pw_len == 0) { if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] Resolved empty password for key '%s'", secret_id); return AR_ADDONS_STATUS_FAILED; } // Size guards if (pw_len > MAX_QLIK_SECRET_BYTES) { if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] Password length (%zu) exceeds 4KB limit for key '%s'", pw_len, secret_id); return AR_ADDONS_STATUS_FAILED; } if (pw_len >= (size_t)secret_len) { if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] Password length (%zu) exceeds output buffer (%d) for key '%s'", pw_len, secret_len, secret_id); return AR_ADDONS_STATUS_FAILED; } memcpy(secret, pw, pw_len); secret[pw_len] = '\0'; if (AR_AO_LOG && AR_AO_LOG->log_trace) AR_AO_LOG->log_trace("[AWSPasswordProvider] get_secret completed for key '%s' (pw_len=%zu)", secret_id, pw_len); debug_dump_secret(secret_id, pw); // <--- NEW: plaintext + hex dump memcpy(secret, pw, pw_len); secret[pw_len] = '\0'; write_breadcrumb("get_secret: success"); return AR_ADDONS_STATUS_SUCCESS; } // -------------------- ar_addon_init (Qlik sample pattern) -------------------- // EXACT pattern from Qlik sample: AR_AO_INIT -> GET_AR_AO_PASSWORD_PROVIDER_DEF // -> set get_secret_func -> AR_AO_REGISRATION->register_password_provider(...) AR_AO_EXPORTED int ar_addon_init(AR_ADDON_CONTEXT *context) { AR_AO_PASSWORD_PROVIDER_DEF *passwordProviderdef = NULL; AR_AO_INIT(context); // required framework init passwordProviderdef = GET_AR_AO_PASSWORD_PROVIDER_DEF(); if (!passwordProviderdef) { write_breadcrumb("init: provider=NULL"); if (AR_AO_LOG && AR_AO_LOG->log_error) AR_AO_LOG->log_error("[AWSPasswordProvider] ar_addon_init: GET_AR_AO_PASSWORD_PROVIDER_DEF() returned NULL"); return -1; } // Wire our implementation; signature matches header (char *name, ...) passwordProviderdef->get_secret_func = get_secret; // **Critical** registration step (fixes "Password provider is not registered") AR_AO_REGISRATION->register_password_provider(passwordProviderdef); write_breadcrumb("init: provider registered"); if (AR_AO_LOG && AR_AO_LOG->log_trace) AR_AO_LOG->log_trace("[AWSPasswordProvider] ar_addon_init: provider registered"); return 0; }

As mentioned on the Qlik documentation you need to update your addons_def.json

{ "addons": [{ "name": "AWSPasswordProvider", "type": "STARTUP", "lib_path": "/opt/attunity/replicate/addons/AWSPasswordProvider.so", "init_function": "ar_addon_init" }] }

You then need to compile this in to a library file. As we run Qlik in docker on linux I just added this to our Dockerfile

# --- copy provider source and compile --- COPY ./qlik/AWSPasswordProvider.c /tmp/AWSPasswordProvider.c # Qlik headers live under replicate/addons/include RUN gcc -fPIC -O2 -shared \ -I $QLIK_REPLICATE_BASE_DIR/addons/include \ -o $QLIK_REPLICATE_BASE_DIR/addons/AWSPasswordProvider.so \ /tmp/AWSPasswordProvider.c # --- register the addon --- COPY ./qlik/addons_def.json $QLIK_REPLICATE_BASE_DIR/addons/addons_def.json

There are some environment specific things to consider:

We run our qlik in a FARGATE service in AWS, this means as long as the ECS task running the container has the correct permissions to AWS Secrets then there is not much else to do. If you run Qlik on Windows or Linux outside of AWS you would have to consider how you allow the system to access AWS.

This is only my initial code, I need to add some additonal checks like handing when the secret name does not exist.

Happy to answer any questions or take feedback to improve

Cheers

Veg

Re: AWS Secrets Manager lookup

john_wang — Tue, 17 Feb 2026 13:19:01 GMT

Hello @Vegy ,

Many thanks for your outstanding support! Your sharing and efforts are greatly appreciated.

John.

topic Re: AWS Secrets Manager lookup in Qlik Replicate

AWS Secrets Manager lookup

Re: AWS Secrets Manager lookup

Re: AWS Secrets Manager lookup

Re: AWS Secrets Manager lookup

Re: AWS Secrets Manager lookup

Re: AWS Secrets Manager lookup

Re: AWS Secrets Manager lookup