[{"content":"DataFusion keeps its core function library small. Aggregates like mode, skewness and kurtosis live in the Rust-only datafusion-extra-functions crate. I wanted them in Python without forking datafusion-python, and it turns out the whole bridge fits in one small lib.rs: datafusion-extra-functions-ffi.\nThe protocol datafusion-python accepts foreign UDFs through PyCapsule protocols, the same pattern Arrow uses for __arrow_c_stream__. When you pass an object to udaf(), it looks for a __datafusion_aggregate_udf__ method. That method must return a PyCapsule named datafusion_aggregate_udf wrapping an FFI_AggregateUDF, a stable-ABI struct from the datafusion-ffi crate. Your extension module and datafusion-python stay separate compiled artifacts, the capsule is the only contract between them.\nThe implementation Wrap the native AggregateUDF in a PyO3 class and implement the dunder:\nuse datafusion_ffi::udaf::FFI_AggregateUDF; use pyo3::types::PyCapsule; #[pyclass] pub struct ExtraAggregateUDF { inner: Arc\u0026lt;AggregateUDF\u0026gt;, } #[pymethods] impl ExtraAggregateUDF { fn __datafusion_aggregate_udf__\u0026lt;\u0026#39;py\u0026gt;( \u0026amp;self, py: Python\u0026lt;\u0026#39;py\u0026gt;, ) -\u0026gt; PyResult\u0026lt;Bound\u0026lt;\u0026#39;py, PyCapsule\u0026gt;\u0026gt; { let name = CString::new(\u0026#34;datafusion_aggregate_udf\u0026#34;).unwrap(); let provider = FFI_AggregateUDF::from(Arc::clone(\u0026amp;self.inner)); PyCapsule::new(py, provider, Some(name)) } } FFI_AggregateUDF::from does the heavy lifting: it turns the trait object into a #[repr(C)] vtable-style struct that survives crossing a shared library boundary. The rest is a name lookup over all_extra_aggregate_functions() and a #[pymodule] exporting it. There are sibling structs for scalar and window UDFs, table providers and catalogs, so the same recipe covers most extension points.\nOn the Python side:\nfrom datafusion import udaf import datafusion_extra_functions_ffi as ffi skew = udaf(ffi.udaf_by_name(\u0026#34;skewness\u0026#34;)) Packaging and the ABI trap maturin builds the cdylib into a wheel. With PyO3\u0026rsquo;s abi3-py310 feature one wheel per platform covers every Python from 3.10 up, no per-version builds.\nThe trap: FFI_AggregateUDF is stable across Python versions, but not across DataFusion versions. Its layout changes with the DataFusion major, and a mismatch between your crate and the one inside datafusion-python fails at runtime with no clear error. Compile against the same major that datafusion-python bundles and pin the Python dependency to it, \u0026gt;=54,\u0026lt;55 in my case. Bumping DataFusion means rebuilding and re-pinning, there is no way around it.\n","permalink":"https://s5dsn-eqee.github.io/notes/datafusion-extra-functions-ffi/","summary":"How to bridge a Rust-only DataFusion crate into datafusion-python with datafusion-ffi, PyO3 and a single PyCapsule dunder, no fork of datafusion-python needed.","title":"Exposing Rust DataFusion UDFs to Python via the PyCapsule protocol"},{"content":"While integrating custom OIDC with OpenMetadata deployed via Helm + ArgoCD, I ran into strange behavior: auth config changes were synced successfully, present in the Secret and the pod environment, yet completely ignored by the server. No errors anywhere.\nTurns out OpenMetadata persists its security configuration in the database since v1.9, and the DB silently takes precedence over YAML/env config. Once the first boot seeds that row, your Helm values are ignored, even after full pod recreation.\nThe undocumented fix:\nopenmetadata-ops.sh remove-security-config then restart.\nI filed an issue with details and suggestions (startup warning, documented precedence, source-of-truth switch): open-metadata/OpenMetadata#31786.\nIf your OpenMetadata SSO config is \u0026ldquo;green everywhere, applied nowhere\u0026rdquo;, check openmetadata_settings in the database.\n","permalink":"https://s5dsn-eqee.github.io/notes/openmetadata-db-config-precedence/","summary":"OpenMetadata silently prefers DB-persisted security config over YAML/env, so your Helm values can be ignored with no errors anywhere.","title":"OpenMetadata SSO config: green everywhere, applied nowhere"}]