Page 1 of 1

Elementary Python SDK

Elementary community members, we'd love to hear what you need!

We're considering expanding beyond dbt to give you full visibility into your data, starting from the source and before it reaches the warehouse.

Our focus is on two main goals:

- Making it possible to monitor non-dbt assets like raw tables and external sources

- Helping you identify the real root cause of issues, whether it's an upstream data problem, a failing Airflow task, or a job that timed out

To support this, we're exploring the idea of building a Python SDK. We'd love to hear your thoughts, ideas, and need. Your input will help shape what we build next.

Do you have any assets outside dbt?

Yes

Do you have data pipelines in Python?

Yes

How do you create/update/insert data to these non-dbt data assets?

Pure Python code

Python that runs native SQL

Python that uses libraries like SQLALchemy

Other

Do you monitor your existing non-dbt data pipelines (the pipeline itself)? If so, how?

I don’t monitor it, and I’m actively looking for a solution

I don’t monitor it

APM like Datadog/Elastic/NewRellic

Home grown solution - please explain

Do you monitor your non-dbt data assets? If so, how?

I don’t monitor it

I don’t monitor it, and I’m actively looking for a solution

Hacked Elementary to support it

Using another data observability tool

Using another OSS python data quality framework like Great Expectations / Dagster / Checks / DQX / Else

Do you have only data pipelines in batch or also real time streaming? Please share any relevant information.

We'd love to know your name to reach out on Slack for more thoughts about the Python SDK.

Thank you for your time!