ZenML LLMOps Roadmap Feedback

Welcome! As one of our 5,000 newsletter subscribers, you're part of the community that's been with us since the beginning.

Now we need your help again—this time to bring the same MLOps standardization to the chaos of LLM development.

Your 5 minutes here will shape the tools we all use tomorrow.

What's your current role?

A

DevOps / Platform Engineer

B

AI / LLM Engineer

C

ML Engineer

D

Engineering Manager / Lead

E

Data Scientist

F

Other

Which challenges are you currently facing with LLM/Agent development?

A

Managing costs across multiple LLM providers

B

Evaluating agent performance systematically

C

Debugging agent failures in production

D

Integrating LLMs with existing ML pipelines

E

Transitioning from dev → staging → prod

F

Compliance/governance requirements

G

Tracking prompts and their versions

H

Other

How mature is your organization's LLM/Agent deployment?

A

Just experimenting/prototyping

B

Planning to work on it

C

Running pilots with select users

D

Production deployment (limited scope)

E

Production at scale

F

Not using LLMs/Agents yet

What best describes the complexity of your LLM/Agent use cases?

We're trying to understand the technical depth of what you're building

What best describes the complexity of your LLM/Agent use cases?

A

Basic automation & assistance - Simple Q&A, content screening, email drafts, meeting summaries

B

Structured workflow automation - Multi-step processes with defined inputs/outputs, template-based generation, basic RAG

C

Complex reasoning & analysis - Research agents, code generation, data analysis, decision support systems

D

Advanced autonomous systems - Self-improving agents, complex multi-agent orchestration, production code deployment

E

Hybrid ML + LLM systems - Combining traditional ML models with LLMs, complex pipelines with multiple AI components

F

Mostly internal/administrative use - HR screening, documentation, internal knowledge management

G

Not using LLMs/Agents yet

Rank these proposed ZenML LLM components by importance to you

Drag and drop to rank from 1 (most important) to 4 (least important) based on which of these (proposed) ZenML Stack Components would provide the most value for your workflows.

Rank these proposed ZenML LLM components by importance to you

1

LLM Provider Component - Unified API management, automatic cost tracking, and provider fallbacks (e.g., LiteLLM, Portkey)

1

Prompt Manager - Version control and templating for prompts across your pipelines

1

LLM Evaluator - Standardized interface, LLM-as-judge, visualizations in the ZenML dashboard (e.g. OpenAI evals, RAGAS, deepeval)

1

LLM Experiment Tracker - Full observability and session tracing for LLM calls (e.g., LangSmith, Langfuse, Braintrust)

What type of LLM/Agent applications are you building?

A

Internal automation/workflows

B

Data analysis/research agents

C

Hybrid (traditional ML + LLM)

D

Content generation

E

Code generation/assistance

F

Customer service/chatbots

G

Not building anything yet

H

Other

How do you prefer to deploy LLM agents in production?

Think about how agents fit into your overall system architecture

How do you prefer to deploy LLM agents in production?

A

Orchestrated steps in pipelines/DAGs - Agents are called as part of scheduled or triggered workflows with clear inputs/outputs

B

Long-running autonomous services - Agents run continuously and independently, handling requests as they come

C

Hybrid approach - Some agents in pipelines, others as services depending on use case

D

Event-driven functions - Agents triggered by specific events but not part of larger workflows

E

Not sure yet - Still figuring out the best pattern

F

Other

Do you currently combine traditional ML with LLMs in your pipelines?

A

Yes, extensively

B

Yes, occasionally

C

No, but planning to

D

No, they're separate systems

E

Not applicable

Why do you keep traditional ML and LLM systems separate? (Optional)

This helps us understand integration challenges

Why do you keep traditional ML and LLM systems separate? (Optional)

A

By design - they solve different problems - Our ML and LLM use cases are fundamentally different and don't need to interact

B

Different teams own them - ML team uses different tools/infrastructure than the LLM/AI team

C

Lack of unified tooling - We'd prefer integrated pipelines but can't find tools that handle both well

D

Legacy infrastructure - Our ML infrastructure predates LLMs and is too complex to migrate

E

Technical challenges - Incompatible frameworks, different deployment requirements, latency concerns

F

Cost/resource constraints - Running unified pipelines would be too expensive or resource-intensive

G

Governance/compliance requirements - Different approval processes, data handling requirements, or risk profiles

H

Other

Which tools are you currently using for LLMOps?

What's your biggest frustration with current LLMOps tooling?

If ZenML offered the LLMOps features described above, how likely would you be to adopt them?

What timeline are you working with for improving your LLMOps practices?

A

Urgent (next 1-3 months)

B

Near-term (3-6 months)

C

Medium-term (6-12 months)

D

Long-term (12+ months)

E

No specific timeline

What's one LLMOps capability you wish existed that no tool currently provides?

Would you be interested in:

A

Beta testing new LLMOps features

B

Participating in a user interview (30 min)

C

Joining a LLMOps community/advisory group

D

Contributing to open source LLMOps patterns

E

None of the above

Size of the team(s) working on your LLMOps projects and features:

A

1-10 employees

B

11-50 employees

C

51-200 employees

D

201-1000 employees

E

1000+ employees

ZenML LLMOps Roadmap Feedback

What's your current role?

Which challenges are you currently facing with LLM/Agent development?

How mature is your organization's LLM/Agent deployment?

What best describes the complexity of your LLM/Agent use cases?

Rank these proposed ZenML LLM components by importance to you

What type of LLM/Agent applications are you building?

How do you prefer to deploy LLM agents in production?

Do you currently combine traditional ML with LLMs in your pipelines?

Why do you keep traditional ML and LLM systems separate? (Optional)

Which tools are you currently using for LLMOps?

What's your biggest frustration with current LLMOps tooling?

If ZenML offered the LLMOps features described above, how likely would you be to adopt them?

What timeline are you working with for improving your LLMOps practices?

What's one LLMOps capability you wish existed that no tool currently provides?

Would you be interested in:

Size of the team(s) working on your LLMOps projects and features:

If you'd be ok with us following up with you, feel free to submit your email below

Anything else you'd like to tell us?