Page 1 of 1

LLM Usage and Evaluation Survey

This survey aims to understand users’ experiences, preferences, and evaluations of large language models (LLMs), including which models they use, how frequently they use them, and how they perceive their performance.


The survey is anonymous, and all responses will be used for research and analysis purposes only. There are no right or wrong answers, so please feel free to answer honestly based on your own experience.

Section 1. Usage

01. Which large language models have you used?

01. Which large language models have you used?

02. Which large language models have you paid for?

02. Which large language models have you paid for?

03. Which large language models are you still paying for?

03. Which large language models are you still paying for?

04. Please rank these models by how often you use them

Please provide a rough ranking based on your usage frequency. If some models are tied, the exact order does not matter. Models you have not used can be placed near the end in their original order.
If you have not used any of the listed models, please rank “I have not used any” first. If you use the listed models about equally often, please rank “I use them about equally” first.
04. Please rank these models by how often you use them
Use the up and down arrow keys to change the rank.
1
1
1
1
1
1
1
1
1
1
1
1
1
1

05. Please rank these models by how much you like them

Please provide a rough ranking based on your usage frequency. If some models are tied, the exact order does not matter. Models you have not used can be placed near the end in their original order.
If you have not used any of the listed models, please rank “I have not used any” first. If you don't have a clear preference on any of them, please rank “No clear preference” first.
05. Please rank these models by how much you like them
Use the up and down arrow keys to change the rank.
1
1
1
1
1
1
1
1
1
1
1
1
1
1

06. Which specific model series or versions do you use right now?

If you are not sure or have never paid attention to the exact model version, just write the name or series you remember. If you do not know, you may leave it blank or write “Not sure.”

07. How often do you use large language models?

07. How often do you use large language models?
A
B
C
D
E

08. Do you cross-check answers across different models?

08. Do you cross-check answers across different models?
A
B
C

09. Do you feel that you rely on LLMs?

09. Do you feel that you rely on LLMs?
A
B
C

10. How much are you willing to spend per month on paid LLM services?

The followings are in US dollars
10. How much are you willing to spend per month on paid LLM services?
A
B
C
D
E
F
G
H
I
J
K

11. Which AI IDEs or plugins do you use?

11. Which AI IDEs or plugins do you use?

12. Is there any model or feature that particularly impressed you?

Section 2. Model Choice by Use Case

13. Which large language models have you used?

13. Which large language models have you used?

14. When you do in-depth research, write papers, or prepare reports, which LLMs do you prefer?

14. When you do in-depth research, write papers, or prepare reports, which LLMs do you prefer?

15. When you write or debug code, which LLMs do you prefer?

15. When you write or debug code, which LLMs do you prefer?

16. When you process long texts or read academic papers, which LLMs do you prefer?

16. When you process long texts or read academic papers, which LLMs do you prefer?

17. When you do creative writing or brainstorming, which LLMs do you prefer?

17. When you do creative writing or brainstorming, which LLMs do you prefer?

18. What are your thoughts on the current state of large language models?

Feel free to share anything you want.

Section 3. Basic Information

This survey is anonymous. We do not collect any personally identifying information, so please feel free to answer honestly.

19. Which country or region do you currently live in or stay in long-term?

19. Which country or region do you currently live in or stay in long-term?
A
B
C
D
E
F
G
H
I

20. What is your gender?

20. What is your gender?
A
B
C
D

21. What is your age group?

21. What is your age group?
A
B
C
D
E
F
G
H
I
J

22. What is your occupation?

22. What is your occupation?
A
B
C
D
E
F
G
H
I
J

Section 4. Evaluation of Common Models

Please rate only the models you have used. You may skip any models you have not used.

23. Please evaluate ChatGPT

1 = lowest score; 5 = highest score.
1
2
3
4
5
Information accuracy
Mathematical ability
Coding ability
Literature review / synthesis ability
Writing ability
Response speed
Hallucination frequency
Image generation ability

24. Please evaluate Claude

1 = lowest score; 5 = highest score.
1
2
3
4
5
Information accuracy
Mathematical ability
Coding ability
Literature review / synthesis ability
Writing ability
Response speed
Hallucination frequency
Image generation ability

25. Please evaluate Gemini

1 = lowest score; 5 = highest score.
1
2
3
4
5
Information accuracy
Mathematical ability
Coding ability
Literature review / synthesis ability
Writing ability
Response speed
Hallucination frequency
Image generation ability

26. Please evaluate Deepseek

1 = lowest score; 5 = highest score.
1
2
3
4
5
Information accuracy
Mathematical ability
Coding ability
Literature review / synthesis ability
Writing ability
Response speed
Hallucination frequency
Image generation ability

27. Please evaluate Qwen

1 = lowest score; 5 = highest score.
1
2
3
4
5
Information accuracy
Mathematical ability
Coding ability
Literature review / synthesis ability
Writing ability
Response speed
Hallucination frequency
Image generation ability

28. Please evaluate Kimi

1 = lowest score; 5 = highest score.
1
2
3
4
5
Information accuracy
Mathematical ability
Coding ability
Literature review / synthesis ability
Writing ability
Response speed
Hallucination frequency
Image generation ability

29. Anything else you'd like to share about LLM usage experience?