Databricks-Generative-AI-Engineer-Associate Free Practice Test

A Generative Al Engineer has created a RAG application to look up answers to questions about a series of fantasy novels that are being asked on the author??s web forum. The fantasy novel texts are chunked and embedded into a vector store with metadata (page
number, chapter number, book title), retrieved with the user??s query, and provided to an LLM for response generation. The Generative AI Engineer used their intuition to pick the chunking strategy and associated configurations but now wants to more methodically choose the best values.
Which TWO strategies should the Generative AI Engineer take to optimize their chunking strategy and parameters? (Choose two.)

1. A. Change embedding models and compare performance.
2. B. Add a classifier for user queries that predicts which book will best contain the answe
3. C. Use this to filter retrieval.
4. D. Choose an appropriate evaluation metric (such as recall or NDCG) and experiment with changes in the chunking strategy, such as splitting chunks by paragraphs or chapter
5. E. Choose the strategy that gives the best performance metric.
6. F. Pass known questions and best answers to an LLM and instruct the LLM to provide the best token coun
7. G. Use a summary statistic (mean, median, etc.) of the best token counts to choose chunk size.
8. H. Create an LLM-as-a-judge metric to evaluate how well previous questions are answered by the most appropriate chun
9. I. Optimize the chunking parameters based upon the values of the metric.

Correct Answer: CE

A Generative Al Engineer is building a production-ready LLM system which replies directly to customers. The solution makes use of the Foundation Model API via provisioned throughput. They are concerned that the LLM could potentially respond in a toxic or otherwise unsafe way. They also wish to perform this with the least amount of effort.
Which approach will do this?

1. A. Host Llama Guard on Foundation Model API and use it to detect unsafe responses
2. B. Add some LLM calls to their chain to detect unsafe content before returning text
3. C. Add a regex expression on inputs and outputs to detect unsafe responses.
4. D. Ask users to report unsafe responses

Correct Answer: A

A Generative Al Engineer is developing a RAG application and would like to experiment with different embedding models to improve the application performance.
Which strategy for picking an embedding model should they choose?

1. A. Pick an embedding model trained on related domain knowledge
2. B. Pick the most recent and most performant open LLM released at the time
3. C. pick the embedding model ranked highest on the Massive Text Embedding Benchmark (MTEB) leaderboard hosted by HuggingFace
4. D. Pick an embedding model with multilingual support to support potential multilingual user questions

Correct Answer: A

A Generative AI Engineer received the following business requirements for an external chatbot.
The chatbot needs to know what types of questions the user asks and routes to appropriate models to answer the questions. For example, the user might ask about upcoming event details. Another user might ask about purchasing tickets for a particular event.
What is an ideal workflow for such a chatbot?

1. A. The chatbot should only look at previous event information
2. B. There should be two different chatbots handling different types of user queries.
3. C. The chatbot should be implemented as a multi-step LLM workflo
4. D. First, identify the type of question asked, then route the question to the appropriate mode
5. E. If it??s an upcoming event question, send the query to a text-to-SQL mode
6. F. If it??s about ticket purchasing, the customer should be redirected to a payment platform.
7. G. The chatbot should only process payments

Correct Answer: C

A Generative Al Engineer is building an LLM-based application that has an important transcription (speech-to-text) task. Speed is essential for the success of the application
Which open Generative Al models should be used?

1. A. L!ama-2-70b-chat-hf
2. B. MPT-30B-lnstruct
3. C. DBRX
4. D. whisper-large-v3 (1.6B)

Correct Answer: D