Review Mode Bonus Set 3 — incorrect answers

[Diyana]

I highlighted my feedback to your answers in italics below:

Review Mode Bonus Set 3 Question 10:

“Deterministic Request Hashing”: as with any hashing goes: slight changes to a text — adding a space, adding even adding a newline would actually change the hash value of that prompt.

Review Mode Bonus Set 3 Question 32:

Eventbridge could trigger Sagemaker AI pipelines natively, directly. You don’t need step functions or invoke intermediate lambda from EventBridge:

https://docs.aws.amazon.com/sagemaker/latest/dg/pipeline-eventbridge.html

Q41: unnamed 3rd party vector — you have to name the exact vector store because direct bedrock knowledge bases integration is dependent on the exact vector store brands/company/type. If it’s not listed, then, technically, you have to migrate your vector store to aws’ supported ones just to use aws bedrock knowledge base. I had this one wrong because I wasn’t sure if your unnamed 3rd party vector store (I don’t ‘assume’) is supported by aws and the answer “Switch the retrieval workflow to Bedrock Knowledge Bases and activate reranking so that retrieved chunks are reordered by relevance before generation.” Implies moving everything to aws, despite the fact that it’s accurately generating responses felt an overblown effort.

Q47:

Chosen answers:

– Step 3: Run controlled A/B testing to directly compare outputs from the new FM and the current production model.

– Step 4: Use AWS Step Functions to enforce automated approval checkpoints before advancing the workflow.

Step 3 & 4 are switched — before you run A/B testing which directly putting your newly trained model with production traffic, you need to actually ask permission (hence step function to ask approval) before putting that new model out for testing in production. That’s how real workplace work.

Q50:

The question mentioned:

The application must consistently deliver responses within 2 seconds to maintain user experience SLAs (Service Level Agreements).

Your answer said: Optimize inference efficiency by aggregating multiple prompts into batched requests and enable Cross-Region Inference in Bedrock using geography-aware inference profiles. Configure the application to invoke Region-specific endpoints that automatically distribute requests across available capacity within each geography.

Batch inferencing is never going to make the 2 seconds SLA because it’s asynchronous. And also, have you considered the security repercussions of batching prompts from different users?

Q58:

The question asked: The application must support multi-turn conversation context across interactions, ensuring that previous exchanges are preserved when generating responses.

The answer chosen as correct:

Use the Amazon Bedrock Converse API with IAM role-based authentication. Preserve multi-turn conversation context by including previous messages in each request payload. Employ language-specific SDKs across Lambda and EKS to ensure consistent access to the same foundation model (FM).

Including previous messages would actually have compounding effect on the payload. That payload would get incrementally bigger as the conversation goes on. It would eventually hit token limit, if you’re not careful enough. You need to actually summarize that previous messages.

[JR-TutorialsDojo]

Hello Diyana,

Please note that all of our scenarios, as well as the options, in our practice tests are randomized. Could you kindly share with us the first statement of the scenario you’re referring to?

• Q41:

• Review Mode Bonus Set 3 Question 32:

[Nikee-TutorialsDojo]

Hi Dinaya,

I hope you’re doing well!

Thank you for your feedback, and I understand your point about the ordering of the steps. I wanted to clarify the reasoning behind the corrected order and why it’s structured the way it is for the model evaluation workflow.

Here’s why the order works:

1. Define clear evaluation criteria that measure attributes such as correctness, relevance, and linguistic quality.
   • This step should always come first because it establishes the standards by which you will evaluate the model’s performance. Without a defined evaluation framework, you wouldn’t know how to assess the success of the new model effectively. It’s essential to set the evaluation criteria upfront to ensure that every subsequent step is aligned with your goals.
2. Assemble a benchmark dataset containing varied prompts, rare scenarios, and regulatory edge cases.
   • Once you have the evaluation criteria in place, you need to gather the right data. The benchmark dataset should be designed to cover all the scenarios (including rare ones) and regulatory edge cases, ensuring a comprehensive evaluation. This dataset should be curated to ensure it aligns with the criteria you’ve just set.
3. Run controlled A/B testing to directly compare outputs from the new FM and the current production model.
   • After defining the criteria and assembling the dataset, it’s time to perform A/B testing. This step involves running both the new and current models against the benchmark dataset to measure and compare their performance. It’s critical to run this test in a controlled environment to observe the results without introducing bias.
4. Use AWS Step Functions to enforce automated approval checkpoints before advancing the workflow.
   • Following the A/B testing, AWS Step Functions are used to automate the workflow and enforce approvals before moving forward. This step ensures that all stakeholders have signed off on the results of the A/B test and that there is a structured process for advancing the model deployment.
5. Review the comparison data, summarize findings, and generate a formal evaluation report with performance conclusions.
   • The final step is to analyze the comparison data, summarize the findings from the A/B tests, and document the results in a formal evaluation report. This is where you draw conclusions about whether the new model performs better than the current production model and whether it meets the desired performance criteria.

Why this order makes sense:

Steps 1 and 2 logically set the stage for Step 3 (A/B testing). You can’t effectively compare the models without first defining what success looks like and gathering the data that will test those criteria. Step 4 ensures that the workflow is automated and that appropriate approvals are in place, helping to maintain governance and compliance standards, which is crucial before any changes are made in a production environment. Finally, Step 5 wraps everything up by reviewing the results and making a formal decision based on the evaluation.

I hope this clears up the reasoning behind the order! If you have any more questions or need further clarification, please feel free to reach out.

Best regards,

Nikee @ Tutorials Dojo