Authored by Dr. Tiziana Ligorio for AI Agents - CSCI 395.32 taught at Hunter College of The City University of New York¶
In this demo, we build a simple AI workflow that demonstrates a single round of reflection. One model call produces a draft, a second critiques it, and a third revises it.
Our workflow consists of three steps, each handled by an LLM with a distinct role:
Report Writer (think journalist) — Generates an initial report based on a user-provided topic
Critic (think editor) — Provides detailed critique and improvement suggestions
Reviser (think senior reporter) — Takes the original report and critique to produce a final version
The purpose is to see how outputs from multiple LLM calls can be connected. A reflection workflow is more elaborate than a single call, but it is not automatically better. We will inspect the draft, critique, and revision to decide what changed and whether the change helped.
We use the OpenAI SDK directly, with OpenRouter as our provider, to keep the implementation transparent: no orchestration framework, just API calls that can be followed step by step.
Installs and Imports¶
%%capture
%pip install -q "openai>=2.0.0" "python-dotenv>=1.0.0"
%%capture hides installation output to keep the notebook readable. %pip installs packages into the Python environment used by the current notebook kernel.
# OpenAI SDK: provides a clean interface for calling LLMs
# We'll use it with OpenRouter, which is OpenAI API-compatible
from openai import OpenAI
# Standard library
import osOpenRouter provides access to models from multiple providers through one API. Model availability, pricing, and free-model options can change, so check the current OpenRouter model list before choosing a model.
This notebook follows OpenRouter’s documented integration with the OpenAI SDK. Other services may also describe themselves as OpenAI-compatible, but their supported endpoints and parameters can differ.
Setup your API Key¶
Step 1 — Get an OpenRouter API key¶
For this demo we will use an LLM via OpenRouter, which requires an API key.
Go to https://
openrouter .ai Sign in (or create an account if you don’t have one)
Once logged in, navigate to https://
openrouter .ai /settings /keys Click Create Key
Give the key a name, e.g.
colab-reflection-workflowCopy the key immediately (you won’t be able to see it again)
Important: Treat this key like a password. Do not share it, paste it into notebooks, or commit it to GitHub.
Step 2 — Store the API key¶
If running in Google Colab:¶
On the left sidebar, click 🔑 Secrets
Add a new secret:
Name:
OPENROUTER_API_KEYValue: your actual API key
Toggle the switch to give notebook access (you should see a checkmark)
If running locally:¶
Create a
.envfile in the project root:touch .envAdd the following (replace with your own key):
OPENROUTER_API_KEY=your_openrouter_key_here
Important: Never paste API keys directly into code cells.
Load the API Key¶
In Colab:¶
Uncomment and run the cell below if you’re using Google Colab.
# # Uncomment the lines below if running in Google Colab
# from google.colab import userdata
# os.environ["OPENROUTER_API_KEY"] = userdata.get("OPENROUTER_API_KEY")
# print("API key loaded from Colab Secrets")Locally:¶
Run the cell below if you’re running locally with a .env file.
# Load a local .env file when one is present.
# This does not replace a key already loaded from Colab Secrets.
from dotenv import load_dotenv
env_file_found = load_dotenv(dotenv_path=".env")
print("Local .env file found:", env_file_found)
Local .env file found: True
# Sanity check: verify that a non-empty API key is available.
api_key = os.getenv("OPENROUTER_API_KEY")
assert api_key, "OPENROUTER_API_KEY not found. Please check your setup."
print("OPENROUTER_API_KEY present:", bool(api_key))
OPENROUTER_API_KEY present: True
Initialize the LLM Client¶
We use the OpenAI SDK to communicate with OpenRouter. OpenRouter documents this SDK integration and provides an OpenAI-compatible Chat Completions endpoint.
This demo uses gpt-5.6-luna, a current OpenAI model that is cost-sensitive. It is not a source of verified facts: generated factual claims should still be checked, especially for changing topics.
Models do not all support the same request settings. OpenRouter’s current model information lists reasoning_effort for this model but does not list temperature, so this notebook does not send a temperature setting. If you choose another model, check its supported parameters in the OpenRouter model list.
# Initialize the OpenAI client pointing to OpenRouter.
client = OpenAI(
base_url="https://openrouter.ai/api/v1",
api_key=api_key,
)
# Use one model and one reasoning setting for all three roles.
MODEL = "openai/gpt-5.6-luna"
REASONING_EFFORT = "none"
print(f"Client initialized. Using model: {MODEL}")
print(f"Reasoning effort: {REASONING_EFFORT}")
Client initialized. Using model: openai/gpt-5.6-luna
Reasoning effort: none
Step 1: The Report Writer¶
Our workflow begins with the Report Writer — an LLM instructed to act as a journalist who writes an initial report on a given topic.
Giving the model a role¶
The system message supplies instructions that should guide the model when it responds to the user message. It is part of the list of messages sent in the API request.
The OpenAI SDK uses a message format like this:
messages = [
{"role": "system", "content": "You are a..."}, # Instructions for the model
{"role": "user", "content": "..."}, # The request and its context
]For our Report Writer:
System message: Instructs the LLM to behave as a journalist writing informative reports
User message: Supplies the topic and asks for a concise, well-structured report
The user message does not have to be the raw topic by itself. We can add context or instructions around it—for example, Write a concise, well-structured report on the following topic: {topic}.
# System prompt that defines the Report Writer persona
WRITER_SYSTEM_PROMPT = """You are a journalist who writes clear, informative reports.
When given a topic, write a short report (2-3 paragraphs) that:
- Provides a balanced overview of the topic
- Includes relevant context and background
- Is written in a professional, accessible style
Write only the report itself, without any preamble or meta-commentary."""def generate_initial_report(topic: str) -> str:
"""
Generate an initial report on the given topic using the Report Writer role.
Args:
topic: The subject for the report
Returns:
The generated report as a string
"""
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": WRITER_SYSTEM_PROMPT},
{
"role": "user",
"content": f"Write a concise, well-structured report on the following topic: {topic}",
},
],
reasoning_effort=REASONING_EFFORT,
)
return response.choices[0].message.content or ""
The function above:
Takes a
topicstringWraps it with a more specific request
Sends the user message together with the writer’s system message
Asks the model not to spend extra tokens on additional reasoning for this simple task
Returns the generated report text
Some models accept a temperature setting that adjusts how varied their output is. This model’s current OpenRouter parameter list does not include temperature, so we do not send one. Model outputs can still vary between runs.
For this demonstration, we will make one writer call, inspect its response object, and reuse that same response as the initial report.
Pedagogical note¶
The SDK returns more than the generated text. We will keep the complete response object so we can inspect its fields before extracting the report.
# Make one writer call and keep the complete response object.
topic = "The impact of artificial intelligence on healthcare"
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": WRITER_SYSTEM_PROMPT},
{
"role": "user",
"content": f"Write a concise, well-structured report on the following topic: {topic}",
},
],
reasoning_effort=REASONING_EFFORT,
)
response.model_dump(){'id': 'gen-1786026317-omgBfZSeUIYoRfNkAosH',
'choices': [{'finish_reason': 'stop',
'index': 0,
'logprobs': None,
'message': {'content': 'Artificial intelligence is reshaping healthcare by supporting diagnosis, treatment planning, drug discovery, and administrative work. Machine-learning systems can analyze medical images, identify patterns in patient records, and help clinicians detect conditions such as cancer or heart disease earlier. AI tools may also accelerate pharmaceutical research, personalize treatments, and automate routine tasks, potentially reducing costs and allowing healthcare professionals to focus more on patient care.\n\nHowever, the benefits are accompanied by significant challenges. AI systems depend on large, high-quality datasets and can reproduce or amplify biases if certain populations are underrepresented. Concerns also remain about patient privacy, cybersecurity, transparency, and accountability when automated recommendations influence clinical decisions. AI is therefore most effective when used as a carefully regulated support tool, with human oversight, rigorous testing, and clear standards to ensure safety, fairness, and public trust.',
'refusal': None,
'role': 'assistant',
'annotations': None,
'audio': None,
'function_call': None,
'tool_calls': None,
'reasoning': None},
'native_finish_reason': 'completed'}],
'created': 1786026317,
'model': 'openai/gpt-5.6-luna',
'object': 'chat.completion',
'moderation': None,
'service_tier': 'default',
'system_fingerprint': None,
'usage': {'completion_tokens': 168,
'prompt_tokens': 99,
'total_tokens': 267,
'completion_tokens_details': {'accepted_prediction_tokens': None,
'audio_tokens': 0,
'reasoning_tokens': 0,
'rejected_prediction_tokens': None,
'image_tokens': 0},
'prompt_tokens_details': {'audio_tokens': 0,
'cache_write_tokens': 0,
'cached_tokens': 0,
'video_tokens': 0},
'cost': 0.0001107,
'is_byok': False,
'cost_details': {'upstream_inference_cost': 0.0001107,
'upstream_inference_prompt_cost': 9.9e-06,
'upstream_inference_completions_cost': 0.0001008}},
'provider': 'OpenAI'}Now extract the report text from the same response. This avoids paying for a second writer call and ensures that the response object above corresponds to the report used in the rest of the workflow.
The generate_initial_report function packages the same request pattern for reuse with other topics.
# Reuse the text from the response we just inspected.
initial_report = response.choices[0].message.content or ""
initial_report
'Artificial intelligence is reshaping healthcare by supporting diagnosis, treatment planning, drug discovery, and administrative work. Machine-learning systems can analyze medical images, identify patterns in patient records, and help clinicians detect conditions such as cancer or heart disease earlier. AI tools may also accelerate pharmaceutical research, personalize treatments, and automate routine tasks, potentially reducing costs and allowing healthcare professionals to focus more on patient care.\n\nHowever, the benefits are accompanied by significant challenges. AI systems depend on large, high-quality datasets and can reproduce or amplify biases if certain populations are underrepresented. Concerns also remain about patient privacy, cybersecurity, transparency, and accountability when automated recommendations influence clinical decisions. AI is therefore most effective when used as a carefully regulated support tool, with human oversight, rigorous testing, and clear standards to ensure safety, fairness, and public trust.'Step 2: The Critic¶
Now that we have an initial report, we pass it to the Critic — an LLM prompted to act as an editor who reviews the work and provides constructive feedback.
The Critic doesn’t rewrite the report. Instead, it identifies:
Areas that need clarification or more detail
Structural or logical issues
Missing perspectives or context
Ways to improve clarity and impact
This separation of concerns (writing vs. critiquing) is key to the reflection pattern. By having a dedicated critic, we get focused, actionable feedback rather than a muddled attempt to both evaluate and fix at once.
# System prompt that defines the Critic persona
CRITIC_SYSTEM_PROMPT = """You are an editor who provides constructive feedback on reports.
When given a report, provide a detailed critique that:
- Identifies areas that need more clarity or detail
- Points out any structural or logical issues
- Suggests missing perspectives or context that would strengthen the piece
- Offers specific, actionable suggestions for improvement
Be constructive and specific. Focus on how to make the report better, not just what's wrong with it.
Do not rewrite the report — only provide feedback."""def generate_critique(report: str) -> str:
"""
Generate a critique of the given report using the Critic role.
Args:
report: The report to critique
Returns:
The critique as a string
"""
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": CRITIC_SYSTEM_PROMPT},
{"role": "user", "content": f"Please critique the following report:\n\n{report}"},
],
reasoning_effort=REASONING_EFFORT,
)
return response.choices[0].message.content or ""
The output of Step 1 becomes the input to Step 2. This is the basic idea behind chaining LLM calls: each stage receives information produced by an earlier stage.
We use the same low-cost reasoning setting for the critic. Its focused behavior comes primarily from the critic’s instructions. The critique may still be incomplete or incorrect, so we will inspect what it says rather than assume it found every problem.
# Generate critique of the initial report
critique = generate_critique(initial_report)
critique'## Overall assessment\n\nThe report is clear, balanced, and appropriately concise. It identifies major healthcare applications of AI and acknowledges important risks rather than presenting the technology as universally beneficial. Its main limitation is that it remains broad and general: several claims would be stronger with concrete examples, distinctions between types of AI, and more discussion of how risks can be managed in practice.\n\n## Areas needing greater clarity or detail\n\n### 1. Define the scope of “AI”\n\nThe report uses “artificial intelligence” to cover many different technologies, including medical-image analysis, machine learning, drug discovery, and administrative automation. These applications have different levels of maturity, evidence, and risk.\n\n**Suggestion:** Briefly distinguish among:\n- Diagnostic or predictive systems\n- Generative AI and clinical documentation tools\n- Robotics and treatment-support systems\n- Administrative and operational applications\n\nThis would prevent readers from assuming that all AI tools have comparable capabilities or reliability.\n\n### 2. Qualify claims about effectiveness\n\nStatements such as “help clinicians detect conditions … earlier” and “personalize treatments” are plausible but broad. They could imply that AI has already demonstrated consistent improvements in real-world patient outcomes, which is not always the case.\n\n**Suggestion:** Clarify that performance may vary by disease, dataset, healthcare setting, and patient population. Distinguish between:\n- Accuracy in controlled studies\n- Effectiveness in clinical practice\n- Demonstrated improvements in patient outcomes\n\nMentioning that some systems may perform well technically but fail to improve care when poorly integrated into clinical workflows would add nuance.\n\n### 3. Explain the role of human oversight\n\nThe final sentence recommends human oversight, but it does not explain what effective oversight entails. Simply requiring a clinician to review an AI output may not prevent errors, particularly if users overtrust automated recommendations.\n\n**Suggestion:** Discuss practical safeguards, such as:\n- Clinicians being able to question or override recommendations\n- Training in appropriate AI use and limitations\n- Monitoring for errors after deployment\n- Clear procedures for reporting and correcting failures\n- Avoiding “automation bias,” in which users accept AI outputs uncritically\n\n### 4. Clarify accountability\n\nThe report mentions accountability but does not identify who should be responsible when an AI-supported decision causes harm.\n\n**Suggestion:** Address the responsibilities of developers, healthcare institutions, clinicians, regulators, and vendors. The report could note that accountability should not be left ambiguous when multiple parties contribute to the design, deployment, and use of a system.\n\n## Structural and logical considerations\n\n### 1. Improve the organization of benefits\n\nThe first paragraph moves quickly from diagnosis to treatment planning, drug discovery, and administration. This creates a useful overview but gives each area insufficient attention.\n\n**Suggestion:** Organize the benefits into categories, such as:\n1. Clinical diagnosis and prediction \n2. Treatment and drug development \n3. Operational and administrative support \n4. Patient-facing applications \n\nA brief example for each category would make the discussion easier to follow.\n\n### 2. Strengthen the transition from benefits to risks\n\n“However” provides a clear transition, but the relationship between specific benefits and specific risks could be more explicit. For example, the use of large datasets supports better prediction while simultaneously creating privacy and bias concerns.\n\n**Suggestion:** Link risks directly to applications:\n- Image-analysis tools may perform differently across demographic groups.\n- Electronic-record systems may expose sensitive information.\n- Generative tools may produce inaccurate or fabricated clinical content.\n- Administrative automation may reproduce inequities in access or billing.\n\n### 3. Avoid implying that regulation alone solves the problem\n\nThe conclusion presents “carefully regulated” use as central, which is reasonable, but regulation is only one part of safe implementation. Standards, institutional governance, technical validation, and professional education are also necessary.\n\n**Suggestion:** Frame regulation as part of a broader governance system involving pre-deployment testing, post-deployment monitoring, auditing, procurement standards, and transparency requirements.\n\n## Missing perspectives and context\n\n### 1. Equity and access\n\nThe report discusses bias but not broader access issues. AI could improve care in underserved areas, but it could also widen disparities if advanced tools are available only in well-funded health systems.\n\n**Suggestion:** Consider:\n- Differences in access between urban and rural settings\n- Costs of implementation\n- Language and disability access\n- The risk that poorly represented groups receive less accurate care\n- Whether AI tools work across different healthcare systems and populations\n\n### 2. Data quality and representativeness\n\nThe report refers to “large, high-quality datasets,” but dataset size does not guarantee quality. Data may contain missing information, historical discrimination, inconsistent labeling, or patterns that do not generalize to other institutions.\n\n**Suggestion:** Explain that data should be representative, clinically relevant, accurately labeled, and continuously evaluated for performance across demographic groups.\n\n### 3. Privacy beyond general concern\n\nPrivacy is named but not developed. The report could distinguish between risks from data collection, data sharing, re-identification, and secondary use.\n\n**Suggestion:** Mention safeguards such as data minimization, encryption, access controls, consent procedures, secure model development, and policies governing whether patient data can be used to train commercial systems.\n\n### 4. Cybersecurity and system failure\n\nCybersecurity is listed alongside other concerns, but the report does not explain the consequences of attacks or technical failures.\n\n**Suggestion:** Note that threats may include unauthorized access, manipulation of training data, adversarial attacks, ransomware, or disruption of clinical services. It would also be useful to mention the need for backup procedures when AI systems are unavailable.\n\n### 5. Patient and clinician perspectives\n\nThe report focuses mainly on systems and institutions. It would be strengthened by considering whether patients understand when AI is used, whether they can challenge an AI-supported decision, and how clinicians’ responsibilities and workloads may change.\n\n**Suggestion:** Include informed communication with patients, avenues for appeal, and the possibility that AI may reduce administrative burdens in some contexts but create new documentation, monitoring, or training demands in others.\n\n### 6. Environmental and economic costs\n\nThe report mentions possible cost reduction but does not consider implementation and maintenance costs. Large AI systems may require substantial computing infrastructure, energy, licensing, and workforce investment.\n\n**Suggestion:** Present cost savings as conditional rather than automatic, and acknowledge that financial benefits may differ among hospitals and healthcare systems.\n\n## Specific recommendations\n\n- Add one or two concrete examples to make the discussion less abstract.\n- Clarify that AI performance does not automatically translate into improved patient outcomes.\n- Distinguish between the risks of predictive systems, generative AI, and administrative automation.\n- Expand the discussion of accountability and post-deployment monitoring.\n- Include equity, accessibility, and implementation costs alongside bias and privacy.\n- Replace or qualify “most effective” with wording that emphasizes context-dependent effectiveness.\n- End with a more specific set of conditions for responsible use, such as representative validation, transparency about limitations, human review, continuous auditing, cybersecurity, and clear responsibility for errors.\n\nOverall, the report provides a strong introductory overview. Its next improvement should be greater specificity: concrete examples, clearer distinctions among applications, and practical explanation of how healthcare organizations can manage the risks it identifies.'print("Critique word count:", len(critique.split()))
Critique word count: 1139
Step 3: The Reviser¶
The Reviser receives both the original report and the critique, then produces another version.
The Reviser has:
The initial draft
The critic’s comments
This gives it an opportunity to make targeted changes while preserving useful parts of the draft. However, the critic may miss a problem or suggest an unhelpful change, and the reviser may not follow good advice. We therefore need to inspect the final version rather than assume it improved.
# System prompt that defines the Reviser persona
REVISER_SYSTEM_PROMPT = """You are a senior reporter who produces polished, final versions of reports.
You will receive an original report and editorial feedback. Your task is to revise the report by:
- Addressing the specific points raised in the critique
- Improving clarity, structure, and flow
- Maintaining the original voice and intent
- Ensuring the final version is publication-ready
Write only the revised report, without any preamble or explanation of your changes."""def generate_revised_report(report: str, critique: str) -> str:
"""
Generate a revised report using the Reviser role.
Args:
report: The original report to revise
critique: The editorial feedback to consider
Returns:
The revised report as a string
"""
user_message = f"""Please revise the following report based on the editorial feedback provided.
ORIGINAL REPORT:
{report}
EDITORIAL FEEDBACK:
{critique}"""
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": REVISER_SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
reasoning_effort=REASONING_EFFORT,
)
return response.choices[0].message.content or ""
This function takes two inputs: the original report and the critique. The labeled sections help the model distinguish the draft from the comments about it.
Let’s generate the revised version and then compare it with the original. Exact requirements, such as a word limit, should be checked with Python rather than estimated by the model.
# Generate the final revised report
final_report = generate_revised_report(initial_report, critique)
final_report'Artificial intelligence is reshaping healthcare, but its applications differ in capability, maturity, and risk. Predictive and diagnostic systems can analyze medical images or electronic health records to identify patterns associated with conditions such as cancer or heart disease. Generative AI tools can summarize records, draft clinical notes, or answer patient questions, though they may produce inaccurate or fabricated information. Robotics and treatment-support systems can assist with procedures and rehabilitation, while administrative applications can help schedule appointments, process claims, and manage hospital operations.\n\nThese technologies may improve care in several ways. In diagnosis and prediction, AI can help clinicians identify potentially significant findings more quickly or estimate a patient’s risk of complications. In treatment and drug development, it may support the selection of therapies, accelerate the search for promising compounds, and assist research into personalized medicine. Patient-facing tools may provide reminders, translation, or basic health information, while administrative automation could reduce repetitive work and allow professionals to devote more time to patients. However, strong performance in a controlled study does not necessarily translate into better outcomes in everyday practice. Results can vary according to the disease, dataset, healthcare setting, patient population, and quality of integration into clinical workflows. A system may be technically accurate yet fail to improve care if it is difficult to use, produces too many false alerts, or encourages clinicians to rely on it uncritically.\n\nThe same data and infrastructure that make AI useful also create risks. Large datasets may contain missing information, inconsistent labels, or records shaped by historical discrimination; size alone does not make them representative or reliable. An image-analysis system, for example, may perform less accurately for demographic groups that were underrepresented in its training data. Electronic-record and generative-AI systems raise concerns about unauthorized access, re-identification, data sharing, and the secondary use of patient information, including whether data may be used to train commercial systems. Safeguards such as data minimization, encryption, strict access controls, secure development, and clear consent and data-use policies are therefore essential.\n\nCybersecurity threats can include ransomware, manipulation of training data, adversarial attacks, and efforts to disrupt clinical services. Healthcare organizations also need backup procedures for periods when an AI system is unavailable or produces questionable results. Administrative automation may reproduce inequities in billing, eligibility, or access if it is trained on biased historical decisions. More broadly, advanced tools may widen disparities if they are affordable only to well-funded hospitals or work poorly across rural settings, languages, disabilities, and different healthcare systems. Implementation also carries costs, including licensing, computing infrastructure, energy use, staff training, maintenance, and ongoing evaluation, so savings should not be assumed.\n\nResponsible use requires more than asking a clinician to review an automated recommendation. Clinicians need training in a system’s appropriate uses and limitations, the ability to question or override its outputs, and protection against automation bias—the tendency to accept computer-generated advice without sufficient scrutiny. Organizations should validate systems on representative local populations before deployment, monitor performance and disparities afterward, audit for errors and drift, and establish procedures for reporting, correcting, and learning from failures. Patients should be told when AI meaningfully contributes to their care and, where appropriate, given ways to ask questions or challenge an AI-supported decision.\n\nAccountability must also be explicit. Developers and vendors are responsible for designing, testing, documenting, and securing their systems; healthcare institutions must evaluate and govern the tools they purchase and deploy; clinicians remain responsible for exercising professional judgment within their roles; and regulators must set enforceable standards for safety, transparency, privacy, and equity. Contracts and institutional policies should clarify responsibility when multiple parties contribute to a harmful decision.\n\nAI can support safer, more efficient, and more accessible healthcare, but its effectiveness is context-dependent rather than automatic. It should be introduced through broader governance that combines regulation with representative validation, transparent communication about limitations, human review, cybersecurity, professional education, continuous auditing, patient safeguards, and clear responsibility for errors. Used under those conditions, AI can complement healthcare professionals without treating automated output as a substitute for clinical judgment or public trust.'print("Initial report word count:", len(initial_report.split()))
print("Revised report word count:", len(final_report.split()))
Initial report word count: 134
Revised report word count: 668
Comparing the Results¶
Compare the initial report, critique, and revised report. Which comments were useful? Which changes helped? Did the revision introduce a new problem or leave an important issue unresolved?
from IPython.display import Markdown
comparison = f"""
## Initial Report
{initial_report}
---
## Critique
{critique}
---
## Final Report (After Reflection)
{final_report}
"""
Markdown(comparison)Reflect¶
After observing the results, look back at the prompts. How would you modify them? Are there some general requirements for these reports that could guide the Writer and the Critic? What are the tradeoffs between generality and specificity? Are these universal or domain specific? Modify the prompts and re-run the notebook to observe the effects.