CIS 3500
Introduction to
Software Engineering or
"how to JOYFULLY BUILD
with a TEAM"
Prof. Jérémie Lumbroso, Ph.D.
lumbroso@seas.upenn.edu

⚠️➡️ RIGHT NOW, go to https://sli.do
Code #23232325
LECTURE 5:
Why Process Matters—
How Software Gets Made

Agenda
- HW1's update
- Programming vs Software Engineering
- Discussion on Technical Debt
- Case study on a public service launch
- What is the "Waterfall" production model?
- ThinkPairShare: Have you witnessed a tech failure?
HW1 due last night
Let's take a look!
Programming vs. Software Engineering
- Programming is about producing code that works in the short term.
- Software Engineering is about maintaining and evolving that code over time.
- Key differences:
- Time: Short-lived vs. long-lived code.
- Scale: Individual developer vs. large, multiperson teams.
- Trade-offs: Balancing cost, complexity, and changing requirements.
-
“Software engineering is programming integrated over time.” – SE at Google


Technical Debt
A Definition
What Is Technical Debt?
-
Coined by Ward Cunningham, technical debt describes the future costs (rework, complexity) incurred when a team takes the “quick/easy” solution now instead of a more robust approach
- Like financial debt, it can accumulate "interest" over time in the form of bugs, difficult maintenance, or slow development speed
-
Why It Generally Happens Intentionally:
- Tight deadlines, pressure to deliver features quickly
- Lack of proper refactoring or code cleanup
- Temporary hacks that become permanent
Ward Cunningham
- 1995: Created WikiWikiWeb
- inspired from Apple's HyperCard
- 2001: Cowrote "The Wiki Way"
- 2001: [one of 17 coauthors of] "Manifesto for Agile Programming"




From HyperCard to WikiWikiWeb






“Everyday Examples (Daily Life Analogies)”
-
Leaky Roof “Patch”
- Quick Fix: Duct tape over the leak instead of a proper repair.
- Interest: The damage grows; eventually the entire roof may need replacing.
-
Messy Closet “Just Slam the Door”
- Quick Fix: Throw everything into the closet so the room looks clean.
- Interest: Finding anything later becomes a nightmare; you spend more time rearranging when you finally clean.
-
In Code:
- Hacking a feature without writing tests → Harder to maintain/test later.
- Not updating libraries → Security holes or version conflicts in the future.
Key Point: If you keep ignoring technical debt, it can snowball, slow you down, or cause major failures.
In modern production methods, regular refactoring and small increments help keep debt from overwhelming you.





Where do you see technical debt in your daily lives?
Debrief Class Activity
Instances
refers to an actual occurrence of technical debt within software, technology, or organizational systems. These are real-world examples where past shortcuts or outdated systems create a burden that must be addressed in the future.
Analogies
refers to instances where the concept of technical debt is used figuratively to describe inefficiencies, poor planning, or shortcuts in non-technical areas of life, such as personal habits, urban planning, or interpersonal relationships.
Where do you see technical debt in your daily lives?
Path@Penn crashing during advanced registration – This is an actual technical issue caused by poor system design or maintenance.
Cramming for exams instead of studying steadily – This is not technical debt in the software sense but is likened to it because of the short-term gain and long-term cost structure.
Instances
refers to an actual occurrence of technical debt within software, technology, or organizational systems. These are real-world examples where past shortcuts or outdated systems create a burden that must be addressed in the future.
Analogies
refers to instances where the concept of technical debt is used figuratively to describe inefficiencies, poor planning, or shortcuts in non-technical areas of life, such as personal habits, urban planning, or interpersonal relationships.
35 examples, often involve:
- University Services / Path@Penn / Administration
- Airline or Public Transportation systems
- Outdated websites or software infrastructure
- Personal or corporate codebases with legacy issues
18 examples, often involve:
- Personal productivity, health, or academics (procrastination, messy notes, junk food)
- Home maintenance (leaky faucets)
- Interpersonal relationships
- Larger societal issues (global warming, urban planning)
Instances
Analogies
- mairiair: "With the rise of AI, some people value convenience and will copy code but can arrive at suboptimal solutions with longer runtimes, taking up more space, etc. Overall, the solutions can be suboptimal."
- kimmia: "One example for me was at a clinic I was volunteering at. The software could only differentiate by name and birthdate when searching, often leading to errors given that there are many patients, some even having the same name and birth date. I believe this was due to the age of the software, and many clinicians had other complaints with it being slow. I wonder if this is due mainly to funding challenges, or also because of the difficulty in transitioning to new systems?"
- wualan: "Many platforms to apply for jobs are so poorly constructed and so slow compared to the platforms and systems used by larger and more modern tech companies."
- wesliu: "I think a lot of government organizations have tons of technical debt. Their websites have poor UIs that are never updated. Some companies also have job application landing pages that look very old and are designed very poorly."
- chaelsey: "I see technical debt in my health. Although I often eat junk food or processed meals since it is convenient, it can lead to long-term health impacts."
- sphia: "i can see technical debt in making food without documenting the recipe."
- domchang: "I can see technical debt happening in my normal life in class, when there may be a very confusing question/topic I don't understand, I might skip it or put a simple answer instead of trying to thoroughly understand. This would eventually cause issues during test prep, when I would have to take more time to actually learn it."
- agvarela: "I just dump 90% of my files in downloads, which makes it much harder to sort, find and discard unnecessary ones later."
- wongang: "technical debt can be seen in interpersonal relationships, when difficult conversations are avoided in order to make interactions smoother in the short term. This can lead to greater problems later on when unaddressed issues begin to pile up."
- mwengel: "procrastinating on tasks even though I'll have to rush and be sloppy at the end to meet a deadline"
Healthcare.gov
An American Software Engineering Meltdown Story
Healthcare.gov Meltdown
Obama (Oct 2013):
Nobody is madder than me that the website isn’t working... which means it’s going to get fixed.

Legislative Context & Unexpected Scope
- ACA (Affordable Care Act) mandated an insurance Marketplace by Oct 1, 2013.
- States could run their own exchange or let the federal govt do it.
- By 2012, most states opted out, forcing the Federal Marketplace (Healthcare.gov) to accommodate millions more users than originally planned.
- The federal deadline was fixed in law—no real option to delay.

The Build-Up—Fragmented Waterfall
- Development started in late 2011. The approach: a quasi-Waterfall with large up-front specs.
- Multiple vendors (CGI Federal, QSSI, etc.)—no single integrator (supposed to be CGI, eventually CMS itself had to function as integrator, coordinating modules from different contractors, but with no experience)
- Frequent requirement changes due to ongoing ACA policy adjustments.
- End-to-end testing was left to the final weeks—typical of big-bang Waterfall.







Launch Day Failure (Oct 1, 2013)
- Crashed in hours. Users faced timeouts, error messages, a “virtual waiting room.”
- Only 6 people enrolled on Day 1, out of tens of thousands who tried.
- The press hammered the site as a “disaster,” overshadowing the ACA rollout.
- States didn’t share the blame; the federal govt shouldered the public embarrassment.

Technical Root Causes
- Login Choke Point: They forced user account creation before browsing, swamping the system.
- Insufficient Load Tests: The site was never tested at realistic scale.
- Late integration of sub-systems. Modules that never “talked” until weeks prior.
- The site was built as though it’d serve a smaller subset (initial assumption: states would run their own exchanges).
“A late decision requiring login to window-shop caused a massive bottleneck at the site’s entrance.”
More detail from the Kansas City Star (Tony Pue, November 08, 2013): “It appears that one of the reasons for the high concurrent volume at the registration system was a late decision requiring consumers to register for an account before they could browse for insurance products. This may have driven higher simultaneous usage of the registration system that wouldn’t have otherwise occurred if consumers could ‘window shop’ anonymously.”
Management & Governance Failures
- No single leader with full authority.
- A bureaucratic chain of command overshadowed real-time problem-solving.
- Cost-Plus contracts: contractors got paid for effort, not outcomes.
- “All or nothing” mentality with no incremental rollouts.
Political & Public Fallout
- President Obama publicly apologized; critics called it proof the government “couldn’t handle large tech.”
- Over $800 million spent by the end of 2013, ballooning from initial $93 million budgets.
- Congressional hearings, demands for Sebelius to resign.
- Mass frustration from people who just wanted to enroll in health plans.
The “Tech Surge” Agile-like Rescue
- Mid-October 2013, a small SWAT team of private-sector engineers (ex-Google, etc.) took over daily ops.
- They used daily stand-ups, real-time monitoring, quick bug fixes (very Agile).
- Key changes:
- Dropped forced login to reduce load.
- Put all dev teams in a “war room.”
- Prioritized bugs & pushed fixes daily.
- By December, the site was stable; millions enrolled.
Quote:
“We did daily triage, day after day, for weeks. It wasn’t rocket science—it was iterative, relentless focus.”
– Mikey Dickerson, Tech Surge
Healthcare.gov—Before vs After (Table)
Aspect | Before (Waterfall-ish) | After (Agile Rescue) |
---|---|---|
Leadership | Fragmented, no single owner | Unified “war room,” daily stand-ups |
Testing | Minimal pre-launch load testing | Real-time monitoring & iterative bug fixes |
Release | Big bang on Oct 1 (unchangeable deadline) | Continuous daily/weekly deployments post-“surge” |
User Flow | Forced login prior to browsing | Allowed browsing w/o account creation (reduced load) |
Outcome | Site meltdown, 6 signups first day | Stable by Dec, millions enrolled, regained public trust |
[Presenter Notes]:
- Walk through each row to contrast Waterfall vs. Agile. This sets up the next portion: “Why was Waterfall used in the first place?”
Lasting Impact – Changing Government IT
- Wake-Up Call: Healthcare.gov fiasco became a cautionary tale
- Digital Service Playbook (2014): 13 “plays” for federal tech
- Procurement Reforms: Agile mandated in some contracts
- Cultural Shift: “fail fast, fix fast” now more accepted
- State-Level Changes: improved launches in state exchanges
Birth of the U.S. Digital Service (USDS)
- Origin (Aug 2014): direct response to Healthcare.gov issues
- Mikey Dickerson: first USDS administrator
- Early Wins: tackled Veterans’ benefits, immigration systems
- “Rebel Alliance” of Gov Tech: USDS + 18F + others
- Legacy: institutionalizing start-small, iterate-fast approach


Waterfall
Was it ever a good idea??
Why Waterfall? – Historical Context
- Origins: 1970 paper by Winston Royce describing sequential stages.
- Waterfall provided a structured, phase-based approach that replaced ad-hoc coding
- For predictable, unchanging requirements, it can be efficient (like building a car the same way every time)





Royce already anticipated revisions and iterative development, but hoped it would be organized
In practice, he acknowledged that software requirements might need to be revisited after testing, but the notion of user is mostly absent



- At the time, software was often for massive defense or NASA projects—similar to how Henry Ford’s assembly line was a major evolution for car production.


Waterfall Diagrams




Fordism & Mass Production
- Ford’s assembly line (1913) revolutionized manufacturing
- High volume, standardized parts, minimal variation
- Built on Taylorism that introduced notion of scientifically studying tasks to achieve maximum efficiency.
- “Plan once, produce many” was extremely cost-effective for identical products
- Waterfall mirrors that logic: gather all software requirements (design once), then code & release at scale


From Ford to Toyota—Just-in-Time & Lean
- Toyota introduced “just-in-time” production, meaning smaller batches, continuous flow, responding to demand.
- Minimizing inventory and discovering defects early.
- This “lean” approach influenced software—flexibility over rigid large-batch.
- Lean is a big inspiration behind Agile: “build what’s needed now, see feedback, adapt.”





The big drawback: real-world software requirements often aren’t stable or fully known up front.
From Ford to Toyota—Just-in-Time & Lean
- Toyota introduced “just-in-time” production, meaning smaller batches, continuous flow, responding to demand.
- Minimizing inventory and discovering defects early.
- This “lean” approach influenced software—flexibility over rigid large-batch.
- Lean is a big inspiration behind Agile: “build what’s needed now, see feedback, adapt.”
Summary
- Software engineering involves thinking carefully about the evolution of the product and its requirements over time
- Healthcare.gov meltdown: a perfect storm of big-bang Waterfall for a quickly evolving, high-stakes project.
- Waterfall historically made sense in stable, mass-production environments. But modern software often needs continuous iteration.
- Lean/Just-in-Time philosophies → Agile: short cycles, fast feedback, user focus.
- Next lecture: Agile + User Stories – the core tool for capturing real user needs in Agile.
Questions?
Do you understand how waterfall fits in? Its strengths? Weakenesses?
Think-Pair-Share: Agile Rescue
-
Think (2 minute):
Recall a recent “technical meltdown” or fiasco you’ve experienced or heard about—something that failed or got stuck in a big way (e.g., a university system glitch, a frustrating mobile app, or a well-publicized national-level issue). -
Pair (3–4 minutes):
Share your example with a partner. Brainstorm specific steps an “Agile rescue team” might take to stabilize or improve the system quickly—what is one (or two) steps that could have made an improvement? -
Share your individual answer with your PennKey (1 mins):
Submit scenario + fix. We’ll collect a couple of examples. Notice how focusing on small, continuous changes can help reverse a major software fiasco and demonstrate why Agile methods thrive in chaotic or fast-changing conditions.
Why This Prepares Us for the Next Lecture:
Much like the Healthcare.gov “tech surge,” Agile’s quick iteration and real-time feedback can rescue struggling projects. Next class, we’ll dive into how these Agile practices work and why they’re so effective.
CIS 3500: Lecture 5
By Jérémie Lumbroso