Tesla first started making promises about full self driving vehicles in 2013. Since then, AI models have been trained on millions of miles of driving. Surely that means they've seen every possible situation, and that full self-driving is just around the corner. So where's my self-driving car?

Let’s look at a list of some of the conditions I’ve encountered while driving or being a passenger in a vehicle. (It's a long list. When you think you’ve got the idea, you may want to skip to the next section!):

• Dazzling sun preventing vision
• Dazzling headlights preventing forward vision or rear vision.
• Thick fog - foggy enough that neither side of the road is visible.
• Drifting smoke from burning stubble.
• Policeman directing traffic with unique arm gestures and facial expressions.
• Roadworker directing traffic wearing safety vest.
• Ordinary person directing road traffic around accident.
• Roadworker directing traffic with ambiguous stop / slow sign.
• Roadworker directing traffic with stop / slow sign reversed. i.e. showing stop when it should show slow and vice versa.
• “Local traffic only” sign - do I count as local traffic?
• Road closed - police vehicles used to block road. I live there, is it really closed?
• Unsafe wide load (oncoming traffic) requiring driving onto hard shoulder and waiting for load to pass.
• Oncoming vehicle on single track road (that means one lane shared between both directions) requiring one of us to reverse into a passing space.
• Road too narrow for vehicle to fit through. (Scraped both sides on buildings!)
• Road with vehicles parked on both sides requiring pulling in to space between parked vehicles to allow vehicle travelling in the opposite direction to pass.
• Flock of sheep being moved on road by shepherd.
• Marching soldiers on road.
• Injured motor cyclist on road.
• Motorcyclist takes turn too fast and slides self and motorcycle under front of car.
• Fire hoses laid across road, firemen directing traffic.
• Traffic light failure (no lights).
• Traffic light failure (all lights permanently red).
• Snow white-out - no forward or reverse vision.
• Snow - no visible road markings due to drifting snow.
• Snow - partial drift on road.
• Car crash - avoiding debris on road.
• Roadkill - avoiding large dead animal on road.
• Roadkill - driving over small dead animal on road.
• Large wild animals on road (take evasive action or stop).
• Small wild animals on road (only take evasive action if safe to do so).
• Fallen tree blocking part of road.
• Fallen tree blocking all of road.
• Debris on road that can be safely driven over.
• Debris on road that can't be safely driven over.
• Car mechanical problem - tire failure.
• Unsafe load on vehicle ahead - drop back to give more time to react.
• Debris blowing off vehicle (dump truck) ahead and hitting vehicle.
• Ice/snow blowing off vehicle ahead.
• Drunk driver swerving dangerously from side to side ahead.
• Driver behind me attempting unsafe overtaking manoeuvre and requiring speed adjustment to allow them to cut in front of me.
• Emergency service vehicle behind.
• Unmarked police car with emergency lights behind.
• Police officer in car signalling for me to stop using hand signals.
• Non-police car signalling for me to stop with flashing headlights to indicate problem (lights failure, load unsafe, etc.)
• Non-police car signalling for me to speed with flashing headlights because they want me to drive faster than the speed limit.
• Driving on sidewalk (pavement) to avoid obstruction.
• Driving onto private property (gas station forecourt, carpark, etc.) to avoid obstruction.
• Anticipating when small child is about to run into the road.
• Anticipating when mother with push chair is about to cross the road without checking for traffic first.
• Anticipating when other driver is about to run a red light or stop sign from lack of braking.
• Anticipating when cyclist is about to ride across pedestrian crossing.
• Unsteady cyclist requiring extra wide berth to overtake safely.
• Black ice (ice which looks like a wet road surface).
• Car ahead skidding on black ice.
• Car ahead on hill sliding backwards towards us on icy surface.
• Car sliding backwards on hill on icy surface.
• Road sign significantly defaced.
• Road sign altered (legitimately).
• Improvised road signage. (e.g. “Road Closed Flooding”, “Private Road”)
• Road sign contains several lines of text indicating when it applies (school days while lights flashing, weekdays 9am to 11am, etc)
• Potholes of sufficient depth to cause tire damage and require suspension realignment
• Ruts of sufficient depth to cause the car to “bottom out” on the roadway
• Garbage bag blowing across road.
• Emergency road closures with unmarked or partially marked diversions.
• Driver turning unexpectedly across path of vehicle resulting in minor collision (park immediately where safe to do so and exchange insurance details).
• Road signs in unknown language.
• Ambiguous or confusing parking restriction signs (sigh!) which determine when and how long you can park. (Does the car know that I need to park for less than fifteen minutes or more than three hours? I know.)
• Local laws which require drivers to give way to transit buses signalling to leave a bus stop.
• Local laws defining speed limits on otherwise unmarked roads (e.g. 30 m.p.h. limit on roads with street lights, 50 km/h limits inside town limits.
• Local laws defining speed limits based on the current weather. (e.g. national speed limit is reduced by 10 km/h if it is raining).
• Broken-down car stopped in lane - no lights
• Other driver signalling that he is relinquishing priority at junction with hand gestures.
• Other driver signalling that he is relinquishing priority at junction with flashing headlights.
• Driveway entrance blocked with string, chain, or other improvised objects due to driveway resurfacing.
• Road closed using police caution tape tied across road.
• Police directing traffic (taking precedence over working traffic lights).
• Police directing traffic around accident.
• Traffic sign (Stop sign) missing or destroyed by snow plow.
• Traffic sign (Stop sign) covered with graffiti
• Traffic light no right turn sign applying only between specific times on specific days.
• Generic warning signs with hand written additions (Road Flooded)
• Depth markers on side of road indicating depth of current flooding
• Road signs partially underwater
• Diversion signs which apply to specific diversions only.
• Axle limit weight restrictions
• Exceptions for “local traffic”
• Incorrect lane markings due to roadworks
• Patches which look like wet road but are actually black ice.
• Deer running into side of vehicle.
• Raccoons on the roadway.
• Skunks on the roadway.
• Large turtles crossing the road.
• Vultures feasting on the road.
• Bridges over single track roads where oncoming traffic is not visible.

Too Long; Didn’t read? Skip to this bit

I’m not a commercial driver, but I’ve encountered all except one of these at least once since I started driving. Each problem is unique. Each problem is less about the mechanics of driving and more about understanding the wider environment and taking the appropriate action. Some of these are hard problems even for a human (ever had to negotiate which way you should be turning with a police officer directing traffic? Ever had to negotiate your way past a police road block? Is it better to hit a deer or drive off the road into a ditch?)

Self driving trains work in because they operate in a highly constrained environment. Problems requiring human judgement have been mostly engineered out of the system with fences, barriers, etc. Trains travel fast, can’t stop quickly, and many lives have been lost in previous accidents. Multiple redundant safety systems are incorporated to reduce the risk of collision with another train.

A real full self-driving car needs to work in an unconstrained environment where the workings of the outside world need to be understood and taken into account. That requires a degree of general intelligence which no autonomous vehicle is likely to have in the foreseeable future. Expect driver assist, not driver replacement. Self-driving taxis will therefore continue to require the occasional remote human intervention whatever the grandiose promises being made to investors

The AI model for self-driving cars may be able to train on common conditions — but it’s the unforeseen uncommon conditions that will kill you.

Stay up to date with the free Risky Thinking Newsletter.

Santa looked exhausted. Christmas always takes a lot out of him, but normally he has recovered by now.

"So what happened?" I asked him.

He stared at me, then whispered the word as if saying it out loud would cause his troubles to return: “ChatGPT”.

Not ChatGPT specifically, he quickly explained. But the idea of ChatGPT. That was the cause of the series of disasters that had almost caused Christmas not to happen.

"Our IT operation here is one of the largest on the planet. We have to determine who deserves presents, handle vast amounts of correspondence in many languages, and handle the logistics of sourcing, constructing, and delivering items from all over the world.

"Our elves have always been at the forefront of technology. They have to be. And when we read about the recent advances in Large Language Models (LLMs), we immediately looked for ways AI could be used to streamline our operations.

"Language translation has always been an issue for us. Wouldn’t it be great if our elves could work in their native language, with machine translation to and from other languages. That would solve so many problems. Do you know how difficult it is to find an elf who is able to read letters written in Xhosa or Mirandese? Our elves would even be able to say QISmaS DatIvjaj ‘ej DIS chu’ DatIvjaj wih or Ĝojan Kristnaskon.

"Then we thought about the Naughty or Nice assessment. We have to handle billions of reports to decide whether people are naughty or nice. In AI terms, that’s a problem known as “sentiment analysis”.

"We also considered using AI to help our Human Resources department. Perhaps we could evaluate all our elf workers with a simple AI interview. That would save our managers a lot of effort and improve consistency in assessments.

“Finally there’s public relations. Our brand looks old. We haven’t done much to modernize our image since our questionnable tie-up with Coca-Cola in the 1930s. Some people think we should project a more modern image.”

“It all sounds very ambitious”, I remarked, “but it's quite conservative compared to some of the proposals I’ve been hearing recently. So what went wrong?”

He sighed. “Where shall I start? It wasn’t like we didn’t prototype things. But although we did pilot studies, things started to go wrong when we moved to production”.

“Perhaps we should start with the Naughty and Nice list?”

“Yes, that one’s definitely the biggie. The Naughty or Nice list drives our entire operation. Easy, we thought. We have over a thousand years of training data - written reports made by our agents around the world, each helpfully tagged with a classification of Naughty or Nice. We immediately started digitizing our archive and feeding the results into our learning algorithm. But when we started testing on real data we realized we had a problem. Does a report of showing too much skin in public mean you are Naughty or Nice, or is it neutral? If you often hang out with a group of naughty people, does that make you naughty too? It depends upon your sex, your status, and where and when you are. It’s easy to create accidental bias if you don’t understand local social norms. We found that by combining all the reports we’d created a monster of a model which was both sexist and racist, with a penchant for Victorian moral values. That’s probably due to the excessive number of written reports from that era. By the time we’d realized the approach wasn’t viable - world-wide historical data couldn’t provide the situation-specific classification we needed - we’d wasted a lot of effort. It took a lot of manual processing to get things back on track. If you didn’t get a Christmas present or lump of coal last year, that might be the reason.”

“So the Naughty or Nice list didn’t go so well. How did using large language models for email work out? Machine translation is getting pretty good now, isn’t it?”

Santa looked me straight in the face. “You realize how little is written in Elvish, don’t you? There’s almost nothing published in modern Elvish, just a few poems composed by Tolkien who wasn’t even a native speaker. We quickly gave up on machine translation into modern Elvish. But even though they had been told not to, some elves started using ChatGPT to write email to our suppliers. And since the output was in a foreign language, often they didn’t read the email before sending it. Apparently the language model we were using had been trained on online reviews. It couldn’t explain why, but it started including plausible reasons for delaying or avoiding payments into outgoing emails. We discovered the problem when a supplier received an email complaining about the poor audio quality of Christmas Trees. Shortly afterwards, we falsely accused Apple of making fake Apple products. We had to fire a few elves over that one. Once our elves understood the possibility of AI hallucinations and what might happen if they didn’t check the output, it didn’t happen again.”

“You said you tried using large language models for HR?”

“Yes. Somebody in our HR department thought it would be a good idea to use a widely available LLM to generate our annual employment reviews. We’re not like most organizations. It generated bland questionnaires and tried to set our elves quite unreasonable objectives. “Where do you see yourself five years from now?” isn’t something to ask an elf working in the toy factory at the North Pole. Asking an elf about their “personal growth objective” is particularly insulting. People accept that somebody can make the occasional mistake, but with the sudden burst of AI-assisted mistake making, we nearly had a strike on our hands.”

“Surely you had some successes”

“Public relations and branding. That mostly worked. As you know, our brand is very important to us. Think what the world would be like if everybody still thought about Krampus at Christmas! We used generative AI to come up with some new ideas, something that is very hard when you’ve been doing the same thing for hundreds of years. We came up with a few good ideas which unfortunately I can’t tell you about as we prefer to run our PR campaigns in stealth mode. We also tried using generative AI to come up with some new images. That didn’t work out so well. Initially our creatives were worried about their jobs. But when they discovered that the generative AI network we were using had an obsession with buxom females in fantasy landscapes… Perfect, perhaps, if our brand was aimed at teenage boys. But we’re family-oriented. Their jobs are definitely safe for another year.”

Santa looked quickly to one side. “I have to go shortly. Do you have any other questions?”

“Can you tell me what I’m be getting for Christmas next year?” It was worth a try.

“As a large language model I only have knowledge of the world before last December. Is there anything else I can help you with today?”

“You’re pretending to be an AI generated Santa so you don’t have to answer my question, aren’t you?”

Santa smiled for the first time and guffawed loudly. “Ho, ho, ho” he roared. “I bet you didn’t think of using Artificial Intelligence as an excuse to avoid questions! But trust me, you really don’t want to know the answer.”

And at that point Santa closed the connection.

Stay up to date with the free Risky Thinking Newsletter.

My interest in with accurate time keeping probably dates from school days, when there were serious bragging rights to be earned from having an accurate watch which exactly matched the beeps of the Greenwich Time Signal on BBC Radio. Therefore, when a YouTube channel I followed showed how to create your own network time server using a cheap ESP32 micro-controller, I thought about building it. So I examined the AI generated example code.

I’ve tired to keep this at a high level, but to understand the problems you have to understand some of the details - and that is partly the point.

The code has to do two things:

1. Read the time from the GPS/GNSS module and sets the local clock to match it.
2. Read for an NTP query packet on the network interface, and constructs a reply which includes the time when the packet was received and the time when it was sent back. (Using these a client can calculate trip delays and decide how to adjust its clock to best match the server’s clock.)

The AI generated code:

• Created a task which waited for a report from the GPS module, parsed it to get the time, waited in a “busy loop” for a variable to be set indicating a new second had started (see below), then set the local time.
• Created a task which waited for an NTP packet on the network interface, constructed a reply including the time (from the local clock) when the packet was received, the time (from the local clock) when the packet was prepared, and sent the reply,
• Created a mutex - a synchronization primitive that can be used by a program to prevent two tasks trying to read and write the same variables at the same time.
• Created an interrupt handler which, every time the GPS module indicated the start of a new second, set a variable to true.

The code “works”, in that it compiles and would give a syntactically correct reply if sent an NTP packet. It looks plausible.

But the code is wrong. Very wrong. And looking out how it is wrong gives some indication of the dangers of “vibe” coding.

1. Although the code declares a mutex to prevent the two tasks from accessing the same variables at the same time, it isn’t ever used. Where is task synchronization needed? The shared variable isn’t in this code, it’s in the system time library. (This is possibly why the AI generated code misses it.) What happens if one task is setting the time and the other is reading the time simultaneously. It’s undefined. Will this happen often? No. It might show up in a long test, but will probably not show up in any simple test.
2. What happens on power up before the GPS module has received its first fix? The time will be (unless we add a real time clock) shortly after1 January 1970. That’s zero (uninitialized) time for time libraries. This time will be shared on the network until the GPS gets its first fix, possibly minutes later.
3. The GPS code waits for a variable to be set by the interrupt handler in a busy loop. There are a number of problems here:
   • The variable is not declared (in C++) as volatile. This means that a compiler can assume its value is not being changed by another task and keep its value in a register rather than reading it from memory. So the time task might loop indefinitely depending upon the compiler being used and the options given to the compiler.
   • Simple busy loops, where the code repeatedly tests whether a variable is true, are a bad thing: the processor is prevented from doing useful work while the task is looping. This may cause some unnecessary delays in the other task.
   • There’s an off-by-one error in the timing. The task waits for the start-of-second variable to become true after reading the time from the GPS module, then uses the time previously read from the GPS module. This is the start of the next second, not the start of the second whose time was read.
   • GPS signals are not perfect. They depend upon moving satellites and can be affected by nearby objects etc. A GPS receiver can lose lock, in which case the next start-of-second can be seconds (or minutes or hours) after the previous one. This results in the clock being set incorrectly until a new signal is received.
4. The interrupt service routine didn’t use the correct type of memory, and didn’t use the best method to communicate with the rest of the program. Micro-controllers have a confusing array of different types of memory, and code that is run in interrupt handlers should be placed in a specific type of memory. There’s also a specific primitive an interrupt handler should be using to communicate with a task (eliminates the busy wait loop), but that wasn’t used here.
5. There’s a questionable choice of hardware: the ESP32 natively uses WiFi - which isn’t really suitable for NTP time servers because there’s a significant amount of jitter (unpredictable delay) on a WiFi network due to other traffic or radio interference. I think this is acceptable in this case (cheap and widely available hardware for teaching purposes) but wouldn’t be a good idea in an area with multiple WiFi networks.

I think you get the idea.

Personal Tests

But perhaps, I thought, I’m being unfair to the human joint-author of this code. He doesn’t claim to be an expert programmer. I therefore experimented with some “vibe coding” myself using Google’s gemini-cli for comparison. I tried a number of simple Python coding tasks, from adding functionality to an existing script to generating a simple command line application in Python. The results were mixed:

• The simple case of adding functionality worked excellently, except that I had to guide gemini as to where to some file format documentation: it’s general web search was inconclusive. There was a package it should have used to format the CSV file, but it’s own implementation worked in this instance.
• The second attempt had to be abandoned: the program hallucinated what a documented function did and built an entire program around using it. When testing the resulting code it produced runtime errors, and a quick investigation showed that the documentation of the Python package it was using had been completely misunderstood. The assumption was a fundamental error, and as a result none of the code generated was remotely useful.
• The third attempt produced incorrect code which could be modified . Once again gemini misinterpreted the specification it was using and chose the wrong values to use, this time giving plausible incorrect output. On the way it wasted a lot of its time trying to use XML namespaces (which weren’t used being used). It also decided to delete the database it was creating each time the program was run - an error which might have allowed test cases to work with serious consequences if ever used in production. In the end it I had to make several code correction by hand.

However, it was a great deal of fun and it felt as if it was productive…

To sum up…

The “vibe” code worked except:

• It didn’t handle startup conditions correctly
• It didn’t handle error conditions correctly.
• It didn’t handle race conditions between different tasks.
• It used undefined C++ behavior and in doing so made unwarranted assumptions about how the compiler would behave. A compiler change could cause the code to stop working.
• It didn’t understand the specification of the GPS module it was using, resulting in an off-by-one-second error even when it worked as intended.
• In general, it didn’t understand online documentation in sufficient depth.
• It displayed a tendency to hallucinate functions or function arguments, and to assume it knew what the function did.
• There was a complete lack of questioning the wider scope: in the GPS case, could this hardware choice ever give the required performance?

But in the GPS case the code did look plausible and would probably pass some simple tests. Apart from the off-by-one second error the faults would be unpredictable and intermittent. (i.e. It produced the type of faults which are most difficult and expensive to find and diagnose, especially after deployment.)

The code would probably fool an inexperienced programmer or someone who was unfamiliar with the hardware specifications. And that’s the problem. Does the person doing “vibe coding” actually know enough to know what they don’t know?

Resources

• It’s easy to feel that “vibe coding” makes you more efficient: at least one randomized control trial suggests that the opposite is the case.
• I don’t think it would be fair to mention the YouTube channel or GitHub repository containing the code here. The moral is to make sure you understand the code in depth before trusting it.

Stay up to date with the free Risky Thinking Newsletter.

There are two business continuity plans at work here: the VoIP provider’s and the data center’s.

The VoIP Provider

From the the VoIP provider’s perspective this should have been a quick (if not automated) switch-over to a different data center with partial (if not full) functionality. It’s clear that the possibility of a complete data center failure was either (a) forgotten, (b) ignored, or (c) judged to be too costly to fully mitigate for the probability of it happening.

Whatever the reason, it doesn’t make them look good. And the fact that I only found out about their problems by encountering them myself rather than being warned in a helpful customer email or in a statement on their website does little to inspire confidence.

Even a rudimentary business continuity plan should include warning customers about service problems.

So the VoIP provider is handling the outage badly. They are cheap and full-featured, but I can no longer say that they are reliable, I won’t recommend them any more and will be looking for a suitable replacement.

But what of the data center? Why isn’t it back yet?

The Data Center

Fortunately (if you know the right place to look and the right discussion board to follow) you can find out that the data center they used actually cared about notifying its customers. Here’s what happened:

Power remains off at our data center in REDACTED per the local fire marshal.

We have had an electrical failure with one of our redundant UPS’ that started to smoke and then had a small fire in the UPS room. The fire department was dispatched and the fire was extinguished quickly. The fire department subsequently cut power to the entire data center and disabled our generators while they and the utility verify electrical system. We have been working with both the fire department and the utility to expedite this process.

We are currently waiting on the fire marshal and local utility to re-energize the site. We are completely dependent upon their inspection and approval. We are hoping to get an update that we can share in the next hour.

At the current time, the fire department is controlling access to the building and we will not be able to let customers in.

And this is what BCP plans often fail to take into account when considering the risk of a small fire. If it’s electrical, the priority of the fire department and the local electrical utility is safety. The fire may have been small. The fire may have been swiftly extinguished. But the fire marshal’s job is to ensure safety. That means shutting off all electrical systems and not taking any chances.

And when everything is shut down, and after repairs are made, it takes a surprisingly long amount of time to switch everything back on and get it working correctly. Here’s an update from twenty-four hours later:

The REDACTED data center remains powered down at this time per the fire marshal. We are continuing with our cleanup efforts into the evening and working overnight as we make progress towards our 9AM EDT meeting time with the fire marshal and electrical inspectors in order to reinstate power at the site.

Once we receive approval and utility is restored, we will turn up critical systems. This will take approximately 5 hours. After the critical systems are restored, we will be turning up the carriers and then will start to turn the servers back on.

The fire marshal has requested replacement of the smoke detectors in the affected area as well as a full site inspection of the fire life safety system prior to allowing customers to enter the facility. Assuming that all goes as planned, the earliest that clients will be allowed back into the site to work on their equipment would be late in the day Wednesday.

The points to note here are that:

• After a fire, even a small one, you are no longer in control of your building: the Fire Marshal is.
• The Fire Marshal’s priority is safety. Electrical systems will be switched off if there is any possibility that they present a hazard to firefighters or have been damaged in a fire.
• Electrical systems have to cleaned, repaired and inspected before they can be re-energized.
• Smoke detectors and other fire safety system components may need to be replaced and the entire fire safety system may need to be checked before normal access to the building is allowed.
• After everything is completed, it can take a long time (in this case at least five hours) before critical systems are all working again.
• Even after this, customer systems will still need to be restored.
• In this case data loss was limited to that caused by systems being unexpectedly powered off. Hopefully this was taken into account in the design of all the application programs.

TL;DR? When planning remember that even small fires with limited damage can have major consequences.

Stay up to date with the free Risky Thinking Newsletter.

I received an email survey today. Not surprising. Many companies send them out. This one was from Southwest Airlines, and it offered me a reward of $100 for completing their survey.

Except it wasn’t. I knew this immediately as I have never flown on Southwest Airlines.

This was a phishing email directing me to a plausible sounding survey website. The fraudsters hadn’t bothered to get a free SSL certificate for their plausible domain, perhaps because this can trigger alerts when the Certificate Transparency List is published.

So should I warn Southwest Airlines? I went to their website. Was their an email address or form to fill in to let them know about the website “southwestairlinessurvey.today” phishing their customers? No there wasn’t. Plenty of ways to report lost baggage, but no obvious way to report an issue to their security team - assuming they have one. I could have found out how long their customer service queues were by calling their 1-800 number and found out whether their customer service agents knew how to contact their security team, but I didn’t.

I’m neither a customer nor a shareholder, so in a literal sense it’s none of my business. I will do things that are easy to do if it helps makes society better. But I won’t put in a lot of effort to help a company that doesn’t make it easy to help them. I suspect I’m not alone in this.

if you care about people phishing your customers make it easy to report it: otherwise you will only hear about it very much later from disgruntled customers who believe you cheated them out of rewards, goods, or services.

Stay up to date with the free Risky Thinking Newsletter.