CS229 is the graduate-level machine learning course I took at Stanford as part of the Graduate Certificate in AI which I completed in 2021. While k-means is my choice as the easiest to understand algorithm in machine learning, it was taught as the introductory clustering algorithm for unsupervised learning. As a TA for XCS229, which I have been doing since 2022 and most recently did this Spring, I know that it is still being taught as part of this seminal course in AI.

We have liftoff!

Unlike previously where I saved the result for the end, let’s start by taking a look at the algorithm in action!

Here is a screenshot of the decision boundary after convergence.

Accuracy is 90% because 1 of the 10 observations is on the incorrect side of the decision boundary.

For debugging purposes, to speed up execution, I reduced the number of samples in each class to 5. (You might note that, on the graph, there are only 4 points in class 1, which are the □s. That’s because one of the points is at (291, 90), which is off the edge of the screen. Gaussian distributions can generate extreme outliers, so I decided to preserve those points rather than clip them to the edge of the screen.) That’s obviously pretty small but you can see the algorithm iterating.

At the end of each loop I draw a line between the latest estimates of cluster centroids. The perpendicular bisector of these segments are the decision boundaries between the classes, so I draw them, too. Some of the code was written to handle more than two classes but here there are only two which makes this relatively easy.

K-means explained

K-means clustering is an iterative algorithm that aims to partition \(n\) observations into \(k\) clusters in which each observation belongs to the cluster with the nearest mean, called the cluster centroid.

Ezpz.

The math is also simple. In step 1, the distance between two points, \(x\) and \(y\), is simply \(\sqrt{(x_0 - y_0)^2 + (x_1 - y_1)^2 + \cdots + (x_{d-1} - y_{d-1})^2}\), where \(d\) is the dimensionality of the observations. In our case \(d=2\) which is why we only have \(x_0\) and \(x_1\). Also, since we’re only using the distances for comparative purposes, it’s not even necessary to take the square root. In step 2, the centroid is simply the sum of all the points divided by the number of points.

Implementation

First, a little housekeeping before getting to the implementation of the algorithm.

10  HOME : VTAB 21
20  PI = 3.14159265
30  GOSUB 1000 : REM  DRAW AXIS
40  GOSUB 100 : REM  GENERATE DATA
50  GOSUB 900 : REM  WAIT FOR KEY
60  GOSUB 2000 : REM  RUN K-MEANS
70  END

100 REM  == HYPERPARAMETERS ==
...
450 DIM P%(2,1) : REM  RANDOM POINTS
460 REM  == K-MEANS DATA TABLES ==
470 DIM DI(NS - 1,KN - 1)
480 REM  -- K - MU-XO, MU-X1, N-K --
490 DIM KM(KN - 1,2)
500 REM  -- K - OLD MU-X0, OLD MU-X1 --
510 DIM KO(KN - 1,1)
...

900 REM  == WAIT FOR KEYSTROKE ==
910 POKE 49168,0 : REM  CLEAR BUFFER
920 IF PEEK(49152) < 128 GOTO 920
930 POKE 49168,0
940 RETURN

At the very top of the program I decided to organize everything into subroutines. The idea here is to enable expansion into other ML algorithms.

The “wait for key” subroutine is the APPLESOFT BASIC method for simply waiting for any keystroke before continuing. (PEEK and POKE are commands for directly accessing addresses in memory. I had those numbers memorized in high school but, naturally, I had to look them up.) I thought it’d be nice to add this pause after generating the data but I might take it out later.

Lastly, at the end of the “hyperparameters” section I declare a convenience array, P%(2,1) to keep track of 3 random points as well as a few arrays I’m going to use in the k-means algorithm. The reason I do this here is because in APPLESOFT BASIC you get an error if you declare an array that already exists. Should at some point I want to call the k-means algorithm multiple times, this won’t be a problem.

Initialize

Getting started, the first thing to do is initialize the algorithm by generating \(k\) cluster centroids. (\(k\) is a hyperparameter that specifies the number of clusters to be “found.” I set it previously with KN = 2.)

2000 REM  == K-MEANS ==
2010 PRINT "RUN K-MEANS"
2020 REM  -- CLEAR PREDICTIONS --
2030 FOR I = 0 TO NS - 1
2040   DS%(I,3) = 0
2050 NEXT I
2100 REM  -- INITIALIZE CENTROIDS --
2110 FOR I = 0 TO KN - 1
2120   J = INT(RND(1) * NS)
2130   IF DS%(J,3) = 1 GOTO 2120
2140   KM(I,1) = DS%(J,1)
2150   KM(I,2) = DS%(J,2)
2160   DS%(J,3) = 1
2170 NEXT I
2200 REM  -- DRAW LINES BETWEEN CENTROIDS --
2210 FOR I = 1 TO KN - 1
2220   HPLOT KM(I-1,0), 159-KM(I-1,1) TO KM(I,0), 159-KM(I,1)
2230 NEXT I
2240 GOSUB 3000: REM  DRAW DECISION BOUNDARY

I start by clearing out the prediction column, \(y\), of the dataset table, DS%(NS - 1,3) because I’m going to use this to make sure I don’t randomly pick the same point twice. Then for each class I randomly pick a point from the dataset. If it’s already been used I randomly pick another. KM(KN - 1, 2) is where I store the means for each cluster along with a count of the number of points in each cluster.

Finally, I draw a line between the cluster centroids. This loop does not take into account all combinations of centroids (it works fine if \(k=2\)) and generates an error if a centroid is off the screen, which is possible, so I might just get rid of this later, since it’s not really necessary, rather than try to fix it.

Step 1 - Assignment

The fist step is to assign every data point to the nearest cluster centroid.

2300 REM  -- COMPUTE ASSIGNMENTS --
2310 FOR I = 0 TO NS - 1
2320   PRINT "POINT ";I;" AT ";DS%(I,0);",";DS%(I,1);
2330   DS%(I,3) = 0
2340   FOR J = 0 TO KN - 1
2350     DI(I,J) = (DS%(I,0)-KM(J,0))^2 + (DS%(I,1)-KM(J,1))^2
2360     IF J >0 AND (DI(I,J) < DI(I,DS%(I,3))) THEN DS%(I,3) = J
2370   NEXT J
2380   PRINT " -> ";DS%(I,3);" Y^=";DS%(I,2)
2390 NEXT I
2500 REM  -- COMPUTE ACCURACY --
2510 CT = 0
2520 FOR I = 0 TO NS - 1
2530   IF DS%(I,2) = DS%(I,3) THEN CT = CT + 1
2540 NEXT I
2550 A = CT / NS
2560 IF A < 0.5 THEN A = 1 - A
2570 PRINT "ACCURACY = "; INT(A*10000+0.5)/100;"%"

The assignment step is also quite easy. I loop through all the data points, computing the Euclidean distance to each cluster centroid. (Since SQRT() is expensive, and unnecessary here since we’re just comparing, I actually just compute the square of the Euclidean distance.) If the distance is less than the previous minimum distance, DI(I,DS%(I,3)), I update the assignment, DS%(I,3) = J.

At the end, I compute the accuracy of the computed assignments by simply counting the number of assignments, DS%(I,3), that match the actual labels, DS%(I,2). Here, however, there’s an interesting wrinkle: with two classes, half the time the label I choose for the assignment is the opposite of the label from the original dataset. K-means doesn’t require the distinction, so at times I was seeing a perfect classification reporting 0% accuracy. The line IF A < 0.5 THEN A = 1 - A addresses this, however, it only works for 2 classes. I’ll need something more robust should I want this to work for \(k > 2\).

Step 2 - Update

The second step is to recompute the cluster centroids based on the assigned data points. Convergence occurred if the centroids don’t change (within a tolerance) from the previous iteration.

2600 REM  -- COMPUTE CENTROIDS --
2610 FOR J = 0 TO KN - 1
2620   K0(J,0) = KM(J,0)
2630   K0(J,1) = KM(J,1)
2640   KM(J,0) = 0: KM(J,1) = 0
2650   KM(J,2) = 0
2660 NEXT
2670 FOR I = 0 TO NS - 1
2680   Y = DS%(I,3)
2690   KM%(Y,0) = KM%(Y,0) + DS%(I,0)
2700   KM%(Y,1) = KM%(Y,1) + DS%(I,1)
2710   KM%(Y,2) = KM%(Y,2) + 1
2720 NEXT
2730 FOR I = 0 TO KN - 1
2740   KM%(I,0) = KM%(I,0) / KM%(I,2)
2750   KM%(I,1) = KM%(I,1) / KM%(I,2)
2760 NEXT
2800 REM  -- DETERMINE CONVERGENCE --
2810 DI = 0
2820 FOR I = 0 TO KN - 1
2830   DI = DI + (KM%(I,0) - KO%(I,0)) ^ 2 + (KM%(I,1) - KO%(I,1)) ^ 2
2840 NEXT
2850 IF DI > 0.01 THEN GOTO 2200
2860 PRINT "K-MEANS CONVERGED"
2900 REM  -- CLEAR GRAPHICS AND REDRAW WITH DECISION BOUNDARY --
2910 GOSUB 1000
2920 FOR I = 0 TO NS - 1
2930   X0% = DS%(I,0)
2940   X1% = DS%(I,1)
2950   K = DS%(I,2)
2960   ON K + 1 GOSUB 1200,1300
2970 NEXT
2980 GOSUB 3000
2990 RETURN

I start by saving the cluster centroids to KO(KN - 1,1). This is used later to determine convergence. I then iterate through ever data point, adding it’s values to the cluster to which it belongs while keeping track of the number of data points in each cluster. Next I iterate through each cluster and compute the mean of each dimension by dividing by the number of data point in that cluster.

Lastly, I determine if there’s convergence by measuring how far all the centroid have moved. (Again, I don’t bother with the SQRT().) If the answer is more than the specified tolerance, \(0.01\), I go back to Step #1. Otherwise, I clear the graphics, redraw the axis and data points and finish by drawing the decision boundary.

Drawing the decision boundary

This code is a slog and it’s not really critical to understanding ML but I thought it’d be cool to drawn a decision boundary while k-means is iterating and then again at the end. Given a point (the midpoint on the segment between two cluster centroids) and a slope (which is perpendicular to that segment), the challenge is to drawn a line inside the ‘box’ of the screen, assuming the line intersects that box.

3000 REM  -- DRAW DECISION BOUNDARY --
3010 FOR I = 1 TO KN - 1
3020   M = 1E6
3030   IF KM%(I - 1,1) - KM%(I,1) <> 0 THEN M = -1 * (KM%(I - 1,0) - KM%(I,0)) / (KM%(I - 1,1) - KM%(I,1))
3040   P%(0,0) = (KM%(I,0) - KM%(I - 1,0)) / 2 + KM%(I - 1,0)
3050   P%(0,1) = (KM%(I,1) - KM%(I - 1,1)) / 2 + KM%(I - 1,1)
3060   GOSUB 3080
3070 NEXT
3080 REM  -- DRAW LINE FROM SLOPE AND POINT --
3090 NX = 1 : REM  -- REM NUMBER OF INTERSECTIONS --
3100 IF ABS(M) > 1E5 THEN GOSUB 3240 : GOTO 3210 : REM  VERTICAL LINE
3110 P%(NX,1) = M * (10 - P%(0,0)) + P%(0,1)
3120 IF P%(NX,1) > 10 AND P%(NX,1) < 149 THEN P%(NX,0) = 10 : NX = NX + 1
3130 P%(NX,1) = M * (269 - P%(0,0)) + P%(0,1)
3140 IF P%(NX,1) > 10 AND P%(NX,1) < 149 THEN P%(NX,0) = 269 : NX = NX + 1
3150 IF NX = 3 THEN GOTO 3210
3160 IF M <> 0 THEN P%(NX,0) = (10 - P%(0,1)) / M + P%(0,0)
3170 IF M <> 0 AND P%(NX,0) > 10 AND P%(NX,0) < 269 THEN P%(NX,1) = 10 : NX = NX + 1
3180 IF NX = 3 THEN GOTO 3210
3190 IF M <> 0 THEN P%(NX,0) = (149 - P%(0,1)) / M + P%(0,0)
3200 IF M <> 0 AND P%(NX,0) > 10 AND P%(NX,0) < 269 THEN P%(NX,1) = 149 : NX = NX + 1
3210 REM  -- DRAW LINE --
3220 IF NX = 3 THEN HPLOT P%(1,0),159 - P%(1,1) TO P%(2,0),159 - P%(2,1)
3230 RETURN
3240 REM  -- VERTICAL LINE --
3250 P%(1,0) = P%(0,0)
3260 P%(2,0) = P%(0,0)
3270 P%(1,1) = 10
3280 P%(2,1) = 269
3290 RETURN

Without delving too far into the details, this routine relies heavily on the convenience array, P%(2,1), that I declared during the “hyperparameters” routine. I start by computing the slope of the perpendicular segment connecting two centroids. I then find the midpoint of that segment. (By the way, this routine also does not account for all combinations of centroids, but it works when \(k=2\).) I accommodate for when the slope is vertical and use P%(0,0) and P%(0,1) to store the midpoint between the two centroids and M for the slope.

I then iterate through the 4 sides of the ‘box’ on the screen, using the corners (10,10) and (269,149) so that the decision boundary isn’t drawn all the way to the edges of the screen. I thought that would look prettier this way. I next determine if the decision boundary intersects, respectively, the left, right, top and bottom edges of the box. I use NX to keep track of the number of sides of the box intersected by the decision boundary and P%(NX,0) and P%(NX,1) to keep track of those intersections. If NX = 3, which means there are two intersections, I draw the line because it’s inside the box.

Can we do better?

Yes! Yes, we can.

While k-means is simple, it does not take advantage of our knowledge of the Gaussian nature of the data. If we know that the distributions are at least approximately Gaussian, which is frequently the case, we can employ a more powerful application of the Expectation Maximization (EM) framework (k-means is a specific implementation of centroid-based clustering that uses an iterative approach similar to EM with ‘hard’ clustering) that takes advantage of this. This post is already long enough, so we’ll deal with that another day. Eventually, perhaps, we’ll also get to deep learning, although developing back propagation for an arbitrary size neural net using APPLESOFT BASIC on an Apple ][+ is not going to be easy.

Social media sharing

I shared this post on Hacker News and, to my surprise and delight, got a decent amount of feedback, including some helpful corrections.

My post on Facebook received significantly less reaction. My friends are apparently a lot less interested in ML algorithms on retro hardware than randos on Hacker News. My share requests to Facebook groups Apple II Enthusiasts and Retro Microcomputers were both rejected, which is a bit disappointing. Still, those are fun groups to follow. My post on the VCF forum also got zilch. (I attend the Vintage Computer Festival at the Computer History Museum, which was a good time.) Perhaps I’ll try Reddit and LinkedIn next to see what happens…

Are you actually writing and running your code on the real physical Apple ][, or are you using an emulator? If you’re using a real one, how are you transferring the code back and forth from the Apple ][ to the blog? I’m guessing you’re not using generative AI to write the code, since it sounds like writing it by hand is part of the fun. Is that right?

I sent him the answers but I thought I might share here, too.

Running on the real hardware

He might have missed my post about bringing the clunker back to life but I’m running on the real hardware from early ’80s, which I recently refurbished with new capacitors in the power supply and new RAM chips on the motherboard.

I can’t remember exactly when my parents ponied up a bought me the Apple ][+ but it was after years of me hassling them relentlessly. In their defense, no one had a personal computer back then and it wasn’t entirely clear what one would do with one. The machine was very expensive and they didn’t want to pay that kind of money for a video game machine. I vaguely recall the Apple ][+ with a pair of floppy drives costing ~$2000 plus a few hundred for the extra 16k RAM on the expansion memory card and ~$500 for the dot matrix printer. That would be ~$9000 today. (For comparison, we had a Magnavox Odyssey² which was under $200. The cartridges were ~$25.) My father was an aircraft mechanic and my mother stayed home to raise two children, so they paid close attention to finances.

My obsession with the Apple computer began during the summer after the 6^th grade. My mother signed me up for computer camp where we learned how to write code on the Apple ][. These machines didn’t even have floppies, so we used cassette tape recorders. You’d press ‘record’ on the tape recorder before entering SAVE to store the program. To get it back you’d type LOAD and then press ‘play’ on the tape recorder. It wasn’t punch cards but that’s wild. My program was a lowres picture of a Star Wars battle (another childhood obsession).

From that moment, I had to have one. Every single birthday or holiday from that point forward, when anyone asked me what I wanted, to everyone’s dismay, I’d say, “Money.” My plan was to save up enough to get one. In the meantime, since these computers were not readily available, I would write BASIC programs on notebook paper (to do what, I can’t remember), ride my bike to the mall and then try them out on the TRS-80 Color Computers at Radio Shack. As I recognized at the time, the sales people there were extremely cool to let me do that.

Saving up that kind of money took years but sometime around the 9^th grade I had enough to get a TRS-80, which was significantly less expensive at about half the cost. My parents, realizing that I was about to drop every penny I had on a computer inferior to the one I really wanted, stepped up and surprised me. I can still remember the boxes sitting in the living room when I got home.

Suffering on the real hardware

While I know that there are emulators, which would probably make life easier, I am not using them. While writing Applesoft BASIC on the original hardware feels authentic, it comes with some challenges:

• The keyboard works but it’s 40+ years old and isn’t particularly ergonomic.
• The monitor is only 40 characters wide because, for some reason, I don’t have the 80-column card. (I vaguely recall having an 80-character wide display but perhaps that was on the school’s computers. I also remember having a color monitor but that must have been the school as well.)
• Reading the code requires the LIST command but without specifying line numbers the entire program will quickly scroll by. Therefore, I have to painstakingly enter the line numbers each time.
• There isn’t a particularly effective way to edit lines of text. This super enthusiastic dude demonstrates how to use the ESC character to edit a line of code but it’s a real hassle. I’ve used this (although on my machine, which I found in the manual, you have to use a, b, c and d, which is confusing, plus hold down ESC the entire time) but it’s almost always easier to just retype the line. (Update: The I, J, K, M works and ESC toggles if you use those keys. It takes a little getting used to but it’s much better.)
• Applesoft BASIC has a “debug mode” with TRACE, which essentially prints out line numbers as they’re executed, but I’ve never used it. It sounds like a nightmare.

Transferring Applesoft BASIC code to this blog

While there’s probably a way to connect the Apple ][+ to my Windows machine, I solve this problem by taking a picture of the monitor with my phone and then asking chatGPT to extract the code. The first attempts weren’t too bad, but I had to iterate quite a few times to get it to output correctly. For example, if a single line of code wrapped on the screen, chatGPT would add extra line numbers. It also messed up the counting on occasion. It sometimes interpreted the percent sign as a dollar sign. That being said, it saves me a ton of work. Interestingly, I think I ‘trained’ chatGPT on how to read the Applesoft BASIC code because outputs have become increasingly higher quality.

I tried taking a video of the entire program scrolling by but the quality was not good enough for OCR.

No generative AI

Or not much, at any rate. GenAI is amazing and Cursor is beyond fantastic, but, as my friend said, part of the fun is writing the code myself. I’ll admit to getting assistance with adpative quota sampling, which isn’t particularly difficult. I’ve also asked about how some commands work (e.g. RND()) although I’ve been relying primarily on the original reference manual. The rest, however, is me.

I suppose there’s a balance between planning out what you’re going to do and just getting started. For my this project, I leaned toward the latter. As such, after the first pass I already decided to change it up. More specifically, the first thing I decided to do was, rather than have a different array for each class of data, move everything into a single array. The reason is that, during training, this will facilitate looping over all the data.

Quick note on line numbers: refactoring BASIC almost always consists of changing line numbers. As such, this code is not going to match up with previous posts. Hopefully that won’t be confusing.

Refactoring the hyperparameters

To make that happen, I start by moving all of my generator hyperparameters into a single array, GE(..,..), where the first axis is the class. The second axis contains, respectively, \(u_{x_0}, u_{x_1}, \sigma_{x_0}, \sigma_{x_1}, \rho\) and \(\sqrt{1 - \rho^2}\). (See Synthesizing data - Box-Muller for details.) The last element is purely for performance since this quantity will be needed frequently. (Upon further refection, I could have used \(\sqrt{1 - \rho^2} \sigma_{x_1}\) but this is a small optimization for later.)

100 REM == HYPERPARAMETERS ==
110 KN = 2: REM # OF CLASSES
120 DIM KN%(KN - 1): REM # OF SAMPLES FROM EACH CLASS
130 KN%(0) = 100
140 KN%(1) = 50

150 REM -- GENERATORS --
160 REM -- CLASS - MU-X0, MU-X1, SIG-X0, SIG-X1, RHO --
170 DIM GE(KN - 1,5)

180 REM -- CLASS 0 --
190 GE(0,0) = 100: REM MU-X0
200 GE(0,1) = 50: REM MU-X1
210 GE(0,2) = 30: REM SIG-X0
220 GE(0,3) = 10: REM SIG-X1
230 GE(0,4) = 0.5: REM RHO
240 GE(0,5) = SQR(1 - GE(0,4) ^ 2)

250 REM -- CLASS 1 --
260 GE(1,0) = 200: REM MU-X0
270 GE(1,1) = 100: REM MU-X1
280 GE(1,2) = 50: REM SIG-X0
290 GE(1,3) = 25: REM SIG-X1
300 GE(1,4) = -0.5: REM RHO
310 GE(1,5) = SQR(1 - GE(1,4) ^ 2)
320 REM == END HYPERPARAMETERS ==

Note: here is a funny thing with Applesoft BASIC arrays. I couldn’t remember if there was 0- or 1-indexed. Turns out, they are both! DIM A(3) will create an array of length 4 which goes from index 0 to 3. I can’t recall seeing anything like that anywhere else. So, I use DIM GE(KN-1,4) because I want the first axis to have KN elements and the second axis to have 5. Funny.

Another note: Applesoft BASIC only uses the first two characters of a variable names, however, KN and KN%(..) are considered different.

Generating the samples

After counting up the total number of samples, I then create DS%(..,..) which replaces AX%(..) and BX%(..). The first axis contains the samples, of which there are NS in total. The second axis is, respectively, \(x_0, x_1, \hat{y}\) and \(y\). \(\hat{y}\) is the label (i.e., the class) and \(y\) is the prediction, which we’ll get to later. Having all the data in a single array will facilitate generating and training.

330 NS = 0: REM TOTAL # SAMPLES
340 FOR I = 0 TO KN - 1
350   NS = NS + KN%(I)
360 NEXT

370 REM -- DATA TABLE --
380 REM -- # - X0, X1, Y-HAT, Y --
390 DIM DS%(NS - 1,3)

Since I’d like for my data to be randomly sampled, rather than scrambled after the fact, I decided to use ‘adaptive quota sampling’ to generate them randomly while ending up with the appropriate quantities. I could have simply taken the end proportion of each class and use that to create probabilities during generation but then there would be no guarantee then the final number of samples for each class would be correct.

400 REM -- REMAINING SAMPLES COUNT --
410 DIM RK%(KN - 1)
420 FOR I = 0 TO KN - 1
430   RK%(I) = KN%(I)
440 NEXT

450 REM == GENERATE SAMPLES ==
460 FOR I = 0 TO NS - 1
470   REM -- ADAPTIVE QUOTA SAMPLING --
480   REM -- PICK CLASS BASED ON REMAINING SAMPLES TO GENERATE --
490   P% = RND(1) * (NS - I)
500   CS = 0
510   J = 0
520   CS = CS + RK%(J)
530   K = J
540   J = J + 1
550   IF P% >= CS GOTO 520
560   RK%(K) = RK%(K) - 1
570   REM -- GENERATE RANDOM NORMAL VARIABLES Z0, Z1 --
580   GOSUB 1400
590   REM -- GENERATE SAMPLE FROM Z0, Z1 --
600   REM X0 = MU-X0 + SIG-X0 * Z0
610   X0% = GE(K,0) + GE(K,2) * Z0
620   REM X1 = MU-X1 + RHO * SIG-X1 * Z0 + SQR(1 - RHO^2) * SIG-X1 * Z1
630   X1% = GE(K,1) + GE(K,4) * GE(K,3) * Z0 + GE(K,5) * GE(K,3) * Z1
640   DS%(I,0) = X0%
650   DS%(I,1) = X1%
660   DS%(I,2) = K
670   PRINT I + 1; TAB(5); "SAMPLE "; KN%(K) - RK%(K);
680   PRINT TAB(16); "IN CLASS "; K;
690   PRINT " AT ("; X0%; ","; X1%; ")"
700   ON K + 1 GOSUB 1200,1300: REM PLOT SAMPLE
710 NEXT

RK%(..) holds the remaining number of samples to be generated for each class. Lines 490 to 550 then produce a random number P% which is from 0 to the number of remaining samples to generate and then figures out which class this should be based on the remaining number of samples to be generated for each. The beauty of this is that the probability of generating a sample in each class will be a function of how many remaining samples need to be generated. It took a while to debug, but in the end this produces exactly the number of samples in each class as specified in the hyperparameters.

The code from lines 580 to 690 is the same as previously, however, the class-specific variables have been replaced by GE(..,..), as mentioned above. Also, I changed X% and Y% to XO% and X1% to avoid confusion and be more consistent with the \(Y = \theta^T X\) literature.

Line 700 is nifty. I don’t recall using ON..GOSUB in high school but it enables branching based on the argument. I use K+1 since the first class is \(0\) and a zero argument doesn’t branch. For the moment, this only supports two classes.

Counting the samples

For debugging purposes, I added some code to count the number of samples in each class to make sure it matched the hyperparameters. In the end, I decided to leave it. The loop is actually a bit slow but it’s only run once.

720 REM == COUNT SAMPLES OF EACH CLASS TO VERIFY RESULT ==
730 DIM CT%(KN - 1)
740 FOR I = 0 TO NS - 1
750   K = DS%(I,2)
760   CT%(K) = CT%(K) + 1
770 NEXT
780 PRINT "TOTAL SAMPLES FOR EACH CLASS:"
790 FOR I = 0 TO KN - 1
800   PRINT CT%(I); TAB(5); "SAMPLES IN CLASS "; I
810 NEXT
820 END

Fixing the plotting routines

The only remaining tweaks were for the plotting routines. The two routines below, which are called from the ON..GOSUB in line 700, are updated to use XO% and X1%. The rest is the same.

1200 REM == DRAW + AT X0, X1 ==
1210 X1% = 159 - X1%
1220 IF X0% < 1 OR X0% > 270 THEN RETURN
1230 IF X1% < 1 OR X1% > 150 THEN RETURN
1240 HPLOT X0% - 1,X1% TO X0% + 1,X1%
1250 HPLOT X0%,X1% - 1 TO X0%,X1% + 1
1260 RETURN

1300 REM == DRAW BOX AT X0, X1 ==
1310 X1% = 159 - X1%
1320 IF X0% < 1 OR X0% > 270 THEN RETURN
1330 IF X1% < 1 OR X1% > 150 THEN RETURN
1340 HPLOT X0% - 1,X1% - 1 TO X0% + 1,X1% - 1
1350 HPLOT X0% - 1,X1%: HPLOT X0% + 1,X1%
1360 HPLOT X0% - 1,X1% + 1 TO X0% + 1,X1% + 1
1370 RETURN

Testing Irwin-Hall as an alternative to Box-Muller

The final chunk of code is interesting. After sharing my Box-Muller routine with my friend, Răzvan Surdulescu, he suggested using the Irwin-Hall Distribution. SQR and COS are expensive operations in BASIC and Irwin-Hall approximates a normal distribution by sampling 12 uniform random numbers.

1500 REM == IRWIN-HALL DISTRIBUTION ==
1510 Z0 = -6:Z1 = -6
1520 FOR Z = 1 TO 12
1530   Z0 = Z0 + RND(1)
1540   Z1 = Z1 + RND(1)
1550 NEXT
1560 RETURN
1600 REM == RANDOM GEN SPEED TEST ==
1610 FOR I = 1 TO 500
1620   GOSUB 1400
1630 NEXT

I gave it a go and ran a comparison by generating 500 pairs of random numbers with each. This took 63s with Box-Muller and 90s with Irwin-Hall, so I will stick with the former. That being said, this was a nifty idea.

An updated final plot

The final result looks pretty much the same as before but it’s flipped horizontally. The new data structure, however, will make implementing ML much easier.

Now onto implementing the simplest and easiest to understand machine learning algorithm…

Every solid machine learning project begins with data. My old clunker, however, is cut off from the world. I never had a modem but I recall one of my best friends using the modem on his Apple ][ to hack long distance access codes. (He since went on to pursue a successful career in telecommunications… naturally.) Even if I had a modem, what would I connect it to? Do those services still even exists? (I’m sure they do but I’m not going to figure that out today.) In the meantime, the plan is to create synthetic data locally.

I’m sure everyone is familiar with the chart comparing different classification algorithms. Since two-dimensional binary classification seems like a great place to start, I’ll need a simple way to graph the data.

Fortunately, the Apple ][+ comes with a high-resolution graphics screen (HGR) that is 280 pixels across by 160 pixels high. (There is another mode with 192 vertical pixels but I wanted leave the bottom 4 lines of the text window visible for running output.) There are 8 color options for each pixel, but I have a green screen monitor, so I set everything to HCOLOR=7 (white2).

Drawing the axes

Deciding to stick to the positive quadrant, I wrote a subroutine (Applesoft BASIC does not have methods) to simply display the axes. Interestingly, coordinate (0,0) is the upper left corner of the screen while (279,159) is the lower right, so HPLOT 0,159 TO 279,159 draws the x-axis.

500 REM  == SET GRAPHICS AND DRAW Y AXIS WITH TICK MARKS ==
510 HGR : HCOLOR=7
520 HPLOT 0,159 TO 279,159
530 HPLOT 0,159 TO 0,0
540 J = 0
550 FOR I = 159 TO 0 STEP -10
560   HPLOT 0,I TO 1,I
570   IF J / 5 = INT(J / 5) THEN HPLOT 0,I TO 2,I
580   IF J / 10 = INT(J / 10) THEN HPLOT 0,I TO 3,I
590   J = J + 1
600 NEXT I
610 J = 0
620 FOR I = 0 TO 279 STEP 10
630   HPLOT I,159 TO I,158
640   IF J / 5 = INT(J / 5) THEN HPLOT I,159 TO I,157
650   IF J / 10 = INT(J / 10) THEN HPLOT I,159 TO I,156
660   J = J + 1
670 NEXT I
680 RETURN

The first FOR I loop adds tick marks along the y-axis. I decided to get fancy and add elongated tick marks at every 5th unit and even longer ticks marks at every 10 unit. I use J to count the ticks. The second FOR I loop does the same for the x-axis. The final result looks like this:

Synthesizing the data

With a particular fondness for all thing Gaussian, I decided to create two sets of data points with Gaussian distributions. They look pretty and they’re fun.

10  HOME : VTAB 21
20  PI = 3.14159265
30  GOSUB 500 : REM  DRAW AXIS
40  DIM AM%(2) : REM  -- MEAN
50  DIM BM%(2)
60  DIM AS%(2) : REM  -- STD
70  DIM BS%(2)
80  REM  == HYPERPARAMETERS ==
90  AN = 100 : REM  # OF POINTS
100 BN = 50
110 AM%(1) = 100 : AM%(2) = 50
120 BM%(1) = 200 : BM%(2) = 100
130 AS%(1) = 30 : AS%(2) = 10
140 BS%(1) = 50 : BS%(2) = 25
150 AC = 0.5
160 BC = -0.5
170 REM  == END HYPERPARAMETERS ==

HOME clears the screen and VTAB 21 moves the cursor to the 21st text line on the screen, which will be the first line under the graphics after HGR. The Apple ][+ doesn’t come with a constant for π so I set that up using all the precision available.

AM% is a two element array of integers to store, respectively, the x-mean and y-mean values for the A data set. (The % sign makes the variable an integer which save space but actually reduced performance because mathematical operations on the Apple ][+ convert integers to real numbers and then back again.) AS% does the same for the x- and y-standard deviations. AC is a real number correlation coefficient ∈ [-1, 1]. Finally, AN is the number of elements in the A dataset. The corresponding B data set hyperparameters are similar. (In Applesoft BASIC only the first two characters of a variable name are ‘considered’ so you have to be careful of collisions.)

Box-Muller to the rescue

The Apple ][+ can generate uniform random variables from 0 to 0.999999999 using RND(), however, to get standard normal random variables we’ll have to use the Box-Muller transform.

Given two independent samples, \(u_1\) and \(u_2\), chosen from a uniform distribution on the unit interval (0, 1), we can get two independent random variables, \(z_0\) and \(z_1\), with a standard normal distribution with the following:

Here is the code to do that.

900 REM  == BOX-MULLER TRANSFORM ==
910 U1 = RND(1)
920 U2 = RND(1)
930 R = SQR(-2 * LOG(U1))
940 TH = 2 * PI * U2
950 Z0 = R * COS(TH)
960 Z1 = R * SIN(TH)
970 RETURN

From there, to obtain a 2D Gaussian with mean \(\mu_x, \mu_y\) and covariance matrix \(\Sigma\), we’ll need to apply the following, where \(\sigma_x\) and \(\sigma_y\) are the standard deviations in the \(x\) and \(y\) directions, and \(\rho\) is the correlation coefficient:

Here is the code that generates all the samples using these transformations.

200 DIM AX%(AN,2)
210 DIM BX%(BN,2)
220 FOR I = 1 TO AN
230   GOSUB 900 : REM FETCH STANDARD NORMAL RANDOM VALUES
240   X% = AM%(1) + AS%(1) * Z0
250   Y% = AM%(2) + AC * AS%(2) * Z0 + SQR(1 - AC ^ 2) * AS%(2) * Z1
260   AX%(I,1) = X%
270   AX%(I,2) = Y%
280   PRINT "POINT IN SET A "; I; " AT ("; X%; ","; Y%; ")"
290   GOSUB 700 : REM  DRAW A +
300 NEXT I
310 FOR I = 1 TO BN
320   GOSUB 900 : REM FETCH STANDARD NORMAL RANDOM VALUES
330   X% = BM%(1) + BS%(1) * Z0
340   Y% = BM%(2) + BC * BS%(2) * Z0 + SQR(1 - BC ^ 2) * BS%(2) * Z1
350   BX%(I,1) = X%
360   BX%(I,2) = Y%
370   PRINT "POINT IN SET B "; I; " AT ("; X%; ","; Y%; ")"
380   GOSUB 800 : REM  DRAW A BOX
390 NEXT I
400 END

I decided to store the data samples in AX% and BX% as integers to save on memory. Since they are all randomly generated, the extra precision won’t make much of a difference anyway. I’m using 2D arrays where the first axis is the number of samples (e.g., AN) and the second axis is the feature dimension (e.g., 2 for \(x\) and \(y\)).

You can see calling GOSUB 700 and GOSUB 800 to plot the points. Here is the final code for that.

700 REM == DRAW + AT X,Y ==
710 IF X% < 1 OR X% > 270 THEN RETURN
720 IF Y% < 1 OR Y% > 150 THEN RETURN
730 HPLOT X% - 1,Y% TO X% + 1,Y%
740 HPLOT X%,Y% - 1 TO X%,Y% + 1
750 RETURN
800 REM == DRAW BOX AT X,Y ==
810 IF X% < 1 OR X% > 270 THEN RETURN
820 IF Y% < 1 OR Y% > 150 THEN RETURN
830 HPLOT X% - 1,Y% - 1 TO X% + 1,Y% - 1
840 HPLOT X% - 1,Y%: HPLOT X% + 1,Y%
850 HPLOT X% - 1,Y% + 1 TO X% + 1,Y% + 1
860 RETURN

I first verify that the data point is on the visible region of the graph (plotting off the screen will throw an error) and then draw either a “+” or a “□”.

Running the code

Putting it all together, it takes the Apple ][+ about a minute to generate and plot 150 data points. The end result looks like this:

Watch this video if you’d like to see it run, in all it’s glory.

Now that this is out of the way, we’ll start by implementing one of the simplest and easiest to understand machine learning algorithms.

Resigned, I looked at my box of archaic floppies and contemplated throwing them into the trash… when a thought occurred to me: what about the backsides?

Virtually all of my floppies are single sided, but as a broke high school student I didn’t care. Using a hole punch to open a write-protect notch on the other side of the floppy, I almost always flipped them over and wrote on the backside. Is it possible that, for some reason, I backed up the game on the “side B” of one of these floppies? Sure, I guess anything is possible.

So, I kept them. Because I am stubborn an optimist!

Another helping hand

A few months later, a good friend, Răzvan Surdulescu (Principal Software Engineer, Robotics at Google DeepMind), came over to dinner with his family. Being a vintage computer enthusiast himself, who enjoys watching videos about restoring computers (who doesn’t?), he was thrilled to see the Apple ][+ sitting on my office desk. When I gave him the whole blah blah about the high school computer game, while my wife rolled her eyes as she left the room, he suggested we try recovering it with Greaseweazle.

Greaseweazle is an open-source USB device and software that allows you to read and write data from various types of floppy disks, including those used by old Apple ][+s. It captures the raw flux transitions from the floppy, enabling it to handle a wide range of formats beyond what is typically supported by standard floppy disk controllers. To get it to work, however, we’d need some new hardware.

New hardware

While Răzvan had the latest v4.1 Greaseweazle hardware at home, we’d need a standard Shugart floppy drive (not an Apple ][+ floppy drive) to read floppies. This is because Greaseweazle uses the Shugart interface to access the raw flux transitions on the disk, rather than relying on the drive’s built-in encoding and decoding. This allows it to read and write various disk formats, including those with custom encoding or copy protection.

No problem. A few eBay searches later, I purchased this Toshiba 5¼” 1.2Mb Internal Floppy Disk Drive ND-08DE for $76. There were different DOS formats for the Apple ][+, but this capacity is significantly greater than the whopping 140Kb with Apple DOS 3.3.

The new drive would also require some new cables to hook it up. They were inexpensive. Răzvan already had a few of the necessary cables and very helpfully made sure that I bought all the right stuff.

The opportunity to give it a go arrived when we were invited to a dinner party at the Surdulescu’s. We plugged everything into Răzvan’s beefy homemade Windows desktop, grabbed a couple flutes of Champagne and got to work.

Problems with the backside

The first thing I noticed was how quickly Greaseweazle was able to read data off the floppies. Unlike the Applesauce USB disk controller that needed to grind back and forth to find the data using an original Apple ][+ floppy drive, Greaseweazle was able to find the floppy catalog in a second. (We used FluxEngine, seen below, which is different from the command line utility that comes with the GreaseWeazle, to test the directory of files on the disk. If that didn’t work, it had an option to read the flux off the disk which involved grinding back and forth to try to locate data. We had to do that quite a few times.)

While we were able to find quite a bit of data, including a few titles from Beagle Bros, it seems the Apple ][+ may have used different kinds of data encodings to lay the sectors on the disk. Even if Greaseweazle can read the magnetic flux, the actual semantics of the flux (what sector goes where and in what order) can be opaque and the FluxEngine software might not interpret them in the right way. This might require more investigation.

After checking a few disks to verify that we were getting the same results as the last time I gave this a go, we decided to flip one over and try the backside. Immediately, we noticed that it wasn’t going to work. To calibrate the rotational timing, Greaseweazle uses the photoelectric sensor on the Shugart floppy drive which receives a ‘pulse’ every time the index hole, near the center of the floppy, passes over an LED. When you flip the floppy over, however, the index hole cutout is now no longer over the LED and so the pulse can’t be read.

One solution was to cut another hole in the floppy case. Given the proximity to the center of the floppy, however, this risked damaging the media. We explored some software solutions but then eventually decided it would be easiest to rip open the floppy and then manually flip the media. I had never opened a floppy before, but it was a relatively easy process. The old plastic was brittle and frequently crumbled in my fingers, but we were able to make it work.

Disappointment redux

Unfortunately, the result was no different from my previous attempt: there was no game to be found. The process wasn’t relatively quick. We were able to rip open and flip the media on about half of the floppies. There are ~25 to go so perhaps we’ll get to those one day. And then I’ll just discard everything. In the meantime, hopefully I’ll have some time to get back to deep learning…

The game logic was written in BASIC but this was much too slow for the graphics. Therefore, I convinced my parents to drive me to a remote bookstore to pick up a book on Apple ][ assembly (there were not a lot of resources in those early days) and then taught myself how to develop assembly subroutines that could be called from BASIC. I had a 2D array which I used as a map and then moved hand-crafted images directly to the graphics memory as the character moved around. This already difficult process was more complicated by the fact the computer’s graphics memory was interleaved and the Apple CPU didn’t have a multiply command. Fortunately, the book offered a routine that would accomplish this with bit shifting and addition.

Side note: After all these years, only today did I learn that Richard Garriott (aka Lord British) coded Ultima in BASIC while his friend Ken Arnold “wrote code in assembly language for the tile-based graphics system, the first game in the genre to do so.”

To this day I’m pretty proud of this little piece of computer wizardry and would love to see it again.

Alas, this is not to be.

Box of floppies

Buried in my parent’s garage for decades, alongside the old Apple ][+, was a box of 5.25” floppy disks. Was my game still on one of them?

For some unknown reason, my high school brain decided to just number the floppies, as opposed to just writing the contents on the stickers. I used to have a sheet that listed the contents of each disk, but that is long gone. Additionally, judging by the enumeration, it looks like some of the floppies are missing. I vaguely recall taking some to college. Was the game on one of them? Perhaps.

A glimmer of hope on Twitter

I don’t recall how I came across @bzotto’s Twitter thread about how he resurrected his high school game by reconstructing the data from an old 5.25” floppy, but it renewed my hope. He purchased some specialized hardware that repeatedly moved the drive head back and forth by fractions of a track to try to pull off enough magnetic signal to reconstruct the data.

Turns out @bzotto is a great guy and extremely generous with his time, so when I reached out with my story he offered to help.

Coffee shop excavation

In February 2022 we met at a coffee shop near his home. I brought the box of floppies and he brought the hardware and his Mac laptop. We spent three hours chatting as we tried one floppy after another.

The floppies almost always started with a bunch of red squares. His software would make an initial pass to see what could be read, and typically that was not very much. The floppy had been sitting in a damp, cold garage for decades and most of the magnetic material had degenerated or fallen off. It did not help matters that I purchased inexpensive, single density floppies in high school, and then recorded on both the front and back. It worked fine back then, but I was clearly not thinking long term.

As we patiently allowed the hardware and software to do its magic, something incredible happened. The red squares became green! Not all of them, but enough for us to identify what, if anything, was on the floppy. We were primarily interested in the catalog which, if memory serves correctly, was on track 14. I had a few floppies that I used for personal development, so the catalog would enable us to find these.

Disappointment

Unfortunately, the old game floppy was not to be found. We were able to identify the contents of every floppy we tried, but the game wasn’t there. As they closed up the coffee shop there were still a handful of floppies remaining. @bzotto graciously offered to check them out at home, but no luck. In high school I had many printouts of my code, but we never saved them. It’s also disappointing that I never made a copy, but this is not how young people think. I’ll have to be satisfied with just the memory of what I put together.

What was the name of this game? In case anyone’s curious, I never got that far. If I remember correctly, the name of the main file was just “game”.

What next?

Nevertheless, I now have a functional Apple ][+ computer sitting on my desk. I don’t have much time to devote to tinkering with an ancient piece of hardware, but it would be cool to see if I could run some modern machine learning algorithms on it. I’ll be giving that some thought and will see what I can do…

Poof

I chronicled my odyssey on Twitter, but I’ll share the highlights here.

After recovering the machine from my parent’s house, where it spent decades in the closet and then the garage, I brought it back to San Francisco. I purchased a brand new set of Apple DOS 3.3 floppies, because I was skeptical that the old ones would still work, and was ready to see if it would boot. It turned on and did… something… but it did not look like it was working.

After playing around with it a bit (swapping the floppy drives), I was able to get the BASIC prompt. I entered and ran a 1-line “HELLO WORLD”, which worked. Then I smelled something awful and saw smoke coming out of the PSU.

Not having been powered on for decades, turns out a capacitor in the PSU dried out and fried.

New capacitors

This is, apparently, a common occurrence with older power supplies. @bzotto, which whom I was chatting on Twitter, quickly identified the likely culprit (the 0.1 µf RIFA capacitor) and suggested a replacement kit. Amazingly, there is an online store that sells a bag a replacement capacitors for the Apple ][+ PSU. According to folks on Applefritter (a forum for vintage Apple computer hobbyists), this sort of thing happens a lot.

With a little advice from the friendly people at Console5.com, who sold me the replacement capacitors, and a ton of help from my best friend, who lent me the work desk in his garage, we rebuilt the PSU. My initial thought was to only replace the RIFA, but I figured, while we’re in there we might as well just change them all.

It was painstaking work that took many hours, and a steady hand, but in the end it was ready to go: the PSU, that is. The rest of the machine was still having a problem. When I booted the machine I got all kinds of strange things on the screen. Most frequently, however, were random green squares.

New RAM chips

The new suspect was the RAM. Another helpful person on Applefritter gave me some code to test the RAM, but after carefully typing it in I wasn’t able to get it to work. Someone else suggested that I buy some contact cleaner and reseat all of the RAM chips, so I gave that a go. I bought the cleaner along with some chip removers and a multimeter. The good news is that every chip on the motherboard is in a socket, as opposed to being soldered to the PCB.

The process was slow and painstaking, but I eventually went through almost every chip on the motherboard. (There were a few under the keyboard that I wasn’t able to reach without removing the motherboard.) The result, however, was disappointing; green boxes were still there. I read on forum that all the rows of chips were not required for the machine to operate, so I took some advice and tried with just a single row of RAM. The machine looked go, so after adding back the floppy drive I was able to get it to boot!

This exercise clearly indicated that the problem was somewhere with the RAM. Taking a closer look at the pins, many of them were obviously corroded. So, I proceed to remove them, again, and gently file away the corrosion with an emery board. This worked well…

… right up until it didn’t.

I tried to repair the broken pins with a soldering iron, but the work was too difficult. Eventually I decided that the hassle here was too great. Following more advice from the forum, I went online and purchased a full set of 24 replacement TMS4116-20NL RAM chips for $100, including shipping. They arrived in no time, with shiny pins, and worked like a charm. The machine booted immediately with no problems and I was able to run software off the new floppies I had purchased.

We have liftoff!

In the meantime, with the help of my new Twitter friend, I tried to recover the old game from high school. That did not go as well.

Everyone is aware of the impressive achievements of modern artificial intelligence and, more specifically, machine learning. Achievements.ai chronicles a growing list of AI milestones, but today virtually every industry has benefited from AI in one way or another. From medicine to entertainment to information retrieval to scientific discovery to agriculture, machine learning algorithms, powered by massive amount of data and computational power, have delivered unprecedented gains in efficiency and user value.

Chip shortages, however, exacerbated by the pandemic, trade, weather, fires and wars, have severely crimped our ability to deploy the compute necessary to train increasingly complex models. In times of scarcity we need to make the most of whatever resources are at our disposal. This requires creativity and gumption. Wired even recently reported on how some companies are hacking their way around the shortages:

Carmakers are using semiconductors taken from washing machines, rewriting code to use less silicon, and even shipping their products without some chips while promising to add them in later. With the shortage of semiconductors now a new normal, everyone is being forced to adapt.

… one large industrial conglomerate had resorted to buying washing machines just to scavenge the chips inside them for its products.

As such, I am going to see what sort of machine learning I can do on my Apple ][+ computer from high school.

The specifications of the Apple ][+ are somewhat daunting:

• Processor: 6502 microprocessor running at 1.023 MHz
• RAM: 48Kb
• Storage: dual external 5 1⁄4-inch floppy drives, each capable of storing 140Kb per side with DOS 3.3
• Programming languages: Applesoft BASIC and 6502 assembly

Nevertheless, these machines, which launched the personal computing revolution, can add and multiply arrays of numbers. As such, they can do machine learning. It is our duty, for the betterment of humanity, to make our best effort to deploy these machines in the service of artificial intelligence.

Let’s see what I can do…