#12 Software Engineering Cycle

somehow i've come full circle back to teaching about software, so here's the latest lesson: i don't know if this is actually how things are done in companies, but it's something i've found to be very useful in developing software of my own. so what exactly is software? it's a solution to some kind of digital problem. now, there's two kinds of problems, ones that you can immediately think a solution for due to your experience, and the other where it's beyond your experience, it's something new something you've never seen before. btw i've found it to be so sexy that you can basically let go of capitalization but still use punctuation and it automatically sounds more human and zero AI slop like. anyway, as i was saying, there's problems you just know are solve-able. like reading a gmail from code, or sending some... idk api call to AI then using it's output to decide on something. the kind of shit that's like known known. but then there's what YOU don't know and can't get that from AI because AI doesn't know for sure. like... i had been building a WhatsApp automation for a client, and i didn't know shit where to even start. AI tells that it's POSSIBLE but like... how? what? where? so the first thing to be done is not to build any software or anything, it's "proof of concept". what i call "labs". i always have a labs folder in my repository. it contains all the stray python files that contain just one thing "this particular method of solving a problem works in this limited scope that it's being tested in in this script". simply put, instead of tackling the BIG WhatsApp problem, which had tons of smaller unknowns like, what number might be used? what if that number already has a whatsapp how'd that work, what would be the pricing, would i even GET to connect some kind of LLM to the messages, how would i get through compliance, how much time it will take, and so on. instead of so many problems, what's the FIRST immediate thing I can test? it's can I sign up on developers facebook platform and create an APP. yes? okay, what's next? can I create a "TEST" number to check if the python program is able to send any kind of message? Yes and no. I GOT a test number, but here's the "findings" from the lab report -> i can't send ANY kind of message, it has to be a pre-approved TEMPLATE that can contain variables that come from code. OKAY, we're getting somewhere. so next step? test if I can do it from python -> open claude code, explain tf I'm tryna do, and bam python script, I run it, OMG I GOT A WHATSAPP MESSAGE TO MY NUMBER! progress. what's next? NOW that we have PROOF OF CONCEPT, that YES, PYTHON -> WHATSAPP is WORKING (i gotta stress the words so I'm capitalizing) we now shift to building the harness. what's a harness? it's the surrounding code that holds the main thing. sure python -> whatsapp works, but my client isn't gonna sit and run python scripts! so, desktop application -> exe that runs 24/7 as long as the computer is running -> bla bla bla. but wait, we can't deploy a TEST number in prod? so THIS is now done parallelly to the harness production. claude code is building the harness, and I'm looking at the facebook's developer page, and ok got it so I have to add a card payment, I have to buy a sim card that doesn't have whatsapp downloaded on it, I have to verify otp stuff, then submit the templates for what messages are gonna be sent, and so on. So there's LABS where a proof of concept, the main bottleneck of the entire problem, the main thing that we gotta prove after which everything else is just a "gotta code it up" problem. Another example: my dad sends me bank account statements that he wants converted to XML in order to import into tally. what's the bottleneck? PDF can't be reliably read. even if TEXT can be extracted from it, it loses it's structure, so PDF -> CSV is HARD. the rest is easy coz after I have CSV, i can deterministically do the CSV to XML using just one python script. So there's the clear elephant in the room. How to address it? Well... solutions come from random ass places so you have to keep your eyes open. YouTube -> got recommended a video about Jev (the latest non-autoregressive AI that's a systems 1 thinker). we just do ONE THING. pdf -> text right... then there's one UNIQUE ID in the row which is something TALLY wants. as in... Tally is a book keeping software, we tell it "yo i wanna add a new voucher for user id X, amt is Y Z whatever" that X is something we use as pivot. We identify all Xs... (hahaha exes don't need identification) all Xs and then we do this: ask Jev "here's the pdf's screenshot, or whatever multi-modal input it has" and here's the thing we're matching: we say, given the following numbers around X, which one's are X's? like... dude it's so hard to convey this through language. imagine the bank statement: columns are date name credit debit balance so name is our X now imagine, that a pdf contains all these rows, but when we convert to TEXT we get them like date1 name1 credit1 debit1 balance1 date2 name2 credit2 debit2 balance2 and so on. now what could happen is this "flattening" of the table can go wrong due to some artifacts in the pdf like a logo here, a url there, whatever. then that fcks up our order so we can't RELIABLY parse this. that's where jev comes in, we say "yo for this name1, our of these values: date1 credit1 debit1 balance1 date2 credit2 debit2 balance2, which ones are name1's" and it gives a probability distribution, and we take the max of that. we repeat over all names. and we're GOODDDD. so this is my "idea". when im implementing this in software i will do labs -> iterate over this idea, see if there's anything better and makes more sense than whatever im suggesting -> improve -> gather results from experimentation -> "ok THIS is the best way to parse a pdf into csv, there's no other better way" -> bam i got the proof of concept and ill put that into the context of the software inside which i wanna use this now. yay? hopefully yay. aight, lesson over. use labs to do experiments, then implement in real software. trial and error in the main repo is BAD, hurts the token usage n AI. prone to hallucination. all focus on "proof of concept" and experiment = good results. if focus splits between integrating into software + making it work, then = bad results. enjoy.