tech blog

October 27

My rant about interview questions and such was posted at roughly 3am. My first version of the IQ test note was posted at roughly 5am. I'm adding to the IQ test after 06:16am. Then more later, which I post higher up the page.

October 12, 2024

September 24, 2024

Sept 20

September 13

September 11

August 24

August 10

August 7

First posted about 2:12am and then another note at 2:15+. Then another at 2:18am.

August 4

July 31

The account manager entry for my Lifelog friend is yesterday's.

July 30, 2024

July 28, 2024

July 20

Null change. I accidentally changed this file rather than my personal blog. As of 21:30.

July 18

New entries going up through 18:18 and beyond, in no order.

July 17

cat * | grep -oP "abcd\:[a-f0-9:]+" | sort -u

July 16

July 15

First post after 02:26. I'm going to keep whittling at this at 03:02 and beyond.

July 10

July 9, 2024

mongosh  --quiet   /tmp/kwqeq10_2021_9ddeee941d9265e105411ffd55d645b2_k.js

print(
	EJSON.stringify(
		db.getSiblingDB('mydbName').getCollection('mycollection').find().toArray()
	)
)

print(EJSON.stringify(db.getSiblingDB('test').getCollection('my_arbitrary_collection_name').find().toArray()))   

[
    {
        "_id": {
            "$oid": "668cec9babdd6dc6cc59daf6"
        },
        "someData1": 1,
        "moreData2": "2 blah"
    },
    {
        "_id": {
            "$oid": "668ceccbabdd6dc6cc59daf7"
        },
        "otherData": "blah blah blah",
        "foo": "bar"
    }
]

June 21 - understanding binary

I have a new apprentice. I have hope that we'll work together for months and possibly years to come. She's already built up 50 karma points in case we start trying to use Reddit to get gigs.

She's taking online classes at one of those private online schools for a BS in CS. There are a handful of such schools in that category I've heard of over the years. I won't name the school because I might wind up mocking them. They go in the category of institution that I wonder if it will do her any good. I tend to think college is generally not worth it even for CS, but that's not my topic right now.

Right now she's taking a Linux sysadmin class, which in theory is great. In practice, there are a bunch of things that annoy me, which is also not my topic now. In any event, she was learning chmod. I was trying to explain the permission bits, such as chmod 755 === rwxr-xr-x .

She quickly mentioned that she does not understand binary at all. I tried to explain that decimal 567 === 5 X 10^2 + 6 X 10^1 + 7 X 10^0, and thus try to explain that binary works the same way. So far, I'm not getting anywhere, but we've only spent a few minutes on it so far. She asked me, "If I don't understand this, am I not cut out to be a programmer?"

I've been pondering that question. I eventually answer it below. For now, though, rather than answer it directly, I'll say this. Forget binary for the moment. It would be a very good idea to start to think about the decimal system that we take for granted, and what it means. As above, what does 567 really mean? Base 10 (decimal) only has the digits 0 - 9. Thus, 567 or 56 or 10 is a different beast. The number "10" is an abstraction in a way that "9" is not. "9" represents something that you can easily visualize--9 objects. One can easily visualize 10 objects, but 10 is a 1 and a 0. It's an abstract usage of 1 and 0. For that matter, 4,598,839 cannot be visualized, but it can be manipulated because the number means something very specific, and their are algorithms to process the number. I mentioned that Roman numerals are very crude and cumbersome because they are not a base system. They do not have a zero. Zero is very important.

It would be great to understand bases, the importance of zero, what 567 or 5678 means, etc. It goes to abstraction and business analysis. That is, when figuring out how to code something, you have to really understand how it works.

I will encourage her to first analyze decimal and really understand what's going on. The way we do addition by hand, for example, is an algorithm. There are reasons why carrying digits over to the next decimal place works. In other words, why does stuff she learned in 4th - 8th grade work? What is really going on? (I think I understood addition when I was a child, but it took me some time longer to think through why long division works. That is, I didn't fully analyze the long division algorithm for a while.)

Computers add and divide in similar ways to how we do it by hand. Again, it's an algorithm.

Once she understands decimal consciously, then binary should be relatively easy.

I don't care if it takes her months. Her circumstances are such that getting a clear head for any length of time is problematic, so it might take months.

If we come at it from different directions over the course of months, and she's still not getting anywhere in her understanding, then the answer may be that we have to start questioning whether she's cut out for programming. As best I understand, some programmers go through their whole careers writing from specifications. I frown heavily upon such things. A programmer should be able to do business analysis / requirements gathering. I should add that I've almost never written from specifications, even at medium-sized software companies (for various reasons).

Hopefully that will do for a start.

June 10

this.bounds = new google.maps.LatLngBounds();
this.bounds.extend(position [object, with lat lng]); // iterated for every location
this.map.fitBounds(this.bounds);

# in another conf file in the real implementation
# live versus dev name
<IfFile			 /var/kwynn/i_am_Kwynn_local_dev_2024_01> 
    Define Gsn dev.example.com
# Define Gsn blah
</IfFile>
<IfFile			!'/var/kwynn/i_am_Kwynn_local_dev_2024_01'> 
    Define Gsn live.example.com
</IfFile>


# ServerName is in a 2nd other file in the real implementation
ServerName ${Gsn}

ProxyPass	 /nodejs http://${Gsn}:3000/
ProxyPassReverse /nodejs http://${Gsn}:3000/

ProxyPreserveHost On
ProxyPassMatch ^/(_next/webpack-hmr)$   ws://localhost:3000/$1
ProxyPassMatch ^/(_next/static/.*)$ http://localhost:3000/$1

/* receive from Node into the parent HTML window with the Google Map */ 
function rcvCustSelectionFromNodeJS(customers) { 
    if (GLIgnoreSelRebound) { 
	GLIgnoreSelRebound = false;
	return;
    }
    /* the customers are shown on the map */ 
}

/* send to node */
var GLIgnoreSelRebound;

function clearCustomersFromNodeSelection() {
    const e = document.getElementById('iframeNode');

	    if (e && e.NodeclearCustomers) {
		     e.NodeclearCustomers();	    

    		GLIgnoreSelRebound = true;
	    }
}

/* the Node side */
// top of file - send selection to Goo Map in parent HTML window
'use client'

import React, {useState, useEffect} from 'react'
export default function setCustomerSelection({ filteredCustomers ...

useEffect(() => {
        if (window.parent.rcvCustSelectionFromNodeJS)
            window.parent.rcvCustSelectionFromNodeJS(filteredCustomers);
    }, [filteredCustomers])
}

// another node file that clears the selection

'use client'

import React, {useState, useEffect} from 'react'
import Checkbox from './Checkbox'

export default function FilterCustomers({ ...

    const clearCustomers = () => {
        setCustomers([]); // original Node
    }
    
    // the window HTML / JS DOM object may not be defined yet, so check
    if (typeof window === 'object' && !window.NodeclearCustomers)  // make available to parent 
                                       window.NodeclearCustomers = clearCustomers;

June 6

June 3, 2024

Looks like at least 3 entries today, and the day is still young from my POV. The first entries were around 5:30am. The 3rd is after 19:21.

May 21, 2024

A few months ago a potential apprentice was looking at Craig's List ads. We only spent a few minutes at it, so I never quite explained my objection to his methodology.

I need help narrowing down to specific ads. He was creating yet more lists of ads, which is what Craig's List already is. He was sending me the results of keyword searches. The results were lists of ads. When he did send me specific ads, he was getting the idea.

There either zero keywords or hundreds of keywords that will help me, where 0 and hundreds amounts to the same problem: a keyword search won't work. In his case, I would have to go through his list and then I would still have to go through the individual cities to find what he missed. He was adding to my workload, not reducing it.

We probably would have gotten that sorted, but his "self-discipline" comment did not go well with me.

May 7

April 28

April 23

April 16

April 14

Looks like at least 2 entries today. The Oracle one is the earlier one and further below.

April 2

First posted 01:50.

I'm not feeling so enthused about updates, but I suppose I do have a very small audience, and there are updates.

I finally upgraded kwynn.com to Ubuntu 23.10. One lesson is that one should beware of setting the TTL of AAAA records too high, at least on AWS EC2. I had them set at 24 hours. I'm trying to be nice to both my registrar and the DNSystem as a whole. AWS EC2 is set nicely to change instances of IPv4 addresses--that's the Elastic IPs. You can set an Elastic IP to any instance you want. You can reassign v6 addresses, too, but, as far as I know, you have to explicitly release them from the old server (instance). It would have been easier to just change it, but instead I took my site down for longer than I wanted to reassign the v6 address.

On a related point, be careful of having more than one subnet (unless you have a reason, of course). I had a second one for no good reason, and I wasn't giving thought to which one to select. That's why I couldn't switch IPv6 easily--I had the wrong subnet. Thus, I deleted the second one.

I think it was 23.10 that upgraded to PHP 8.2. (I upgraded my local computer months before kwynn.com.) In any event, I got hammered by "Deprecated function: Creation of dynamic property className::$varName is deprecated..." I wasn't prepared for that sort of workload when I upgraded Ubuntu. I would assume there is still stuff broken on my site. I went over the stuff I use; that's the best I can / will do for now.

Years ago I heard that AWS us-east-1's original AZs were becoming old and decrepit. I've been on -1a for a while, and, with one possible exception, it works fine. It seems too often that the network connection gets hung up for minutes upon (re)boot. I don't have enough data to correlate that to an AZ, though. It's just a thought.

So, when I create a new instance, a new AZ is one consideration. Then there is the warning flag, "EC2 recommends setting IMDSv2 to required." I wondered what the heck that was. It's the http://169.254.169.254/ service within an instance--Instance MetaData Service. It lets programs figure out which AZ they are in, and what type of instance it is, and such. Around 24 hours ago I upgraded my metric tracking code to IMDSv2. I'd have to move instances, though, to turn v2-only on. At least, I'm pretty sure I have to move instances. Actually, no; one can do it on the fly, but my first glance shows it's too complicated to bother with.

AWS' discussion of v2 is interesting. They set the TTL of the result to 1 hop, so that they data can't leave the instance. Also, they observed that most proxies and firewalls and such don't forward HTTP PUT, so they use that.

In somewhat unrelated news, I got Discourse running for a client (a few days ago) on EC2. That went smoothly.

In unrelated news, I notice that the latest HTML validator complains "Trailing slash on void elements has no effect and interacts badly with unquoted attribute values." I suppose I should deal with that. I may do so now, in fact.

March 13

False alarm / false update.

March 11 - thoughts on cryptocurrencies

This begins the original post:

I've been asked about crypto at least twice in the last few weeks, so I'll try to answer usefully.

First, as an abstract concept, independent of specific tokens / coins, crypto has the potential to be a true revolution along the lines of the industrial revolution. Crypto has proven that a large number of people will accept a currency(ish) outside of governments, banks, or other "official" institutions. I don't quite consider crypto to be true currencies, because one definition of a proper currency is a unit that maintains stable purchasing power for years, decades, or centuries. The cryptos spectacularly fail at stability.

There were supposed to be "stable" tokens, but my thumbnail understanding is that a handful of those totally failed recently--they became so unstable that they went to permanent zero. Also, at least one was based on the Federal Reserve Note (FRN - fake "dollars"), which is quite a joke.

To back up and emphasize that point: the green bills and stuff in bank accounts are not legal dollars. They are falsely denominated in dollars. The US Mint still issues a true one dollar, one silver (Troy) ounce, 2024-stamped coin. I link to the proof--reflective, polished, for collectors--version. They also mint many more non-proof coins that are theoretically meant for general circulation. You generally can't buy those directly from the Mint because they only sell in bulk. Last I checked (many years ago), you could buy those coins for 1 - 5 Federal Reserve Notes (FRN) above the spot price of silver. That is, right now a true dollar will cost you perhaps 28 FRN with a spot price right now of 24.73 FRN / Troy ounce.

The Mint also issues 1 (Troy) ounce gold coins that are stamped 50 dollars. Hopefully they didn't discontinue the American Eagles in 2024. I can find a 2023 gold coin, though. Interestingly, that one is uncirculated rather than proof. I didn't know they sold those directly, although I haven't looked in years or even decades. In any event, I'm sure I could get a much better price, but the spot of gold is 2,190 FRN right now. I would likely have to pay 2300 FRN for a 50 dollar coin, so that's a 46:1 ratio rather than > 25:1 for silver.

So, how do the banks get away with denominating their account money in dollars? I damn sure can't get gold or silver at the legal ratio. That's a rhetorical question, of course. The banks are at the core of the Satanists / J--ish "elite" / Illuminati system.

Worse yet, 99.99999% of all fake "dollars" are literally created by a bank when a debt is incurred. Every (99.999999%) unit of the so-called national currency is lent at interest, which means the whole "national" currency is borrowed, which means that the currency can never be repaid because more debt exists than units to pay it. The 2018 Swiss sovereign-money initiative (Vollgeld) tried to address this issue, shockingly. Meanwhile, Quiet Weapons for Silent Wars blithely admits this is "slavery and genocide" (page 4), "counterfeiting" (14), and "... presented as ... 'currency,' [that] has the appearance of capital, but is in effect negative capital. Hence, it has the appearance of service, but is in fact, indebtedness or debt" (10).

So, on one hand, cryptos are a true representation of value as opposed to Rothschild's debt-slave tokens. Cryptos are created by intense computation (mining, like Bitcoin -- proof of work) or lower-power computing backed by proof of stake (like Cardano Ada, at least in the summer of 2021). Similarly, it's much harder to turn someone's account off. My understanding is that during the Canadian Trucker's Convoy (early 2022), the Satanic occupying power ("government") targeted Coinbase and other large exchanges. That only works if one kept one's wallet with an exchange. A crypto holder is free to keep their own security tokens / wallet.

I should also address the notion that crypto is used for evil purposes. What a joke. The banks have been deeply involved in the "illegal" drug trade since banks existed. The same people who own banks (Rothschilds, etc.) run the drug trade, sex trafficking trade, etc. They are the ones who decide what's illegal. Cocaine and heroin were over the counter products in America in 1900. Duh. JP Morgan has been successfully sued by Jeffrey Epstein victims in the last few months, recently for 290M FRN.

Bayer marketed diacetylmorphine as an over-the-counter drug under the trademark name Heroin.... From 1898 through to 1910, diamorphine was marketed under the trademark name Heroin as a non-addictive morphine substitute and cough suppressant.... In the US, the Harrison Narcotics Tax Act was passed in 1914 to control the sale and distribution of diacetylmorphine and other opioids...
-- Heroin, WikiP, unchanged as of Mon, 11 Mar 2024 23:32:35 GMT, captured around 21:15 EDT.

By 1885 the U.S. manufacturer Parke-Davis sold coca-leaf cigarettes and cheroots, a cocaine inhalant, a Coca Cordial, cocaine crystals, and cocaine solution for intravenous injection.
-- similar to above, apparently unchanged since Mon, 11 Mar 2024 13:46:30 GMT, captured ~3 minutes after the above

Ronald Bernard talked about banks. At a quick glance, he's talking about them at 17 minutes. At one point in that interview, he talks about how his mission was to launder tractor-trailer loads of FRN green bills in the basement of a large European bank. That was for Iraqi oil during sanctions, but the same applies to actual crimes, as opposed to the tyranny of the decade that the US Inc. makes up.

On a similar note, I spent perhaps 10 minutes browsing AlphaBay, a dark web site. Every "illegal" drug under the sun was available. I just laughed and laughed. Why I thought it was funny is another topic.

Wall Street Journal: "Instagram Connects Vast Pedophile Network" by Jeff Horwitz and Katherine Blunt. June 7, 2023 7:05 am ET. Another example of large companies being far worse than anything crypto is accused of. I haven't read that specific article. I assume it's a limited hangout, and the truth is far worse.

March 8 (started 4th)

This is more on moving computers.

• install the HTML validator - first sudo apt install default-jdk then, after after cloning, python ./checker.py all
• The way I have it set up, the validator needs the proxy and proxy_http apache mods
• Here is how I set up the validator in Apache config to run at /htval
• the validator involves systemctl if you want it to start at reboot. That URL might change, so here is a specific version of nu / validator systemctl.
• replace "nobody" user with DynamicUser=yes # done in repo above
• I have been and will be adding to my sysadmin examples repo
• I will have to activate the Apache rewrite engine: sudo a2enmod rewrite ; sudo systemctl restart apache2 -- note restart not reload
• sudo systemctl status apache2 | fold -- the fold command wraps lines readably
• sudo a2enmod ssl and restart
• put locally run, SSL-enabled domain name in /etc/hosts : add the line : 127.0.0.1 example.com
• set permissions of Apache DOCUMENT_ROOT aptly
• it would appear I have symbolic links in kwynn dot com /t/23/02. That may not be the best way to do it, but I will probably stick with it for now. The specific links are below. Need to change underlying permissions, too.
  1. homecms points to my CMS project / home (explicit /home)
  2. htva points to cms/htva
  3. webhook_git points to my GitHub webhook project -- I rather doubt this still works, but that's another issue
• That in turn leads to issues with isKwGoo and the positive email [Big Evil Goo Mail] check. See "notes" below.
• looks like kwynn dot com /index.php and the cms index.php should be a hard link
• need to install GMail / Goo composer. I keep these in /opt/composer as is shown in kwutils (I refer to the whole repo as kwutils because of the one file with that name):
  composer require google/apiclient
  composer require google/apiclient-services
  composer require google/auth
  
• positive email check read-only secret file - set permission - That's the Goo Cloud Console API key stuff.
• isKwGoo email hash file
• looks like I have yet more symlinks in my website to resolve, or just use the actual repo:
  • PHP date format repo is in /t/20/08/dates/
  • To do it backwards, one of my user agent parsers is in this web server access log repo directory
  • textIO example from my "permanent examples" repo.
  • /t/22/03 was probably the start of a new reference to /t/22/02 sessions, but I've had enough for now

February 19 (not posted until probably the 29th)

Some thoughts on moving computers:

• GitHub keys
• get Apache config in repo
• create necessary (perhaps obfuscated) paths / sym links - such as /var/kwynn/...
• install chrony - After being asleep for something like 9 days, the new computer is 20 SECONDS fast. That won't do.
• put my chrony.conf in public repo and swap it for the default
• move selected passwords - GitHub, client stuff, etc.

Feb 5

January 15

January 2

I just posted the previous entry that I wrote on December 26 but didn't post until now. I had to install a couple of plugins and restore the theme, and then I did get paid. That client should have more work for me soon, too, but I plan to spend more time on non-technical jobs for at least a handful of months. Probably more to come on that in my personal blog soon.

I'm writing and hopefully posting now because I finally took the moment needed to track down the "general store" joke. "I went to a general store but they wouldn't let me buy anything specific." That's American comedian Steven Wright, born December 6, 1955. So the joke can't be as old as I thought it was. It probably still is decades old, but I thought it was from the 1920s - 50s. I misquote it at least once below. I had the idea perfectly right, but not the exact wording.

December 26

November 28

August 29

August 21

August 15

August 12

I just got paid $50 as agreed. It's definitely something. I installed a website from a code file and SQL dump through cPanel. There will probably be at least a bit more work, and perhaps much more work eventually. We'll see. Overall, sales are going very badly, though.

July 18

July 14

It gets off topic from tech, but perhaps today the French began a useful revolution, rather than one motivated by the usual evil (Satanic) suspects. I haven't followed my "news" sources for the last few days. I know there was rioting, but I don't know if anything special happened on Bastille Day.

Now on to tech.

June 22

This is one of my checklist entries. My changes to /opt/kwynn have been growing. /opt/kwynn is also known as "php general utilities." That is, I have been changing /opt/kwynn for a paid project and not merging the changes back to kwynn.com. So, changes:

• I think just the testing timestamp on kwynn email. Should not be important. Just make sure I get an email in the next few days. "Kwynn email" is the object I use to email myself sysadmin messages such as Ubuntu needing updating or my time server not being able to get enough NIST polls.
• isKwGoo change - This protects admin parts of my site. Lots of ways to test it.
• inht() change - I'll have to think about that. It's unlikely to break anything because it's likely been tested locally.
• kwifs() - same
• dr() - document root. Hmmm...
• lots of testing needed for cron and web SNTP.

June 16

I got a complaint that my previous entry was not clear enough, so I will try to clarify. With the exception of a bit of SQL to clean up a small mess on May 31, I didn't write a single line of code from May 7 until today. With those exceptions, I have not been coding at all. I can't justify unpaid coding right now.

In the last few hours I have done a bit of simple coding to rearrange which NIST servers I'm using to track my server's time. The record is in GitHub.

June 14, 2023

June 3

I saw an ad seeking computer services. The ad specified a certain religion. I have no problem with that, especially given the context. The potential client thought he was going to get Ubuntu Linux. That decision was made before I got involved, so it seemed a good start. Then he writes back to say that it turns out someone installed SatanSoft on his server.

I got stuck on his religious question because it's not precise enough. The religion, like almost all religions, can be interpreted too many ways. I got a long ways into an essay trying to explain my belief to find out whether it was close enough to his. Then I find out about SatanSoft. What good is his religious filter if that's the result? I asked him if it was an approved member of his religion who installed SatanSoft.

This is another demonstration of why I should not be freelancing, or that I at least need an intermediary in many cases.

May 26

I mentioned UTC to one of my former apprentices. He said he uses it because he works with servers in different timezones. This is my reply.

I've very rarely worked in different timezones. I'm often dealing with the UNIX Epoch timestamp, though, which is in UTC. As best I've ever been able to tell, MongoDB doesn't have a date data type, so I use the UNIX Epoch. Now that I'm writing this publicly, I should add that the Epoch has served my purposes just fine. Occasionally I look again for a MongoDB date type, and I'm still reasonably sure there isn't one.

Actually, I use the Epoch as the true / primarily / always-used field and then I generally have a denormalized / human-readable / redundant secondary field so that I can keep track of what's going on in the database more easily. I usually have a human-readable date as part of the _id / required primary key, too. The date may change, and one is not allowed to change _id, only delete and insert a new row, but that's fine. It still makes it easier to keep track of what's going on. Again, I use the Epoch as the true field on the matter, and everything else is an aid for me when looking visually and not used in calculations.

Also, I have my small suite of time services, and I use the Epoch with those, too.  JavaScript does the Epoch in milliseconds, and I wrote my PHP extension to do it in nanoseconds.  I've seen various Epoch timestamps literally 10,000s of times, I would guess, although it's not like I'm paying close attention usually.  I've at least glanced at timestamps many, many times.

A few years ago I got bent out of shape dealing with the NTP timestamp format which was formalized in 1985. At that point, 32 bit PCs were a few years away, let alone 64 bit. That format is 2 X 32 bit fields. "NTP timestamps are represented as a 64-bit fixed-point number, in seconds relative to 0000 UT on 1 January 1900. The integer part is in the first 32 bits and the fraction part in the last 32 bits..." I've spent a bit of time in at least two languages and a few rewrites turning that into the UNIX Epoch.

One interesting note is that the NTP format will roll over in late February or early March, 2036, if my rough calculations are correct. Perhaps I'll shepherd the new NTP format and the transition code for that. If there are any signed 32 bit implementations left in January, 2038, the UNIX Epoch will roll over. A reading of that article shows that it's a tricky problem. I may be dealing with it, too.

I am well aware of the difference between hard technical problems and seemingly fantastic tales, but John Titor either implied or stated that one goal of his time travel missions was to bring back (forward) 1970s computers to help them deal with the 2038 problem. I'm not sure that it was of vital importance, just helpful. His timeline had a degree of civilization breakdown, so perhaps it makes more sense. I recently mentioned Titor in my personal blog.

May 14

April 29

A new Ubuntu upgrade checklist:

100. After the Ubuntu upgrade, upgrade composer for MongoDB. On my local machine, I saw a subclass definition conflict when I upgraded from Ubuntu 22.10 to 23.04.
200. Test web message form and email at the same time.
300. clock and stats - looks like /var/log/chrony may have a new permission issue
400. quick email

April 28

I have likely found an investor for projects. There is no need to go into details as to what we're working on. For one, we're not entirely sure yet. This will not be a full-time job, I will keep looking for "real" but night jobs, and I'll almost certainly still be available for at least small projects. Or I'll be available for larger projects if you can outbid the investor. That may not be hard, as I'll explain. Meanwhile, there was no feedback on the IQ test today, so the business week has passed on that one.

My investor wants a "dollar" (Rothschild debt note / Federal Reserve Note) number of what I need to stay afloat. I told him that he could get a lot of work out of me just by accomplishing "flotation." That's one of several reasons I get a bit irritated with the question "What's your rate?" Expenses are by the day and month, not by the hour. I have always been willing to do quite a bit of work for a steady monthly fee or a barter, even if it's 1.2 orders of magnitude lower than dev pay. I don't want to do that forever, but it would be a big improvement over what I've been doing, or failing to do.

I've never calculated that exact number. I just did. I'm not going to give the number, but it's depressing. Real dev pay would solve my problems several times over; it's not like I spend that much. Freelancing has just been that disastrous from a sales point of view. A summary of how I got into this mess is in order, or at least some highlights / lowlights.

I'll somewhat arbitrarily start in 2020. I want Billuminati Gates and at least a handful of others tried and executed just for one specific series of events they urged the shod, masked zombies to destroy in 2020. I have probably not grasped how bad that was for me. It is my equivalent of an annual religious renewal of faith. It was totally destroyed in 2020, and effectively destroyed in 2021. In 2022, kids who were 22 years old and in perfect health were wearing masks. I wouldn't have tolerated that. Worse yet, there may be "vax" requirements involved, in which case I would probably be in some people's faces for their gross idiocy. One way or another, I might manage to get myself arrested arguing with others involved. Their official policy needs to be for Gates and company to be tried and executed, or I can't stand to be around them until his execution is made public.

The point being my faith has not been renewed since 2019, and now I'm getting increasingly angry at my peers in the matter, but I don't even see them as peers but shod, masked, vaxxed zombies. (They weren't vaxxed in 2020, of course, at least not with the latest deadly cocktail.)

I think I've mentioned in this blog the one project disaster in the summer of 2020, then 3 disasters in the summer of 2021. I got a couple of small projects last year. Also last year I tried as hard as I've ever tried to get freelancing projects. I pushed hard, relative to my limited capacity on the matter, a number of times. In November I didn't refresh my paste buffer and sent the wrong email to a potential client. The email was meant for someone else. I was deliberately sarcastic to the "someone else" because they deserved it. I figured there was some chance they'd see what devs were reading into their ad and change their tone. The actual recipient was understandably offended. That was the point I realized that I'd better stop the freelancing route. My sales skills really are that bad.

Overall, I just kept thinking that my sales couldn't go so badly for so long. If I just worked a little harder, I could make it work. In November I started look for "real" but night jobs. I've applied to 55 jobs since then. That resulted in 2 interviews and perhaps 1 - 2 other human responses. One interview series is still in progress, as I've previously described.

So, another point being that I didn't realize the real job path would also go this badly for this long. The ongoing interview series / candidate process may wildly succeed 3 weeks from now, but, until then, the hunt as a whole is a failure.

There is almost always more to say. Part of the point of this blog is to whine. That's enough whining for now, though. I need to get it through my head that my investor and I may accomplish great things in the next few months, or at least much better than what I've been accomplishing.

April 27

April 13 - status report

Looks like I never followed up on my January 20 entry. The result of that interview was "We don't think you're right for this job, but we like you as a candidate, so please find another one at our company to apply to." Grrrrr... In terms of potential earning power, I was generally way over-qualified for that job and specifically slightly under-qualified. So, they might actually mean it when they say apply to something else. The part that I'm growling about is that they should be the experts in what I should apply for. Why would I spend the time starting the whole process over again?

With that said, I finally found a company that agrees with that sentiment; I'll explain in a moment. I am now up to 52 job applications from November 1 to March 10. I got a second interview. I will not name the company for at least two reasons. In any event, this company very specifically says that may route me towards another job during the hiring process.

The interview was written with no time or space limit. I spent a long time and finally submitted it this morning. I don't expect to hear from them until Tuesday or Wednesday.

I also got a great response to my Gab ad for my web services. That's just a link to Gab itself; I don't want to ponder my ad right now. My ad is under "Marketplace." My potential client is already running into potential regulatory issues. I offered him another perspective on the matter, but his original idea is probably on indefinite pause. I think he'll be a great contact, though.

I also also (sic) encountered a handful of potentially useful recruiters, but no results yet. I am not quite sure where I'll look for work next. I hope to get back to relatively focused activity in the next few days.

In answer to one question from a potential apprentice, I am still living exactly where I have been for a few years, despite several periods of serious concern that I'd have to move.

March 4 - AWS EC2 upgrades

An instance I am responsible for is still on Ubuntu 18.04 LTS. Support expires in a few weeks on April 30. I know why I'm not on 22.04: sometime after 20.04, Ubuntu upgraded to PHP 8.x, and the software involved will not work in 8.x. I don't remember how long I tried to get it to work, but I concluded it was time to do something completely different. I'm in the process of that, but it will take a few more months. I'm being vague because there is no reason to paint a target on this system.

With that said, I am struggling to some degree to remember how I never upgraded to 20.04. Actually, I do know, but that's another story. In any event, I have been using test instances to make sure that the system will work in 20.04. The tests look good so far. 20.04 uses PHP 7.4.

{
"Version": "2012-10-17",
"Statement": [
		{
			"Sid": "VisualEditor0",
			"Effect": "Allow",
			"Action": "ec2:ModifyInstanceAttribute",
			"Resource": "arn:aws:ec2:*:12ishdAWSAcct:instance/*"
		}
	]
}

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
sudo apt update
sudo apt dist-upgrade

February 21

January 20

December 31 (posting 23:01)

I am using ancient hardware. I bought this computer (that I'm typing on) used for something like $160 in 2017, but it came out in roughly 2009. It may have been a $2,000 - $3,000 system then; I'm not going to bother to look that up.

In any event, this has caused me at least two problems, and I'm sure more will come. One problem I list way below regarding MongoDB 5.0 and a CPU instruction I'm missing.

I *think* I have now solved the other problem, or at least plenty well enough until I get new hardware. First of all, the Ubuntu package log can be accessed with "tail -n 500 /var/log/apt/term.log" When the grub-efi-amd64-signed package gets updated, I've been having problems. Specifically, "mount: /var/lib/grub/esp: wrong fs type, bad option, bad superblock on /dev/sda1, missing codepage or helper program, or other error."

I have found the hard way that you do NOT want to leave those package errors hanging for days. Eventually your system will break in ways that even I have given up trying to unscramble.

Today, after going around and around (for only 20 - 30 minutes or so), I found that the problem is in /etc/fstab . More specifically, as I've copied partitions when I get new drives, the old fstab is no longer accurate. (If you have an old UUID entry, you don't need a UUID. You can use the /dev/ name.)

The solution to all this involves having both an old fashioned "i386-pc" partition for grub and an unused (as far as I can tell) but properly set up efi / uefi / esp partition.

This part is unnecessary to fix the package install problem. I'm just listing for the record what you need for grub to work in this situation. In the "gparted" program (runs from the GUI / "sudo apt install gparted"), the "old fashioned" partition I have is 250MB (MB, not GB), although I think it can be much smaller. I give it the "bios_grub" "flag" - after creation, right click and "Manange Flags." As I remember, you don't have to give it a filesystem. After grub runs (see below), the filesystem type will be labeled in gparted as "grub2 core.img." I think that's just a label / name and not any sort of filesystem type. (I don't think the filesystem commands show any filesystem at all.) The "bios_grub" flag is what is needed for grub to find the partition and install itself. The partition should only have that flag set.

Today's solution was to create a partition for efi so that the above package install will shut up and work. You (may) need 2 packages installed to create an old-fashioned FAT32 filesystem. "sudo apt install dosfstools mtools" Before I didn't have mtools, and that resulted in a red exclamation point in gparted.

Once I had those 2 packages installed, I created another 250MB partition (9.1 "MiB" used right now) for efi. Format it to fat32. Give it only the "boot" and "esp" flags. Then make an entry (add a line) in fstab such as

/dev/sda6   /boot/efi  vfat   umask=0077      0       1 

Where /dev/sda6 is the efi partition I just created. (You may have to create /boot/efi .) Once your partition and /boot/efi exist, either manually mount or automount or reboot. Then either redo your install via something like "sudo apt dist-upgrade" or reinstall specifically with "sudo apt --reinstall install grub-efi-amd64-signed"

When I reinstalled, I still got warnings:

Setting up grub-efi-amd64-signed (1.186+2.06-2ubuntu13) ...
Installing grub to /boot/efi.
Installing for x86_64-efi platform.
grub-install: warning: EFI variables cannot be set on this system.
grub-install: warning: You will have to complete the GRUB setup manually.
Installation finished. No error reported.	

I don't care about the warnings. It ran without breaking my packages. I don't think there is any need to run grub commands or worry about further setup or EFI variables or anything else. I did run grub only as a check. Partial results are below, just for reference.

$  sudo grub-install /dev/sda ; sudo update-grub
Installing for i386-pc platform.
Installation finished. No error reported.
Sourcing file `/etc/default/grub'
[a bunch of lines where it finds various Linux installs]

Recap below. The first part isn't necessary to solve the package problem; it probably already is set up as such. I'm just listing what you need for both i386-pc and EFI. Remember that this is for a situation where you have very old hardware with no EFI capability.

1. An old bios partition of 250MB is probably way too much, but it works. It should have one flag "bios_grub". No need to worry about its filesystem format / type.
2. Make sure mtools and dosfstools are installed (Ubuntu packages).
3. Create a 250 MB (or perhaps only 25MB - see above) partition, formatted to fat32. Give it flags "boot" and "esp." This is the EFI partition.
4. Enter the EFI partition in /etc/fstab as shown above.
5. Manually mount, automount, or reboot. (If you manually mount, you can't be 100% sure your mount command is the same as the fstab entry. )
6. Reinstall the grub-efi-amd64-signed package as above.
7. Only if you are "starting over" without a pre-existing bios partition, install and update grub as above.

December 2 - continuing the job application

I just submitted the job application from my previous entry (Nov 22).

November 22 (posting 11/24 00:20 or later)

Note that I'm posting something new, further below, that I wrote on October 31 and didn't post until now.

I have given up on freelancing. There is a subset of sales skills that I don't have, or perhaps it's more accurate to say they were beaten out of me. I have been searching for nighttime tech jobs, and I'm finding a wide range. One of them has a prompt / question that I haven't seen exactly the likes of. I'm going to write / post my answer here. I won't give the exact prompt for a number of reasons, although you should be able to more or less figure out what it is. There is a character limit. I hope to stay well away from the limit, but we'll see what happens.

It appears I never linked this here, but a relevant document is "Why I'm a master software developer."

October 31 - November 22 - a system design sketch (micro-specification), a job application, and a rant, all in one

I wrote the following on Oct 31 and then got sidetracked. I'll skip the rant for now, although a rant is in order. I should post late on November 22.

I came across someone who has a plan for prosecuting the "Covid" criminals. It involves a request for a software specification, so I'll start with that.

The request is for a searchable database of evidence and software that can help organize the evidence. One question is whether we can assume that phase one will only deal with data that can be widely / publicly distributed. That is my vision for phase one. Otherwise put, I don't want the burden of trying to protect the identity of anonymous sources and confidential data, at least I don't want it to start. Someone else would have to take that thread, or we'd have to wait until I am ready to be paranoid enough, which may be a matter of a few weeks. I think a lot of this could be done in very few weeks, even if I'm working alone.

I am assuming that the database will eventually come under heavy attack, and we should assume the original store of the data will be destroyed-- thus the part about wide distribution of both the data and metadata (searchable index / various ways of looking at the data). A mechanism to distribute the data should be there from nearly the beginning.

The base product could be a web page with file upload capability, a text form (for several thousand characters at least), and smaller text fields for links. The most basic output can, for one, be a list of everything submitted with digital fingerprints (sha512 "hash") that can always verify the validity of the data. (Note that a URL by its nature is a fingerprint--unique.) Then the other part of the basic output can be a list of all the documents for download (and thus the documents themselves). A very early goal will be to encourage the relevant followers to download it early and often. A next step would be to automate distribution / backups / redundancy.

October 26 - first entry around 4:30am, 2nd - 3rd after 18:40

October 17 - updating various software - checklist

check web form / email
test that the new database email audit feature works
JS utils - stdio - not entirely sure what to check
get_oids()
*****
SNTP
make sure /var/kwynn has www permission along with all the files
make sure current version of wrapper, runner, and base
check cron
make sure FIFO mode
make sure no runaways
check quota DB records
perhaps put up sign that chm is likely to break

	

October 13 - apprentice timekeeping assignment

Short version: assignment 1: compare your computer's time to Time.gov and my own clock.

To give the context going back roughly 3 years, I asked myself the simple question of "What time is it?" By which I meant how does one obtain the time from accurate timeservers in a way that is "programmatically" useful. I'll come back to "programmatically" in a few moments. Time.gov is very useful to an end-user setting various (wall) clocks. Some of my frustrations with time.gov, though, led me to creating my own clock that I assert is better than time.gov assuming kwynn.com is accurate, which I'll get to in a moment. (Why it's better is a discussion for later.)

My clock web application is only as accurate as kwynn.com. How do I know kwynn.com is accurate? That has caused a storm of coding at various points, all of which is in my GitHub in several repos.

Part of the answer is to install an NTP client. (Look up NTP, but I'll get to the specifics in a moment.) Or at least that's an answer that doesn't involve purchasing some inexpensive hardware and doing some fiddling with it. I can't install hardware on kwynn.com, though. It lives a few hundred miles away. Also, NTP is plenty good enough for my purposes. "My purposes" are to be accurate to something like 0.4ms at worst. Right now, and usually, kwynn.com seems convincingly within 0.1ms.

So, for a first (sub)assignment, compare your computer's time to my clock and / or time.gov. I've rarely tested a desktop that's not well-synced. Right now, from my clock, this computer I'm typing on shows 1ms off. My clock application runs on JavaScript, which only keeps time to a precision of 1ms, so it's probably only 0.5 or 0.6ms off, and that's rounded to 1. Or maybe I used ceil() rather than round()?? For a comparison, my cell phone shows 170ms off, from my clock web app. Cell phones are synced from the tower to some degree, but not a high degree, or they would never be that far off. I'm not sure what to expect from an un-synced desktop. You'll tell me.

I never got to "programmatically" in this lesson. Perhaps next time.

October 3 - "What is your specialty?"

October 2

AWS fixed their clock by yesterday evening. My latest chrony.conf has this:

server 169.254.169.123		minpoll 2 maxpoll  9 iburst xleave
server 129.6.15.29			minpoll 4 maxpoll 10 xleave
server 2610:20:6f15:15::26	minpoll 6 maxpoll  9 xleave
server 2610:20:6f15:15::27	minpoll 5 maxpoll  9 xleave

I found that if the .29 NIST server had a minpoll of 2, it would "take over" and become the reference time. I want the NIST servers as a backup, now that I can't trust AWS anymore, but I want to give most of the work to the AWS server.

I also updated my system status email notifier to keep track of time in addition to disk space, CPU usage, net usage, Ubuntu updates needed, etc.

September 30 - October 1 (started 9/30 02:21, continuing 20:27, first posted around 9/30 22:01) - continuing 10/1

Below is my first entry from early this morning (9/30), where "early" means 2 - 3am.

For at least the last day or so, the NTP server(s) for AWS EC2 us-east-1 have been about 4ms slow. I actually have some trouble believing it, but I have verification; I'll come back to that. I am reasonably sure that some systems would be having problems with that much of a difference. I would think someone paying for AWS support would be howling by now.

I verified the problem in us-east-1a, 1b, and 1e.

Notes to self on working AWS: Auto assigning a v4 IP address is at the subnet level. The availability zone (AZ) is also there. It seems that at subnet creation I have to enable IPv6. I'm going to try with AZ -1e now. I didn't know that e and f existed. They must be a year or two old at most. (This was written before the above.)

My own various SNTP clients tell me that us-east-1 is 4ms off. Also, when I configured chrony with both the AWS NTP server and the Maryland NIST servers, I could see the conflict. I think the conflict even "broke" chrony, perhaps because I did not select a preferred server. I could see the different results in the chrony log, and chryonc tracking failed due to the failure to sync. (Upon further review, this probably only happens from roughly 30 seconds after launching chrony to 90 seconds.)

Check the 2 subnet boxes, "Enable auto-assign IPv6 address" and "Enable auto-assign public IPv4 address." Those settings are then confirmed / changeable at instance launch.

Same problem with -1e. Seems all of -1 syncs with the same server. Very weird. As I've nattered about at some length, US East 1 is in northern Virginia and less than 2 network ms one way to the NIST servers in Maryland. It's about 30 miles, as I remember.

Roughly 27 hours ago I was working on my page that shows my server's time "readings." I noticed the 4ms consistent difference. Within a short period I had given up on the AWS server and synced directly with NIST.

NIST says never poll more than once in 4 seconds. I originally synced to all 7 Maryland NIST servers (that don't require encryption) and set the polling way, way high such that I wouldn't be anywhere near that limit. That led to some very odd results. I would expect chrony to phase the polling, but it doesn't. The polls would all fire at once, and the chrony log was just weird to watch. I'm sure it was syncing just fine, but it was way, way overkill.

I decided to sync to 3 servers so there is always a majority in case of disagreement. I randomly picked 3 servers. I realized I really needed to set one of them to a min poll of 4 seconds so that the initial sync happens in reasonable time. I could use "iburst" (initial burst of polls upon start) with AWS but, off hand, I saw no such invitation at NIST. It seemed that the sync took a while with a high min poll. I may play with that a bit, in fact.

In any event, I wound up with this on kwynn.com:

server 129.6.15.29		minpoll 2 maxpoll 13 xleave
server 2610:20:6f15:15::26	minpoll 4 maxpoll 13 xleave
server 2610:20:6f15:15::27	minpoll 5 maxpoll 13 xleave

The minpoll of 2 means 2^2 === 4. I find "min" and "max" confusing in that it seems the terms should be opposite. Those are the 3 NIST servers. I've found that the polling doesn't get anywhere near 2^13 seconds. The logs show that the polls are every 13 minutes at "worst," and my offset (error) times are on the order of 50µs, so that polling is just fine. (Revision: it does go towards 2^13, so I'll reduce it later.

I started up a new instance to play with the 2^2 versus higher settings, and I got an sntp "ping" of 2ns to the NIST server! A ping is what I call the measured error between my system and the server, based on an SNTP time request. I wonder if I'll ever see that again. I often see 7µs but rarely in the ns range.

It's like a 4 leaf clover. I'll enter it into the record:

$ php s.php
1664524061841247666
1664524061843091158
1664524061843092294
1664524061844935789
2610:20:6f15:15::26
0
1,843,492
1,844,628
3,688,123
-2

Perhaps I'll explain all that some other time. It's in my SNTP GitHub and such, to a large degree. Well, it's there to a total degree in that the code is there. My readings to AWS are around 4,000,000 (4ms) rather than 2 (2ns).

If I set 3 servers to a minpoll of 5, it takes the 3rd poll at 64 seconds to sync. If I set the 7 servers to the same, it still takes 64 seconds.

I am also confirming that the AWS server "breaks" chrony. If I put one NIST server and AWS on 2 seconds maxpoll, it takes 64 seconds to sync, when the other servers break the deadlock. If I keep that that the same but set the AWS server to "iburst," then chrony syncs very quickly and then loses sync. Then it gains it in 64 seconds. Then it can't increase its polling interval, or it keeps going up and down. I can see the 4ms error versus 2 - 3 orders of magnitude better for NIST. That is base 10 orders, not 2. It looks like in this case chrony throws out AWS and starts to increase poll interval. I'll try to reproduce "breaking" chrony.

Now I have AWS on iburst and default polling intervals--not explicitly set. I set 3 NIST servers to maxpoll of 5. AWS does the initial sync, then NIST takes over, probably after 64s, although I didn't notice it for sure. I restarted chrony, and for now I see that the NIST servers are off by 3ms. Then the NIST servers "take over."

Maybe the break I saw was very brief. In any event, it seems definitive that the AWS server is around 4ms off. 4.2 seems the most common number. The alternative is that 2 NIST servers are broken. There is a thought. I'll try the settings as above except for 3 "new" servers. It takes 3 cycles--96 seconds--for NIST to "take over."

Even if I set AWS to "prefer," NIST "takes over" in 2 - 3 X 32s cycles.

If I set AWS to the default polls and the generic time.nist.gov to maxpoll of 5, the NIST load is "balanced." In this case, chrony got latched on to one of the Colorado servers. Then chrony is looking at a long net time, it can't compensate, and AWS wins.

Some digging shows 4 Amazon servers 0.amazon.pool.ntp.org though 3... The ping times to all are awful, though, for my purposes.

September 26

September 24 (posting 9/26 or later)

September 17 - 19

September 7 - Mr. Zoom II (updated after 20:46)

First posted roughly 19:30. My next reply after 20:46.

I had an encounter with Mr. Zoom II--probably not the genetic son of Mr. Zoom I, but conceptually related. Looks like I first mentioned Mr. Zoom [I] on May 30 and updated that on July 4, but your search of this page is as good as mine. May 30 seems later than I remember.

Below is our exchange. I am removing one technical keyword to make it a bit harder to find him, and because the word isn't relevant.

My blood pressure went up, and I started to blow my stack. Then I decided that his reply was well-written, so I tried a compromise.

His ad:

I am looking for someone to help with a ... project I am working on. The gig has potential of becoming a full time paid gig. If you are interested please let me know and I can give you the details of the gig and what is entailed with the project. I would keep at it on my own but other project keep pulling me away and I don't want to get behind on this project.

his reply 1

Hi kwynn, I am still looking for some help. I would need to do zoom meeting so I can meet you. If your still interested. [signed]

September 5

August 28 - email server, IMAP, etc.

This is part of my gig application for a Craig's List gig / "temp jobs" / "computer temp jobs."

To take your ad literally from the top and start with Linux, I am typing on Ubuntu Linux, and Kwynn.com runs on an Ubuntu Linux VPS. My /t directory tree is evidence that this site is over 11 years ago. A whois lookup on my IP address shows AWS, and thus this sever is a Linux VPS.

I am still on the front page of Big Evil Goo for the search term "Python IMAP IDLE" without quotes. I'm #7. I've held #6 - 8 for over 7 years. Here is the direct link to my IMAP IDLE.

On the encryption end, here are the steps to do "by hand" a large portion of what TLS does. From memory, I create a long symmetrical key and then encrypt it using "Bob's" public key. I send the payload to Bob encrypted with the symmetrical key and then the sym key encrypted with the public key.

I had a problem with Goo OAUTH working with the PHPMailer class. I started with someone else's solution, but I had to do some hard thought to modify it. I considered posting a sample of this code, and I probably will if it's round two of my application. For now, I decided it would take too much work to make it generic enough for an example.

My GMail "positive email check" is using a Goo PHP library and is therefore somewhat removed from the raw server, but that is just as hard to get working as dealing with the raw server.

Regarding Postfix, a funny story about that. About the same time I wrote that encryption script for a friend, we were discussing running our own mail servers. It turns out that I did get incoming email working, as I explain: I was using port 25 for SMTP, incoming in this case. There was some talk that residential ISPs blocked it, but it wasn't blocked. Then I created an MX DNS record, set up Postfix, and sent email to the domain I used in the DNS record. I think what went wrong is that root received the mail a la 1991 Sun Solaris UNIX. I hadn't sent the email to an existing Linux user, so it went to root. The sudo command won't show you that root has mail. I didn't realize the email receipt worked until I actually did "sudo su" to open a root-user shell. Then I saw the notification a la 1991 at the command prompt. I should have been looking in, as I recall, /var/spool/...

I dealt briefly with SPF. Goo is doing the work, but it shows that I can implement instructions and such. For that matter, I stepped my client through adding the DNS record. I may send the actual domain name in the email; I shouldn't include it here.

$ dig example.com TXT

; ... DiG 9.18.1-1ubuntu1.1-Ubuntu ... example.com TXT
...
;; ANSWER SECTION:
example.com.		300	IN	TXT	"v=spf1 include:_spf.google.com ~all"

;; Query time: 188 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Sun Aug 28 21:29:01 EDT 2022
;; MSG SIZE  rcvd: 171     

Regarding DKIM, I know what they are, and I'm sure I could figure it out quickly.

August 24 - the Gab troll

Weeks ago I posted a variant of my apprentice ad to Gab's "Programming, Computing, and Electronics" forum. As you can see, I acquired a troll. I'm not sure I've ever been trolled before. If I have, it's been a long time. I didn't even register "troll" for a while. At first, I thought it had a following. Its header says "1.2k Followers," but when I click, there is no one. I see that inconsistency with other people, so I'm not sure what to make of it. I also see lists of followers with some people, meaning that the following header does work in some or most cases.

Now that I have partially grokked "troll," I'm continuing to engage it because I have evidence of the phrase, "There is no such thing as bad publicity." And that's assuming anyone is taking the troll seriously and thus it's bad publicity. I have a few pieces of evidence that engaging it has kept my apprentice post alive. A few days ago I made contact with my first potential apprentice from Gab. I've also seen Gab HTTP referrers entering my site since the troll engagement.

In addition to keeping my post alive, this is an exercise in engaging a troll while keeping my blood pressure down. As I said a couple of days ago, I have almost no reason to think that my blood pressure will lead to any problem; I haven't taken the clot shot. This is simply an exercise in emotional control.

So, to address the troll's points: This is not a matter of being offended. If I thought anyone took it seriously, I could probably sue it for defamation. I suspect Gab would back me rather than fighting giving me its identity, or leads to its identity.

When contrasting "the internet" with the real world, I suspect it is being deliberately obtuse. My real name is right there, and the troll is potentially damaging my ability to do business. By contrast, the troll is anonymous and thus takes no risk unless I went to the courts.

As I told it or at least implied, anonymous trolls are fine and useful in quite a few contexts. Public figures are fair game. Defamation law distinguishes public and private figures. I am not a public figure. In my post I am not advocating policy or trying to affect anyone other than a potential handful of people who might want to become apprentices.

The troll is only worth so much time. I should boil this down for posting to Gab, though.

August 23

A chat with Gab potential / probable apprentice...

Reasons to move from PHP to Node.js:

• end to end client to server JavaScript
• It's the natural language for and of MongoDB. PHP can be rather painful with Mongo when the queries get complicated, such as the $ symbol conflict.
• I noticed a few hours ago that AWS Lambda does not natively process PHP. Similarly, I have seen instructions and even tests that have several languages, but now one of them is Node and not PHP.
• Alpaca had Node and not PHP. The hired.com test quiz had Node and not PHP. Those are two more that come to mind.

A restatement: Kwynn's Rule of Software Development #1: NEVER develop without a debugger. I found that one of the links at least in my apprentice 2 paths page is still useful for showing what a debugger is. That's client-side JavaScript, but the concept is the same in any language. I use NetBeans and php-xdebug for PHP. I held my nose and use the free and open source SatanSoft Visual Studio Code for Node.js. I've used PyCharm for Python. A debugger is something that lets you run the code line by line and see what it's doing, including watching the variables change and the call stack go up and down. You can also set "breakpoints" on specific lines such that the code runs until you "break" (pause) at a given line. Then you can go step by step. You "step into" functions or just "step over" them and see the result on the next line. Or you can "step out" of them. That's the basics. The quotes indicate terms that many debuggers use, if not all of them. I've also used Eclipse for PHP.

I should add that a debugger does not solve your bugs or literally debug them. It's a tool to let you debug them.

August 22

The potential apprentice situation that I was ranting about for the last few days seems to have come to a bad end on his end. I suppose you could could say he withdrew his application for apprenticeship. Perhaps I'll come back to that.

The potential apprentice from Gab is coming along. He can write! He passes my writing tests. His first question today is "Do you mind sharing any details in how you got burnt in previous work?"

Divorced, beheaded, died, divorced, beheaded...

Let's make a list of codenames / brief summaries, then perhaps I'll randomize their order and expand.

• the opposite of the British "peer" telling the truth
• meltdown over $100 - $200
• vanishes when I start talking about charging (instance 1 - filthy edition)
• 8 year deprecated PHP code
• 7 hour talk / the traveler
• Mr. disorganized in several ways and bad communication with everyone, not just me
• doesn't feel needs to pay minimum wage
• pay me peanuts then Number 1 makes assumptions
• vanishes when I start talking about charging (instance 2 - laborious edition)

August 21 (at least 4 entries; the first just after midnight, the second starting 03:05am, 3rd at 20:42, the 4th soon after)

August 20 (starting 23:42)

My potential apprentice answered my question about back end projects. "What backend projects can you help me with?" I read that about 10 hours ago. My thought for most of that time has to been to roll on the floor laughing. (Yes, I am aware there is an acronym for that phrase.) Or perhaps I should cry.

I'm starting to appreciate the front end more and more, but I had to be dragged there. It's where the projects I found led me. I think it's safe to say my heart is still in the back end. So the only question is do you want ideas or do you want to present me with ideas?

If you want ideas from me, I'd have to resist writing dozens of ideas. I'm not even going to start until asked. It's too dangerous in that I could be at it for hours.

To elaborate on your presenting me with ideas, I have done and will do a huge variety of projects, so let's hear it. I'd imagine I'll entertain it.

August 19 (starting around 23:15)

Note that my previous entry was updated a few hours ago, during the night of the 19th.

A potential apprentice wrote me a few days ago. I replied here, but I made one imprecise statement that needs correcting, and I otherwise want to rephrase.

I mentioned a website to showcase his work. That was imprecise. I'll try again:

One way I can help apprentices is to get them work experience. The work does not have to be for me; it can be something you want to do. I can play the role somewhere between boss and thesis adviser. That is, I can make sure you're doing it well enough that it is legitimate experience. I don't have a corporation or other formal artificial entity, so that presents a very small but management complication. One option is that you say you worked for Kwynn.com. I'm fine with that, but I would understand if you don't want to be associated with my louder and louder "irritation" at the powers that be.

The alternative is to create a website that does not have to list me publicly, but I can speak as your managerial reference. The website doesn't have to have much content. Ideally I would want whatever you generate to be public, but, if not, the site only has to make reference to what you're doing.

August 18 - 19: "hello world" for the Alpaca stock and crypto trading API (in progress 00:57 8/18, continued 8/19 21:16+)

updates 8/19

Updates during the night of 8/19: Here is the latest code and what should be a permanent repo. Maybe I finally solved the code-fragments problem.

To add to yesterday's (this morning's) commentary, I was a bit concerned when Alapca said they wanted NodeJS 14+, and the command line shows 12. I added version output which currently shows v16.15.0. NodeJS installs versions of itself in ~/.npm I believe. It likely does this based on the package.x files. Visual Studio Code is smart enough to know which version to run.

original 8/18 - early 8/19

The Alpaca Trading API hello world documentation is pretty good, but it contains an annoying bug. I was following the JavaScript instructions, but the bug may be in all of them. The first trading example is to buy BTC, but if one tries to buy BTC with a "day" "time_in_force," you get an HTTP 422 error. The "gtc" setting rather than "day" works. GitHub says the settings are 'day' | 'gtc' | 'opg' | 'ioc' "Time in force" is a general stock trading term.

I think I mentioned that I held my nose and am using Visual Studio Code, the free and open source version, to debug JavaScript (server-side Node.js). It works. I'll probably live. (Those taking Bill's vax may not live. SatanSoft should be seized and sold off to pay damages.)

I can manage in Node.js, and one day very soon I may really dig in, but for now I still run into unexpected issues. If you want to keep your API key out of the public code, it takes some unexpected syntax. Also, for debugging purposes, the example code needs to be modified a bit to deal with asynchrony.

Once again I'm going to refer to a specific version of code. The code is subject to updates and the new versions being moved entirely. In fact, that is the broken version for purpose of demonstrating the time_in_force error. I just pushed a new, working version that should be easy to find from the above link.

Note how I use module.exports.publicSampleCreds in public_example_creds.js to make the data accessible to the main program. I am not certain this is the exact right way, but it works for now. I'm sure I'll learn more eventually.

Then I write a couple of async functions to make sure the the program doesn't end before I see my output.

Here is more about the 422 error. Note that in the createOrder.then() I need both the success and failure parameters to be able to see the error. The various levels are

Error: Request failed with status code 422
        error.response.data: {
            code: 42210000,
            message: "invalid crypto time_in_force",
        }

Out of (sandbox, fake) money gets an HTTP 403 error, and error.response.data explains this. Oddly, when you put more money in your account, you need to use new API keys, and trying to do anything will get you another 403 error much earlier in the program.

August 14 (first [and only] post after 22:35)

This is to a new potential apprentice. Your comment about websites versus APIs reminded me of an XKCD. Here is the Explain XKCD version, which is more useful for a handful of reasons, and the original. The main text is "If you do things right, it can take people a while to realize that your 'API documentation' is just instructions for how to look at your website." The server side (back end) of a website to a degree is an API. You know the term "back end," and there is such a beast as a "back end developer" who focuses on the back end of a website, which is (in part) the database and more-or-less explicitly an API.

(I doubt I understood the IERS reference at the time. Now I do. That IERS reference is quite funny when one gets it. About 22 hours ago, though, I was writing in my personal blog about how dangerous XKCD has become. Anyhow, those are other topics.)

For the moment I'll interpret your request as "How can I (Kwynn) help you get an 'IT' position?" I'll take this in a handful of steps. We'd have to figure out what project(s) you want to work on. I have lots of of ideas that are back-end only, but they don't have to be my ideas or even anything I'm interested in. You work on projects, I periodically evaluate them, where "periodically" might range from a few times a day to monthly. Then you can list experience working for this site or some new site we create just to showcase what you're doing. You can list on your resume n months of experience working for kwynn.com or XYZ.com or blah.com. I serve as managerial reference to confirm it, and job recruiters will love talking to me; they'll want to recruit me, too. It probably would not take very many months to start getting contacts from job recruiters (headhunters).

That's the first notion that comes to mind.

August 9

August 8 (last entry roughly 22:30)

ENTRY 1: I created a utility to test my simple timeserver. For my own searching purposes: port 8123, TCP, UDP. (02:11 AM then slightly revised 02:12)

August 2

I think I was about to launch into a new entry on 8/2, then I didn't. This has not been substantially changed since some time on 8/2.

Yes, I am up very, very early by my 25 year usual.

Failed to describe disableApiTermination for instance i-0123...
Failed to describe instanceInitiatedShutdownBehavior for instance i-0123...

July 31

A few hours ago I posted this site's update / change log. The source code is linked from that page.

My blood pressure just went up while cleaning my inbox of calls from a job recruiter. I've already complained about this one, I believe. Several weeks ago I sent him my SatanSoft formatted resume exported from LibreOffice. One of many irritations is that we are 30+ years into the web era, and I am still asked for SatanSoft resumes. I answered his question about rate, citizenship, availability, blah blah blah. Then, like almost all recruiters always do, he went dark.

Note to job recruiters: you lose whatever smidgen of credibility you have when you go dark for 2.5 weeks. "Thank you for your information, but the job is taken" is all it takes. Apparently the job became available again, but I am not jumping up and down to waste more time. It's not just answering his questions and dealing with SSoft resumes. I generally explain that the first thing I care about is the night owl thing. (I'm writing this sentence at 4:23am.) I am very sick and tired of saying that over and over and having it ignored.

I am reading the transcription of his voicemails. Part of my irritation is that I really don't want to have to listen to him. He's Indian, and I really resent that I am dealing with Indians. I also find it interesting that Big Evil Goo Voice's transcription doesn't do as well with Indians as Americans. There must be hundreds of millions of hours of Indian voices to train the "AI" on. I suspect this goes to the "racist robots." Perhaps most Indians simply OBJECTIVELY cannot speak English clearly enough.

There are hysterical articles on "racist robots" / racist AIs. AI essentially proves that racism is objectively, mathematically valid. It seems the AIs have become "racist" on quite a number of occasions. I could go on about that; perhaps some other time.

Anyhow. Good. I feel a bit better now. I did respond to him and sent my resume again. I asked if he wanted me to answer his other questions again. This should be amusing if not blood pressure raising.

(posted 4:44am)

July 27

New utilities:

1. my own copy of the HTML validator
2. ipv4 echo
3. ipv6 echo

July 26

I'm posting my July 26 and July 25 entries at the same time.

When I start a new instance, any sudo command, including the innocuous "sudo echo blah" results in "sudo: unable to resolve host ip-10-1-2-3: Name or service not known" where ip-10-1-2-3 is whatever is after @ at the command prompt and / or the result of "cat /etc/hostname" The solution is to change /etc/hosts such that the 127.0.1.1 entry (yes, 1.1) corresponds to /etc/hostname

The above problem will happen with every new instance. This next problem has to do with a new install of MongoDB. Now "sudo apt update" results in "W: https://repo.mongodb.org/apt/ubuntu/dists/focal/mongodb-org/4.4/Release.gpg: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details." I think I'll sit down and fix that now.

Well, the instructions were close, but it took a bit more doing. The standard but now slightly out of date instructions tell you to create a sources file and a public key file. The key file command needs to be modified as I show below. The command to create the sources entry is fine except that you need to modify the sources file as below, too.

wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo tee /etc/apt/keyrings/mongodb-server-4.4.asc
ls /etc/apt/keyrings/
# confirm the file exists
mongodb-server-4.4.asc
head -n 4 /etc/apt/keyrings/mongodb-server-4.4.asc
# confirm there is the beginning of a PGP public key there
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: GnuPG v1

mQINBFzteqwBEADSirbLWsjgkQmdWr06jXPN8049MCqXQIZ2ovy9uJPyLkHgOCta
# at this step YOU HAVE TO MODIFY the sources file so that it looks like the following
# the next command simply shows what it should look like when you're done modifying
cat /etc/apt/sources.list.d/mongodb-org-4.4.list
deb [signed-by=/etc/apt/keyrings/mongodb-server-4.4.asc] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse
# old version with my failed attempt at modification commented out below
# note that the signed-by completely replaces arch
# deb [ arch=amd64,arm64,signed-by=/etc/apt/keyrings/server-4.4.asc ] https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 multiverse

sudo apt update
# now should result in
# ...
# Ign:4 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 InRelease
# Hit:5 https://repo.mongodb.org/apt/ubuntu focal/mongodb-org/4.4 Release  
# ...
# All packages are up to date.     

Note that the "gpg" command itself does not come into play. There may be ways to get it to work, but I didn't get that far.

July 25 - a knock down, drag out fight with Apache

I won the fight, but it was tiring. Now that I have a micro instance with 1GB of RAM, I should be able to (more easily) run my own copy of the W3 HTML validator. I was working towards that. My previous solution, months ago, on my local machine, was to run the validator on port 9999 and do all this "Proxy" directive and such. In hindsight, it was a clumsy solution, but that's what "the net" proposed, and it did work.

I decided I want my validator to live at https://example.com/htval. After much gnashing of teeth, I have two solutions, one somewhat more elegant than the other.

To back up a bit, I have restructured my /etc/apache2/sites-available files. Hopefully I'll post the latest versions soon. Earlier versions are in GitHub under a repo with sysadmin in the name or something. I separated the files into .conf and .inc files. .conf is the Apache format for config files. .inc is my arbitrary format meaning "include" files. The .conf files enable a specific site. The .inc files are included with, not surprisingly, an Apache Include directive.

I have a "private...inc" that uses the Define directive to define the server name and document root. That is, the Define var becomes the literal ServerName directive / variable, and the DocumentRoot. (Actually, I am still using /opt/blah for the DocumentRoot, but it could be done that way.)

For the local copy of kwynn.com, the kwloc.conf only has 2 lines that are the private..inc Include and a "pubcommon.inc" file. The pubcommon defines the non-secure http redirects and DocumentRoot and various other directories (Directory) and Alias(es) and such.

The following Location entry can be included from anywhere in the common file or the .conf file. The following is my second and somewhat more elegant version because it can be defined anywhere.

First I'll explain my solution, then I might get around to explaining the various difficulties I had to pick through.

I post my earlier "Directory" version further below. Note that the context of what you get from the URL changes between Directory and Location. That is, you're getting a different string into the regex and thus the regex must be different.

The biggest problem I had was a URL of /htval versus /htval/ . The Nu validator uses "style.css" rather than "/style.css" as the path. Given that just about everyone is going to access their system indirectly rather than through their default 8888 port, I really wished they would have used the /. Given that it's open source, I'll add it to my stupendously long to do list. That is, I should lobby for the change and / or make it myself when I have time for such things.

Anyhow, the first rewrite rule turns /htval into /htval/ This should NOT use the P (proxy) option. The proxy option does not change the URL, but we need the URL changed so that Nu interprets style.css as /htval/style.css or /style.css, depending on how you want to look at it.

The second rewrite rule solves a somewhat puzzling problem. Even though there is an index.html in Nu, trying to access it directly results in a 404 error. If you want the Nu home page, you have to ask for precisely / . The third rewrite is a fairly standard rewrite of the rest of the URL.

I can't remember if 302 is the default over 301 or not. In any event, I think it's best to explicitly use 302 (temporary redirect). I guess once you're certain it's working it doesn't matter, but I think it's best to err on that side. Maybe in a few months I'll change it.

Remember that if you do use 301, if you want to change anything you'll have go the History in Firefox and "forget" the site to make it stop using the 301. Also remember that you'd better do a hard refresh when testing: control - F5 in Firefox. Also also (sic) remember that you have to restart or reload Apache with every change.

I don't claim to fully understand the L (last rule to execute) option. It works. I'll leave it.

This is the sort of development (sysadmin) that you literally have to take one line at a time and make sure the first one works before you keep going. My earlier versions dealt with the Proxy and ProxyReverse and all sorts of unnecessary code. I was changing too many things at once. I was previously using port 9999 and redirecting to 8888. I was using something like 15 lines. The following is a huge improvement on a number of points.

I thought I was going to show the code then explain it, but I did it the other way around.

# begin included file
<Location /htval>
	RewriteEngine On
	RewriteRule htval$   /htval/ 			 [R=302,L] 
	RewriteRule htval/$ 	http://localhost:8888/ 	 [R=302,L,P]
	RewriteRule htval/(.+)$ http://localhost:8888/$1 [R=302,L,P]
</Location>
# end included file
    

The following earlier version was in the Directory entry of my DocumentRoot directory.

RewriteEngine On
RewriteRule ^htval$    /htval/ [R=302,L]   # do NOT use P for this!!! ****
RewriteRule ^htval/$ 	 http://localhost:8888/ [R=302,L,P]
RewriteRule ^htval/(.+)$ http://localhost:8888/$1 [R=302,L,P]
    

July 24 - a second, recent kwynn.com instance / MongoDB upgrade

After getting the new home of kwynn.com working, I realized I was still using, rather embarrassingly, MongoDB v3.6. It seems that hasn't bitten me yet. I decided, though, not to wait around for it to bite me. Locally I am running v4.4 which is the last of the v4.x. Version 5 has been out for a while, and they are up to 6. I mentioned months ago that I cannot run v5 locally because my CPU is too old. The good news is that version 4.4 reaches end of life in February, 2024. This situation irritates me, but if I solve my basic problem of finding enough work, everything else gets solved. It's probably not worth even digging at a Raspberry Pis for the moment.

In any event, kwynn.com already has another new home over the course of several days. This time I set the time to live (TTL) on my IPv6 AAAA records to 1 hour, very early this morning. Just as I sat down to create the new instance, I set it to 5 minutes. Once the new instance was ready, I gave the new records a 5 minute TTL and left them both "on." I found that it's probably a good idea to close one's browser after 5 minutes because "old" tabs (20 minutes old or less) will still go back to the old address.

MongoDB v4.4 would not auto convert the old data files. I think I went back to 4.2, but I don't think I went back to 4.0. I may not have taken the time to realize that 4.0 existed. After the second version failed, I did this:

1. Export (mongodump command) the data from the earlier version
2. stop the server
3. remove all the files under /var/lib/mongodb (Don't remove the directory because Mongo will not necessarily create it.)
4. upgrade Mongo (and make sure it's running)
5. load the data (mongorestore command)

The careful and knowledgeable reader may have understood the context of the instance change, but I'll clarify. I made a no-reboot image of my live instance, fired it up on a c5ad.large instance, did the Mongo upgrade, and tested everything with a subdomain of an extra domain I lease. Then I made an image of the upgraded system, loaded it as a t3a.micro instance, tested again, and then created DNS AAAA records for the new system. I watched for signs that the browser was picking up the new records at least part of the time, then I turned off the old records and switched the IPv4 via the AWS EC2 Elastic IP. I watched the web server access log of the old system to make sure requests stopped coming in, then I shut the old (virtual) machine down.

July 23

I originally posted this around 13:48. At 13:52 I added one word to my email to him and emailed the jackass. I added the word below, too.

In my own pure tech news, the new instance of kwynn.com went live last evening around 8:29. It's run fine since, and the CPU credits are still charging. I should make some more notes on that, but I have other topics I want to address.

In non-technical news, I was considering beginning my job application to be an influencer for the Russian government. I'm somewhat serious about that. Unfortunately, I should deal with a certain jackass first.

July 22 - yet another chapter in the upgrade

I'll try again to recap what needs doing for this and future upgrades. I may be a few hours from throwing the switch to my Ubuntu 22.04 image.

When doing the AMI (image) quick launch from image ("Launch instance from AMI"), the IAM role is under "Advanced details" > "IAM instance profile."

As I approach go-live:

• Using the entry in /etc/hosts, I should be able to test kwynn.com on the new machine before it gets DNS routing. I already have a copy of the SSL cert. (Yes, this works.)
• Give the instance some time to charge up CPU credits.
• quick link "cpu" must be tested on an instance that has limited CPU credits
• rsync to new machine
• update /opt/kwynn if needed
• reset /var/log/apache2/access.log and error.log as to not conflict with the last entries of the old one.
• test all services in launch-wizard / security group / firewall rules, including 4 and 6 ping

General testing / steps:

• update composer libraries both locally and in the cloud - update ALL libraries
• test my contact form
• go through my quick links on my local machine - it seems I was very bad at that
• the web random generator (started 2017) tests nanopk() and nanotime() and such
• remove older keys from ~/.ssh/authorized_keys
• set up the Apache conf files to test both the test name and permanent name

certbot

sudo certbot certonly --dry-run --apache
sudo certbot --apache
    

It's probably easiest to let certbot mod the Apache conf files as it wants, then go back and set them up for both test and live domain names.

July 21 - Kwynn.com Ubuntu 22.04 upgrade update

July 20 - database theory

I have updates from my "usual apprentice" who got a title something like that about 8 months ago. For about 10 months he's been working for one of the big tech companies. It's not one of the really big household names, but it is very famous in tech circles, and it was a major power decades ago.

His first job did not go very well, but he transferred within the company a few weeks ago. Now he's very happy.

His new job led him to some questions about databases. They were of the sort that I suppose should get a blog entry, based on our tradition months ago.

He's asking about ACID and normalization (3rd normal form). "[Do] you apply these principles when doing database design[?]" Yes, I do, with a lecture and perhaps a long lecture.

I'm getting somewhat away from the question, but ultra-normalization is one of the designs that drove me crazy with Drupal. I guess it's still driving me crazy because I'm still writing the migration scripts from Drupal MySQL (MariaDB) to MongoDB. Drupal version 7 has this concept of a "field collection" which is one of the more perverse designs I've ever seen.

Thankfully there were only 2 field collections before I got involved with the project. The most annoying one contained the main billing fields--which legal case the hours are billed to, notes on what was done, the broad category (jury trial, trial prep, writing legal briefs, interview with client, etc.), and then the hours billed. (There are other fields that made for an interesting design spec from my client, but those are the basics.)

I'm not going to take the time to think through the formalism, but the field collection is probably very correct in terms of 3rd normal form. There are likely other ways to have achieved that, though. What I had to deal with was a case of people following rules in an inappropriate context. Although that isn't even quite correct. The data format (schema, design) worked well enough for its one intended purpose, but the problem was that I needed to use the data in another context. The main purpose was to interact with the data on the screen. I needed to do batch, back-end-only operations, though. It took insane queries to get at the data directly with SQL.

That sort of problem is one of several reasons why I say that CMSs are only worthwhile if they can do 90% of what you want with the core product plus existing plugins. Drupal did maybe 40%, and some of that it did badly.

A large part of the problem is that Drupal needed to be very, very general and handle any type of data. This goes to my saying about CMSs that they are so general that they don't successfully do anything specific. (The saying is based on a comedian's saying from decades ago. I think I've referenced that elsewhere.) Whatever benefit Drupal gave to get the project started, it's on net cost the project an enormous amount of money as I undo the damage of starting with Drupal.

One point being to think through all the things your data needs to do and the ways it needs to do it. If you're first designing for user interaction, test the back-end queries you may need to use to make sure that your design covers both scenarios.

To head back towards the original question, on one hand, I barely remember the formal definition of 3rd normal form. The saying I learned was that the data in a table should be relevant to "the key, the whole key, and nothing but the key." "Key" in that sense means the unique index, which may be several fields.

It's been a long time since I did RDMBS table design because I've done all new work in MongoDB for 5+ years now. One of the benefits of MongoDB is that you don't have to worry about the design while you're prototyping. You don't have to "CREATE TABLE" in MongoDB--you just toss data into a collection. That has the theoretical potential of being a horror show, but my RDBMS training helps me stay disciplined. The biggest problem I've had is that I make mistakes around integers coming from an HTML form as strings ("3"), writing that string to MongoDB, and then that fails a query match with integer 3. That zapped me enough times that I get more and more disciplined about always putting my IDs through a validation function. I have one in my client's system called vidord()--valid ID or die.

I should add that there is a question whether IDs should ever be integers. You never do math on IDs, so why should it be a number? A system may be more flexible if it doesn't require integers. First of all, I'm not talking about the '_id' in MongoDB. The '_id' is the required unique index in MongoDB. That is almost always a string both in my case and by default. I tend to use separate IDs for querying purposes; I'm talking about that separate ID. In my systems, I make the '_id' a human-readable string so that I can see what's going on at a glance in the Robo3T MongoDB tool. Robo3T is a direct equivalent of MySQL Workbench. By default, the rows (documents) list themselves by _id in Robo3T (or the command line), so it's nice to have something human-readable. I can make my own IDs unique indexes (keys) with createIndex(..., ['unique' => true]).

In any event, Drupal used sequential integer IDs, and in this case I don't see a problem continuing that. For one, it would be more difficult than it's worth to change it, even though I'm migrating to MongoDB. For another, I got burned decades ago by inflexible numeric IDs, but this is my system with a specific purpose; I don't need to be general.

The integer IDs in MongoDB brings up the ACID issue. The question of sequential IDs in MongoDB was a thorn for a few years. That doesn't mean I spent 4,000 hours trying to solve the problem, just that it rolled around in my head on and off for years. MongoDB does not have a native AUTO_INCREMENT, nor can you lock a collection (table) like you can in an RDBMS.

AUTO_INCREMENT is a built-in guarantee of atomicity, the A in ACID. The RDB takes care of the potential race condition for you.

In MongoDB, the answer is most likely to avoid sequential IDs *IF* you're starting from scratch. Otherwise put, if you're starting from scratch, you might as well use the required _id, although I think it's a good idea to add human-readable elements as opposed to the default _id (a hex number that turns out to have useful properties, but that is not at all obvious, nor is it at all human-readable).

As I said, though, I am going to continue using integers with my lawyer client's system. There are a number of decent ways to solve the problem in MongoDB. The one I'm using is to do a Linux level semaphore lock using the DAO file's path to ID the lock itself. I do all the processing I can without the ID, then I get the lock, find the max existing integer with a findOne sorted by descending index, add one for the new ID, write the new row, and then unlock. It's a really good idea to already have a descending-sort unique index on the ID field anyhow, for data integrity purposes. Thus, that query will always be as fast as it can get.

To go back to the original question again, yes, when I was using MySQL I thought about what needed to be a bridge table (many to many) and otherwise how to arrange the data. Something close enough to 3rd normal form was burned into me decades ago such that I'm not consciously thinking about it, but yes, I am unconsciously thinking about it.

Another benefit to MongoDB is that in many cases you don't have to normalize. The fundamental structure is different, or at least it *can* be different. You could write MongoDB collections just like RDB tables. MongoDB does not have referential integrity constraints, though.

You don't have to normalize in the sense that often you can get the data you need in one document without the equivalent of joining and pulling from related tables. Put another way, I moved to MongoDB in 2017 in part to avoid normalizing.

With that said, a few weeks ago I found myself normalizing in MongoDB, and I'm probably about to do it again, both within my lawyer client's project. I recently posted a drag and drop example where the ordering is kept as a float. I decided that the ordering table (collection) should stand on its own rather than being part of the billing data collection. This goes to "the key, the whole key, and nothing but the key." The billing data is associated with a bill line item ID. The ordering is based on the ID of the whole timecard because the order represents the order of the line items on the screen.

It would probably take me a few minutes to think through why I decided to "normalize" the ordering, but not now.

In some cases I've needed to consider lots of data coming from an HTML form nearly at once, and I had to consider how to reject network packets getting out of order. In other words, I don't want older data to overwrite newer data. I use HTML "data-blah" attributes / JavaScript DOM element.dataset.blah variables to add both a sequence and a ms timestamp, from the client's clock. I know for a fact that two keystrokes when typing can get the same ms. If you go read the fine print of JavaScript threading, I *think* with the sequence you are safe from race conditions. For further projection, though, I also have a feedback loop that doesn't turn the field green ("OK" color) until the data that the database wrote matches the current value of the field. In some cases I wait until 2 seconds after typing stops and then check one more time.

So, that is all to say that yes, I consider normalization and ACID, even in MongoDB.

One day perhaps I'll add links, but I should at least list all of the solutions I came up with to the MongoDB sequence problem. They are all in my GitHub, but they are scattered among several repos and obscure corners thereof. You can easily write a "find one and increment" in MongoDB. That process is different than an AUTO_INCREMENT, though. It's a loop that runs either once or twice. I don't remember that exact logic, but you are guaranteed to get the next sequence the second time around if not the first. I ran it with all my cores at once and proved the before and after. It works, and the logic is sound, too, although I don't remember exactly what it is right now. This solution assumes that you have a document whose only purpose is to keep track of the next integer.

The "before and after" means that without the loop, the sequencing would trivially fail with all cores banging. With the loop, every number between 1 and several million were assigned.

As above, the Linux level semaphore works as along as all code doing the increment is using the same lock, which in my case is the same file of PHP code.

I wrote a PHP extension to get each core's clock tick plus the core ID. That will get you unique integers that always ascends (between boots), but they will not be sequential. The clock tick resets with each reboot, so you need to add another field such as ns time or even s time. Second time will work fine assuming your system cannot reboot and start running the application within 1 second. To be unique among different machines or VMs, you'll also need a field to ID the VM.

As I remember, if you're banging away with all cores in a tight loop all trying to get the next sequence, the core tick is much, much faster because there are no locking issues.

That's plenty enough nattering.

July 18 - Kwynn.com system upgrade, continued

I think I started this on June 10. I started again to upgrade to Ubuntu 22.04. I found 8 links in my quick links that were hard-coded to kwynn.com. That is usually a bad idea. While I was testing that, I ran into just the sort of problem I was looking to check: the sunrise function in PHP is deprecated, so I have to fix that.

July 16, 2022 - I crashed this site this morning

It seems that the latest updates of Ubuntu 20.04 and / or around the 5.15.0-41-generic kernel finally exhausted the 0.5 GB RAM of a t3a.nano instance. Ubuntu 22.04 server requests 1 GB RAM, but I didn't go back to look at 20.04.

In the AWS EC2 web instances list, if you're on the instances list with an instance checked, or else drilled down on a specific instance, Actions -> Monitor and Troubleshoot -> Get system log is helpful. It seems that it takes several minutes to update, though. The message
"[ 3.159381] Out of memory and no killable processes..." is a big hint, but it was somewhat hard to find. The last message,
"[ 3.169146] ---[ end Kernel panic - not syncing: System is deadlocked on memory ]---" is less clear, but it helped me figure it out.

I almost always reboot by creating an image. By default, creating an image causes a reboot because then the image can be completely consistent because it's not a running instance. (You can change that option in the image screen.) It was very handy to have created an image because I picked up where I left off, at least in terms of data. I upgraded a number of items while I was at it, but I'll come back to that.

I have found that something like 1 in every 100 boots goes badly for who knows what reason. I've found that most often on desktops. With EC2, sometimes boots go badly due to the network card not syncing correctly. That happens once every year or two.

In any event, when several minutes went by without ping starting to work or any other sign of life, I first tried a combination of rebooting and stopping and force stopping and then rebooting. Given the log delay, I eventually saw that I was going around in a circle with the same failure.

Then I tried launching an earlier image. That worked. I started upgrading the software of the image to see if I would indeed have the same problem. I eventually confirmed that I did. At one point, the update took a very long time--maybe 7 minutes. Once upon a time, some types of instances started with CPU credits. The t3a.nano and t3a.micro do not. That was probably part of the problem, but I think the settings were such that I was simply charged a few cents for the extra CPU rather than the system slowing way down waiting for the credits to rise. In any event, I seemed to be pegging the CPUs (I didn't even run top to be sure.) Although I may have been pegging the EBS disk. See below.

I started a c5ad.large image to see if the horsepower (a funny image) would help. Then I think I was up against EBS delay because it took a long time there, too.

I think I found that the upgrade worked on the .large, so right around that time it occurred to me to boot the image I had just made with a t3a.micro with 1 GB RAM. That worked and had all my data, minus a few minutes when I think kwynn.com was pointed at another image.

I was down for roughly 30 minutes. I could look at the web logs to be sure, but meh.

Lesson learned: Consider your DNS TTL (time to live) settings, especially for IPv6 in the case of AWS. More below.

Lesson learned: Periodically clean up the various "Security Groups" that crop up even with light usage such as mine. The have names like "launch-wizard-22." Those are the firewall rules. By default, one is created with each instance launch. I was having mild trouble finding / remembering which one I was actually using and selecting it. They don't let you delete the "default" even though one is almost never using it. Then label the remaining ones by clicking in the "Name" field that is "-" by default. I just took my own advice and deleted roughly 23 of them and labeling the one "live20to2207etc".

I add this because selecting among the security groups while I kept launching instances was very annoying.

Lesson learned: When you select an image to launch, the quick launch screen does not by default prompt you to select a role. It's in the advanced screen. This is generally not a big deal. It just affected my CPU usage widget. A role gives an instances permissions like a specific user, or a role can have an IAM policy, where IAM is a very fine grained permissions mechanism in AWS.

July 11, 2022

I expanded my June 11 entry into more front end examples.

July 4, 2022

part 1

I am adding a forward to my May 30 "Mr. Zoom" entry.

part 2 - more ranting about those who ignore the night owl request

In the last few weeks, several likely American job recruiters have acknowledged my night owl request in a useful way. For that matter, so did a lady who is probably Arab by both lineage and location. By contrast, I've had 2 - 3 developers tell me that dealing with Indian job recruiters is pointless, and they all have less reason or far less reason to be "racist." (Depending on what one means by the term, I am to some degree unashamedly racist.) So perhaps I am a fool for even engaging them.

One Indian recruiter made a vague reference that may have been an acknowledgement, on a voicemail. He won't acknowledge it in writing, though, despite my having gone into some detail. He's not responding to my emails at all, in fact; he just keeps calling me. The number he's calling is meant to be a tar pit for job recruiters. It's not set up to ring.

In roughly 2013, I posted my resume to Dice.com and got 30 - 40 calls the next day. I will go batpoop if people did that on my cell phone. I am not talking to a recruiter until they clearly acknowledge the point and indicate there is some small chance I can get what I want. It seems, by the way, that Dice has now been overrun by Indians. That's why I'm not making a live link.

The tar pit is a Big Evil Goo number. If Goo wants to suck in Indian job recruiters for "free," why not let them? One interesting point is that Goo's voicemail transcription doesn't do Indians very well. I could go on about that a some length, but perhaps not now. I got one transcription that started with "This is God." It would be disturbing if God speaks to 3rd+ generation Americans with an Indian accent.

Anyhow, this latest guy might have acknowledged my point in his first voicemail, but I can't make much out of the transcription of the second one. I haven't listened to it yet, for at least two reasons.

His accent is fairly understandable. It seems unlikely that he could speak English well enough but not write it. I am tempted to inquire whether he is literate, although I have wondered that about quite a few Americans, too.

I'll stop this entry. The main reason I bring it up is because I updated the May 30 entry on the same topic.

July 3, 2022 - creating an AWS IAM user to stop and start a test instance

I left my main client's test machine running overnight. This cost roughly 20 cents, but it's the principle. I get a bit paranoid turning it off for fear of turning the live machine off. Note that you can easily "Name" instances (virtual machines) on the far left side of the instances list. Just hover over the space and you'll get an edit icon. Naming "live" and "test" helps, but it's still not good enough.

My IAM solution was not all that I hoped for, but it does the job. If I had enough budget, the answer would be to write a program to automatically shut it down and use the same IAM policy.

You can use the visual policy maker, but note that some of the relevant permissions can't be restricted by a resource such as an instance ID. If you try to restrict them, the policy will break. This is what I came up with:

Updated August 2 (notes) - added "ec2:DescribeInstanceAttribute"

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances"
            ],
            "Resource": "arn:aws:ec2:us-east-1:123456789012:instance/i-abcdef012345"
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstanceAttribute"
            ],
            "Resource": "*"
        }
    ]
}   

The permissions with the "*" resource are the ones that can't be restricted. A login user with that policy can see the instance list and stop and start the relevant instance. One can also see other instances in the account, and the "stop" button is there, but pushing the stop button leads to an error; in this case, an error is the correct answer.

When using the limited user, note that you'll see all sorts of "API Error" messages and other red-colored errors because the user has very limited access. It also helps to click the instance you want, and then you can stop it. That's probably a good idea in any event--using any sort of user.

June 18, 2022

I made 2 changes to my June 11 "job / gig application" entry. I added "project flag" as an example, and I removed my defense of a usage of "he." I comment on that removal on my personal blog.

note on June entries through late June 17, 2022

As of June 17, 22:50 my time, I hadn't posted any June entries until a few minutes ago. I left them with their original dates, though, as they sat lonely on my SSD. I had backed several version up to GitHub, though. As I note and link to at the top of this page, I sometimes save drafts there.

June 14, 2022 - ranting about the "real world" job hunt

This guy was Indian, although American recruiters have rarely proven to be any better.

June 11, 2022 - This is a job / gig application.

Update: a month later I expanded this more front end examples.

"Please send at least 5 examples of [relevant] sites you have made, or worked on stating what your role was."

EXAMPLE: For the last six years, 5 - 10 hours a week, I have been developing and enhancing a web application that is a literal digital assistant for a law office. I say enhancing because I didn't start the project, but I've reworked it so much that I can call it "mine." My work on the system is cheaper than hiring a human assistant, and it makes a human assistant unnecessary. It's Linux, Apache, PHP, and started as MySQL. I have been moving the data to MongoDB. Soon I might get around to moving some of the code to Node.js, which is more compatible with Mongo than PHP is. The application is hosted on AWS EC2, so that makes me a perhaps-beyond-full stack developer down to the Linux sysadmin level. The system started in Drupal, but I am liberating it from Drupal. More details are on my resume.

The job description on hand "might" require references. My lawyer client will certainly serve as a reference, and I can think of several others.

Regarding troubleshooting skills, I did that more or less full time from 1997 - 1999, and those skills have only gotten better. I've done plenty of troubleshooting since.

Given the zillion lines of code I have in GitHub, I think that demonstrates I can use Git.

I list some (client-side) JavaScript examples below. In part to demonstrate that I am not delusional, as I implied, I tend towards full-stack business web applications rather than front-end "make me a pretty, public-facing website." I make no claim to be any sort of visual artist. There have been a number of times, though, that my "art" was close enough if the client knew more or less what he wanted to see. I have zero to little original artistic vision, but I can at least sometimes implement another's vision.

EXAMPLE: Speaking of artistic vision, I'd like to think that my numerology calculator "icon" is somewhat pretty:

That is HTML, not an image. It's also a live link to my numerology calculator. I haven't substantially touched it since 2010, but it's a useful piece of JavaScript and is written to an objective standard.

Note that I take neither credit nor blame for any visual aspect of my father's website mentioned below. I did NOT create the site and have done minimal work on the front end, but I've done quite a bit of work behind the scenes keeping it going over the years and almost decades. The numerology calculator on his site got caught in the visual crossfire between my original and the WordPress "developer." Again, I take neither credit nor blame.

EXAMPLE: The "flag project" parts 1 and 2 were visually demanding for my talent, or lack thereof. A client gave me a very detailed visual spec, and I did it such that he paid me for the mockup.

EXAMPLE: I hope my clock is a nice JavaScript example. There is some work on the back end of that, too. As I argue at some length, it should be a very accurate clock, and I make arguments that it's probably better than the official US NIST clock.

EXAMPLE: I wrote a Firefox extension for Facebook years ago. I didn't maintain it for quite a few reasons, but I believe I made my case (and the source code is still there) that it worked quite nicely, and it wasn't easy to accomplish.

EXAMPLE: In April, 2013 I wrote a Chrome extension that plucked phone numbers, email addresses, and snail addresses out of GMail and put them in Google Contacts.

EXAMPLES: my quick links / favorites. Roughly half of those are little, or not-so-little, applications I wrote. Some of them are just links to external sites.

EXAMPLE: Regarding Bootstrap, here is an example where I boiled down a zillion characters of Bootstrap CSS until I had what I needed. I was horrified at the thought of all that CSS for one example.

June 10, 2022 - moving my AWS instance

I'm in the process of moving this site's AWS EC2 instance from one EBS block to another. My goals are twofold. One, I am complying with my own dev rule #2 that the computer I'm typing on should be as close as possible to Kwynn.com's system. My site's Ubuntu version is getting behind, and thus so is my PHP version. I'm typing locally on Ubuntu 22.04 with PHP 8.1, and my site is on 20.04 and 7.4. My other goal is to increase my disk space. I am running on 12 GB right now, and that's getting too close to filling up the disk. My main client has been running on 16 GB with no problems, but I'm going up to 18 GB. I actually don't completely remember how I wound up with 18, but I already have an Ubuntu-upgraded EBS block stored, so I'm committed in terms of the time I spent on the upgrade. I moved it to the smallest "compute" type instance (that I list way below) to do the upgrade, and it still took over an hour, as best I remember. It may have been 1.5 or maybe 2 hours.

I immediately came across one bug in my testing. My BitCoin price widget died in PHP 8.1 with " file_get_contents(): SSL operation failed with code 1. OpenSSL Error messages: error:0A000126:SSL routines::unexpected eof while reading ..." There is a fix moving down the PHP pipeline, but I fixed it my using cURL instead of file_get. See the specific version of my BTC widget. To confuse the issue a bit, I had another problem to solve, too.

For the record, here is the GitHub listing of the OpenSSL problem. Last I checked, this appeared to be fixed in php-8.1.7RC1 dated May 24. I am current for Ubuntu, and I am still on 8.1.2.

So here is a checklist of items to do / test / consider as I continue my testing, before making the new 18GB Ubuntu 22.04 block live.

• rsync the local version of my DOCUMENT_ROOT tree
• git pull /opt/kwynn from my general utilities
• test my clock. Usage seems to be down, but people still use that, I think.
• test chrony and the accuracy of the clock
• test the redirect / rewrite for my resume
• route a temporary URL with SSL cert
• add that URL to my email check widget so I can test that
• test my own other "favorites"
• test the numerology calculator, I suppose, but that is very unlikely to break (It's all client-side.)

I started this entry and then cut if off before its time. I'll probably come back to it.

June 6, 2022 - Mr. Zoom continued, and Mr. D. (started June 2)

Update: On July 3, I'm rewording a point several paragraphs down. I'll mark the point below.

My entry on the 31st was a rant on various problems I have with potential clients. I did feel better. It brought me some sense of completion, even if the completion is to pound deep into the ground why I should avoid certain potential clients.

I suppose the difference between my personal blog and tech blog is getting thinner, at least for the moment. Obviously I may / will alienate some people. Hopefully I can filter out nonsense and come out with a few people I can work with.

May 30, 2022 - processing a potential client

back to the original post

"Processing" as in processing their emails in my inbox, for good or ill. I suppose this can be in my tech blog because this is about the acquisition of clients, or miserable failure to do so.

I'll back up a bit. I have known for years that I'm not cut out to be a solo freelancer. I don't have the people skills. As I remember, the whole freelancing thing started out relatively well, but then I got hammered with a series of "divorced, beheaded, died, ..." After being bitten a number of times, my situation has been, to a degree, a downward spiral--a positive feedback loop. I'm afraid I'm going to get bitten, and it's self-fulfilling. (Yes, the engineering term positive feedback loop is correct.)

The "real job" path resulted in more serious bites. The major reason I attempt to freelance is that it's 10:10pm (and now 11:30), and I'm early in my work "day." The latest attempt to solve the night owl issue in the "real world" went very badly. I figure I was operating at 30% capacity, but the joke is that my boss wanted to keep me. However, I was so exhausted that I wasn't thinking clearly. Also, I had been set up to fail by the recruiter. He's supposed to be the people person, so I took his advice. I was then so angry at him that I wanted to make sure he didn't get paid, so I quit. I can't remember how much of that story I've posted, and I want to limit digressions right now, so I'll stop that thread for now.

LG T-Mobile (MetroPCS) "Unfortunately, LG IMS has stopped" - May 23, 2022, 20:07 (updated after 20:24)

This started showing up Sunday, May 22 around 10am my time. I didn't encounter it until 5pm or so yesterday.

I finally fixed this with several variations on what's out there. I have an LG Aristo 2 with Android 8.1.0. I tried the instructions of clearing data, cache and force stopping 1 of the 2 apps I list below. After several attempts, that didn't work. Below is what I did that wasn't in the instructions:

• I cleared the cache, data (for 1 of the 2), and force stopped BOTH "LG IMS" AND com.lge.ims.rcsprovider (I cleared this com.lge... first). There are separate entries for each app. (The "LG IMS" app will not let you clear data, but that did not matter. 3 dot menu / hamburger menu "Show system" lets you see the system apps. Clearing cache and data is under "Storage" for each app.)
• I had the SIM card out (removed) and was in airplane mode one of the times when I did the clearing / force stop step. Then I'm pretty sure I turned the phone off, re-inserted the SIM card, turned it on, got the error again, did the above step again, and then pulled the SIM card out with the phone on to force a reboot (see below). Then the error went away. After roughly 1 minute to confirm the error was gone, I think I turned the phone back off, then put the SIM card in, then turned it on again. I made a call to check phone service, 4G LTE shows connected, and I have not seen the error in roughly 30 minutes, which almost certainly means it's fixed.
• For the final reboot, I removed the SIM card while the phone was on and in airplane mode. Within roughly 15 seconds, Android gave me a message about no SIM card and then rebooted itself. Previous reboots had failed to fix the problem, so forcing it to reboot due to no SIM card might have been the difference.
• I took the battery out for about 40 seconds before the previous-to-working boot. That is unlikely to have made a difference, given that it was a previous boot, but I've heard of that solving such problems in the past.
• From the time I put the battery back in until I confirmed the fix worked, I'm pretty sure the phone was in airplane mode.

Updates, 20:24

I just noticed that some people are saying it fixed itself after a reboot, so it is possible I didn't do anything useful except reboot 7+ times. On the other hand, I am reasonably sure nothing updated itself. At a glance, though, it's hard to figure out timestamps on Reddit, so I'm not sure when that was.

For sake of documentation, I started from this post on "Android Police." Also, here are a handful of Reddit posts: 1. Megathread, 2. "Unfortunate" thread, 3. wake up call thread.

To repeat my "apology" in the 17th Century sense of the term, the only reason I linked here in my Reddit post was that when I tried to copy and paste my pre-written instructions (as above), my text kept doubling and trippling in the Reddit HTML box.

April 27, 2022 (00:07)

I have started an Ethereum / NFT project.

April 11, 2022 (01:14)

It's been an interesting few weeks. There is a lot to report, but not sure how much I'll get out now. My immediate purpose was to record for myself and whoever finds this the details on the MongoDB 5.x CPU requirements. For Intel, it needs the "Sandy Bridge" (micro) architecture. WikiP tells us this came out in 2011 with the Core i3.

It would appear a lot of people didn't get to the bottom of this. Fortunately, I figured it out fairly quickly and reverted to 4.x. Here are some of the error messages I see, in case Big Evil Goo picks it up, and this helps someone. I probably figured it out because I'm well aware I have ancient hardware. It's about 2009 vintage. It was a $3 - $5k computer at the time, I'd imagine. I got it for something like $160 in 2017.

(core=dumped, signal=ILL)
status=4/ILL
core-dump
core-dumped
Illegal instruction (core dumped)
kernel: traps: mongod trap invalid opcode
mongod.service: Control process exited, code=dumped status=4
mongod.service: Failed with result 'core-dump'.
dumped core

March 18, 2022

web server access log analysis - README redone

In answer to "usual" (recent CS-grad) apprentice, I rewrote the README of my web logs repo.

create a git branch

the right way

I found the note file where I saved the commands, but it took me way too long to figure out what I had branched. "grep -R branch | grep -P "0\.5"" It was my cms. Note that the comments from further below on "master" versus "main" and label versus branch apply.

git branch 0.5
git checkout 0.5
git add -A .
git commit -m "trying to create branch 0.5"
git push --set-upstream origin 0.5
git checkout master

the long way / sort of wrong way / correcting mistakes

The following were the actual commands where I went around in a circle. It did the job, but it's not the perfect commands. I'm "saving" this here because I'm removing it from the README (see link below) . Note there is ambiguity between the words "main" branch and "master" branch. The libtard speech Stasi deemed "master" inappropriate. Git seems to be in the middle of transitioning both its gender and use of those words. I don't remember which is the right word as of that version or the current version. Here is the link to the branch I created. In hindsight, a label rather than a branch may have been more apt.

Note on branching:

The key is to create the branch and THEN to change the branch.  I think you have to commit it.  Make sure it shows up in origin / on github.  

git branch 0.32
git checkout 0.32
git add -A .
git commit -m "trying again to create branch"
git push --set-upstream origin 0.32
git checkout main
git add -A .
git commit -m "removing all from mai[n] temporarily"
git checkout 2a7231bda956def5e205e910062b7f3f4b23c046 cli/t1.php
git checkout 2a7231bda956def5e205e910062b7f3f4b23c046 README.md
git checkout 2a7231bda956def5e205e910062b7f3f4b23c046 parse.php
git add -A .
git commit -m "new main or master branch"
git push

March 17, 2022 - web log introduction for "sales guy" apprentice (started ca 22:40, updated: 3/18 17:03)

Note that on 3/18 00:15 and possible later, I am adding stuff at various points. It's not always the furthest down is the newest. At 17:03 I added one last note just below, and now I will almost certainly close the entry.

One more note, out of order. The line below shows a fetch / "GET" of my apprentice page. I should not assume that's obvious.

Below is a line from my web server access log. I'm going to separate it into 2 lines for page aesthetic purposes.

66.249.70.62 - - [17/Mar/2022:16:33:55 -0400] 689777 "GET /t/9/02/apprentice_steps.html HTTP/1.1" 200 3646 "-" 
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 

This is what I've been nattering on about and built a vast edifice to process.

The lines represent every page and other object query to a site. That is, the fetch of the .html page, all of the JavaScript files individually, all images, all external css, etc. That particular line is from the Big Evil Goo Bot. That's how Goo builds its search data: its robot queries pages, runs them through the wrongthink filter, and then maybe indexes the page for the search engine, if the page doesn't threaten various of their sacred cows.

The first field--66.249.70.62--is the IP address of the robot--where the robot is "calling from." I picked a line from the Goo Bot so that I could reveal the IP address. I feel much less than zero moral obligation to protect the privacy of their IP addresses. If you run "whois 66.249.70.62" without quotes from the Linux command line, after perhaps installing whois, you'll see that Google owns that IP address. There are also web services for whois.

The next 2 fields "- -" are for if a user is logged in with Apache's ancient password method that very few use anymore. At least, one of the "-" is for that. (Apache doc.) As I think about it, I should probably remove those from future logs, but not now. Anyhow... They are always "- -" in my case, which is why I should consider removing them.

Next is the time with GMT / UTC offset, set to my local time. Next is where I added microseconds. My EXACT log format is displayed on GitHub. The "%u" is where I add microseconds. This goes to my obsession with unique index methods. I was trying to uniquely define lines, but it didn't work. There are HTTP "408" response code lines that can happen in the same microsecond, or at least are logged as the same microsecond, so it doesn't work.

the "ancient" Apache passwords

Sales guy apprentice (SGA) asked about the ancient Apache login method. I call it ancient because I read about it and used it briefly years and years ago, and I would guess it's much older than when I used it. For reference, here is the password feature in Apache's doc.

SGA asked for clarification. Their password feature allowed you to create a password in a file on the same server as the web server. When you say "their side," Apache as an entity was not involved. That's not one of the "sides." It's "their side" in the sense that the password file was on the same side (server-side) as the web server.

I think you can integrate their password into a database, but I'd imagine that was / is clumsy. That's one reason it wasn't / isn't used. Another is that I have no idea to what degree they kept up with password hashing. The PHP language has built in more and more sophisticated hashing, for example. I don't remember what the permission issues were around the password file, which is another drawback. I could probably think of more. I'll just say that putting user data in a database is just how it's been done for decades, and this is one instance where I have no objection to "how it's done."

back to log files

Back to the log file, I have debated removing microseconds because they don't do what I want, but, then again, there is very likely information to derive from the microseconds, so I'll keep them.

The next field is "GET /t/9/02/apprentice_steps.html HTTP/1.1" GET is one of a handful of "HTTP request methods" or actions or verbs. I have not crunched the data on it, but I believe it's safe to say that GET is by far the most common. GET is what the browser (or bot) uses to get a page in most cases. Then "HTTP/1.1" is the HTTP version used. I suspect that all my requests right now are HTTP/1.1. HTTP 2.0 exists but is early in its support, last I checked. HTTP 2 goes to a binary rather than human-readable text format, which is surprising. I am not excited about a binary format, so I am in no hurry to adopt. As of now I see no technical pressure to adopt.

200 is the HTTP response code, such as listed / linked above. 200 is the famous "200 OK" It is probably / hopefully the most common response. Although it would be interesting to crunch numbers on that. It probably is the most common, but there are quite a few hack attempts that result in "404 not found." The 404s might be 5 - 10%, as I think about it. I call it famous because I know someone with a bumper sticker "200 OK," so it must be famous, right? To elaborate a bit, the 200 in this case means that the page exists and is accessible / has proper permissions, and it was served up successfully. (Successful in that Apache sent it. It doesn't mean it was received, although it likely was.)

3646 is the number of bytes returned. When I come back, I'll compare that to the HTML on the disk. "$ ls -l /[document_root]/t/9/02/apprentice_steps.html" shows 7984. If do control-shift-I (captial I / India) in a browser (Firefox and others) and go to the "Network" tab and refresh a given page, you'll often note in the "Response Headers" an entry "Content-Encoding gzip" Gzip is a compression algorithm like the old WinZip / .zip. So the 7984 compresses down to 3646. I don't remember if that number includes the size of the headers.

"-" is the referrer, or the site that referred the browser to that page. For example, sometimes I see "https://www.google.com/" which means someone found the page from Goo Search. Sometimes I'll see referrals from DARPA LifeLog (Big Evil Facebook) or Goo's "Tube." There are also internal referrals such as JavaScript pages being "referred" from the HTML page.

"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" is the now-infamous-to-me "User Agent." "User Agent" is an odd term. It means the program that sent the request. In this case, it's the Goo Bot. I list lots and lots of user agents elsewhere.

return to Apache's passwords

SGA is curious, so, here we go. "$ htpasswd -c /var/www_ex/apache_pwd_ex sga" creates the password, with user sga. I type the password as I create the file entry. The super-secret password is "password123" without quotes. The "protected" directory / file is here. I'm following the instructions as linked above, and I'll link again. I created www_ex for this purpose. You want to create a file outside of the Apache file tree (DOCROOT / document root) because you don't want Apache to serve the password file. Apache (www-data user) needs permission to read the file, though.

March 5, 2022, starting 13:04, winding down (maybe) at 16:26, with many long breaks in between, maybe posting 19:06

"PHP Fatal error: Uncaught Error: Call to undefined function mysql_connect() in /...path/[implied index.php]" You are using PHP 8.x, right? Or at least 7.x? Have they already aliased the 1990s "mysql_connect()" to be mysqli_connnect()? (MySQL goes back to '95.) The PHP documentation does not indicate that. In any event, I would use mysqli_connect(). You'll give old-timers like me hives just looking at the thought of the old mysql_connect(). Note that the "old" mysql_connect() is gone, per the doc.

I ranted about this several weeks ago. CERTAIN CLIENTS who sent me screaming were still using the original mysql... functions in PHP 5.x. The original mysql functions were deprecated in roughly 2013 and removed from the language in 2015. (The years are from memory.)

apt-cache policy php-mysql
php-mysql:
  Installed: 2:8.0+82~0build1

This leads me to believe that mysql_connect is an alias, but, still, for my peace of mind, if nothing else, consider using mysqli. If you want to do some research and see where it's documented as an alias, I'd probably live with it, but, given that the documentation still shows deprecation, I would avoid the mysql functions.

With all that said, I just got a reply that he's using mysqli now. Good.

Now, back to the error itself. I'm not sure when (or if) it became relatively clear to me that it meant you had to install something separately. I've had variants of this drive my nuts fairly recently, though. I'll come back to that.

For future reference, consider the difference between the mysqli doc and substr() (or string functions generally) doc. The mysqli doc natters on at some length. That's a hint that something needs installing. I would probably not recommend reading all that nattering. Given that you're reading the general PHP doc, it may not be helpful towards the simple question of "How do I install this in Ubuntu?" Also, there are all sorts of references to "--with-mysqli" and related switches that I have never needed to worry about because I'm not compiling PHP from scratch. When you install php-mysql, it does the same thing as the switch.

It is harder than it should be to know which package to install in Ubuntu. Big Evil Goo usually answers it easily enough, but it's the sort of thing that may be quicker when I'm around to show you what I have installed.

By contrast, the string functions say, "There is no installation needed to use these functions; they are part of the PHP core."

One variant that drove me nuts was when one of the MYSQL(I) constants was not defined. I didn't know if that was a PHP package thing or a Stoopal thing. It turned out that I hadn't installed mysqli. That was not obvious to me.

A worse variant is when the rewrite engine isn't enabled. The manglement (sic) systems should go out of their way to check that it's working. I wrote code to do it fairly quickly, so they can. Otherwise, you get the damnest errors. I might have spent 3 hours fussing with one of them until I finally realized from relatively subtle signs, using the debugger, that the lack of rewrite engine was the cause. These issues are some of the many issues I have with manglement.

On an apprentice procedural note, my theoretical standard for something like the error above is to spend 20 minutes thumping on it. Then ask me. It's not worth that sort of frustration. I'm not sure you learn anything from it. I've said it before, but it's worth repeating: I remember spending 3 hours in 1992 trying to figure out SYNTAX errors. It would have been lovely to have someone point them out. I don't think I learned much from all that. I just got very frustrated and probably would have gotten out of dev if I weren't so damn stubborn. I think I've told the story here of someone who did quit dev in 1985 because his compiler simply said "syntax error" with no line number or hints or specific syntax violations.

In 1992, I sort of kind of learned to break code into smaller and smaller pieces--isolate the problem more and more tightly--to figure out where the problem is. It would have been nice to have someone state that in the context of my problem as a general rule. I sort of learned it, but I'm not sure it really stuck for a while. I think I've complained before of people who post to StackOverflow with 100 lines of code. Their problem is within 3 lines, and they should have isolated that before asking the question. That applies to StackOverflow, though. In the case of asking me, I'll go with the 20 minute rule for now. And maybe that should be 10 - 15 minutes.

To recent CS grad apprentice, do you want me to link to your blog? I'll do it if you want, but... I suppose I can ask this openly because it goes for any apprentices. You may have noticed that I'm getting more and more, shall we say, testy, on this website. I'm getting closer to the point where I say publicly what I might have only said privately in the past. I stand the risk of eventually being deplatformed or condemned by the ADL--those being about the same thing. I am at risk of being put in the same category as Putin and Foreign Minister Lavrov are right now. So, do you want to be literally linked to (or from) me? I would not be at all offended if you said "no."

In other news, CS-g apprentice wore a hoodie to a recent job interview, and even "swore a little bit." He is doing an experiment in being one's self. Maybe there is hope for me, but maybe not. He's not a fan of Putin and Lavrov. I'm not trying to be insulting, but I doubt he has consciously heard of Lavrov. Indeed, Lavrov has likely been somewhat obscure until the last few weeks. I know of him because friends who keep much better track than I do have been a fan of Lavrov's for many years. I didn't know until just now that Lavrov has been FM since 2004, so I've heard about him for quite a while.

Russian military fandom aside, I suppose my equivalent would be barefoot for an interview. Decades ago I heard a reliable report of an office in Orlando being commonly barefoot. Actually, I would be wearing my 2009 Holy Name Cadets Drum and Bugle Corps hoodie and barefoot during a light snow.

Anyhow, where were we?

You mentioned PHP and (client-side) JavaScript and loosely implied that learning them was in conflict, or that server-side (PHP) code had much higher priority. Many weeks ago I talked about the potential for 4 layers of code filtering from the database to the front-end. PHP and client-side JavaScript are part of the same team to process web data. Sometimes it makes sense to do most of the processing on the server-side and sometimes on the client side, and sometimes it isn't clear which is better.

Things are processed on server side (whatever the code / language is), and then the result is then sent over to the client. The client doesn't see any code, just the output. You'd think i would already have known that but man it didn't click until a couple days ago.

Now, i'm guessing JS is processed on client side?

Correct on both points. More specifically, client-side JavaScript, which is what you're talking about in context, is processed in the browser. That's why you see it debugged in the browser's dev tools, and you can't see the JS execute on the server-side.

Which leads me to Node.js. Node.js is server side JavaScript. Just in the last few weeks it's finally started to fully sink in WHY the MEAN stack goes together. It's somewhat of an equivalent to your "revelation" above, but I don't have an excuse for not having my revelation much sooner. An apprentice guideline, to repeat, is that Kwynn is not all-knowing, and I suppose I can sometimes be kinda dumb.

Actually, let's leave Angular and Express aside. Angular is a client-side JavaScript library, so the JS part matters. MongoDB's equivalent of SQL is JSON (BSON) formatted. The distinction of what exactly it is--JSON for BSON--isn't at issue for this limited discussion. I'll just call it JSON for now. In any event, queries including data entry are JSON formatted. A query result is essentially a JS object with some portion of the JS language to operate upon it. I don't yet know the extent of JS in Mongo objects.

One point being that I've been doing all sorts of machinations to harmonize PHP and Mongo. My life might be easier if I just learned Node. To this end, I installed a Node debugger early this morning. Let us all repeat in chorus, Kwynn's Rule of Dev #1: never dev without a debugger.

I spent 2 - 3 hours researching various debuggers, re-installed Eclipse, and tried getting Node to work in Eclipse. So there's Rule #1, and then there is my dedication to open source tech. So, when those two come up against my disgust and perhaps hatred of SatanSoft, it seems that I go with open source and Rule #1. After resiting it, I installed the MIT / open source licensed Visual Studio Code for Node. It works. Perhaps I should set some apprentice or another on finding a non-SatanSoft product.

"You keep saying don't code (such as php) without a debugger. How important is this over say doing 'php index.php' and having it give you the errors at the terminal?" They are not equivalent. Running PHP as CLI is different than using a debugger. There is no substitute for a debugger. Just to be clear on terms, by a debugger I mean software that allows you to set breakpoints, step through code, and watch variables as you step through.

I should explain why they are different. First, I'll back up and discuss CLI PHP versus "web" PHP. Remember that those two types of PHP have separate php.ini files. Also remember that if you change the /etc/php/8.0/apache2/php.ini, you have to restart Apache for the change to take effect. Anyhow, for dev purposes you should turn on "Display_Errors" or whatever the .ini variable is for showing errors in web PHP. That is, you should see the same error in both CLI and web.

By default, display errors is off in the web version because in theory you don't want the public to see the errors, because it could give an attacker insight into how to attack your site. Depending on what you're doing, I vote for turning display on even for a public site. I am almost certain display errors is on for kwynn.com.

With that said, one of the reasons why it's helpful to run CLI on a web application is that the error may not show up in the HTML output, even if display errors is on. Sometimes you'll get a totally blank HTML page. kwutils.php changes the way errors are handled, so I'm somewhat confused on this point because I'm usually using kwutils these days. But, as I remember, you get blank when the error is upstream of the page that was GET / POSTed from the browser. I think this is because by default errors go to stderr and not stdout. The HTML display is from stdout. That is, the PHP program "crashes" before there is any stdout.

Similarly, you won't see an error sometimes (even if display errors is on) unless you view source, because the error message may not be proper HTML. There is also a setting where errors are specifically HTML formatted, but, again, I've kinda lost track of this because I handle errors differently.

Anyhow, using CLI rather than web mode to "debug" is a minor point versus using a debugger for CLI, web, or ANYTHING ELSE. But, let me finish on the CLI versus web part, first. The reasons that it's often helpful to use CLI to "debug" are something like:

As above, display errors might be off (although you should turn it on), the error might not otherwise show in the HTML (stderr vs. stdout), or it might be embedded within HTML in a way that doesn't display. There are other reasons. Even if you're debugging, you're going back and forth to the browser, and just the browser being involved at all makes things slightly more complicated. I'll try to think about this. My guideline on this weeks ago was that it's often best to dev in CLI mode until you're ready to bring it into HTML. And it's often best to separate all the processing until you combine the HTML and PHP.

On a related point, don't forget /var/log/apache2/error.log or whatever you may have overridden error.log to. Sometimes very helpful PHP errors and warnings will show up in error.log. And, on a related point, when you turn display errors on, also turn on display E_ALL types of errors / warnings / notices. Occasionally you'll have to hide notices because they pop up in places that are not useful, but I've had to do that 3 - 4 times in many years.

So back to debugging with a debugger. In some cases seeing an error message is all you need. The debugger comes in for situations were it's not working but there are no messages. In other words, a debugger helps solve your logic errors more so than PHP language errors. I think you once asked about echo / print / console.log. Your code will get cluttered with more and more of such if you go that route. Then there is the problem of your code never getting to the output points. I had that happen the hard way when I was trying to debug Ruby without a debugger. I was getting a distorted picture of what was happening because my biggest problems bypassed my "print" (or whatever it is in Ruby) entirely. In a debugger, you're running the code line by line and know EXACTLY what is happening and where in the code.

In other news, I hope I can declare my XOR processing as good as it's going to get in any reasonable amount of dev time. One tentative conclusion is that mongo CLI / mongosh is not particularly efficient at output. Although it's not apples to apples because I was running that output as a "$ mongo" shell command, and running any command through shell_exec() or the equivalent is relatively slow. In any event, I went back to queries being done from PHP and outputted downstream from proc_open(). With 12 hyperthreads / "cores" (6 real cores X 2 for hyperthreading), my XOR processing takes about the same time as one core does to XOR the raw file. Although that is also not apples because I have an old computer versus Amazon's spiffy new ones.

I also did some buffering both to limit the number of cursor queries and the number of fwrite()s. It appears that my entire CPU capacity pegs for somewhere around one second. Perhaps that's the best one can get, as opposed to the CPUs waiting on RAM or disk. There were some indexes that help a lot. Of course those same indexes slow down inserts. I need to whittle down my indexes and try to find that balance.

The Mongo .explain() is helpful, although I'm not used to how it works versus relational. It was making some odd index choices that I had to curb. I simply deleted the index it was using and added another one that improved the situation dramatically.

Back to the email series. As I said, messing around with data types in advance in a database table is one reason I switched to MongoDB. In Mo, you simply chuck an array or object into the database.

The number in VARCHAR(30) is the maximum number of characters--in this case, 30--the field can hold. The number is up to you. How many bytes are in each character is a separate issue. In Mongo, you don't have to worry about this stuff. What is UNICODE up to now? Is it up to 8 bytes? Once upon a time, a character was a byte. The modern issue with VARCHAR and related data types is that one byte doesn't remotely cover all the characters in all the languages on earth. You need a set for Cyrillic, Mandarin, Hindi, Arabic, etc. You can restrict your database to the "original" characters, though.

This issue might have cost me a lot of money. I was once offered a 30 hour project, with possible extension. I was loaned a laptop with a working MySQL or MariaDB. It was enough years ago that it might have been either one of them. My KVM connectors and such were buried at the time. (I'm not sure if I could get at them now or not.) A laptop keyboard is almost useless to me. I couldn't get the database to load on my desktop. Given the nature of the restore program, the specific error was not easy to figure. I finally found out that it had to do with UNICODE. The other dev had left an email address as 255 characters. It was something that did not have to be anything close to 255 chars. He had an index on that field. His database was set to Latin-whatever-number-it-is or some such. By default, my database was set to multi-byte UNICODE, and I think it was 4 bytes at the time. The problem was that MySQL or MariaDB didn't allow an index on a field above something like 768 bytes. Not chars, but bytes. So it wasn't allowing the table creation with unique index. Now that I think about it, I didn't see the error because the import did not terminate. It just kept going after failing to load the test email addresses. It took me a while to set things up to see the error properly.

I had already been questioning whether I wanted to use relational again, and I wasn't sure I wanted to deal with Laravel, either. By the time I got the db loaded, I decided to walk away from the project. If he had given my access to the data file and not loaned me a laptop, I probably would have solved it. I kept going back and forth from desktop to laptop, though, and everything about the laptop was painfully slow. One problem was I didn't have a good place to put the laptop. It was a weird combination of events. It's one of those "What ifs?"

March 3, 2022 - commutative hashes / XOR by line (begin 18:45)

To quote Colonel Hannibal, "I love it when a plan comes together." (Although Captain Reynolds naked in the desert--"Yeah. That went well."--that might be yet better.) I suppose, while I'm at it, I should mourn the loss of General Hannibal of Carthage. The world would likely be a better place if Carthage had won, but I'm not sure they stood a chance under the circumstances. My vague memory is the world might have been a better place if Carthage had committed to total war. Hmmm.. There is discussion on the "What If?" and it is likely infinitely more honest than equivalent situations in the last 100 years. Might make for interesting reading. But I digress.

So I have been musing over this issue of quickly validating my web server access log database versus the source file(s). One problem is that all the hashes I can quickly find are linear in the sense of a VHS tape--they are only useful with one long string. They can't be parallelized. So I got the notion of XORing each line and then XORing the lines together. That is parallelizable. (That may not be a word, historically speaking. It is now.) I started in PHP. Right this moment, I don't remember why I went to C. At first I was concerned about 64 bit unsigned versus signed. That may have been it. PHP doesn't have an unsigned. Then I, dare I say, "circled back" and used signed in C. (I use "circle back" in mockery.)

In any event, I now have XOR working as planned in both C and PHP. (The code was in GitHub before I started writing this.) C and PHP match both forwards ("$ cat") and backwards ("$ tac"). This heavily implies they will match with the lines in any order--thus, parallelizable. C is 100+ times faster than PHP. I'm not sure I've ever known that the situation was quite that "bad." Our hardware is just stunning these days to create a situation such that I don't know that.

March 2, 2022 - .php versus .html (start 23:03)

My apprentice is about to bring MariaDB (MySQL fork) data into a web page. That is, he's going to try. I'm sure he'll get it fairly soon, but that's the sort of thing that doesn't go smoothly the first time you ever, ever do it. With that said, he may have accomplished that particular "Hello, world!" already.

His page was index.html. He couldn't get any PHP to work inside a .html file, which is not surprising. By default, Apache won't execute PHP inside a .html file. In almost every case I've seen, PHP runs from a .php file. I think there are a small number of extensions that will execute PHP. I *think* .tphp or .phpt will work, where that's a PHP template, but I'm pushing my knowledge / memory. Remember that included (required) files are already in the PHP context or they won't work, so you could call an include file anything, although I can think of very few reasons not to stick with .php. That is, an include file is only meaningful if it's coming from a PHP context, so there must be a .php file somewhere down that call stack. Similarly, if PHP is file_get[ting]_contents() a file, it can be called anything.

So the short answer is change it to .php. I have never done anything different. You asked about speed. I'll address that further below. There are some interesting issues around that, but they are all very, very minor relative to what you care about right now.

He asked whether it is possible to run PHP from .html. I'm fairly sure you can override something in Apache, but I don't think I've ever seen it done.

With that said, some sites suppress .php. I just tested and found that one of the systems that shall not be named will let you do example.com/index.php . I find that the most "popular" system that shall not be named will not. ("WordPest" is not strong enough.) It will redirect (301) /index.php to / I think the redirect is done within the bowels of the application and not in the Apache .htaccess, but I'm not entirely sure because I'm not fluent in rewrite rules. To the extent I understand them, I put it at a 70% chance that it's done in the "bowels."

I'll call the "popular" one WordPlague. Word pest, word pestilence, Word Plague, or simply The Plague. That works. I'll call the other one Stoopal because it will cause your development skills to stoop until you're dependent on the cane / crutch of it. You'll never be able to walk properly or run after too much use.

In any event, as best I understand without digging too much, often you don't see ".php" because all of the URLs are written with the assumption of a "single page" application and / or other redirects / rewrites. In both the cases of The Plague and Stoopal, most HTTP requests are rewritten and sent through index.php. index.php in turn starts a series of includes and conditional includes that processes the request. This is a case where looking at the superglobals in a debugger would explain a lot, but I'm not sure I can bring myself to care much about the content manglement (sic) systems.

You asked about a speed difference. That is not worth a second of thought or hesitation. I will delay renaming .html to .php until I'm sure using PHP directly in the file is the right answer. That's a matter of clarity, though. It doesn't make sense to use .php unless you're specifically using PHP.

Months ago, I might have leaned towards .php to leave myself the flexibility, but I'm getting better at rewrite rules, so I'm not as worried about that. In fact, I did a rewrite from .html to .php recently. It's very profound, isn't it? Note that using .htaccess files in various ways requires a specific "AllowOverride" in the virtual host's .conf file. All that stuff is in my sysadmin repo. If the machine in question is yours, I don't see a problem with very loose AllowOverride. The potential problems come with shared hosting where you are the host.

To do a fair test of a speed difference, you would want to run precisely the same content with .php and .html. You can set your /var/log/apache2/access.log to show microseconds. I've done it, as should not be surprising. The example is in my sysadmin repo. If you call a page twice using the same method, it would give you an idea. Or call it 1,000 times.

I should add that I've given some thought to turning microseconds back off. I was hoping they would provide a unique line identifier, but they do not. A 408 HTTP response (request timeout) will show up twice in the same µs. As best I understand, a 408 is Apache not-so-politely telling a browser to shove off and disconnect. That's all deep inside Apache. I'd have to dig to start to understand the context. In any event, those two lines are completely identical including the microsecond.

In any event, I'm not sure you would detect a time difference. If you did, it would be way too small for human detection. It's just not a consideration. There may be situations where you want to load some of your data with AJAX after the page has loaded, but that's a mostly separate issue. You're asking about very similar files as .php versus .html. You're also asking about tiny amounts of data pulling from only slightly larger data into a page. I see no problem pulling it directly from a speed perspective.

I have never tried to detect those sorts of speed differences and have not worried about that. There is an interesting consideration, though, between .html and .php that has been in the back of my mind for a very long time. When Apache serves an .html file, the Date-Modified in the HTTP reply header is the date of the file modification, and the ETag is based on the contents. That makes it easy for Apache to serve up a 304 response. 304 means that the browser sent an "If-Modified-Since" and / or "If-None-Match" the ETag. The browser is reporting what it has in its cache. Apache serves the document if it has changed or else send back "304 Not Modified.

Contrariwise, if the exact same content were in a .php versus a .html, I don't think the .php has an ETag automatically, and the Date-Modified will be "now." If you want to generate those properly and serve "If-None-Match" and such properly, you have to do it yourself in PHP. I have dug at that a little bit in kwutils.php, but I haven't fleshed it all out.

If it's a borderline case between .html and .php, I consider it obnoxious to leave off those features. It's been one of my hesitations in creeping towards a "single page" website. I see the merit in single page. That is not one of my issues with manglement (sic) systems.

more immediate issues

I'll try to wind this down. I'll remind you again that there are whole sections of this blog--perhaps 70%, perhaps more--that I don't expect to make sense right now. I'm recording thoughts for the long term. I'm not sure there is a point in reading it over and over. Perhaps once every 6 - 10 weeks at the rate you're going. Your rate is fine; I just don't want to waste your time. Perhaps we should make an interactive application to help you track where you are with each subject.

As for your comment, "I feel like it's gonna start coming to me really fast, really soon." It has been my guess that there are parts of the learning curve that are exponential or perhaps x^3. But then you are pretty much guaranteed to be banging your head on various new issues, unless you ask for help (hint hint). Yes, I think quite a few things will start to make sense soon.

March 1, 2022 - continuing from yesterday (starting 21:07)

The credit card processing is now a very few steps from live. One big accomplishment is that I have for the first time used Apache virtual hosts exactly how they were meant to be used. I've used the "VirtualHost" tag a zillion times, but I've never set the same machine (and IP address) to serve two domains. I've never had a reason to before. It's sort-of-kind-of as easy as Apache's very first example. I give some "gotchas" below. One criticism I would make of their example is that www. should work via the "*" DNS A record. There should be no mention of www anywhere in DNS or Apache config or anywhere else. That's so 1990s. Oh wait... It's not that simple:

I would about to say that www.kwynn.com works just fine, but given my recent http to https redirect, it does not work just fine. I have the RedirectRule set to preserve the www or anything else. The routing works fine, but given that "www.kwynn.com" does not exactly match "kwynn.com," the security cert process rejects it. I have a solution for that!

/etc/apache2/sites-available/000-default.conf  
# old version: 
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,NE]
# New version: 
RewriteRule ^ https://kwynn.com%{REQUEST_URI} [L,NE]
# end change

sudo apachectl configtest
# Syntax OK
sudo apachectl graceful

Then it works. Ok, with that said, I repeat my statement. "www" is so 1990s. STOP USING IT! STOP REFERENCING IT!

With that said, my client's site still has multiple references to www. I vaguely remember this coming up years ago. At the time, I was too inexperienced and /or chicken to deal with it severely. Right this moment, I don't want to confuse issues. I want the credit card code to be fully implemented. Then one day I will deal with extreme prejudice with "www."

I started the process of VirtualHost "B" by copying a .conf file. I removed the security references to the exact site, but it turns out that leaving the "Include /etc/letsencrypt/options-ssl-apache.conf" was a bad idea when I was just setting up site B. It passed configtest, but it brought down both sites when I activated it and restarted Apache. So no references to security until you get farther down the path.

So, to get VirtualHosts working for both sites A and B, start with a stripped-down .conf file for the new site B. The following worked. Note that before you run certbot, the new site B has to work with http (see gotcha below). That is, the "certbot" process has to be able to access a non-secure version to verify ownership. By "working" that means in part you have your DNS A and / or AAAA records pointed to the right IP address.

ubuntu@ip...:/etc/apache2/sites-available$ more payt1.conf
# the following should be the result once created / edited:

<VirtualHost *:80>

        DocumentRoot  /home/ubuntu/payments_www
        <Directory /home/ubuntu/payments_www>
                DirectoryIndex index.php index.html
                Require all granted
                AllowOverride All
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

ServerName payments.example.com
ServerAdmin buessresume@gmail.com
</VirtualHost>
# end file

sudo a2ensite payt1.conf
sudo apachectl configtest
sudo apachectl graceful
sudo certbot certonly --apache --dry-run

# for real:
sudo certbot --apache

# it's fine to let certbot do the rewrite / redirect from http to https
# that last command does the a2ensite and restarts apache

After that, there is one last annoyance. It's not really a "Gotcha!" but it acts like one, and you could do a lot of wheel spinning if you don't realize what's happening. Now that I think about it, I'm pretty sure the only annoyance comes if you are testing the non-secure http when any https sites are involved. Firefox and presumably other browsers will be assuming a previous 301 permanent redirect from a given IP address to a URL. If you try to go to the non-secure site B and maybe the secure site B, you'll get redirected to the original site A. The solution may or may not be as simple as going directly to the https URL of the new site B. My solution was to use Brave instead of Firefox. I hadn't been to either site in a long while on Brave. After 10 - 20 minutes, the https version of site B worked fine in Firefox, but I'm not sure if that was due to some sort of timeout / expiration or because I went to the secure URL.

Next is another solution for 301 problems. Note that in the Control - Shift - I "Network" tab you can see the 301 redirect. Anyhow, another solution in Firefox: go to history, then hover over the site in question, then right click and "Forget About This Site." You may have to do that for several variants of the site. I don't think that messes with your saved password, but I'm not entirely sure, so keep that in mind.

Feb 28, 2022 - a day or a few hours in the life (starting 17:59; update starting 18:25)

Today / very soon I hope to get back to my main client's credit card processing. That is the main client whose project has always been part-time. I still need more work. I have one source that's looking promising for a bit of work, but it will only be a bit. I've also been getting nibbles for fairly big projects, but we'll see what happens.

For years my client has used the same credit card processor. (I only got involved with that side of the project very recently.) Apparently it has worked fine, so that's much of what matters. They are not a big name, though, and their documentation is not public. Non-public documentation is generally enough reason in my mind not to use a provider. In fact, I was heading towards suggesting this to my client. He created a login for me, though, and the documentation behind the login wall was quite reasonable. Their PHP example was straightforward and worked right out of the box, which is more than I can say for many such situations.

I have whittled on the example and am approaching an end to that phase.  "Whittled" means I confirmed that I can assume a US billing address and whole-dollars only (no cents).  I removed the fax number field (!?!?).  And I'm otherwise cleaning up their code.  It was plenty good enough as an example, but I want it to be cleaner.

The immediate task is to improve their final result handling.  The example simply dumps 1,000 characters of raw XML, which is not exactly comforting or useful to shod muggles.

I have never worked on the site that will use the credit card processor. The site is in the much-despised Drupal. I more-or-less flatly refused to touch Drupal anymore. What I did is took one of his pages and saved it to disk. Then I whittled out all the stuff specific to that page so that I could turn it into the payment page(s). I cleaned up the HTML to some degree, but it was deemed not worth the time to clean it completely. My plan is to put the form on a subdomain. I made my "new" page pixel-perfect such that no one will notice from the page itself that they have moved to a subdomain. Given that it's a subdomain, I can stay away from damned Drupal.

I should also mention that the example and thus working version is a different technique than my PayPal form. The PayPal form is meant to be very "simple" to implement with front-end JavaScript only. It accomplishes that, but I don't really like that format. Neither does my client. In the case I'm working on now, everything is on the back end, so I have much more and easier control. I am almost certain PayPal can do it that way, but at the time I was trying to do it the quick way. Given that their sandbox was having major problems, though, I probably would have been just as well off to do it in the back end. Even if the sanbox was having problems, the nature of the problem would have been clearer from the back end, too.

update, starting 18:25

For years my client has been using a 3rd party's form to interact with the credit card processor. When he sends people the link to the form, though, it's usually eaten as spam. I suggested linking to it from his website, but he said that it's dated anyhow, and he wants the form(s) redone. When I looked at the existing form, I was having a lot of trouble figuring out where the money went--who are these people? Obviously it's going to him, but it shouldn't be that hard to figure out from the outside. I couldn't figure out who they were, let alone find documentation. That's when I suggested using someone else. That's also when he said he doesn't like PayPal's "simple" form. And that's when he made me account that I found very useful in making progress.

Even though their lack of public doc annoys me, I was about to link to them here. Their system has worked for years, their example is quite good, and their documentation is somewhere between plenty good enough and near perfect. When I went into the login system, though, I STILL had trouble finding their public site. Someone in fact has registered a similar domain that outranks the real one on Big Evil Goo; the fake one is a God-knows-what probable mischief site. The second result is very likely the real site, but they aren't even using an SSL cert. Given that even I finally started using an SSL cert some time ago, and then I finally did "the redirect" a few weeks ago, I'm going to discriminate. (I chickened out and did the 302 redirect so far. I suppose the 301 is in order very soon, given that the 302 shows every sign of working.)

So I'm going back in part to my original position. I would imagine they will continue to work fine long enough, but I have gotten the impression from the start that it's a dying product. They are supporting current customers, but they aren't looking for more under that brand. I have some loose indication that they are associated with Wells Fargo. Maybe they got bought out and re-branded.

Feb 22, 2022, starting 18:33

The madness has been on me for several days, although both my moon page and my eyes tell me the moon is days (waning) away from full, so presumably the madness is neither lunacy nor lycanthropy. The madness meaning that I am obsessed with various tasks of my pet, personal, non-paid projects.

So my web log loader loads efficiently, and in the last few days the verification is working more and more efficiently. I decided, though, that a 2 - 3 second delay due to ssh making the connection is not good enough. I also decided against a web app because there would be all sorts of permissions mucking between the www-data Apache user and the "adm" group that can get into /var/log/apache2. In the past I solved web access with a tiny C program that uses setuid() / the setuid bit. In that case I wanted C to call a very simple application with no parameters or complexity. This would involve empowering a relatively complex program, so I decided against web access. Also, given that I'm wanting to whittle down from a few seconds, the web path would cause its own delays.

So I have created a dedicated background PHP program / process that listens on a highly secret port number (that is sitting right there on my public repo). I minimize RAM with an input buffer and proc_open() into "$ openssl md4" I am using md4 because a combination of web searching and testing demonstrated it's fastest. In other words, the program sits there already running with the log file already open waiting to md4 hash a section of the web log to compare against what's in my local database.

While I was at it, I refined my password hash interface to more easily create passwords and hashes. The socket listener authenticates the request with 0.02 seconds of password hashing. I want to spend minimal time on that, but I want it to be some time to turn cracking the password into an age-of-the-universe level task.

In short, it's pretty spiffy. I think it's done, but I haven't set it live yet. On second thought, I can probably live with 3 seconds. The app I created is a good model for future "listeners," so it was helpful and fun even if I never deploy it.

Several days ago I set the verification process to fork such that the database md4 and file md4 happen at the same time. I created a new forking class for this purpose that is more specific and thus simpler than the more general fork process in kwutils.

On my part-time, few-hours-a-week paid project, I've had a productive few days in terms of efficient coding. The timecard specification has plagued me. It seems that I now have complex logic whittled down to about 11 lines. It only took me 5 years of intermittent work to achieve that. The overall phase of that project is that I am partially rewriting the application to extract it from Drupal. A quick search says that I've talked about this to some degree, going back to last July.

Only extracting from Drupal, though, I have deemed not good enough. I spent a lot of time making sure my save-on-edit (keystroke) process was totally reliable. When I started dev'ing again, I immediately got 2 cases of false positives--a save indication when there was none. That makes a certain amount of sense given how drastically I have changed things, but I decided it wasn't good enough, so I have dug into a deeper rewrite, which of course led yet deeper. I should have vastly better code when I'm done.

Between what I call Brook's Second Law and Raymond's Corollary, plan to keep rewriting it, because you'll keep rewriting whether you plan on it or not (Brook's [First] Law). What I call the Second Law involves rewriting. What I call Raymond's Corollary is in his classic Cathedral and the Bazaar.

Feb 19, starting 18:52

Going back to my previous entry, the sequence process I had would work fine assuming that the sequence existed before multiple processes would start using it. Last night I put more code in GitHub where I trivially demonstrated the potential problem. If the sequence did not exist before a heavy multi-process load, there would be many failures.

I am usually referring to codename "recent CS grad" apprentice. Two nights ago "sales guy" apprentice read my previous entry. In answer to some of his questions, and to elaborate on this issue:

A real-world example is how to generate a sequential invoice number (1, 2, 3, ...) when there are multiple users who might hit "buy" at the same time. Some physical few bits of some computer somewhere must be dedicated to keeping track of which number the sequence is on. But it gets more complicated than that. The operation to check and then add to that number must be "atomic," or it must be a "transaction" in the database technical sense. An atomic operation is one that breaks or makes a mess if it does not complete all its steps. In this case, the operation is checking the existing number and then adding one. If one process checks and then another process checks and adds before the first one adds, then two customers will get the same number.

At some point, sitting on a toilet (under some conditions) becomes atomic. You have to finish the cleanup step, or you have a mess. In the data world, the data is either wrong or otherwise corrupted if an atomic operation fails and does not "roll back."

If you go looking, the classic example of an atomic operation / transaction is when a husband and wife with a joint account show up at two bank branches at the same time. If the timing of checking the account balance versus giving them their cash is precisely wrong, they overdraw their account. (Look it up.)

My code last night demonstrated the equivalent problem with a brand new sequence. I have a bit of code where if I only do the loop body once, the sequence can fail. If I loop twice, then one of the two iterations is guaranteed to work, and my code bears that out.

Some time perhaps I'll elaborate on this. The point for my geeky purposes is that I finally have a sequence process that will work under all reasonable conditions. As I said in my previous entry, my musing over this issue has led me to all sorts of fun and games. I might have saved myself some time if I had done last night's code roughly two years ago. Then again, I did some cool stuff along the way.

Getting back to the real-world problem, note that Amazon makes absolutely no attempt to give their customers sequential numbers starting (decades ago) with 1. A quick search shows that as of several weeks ago, Amazon does almost 19 orders per second. We are at 1.6+ billion seconds and counting into the UNIX Epoch (seconds since the start of 1970 UTC). That's a 10 digit number. Even if Amazon had done 19 orders a second going back decades before it existed, that's an 11 digit number. As of March, 2021, I have an Amazon receipt with a 17 digit order number.

The format is 3 numerical digits, then 7, then another 7. The first one might be a server number. It would make sense that the individual server would be preventing duplicates.

Feb 18, starting 01:21

I got my PayPal donation page working. For most of my work "day," Paypal's systems were working fine. Based on their system status, it seems they were having problems late on the 16th which caused me problems. Later in my day, the sandbox system died again, so I stopped messing with it. It's working fine.

Today my bigger irritant with PayPal is that it seems very difficult to correlate the "Transaction ID" with anything you can access "programmatically." The transaction ID is what both the payer and payee see, but it's hard to get at. I thought the difficulty was only because I'm using the simple client-side JavaScript button widget. Upon research, though, it's not necessarily easy to correlate with the full API.

As I investigated the API, it was hard to tell again whether PayPal was having problems or I wasn't getting my queries right. There is a transaction history function / API call, but I was either having difficulty formatting the time right, or there is a long delay until the data is available, or the sandbox was having problems.

I did some experiments with an "invoice ID." Both the payer and payee see that, too. I got that working (in the test version) in that I could create an invoice ID, but then there is the question of uniqueness. That question has been plaguing me in various forms for something like 2 years now.

So, my latest sermon on unique IDs... Most of the IDs I've been exploring lately are absurdly too long for an invoice ID; they are meant to be unique in all of time and space. I want the ID to be intelligible to my customers. So that brings me back to sequences (1, 2, 3, ...).

It was in part my uncertainty over getting sequences out of MongoDB that set me on the path towards my PHP extension to get a unique ID from time, the CPU clock's tick since boot, and the CPU (core / hyperthread) number.

So, after all this time, I decided to revisit MongoDB sequences. My fork class is easier and easier to use, so that means I can fairly easily set any task to use all my CPUs / cores / hyperthreads. So the setup is to set all the CPUs "demanding" a sequence at once, in a very tight loop. In my "code fragments" repo, I have posted the code.

MongoDB's instructions (indirectly) on sequences are somewhat vague, but now I can say with a reasonable degree of certainty that they work. MongoDB went to 500% CPU usage (6 of my 12 hyperthreads: 6 cores X 2 for hyperthreading). My 12 PHP threads divided the rest of the CPU usage. That was MongoDB doing a lot of work resolving locking issues. I demonstrated again that it takes a long time to resolve locking issues. That is, a sequence is not an option if the system is going to be banged on. However, if I asked for 950,000 sequence calls, the sequence at the end was precisely 950,000. (I started the sequence with 0 internally; the first call would get a 1.)

When I just "asked" Mongo for the sequence, that took much longer than actually writing rows with Mongo's default _id or my nanopk(). I will try to remember that I can "objectify" my array and use that directly as an _id. I'm almost certain that arrays aren't allowed. I suppose in future editions of nanopk() I should see how hard it is to return an object.

Feb 16, 2022 (starting early 17th)

I was battling PayPal's simple, few-lines-of-JavaScript button for about 5 hours. I first wrote 8 hours because it feels like that and more. Then I went back to look at a discussion I was having, so 5 hours is correct.

I'll come back to my shrieking about PayPal. I got BitCoin and Ethereum working, I hope.

As for PayPal, I was having problems in sandbox mode. I hope that sandbox mode is far more problematic than live. The problem I was having was that the popup window kept crashing / closing. There was a 500 error, but that was rather useless for diagnosis. Even when it worked, various steps could take something like 40 seconds. The solutions, in approximate order of importance, were something like:

• Clear all cookies of all sites on the button page.
• Hard refresh the page (shift-F5 in Firefox).
• In Brave, take shields down.
• It may or may not be useful to remove all cookies from the PayPal popup.
• Make sure you accept cookies on the popup. The cookie prompt can be slow, and I suspect trying to skip ahead of it was part of my problem.
• You'll wait. Perhaps 40 seconds for certain steps, perhaps more.

Here is my release candidate code. Hopefully I'll go live in around 16 hours.

Feb 13, 2022 (starting 18:58, PS started 19:55) - Zoom

I see an ad for someone who wants to learn a given dev language. It's not one that I've done, but I've done 8 professionally and two others without pay, so I can probably help. I tell him just that, and I ask for some source code, so that I can get a feel of the situation. He mentioned he had a bug, so I figured the place to start might be to solve the bug. Given that he's just learning, I can't imagine that the code is sensitive.

So I ask for source code, and I get back precisely, "Can we do a zoom [sic] session to discuss?" This is an example of why I shouldn't be gig hunting without help. Here is the ultra sarcastic version of my response. I have no intention of sending it to the unlikely to-be client.

I understand that the ad was for teaching, but I need to deal with the objective part first. I'd like to see some relevant snippets of the language first. Software dev is not improv; it's not live stage acting. Software dev in itself is not real time like driving. I like time to think about what I'm doing. There is simply not much to say to this person until I see some code and decide if I want to go down the path.

Furthermore, what the hell is it with Zoom? I suppose I'm going go off the "professional" reservation, but the whole world is off the reservation. I'd never heard of Zoom until Billuminati Gates' Satanic Crusade. I can't get excited about the crime of war profiteering because arguably no wars fought by the US after 1815 were legitimate. I was trying to find historical instances of executions for such, but the closest I found off hand was 17 years in prison for David H. Books of DHB Industries, who died in prison in Danbury, CT in 2016. He sold ineffective vests that were not bulletproof. (Oh, of course, the $10M party was a bat mitzvah party. See my personal blog for "the preface" on that.)

Twitter and YouTube and company are so fond of their "misinformation" disclaimers. If Zoom wanted to make the following disclaimer, that would be one thing:

After due process of law, we deem it near certain that Dr. Fauci would be convicted of multiple capital offenses and possibly executed. Until that time, however, we will try to help keep life going with our service.

If they did that, I would probably be satisfied. Off hand, I see no such statement on their home page. Surprisingly, I don't see masks, so that's something, but not enough. The point being that I see Zoom as "Covid" profiteers, and I do get excited about that.

Also, Zoom goes in a similar category to SatanSoft and Satan's Operating System even before SatanSoft was obviously funding a war against humanity. Why on earth do people install proprietary software when there are free and open source alternatives? Even if you were using Zoom 3 years ago, why would you do that?

I asked you for source code. Until I have source code and have studied it and debugged it, no, I don't want to go live to discuss, and I certainly don't want to go live on Zoom.

possible solutions

To my apprentice, no, lurking in the background won't do. As I said, I am not going live until I get what I asked for. One potential solution is that you contact him and tell him we're connected. You can call me autistic or Asperger's or anti-social or shy or even a crazy conspiracy theorist. You can call me quirky and touchy and eccentric or even an asshole. You can say that I am incapable of politely asking for source code AGAIN, so you are politely asking for source code.

Another option is you do what you did with Flutter. At least this time we have a response. You don't have to mention his response to me; the point is this guy is responding. See what you make of the language. Possibly install a debugger. Then see if he'll talk to you. You can make a similar arugment to me in that you've been at it in a number of languages for a very long time. I'm not necessarily encouraging you to take that risk, although as I said in the email to him, R looks like it's worth learning. I've responded to perhaps 5 R ads in the last year or two, but I think this is my first response.

PS - 19:55

I mentioned a checklist. In some contexts, that would be useful. In this case, I've already made my checklist. I want some blanky blank source code. That's the one item on my list.

Feb 5, 2022 (starting 22:57; posting 23:45)

That may be some of the more satisfying dev I've ever done. The code is in GitHub. For reasons that would depress this entry, I corrupted my local database of my web server access logs, going back to late October. When I reloaded the file, it had become 400MB and 1.7 million lines. My loading program ate 2 - 4 GB RAM and was swapping and was depressing to watch the "top" results of and took 5 - 10 minutes. Now it takes 6 seconds and consumes minimal memory. I am not quite comparing apples to apples, but I'll get there.

I'm not quite comparing apples to apples because the old version did the line parsing--separated the parts of a line and give them digital meaning where appropriate. The new version only chops up the file into lines and puts them in a database with enough line number information to reconstruct the line numbers later.

The old version loaded the whole file into memory, so that was one problem. The new version sets as many cores / hyperthreads as one has digging into the file at the same time. The processes are reading the file line by line, so I am not loading multiple lines into memory. More precisely, I may load parts of two lines into memory, but that's it.

Another large improvement was to abstract "insert one" into "insert many." I learned that trick over 20 years ago, and it still works. If you insert a few hundred rows in an array with one command, that is tremendously faster than inserting hundreds of rows with individual insert commands. I created an "inonebuf"--insert one with buffering--such that the user can queue individual rows, and the buffering and inserting is done for them.

I created classes to simplify forking (multi-process / multi-core) perhaps 2 years ago. Now I've put those in kwutils (in the general sense, not the file). They work splendidly as written years ago.

Feb 3, 2022 (starting 03:32)

The moon phases are now in the user's / browser's local time.

Feb 3, 2022 (starting 01:01)

In answer to my disappearance over the last several hours, the lunacy took me again. Another apprentice came online who is in another timezone. So, my fun tonight started out considering adding a timezone offset from JavaScript (the user's browser) so that he would get times in his local timezone. This led to separating calculation from data acquisition (from the Python ephemeris) and storage. That led to cleaning up the various "magic numbers" where I'm trying to make sure to always have enough days of data. I also took my advice to use AJAX: I created a data buffer so that any given user is unlikely to suffer a delay when the almanac (ephemeris) loads. The almanac takes about 0.5s to load, so after every call to the app, there is an async call that makes sure there are plenty of days in the database. The user won't see any of that unless they are using dev tools, although they might see a spin somewhat away from my HTML doc. Do browsers show any spin (spinners) under those conditions? I don't know. I'll see what it looks like over time.

The alternative is to set a cron job. I might do that.

Anyhow, when I had everything working locally, I broke the live system for roughly 15 minutes. A violation of Rule #2 burned me. Locally I'm running MongoDB 4.4. (I can't run 5.0 because my CPU is too old.) Kwynn.com is running v3.6.

The violation of Rule #2 led to my having to seriously bend Rule #1. One can run a debugger remotely, but it strikes me as a bad idea. So I had to go do what is generally a gross violation of Rule #1. My current code in GitHub still has a file_put_contents() on line 29. The error was on the current calc.php (not data) line 41. $ala[0] did not exist. I did the file_put to see the data. The algorithm assumes that the data is in earliest to latest timestamps. It would appear that locally MongoDB was sorting in the order the data was put in the database, which is what I wanted. I could not at a glance figure out what the earlier MongoDB version was doing, but it wasn't in the order I needed. So the solution was to add an explicit sort, which is what I should have done in the first place.

The code is somewhat cleaner now, but I'm not sure that whole exercise was worth it.

February 2, 2022

I have posted my working lunar calendar.

You said, "I desperately need to start adding repos to GitHub in case these clients want to actually look at it." I'm not sure anyone in that context has looked at my GitHub, despite my mentioning it a number of times. It seems the people I most wanted to look didn't respond at all. I doubt that's because they looked at my GitHub and then decided not to respond, although I can't be sure.

I have learned over decades to be very careful what I do for that sort of reason. If you are working on something that does not absolutely have to be secret, then post it publicly to GitHub. Why not? *Maybe* someone will look at it one day. It serves as a backup and version control if nothing else. The effort to go backwards in time to post stuff should be nowhere near "desperate," though.

I have found GitHub to be very motivating, and it continues to be motivating even though it doesn't seem to have served the purpose you mentioned. When I have posted to GitHub, I have "published" in the academic sense, even if there is no peer review and even if no one ever looks at it. It's on the record. I also like the idea of SatanSoft paying for it. If they want to host my code for free, I'll take them up on it.

January 31, 2022 - Earth counters Luna (02:07)

Going back to yesterday's moon entry, I have begun my counterattack against the moon. There is nothing live yet because it's all back end so far. I have the Python ephemeris working. I feed Python output (painfully) into PHP and load it into the database for better sorting and querying. I have all the data I thought I had earlier. This time I've tested it down to the minute, several weeks into the future. I have a notes file with some installation notes.

In Python, SkyField deals with something called NumPy (?) that seems to have a problem serializing a simple array. Specifically, I can't simply json.dump() it. So I preg_match_all() in PHP, which is an obnoxious way to do business. It seemed faster than decoding NumPy, though.

There is a noticeable calculation time with the almanac SkyField function; less than a second, but noticeable. That's not a complaint; I'm starting to get some notion of how complicated that calculation is. That's one reason I'm saving the result in the database.

I'll eventually write code to tell JavaScript when Python should be called again for data days enough in the future. I'm starting to think the easiest way to do asynchronous calls is not surprisingly with Asynchronous JavaScript and XML (AJAX). The alternative is to try to exec() in the background, but weird things generally happen when I do that--either directly or the debugger won't work. One day maybe I'll figure that issue out in PHP. Anyhow, when the data goes "down" to JS, the JS is "told" if it should do an AJAX query back to PHP to update the database. That's another weird way to do business, but, again, it's called async for a reason.

January 30, 2022 - when to collect from a client

This is the 3rd topic today. Should I have a separate blog for the business of technology? Mostly what I'd have to say is what not to do.

Financially and perhaps more importantly psychologically, for a project of any substance, I need some payments before it's done. The project needs to be divided into benchmarks where some aspect of it works--a proof of competence. Also, we can use an escrow service. Escrow.com charges a lot less than what I saw years ago. They have an "A" rating with BBB, last I checked. An escrow service would help, but I would still need interim payments.

Put another way, bill early and often in small increments.

I can go on about this at some length--both historically and for the future. I'll wait for your reply. How big a problem is this for you?

January 30, 2022, one of several entries today - new topic

I had never thought to check, but the "mysql" command is a symbolic link to "mariadb" Yes, you are correct, I should start referring to the mariadb command line as such. I'm sure I'm going to slip, though, and refer to "MySQL." It should be understood that I always mean MariaDB.

Did you set the MarDB password with sudo mariadb? You must have. The logic may have changed over the years, or it might not have. Apparently once you set a MarDB root user password, you have to use that password even if you're Linux root. Maybe you've come across something I haven't, but that seems it could lead to trouble. Is there an override as Linux root? You might consider unsetting the MarDB root password and using specific MarDB users for each database or several databases. I have done it both ways over time. When I set a MarDB root password, I save it in a file only accessible by Linux root in the Linux root user's home directory. I do not have a MarDB root password set on my dev machine. I created a user for "main project."

January 30, 2022 - return to the moon

part 1 - when I still thought it worked or was about to work

That would be returning to the moon in an anti-gravity ship, not that rocket silliness. To quote the Cthaeh, "Such silliness."

The moon phase UNICODE characters didn't show up that night because I hadn't gotten that far yet. It was almost certainly not a mobile issue.

In the following, I'm much more making fun of myself, not you. You asked about "hundredth millionths of a second" precision. Given that I have spent an enormous amount of time on time (sic), I will seriously address your question. I realize you probably meant the point hyperbolically.

Yes, the count from 0 to 1 is in real time. No, it's not displaying "hundredth millionths of a second" precision. It's not that precise for several reasons. For one, pop quiz: what is the refresh interval? The numbers are not blurring to the eye. I wrote the refresh interval specifically not to blur. I want to user to immediately see that the calculation is in motion, but I don't want it to be blurring. The refresh interval is 7 orders of decimal magnitude from what you said.

To continue the pop quiz, what is the unit of the numerical argument to the function in question--the "refresh" function? That's 5 orders of magnitude. If you send an input below 1 unit, will it make any difference, or will the unit be interpreted as an integer? That's a very good question. I don't know the answer.

Also, 1 represents a lunar month. I am displaying 8 digits to the right. If I have the math right, that's displaying down to increments of 25ms out of the lunar month (7 orders of magnitude), but the refresh interval is several times higher than that.

I'm pretty sure the answer is the precision is equal to the refresh interval, but I may have lost track of the question.

Then there is the question of whether I could keep track of hundreds of millionths if I wanted to. In a compiled language, I am fairly sure the raw numbers in the CPU would keep up with real time to that precision, but by the time it was displayed in any manner, it would be several orders of magnitude off. If it were a fancy display using a GPU, then it would come down to that max refresh interval. I understand those are getting faster and faster, but when do they stop and conclude the human eye can't perceive anymore? Or are they going beyond that as supplemental CPUs? I guess the latter, but I'd have to dig.

part 2 - such silliness indeed

Hopefully I am not under the influence of the Cthaeh. On one hand, he would do much worse. On the other hand, it's hard to say what the effects will be.

After mucking about way too much, I did a check against a reliable source and realized that the "lunation" of the moon is rather complicated. I had assumed it was close enough to linear only in the sense that earth days and years are constant down to an absurd number of decimal places. (There are leap seconds every few years.) I thought the lunation period was the same. Silly me. Such silliness. "...the actual time between lunations may vary from about 29.18 to about 29.93 days" (WikiP). Any time I cite the WikiP I must warn that quite a few of its entries in absolute numbers, if not relative, will get you killed if you believe them. I most certainly checked against another source.

As for your second email on the subject, hours ago: like many applications, there is the external goal that you are trying to calculate or track or assist with, and then there is the implemenation. In answer to one of your points, the "technical part" in terms of the orbital mechanics went over my head, too. Perhaps worse yet, I made a silly assumption and didn't even think about or look into the orbital mechanics. With that said, there are probably some decent tricks in my code such that once I have valid input, the processing of that input had some merit.

When and if I ever have valid input, Rob Dawson's "Moon/Planet Phases in JavaScript" code looks very interesting. It would be more realistic than my using a handful of UNICODE characters and opacity CSS.

As for getting valid input, I will hold my nose and grudgingly admit that this is one of a handful of situations where the best code is in, ..., rrrrrrr..., I don't want to type it...... Python. Not too deep in the bowels of my "code-fragments" repo, there is a Python script that uses a powerful ephemeris library.

I would likely write everything I can in JavaScript and PHP and then call Python with the exact data in the exact format that the library needs.

January 29, 2022 - howl-dev-ing at the moon

It became critical to know when I turn into a werewolf. Although, come to think of it, I didn't add that feature. Hmmm... We need a definition. One becomes a werewolf when the moon closest to full rises at one's location? That will take more work involving location--fairly easy--and location-specific moon calculations, probably not as easy, perhaps not so hard, either.

The whole disk of the moon rises, I guess? Seems that a sliver isn't enough.

It should be emphasized that the Quileutes are not werewolves. Edward pointed out to Caius that it was the middle of the day. This fact surprised Jacob. (I don't remember if that fact made the movie.) Me, read the book? Nah.

Anyhow, I worked on it for about 2 hours after you signed off. The version at that point has 7 different UNICODE moon symbols with the sky going dark at new moon and brighter towards full moon. It's only 3 columns now--date, UNICODE character on a background of varying blackness, and the 2-word description of the phase--waning crescent, waning [and close to] new, waxing crescent, etc.

Oh yes, here is the live version, and here is the snapshot of that code. Just after linking to that version, I moved the style tags back into the HTML rather than an external CSS. I'll get around to deleting the CSS eventually. Then I made a few more small fixes.

This is one main way to build HTML elements--by creating them as objects in JavaScript and appending them to the preexisting document. Note that cree() is defined in /opt/kwynn/js/utils.js. I mentioned this weeks ago in the context of various ways to handle MVC and 4 layers of code. The other way is to write the HTML directly in PHP, keeping HEREDOC string format in mind. They each have their cases and merits.

In this case, everything is (client-side) JavaScript. If you were building HTML in JS starting from a database, one method goes something like the following (all source code):

<script> <?php	echo("\t" . 'var KW_G_CHM_INIT  = ' . json_encode($KW_G_TIMEA    ) . ';' . "\n" );  </script>

A possibly better example would be when you init a larger array that the JavaScript cycles through and creates a row for each member of the array. You still init it the same way as above.

January 28, 2022 (first entry 17:14, entry 2: 17:52)

Entry 2: Note to self regarding my goaround with OAUTH2 recently: a rewrite rule would partially solve the XDEBUG flag from Google when it sends an OAUTH code. That code expires in a matter of tens of ms or less, though, so if the debugger stops processing before a certain point, the code won't work. At least, in my reading I learned that a system clock being off by 5 ms could invalidate the code. So maybe there IS a practical reason to keep good time; imagine that.

Yes, setting up an environment--databases, libraries, debugger, etc.--is painful the first few times you do it, and even then it can be painful. Helping with that is one my jobs, so it's best not to do such things while I'm asleep.

Otherwise put, expect such things to be a pain for the first several times you do them. Take notes; keep those notes in a very accessible place--perhaps publicly online or in GitHub. If you don't do it for months and months, then it can still be a pain when you forget.

When you ran sudo systemctl status mariadb , the results were somewhat puzzling. I had assumed that installing the mariadb server will install the client. Now I'm not so sure. So, for the record:

apt list --installed | grep mariadb
mariadb-client-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-client-core-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-common/impish-updates,impish-updates,impish-security,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 all [installed,automatic]
mariadb-server-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-server-core-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-server/impish-updates,impish-updates,impish-security,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 all [installed]    

When you see version numbers, you generally do not want to use those. You almost always want the latest available. The situation gets even more puzzling:

sudo apt install mariadb-server
# already latest
sudo apt install mariadb-client
# something new was installed.  Huh?

Whatever the case, does the "mysql" command exist? I realize you are going to use MySQL Workbench as a client, but you'll want that mysql command. "sudo mysql" will get you in as the root mysql user so that you can set the root password for non-root Linux users. That is, mysql (mariadb) assumes a relationship between the Linux user and the db user unless you specify otherwise. You can get into mysql as mysql-user-root with sudo, but MySQL Workbench will need a MySQL root user password because you should not be running MySQL Workbench as root; it's just bad form.

I am puzzled why your mariadb status talked aobut the "mysql" (client) program at all. My does not. It was somewhat unconnected that I suggested what was wrong with your Workbench connection.

For future reference, I'm not sure how clear the distinction is in MySQL Workbench between "server not running" and "server is running but I, Workbench" cannot connect to it. In some ideal world one of us would take the time to document that. That distinction led to some confusion over the last few days.

As I think about it, your status results were even more puzzling. This is what I get:

systemctl status mariadb
● mariadb.service - MariaDB 10.5.13 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-01-28 02:50:44 EST; 14h ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
   Main PID: 1179 (mariadbd)
     Status: "Taking your SQL requests now..."
      Tasks: 9 (limit: 9458)
     Memory: 217.9M
        CPU: 1.779s
     CGroup: /system.slice/mariadb.service
             └─1179 /usr/sbin/mariadbd

Jan 28 02:50:44 ubu2110 mariadbd[1179]: 2022-01-28  2:50:44 0 [Note] InnoDB: 10.5.13 started; log sequence number 862675097; transaction id 15437
Jan 28 02:50:44 ubu2110 mariadbd[1179]: 2022-01-28  2:50:44 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
Jan 28 02:50:44 ubu2110 mariadbd[1179]: 2022-01-28  2:50:44 0 [Note] Plugin 'FEEDBACK' is disabled.
Jan 28 02:50:44 ubu2110 mariadbd[1179]: 2022-01-28  2:50:44 0 [Note] Server socket created on IP: '127.0.0.1'.
Jan 28 02:50:44 ubu2110 mariadbd[1179]: 2022-01-28  2:50:44 0 [Note] Reading of all Master_info entries succeeded
Jan 28 02:50:44 ubu2110 mariadbd[1179]: 2022-01-28  2:50:44 0 [Note] Added new Master_info '' to hash table
Jan 28 02:50:44 ubu2110 mariadbd[1179]: 2022-01-28  2:50:44 0 [Note] /usr/sbin/mariadbd: ready for connections.
Jan 28 02:50:44 ubu2110 mariadbd[1179]: Version: '10.5.13-MariaDB-0ubuntu0.21.10.1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Ubuntu 21.10
Jan 28 02:50:44 ubu2110 systemd[1]: Started MariaDB 10.5.13 database server.
Jan 28 02:50:45 ubu2110 mariadbd[1179]: 2022-01-28  2:50:45 0 [Note] InnoDB: Buffer pool(s) load completed at 220128  2:50:45

In my case, it tells you the port is 3306. Or:

sudo netstat -tulpn | grep -i mariadb
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      1179/mariadbd  

Your router is not at issue. The router has no part in the "network" traffic inside your computer. One diagnostic would have been to run "mysql" and / or "sudo mysql"

January 27, 2022

As with a week or two ago, cue "*autistic screeching*". If I haven't said it before, I hate OAUTH2. So, for the record, if you add a redirect URL in Google Cloud Console, you have to add the URL to your client_secret file IN THE SAME ORDER!!!! Otherwise, you get '{"error":"redirect_uri_mismatch","error_description":"Bad Request"}'. Here is my debugging code.

To make matters worse, I am using debugging code in part because I could not think of a good way to conform to my rule #1. That is, given that the redirect URL is called from Google, how do you get Google to put a precise URL query in the URI such that it invokes the debugger? I tried putting the xdebug query in there, and I'm fairly sure it simply told me it was an invalid URL. I found instructions for getting Google to pass data in a "state," but not a literal, precise URL query. The debugging code shows were I use file_put_contents with FILE_APPEND. It led me close to the problem, and then it occurred to me that order might matter. At least, I'm 87% sure that's the problem. I am not going to reverse it right now just to check.

This is almost reason in itself to get away from Big Evil Goo and run my own mailserver on my own domain.

January 26, 2022 - ongoing entries, 17:15, then 19:26

debugging MariaDB connect issues

If MySQL Workbench can't find the service, the service probably isn't running. Usually a service starts when it's installed, but not always.

# no harm in doing a restart if it's already running
sudo systemctl restart mariadb
sudo systemctl status mariadb
# ...      Status: "Taking your SQL requests now..."
# That's funny.  I've never noticed that before.
ps -Af | grep -i mariadb
# 7278 is the process ID.  Yours will be different in all probability
# mysql       7278       1  0 19:27 ?        00:00:00 /usr/sbin/mariadbd
sudo netstat -tulpn | grep -i mariadb
# 3306 is the port I would expect
# tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      7278/mariadbd       

snap versus apt 2; Firefox 2

A double installation was not the problem I had. It was something much more subtle. If I may be so bold, your whole email should be entered into the record. You're anonymous for now, so I'll just do it.

I have spent well over half and hour doing all sorts of such things.

funny, the only issue i had with snap vs apt was exactly that, firefox. i just had to apt remove firefox, because somehow i had 2 copies of firefox floating around... randomly, (well probably not random at all) in certain apps when I would click a link (like discord or thunderbird) it would open in the older firefox (apt) version, and the rest in my new firefox (snap install). the only reason I could differentiate the 2, and know which one to uninstall, was when I went to help/about in firefox, it actually said it was a snap package.. so I knew to remove the apt version.

previously to that, i wasted about a half hour trying to find out how to set the default browser to load in thunderbird when a link is clicked, and sadly i didn't find any way to do it.

snap versus apt

The trend appears to be towards snap, so if there is a snap, go with it. I was playing with something once that was causing trouble as a snap versus an apt, but it wasn't that important an issue. I think it was Firefox, but I'm not even sure.

MySQL, MariaDB, and MongoDB continued

In response to your email a few hours ago... You may have misunderstood the choice I was positing. If you're using Ubuntu, MariaDB is the way to go. There are supported packages for MariaDB. MySQL was abandoned by much or most or almost all of the open source community over concerns that Oracle would corrupt it. (Oh, Oracle is a funny coincidence, huh?)

As best I remember, every site I've seen over years has moved to MariaDB, except perhaps for the insane people who were still using PHP 5.x.

Yes, I do recommend installing MySQL Workbench directly from Oracle. I don't think there is an Ubuntu package for it anymore. MySQL Workbench gives you an enormously prettier SQL command line to work with, as opposed to the pure "mysql" command line. Even though I'm using MariaDB, the commands have stayed the same, so it's still mysql. MySQL Workbench will "express" "confusion" over connecting to MariaDB, and it will warn that it's not supported, but it will work.

So, going back, the choice I was positing was not between MariaDB and MySQL. The choice was between MariaDB and MongoDB. I have probably a few hundred lines of working MongoDB code in my repos. I have 0 MySQL code on GitHub. I drank the MongoDB (object oriented database) Kool-Aid in 2017, and I've started using it whenever I have the choice. I won't be using MySQL for my main (part-time) project for much longer. As soon as I free it from Drupal, I'm done with MySQL.

MySQL is not going anywhere, though, and I'm in the minority of PHP and MongoDB users. MongoDB is much more closely associated with Node.js (MEAN / MERN stack). MySQL will have much better examples out there, too.

I said that there would probably be some grumbling on my part if you chose relational (MySQL) over OO. We'll see when and if the grumbling begins.

January 24 - 26, 2022 (20:06 - entry 2 1/24; first entry of the day drafting at 02:14 1/24, posted 1/26)

RewriteEngine On
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,NE]           

January 22 (first entry after 17:03, continuing 18:35)

Apprentice is trying again to run my chat code, as mentioned yesterday. First, I have a chuckle that "php fatal error" must seem ominous to him. On some days I probably get several fatal errors an hour while dev'ing, usually syntax errors. I'd imagine I've mentioned it before, but I remember in the C language in 1992 syntax errors could take me 3 - 4 hours to solve. Helping with such things is what I see as one of my major jobs as master dev for an apprentice. I don't think I learned all that much taking 3 - 4 hours to solve syntax errors. I probably just got that much closer to getting frustrated and changing majors.

In fact, I know someone who did that in 1985. At that point, the compiler simply said "syntax error." No line number. He'd had enough at that point. I at least had a better indication in 1992.

It's rare that syntax errors take me more than a handful of seconds to solve these days. Some of the most puzzling instances that have taken longer involved my violating my own rule #2 to the effect of your dev environment should be as close as possible to the live system. In those cases, the "Heredoc" PHP string syntax worked differently in PHP 7.4 versus 7.2. Thus, something worked on my system and gave a mysterious syntax error on the live system. Worse yet, the include / require_once() tree was such that a syntax error crashed the whole system. I only have one user of that system, and it was late enough at night that I doubt he had any intention of using it, but still.

The difference in "heredoc" was something about whitespace--the higher version of PHP allowed whitespace in places that 7.2 did not.

Anyhow, I have already emailed him back. I am not tormenting him waiting for this entry. I still don't know exactly what is wrong because I'd need to be looking at his system. However, I can give some (more) general advice and repeat what I said in email.

When I say "include," I should clarify even more than I did in email. Generally speaking in a number of dev languages, an "include" means to incorporate code. The precise keyword "include" goes back to C if not before. To make the history somewhat more complicated, includes work differently in C because C is compiled. I'll stick with PHP from here on out. If I say "include" in PHP, I am using the word in the general sense to mean incorporating code. In terms of precise syntax, I almost always mean require_once(). I would state it as a general rule that one should use "require_once()" unless there is a really good reason not to. I would guess that I use require_once in 99.7% of cases. I can think of one specific case of using include and one case that I can't quite remember the details of.

Anyhow, on to his specific problem and the general debugging methodology. Once again, in this case I'm using "debugging" more generally because a debugger may or may not help him in this case. It may not help because his situation is akin to proving a negative. He's getting an error very close to "Class 'dao_generic_3' not found in line 54."

His first problem is very likely my fault to a degree. dao_generic_3 is my creation in this file. I probably should have called it kwynn_dao_generic_3 or something to make it clear that it's my creation.

So let's back up to the more general case. If a function or class isn't found, the first question is how far from the PHP core / default it is. If the function is in the PHP core / default install, then you should never get a not found. There are a few classes of functions, though, that are not installed by default. So, step 1 in debugging is to search the function / class on php.net.

To take an example that I encounter all too recently, let's try "mysql_escape_string." The PHP doc will inform you that the function and "the entire original MySQL extension was removed in PHP 7.0.0." What part of that is unclear? This is one of several situations that wasted enormous amounts of my time. The PHP release history page informs us that PHP 7.0.0 was released on December 3, 2015. Let's just say that I came across the mysql... function fairly recently. When I spluttered incoherently about encountering this problem for the first time in YEARS, I was told "no other developers have complained about this." Yes, those would be the developers who are happily charging you $150 / hours (a real figure) to maintain an ancient environment.

Similar indications of madness were that one of the systems--perhaps the live system--had a clock that was several hours off in part because the system hadn't been rebooted in 900 days or something. A few hours later I noticed that someone had used the "date" command and reset the clock.

I don't expect other people to be insane as I am about timekeeping, but *hours* off!?!?! Having to use the "date" command?!?!?!

Anyhow, I digress. The point being that if you ever have the misfortune to see a "mysql..." function, the PHP documentation will at least tell you what's wrong. Solving your problem is a rather bigger issue, especially when the people involved are likely insane. This brings up an interesting question. Can developers diagnose insanity based on years-old functions or hours-off clocks? Perhaps so.

Now I'll give a much more reasonable example. In kwutils.php is getDOMO(). It calls DOMDocument. The PHP documentation within a few clicks mentions "This extension requires the libxml PHP extension." This situation is admittedly at least mildly painful until one gets used to such issues. The solution in this case is "sudo apt install php-xml" in Ubuntu. Hopefully Big Evil Goo gives that solution simply. I knew it from searching my install list.

A related problem I've had involves one of the MYSQLI constants, and that's MYSQLI, the current version, as opposed to MYSQL. I remember chasing my tail for quite a while because a MYSQLI constant wasn't found. The solution was as above to install mysqli. Ironcially, it's been so many years that the install is "sudo apt install php-mysql" rather than mysqli. Both Ubuntu and PHP assume that it's been so long that if you're still running the old mysql library you are insane and beyond help.

Anyhow, to continue down the path of undefined stuff, and back to apprentice's specific problem, he would hopefully find that dao_generic_3 is not defined by PHP or anything he can find. Again, this is an argument that I should have called it kwynn_dao... I should also perhaps make it more obvious more often that /opt/kwynn on my system is a clone of what I generally call "kwutils." In his case, he does know these 2 points. We just had another exchange while I'm writing this. My new advice is of general importance.

To back up a bit, though. Sometimes I do something like "cd / && sudo grep -R dao_generic_3" That would quickly result in "opt/kwynn/mongodb3.php:class dao_generic_3 {" Much could be said about that, but let's stick with this specific debugging path:

I didn't demand he recursively grep. Given that the details of the above are not obvious, I mentioned that the include execution should go from kwutils.php to mongodb3.php. The fact that it's not in his case is baffling. We're both fairly certain he has the latest of both repos of code.

Which brings me to:

minimal functionality debugging technique

I want to run around in circles waiving my hands in the air and screaming when I see people post 100 lines to a forum and want debugging help. First of all, I not criticizing my apprentice nor even the people posting to a forum. Because a main job of mine is to reduce frustration, my apprentice should be asking me sooner than later. And here we are a while later, and I still don't know what's wrong in his case. The solution is not obvious, and even if it was obvious, I don't want him spending large amounts of time on such things.

With that said, for future reference for forum posters and apprentices, the minimal technique goes something like the following. I am assuming he is using the debugger and stepping through line by line. Actually, I'll put this in GitHub. One moment please....

Here is my minimal debugging example.

January 21 (2 versions -- 16:39 and 16:56)

This is the part several minutes later. So that means you have "base" PHP / cli PHP running or you would have gotten an error of program not found. As I mention below, server.php has no output yet--that is, it has no output when it's run as PHP. I should add while I'm at it that sometimes I'll use "exit(0);" or "return;" (or "continue;") for no apparent reason. I'm not entirely sure about NetBeans 12.4 and / or the latest xdebug, but slightly earlier versions needed some code on a given line to make it a breakpoint, so I will write exit or return or continue. That's why I also created the null function kwynn(); Sometimes I would use "$x = 1;" which causes great gravy grief is one is processing weather radar data with x and y coordinates. *sigh* So THEN sometimes I'll use "$ignore=34523;"

The incident with the weather radar took me way too long to debug.

Back to server.php. No, you don't have to run server.php separately because if Apache PHP is running, the browser calls server.php and thus runs it. However, running server.php from the command line is a good idea for debugging purposes, as I mentioned a few weeks ago. Generally speaking, there is an argument to be made that web code should also run seamlessly in CLI mode because it's often easier to deal with / debug code in CLI mode. That's why I created the kwutils.php function "didCLICallMe" because I set up various files to run standalone in CLI mode.

Below is the first ~16:39 entry:

I started a web chat app. I'll point to the specific version in question. When apprentice hits send, he gets an XML parsing error, unclosed tag, in line 1, column 1, from the output of server.php. That is, he gets that in the JavaScript console. I assume that PHP is not running at all, but Apache is very likely returning the Content-Type as text/html in the return HTTP header. Apache is doing that based on the .php extension. So the browser is expecting HTML which these days should be in the form of XML. If PHP is working, the response would be completely blank because the whole server file is a PHP application with no output. I have put my debugger on line 6 and confirmed that the server is receiving data, so that's a form of output to me for the moment, to know that it's working so far. kwjssrp() in PHP and kwjss in JS are both in /opt/kwynn/js, and the relevant PHP file is included (require_once()) from kwutils.php. As a reminder to the hordes of people reading this, /opt/kwynn is a clone of my general utils repo.

Back to the browser--it sees the < opening tag of php, and the PHP tag is not closed for reasons I have gone over rather exhaustively, below. If you look at the server.php output in the browser (the HTTP response), it will almost certainly be the precise PHP code rather than executed code (which in this case would result in ''). The browser is trying to interpret this as HTML / XML. So, again, my assumption is that PHP is not running at all through Apache; Apache is just running the raw file. Given that this is your brand new install, I assume the solution is

sudo apt install libapache2-mod-php8.0
sudo systemctl restart apache2

If you haven't installed base PHP yet, I think the above will do it.

January 15, 2022

As I'm writing this, as of 18:37, apprentice reports a successful install of Ubuntu 21.10. It should be recorded as 21.10 rather than 21.1 because the Ubuntu format is YY:MM. Ubuntu releases every 6 months in April (04) and October (10). The even-numbered years in April are LTS (long term support) releases, meaning support for something like 5 years. The other 3 releases in the cycle are supported for something like 9 months, so that you have 3 months to upgrade.

As an action item, does the following work?

sudo apt update ; sudo apt dist-upgrade

When I say "work," I mainly mean do you get any indication of broken packages? You may have no packages to install. You may also get an autoremove message. There is no need to do the autoremove step, nor is there any harm. I have never had a problem with autoremove. Also, it's possible you'll get a message about another process having the lock because "unattended upgrades" may be in progress. Generally, that's fine; just let the unattended finish. Sometimes unattended dies, though, and you'll sit there forever waiting. Usually that only happens when I'm updating a system running on a USB drive plugged into USB 2.0. (I have old hardware. I do have a USB 3.1 PCI card and use it all the time, but my hardware is old enough that it won't boot off of PCI devices.)

Which brings some more commands, in this case to know when a process like an upgrade has died. One is "top". I would play with that relatively soon. It goes in the category of you should know what it usually looks like, so you know how to look for anomalies. When an upgrade is working, or any other disk-intensive process, the "wa" (processes waiting) number will be very roughly 16 - 25% as opposed to near zero.

"sudo apt install iotop" and then "sudo iotop" is similar in that it does the same thing with disk data transfer. Again, I recommend looking at what's normal.

When I start a new partition / system / install, I don't bother moving data as part of the process. I move the data from the previous partition as needed, which is a good way to keep things clean.

Ubuntu comes with backup software. I have never played with it, but it might keep track of full backups versus incremental backups. That would solve your redundancy problem. The "rsync" command does that, too. I've been using rsync for the last several weeks to upgrade kwynn.com with this local copy of the DocumentRoot tree.

Please tell me you didn't install 3rd party software? If you did, just stay on top of broken packages.

In answer to one of your points from weeks ago, I just moved the Ubuntu Symphony JavaScript to an external file. I did this in part because I have no plans to touch the code ever again, and it was sitting there in this file in my way.

written before I knew you were done with your install

As for your immediate problem, I wasn't clear on at least a couple of points. Just below I go back to immediate solutions. But, first, to clarify one point: not installing 3rd-party / proprietary software is for future reference; at the moment, that damage has already been done. For future reference: when you install Ubuntu, there are only a handful of (sets of) options as you start the install. One of them is whether you want to install 3rd party software. It asks you this about the same time as it asks whether you want to update packages as part of the install. For future reference, I recommend against installing the 3rd party software, in part because I think it caused your problem. I suspect it's caused me problems in the past, too.

But back to the immediate problem, you may be able to solve everything if you just remove those 2 packages that are causing you problems. In case it's not clear, it's possible everything you're describing traces back to that. I will reiterate the point: if you see any indications, ever, of broken packages, back up your data and then attack that problem.

In the last few hours, I booted into my partition that is broken in a similar manner as yours. I was hoping to start removing packages to definitively record how it's done. My partition is far more broken than yours, though. The NetworkManager service was apparently "masked" by the cascading failure I had weeks ago. I had to look "masked" up. It's a step beyond "disabled" in systemctl / systemd. That means I had no network. My attempt to "unmask" it deleted it rather than unmasked it. I have almost no clue what that's about. Without a network to look things up and let Ubuntu load software, I decided it was time to give it up on that partition.

"$ man apt" indicates that it may be as simple as "$ sudo apt purge <package>" or "$ sudo apt remove <package>" After messing around with that broken partition, though, I decided I am not removing anything at all on any of my partitions.

If you decide to start over, I would install 21.10, the Impish Indri; I've been running it since soon after it came out. Impish is not LTS, so the consequence of this is that support will end in roughly July of this year, so by that time you should install 22.04, which will be LTS. My jury is still out on this decision in general, but I am leaning towards keeping up with every 6-month, full version. Probably the way to go is to upgrade your dev machine to every full version and then upgrade any clients once you're satisfied that their software will work.

One of the reasons I say this is because in addition to 3rd party software, I think you were zapped by Ubuntu's deprecation of 32 bit systems (i386). I'm assuming Ubuntu deprecated them some time after 20.04 based on observation; I'm sure you can find the official doc.

"Try Ubuntu" and a permanent USB install to let you boot

A USB installer has a "Try Ubuntu" mode. I recommend always having an installer USB on hand to give you something to boot from. Note that the "Try Ubuntu" mode will not save your data, though. Any data saved to the USB only lasts until reboot. You can of course save to a standard filesystem.

I also recommend installing from one USB to another USB such that the second one has a permanent install. Then you can do work on the full install. Also, the full install can be set to boot to your SSD or hd partitions. (You can do multiple installs onto a USB just like onto any other disk.) Note that when you are trying to boot from A to B, though, A must have a common kernel with B. Thus, you need to periodically update the USB. Also, when you are in Grub (just after power-on-reset) you'll notice that there are various options. Several of those options allow you to look for a common kernel. If you don't have a common kernel, when you try to boot into a partition, you'll get a message to the effect of "cannot load."

After you update the USB, "sudo update-grub" is what syncs the USB with what is available to boot to on other disks. Note that update-grub will set the partition running grub as the default boot partition from that physical device. This can get confusing if you run grub from an SSD or hd. If you had 2 partitions on an SSD and you update-grub from one, it would then be the default, which may not be what you wanted.

As I mention this, you should look into partition numbers and disk names (/dev/sda7). You should know what your / actually is on the /dev level. The spacing of this is very precise: "mount | grep ' on / '" and / or just run mount to look at it. I can't imagine what harm you could do if you run mount as the standard user (not sudo).

data management, commands that might save you in case of trouble, and one means of backup

You may do this anyhow, but consider various degrees of separation between a filesystem needed to boot / run and large data that is slower to copy. That is, your data and running files go into at least 3 categories: the files needed to run Linux, your small data such as code you type, and then various degrees towards your large data. Consider keeping your large data in one file tree that can be isolated or better yet on another partition. It makes it easier to know what you really need to back up, and it lets you copy a whole partition and then set it up to run.

The following is not definitive in that I am not going to take the time to test it over and over. It's something to explore.

One reason I mention all this is that now that you have a running, pure partition, possibly with little of your data, you might want to make a copy and learn how to make that copy runnable. First, I highly recommend doing the copy itself from another booted system, such as an install disk running in "Try Ubuntu" mode.

I often do the copy with the GUI: "sudo nautilus" Nautilus is the "Files" manager. If you run it as root, it can copy anything. There is a recursive "cp" that does this and preserves dates and users and such, but I lose track of what all those switches / arguments are.

So, make your copy. You should probably give the copy's partition a name and partition label. Otherwise, in /media/<your user> you'll get a UUID and have to figure out which one is which. In any event, navigate in the command line to the root of the other partition, which will start as /media/user/something. Then I give you the commands that are not harmful as long as you exit the shell when you're done, or at least exit from chroot. Then I give the last command that is probably not harmful but may be confusing.

First, you might want to mark both / and both /home/user with a file indicating which copy is which such as "iam_2022_0115_18_install"

# be in the root that you want to make THE root.  That is, pwd shows the relevant root
sudo mount -t proc /proc proc/
sudo mount --rbind /sys sys/
sudo mount --rbind /dev dev/
sudo chroot .
      

At this point you will be the root user (indicated by #) and the filesystem root will be the "new" root that you want to make independently bootable. Then I *think* a "sudo update-grub" is enough to make it independently bootable. I would not do this, though, until you are sure you can boot into a working partition from USB as above.

The reason you need to set grub from the new partition is that otherwise all of its mount points are set to the old partition because it is a copy. Before grub, the chroot sets / to the new partition, then grub sets the bootloader to recognize this new partition as bootable.

NetworkManager - something I learned today.

I already knew the systemctl commands below, but I didn't know the name of the network service.

Somehow when I booted into the broken partition, my active NetworkManager unset itself to a degree as I'll explain. That is, when I booted into the active one, I didn't have any network. I had to both (re)start it and re-enable it to start on boot:

sudo systemctl start NetworkManager
systemctl enable NetworkManager
      

January 14, 2022

Ubuntu package problems and some solutions

My currently active apprentice is having package (software install) problems. These are some thoughts.

For one, when installing Ubuntu and probably other Linux distributions / "distros" / sub-brands, the 3rd party / proprietary packages / software are probably not worth it. That's one question Ubuntu asks by default upon install. I suspect that's the root of his problem. Specifically: when he runs "sudo apt install --fix-broken" these 2 packages have problems: "Errors were encountered while processing: libgl1-amdgpu-mesa-dri:amd64 libgl1-amdgpu-mesa-dri:i386"

When I run "apt list --installed | grep libgl1" I get libgl1-mesa-dri and libgl1/impish. I am reasonably sure this means he installed AMD specific drivers and I did not. I can run FlightGear reasonably well on very old hardware. I'd have to go digging for my graphics card specs, but I remember it only has 1 GB RAM which is old. Maybe the proprietary driver would help, but, as above, it's probably not worth the cost benefit, and I should get better hardware when I can.

I suspect he's also having problems because Ubuntu is pulling away from i386 (32 bit). That is an argument for installing the latest Ubuntu rather than the LTS version, but that's another discussion.

The GUI / Gnome / desktop was probably almost literally making noises about a package problem. Whether it was or wasn't, there is an argument to be made for checking by hand perhaps once a week:

sudo apt update
sudo apt dist-upgrade
       

If you get the package error message, do not ignore it. In fact, it's probably time to backup your computer. This has not happened to me often, but it has gotten out of hand perhaps 3 times in 12 years. Usually I was asking for it, but that's another discussion. (Below I list one instances of trying to downgrade PHP.)

In any event, solve the package error ASAP.

The solution to the problem is usually to remove the packages in question, and really thoroughly remove them. I can't think of a way to quickly set this up to demonstrate, so I'll have to refer ya'll to Big Evil Goo. The commands involve such terms as "purge," and then I can't remember if the usual term is delete or remove.

Note that you may have to use the dpkg command for this process, and you may have to use aptitude rather than apt, although if it's not already installed, you probably can't install anything while that error is pending.

Command list:

# a list of commands or their close equivalents for package management
# use these 2 often, perhaps once a week or more:
sudo apt update
sudo apt dist-upgrade
# the rest are not harmful but only useful in context
sudo apt install --fix-broken
# I have never used the audit option before; I was simply looking for something harmless but to remind one 
# of the dpkg command
# You may not have this package at all.  
sudo dpkg --audit libgl1-mesa-dri
apt list --installed      
# which package did x file come from:
# this is a small part of answering another question of yours
dpkg -S /usr/bin/find
# not directly relevant, but in answer to another issue
tail -n 20 /var/log/syslog       

For problems with running software as opposed to their packages, many pieces of software have a verbose mode or variously levels of verbosity. Sometimes you can shut down the systemctl version of the program and run it "by hand" with the direct command and verbose turned on.

You asked if some of these commands are the direct equivalent of what the GUI does. Well, update and dist-upgrade are the equivalent of some of it, and as you discovered, sometimes doing it by hand is an improvement because the GUI will lose track of changing IP addresses. That is, always do an update before an upgrade.

The above will not upgrade you a full version such as 21.04 to 21.10. There is of course a command line for that, too.

You mentioned digging around on various errors in the syslog. Unless the timing makes it certain it's what you need to focus on, I'm sure there is all sorts of junk in syslog that a perfectly running systems spits out. You could spend forever chasing such things.

software (un)installs versus Satan's OS (SOS) / Linux file organization

I only have a fuzzy understanding of how the files are laid out--libraries, binaries, datafiles, etc. In the immediate context, solve your package problem first. I wasn't trying to discourage you from learning more; I was just saying solve that problem first. I don't think a better understanding of the overall system will help you solve the immediate problem.

You pointed out in SatanSoft that you can delete software more easily. There are a number of reasons for that. For one, almost all software is using SOS code. In Linux, you have an ocean of free software to choose from. The Nu HTML Validator uses both Java and Python, for example. There are so many choices of how to build software that it leads to so many more packages. Open source software is build on layers and layers and layers of open source software. The dependency graph is much more complex.

For a number of reasons, SOS apps are already compiled for a known OS, and SOS takes care of any small differences in hardware. The software is already a more-or-less standalone set of files when it ships. Lots of Linux software is interpreted rather than compiled, so its dependencies are not compiled in. Also, Linux runs on such a variety of hardware that an app couldn't compile for a known hardware target anyhow.

Another way of phrasing the above is that SOS software can make assumptions about what libraries are available because those libraries come with SOS. With Linux, one can make very few assumptions, so one has to make a package dependency list which is turn invokes other dependencies.

Again, many more choices and possibilities.

Linux packages do have an uninstall. I just haven't listed the command because I don't have anything I want to uninstall that badly. Also, with Linux, how deep down the dependency tree should the uninstall go? It can't uninstall stuff that other software depends upon.

Upon thought, if you continue to have trouble, I can do LOTS of uninstalling on one of my partitions. This was weeks ago when I tried to downgrade to PHP 7.4. I used a PPA that I had success with before going forward, but going backward got out of hand quickly. I wound up with errors such as "yours" of the day, and I abandoned ship and built another partition. I can go back to that old partition and start stripping it down to get rid of stuff that doesn't work anyhow. I did NOT lose data. It would be difficult, in fact, to use data due to package issues, even if the system was unbootable. You could still get at the data with another running system whether on another partition or a USB.

several emails ago

You said that you've read some of these entries 3 times. That doesn't surprise me. In case there is doubt, I realize that a lot of this simply cannot make sense yet because you don't have the context. Hopefully in a small number of months, you can re-read these and get more out of it.

January 13, 2022 - broken HTML validator - continuing 02:38

I have the Nu HTML validator working locally. I cloned it to /opt and then this works: python3 ./checker.py all and then python3 ./checker.py run .

Days ago nattered on about the referer HTTP request header as it applied to the W3 HTML validator. Coincidentally, I just realized that the referer is essentially broken for validation purposes. It appears that in March, 2021 (version 87), Firefox started to "trim" the referer down to the domain name. It would appear that I have a lot of work to do to fix this.

I already fixed it for this page, or at least I gave it the quick-and-dirty fix. I just hard-coded the URL to kwynn.com. That won't work for a fully online test system--that is, a test system accessible from the outside.

The right solution is JavaScript populating the URI / URL on page load. But I have to add that JavaScript to 156 pages according to my grep -R. That is one of the rationales for a single page system--route all page requests through one program that adds / changes whatever needs adding / changing.

January 12, 2022, one of several entries today, starting this at 23:26

Updated 23:47 with some answers. Updated 23:54 regarding http://validator....

I'm consider several changes to my Apache config, several of which I've tested. Some notes below. I am not using newlines in a realistic way--that is, the following is not precisely correct.

• I must fix http://validator references before doing the SSL rewrite
• "$ sudo apachectl configtest" - keep this in mind
• apachectl above seems to want you to define the ServerName in apache2.conf itself, in addition to elsewhere
• This works quite nicely: <VirtualHost *:80>Include sites-available/common.conf
• Does certbot need ServerName in sites-available if it's in Apache.conf? - YES, if you want to auto-discover. But it CAN be in the include / common file.
• Will certbot find my email address if it's in the include file? Yes.
• If you are already in *:80, then everything needs rewriting: RewriteEngine On RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,NE]
• Apache Rewrite Rule config flags: L is the last rule, NE is no escape of chars in the URL because you want to pass them on as-is
• If you're in an .htaccess file: RewriteCond %{HTTPS} !=on and perhaps ReWriteRule ^(.*)$ but I think ^ above would work fine
• Note that the above is for a 302 (temp) rather than 301 (perm). I can do 301 later.

January 12, 2022, one of several entries today, starting 20:43, first posted 22:20

(I made small revisions at 22:38.)

And back to my web dev Q&A with my apprentice. He is the "you" to whom I'm originally speaking.

First, something I should emphasize again about sending email. Among basic features that should be simple, it is one of the harder things I've done over the years. I'm sure I've done lots of harder things, but nothing that is both such a basic feature and so hard. Part of the problem is that there are Federal and presumably many other laws around the world around spam, so the providers are very cautious.

To clarify for myself and everyone else, your point about "from scratch" was about memorizing stuff. There are basic things I've done several thousand times that I still look up in some sense of the term "look up" (see below). That's in part because PHP is inconsistent in the order of function arguments in a few places, but that's not by any means the only example.

I'm trying to stop beating on php.net and trying to grep -R my own code. I will suggest something that I have only done sporadically:

Create your own page and / or your own program that demonstrates all the things you keep looking up. Exaggerate the clarity of the variables. I tend to use very short variable names because I get tired of typing them, but I should take my own advice when I'm giving examples.

I think it'd be very funny and geeky and cool to have one bigger and bigger program that demonstrates all of the little "primitives" of coding, where I'm using the term primitives to mean the syntax, the functions, the order of the function args and what they are, snippets of doing a bunch of simple little things, etc.

Alternatively, I have downloaded all the PHP doc, but I never fully installed it. I've also considered installing it on Kwynn.com. I've also also (sic) considered installing the "Nu" HTML validator that W3 runs. Recently I noticed a reference to installing it yourself.

Other than my not understanding the memorizing key to your question, we seem to be on the same page.

What, me using uncommon words such as Quixotic? I would never do such a thing.

WINE is a mixed bag. Around 5 years ago an apprentice got Call of Duty working in Linux running on iCult hardware. I don't remember if it was perfect, but it at least worked fairly well. I've had some success with WINE, but I'd say it's still a pain. A potential apprentice recently mentioned Zorin Linux, which is based on Ubuntu. Apparently the emphasis is on making it easier for people to migrate from SatanSoft. The Zorin WikiPedia article mentions both WINE and PlayOnLinux that Zorin encourages.

HTTP headers

Regarding the "header thing" or the "DO NOT CLOSE PHP TAGS UNLESS..." thing, which I got into in the last few weeks, this issue is a special case, just like sending email is a special case. The email thing you came up with on your own, in that you brought it to me. The tag thing I'm shoving a screeching at you because it has caused me enormous damage. For one, it's a special case because it goes in the "hear me now and believe me [and understand me] later" category. It's probably not worth taking the time to reproduce the dreaded "...cannot be changed after headers have already been sent" error. With that said, I'll give some explanation.

Below is a relevant example. I have removed a number of lines to make it smaller.

$ curl -i https://kwynn.com/robots.txt
HTTP/1.1 200 OK
Date: Thu, 13 Jan 2022 02:11:22 GMT
Server: Apache/2.4.41 (Ubuntu)
Last-Modified: Sun, 04 Oct 2020 04:13:01 GMT
ETag: "195-5b0d094ed72a3"
Content-Length: 405
Content-Type: text/plain

User-agent: *
Disallow: /t/8/01/wx/wx.php
Sitemap: http://kwynn.com/t/1/01/sitemap.xml
Sitemap: https://kwynn.com/t/20/10/sitemap_begin_2020.xml

When your PHP runs in "web mode," it is usually responding to an HTTP GET or POST, which I'll demonstrate more specifically in a moment. It must respond to an HTTP request with a result specified by the protocol (HTTP). It's doing some of that behind the scenes. PHP *MUST* generate headers for the browser to accept the result. The curl command above generates a GET, and that is the result including the headers, minus a few lines. I'm not sure which of the above are absolutely required in an HTTP response, but, whatever the case, the browser is expecting a number of lines and a number of types of lines and then a double newline. Everything after the double newline is the body which is often the HTML itself. If you output anything, even accidentally, from PHP before the proper headers go out, you may break your page.

If that's all there were to the "cannot be changed" issue, it wouldn't be so bad. But, believe me, at least years ago the results were unpredictable to the point that it seemed like a virus. (A real computer virus that does damage to data, as opposed to fake viruses that do nothing to people.) At that time, I was just starting to do hard-core PHP dev, and I could not figure out what was going on. (I think I already was using a debugger.) I'm sure I Googled, but I guess it still took me a while to figure out what was going on.

I was going to show you what I thought was a more "pure" example of an HTTP GET, but it doesn't work quite like I expected. I think that's because I'm not sending a proper HTTP request packet, and my brief attempts to do so didn't get me anywhere. But it hopefully the follwoing gives you more insight. Note that I'm removing parts of HTML tags because it seems that the validator doesn't like CDATA, or maybe CDATA has been deprecated.

$ telnet kwynn.com 80
Trying 2600:1f18:23ab:9500:acc1:69c5:2674:8c03...
Connected to kwynn.com.
Escape character is '^]'.
GET /
!DOCTYPE html
html lang="en"
head
[...]
titleKwynn's website/title
[...]
/body
/html

You can see the request and response headers in control-shift-I network, and there are other ways to get at it. Note that if you ever parse a raw HTTP response, all of the newlines are old-fashioned \r\n rather than just \n in Linux. This becomes important if you try to separate header and body. I parse it with "$hba = explode("\r\n\r\n", $rin);" on line 34.

As for your having no experience with PHP and "header()," hopefully I've shown that this is a much wider question than PHP. Everything on the web uses headers.

As for the header() PHP function, that lets you add headers when needed. I hope you have realized that all headers must come before the main body of the output. :) Reasons I've had to use headers are:

• "Content-Type: application/json" - You need to tell the recipient it's not HTML.
• You output a lot of headers if you're doing a direct download such as PHP reading an audio file and then allowing a download, or downloading a PDF. I'd have to go digging for those examples, or I'm sure you'd find them.
• If I ever decide to rewrite all URLs to a "single page" system like WordPress does, I should output my own modified dates and etags because PHP assumes the output is dated "now," when the date should be when the content changed.
• header('Location: https://' . $_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI']); -- that's one way of doing a redirect, in this case to HTTPS from HTTP, although this is not the best modern method for forcing HTTPS.
• You have to use headers in cases of cross-scripting issues / CORS / Cross-origin resource sharing. Browser security says the browser can only accept data from one domain unless you output headers telling it that cross-origins are OK.

Hopefully that gives you a lot more under the hood.

kwas()

In your kwas() example, you'll want a line before exit(0); that does echo('OK' . "\n"); I assume it's not outputting anything because you got sendmail installed, and sendmail is accepting emails for processing and thus mail() returns true. (As I said at great length below, that in itself won't get you much closer to sending an email, but anyhow...)

kwas() is about checking your assumptions ALL THE TIME. In your example, you didn't output anything in the event of success.

You did incorporate kwutils into your code just fine. You just didn't do anything to output in the event of success.

There is more to say on this, but I'll wait until you have more examples.

Here is an example of one consequence of require_once'ing kwutils. First without then with:

<?php
$a = [];
$b = $a['a'];
echo('got here' . "\n");
// RESULT:
[stderr in red:] PHP Warning:  Undefined array key "a" in /home/k/sm20/frag/kwuex.php on line 4
got here

<php // new program
require_once('/opt/kwynn/kwutils.php');
// then same 3 lines as above
// RESULT:
ERROR: kwuex.php LINE: 4 - Undefined array key "a" /home/k/sm20/kwuex.php

When I change the error handler in kwutils.php, warnings become fatal errors. I am certain this is the right answer in the real world. Just about all of my own code uses it. I haven't been able to fully prove it's the right answer in my main paid project because Drupal throws warnings right, left, and center. But the new version is most certainly going to take the position of forcing fatal errors. I made this change in response to annoyance at Drupal throwing warnings and continuing execution.

There is as always more to say, but that will do for now.

January 12, 2022, posted at 20:33 (started 18:11) - installing an HTTPS SSL cert locally

This is my second entry started today and the 3rd entry that either started or bled into today.

You have to have a domain for an SSL cert. For local dev purposes, I highly recommend using a subdomain (testinglocal.example.com) even if you're not using the base domain. One reason is that when in development, you change things so much that you might go over certbot's limits on certs. It's something like 5 certs a week for a given fully qualified domain. Thus, if you're using a subdomain, you can just use another subdomain (testinglocal2.example.com) rather than losing access to the base domain for several days. This isn't theory. I went over the limit several months ago. It snuck up on me.

As I muck around with my internet service provider's modem / router, I'm finding that my local system does not have a 32 bit IPv4 identity. This is important for firewall reasons. So, let's see if this works: "$ ifconfig | grep global" That results in 3 global IPv6 addresses. The first one didn't seem to work, then I added a second. Then as I ran curl from kwynn.com (external to my local system) and found which address it was using, I went back down to only one IPv6 address in the DNS AAAA record for the subdomain. You register DNS records with your domain name registrar. I use Hover.

My router's ping / icmp settings were somewhat confusing. The setting "Drop incoming ICMP Echo requests to Device LAN Address" had to be turned off for the "Global Unicast IPv6 Address" of the router itself to respond to ping. In order to ping my local system, "Reflexive ACL" had to be turned off. That needs to stay off through the certbot verification because the process needs a system passively listening on port 80.

Turn off any active local sites except the one in question. That is, disable them in Apache then restart. Below, "irrelevant" is the site / virtual host defined in /etc/apache2/sites-available/irrelevant.conf

sudo a2dissite irrelevant
sudo systemctl reload apache2

I set the one active virtual host to simply be receiving *:80: <VirtualHost *:80> Putting IP address in the VirtualHost top line did not work--certbot did not find anything listening, even though curl worked. You also need to set the ServerName to the fully qualified domain. Don't forget to restart Apache.

Note that the ufw default settings were not causing a ping problem. As for getting ready for the certbot: "$ sudo ufw allow http" and "$ sudo ufw allow https"

Certbot home, and relevant-to-me instructions.

First, do the "dry run": "$ sudo certbot -v certonly --apache --dry-run" Then do the real thing: "sudo certbot --apache"

Once SSL is working, you can put an entry like this in /etc/hosts:

127.0.0.1   blah.example.com

Once you do that, you can reverse all the security and even the AAAA DNS record because the site is now self-contained. Note that you have to open things up again to renew the security cert before 90 days.

If needed, set your firewall back to "Reflexive ACL" on. Then "$ sudo ufw status numbered ". Assuming you want to delete all the rules, "sudo ufw delete 1" to delete all the rules until they're all gone. Delete the AAAA record.

For cleanup later, when you're done with the cert: $ sudo certbot delete --cert-name blah.example.com

January 12, 2022 (AM) - the concept of "from scratch"

Note that my January 11 entry on email continued into today, January 12. This is a new entry started on Jan 12.

Yet again I'm continuing my web dev Q&A, usually answering emails from one apprentice. He is the "you" when I use "you."

You asked about Googling HTML and CSS and "from scratch." I fear you may have seriously misunderstood what I mean by "from scratch."

Before revisiting "from scratch," I'll separate what I do and do not recommend:

DO

• Use Google and any other examples / sources you can get your hands on. "Program by Google" is fine as long as you eventually understand the core code you're using. Again, start anywhere / however you want / however you can / whatever makes progress. Do something--anything--to start.
• Use others' code for specific tasks such as PHPMailer for email. You do NOT have to understand its internals at all.
• Start with others' code as the entirety of your app, as long as you eventually understand the core logic.
• If you see something using Bootstrap CSS, find out what Bootstrap is doing and use that specific part without installing a zillion bytes of Bootstrap

do NOT, at least at first

• use Angular, React, Bootstrap (CSS or JS), WordPress, Drupal, or the like

When I say "from scratch," in part I mean use out-of-the-box JavaScript as opposed to a general library like React or Angular or even jQuery, at least to start. You should know how to use basic JavaScript. After gaining some understanding, perhaps in a month or two or so, then by all means experiment, probably starting with React. For now, for my own purposes, I have turned away from all of the above including React. I still want to whittle on my own JavaScript. At this rate it will be 6 months at least before I reconsider that. I put React highest on the list because I have not tried it and have heard good things. I have tried Angular and I found it to be a waste of time in the short-term. It took longer to learn the "Angular way" than it would have taken me to do it myself and even create my own portions of a library.

The same applies to using WordPress and Drupal and such in PHP. They fundamentally alter the dev landscape. Using WordPress in some cases may be a necessary evil, but you should still know how to do various things yourself.

You're going to be Googling HTML and CSS and many other things for your entire career (or at least until Google is renamed after being seized for crimes against humanity). Starting with "program by Google" is fine as long as you eventually understand enough to modify it. The long-term problem with "program by Google" is people who do that and barely make it work and then have no idea what to do when it stops working.

I need to fully distinguish PHPMailer from Angular. Angular fundamentally changes how you do things. Angular is a very general library that does a lot of stuff. Again, if you're going down that route, you should know how to do basic things in pure JavaScript first. PHPMailer does something very specific; it does not alter the entire dev landscape. You don't have to understand the internals of everything you use, or you'd never get anything done.

Perhaps another way to come at it is that Angular is an overlay with which you do general dev. It's an overlay of pure JavaScript. You should know the basics of pure JavaScript first. You should be able to implement basic logic in pure JavaScript first. PHPMailer is an overlay of SMTP, but it does a very specific task. There is no reason for you to implement anything that specific. If you implemented everything yourself, you'd never get anything done.

Another way: your current goal is to write a web form and get email notification of the entry. You should understand a basic web form and be able to modify it yourself, even if you copied the code. You should have fluid control over your core product--the web form. A web form is very general and can have many field and do many things. PHPMailer sends emails. If it "just works," great. It's not the core of what you're doing.

Ideally, "from scratch" means you typed all the core code yourself, or you copied it from yourself. You may be typing it yourself but looking every single detail up. The next nuance that is good enough is that you copied it from someone else but you come to understand it well enough to modify it. Then the next time you are copying from yourself.

Going back to the Bootstrap CSS example, one problem with importing the entire Bootstrap CSS is that it formats every p, div, li, tr, td, etc. You wind up trying to override it and otherwise fight it. I addressed this roughly 2 - 3 weeks ago below when I talked about PHP date formats. The entirety of Bootstrap.css is huge. I whittled down to what I wanted and it was a tiny fraction of the whole thing.

Another way: all of the "bad guys" above are very general libaries or systems or whatever that put a layer between your code and the fundamental code below it--whether that's PHP, JS, CSS, HMTL, or whatnot. You don't want to distort your dev environment like that until you at least know what a pure environment looks like.

January 11 - 12, 2022 - sending email "programmatically" (maybe done at 01:39 Jan 12, my time, UTC -5 / New York / Atlanta)

Continuing again with my web dev Q&A...

One lesson is that I realized below that my stack trace in the kwas() example revealed my username by way of revealing its path. I removed that part, but it's a lesson in security. Knowing my username should not matter too much, for a number of reasons, but there is no reason to reveal it, either. It's good to consider such things for cases in which it does matter.

As of the Jan 12, 00:27 version, I have reworked this somewhat since anyone last saw it. For one, I moved the section on email providers up above the details of the PHPMailer class.

My first comment goes back several entries, including an indirect reference in my Jan 9 entry: DO NOT CLOSE PHP TAGS UNLESS THE CONTEXT DEMANDS IT! I will try not to boldface and put that in red, but we'll see what happens. Just after your mail(...) function, you close the php tag. In your code snippet, there is no HTML or hint thereof, so there is no need to close the PHP tag.

Regarding the mail() function, what is the date on that code snippet? Using the mail() function has become more and more problematic over the last several years. I hope no one is posting that recently.

As for the gory details of the mail(...) function: as I read the documentation, I'm somewhat surprised that I have to read quite a bit before getting a hint as to your problem. I know what you're problem is, more or less, but I'm looking at it as if I didn't.

To take the problem in parts: this is a case where kwas() would help. Also, I mentioned that you usually want to be able to run code in CLI mode for debugging purposes. There is what happens when I use kwas() in CLI mode, and something similar would happen in "web" mode and kwas().

First, the code, then running the script, below, and here is an active link of the following: https://github.com/kwynncom/kwynn-php-general-utils

<?php
require_once('/opt/kwynn/kwutils.php'); // a clone of https://github.com/kwynncom/kwynn-php-general-utils
// kwas() is current defined on line 50, but that of course is subject to change
kwas($res = mail('bob@example.com', 'testing', 'test'), 'mail() failed - Kwynn demo 2022/01/11 21:58 EST / GMT -5');
exit(0); // I am simply emphasizing that this is the end of the code -- do NOT close the PHP tag!!!!
        

Running the script--the same thing happens in NetBeans CLI in the Output / Run window:

$ php mail.php
sh: 1: /usr/sbin/sendmail: not found
PHP Fatal error:  Uncaught Exception: mail() failed - Kwynn demo 2022/01/11 21:58 EST / GMT -5 in /opt/kwynn/kwutils.php:51
Stack trace:
#0 [...] mail.php(4): kwas()
#1 {main}
  thrown in /opt/kwynn/kwutils.php on line 51        

I'll come back to this. It occurred to me that nothing says you have to use kwutils in its entirety. There is an argument for using your own equivalent step by step as you understand the consequences. The two points that I want to emphasize are kwas() and the two functions where I change the error handling such that notices and warnings, and whatever else becomes an exception. Those two functions are my own kw_error_handler() (current line 69) and set_error_handler() which is a PHP function on line 77.

Back to the error at hand, a related technique to "kwas()" would be to note that the mail() function returned false. You'd have to assign a variable to the return value to see that, though:

<?php
$mailResult =  mail('bob@example.com', 'testing', 'test');
if (!$mailResult) die('mail() fail'); // kwas() does the same thing with less lines, vars, and chars :)     

Also, in web mode, /var/log/apache2/error.log does show the error: "sh: 1: /usr/sbin/sendmail: not found"

You mentioned PostFix. It may or may not install sendmail. I don't remember PostFix' relationship to sendmail. With email, there is both incoming and outgoing. Even if you got sendmail installed, though, then there is the matter of configuring it. I'm not sure I ever got that working right. I got incoming sort of working, years ago.

Even if you got sendmail working and the email got farther in the process, you have another big problem or several related ones. When you use sendmail, it is going to (try to) connect to the server of the domain name of the recipient as specified in the MX DNS entry of the domain name. Let's say that's gmail.com. GMail may not accept the connection at all. Years ago, sendmail would have worked fine, but then spam came along, and then SSL came along. And then domain name keys came along, and related stuff around email and validating email. Even if GMail actually accepted the email, it would send it to the recipient's spam box unless you did a LOT of work. The work would involve DKIM and whitelisting and God knows what these days.

So the mail() / sendmail path is a very steep uphill battle. I've never tried fighting it very far. I have also been bitten by this exact problem in a real, paid project. It took until roughly 2 years ago to start causing bigger and bigger problems to the point of total failure. Before that, there were spam problems.

Far be it for me to call something like getting sendmail working a Quixotic quest. I have made some motions to that effect. However, in terms of an actual real-world solution, even I have ruled it Quixotic.

I have used 3 solutions in the real world. All of them involve the PHPMailer class that I address further below. First, though, you have to decide on an email sending provider, unless you want to fight the aforementioned uphill battle.

email service provider (sending email)

As I rework this, I realize that all this is just sending email. That is your immediate problem. I'm not even going to address receiving because I'm not happy with my solutions.

I said that I have used 3 solutions, all involving PHPMailer. I do NOT recommend this first one, but I want to address it because it shows you historically how things have gone along different paths, and it gives you basic info before getting somewhat more complicated.

If you wanted to use GMail to send, there is at least one hoop to jump through even with the not recommended path. If you want to do it the 2010 way, you have to change a setting in your overarching Google account. (I thought you had to specifically turn on SMTP, but perhaps not. I am probably thinking of IMAP and not SMTP.) You have to set your overarching Google account's security allow "Less secure app access." With this option, you would do that to avoid the infamous OAUTH2. I'll leave it at that short sketch because I don't recommend it anyhow, for several reasons.

I used that above option in the real world until several months ago. One problem is that Google will eventually and intermittently cancel the "allow" option. It's just not a viable option anymore. The next option, which I still don't recommend, is to use GMail with the infamous OAUTH2. I started doing that a few months ago when I stopped using option 1, so I am currently doing it. There are a variety of problems using OAUTH(2), however. I'll mention it as a possible option and then skitter away from it because it's a pain. I have a specific reason for using it right now, but I'm still on the fence as to the cost-benefit. In your case, I would strongly consider option 3:

Here I will propose something that may be mildly surprising or very surprising. I like both free as in speech and beer, but in this case I'm going to recommend a paid option, although it's almost literally dirt cheap for our purposes.

Yes, it's tempting to use Big Evil Goo for free as in beer (where you and your data are the product - TANSTAAFL), but it is a pain. I would probably step you through it if you really wanted to, but it borders on Quixotic even for me.

So I use AWS' Simple Email Service (SES), even for my own, non-paid-project notifications. The cost is so low that I don't think I've actually paid a cent for it even though I use it with a paid project. The project emails ~4MB files that add to the very, very low cost calculation. The price is something like 1 cent per 100 emails or maybe even 1,000 emails plus 1 cent per 100 MB of size, or something like that.

For purposes of being thorough from a tech point of view, MailChimp Mandrill is an equivalent service. I am almost certain MailChimp has gotten into the deplatforming / censorship game, though, so I don't recommend them on that basis. I did some testing with Mandrill years ago when it was free, but I also can't recommend it beyond roughly 6 - 7 years ago because I haven't used it since.

SendGrid is another alternative. I would not say I have used it so much as I have seen it used, but that was over 3 years ago.

Getting back to AWS SES, I need to add another few steps. You create a user in the SES screen. That user includes the username and password that you'll use in PHPMailer. Note that the user you create in the SMTP screen is an IAM user, but you do NOT want to interact with that user in the IAM screen, as I further address below.

Also note that the PHPMailer username is not the IAM user with dots in it (by default). The email PHPMailer username is of the form AKIA3ZN... continuing with several more uppercase and numbers. As the instructions tell you, you only get to see the password or download it once upon creation. Otherwise you have to create a new user, which is no big deal, but just to save you frustration. Note that I have found that renewing the credentials of an SES user in the IAM screen does not work. If you want to change the password, just create a new user in the SES screen and change both the username and password. If you change just the IAM password, you get silent failure. That is, you get silence at first glance. I never even set the debugger on it to see when or if the "silence" ends. I just went back to the SES screen rather than the IAM screen.

Another small potential problem with AWS SES is that you STILL have an issue emailing to arbitrary users--yet another layer of spam protection. By default, when you start using AWS SES you are in "sandbox" mode. In sandbox mode, you send a potential recipient an email from an SES screen, and he clicks an activate link. THEN you can email that address.

The SES screens list the port number and SMTP server and SSL / TLS / whatever settings, too, and they are in my code I mention below. Once you have a username and password and approved recipient, you're getting yet closer to actually, like, SENDING AN EMAIL. Amazing, huh?

PHPMailer class and composer

All of my solutions involve the PHPMailer class. I install it with the "composer" command. "composer" itself is mildly irritating to install, as I remember. You can start with "$ sudo apt install composer" but I'm not sure it's going to work. This is one of the roughly 20% - 30% of cases where "apt" / Aptitude is either not the entire solution or the recent-enough package just doesn't exist for Ubuntu / Debian. See what happens. This is a case where I can probably help quite a bit. Yes, the solutions are of course out there, but I still remember that it was irritating.

Composer is a tool specific to PHP. It's a package management system for PHP (source code) libraries. When you install a composer library, by somewhat circuitous steps it's a series of includes / requires / require_once() that pulls the PHP source code into your own code. That means that you can debug a composer-installed library. I don't think I've had to fix a bug in a composer library, but I have debugged into several of them in order to understand a problem and / or learn about how the library works.

As an aside, I specified that composer installs libraries that are included and can be debugged. That's as opposed to a library / extension that adds native PHP functions. For example, my nano extention is a PHP extension written in C that creates a few native PHP functions. Once it's installed, you simply call "nanotime()" like any other PHP function with no include / require / require_once needed. You cannot debug nanotime() just like you can't directly debug mail() by stepping into it.

Getting back to your original problem, first you have to get composer installed. Then you need to decide where to put composer libraries. I use /opt/composer I had to create the "composer" directory. Then note that composer wants you using a standard user, NOT root or sudo. Therefore, going back to your lesson on permissions, I recommend changing "composer" to be owned by your own user and give 755 permissions (rwxr-xr-x). The world / "other" users need to be able to read and pass through. There is no security issue with reading because the composer libraries are "public" code in the same sense that the "ls" command is public.

Once you have your permissions right, do the following. In my case, it's already installed, so your results will be different:

/opt/composer$ composer require PHPMailer/PHPMailer
Using version ^6.5 for phpmailer/phpmailer
./composer.json has been updated
Running composer update phpmailer/phpmailer
Loading composer repositories with package information
Updating dependencies
Nothing to modify in lock file
Installing dependencies from lock file (including require-dev)
Nothing to install, update or remove
Generating autoload files
5 packages you are using are looking for funding.
Use the `composer fund` command to find out more!  

Unfortunately, you're not quite done with your intro-to-composer experience. After I just finished saying above that I wanted to emphasize 2 parts of kwutils, I need to add a 3rd. ("Our three main weapons are fear, surprise, ruthless efficiency...") Once again, you don't have to use my kwutils, but you need to know how to use composer. If you go grubbing (grep'ing) around in kwutils, you'll see I do it like this... Actually, this brings up an interesting question for you. If you use the whole kwutils, I think PHPMailer will "just work" once you have it installed under /opt/composer. Let's see...

<?php
require_once('/opt/kwynn/kwutils.php');
kwas(class_exists('PHPMailer\PHPMailer\PHPMailer'), 'class does not exist');
echo('OK' . "\n");

Yes, it just works. If you think that the 'PHPMailer\PHPMailer\PHPMailer' syntax is one of the weirdest things you've ever seen, I agree. It gets into PHP "namespaces." I understand the concept, but I have barely studied them and have barely attempted to ever actually use them for my own code. One of the lessons I like to convey to apprentices is that I am very far from all-knowing, even when I should be a PHP "expert."

There may be "gotchas" just with require_once'ing kwutils. Maybe you'll find out. Either way, you should still understand what's going on behind the scenes:

<?php
set_include_path(get_include_path() . PATH_SEPARATOR . '/opt/composer');
require_once('vendor/autoload.php');
if (!class_exists('PHPMailer\PHPMailer\PHPMailer')) die('class does not exist');
echo('OK' . "\n");   

That works. As for actually USING PHPMailer, that is yet another step. Isn't this fun!?! Actually, in terms of something that should be simple like sending email, this is one of the harder tasks I've had over the years. Be happy that you're learning from my experience. :)

So, with that said, here is another decision point. I have created my own email class to use PHPMailer. There are most certainly "gotchas" on that--that is, if you use my class precisely, you have to set up the credentials like I did, and there are probably other gotchas. Hopefully I give instructions. (It's been long enough that I don't remember.) And if you want to do it "your way," that's fine, too. Also, I just created a web form with email notification a few days ago. Yours does not have to be that complicated. You can just use an HTML "form" for now. I get all fussy about save-on-edit (AJAX) because it was a specification of my main client. It was a lot of work to implement such that I'm still perfecting it.

Actually, to digress again, the save-on-edit went in 2 phases (so far). For the most part, I got it working several years ago and that is still working. Months after one of my revisions, we learned the hard way that my solution lost way too much data in some cases. I never did figure out what the "cases" were; I just reconceived and rewrote part of it. This problem wasn't catastrophic but it was of course annoying. I rewrote the one field that was causing problems. Since then, it has worked to the point that my client hasn't reported any more problems. I have reason to believe that small bits of data are still being distorted, but it's obviously not critical. Obvious because nothing bad has happened in a long while.

Because I got tripped up over that, I've kept whittling on my save-on-edit technique. I will probably rework it yet again with my main client in the next few weeks, as I partially rewrite the whole application to escape from Drupal and be compliant with PHP 8.0.

Back to your email problem. As for PHPMailer, you have my examples, and there are plenty more examples out there. I'm going to try to wind this down.

ALL THAT is to say that email is no longer easy because of nearly 30 years of spam wars.

January 9

Cue Rage After Storm's "*autistic screeching*" that I address at some length in my new personal blog. Several days or perhaps a few weeks ago I address php tags and the infamous output before headers issue. Now I can quote it precisely because I encountered it again, "...cannot be changed after headers have already been sent." I'm not sure that was the exact wording I saw many years ago, but it's close, and it's the same problem. In this case, the exact quote was "ERROR: kwutils.php LINE: 201 - session_set_cookie_params(): Session cookie parameters cannot be changed after headers have already been sent /opt/kwynn/kwutils.php" For the record (again), /opt/kwynn is my clone of my general PHP utils file and repo. Note that the link is to a specific version of the file--the relevant one.

I felt like "*autistic screeching*" when I saw that. The good news is that now I know what to do.

I'm going to get lazy and stop linking stuff. You'll see changes in my GitHub in at least 2 repos in the near future. I'm writing this 1/9 at 00:26 my time. The short version is that you call a parent PHP file as the link target and then require_once() the template. The session stuff goes in the parent file before the template is called.

2022, January 5 (PM) - at least 2 entries

"side exit" from the shopping cart (16:59)

Continuing again the web dev Q&A...

The principle of "do something" includes taking "side exits." It's fine to divert from the shopping cart to do something simpler with a database. Any understanding you gain is "doing something."

MySQL became MariaDB...

...and "Istanbul was Constantinople."

I should have thought to mention this earlier. If you take the relational route, MySQL became MariaDB. For Ubuntu installation, I *think* all you need is sudo apt install mariadb-server
In case it helps, I list what I have below. The one command above should kick off the rest, though. You'll need to download MySQL Workbench directly from Oracle, though.

   apt list --installed | grep -i maria
[...]
libdbd-mariadb-perl/impish,now 1.21-1ubuntu2 amd64 [installed,automatic]
libmariadb3/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-client-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-client-core-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-common/impish-updates,impish-updates,impish-security,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 all [installed,automatic]
mariadb-server-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-server-core-10.5/impish-updates,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 amd64 [installed,automatic]
mariadb-server/impish-updates,impish-updates,impish-security,impish-security,now 1:10.5.13-0ubuntu0.21.10.1 all [installed]  

To add to the confusion, MySQL is still being developed, as far as I know, but when Oracle bought the MySQL company, the open source community forked MySQL into MariaDB. When people speak of MySQL these days, they probably mean MariaDB in most cases, or perhaps 85% of cases.

entry 2 on the 4th then into the 5th - sessions, etc. (into Jan 5 01:10)

Regarding sessions, my update to my pizza code gives an example. I'm only using a handful of function from /opt/kwynn, so you can either extract them or use my whole utility. A reminder that I addressed this at some length days ago. Some of the usage in my little functions are very hard won information.

The session ID returned by my function keeps track of one user. Behind the scenes, a cookie is going from server to client and back. Keeping track of the session is really that easy. You can just call my "start" function every time because if the sessions is already started, my function will return it. Perhaps my function needs a better name, in fact, or an alias.

The cookie goes back and forth in the HTTP headers. You can see both the headers in the network traffic and the stored session on the client in Control-Shift-I as in India.

The session ID is very helpful, but it's only a portion of the shopping cart code. You addressed some of the rest of it in your other questions. I'll come back to them.

You asked about echos within a PHP / HTML file. In any entry roughly 10 days ago, I suggested up to 4 layers of PHP code from back to front. The echos go in the front most layer. An example is my user agent template file. The variables are being build deeper and deeper and come out with the echo.

More generally, a .php file can be all HTML, all PHP, or both. If there is no php tag, then the HTML is going straight to output--straight to the client / browser--from top to bottom as any other code. When there is a php tag, that code is run, and any HTML below isn't run until the php tag ends.

You can even do conditional logic on the HTML. You can surround the HTML with { } of an if and conditionally output the HTML. I have an example of that somewhere. Remind me to find it if you don't find one.

Whether you can do the same thing in JavaScript is both simple and more complicated. The short answer is yes, but if the data is coming from the server, then it still has to get to JavaScript somehow. But yes, you can do the same things in PHP (server side) or JavaScript (client-side) with the cavaet that the data has to get to the JavaScript. I discussed this at some length "below" such as when I discuss using one big JSON rendered by JavaScript versus writing the HTML in PHP.

How the client and server interact is a big question in that there are at least several good answers.

You mentioned clicks in JavaScript. Yes, detecting clicks and what was clicked and what the click means more or less has to be done in JavaScript, or at least it makes more sense. You mentioned writing to a local JSON. Note that client side JavaScript can't write to an arbitrary local file. JavaScript is very limited for security reasons. There is "local storage" in JavaScript, but I'm not sure there is a point in using it in this case because everything has to go to the server anyhow.

As I mentioned several days ago, I tend to think you want to account for the user moving off the site and then coming back to it, so the cart should primarily live on the server keyed by the session ID. With some exceptions and alternatives, JavaScript data is lost when the user clicks away from the page.

Getting back to the cart more generally, it's probably time to start learning about databases. If you want to punt that, you can save things to your own files or whatnot. You are probably correct that the basic cart concept is harder than you thought. You'll have to learn about client-server interaction and databases, and that's before you do the payment / checkout.

I should probably bang together the simplest shopping cart I can manage--perhaps 2 items with arbitrary quantities. I assume you're done for the day, though. I'm not sure I can be so inspired if you're going to bed. Also, you might need to back up and do some more general playing around with a database. MongoDB would make my life easier. I could live with MySQL, but it would cause grumbling on my part. Relational databases are so 1990s. If you install MongoDB, I recommend Robo3T as a GUI.

Yeah, as I think about it, learning basic database stuff and "hello world" both for the command line and programming databases is probably going to be a detour for the shopping cart. We'll probably do a lot of back and forth on this. For now, I'm not sure how helpful it would be for me to create a shopping cart.

Need a database for the shopping cart?

Do you need a database for the shopping cart? The short answer is very likely yes. The longer answer is something I hinted at above and you did in your email. You mentioned the shopping cart as a JSON file. Yes, the cart can be a JSON file. It's somewhere between impractical and not particularly sensible to save that JSON file on the client side, but you could save it on the server side.

You could do something like that during development, but it's probably time to bite the bullet and learn databases. For one of many examples, if you had a bunch of shopping cart files, it's harder to answer the simple question of "How many orders need to be fulfilled right now?" As its name implies, the purpose of a database is to organize data.

which database?

MySQL is not installed on Kwynn.com. MongoDB most certainly is. I only have MySQL installed on my latest local system because I could not quite justify moving my main (but part-time) client off of it until a few weeks ago. Now I am moving him off of it, but that's in the present tense. I will be happy when I can delete MySQL from my system.

Part of my pitch for Mongo is that you've mentioned a JSON file, and that is one lovely thing about Mongo--you esentially toss JSON files right in the database. To get it into MySQL in a logical format is a lot more work.

With that said, you pointed out that all the PHP examples you've seen so far are MySQL. MongoDB works in PHP just fine, but I very likely am in a (small? tiny?) minority of PHP users. I assume the common PHP stack is still LAMP (Linux Apache MySQL PHP). Mongo shows up in the MEAN and MERN stacks (MongoDB, Express, Angular / React, Node.js), to name a couple.

On one hand, I leave it as an exercise to the apprentice to research trends in relational versus OO DBs. On the other hand, I can be reasonably sure that MySQL in particular isn't going anywhere anytime soon. There may or may not be a slight trend towards OO, but it must be slight at best.

This is yet another instance of "do something." There will be grumbling from me over MySQL, but just your point about examples is an argument for starting in that direction. (All of my GitHub code is MongoDB, but my code was not written as a tutorial.) Also, I might drop support for MySQL when I drop / delete the whole thing, but that may be months away. Right now I won't even bother with an estimate beyond "months."

On the other hand, just in the last few days I've started writing some of my Mongo code to be executed from the MongoDB command line starting from PHP. In other words, if you could find good lessons on Mongo, the first concern would be learning it generally from the prompt or better yet from Robo3T. If you can learn it generally, running it from PHP is almost identical to running it from Robo3T, now that I have libraries that do that more and more easily.

I'll stop there and see what you come up with.

Asterisk

I answered an ad about VOIP. The key of the project was that the client needs to be able to leave more-or-less arbitrarily long voice messages. I haven't gotten to the point of just how long, but definitely well over 10 minutes. I would guess that an hour is needed. The problem they had is that they talked to 15 VOIP providers and noone went over 10 minutes.

I had a brush with a VOIP project in early 2016, and I've always wondered "What if?" I played some with the Asterisk software but couldn't make much of it. I compiled it and had it running in the barest sense, but didn't get it to do anything. Asterisk is of course free and open source.

In part because I had unfinished business from 2016, I started experimenting. Then I got obsessed and started chasing the rabbit. After about 21 hours of work spread over a week or so, I have most of the critical elements I need in two "pieces"--part in the cloud and part on my own server.

UPDATE: I greatly improved the following on August 28. I eliminated all "tail chasing." I also wrote up new notes in a new blog entry.

Here is an attempt at an edited version of my Asterisk install command history. One important note is that some of that was probably tail chasing versus:
sudo ./install_prereq
sudo ./install_prereq install

Then I changed 4 config files.

Probably more to come, but I have an apprentice live right now reading this.

AWS

In almost all cases, the AWS documentation is excellent. In this case, I chased my tail around. In the end, I got somewhat lucky. Of all the weird things, I have the darndest time finding the right AWS console. The link is for the AWS Chime product including "voice connectors." So THERE is the console link.