Following the guidelines from various sources:
Note that in the second howto, replace KERNEL with DEFAULT, remove APPEND and put the content of those 2 line in to one:
KERNELDEFAULT vmlinuz-4.4.0-139-generic APPENDroot=/dev/nfs ....
a. I already have a router, which is a nice Ubiquiti USG-PRO-4, and it has support of PXE.
b. I don't want to use 2 Raspberry Pis, i want to setup the DHCP and TFTP on the Ubuntu VM.
I have encountered a few roadblocks initially...
1. Setup a DHCP server without interfering with my router (Ubiquiti USG-Pro-4)
2. Serve different boot config to different hosts, by their MAC addresses
3. Serve different bootloader to different hosts, by their MAC addresses
Problem 1 was solved swiftly (seemingly, maybe it'll screw my network after previous DHCP lease expiring?) from installing the isc-dhcp-server in the second howto. but at least I set it up to only work for certain MAC addresses.
Problem 2 should be fixed by the config itself, at least from this document. I have not test this yet.
Problem 3 is the one that I am struggling now, as the howto is aimed at x86 hosts, which Raspberry Pi is not. I presume it should be easily configured if the isc-dhcp-server is the master dhcp server for my home network. Or I pull out another Raspberry Pi and strictly follow the first howto, or I figure out how to use the advance DHCP options in the router.
My thought from setting these things:
1. PXE boot would speed up spinning up new metal hosts, nowadays maybe they are used for setting up Kubernetes Nodes as we move to a docker world?
2. In real world applications, if you need to setup PXE, you probably should also have a dedicated DHCP/BOOTP server, so it is easier to configure it without messing with your existing networking.
3. Also, I believe it is unlikely that in a real world to have a PXE to support heterogeneous hardware architecture as it would complicate things and make sysadmin's life harder?
On a side note, that Ubuntu VM that runs in bhyve in my 3 years old FreeNAS, is starting to look like a single point of failure.