"}
24 | $devices = $devices | Where-Object {$_.pathselectionpolicy -like 'vmw_psp_rr'}
25 | foreach ($device in $devices)
26 | {
27 | # Create an argument's list to invoke against the ESXCLI
28 | $d = @{}
29 | $d['iops'] = 1
30 | $d['device'] = $device.Device
31 | $d['type'] = 'iops'
32 | $result = $cli.storage.nmp.psp.roundrobin.deviceconfig.set.Invoke($d)
33 | Write-Host $vmhost.name $device.device $result
34 | }
35 | }
36 | ```
37 |
38 | ### esxcli
39 |
40 | I discovered ESXCLI years ago. It's great. It gives you access to the deepest levers and buttons on ESXi but over remote. No SSH or anything necessary. The only wonky part that takes some getting used to is creating the arguments. There's more than one way to skin the cat, so please don't judge my clunky method too harshly. The key takeaway is that you compose the argument in the form of a hashtable and then send it with the "invoke" method.
41 |
42 | ### devices
43 |
44 | Your devices should have some naming convention or other identifying thing. In most cases, the `DeviceDisplayName` will include the vendor name of the LUN. This can be a great way to filter out LUNs that you don't want to monkey with. Generally speaking, though, you can probably just use the `vmw_psp_rr` filter. You can assume that any LUN with Round Robin might be affected. Though you might have multiple arrays on the backend. Do whatever you want here!
45 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-01-singularity-by-2030.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "Singularity by 2030?"
4 | date: 2020-10-01
5 | description: I think we will see the singularity by 2030...
6 | categories: [Singularity, Deep-Learning, Quantum-Computing, Automation]
7 | ---
8 |
9 | # What is the singularity?
10 |
11 | Before I can make any assertions about an event, I have to define the event beyond just a buzzword label. The [technological singularity](https://en.wikipedia.org/wiki/Technological_singularity) is a hypothetical event whereby technological advancements compound and snowball.
12 | By this definition, you could easily assert that we're already in the singularity. Computers, for instance, accelerate business, research, and technological deployment. This, in return, accelerates the advancement of computers. It's a positive feedback loop.
13 |
14 | So what sets apart *The Singularity* from today?
15 |
16 | My personal definition is: *When my life drastically changes.*
17 |
18 | Sophisticated, I know. But there's some value in evaluating global technological shifts from the subjective perspective. The singularity is fundamentally about disruption. Disruption of economics and commerce, disruption of lifestyles and life trajectories.
19 |
20 | # Key disruptors
21 |
22 | ## Deep learning
23 |
24 | The internet is presently losing its composure over GPT-3. An old college buddy of mine works at OpenAI and, almost two years ago, he told me that they were getting to the point of fundamentally understanding the nature of knowledge and intelligence. I didn't really believe him at the time, but I also didn't understand the groundbreaking Transformer architecture upon which GPT-3 is based.
25 |
26 | I first got my hands on a Transformer in the form of Google's [Universal Sentence Encoder](https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/3). This technology alone will disrupt the world once it gets fine-tuned and deployed globally. How? Why? That's a post for another time. Suffice to say, I now agree with my buddy - we are starting to grasp the fundamental nature of intelligence.
27 |
28 | I have been listening to podcasts and YouTubers talk about GPT-3 and even a lot of tech savvy folks can't seem to make heads or tails of it. They make assertions like *oh, this is just some sort of compression*. Is it actually processing and *thinking*? Or is it just storage?
29 |
30 | It's actually *both*.
31 |
32 | I first started tinkering with neural networks way back in 2009 and 2010, before the current wave of deep learning. Back then, I was experimenting in C++, manually creating what are now known RNNs and such. I had no idea what I was doing and was way in over my head, but it's nice to know that I was just a couple years ahead of the curve. During that time, I had some fundamental realizations about neurons (biological and artificial). Neurons (or nodes) are responsible for both memory and processing.
33 |
34 | For conventional computing, it's difficult to grok the idea that a unit does both storage and computing. But that's just how it is. There is no hard drive in your brain, no CPU. You have gray matter (the "thinking" neurons) and white matter (the wiring) and a few regions that tend to specialize. None of those regions specialize in either processing or knowledge, though some do specialize with forming cohesive memories.
35 |
36 | But I digress. The point is that deep learning can and will disrupt pretty much every domain of human expertise.
37 |
38 | ### Medicine
39 |
40 | Diagnosing patients is fundamentally a pattern matching problem. Deep learning is the most sophisticated pattern-matching tool we have, regularly outperforming humans once you have the right training data. I suspect that we will see early disruption in rural and poor regions of the world where need for medical care trumps the status quo. With the launch of Starlink, everywhere in the world will have access to quality, reliable internet. This will create a huge market for cloud-based medical diagnostic platforms.
41 |
42 | A traveling physician with the most basic training in the middle of the Sahara will be able to upload pictures and patient history to such medical portals via Starlink and rapidly get back a list of most likely diagnoses as well as follow-up tests that can nail down the diagnosis. Think about Star Trek medical tricorders. That kinda thing. Wave a bluetooth smartphone attachment over someone and, within moments, get a detailed and sophisticated state of their health.
43 |
44 | Will this technology cost people jobs? I think so. Physicians and specialists are expensive. I suspect that hospitals will find that they can have fewer physicians on staff and go with cheaper labor, such as nurses and physician's assistants who can leverage these cloud-based services to create better, more consistent care for cheaper.
45 |
46 | ### Transportation
47 |
48 | I was getting an Uber back from the car dealership a few weeks back and was quietly contemplating how my driver would be out of a job within 5 years. Uber, Tesla, Lyft, Ford, and pretty much every other major manufacturer are working tirelessly to bring full self-driving vehicles to the market. These vehicles will ultimately be safer and cheaper than anything on the road today, by a long shot.
49 |
50 | Economic forces will mandate that these technologies be adopted, though the entrenched industries will resist it. Still, there are something like 7 million drivers in America alone who will be out of work within a decade. When combined with electric vehicles, we are going to see some huge knock-on effects, such as a collapse in demand for automotive mechanics and parts, since EVs are far more reliable and cheaper to maintain. Though, this lower cost could be offset by unment demand for transportation. Cheaper transport could result in vastly more passenger-miles per year.
51 |
52 | Transportation costs will drop drastically, which will probably cause people to travel more, but will also reduce the cost of many goods, including materials for homes.
53 |
54 | ### Science
55 |
56 | Embedding true knowledge and intelligence into tireless machines is already starting to revolutionize science. Deep learning is already aiding in drug discovery as well as fundamental physics at places like CERN and the LHC. As these tools become more commercialized, we will see every researcher benefitting from deep learning. If nothing else, it will help consolidate the state of the industry, reading every paper in existence and summarizing it, allowing researchers to focus on the gaps and also to keep up to date much quicker.
57 |
58 | As deep learning proliferates, science will accelerate in most areas, creating compounding returns for better computers, better AI, and the snowball will continue. We're already seeing deep learning being used to streamline data acquisition, used to create better models. This can be see in concepts like *AutoML*.
59 |
60 | ## Quantum computing
61 |
62 | IBM, Google, and D-Wave are all currently locked in an arms race for *Quantum Suppremacy* (which sounds cyberpunk AF). I don't think I can overstate how disruptive the proliferation of quantum computing will be. You can already sign up and use [quantum computers as a cloud service](https://quantum-computing.ibm.com/).
63 |
64 | What is the bottom line?
65 |
66 | Quantum computing excels at difficult problems. I won't explain how it works but you can check out [this great video from IBM](https://www.youtube.com/watch?v=zOGNoDO7mcU) for a deeper dive. Quantum computing is hundreds of millions times faster at some problems than conventional computing. I suspect, as qubits increase, we will see quantum computing become billions or trillions of times faster.
67 |
68 | ### Optimization
69 |
70 | Maybe you've heard the "traveling salesman" problem. Maybe you haven't. What you might not have heard, though, is that training AI is all about optimization. OpenAI allegedly spent about $7 million just to train GPT-3 in cloud computing. That is prohibitively expensive, obviously. What if the training process of advanced AI models dropped precipitously? What if the most process-intense parts of optimizing a deep neural network became millions of times more efficient? Billions? Within 10 years, I think that training the equivalent of GPT-3 will be trivial, costing pennies rather than millions of dollars.
71 |
72 | What happens when literally everyone can train something as powerful as GPT-3 in an afternoon?
73 |
74 | It's hard to even wrap my head around this possibility. Anything that you have enough data for, or a good enough simulation environment, can be automated to super-human levels in an afternoon. The economic value here, let alone the disruption to everyday life and work, is incomprehensible. This is why the biggest companies in the world are locked in this arms race.
75 |
76 | ### Simulation
77 |
78 | Right now, the largest networks of computers, such as those for Folding@Home and those used to decode LHC output, are slow and expensive. Ludicrously expensive. These kinds of networks have been used to help develop the COVID-19 vaccines.
79 |
80 | What if the entire backlog of F@H could be done in a few days? What if new vaccine and drug candidates can be tested in massive batches? Thousands of candidates tested against tens of thousands reactions? What if you can then simulate the entire genome of a patient and their drug interactions during an office visit?
81 |
82 | This is not even entirely hypothetical, quantum computing is already being used in material science to help create the next generation of lithium batteries.
83 |
84 | # What life might be like in 2030
85 |
86 | ## Where we live
87 |
88 | Between Starlink, remote work, and job destruction, I suspect the current exodus from cities will explode. I think we will see more and more communities popping up in the cheapest places to live, such as the mostly empty central states. Self-driving electric vehicles are expected to reduce the cost of passenger-miles to 5 cents or less. You'll be able to summon a car to take you from your home to anywhere for just a few dollars. I expect a lot of people will also switch to a more rural, homestead style of living without their conventional jobs. People still need occupations and UBI will create a safety net.
89 |
90 | Beyond that, I suspect economic pressures will force more people to cohabitate. I anticipate a rise in intentional communities and cohousing projects. We may very well find ourselves living in eco-villages before too long.
91 |
92 | ## Where we work
93 |
94 | Honestly, I think that unemployment will skyrocket by 2030. Simply put, many people will be unemployable. This is not due to personal failing or laziness. It will simply be due to the fact that people cannot compete with the kinds of AI and automation I've outlined above. Only those who are capable of gaining domain expertise will be able to find jobs conventional jobs.
95 |
96 | I do suspect the gig economy will continue, though. Homestead life will see more people switching back to gardening, crafts, and other more personal services as a way to make some extra money. Think of things like childcare, pet grooming, and other domestic service. A good friend of mine is a massage therapist - I suspect she will be immune to job losses due to the singularity.
97 |
98 | If I do lose my job to the singularity, I plan to focus on writing.
99 |
100 | ## Health and medicine
101 |
102 | With a mass exodus from cities, the introduction of UBI, and hopefully universal healthcare, we will probably be under far less stress as a society. One can only hope! Beyond that, I suspect that the combination of quantum computing and deep learning will result in massive breakthroughs in regenerative medicine, de-aging medicine, and remediations for chronic conditions. We're already seeing the beginnings of some of this with nanotech based medicine.
103 |
104 | Taken all together, I think we will be far happier and healthier in 10 years, and we will probably be approaching *indefinite lifespan*. Indefinite lifespan is the idea that all common causes of death will be "solved". Infectious disease will be a thing of the past as well as chronic disease.
105 |
106 | Back around 2010 is when I started paying attention to stem cell therapy and regenerative medicine in general. I predicted then that we would see the first major breakthroughs by 2020. Today, there are a handful of therapies going through clinical trials. The market for regenerative medicine is huge. The first to market will make a bloody fortune, so you can bet your biscuit that someone is working on it!
107 |
108 | A series of implants and/or external sensors will be able to evaluate your health on an ongoing basis. Heart attack, stroke, and aneurysms are gone for good. Early detection of and outright prevention of all cancer will be in place.
109 |
110 | ## Entertainment
111 |
112 | GPT-3 can procedurally generate just about any text, this technology will only get better with time. Imagine you fire up your Nook or Kindle and say you want to read a brand new high fantasy story. *I want to read something like Game of Thrones*, you say to your device. It goes and thinks for a moment and generates the first page of a story. You start reading and, as you do, biometric sensors track your response to what you're reading. When you finish each page, the next one is generated on the fly based upon your reaction to the previous page. You read and entire story that is custom-tailored to your exact preference and reading level.
113 |
114 | Take that one step further.
115 |
116 | Let's imagine that the fourth or fifth generation of GPT technology can generate movie and TV scripts in real-time. Then you have some other generative deep learning models that can translate the screenplay directly into audio and video. Yes, I'm saying that you'll get highly customized TV shows and movies generated in near real-time just for you!
117 |
118 | Some commentators on Reddit suspect that Microsoft licensing GPT-3 means that videogames will have procedurally generated dialog before too long. Take that to a few more evolutions and you'll get entire games that are procedurally generated.
119 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-06-enumerate-esxlunpaths.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "My Enumerate-EsxLunPaths function"
4 | date: 2020-10-06
5 | description: Get-ScsiLun is slow, this way is much faster!
6 | categories: [VMware, ESXi, Storage, PowerCLI, PowerShell, KB]
7 | ---
8 |
9 | # The fastest way to get all LUN paths in vSphere
10 |
11 | The larger your environment gets, the more unwieldy it can become. Ideally, you have good process controls in place so only expert hands are modifying the storage environment, but sometimes mistakes get made even by the most veteran engineers. In worst case scenarios, you have very low level techs with insufficient training and too much pressure to just close tickets without checking their work. If you've never worked in such an environment, pray that you don't! But since I have in the past, I developed the fastest way to quickly check all LUN paths for ESX hosts.
12 |
13 | ## Case 1: Needle in a haystack
14 |
15 | For whatever reason, you're looking for a few paths that are down and you need to correlate it back to an NAA ID or a datastore. For those of you that have tried to do this through the GUI I can already hear you groaning. This is the kind of thing that you pitch over the wall for the storage guys to track down, or else the FNG on your team. But with this function, you can easily iterate through all your hosts and look for any paths that are not in the desired state.
16 |
17 | ## Case 2: Daily health report
18 |
19 | One time I worked from dawn until dusk for a week straight because of a neglected storage environment. It turns out that if ESXi cannot use a path, it will keep trying to bring the path up. I'm paraphrasing poorly but this was a behavior change between ESXi 5 and ESXi 6. My organization at the time had decided to be rather lazy when it came to deprovisioning LUNs so there were a LOT of orphaned paths out there that our hosts were dutifully trying to reconnect to. It turns out that if this goes on long enough, or you get enough orphaned paths, you can start crashing services! We had hosts dropping into disconnected state randomly.
20 |
21 | After that, I figured out that a daily health report, looking for orphaned paths was a way to go. This has become a staple for me. I want to know the state of all paths in my environments at all times. I trust my storage guys but I also like to verify. Plus, the richness of the information provided by this function makes troubleshooting and narrowing it down a breeze.
22 |
23 | ## Case 3: A jumping-off point
24 |
25 | In another case, I had a client environment that was showing periodic storage latency issues. I used this function to rapidly enumerate all LUNs on each host and piped that into `Get-Stat` and created an hourly report to see which LUNs were showing increases in latency and throughput.
26 |
27 | # The Code
28 |
29 | Without further ado
30 |
31 | ```powershell
32 | function Enumerate-EsxLunPaths
33 | {
34 | param($vmhost)
35 | $data = @()
36 |
37 | # This is the meat and potatoes here!
38 | # These 4 lines are lightning fast and gather all the information you need
39 | # The rest of the script is just massaging it into an object to return
40 | $storage_view = Get-View $vmhost.ExtensionData.ConfigManager.StorageSystem
41 | $luns = $storage_view.StorageDeviceInfo.ScsiLun
42 | $multipathdata = $storage_view.StorageDeviceInfo.MultipathInfo.Lun
43 | $mounts = $storage_view.FileSystemVolumeInfo.MountInfo | Where-Object {$_.Volume.Type -like "vmfs"}
44 |
45 | foreach ($mount in $mounts)
46 | {
47 | try
48 | {
49 | $extent = [string]$mount.Volume.Extent.DiskName
50 | $lun = $luns | Where-Object {$_.displayname -like "*$extent*"}
51 | $paths = ($multipathdata | Where-Object {$_.Lun -eq $lun.Key}).Path
52 |
53 | foreach ($path in $paths)
54 | {
55 |
56 | # This is where I create an object to capture all the information
57 | $info = "" | Select-Object vol_name,vol_path,vol_uuid,vol_extent,lun_uuid,lun_device,lun_key,lun_name,lun_vendor,path_name,path_state,path_working
58 | $info.vol_name = $mount.Volume.Name
59 | $info.vol_path = $mount.MountInfo.Path
60 | $info.vol_uuid = $mount.Volume.Uuid
61 | $info.vol_extent = $extent
62 |
63 | $info.lun_uuid = $lun.Uuid
64 | $info.lun_device = $lun.DeviceName
65 | $info.lun_key = $lun.Key
66 | $info.lun_name = $lun.CanonicalName
67 | $info.lun_vendor = $lun.Vendor
68 |
69 | $info.path_name = $path.Name
70 | $info.path_state = $path.State
71 | $info.path_working = $path.IsWorkingPath
72 |
73 | $data += $info
74 | }
75 | }
76 | catch { continue }
77 | }
78 | return $data
79 | }
80 | ```
81 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-07-deep-learning-firefly.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "How deep learning could give us more Firefly"
4 | date: 2020-10-07
5 | description: Take my love, take my land. Take me where I cannot stand.
6 | categories: [Singularity, Deep-Learning, Entertainment]
7 | ---
8 |
9 | We all want more Firefly, so let's do a thought experiment as to how deep learning could achieve this!
10 |
11 | # The Screenplay
12 |
13 | Technologies like GPT-3 show that we now have deep learning models that can generalize and reproduce a broad variety of text outputs. For instance, [GPT-3 allegedly went undetected on r/AskReddit](https://www.kmeme.com/2020/10/gpt-3-bot-went-undetected-askreddit-for.html) for a week or so.
14 | GPT-3 is what's known as a "one shot" or "few shot" technology, where it has baked in ability to recognize the type of output you want from a single example. So let's say we fine tune GPT-3 with a bunch of TV show scripts and sci-fi novels and then show it the actual original Firefly screenplay and see what it produces.
15 | "Fine tuning" is something you can do on GPT-2 whereby you [add more data to make it more purpose-built for your particular task](https://openai.com/blog/fine-tuning-gpt-2/). Fine-tuning allows you to specialize a big giant model without starting from scratch every time.
16 |
17 | Stay tuned, I think I will give this a shot on my own sometime soon! If I get around to it, I will post an update here, with the results and the code!
18 |
19 | EDIT: [Oh look, someone already did it!](https://towardsdatascience.com/film-script-generation-with-gpt-2-58601b00d371). There's a lot of attention on [GPT-3 for this task already](https://www.gwern.net/GPT-3)
20 |
21 | ## What a screenplay looks like
22 |
23 | Just for reference, screenplays are highly standardized with very specific syntax and formats so they are universally accessible and interpretable by directors and producers. They include descriptions, actions, and dialog.
24 |
25 | [Here's an example of what one looks like on paper](https://www.raindance.org/scripts/Firefly_1x02_-_Bushwhacked.pdf)
26 |
27 | # The Video
28 |
29 | First we need to break it down into chunks. We don't need to use a deep neural network to make an entire 44 minute episode. Episodes are broken down into scenes, and scenes are broken down into cuts, or "transitions".
30 | Everytime the camera cuts away to a new perspective is a film segment. Some directors and producers make use of rapid cuts, meaning we would only need to generate a few seconds of video at a time, and then stitch it together.
31 | Firefly, which I just rewatched with my partner, seems to be pretty standard. A long cut in that show would be 30 to 60 seconds, but most are shorter, close up of faces and dialog with some action sequences, establishing shots of Serenity and scenery.
32 | [DVD-GAN](https://medium.com/syncedreview/deepmind-dvd-gan-impressive-step-toward-realistic-video-synthesis-12027d942e53) is already on the way to full video synthesis.
33 |
34 | ## Scenery and Settings
35 |
36 | GANs (Generative Adversarial Networks) have become exceptionally good at [generating realistic images](https://www.marktechpost.com/2020/10/06/nvidia-releases-imaginaire-a-universal-pytorch-library-designed-for-various-gan-based-tasks-and-methods/) from basic information.
37 | NVIDIA released a library called Imaginaire that attempts to standardize this technology, making it more and more accessible. I think it's only a matter of time before this increases in sophistication and quality.
38 | Right now, [text-to-image](https://deepai.org/machine-learning-model/text2img) technology leaves a bit to be desired! As GPUs get more powerful and data increases, we will inevitably see better models.
39 |
40 | ## Characters
41 |
42 | We humans (and yes, I'm a human) are finely calibrated to recognize faces. The uncanny valley has been the death of many early technologies, from CGI to video games. Sites like [This Person Does Not Exist](https://thispersondoesnotexist.com/), however, demonstrate that we are well past the uncanny valley of face generation.
43 | Nathan Fillion has already been [deep-faked into live-action footage](https://www.eurogamer.net/articles/2020-05-02-uncharted-4-deepfake-starring-nathan-fillion-is-as-impressive-as-it-is-scary). So why not Firefly?
44 |
45 | # The Audio
46 |
47 | ## Speech
48 |
49 | Text-to-speech is nothing new. The latest and greatest speech synthesis adds inflection, tone, and style tags. The subtle quality of human emotion poured into speech can be summed up as "prosody". [Here's a creepy-realistic example of prosody embedding!](https://ai.googleblog.com/2018/03/expressive-speech-synthesis-with.html)
50 |
51 | ## Sound Effects
52 |
53 | Back in 2016, [MIT published some work](https://www.wired.com/2016/06/mit-artificial-sound-effects/) about natural sound generation for video. Today, Adobe can provide this as a service, and it's eerily high quality. Seriously, check out [this train](https://research.adobe.com/news/ai-can-generate-realistic-sound-for-video-clips/).
54 |
55 | ## Music
56 |
57 | Synthetic music is also nothing new, it's just getting [way better](https://towardsdatascience.com/neuralfunk-combining-deep-learning-with-sound-design-91935759d628). And today, music AI is getting to an entirely new [level of beautiful](https://www.zmescience.com/science/ai-musical-composer/).
58 |
59 | # Implications
60 |
61 | ## Netflix and GAN?
62 |
63 | So what does this mean? I think the only logical conclusion is we're going to see a massive explosion of consumer media. If they aren't already, I suspect that Netflix and Amazon are hard at work creating fully synthetic consumer media. We'll probably see books and short stories first, but it's only a matter of time before that morphs to TV and movies. Imagine this: Netflix creates a library full of tens of thousands of movies and shows, all procedurally generated and rated by the masses. Those that are good percolate up. They are filled with actors that never existed, written and produced by directors who never existed. This level of automated media production is still prohibitively expensive. It took millions of dollars just to train GPT-3, which can only do generalized text tasks. It will take something a bit more powerful and sophisticated to do high quality TV screenplays and 4k 60fps film.
64 |
65 | ## IP Laws
66 |
67 | I haven't the foggiest clue as to how this is going to play out. I suspect that IP (intellectual property) laws will say that AI models trained in-house are propriety and that any data they use is part of the model. Thus, they will need rights or license to train on other TV and movies. You can't just copy the screenplays of every Marvel movie and not expect Disney to sue your pants off! Even if an AI is then just learning from the screenplay, the same that another director might. I could see that going to the Supreme Court.
68 |
69 | ## Personalized TV, Movies, Music, and Books
70 |
71 | As GPU technology advances, which it is rapidly due to demand, it will become cheaper and cheaper to train giant models. My personal desktop computer, with its NVIDIA RTX 2070, is more powerful than ASCI Red, the top supercomputer from 1997. Commercial industry is usually about 10 years behind the cutting edge of massive computing power and private homes are about 20 years behind. It took supercomputer level processing to train GPT-3, so we can expect run-of-the-mill businesses to be able to do that by 2030, and everyone else by 2040.
72 | As the cost of producing these giant models comes down, and the quality of data and output increases, we will soon see hyper-personalized entertainment. Want more Firefly? Just ask Netflix or Amazon. Want more Game of Thrones, except the ending is way different? That could be possible, too!
73 | And I don't mean recommender systems, either. I mean stuff that is generated *just for you*.
74 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-12-vmware-network-disk-throughput.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "Monitoring VMware VM and ESXi Throughput"
4 | date: 2020-10-12
5 | description: Network and disk IO can kill your environment
6 | categories: [VMware, ESXi, Storage, PowerCLI, PowerShell, Monitoring, UCS, KB]
7 | ---
8 |
9 | As a VMware admin or engineer, I'm sure you know that the first thing people ask about when VM performance is slow is for more CPU and RAM.
10 | It is true that there are some memory-hungry applications out there and right-sizing for CPU is critical. But these are only two of the four food groups of computing.
11 | Just as critical are network and storage, and in datacenter environments, both are subject to shared networks in the form of TCP/IP, FC, SAN, and FCoE. By extension, they can be subject to latency and congestion problems.
12 | Most people's experience with computers is with local storage, which is almost never going to be a bottleneck. Even more confounding is the fact that a bottleneck on any one of these resources can appear to impact performance elsewhere.
13 | For instance, if storage is too slow, you might start slamming your CPU and memory as SQL queries pile up. Or if you're short on RAM, you might start swapping to disk, slamming storage speed.
14 |
15 | # VM Throughput Report
16 |
17 | Here, I gather usage statistics for a period of 24 hours and find the peak usage. I then pare it down to the top talkers in each category.
18 | This kind of report is helpful to have on hand in case something changes, you can just glance at the report every morning to see if a new VM is spiking.
19 | More often than not, this will help you identify problematic VMs before they percolate up to the awareness of management, leadership, and even other teams.
20 | I cannot tell you how many times I've spotted an errant database or backup server before anyone else noticed it and headed off a high visibility issue.
21 | An ounce of prevention is worth a pound of cure! The ideal crisis is the one that never happens!
22 |
23 | ```powershell
24 | $vms = Get-VM | Where-Object {$_.PowerState -like "*on*"} | sort-object
25 | $data = @()
26 |
27 |
28 | foreach ($vm in $vms)
29 | {
30 | $net = $vm | Get-Stat -Stat net.usage.average -Start (Get-Date).AddDays(-1) -Finish (Get-Date) -MaxSamples 5000 | Sort-Object -Property value -Descending | Select-Object -First 1
31 | $disk = $vm | Get-Stat -Stat disk.usage.average -Start (Get-Date).AddDays(-1) -Finish (Get-Date) -MaxSamples 5000 | Sort-Object -Property value -Descending | Select-Object -First 1
32 | $info = "" | Select-Object vm,max_net_kbps,max_net_time,max_disk_kbps,max_disk_time
33 | $info.vm = $vm.Name
34 | $info.max_net_kbps = $net.Value
35 | $info.max_net_time = $net.Timestamp
36 | $info.max_disk_kbps = $disk.value
37 | $info.max_disk_time = $disk.timestamp
38 | $info | fl
39 | $data += $info
40 | }
41 |
42 | Clear-Host
43 |
44 | $data | Sort-Object -Property max_net_kbps -Descending | Select-Object vm,max_net_kbps,max_net_time -First 50 | ft
45 |
46 | $data | Sort-Object -Property max_disk_kbps -Descending | Select-Object vm,max_disk_kbps,max_disk_time -First 50 | ft
47 | ```
48 |
49 | At the end you can do whatever you want with the data. I send it out via email once a day. Usually, I'm really lazy and just do `Out-String` and set the output into `` tags in an HTML email. It ain't pretty but it's fast and valuable!
50 |
51 | I also recommend removing the additional console output if you do schedule this as an unattended job.
52 |
53 | # ESX Host Throughput
54 |
55 | This is the same exact information just with a different focus.
56 |
57 | ```powershell
58 | $vmhosts = Get-VMHost | Where-Object {$_.ConnectionState -like 'connected'} | Sort-Object
59 |
60 | $data = @()
61 |
62 | foreach ($vmhost in $vmhosts)
63 | {
64 | $net = $vmhost | Get-Stat -Stat net.usage.average -Start (Get-Date).AddDays(-1) -Finish (Get-Date) -MaxSamples 5000 | Sort-Object -Property value -Descending | Select-Object -First 1
65 | $disk = $vmhost | Get-Stat -Stat disk.usage.average -Start (Get-Date).AddDays(-1) -Finish (Get-Date) -MaxSamples 5000 | Sort-Object -Property value -Descending | Select-Object -First 1
66 | $info = "" | Select-Object vmhost,max_net_kbps,max_net_time,max_disk_kbps,max_disk_time
67 | $info.vmhost = $vmhost.Name
68 | $info.max_net_kbps = $net.Value
69 | $info.max_net_time = $net.Timestamp
70 | $info.max_disk_kbps = $disk.value
71 | $info.max_disk_time = $disk.timestamp
72 | $info | fl
73 | $data += $info
74 | }
75 |
76 | Clear-Host
77 |
78 | $data | Sort-Object -Property max_net_kbps -Descending | Select-Object vmhost,max_net_kbps,max_net_time -First 20 | ft
79 |
80 | $data | Sort-Object -Property max_disk_kbps -Descending | Select-Object vmhost,max_disk_kbps,max_disk_time -First 20 | ft
81 | ```
82 |
83 | You can add other helpful information such as Cluster if you like.
84 |
85 | # Datastore Latency
86 |
87 | Storage latency is one of those things that is horribly counter-intuitive to pretty much everyone except virtualization and storage folks, and some cross-pollinated network folks.
88 | App folks, developers, and DBAs tend to not grasp this problem unless their domain deals specifically with storage technologies. This is nothing against them - we all have domains of expertise for a reason.
89 | Datastore latency is one of those more arcane metrics that is harder to get but incredibly critical when you need it. Hence this script!
90 |
91 | IMPORTANT: This script relies upon another function I have documented here: [Enumerate-EsxLunPaths](https://daveshap.github.io/DavidShapiroBlog/2020/10/06/enumerate-esxlunpaths.html)
92 |
93 | ```powershell
94 | $stat_list = "disk.deviceReadLatency.average","disk.deviceWriteLatency.average"
95 | $host_data = @()
96 |
97 | foreach ($cluster in Get-Cluster)
98 | {
99 | $vmhosts = $cluster | Get-VMHost | Where-Object {$_.connectionstate -like "connected"}
100 | foreach ($vmhost in $vmhosts)
101 | {
102 | $stats = $vmhost | Get-Stat -Realtime -Stat $stat_list
103 | $stats = $stats | Where-Object {$_.Value -gt 200} # change this threshold to squelch noise
104 | if ($stats -eq $null) { continue }
105 | $lun_data = Enumerate-EsxLunPaths $vmhost
106 | foreach ($stat in $stats)
107 | {
108 | $lun = $lun_data | Where-Object {$_.lun_name -like $stat.Instance} | Select-Object -First 1
109 | $info = "" | Select-Object host,cluster,max_latency_ms_15_min,datastore,device_id
110 | $info.host = $vmhost.Name
111 | $info.cluster = $cluster.name
112 | $info.max_latency_ms_15_min = $stat.value
113 | $info.datastore = $lun.vol_name
114 | $info.device_id = $stat.instance
115 | $info | fl
116 | $host_data += $info
117 | }
118 | }
119 | }
120 | ```
121 |
122 | The biggest problem with this set of stats is that they expire very quickly so you may need to set this to run repeatedly, regularly checking for problematic LUN paths.
123 | If you've ever had a major issue with storage latency, I'm sure you're already salivating. This script relies upon my `Enumerate-EsxLunPaths` function, which is lightning fast.
124 | You can run this script on demand or as a scheduled task. Generally, I have used it as an on-demand tool to help troubleshoot big issues while on calls real-time.
125 | You're most likely to need this particular gem when backup jobs are taking too long or Oracle queries are bogging down.
126 |
127 | # BONUS: UCS Statistics one-liner!
128 |
129 | Okay, so you know how VMware basically nests the OSI model inside the OSI model? Cisco UCS takes that to the *nth* degree by abstracting all storage and networking over the IOM uplinks.
130 | I once discovered just how crazy this is when a network engineer discovered one particular IOM port was saturated during backup jobs. I traced it down and discovered that `Chassis/FEX Discovery Policy` was not set to `Port Channel`, which meant that every other server was pinned to a given IOM uplink on one side. Yikes!
131 | It had taken a few weeks of complaints getting louder and louder until we discovered that **storage**-intensive backup jobs were trashing **network throughput** because of this shared nature of UCS.
132 |
133 | Without further ado, the magical command that can enumerate more UCS statistics than you ever wanted!
134 |
135 | ```powershell
136 | Get-UcsStatistics -Current | Where-Object {$_.Rn -like "tx-stats"}
137 | ```
138 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-14-powershell-email-functions.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "PowerShell Email Functions"
4 | date: 2020-10-14
5 | description: Since I'm a monitoring addict, I use email reports a lot...
6 | categories: [PowerShell, Email, Monitoring, KB]
7 | ---
8 |
9 | These are some functions I keep in a PowerShell module on my Rundeck server for easy reuse. I can import the module at the beginning of jobs like so:
10 |
11 | ```powershell
12 | Import-Module -Force -DisableNameChecking "C:\\EmailFunctions.psm1"
13 | ```
14 |
15 | # Send HTML Email
16 |
17 | For the most part, you will want some formatting, which means you need HTML support. In my experience, most organizations have internal SMTP relays that will relay to local (domain) addresses without authentication. This is pretty typical as many servers, applications, and hardware support email notifications but many do not support SMTP authentication.
18 |
19 | ```powershell
20 | function Send-EmailHtml
21 | {
22 | param($to, $from, $subject, $html, $smtp)
23 | $message = New-Object System.Net.Mail.MailMessage $from, $to
24 | $message.Subject = $subject
25 | $message.IsBodyHTML = $true
26 | $message.body = $html
27 | $smtp = New-Object Net.Mail.SmtpClient($smtp) # FQDN of your SMTP server or relay
28 | $smtp.Send($message)
29 | }
30 | ```
31 |
32 | # Make a pretty HTML table
33 |
34 | PowerShell already has a default function for converting data objects to HTML tables but it's ugly.
35 |
36 | ```powershell
37 | function Make-HtmlTable
38 | {
39 | param($data)
40 | $t_header = @"
41 |
54 | "@
55 | $table = $data | ConvertTo-Html -Head $t_header
56 | return $table
57 | }
58 | ```
59 |
60 | # Send email with attachment
61 |
62 | I have, on occasion, been asked to schedule reports for other people. Sometimes they don't want it in a pretty HTML table, they want it as an Excel doc or something. For that, I recommend [PowerShell Galleries ImportExcel](https://www.powershellgallery.com/packages/ImportExcel/7.1.1).
63 |
64 | ```powershell
65 | function Send-EmailAttachment
66 | {
67 | param($to, $from, $subject, $body, $attachment, $smtp)
68 | # attachment must be in the form of full file path to attachment
69 | $message = New-Object System.Net.Mail.MailMessage $from, $to
70 | $message.Subject = $subject
71 | $message.IsBodyHTML = $true
72 | $message.Body = $body
73 | $file = new-object Net.Mail.Attachment($attachment)
74 | $message.Attachments.Add($file)
75 | $smtp = New-Object Net.Mail.SmtpClient($smtp)
76 | $smtp.Send($message)
77 | }
78 | ```
79 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-16-hyperflex-rest-api-python.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "Monitoring Cisco HyperFlex REST API with Python"
4 | date: 2020-10-16
5 | description: I'm actually super salty about this
6 | categories: [UCS, HyperFlex, Python, Monitoring, KB]
7 | ---
8 |
9 | # The REST API Explorer
10 |
11 | I judge companies and products harshly by their documentation. API documentation is especially stringent because, well, an API is meant to be standardized. There's no excuse for sloppy REST API documentation.
12 |
13 | Cisco HyperFlex documentation... [leaves a lot to be desired](https://developer.cisco.com/docs/ucs-dev-center-hyperflex/#!connecting-to-the-hyperflex-rest-api-explorer). Seriously, just compare that to [Twitter's API documentation](https://developer.twitter.com/en/docs/twitter-api). It's pitiful.
14 |
15 | To make matters worse, the HyperFlex REST API apparently changes drastically from release to release so it's increasingly critical that you get familiar with the REST API Explorer. There is something to be said for having a robust API explorer, so they get some points back for that one. Still, you can see that I have a lot of debug output in these scripts, and I decided to keep them to show you just much effort it was to figure this out!
16 |
17 | # Authentication
18 |
19 | This is what I got to work after cobbling together some snippets from around the internet and gaining access to my own instance of HyperFlex. A few things to note: The `client_id` and `client_secret` are apparently mandatory but not really documented anywhere that I recall.
20 |
21 | ```python
22 | import requests
23 | import json
24 |
25 | def auth_to_hx(fqdn, username, password):
26 | url = 'https://%s/aaa/v1/auth?grant_type=password' % fqdn
27 | headers={'content-type':'application/json'}
28 | payload = {'username': username,
29 | 'password': password,
30 | 'client_id': 'HxGuiClient',
31 | 'client_secret': 'Sunnyvale', # this is the default, you can change it
32 | 'redirect_uri': 'http://%s' % fqdn}
33 | try:
34 | response = requests.post(url,headers=headers,data=json.dumps(payload),verify=False,timeout=40)
35 | if response.status_code == 201:
36 | #print('Login succeeded to', fqdn)
37 | return response.json() # this is your authentication token
38 | else:
39 | #print('LOGIN FAILED', fqdn.upper())
40 | #print(response.status_code)
41 | #print(response.text)
42 | return None
43 | except Exception as oops:
44 | #print('LOGIN FAILED', fqdn.upper())
45 | #print(oops)
46 | return None
47 | ```
48 |
49 | # Enumerate Clusters
50 |
51 | HyperFlex is organized in a few ways. There are the Clusters and the Platform. We will get to the Platform in a moment. The Clusters are the meat and potatoes, though. Pass the `token` you get from authenticating to this function.
52 |
53 | Note: The `timeout` option became necessary because sometimes HyperFlex doesn't respond as fast as you'd like. Sometimes it's lightning fast. I don't really know why there's variance.
54 |
55 |
56 | ```python
57 | def get_hx_clusters(fqdn, token):
58 | url = 'https://%s/rest/clusters' % fqdn
59 | #print(url)
60 | headers={'content-type':'application/json','Authorization':token['token_type'] + token['access_token']}
61 | response = requests.get(url,headers=headers,verify=False,timeout=40)
62 | if response.status_code == 200:
63 | return response.json()
64 | else:
65 | #print(response.status_code, response.text)
66 | return None
67 | ```
68 |
69 | Another note: The `cluster` object returns has child elements `/entityRef/id`. This is the `Cluster UUID` or `CUUID` that you will need to reference the cluster by later. Again - this was not documented anywhere! To make matters even worse, the CUUID needs to be URL encoded so you have to manually convert it back like so:
70 |
71 | ```python
72 | cuuid = cluster['entityRef']['id'].replace(':','%3A')
73 | ```
74 |
75 | Starting to see why I'm grumpy about the HyperFlex REST API?
76 |
77 |
78 | # Get Platform Alarms
79 |
80 | The Platform is the management/controller portion. As best I can figure, this is roughly analogous to vCenter in vSphere. It's similar to the methodology used in other hyperconverged systems, such as HPE Simplivity, where you've got a VM embedded that runs the software and manages the backplane. In UCS-world, the closest thing is likely UCSM that runs inside the Fabric Interconnects.
81 |
82 | The bottom line is that you will want to monitor the HyperFlex platform as well as the clusters/hosts. Most of the time there's going to be nothing reported, but that's okay. Still, I'd much prefer the old fashioned `Get-UcsFault` style. One request, all alarms. Period, end of story. Even VMware doesn't have anything that good. Alas, it seems like we were lucky only the once. Maybe I'll combine all these functions into a single `Get-HxFault`?
83 |
84 | The platform layer will return alarms dealing with things like the manager configuration, backups, and the like.
85 |
86 | ```python
87 | def get_hx_alarms(fqdn, token):
88 | url = 'https://%s/rest/virtplatform/alarms' % fqdn
89 | #print(url)
90 | headers={'content-type':'application/json','Authorization':token['token_type'] + token['access_token']}
91 | response = requests.get(url,headers=headers,verify=False,timeout=40)
92 | if response.status_code == 200:
93 | return response.json()
94 | else:
95 | #print(response.status_code, response.text)
96 | return None
97 | ```
98 |
99 | I actually don't know what these might look like because I didn't record the one I had, I just fixed it. If I recall correctly it was something like "email alerts not configured" or "phone home support unreachable". Those kinds of things.
100 |
101 | # Get Cluster Alarms
102 |
103 | Cluster alarms are separate from platform alarms, which are yet still separate from cluster health. I really wish it weren't so, but it is what it is. Cisco did a great thing by giving you all UCS faults in one endpoing via `Get-UcsFault` and I will forever be salty that no other platforms seem to work this way. I just want to monitor everything and fix things before they blow up. Why is that so hard?
104 |
105 | Okay, I'll quit whining. Here's the cluster alarms function. This will return things like host and VM level errors.
106 |
107 | ```python
108 | def get_hx_cluster_alarms(fqdn, token, cuuid):
109 | url = 'https://%s/coreapi/v1/clusters/%s/alarms' % (fqdn, cuuid)
110 | #print(url)
111 | headers={'content-type':'application/json','Authorization':token['token_type'] + token['access_token']}
112 | response = requests.get(url,headers=headers,verify=False,timeout=40)
113 | if response.status_code == 200:
114 | return response.json()
115 | else:
116 | print(response.status_code, response.text)
117 | return None
118 | ```
119 |
120 | This is what a cluster alarm might look like:
121 |
122 | ```json
123 | [ { "acknowledged": False,
124 | "acknowledgedTime": 0,
125 | "acknowledgedTimeAsUTC": "",
126 | "description": "Default alarm to monitor virtual machine memory usage",
127 | "entityName": "",
128 | "entityType": "VIRTUALMACHINE",
129 | "entityUuId": "vm-1111",
130 | "message": "Default alarm to monitor virtual machine memory usage",
131 | "name": "alarm-6.vm-1111",
132 | "status": "CRITICAL",
133 | "triggeredTime": 1002709437316,
134 | "triggeredTimeAsUTC": "2010-07-16T00:50:37Z",
135 | "uuid": "alarm-6!!Alarm!!alarm-6!!vm-3517!!VirtualMachine!!vm-3517"}]
136 | ```
137 |
138 |
139 | # Get Cluster Health
140 |
141 | Cluster health is fun. This primarily focuses on the storage replication status.
142 |
143 | ```python
144 | def get_hx_cluster_health(fqdn, token, cuuid):
145 | url = 'https://%s/coreapi/v1/clusters/%s/health' % (fqdn, cuuid)
146 | print(url)
147 | headers={'content-type':'application/json','Authorization':token['token_type'] + token['access_token']}
148 | response = requests.get(url,headers=headers,verify=False,timeout=40)
149 | if response.status_code == 200:
150 | return response.json()
151 | else:
152 | print(response.status_code, response.text)
153 | return None
154 | ```
155 |
156 | This is what it might look like:
157 |
158 | ```json
159 | { "dataReplicationCompliance": "COMPLIANT",
160 | "resiliencyDetails": { "dataReplicationFactor": "TWO_COPIES",
161 | "hddFailuresTolerable": 1,
162 | "messages": ["Storage cluster is healthy. "],
163 | "nodeFailuresTolerable": 1,
164 | "policyCompliance": "COMPLIANT",
165 | "resiliencyState": "HEALTHY",
166 | "ssdFailuresTolerable": 1},
167 | "state": "ONLINE",
168 | "uuid": "",
169 | "zkHealth": "ONLINE",
170 | "zoneResiliencyList": []}
171 | ```
172 |
173 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-18-predictions-self-driving-ev-cars.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "Some Predictions On Self-Driving EV Cars"
4 | date: 2020-10-18
5 | description: TLDR - lots of cost savings and job changes plus a bonus homestead house!
6 | categories: [Deep-Learning, Electric-Vehicles, Self-Driving]
7 | ---
8 |
9 | # Autonomous is Coming
10 |
11 | Fully automated cars are coming. It's not just Tesla. There are a number of other startups and small companies getting in the game. So are the giants.
12 | [General Motors](https://www.theverge.com/2020/10/15/21517833/cruise-driverless-cars-test-permit-california-dmv) has entered the field, as have the likes of Ford, Nissan, and Honda. The writing is on the wall: Fully autonomous cars are coming.
13 | The big buzz right now is on *robo-taxis*. These are autonomous fleets of cars that you summon via app just like Uber and Lift, except there's no human driver. Automobile-as-a-Service.
14 |
15 | The investment is there. We are now inexorably heading for a driverless future.
16 |
17 | # The Good
18 |
19 | ## Latent Demand
20 |
21 | Life is hard and getting harder for many folks. Reliable transportation is listed as a job requirement on many jobs and is simply outside of the reach of some people. Owning a car is expensive and complicated. Insurance, maintenance, inspections, taxes. It adds up.
22 | Even those of us with cars might sometimes question the reliability of our cars and avoid long hauls for fear of getting stranded somewhere.
23 |
24 | All that is about to change.
25 |
26 | Individual transit autonomous EVs are expected to get down to 5 cents per passenger-mile. That means a thirty-mile commute (sixty for round trip) will cost you a whopping $3 per day. Let's say that rate is absurdly low and off the mark by a factor of 3.
27 | That's still only $9 per day to get to and from work with no further risk or expenses. Even though these cars will be getting much more use, reliability is not a concern. Why? The car itself can request a replacement automatically if it breaks down.
28 | Worst case scenario: You summon another replacement with your phone.
29 |
30 | This will result in far more people gaining access to work that may otherwise be presently unavailable. Mass transit models of autonomous EVs are expected to get as low as 1 cent per passenger-mile. Suddenly transport is no longer a barrier for most people. I suspect th is will have many knock-on effects on the economy.
31 | Imagine dozing off every commute. Or reading, or continuing to work. Suddenly, a longer commute doesn't sound like a bad idea. Now you can buy a rural dream house and still have the urban job. Moving across the country is suddenly more feasible. Just summon a bigger truck!
32 |
33 | Stress will decrease for millions of people, access to goods, services, and jobs will increase. This is a huge win for everyone.
34 |
35 | ## Climate Change
36 |
37 | It goes without saying that getting off fossil fuels is a good thing. There are a few caveats, namely that we have to deploy enough renewables and batteries to go completely carbon neutral. That also means reducing the carbon intensity of producting renewable hardware and batteries.
38 | I consider this a problem that will inevitably be solved for economic reasons. Rare and conflict minerals are intrinsically more expensive. It just makes economic sense to find abundant alternatives.
39 |
40 | A more immediate impact will be increased cardiopulmonary health, particular in dense urban regions. This will decrease health costs and increase quality of life for millions of people.
41 |
42 | ## More Automotive Jobs
43 |
44 | American motorists presently drive about 13,000 miles per year on average. I think that could easily double due to cheaper and safer travel, as well as that aforementioned latent demand. More miles means more wear and tear. You know what else that means?
45 |
46 | More mechanics.
47 |
48 | A lot more. Sure, EVs are a bit cheaper to maintain, but EVs are going to rack up miles far faster than ICE vehicles. The only thing fundamentally different is the power train. Everything else is still just an ordinary car.
49 | Brakes and tires wear out. Electric windows and ACs fail. Windows and windshields break. Calipers and bearings all need replacing. While a lot of things will change with EVs, there's a lot that won't when you look at the nuts and bolts.
50 |
51 | The aforementioned latent demand, I think, will see the total passenger-miles per year skyrocket. The same thing happened when electricity became cheaper - people just used a lot more of it. I would argue that electricity still has a lot of unmet demand.
52 | I think that the EV repair shops are going to be hopping busy. This will bode will for the [presently-declining automotive tech industry](https://www.cnbc.com/2017/05/22/goldman-sachs-analysis-of-autonomous-vehicle-job-loss.html).
53 | By extension, I think this could be a boon to the autoparts industry as well.
54 |
55 | A more speculative new job that could be created would be remote drivers. These are folks who work in a remote call-center of sorts and can remotely control vehicles that are in distress.
56 |
57 | ## Childcare and School
58 |
59 | The NYC subway MTA says that children as young as 12 can ride by themselves. I suspect that EVs, with their plethora of cameras, could allow for children much younger to ride without supervision. This could be a game-changer for childcare and education.
60 | Instead of waiting for the schoolbus in the morning, a vehicle is summoned for the kids, taking them to the school that is best suited to their needs, not where they happen to be geographically constrained. This yields the possibility of building more schools in cheaper areas.
61 |
62 | An extension of this includes access to after-school programs. I think this will be particularly important for children in high-risk neighborhoods. Suddenly, it becomes a lot easier for a kid to stay somewhere safer and more accomodating for studying after school, rather than going straight home to the rougher neighborhoods.
63 | This extra mobility could very well be the difference in getting more children out of the cycle of poverty.
64 |
65 | ## Savings for Everyone
66 |
67 | People who own cars spend anywhere from 10% to over 30% of their income on their cars. Once car ownership becomes optional, that's a LOT of money that folks can spend elsewhere. That reallocation of funds will likely have dramatic effects on other segments of the economy.
68 |
69 | ## Safety for Everyone
70 |
71 | It's true that autonomous cars make mistakes. It's also true that they make far fewer mistakes than humans already, and that will only get better as the technology improves. Presently, about 38,000 people die each year from traffic accidents and another 4 million are seriously injured.
72 | That's a lot of avoidable death and injuries. Traffic fatalities and injuries are incredibly traumatic. They don't just create permanent loss of life and limb, but permanent emotional harm to survivors and family. I suspect traffic deaths will fall off exponentially over the coming years.
73 |
74 | # The Bad
75 |
76 | ## Millions of Jobs Gone
77 |
78 | There are currently about 4 million driving jobs in America. Machines tend to be more reliable and safer than humans once the kinks are worked out. They can also work tirelessly, hence why industrial robots have been assembling cars for several decades now. This reliability and safety translates to one thing: better bottom lines.
79 | Before long, human drivers are going to be antiquated - a privilege for concierge-level services only. Pizza delivery? Gone. Long-haul truckers? Gone. Amazon Prime drivers? Gone. FedEx, UPS, and USPS? All gone.
80 | It will become economically infeasible to hire humans for these jobs.
81 |
82 | Petroleum figures into a lot more than just automobiles. Figures vary wildly depending on who you ask, but the petroleum industry supports somewhere between 2 million and 10 million jobs. Any significant disruption in that domain can have huge knock-on effects.
83 | It's difficult to imagine a simple antidote to this problem. With the rise of automation, I suspect many of these folks will permanently transition out of the workforce. These jobs include oil and gas workers, as well as gas station employees.
84 |
85 | With so many people permanently unemployed, I don't see any solution other than redistribution of wealth via Universal Basic Income.
86 |
87 | ## Rare and Conflict Minerals
88 |
89 | Batteries, cameras, motors, and deep learning hardware all require some exotic elements to manufacture. Presently, that presents a huge humanitarian and moral dilemma. We don't want to fund crimes against humanity, slave labor, human trafficking, and child labor.
90 | Unfortunately, we have few other options at present. Some of the rarest minerals on the planet are controlled by some of the most despotic people. As the demand of autonomous EVs rises, those despots will become even worse to maximize their own profits.
91 |
92 | The only solution is to develop alternative materials through things like nanotechnology and other advanced materials science. This takes time and, more importantly, a lot of investment.
93 |
94 | ## Privacy Goes Out the Window
95 |
96 | Busses and other public transit already have cameras. EVs and autonomous cars are going to have even more. I am willing to bet that the EULA of every autonomous car service includes signing over your data. This probably includes biometrics about your body specifically as well as your destinations and pickups.
97 | If push came to shove, I bet this could be used to harm individual liberty.
98 |
99 | One way to mitigate this would be something akin to GDPR. I would expect to see legislation coming before too long that restricts what kinds of data car companies can collect, or require some kind of compensation or ability to pay extra to opt-out.
100 |
101 | # In Conclusion
102 |
103 | The cost savings for people will all but guarantee the transition from manual ICE vehicles to autonomous EVs. The increase in safety paired with decrease in per-mile cost makes this a no-brainer.
104 |
105 | ## Reallocation
106 |
107 | The cost-savings to individuals will get allocated elsewhere. Where exactly? I can't say. Perhaps home ownership? I suspect that cheaper homes in less desirable areas will become more popular with the rise of autonomous EVs.
108 | Perhaps we will see some folks setup a trucker-to-homebuilder training pipeline? Perhaps we will see a huge rise in demand for electricians, plumbers, and general purpose handymen to build and maintain these houses.
109 |
110 | Academics predict that jobs requiring improvisation, manual dexterity, and creative problem solving will be the last to get automated away. This supports the idea that plumbers, electricians, and repairmen jobs will be robust against automation.
111 |
112 | Other types of jobs that will remain robust against automation include teaching, childcare, therapy, massage, nursing, and other human-oriented caregiving services. I think it would be wonderful to see classrooms with fewer than 10 students per teacher.
113 |
114 | ## Relocation
115 |
116 | With reliable transportation a solved problem, and more money in people's pockets, we could see a surge to rural life. This may have additional knock-on effects in terms of physical and mental health. Allergies and auto-immune disease are reduced by increased contact with nature. More time in nature means better mental health. Smaller communities also promote more regular human connection.
117 |
118 | ## Health And Safety
119 |
120 | The one-two punch of increased physical safety and decreased environmental hazards will do wonders for the loss of human life. I expect there will come a time that people aren't allowed to drive manually and that ICE vehicles are banned outright. This won't be seen directly, but rather will be felt as an absence - like how we don't have to worry about polio today.
121 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-19-neuralink-predictions.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "Neuralink Predictions"
4 | date: 2020-10-19
5 | description: We're a long ways off from Ghost in the Shell
6 | categories: [Neuralink, Singularity]
7 | ---
8 |
9 | # I've seen this movie before...
10 |
11 | Neuralink appears to be the first practical step towards a *Ghost in the Shell* type world where our brains can be plugged in and hacked. Ghost in the Shell is a cyberpunk masterpiece.
12 | I lowkey place *The Matrix* in the same world as Ghost in the Shell, just a couple decades ahead. In *The Matrix*, machines can plug into our brains and give us a complete VR simulation world.
13 | In *Ghost in the Shell* you can dive into more abstract cyberscapes to do your hacking and social media. Perhaps most impressively, *The Matrix* postulates that you could learn anything in a matter of seconds.
14 |
15 | Elon Musk has been careful not to make any such grandiose promises, except on [Joe Rogan](https://www.youtube.com/watch?v=Jtn4tr202Ko). That being said, he has let slip some extraordinary claims. Still, I'm not here to debate the man, I'm simply going to write my own predictions for this technology. Here's a non-exhaustive list of movies that immediately come to mind:
16 |
17 | * The Matrix
18 | * Ghost in the Shell
19 | * Surrogates
20 |
21 | # Rehabilitation and Diagnostics
22 |
23 | Rehab is probably going to be the top use for Neuralink. The human brain is just too big and too complicated for a few thousand electrical probes to get a good picture of what's going on.
24 | We've had people controlling computer cursors and other very basic tasks with brain-machine-interfaces (BMIs) for quite a few years now. I think that Neuralink will, at best, make such technology more accessible and sustainable.
25 |
26 | Will it help restore sight and sound to the blind and deaf? Certainly not the early versions. Will it help restore motor function to paralysis and ALS victims? That's probably even farther away.
27 |
28 | Sensing and interpreting output is one thing. Every brain is different and so you'll need a lot of data and computational power outside of the Neuralink implant to even make sense of a lot of the output.
29 | It's simply not possible to jam enough computer power into a quarter-sized implant to do that. I am skeptical if phones even have enough horsepower to do this. I suspect that you'll need to integrate with cloud services, and more powerful computers, to get many of the benefits Elon has suggested.
30 |
31 | However!
32 |
33 | There would be a huge value to everyone walking around with high quality sensors in their heads (as well as the rest of their bodies). Stroke, aneurysm, dementia, depression, PTSD - all these things could be detected very early.
34 | The impact on the quality of life for people by avoiding suffering would be phenomenal. I suspect that Neuralink will ultimately find a lot of value by integrating with the ENS (Enteric Nervous System) as well.
35 | The brain-gut-axis is turning out to be extraordinarily important for mental and physical health, carrying huge implications for chronic illness.
36 |
37 | # High Bandwidth BMI
38 |
39 | I seriously doubt this will come about. Here are some reasons why.
40 |
41 | ## Limits of the human brain
42 |
43 | It can take us a while to formualte cogent, cohesive thoughts. We recruit specific regions of the brain when we intend to speak, and when we are listening and reading. Some of these regions are quite large, far too large for Neuralink to adequately survey.
44 | Elon says that writing, reading, speaking, and listening are very low-bandwidth. From a strictly numerical standpoint, I agree. But I also think that humans can only get so much faster. We have evolved to ingest general purpose information by listening to other humans.
45 | We simply lack the hardware and software to ingest data through other means. Even our reading and writing is just a facsimile of our speech, representing the sounds of speech with visual symbols.
46 |
47 | Sure, you can watch a lecture at 2x speed and still get most of it. We even have multimodal learning, where you see, do, hear, and practice all at once. It's simply a fact that human brains take a while to acquire new skills and knowledge.
48 |
49 | Neuralink would have to fundamentally alter the way the human brain works - which I don't think it will do with its first iterations. I would absolutely LOVE to be wrong. Maybe using Neuralink is a skill, a talent that we will have to develop.
50 | Perhaps we will have to learn to communicate with and through the device. Maybe it will ultimately be faster. It would be wonderful if I could type up this blog without a keyboard.
51 |
52 | Still, I think there are fundamental limits to how fast the human brain can assimilate and integrate new information. I think those limits are biological and won't be changed by a tiny implant.
53 |
54 | Sorry Neo, no helicopter program for you.
55 |
56 | ## Computer horsepower
57 |
58 | The human brain's processing power is roughly the equivalent of a 1 petaflop computer. It's a really bold claim to say that a pocket-sized machine could communicate with that faster and more effectively than it already can.
59 | The one caveat is if the power of the human brain is actually what the Neuralink device relies on. Still, I'm highly skeptical.
60 |
61 | At full power, my Pixel 4 operates at 954 GFLOPS, just shy of 1 TFLOPS, which is 1000x less powerful than my brain.
62 |
63 | Why do I use this comparison?
64 |
65 | I anticipate that Neuralink will have to build a model unique to everyone to essentially simulate their brains in order to calculate the exact pattern of neural stimulation required to communicate with us at a high bandwidth.
66 | In principle, I do believe that *Ghost in the Shell* and *The Matrix* technologies are possible. The human brain relies solely on neural impulses for all input and output (IO). We know this for a fact.
67 |
68 | When you want a human brain to learn something, you want to change the state of neurons, their individual "memory" in the form of synaptic connections. There's an initial state and an end state.
69 | We will probably need to be able to simulate or model a large portion of your human brain in order to communicate quickly and effectively with just you. If Moore's law holds, then it will be 20 years before a smartphone reaches the petaflop mark.
70 | That's 2040, which will be approaching Singularity anyways. If the overall goal is to prevent an AI holocaust, that's likely to be too late.
71 |
72 | # Direct sensory stimulation
73 |
74 | While I do believe the high bandwidth BMI is 20 years away, I think we could see direct audio and visual stimulation much sooner. Paradoxically, the part of the brain that handles optical processing is at the back of the head.
75 | This makes it an easy place to drop some electrodes. Perhaps this is why Neo jacks in at the back of his head? In *Ghost in the Shell* the connectors are on the back of the neck, closer to the brain stem.
76 |
77 | This could ultimately allow for hardware sensors being integrated, used to replace defective or missing eyes and ears. Wouldn't that be cool! You can upgrade your eyes to have telescopic and thermal vision! You could hear better than cats and dogs.
78 | Maybe you'll be able to relay extra sensory information via BlueTooth on your phone.
79 |
80 | In this way, I think Neuralink is far more likely to result in some cool augmented reality abilities rather than full VR. At least for the foreseeable future.
81 |
82 | # Telepresence robotics
83 |
84 | Did you ever see the creepy Bruce Willis movie *Surrogates*? Basically, I think we're far more likely to see that sort of thing than anything else. Elon Musk already demonstrated the ability to detect motor movements from neural signals.
85 | With motor output and sensory input, you could remotely inhabit any machine, not just human-like ones. In that movie, a bunch of soldiers are sitting in a call-center, remotely piloting battle mechs.
86 |
87 | The movie was really cool in the premise but the villain was very 1980's. Spoiler alert: *He's going to take over the world by enslaving everyone with his telepresence robotics technology! Bwahahaha!* Yeah, that was dumb.
88 |
89 | I recently played through a game called *Lone Echo* where you occupy the perspective of a robot working on the outside of a space mining station. I think that Neuralink could enable that sort of remote work for humans.
90 | *Dr. Who* even had an episode about this, except the telepresence robots were learning polymer goo that became sentient and rebelled, killing the workers.
91 |
92 |
93 |
--------------------------------------------------------------------------------
/docs/_posts/2020-10-22-infrastructure-dashboard.md:
--------------------------------------------------------------------------------
1 | ---
2 | layout: default
3 | title: "My Infrastructure Dashboard"
4 | date: 2020-10-22
5 | description: When you're responsible for dozens of systems and hundreds of hosts...
6 | categories: [VMware, UCS, Python, PowerShell, Flask, Monitoring, Dashboard]
7 | ---
8 |
9 | # Enterprise Tools are Expensive
10 |
11 | I'm not gonna knock tools like vRops and UCS Director and SolarWinds. They are awesome tools, but they are expensive.
12 | They also tend to offer a lot of functionality that I might not need. Lastly, they might not integrate with all my systems. Here's a summary of what I want in a dashboard:
13 |
14 | - Single pane of glass for EVERYTHING I'm responsible for
15 | - Inventory of hardware, servers, switches, blades, etc
16 | - All active faults, alarms, etc, across all my systems
17 | - Capacity and throughput for all my systems
18 | - I want it to be FAST and CLEAN
19 |
20 | Basically, I want the ability to glance at one thing to quickly ascertain the state of my entire environment.
21 | I set up a previous version of this at a while back and several of my team members found it indispensible - a great way to keep dynamic track of all inventory.
22 | This is especially critical when you have blades, chassis, and vCenters that don't necessarily all talk to each other.
23 |
24 | My first iteration of this was something like 7 years ago, when I tried to make a PowerShell GUI app that I called "vCommander".
25 | It didn't go too far because I didn't have clarity of purpose. I thought I wanted a fast way to run all my scripts, but for that I just use PowerShell ISE and now Rundeck.
26 | [Rundeck](https://www.rundeck.com/) provides all the script-via-web functionality you could ever want and it's open source!
27 |
28 | # Python Flask
29 |
30 | Ever build a web page from scratch? Well, now you can! With Python and Flask! Personally, I've created a few utility tools and REST APIs with Flask.
31 | It's one of my favorite little things.
32 |
33 | ## Bare Bones Version
34 |
35 | ```python
36 | import flask
37 | import json
38 |
39 |
40 | def json_to_html_table(data):
41 | # data must be list of dicts, where all dicts have same keys
42 | result = ''
43 | keys = data[0].keys()
44 | for k in keys:
45 | result += '| %s | ' % k
46 | for row in data:
47 | result += '
'
48 | for k in keys:
49 | try:
50 | if 'http' in row[k]:
51 | short_name = row[k].replace('https://','').replace('http://', '').split('.')[0]
52 | result += '| %s | ' % (row[k], short_name)
53 | else:
54 | result += '%s | ' % row[k]
55 | except:
56 | result += '%s | ' % row[k]
57 | result += '
'
58 | result += '
'
59 | return result
60 |
61 |
62 | def load_json_content(filepath):
63 | html = ''
64 | with open(filepath, 'r') as infile:
65 | content = json.load(infile)
66 | for section in content:
67 | html += '\n\n
%s
\n' % section['Header']
68 | html += json_to_html_table(section['Data'])
69 | html += '\n'
70 | return html
71 |
72 |
73 | app = flask.Flask('InfrastructureDashboard')
74 |
75 |
76 | @app.route('/', methods=['get'])
77 | def home():
78 | html = base_html
79 | html += '\nFast Links
\n'
80 | html += load_json_content(
)
81 | html += '\n