Articles that don’t fit in to any other categories.
If you’ve noticed that you can’t search for locations on the AccuWeather iOS if you are on a network protected with a Pi-hole, you aren’t alone. To fix it, you need to whitelist the following domain:
api.radar.io
After that, search will start working with AccuWeather again!
For the past decade I was a software engineer at a number of different companies. At each company I learned a lot of stuff, honed my craft, and eventually decided that I wanted to be a manager. I knew that it was a much different position from being a senior or lead engineer, but I wasn’t quite prepared for the number of things that I was going to learn.
Overall it was a great experience, so I thought I’d write down some of my learnings here for posterity. So here we go, in no particular order.
One last note before I go: Imposter syndrome is real. I’ve always known I was a good engineer. My track record delivering, career, and salary path all back up that idea of myself. As an engineering manager I have no track record, so I feel like an imposter sometimes. You get over it, but it hits hard.
I’ve spent the last few weeks writing a data migration for a large high traffic website and have had a lot of fun trying to squeeze every bit of processing power out of my machine. While playing around locally I can cluster the migration so it executes on fractions of the queryset. For instance.
./manage.py run_my_migration --cluster=1/10 ./manage.py run_my_migration --cluster=2/10 ./manage.py run_my_migration --cluster=3/10 ./manage.py run_my_migration --cluster=4/10 |
./manage.py run_my_migration --cluster=1/10 ./manage.py run_my_migration --cluster=2/10 ./manage.py run_my_migration --cluster=3/10 ./manage.py run_my_migration --cluster=4/10
All this does is take the queryset that is generated in the migration and chop it up into tenths. No big deal. The part that is a big deal is that the queryset contains 30,000 rows. In itself that isn’t a bad thing, but there are a lot of memory and cpu heavy operations that happen on each row. I was finding that when I tried to run the migration on our Rackspace Cloud servers the machine would exhaust its memory and terminate my processes. This was a bit frustrating because presumably the operating system should be able to make use of the swap and just deal with it. I tried to make the clusters smaller, but was still running into issues. Even more frustrating was that this happened at irregular intervals. Sometimes it took 20 minutes and sometimes it took 4 hours.
My solution to the problem utilized the clustering ability I already had built into the program. If I could break the migration down into 10,000 small migrations, then I should be able to get around any memory limitations. My plan was as follows:
One of the issues with doing concurrency work with Python is the global interpreter lock (GIL). It makes writing code a lot easier, but doesn’t allow Python to spawn proper threads. However, its easy to skirt around if you just spawn new processes like I did.
Borrowing some thread pooling code here, I was able to get pretty sweet script running in no time at all.
import sys import os.path from util import ThreadPool def launch_import(cluster_start, cluster_size, python_path, command_path): import subprocess command = python_path command += " " + command_path command += "{0}/{1}".format(cluster_start, cluster_size) # Open completed list. completed = [] with open("clusterlog.txt") as f: completed = f.readlines() # Check to see if we should be running this command. if command+"\n" in completed: print "lowmem.py ==> Skipping {0}".format(command) else: print "lowmem.py ==> Executing {0}".format(command) proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = proc.stdout.read() # Capture the output, don't print it. # Log completed cluster logfile = open('clusterlog.txt', 'a+') logfile.write("{0}\n".format(command)) logfile.close() if __name__ == '__main__': # Simple command line args checking try: lowmem, clusters, pool_size, python_path, command_path = sys.argv except: print "Usage: python lowmem.py <clusters> <pool_size> <path/to/python> <path/to/manage.py>" sys.exit(1) # Initiate log file. if not os.path.isfile("clusterlog.txt"): logfile = open('clusterlog.txt', 'w+') logfile.close() # Build in some extra space. print "\n\n" # Initiate the thread pool pool = ThreadPool(int(pool_size)) # Start adding tasks for i in range(1, int(clusters)): pool.add_task(launch_import, i, clusters, python_path, command_path) pool.wait_completion() |
import sys import os.path from util import ThreadPool def launch_import(cluster_start, cluster_size, python_path, command_path): import subprocess command = python_path command += " " + command_path command += "{0}/{1}".format(cluster_start, cluster_size) # Open completed list. completed = [] with open("clusterlog.txt") as f: completed = f.readlines() # Check to see if we should be running this command. if command+"\n" in completed: print "lowmem.py ==> Skipping {0}".format(command) else: print "lowmem.py ==> Executing {0}".format(command) proc = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) output = proc.stdout.read() # Capture the output, don't print it. # Log completed cluster logfile = open('clusterlog.txt', 'a+') logfile.write("{0}\n".format(command)) logfile.close() if __name__ == '__main__': # Simple command line args checking try: lowmem, clusters, pool_size, python_path, command_path = sys.argv except: print "Usage: python lowmem.py <clusters> <pool_size> <path/to/python> <path/to/manage.py>" sys.exit(1) # Initiate log file. if not os.path.isfile("clusterlog.txt"): logfile = open('clusterlog.txt', 'w+') logfile.close() # Build in some extra space. print "\n\n" # Initiate the thread pool pool = ThreadPool(int(pool_size)) # Start adding tasks for i in range(1, int(clusters)): pool.add_task(launch_import, i, clusters, python_path, command_path) pool.wait_completion()
Utilizing the code above, I can now run a command like:
python lowmem.py 10000 3 /srv/www/project/bin/python "/srv/www/project/src/manage.py import --cluster=" & |
python lowmem.py 10000 3 /srv/www/project/bin/python "/srv/www/project/src/manage.py import --cluster=" &
Which breaks the queryset up into 10,000 parts and runs the import 3 sets at a time. This has done a great job of keeping the memory footprint of the import low, while still getting some concurrency so it doesn’t take forever.
Over the past few years I’ve slowly accumulated some big side projects. They weren’t done for clients, but just for myself. At some point maintenance of these side projects isn’t fun anymore and hinders the creative juices. I have other things I want to work on, but having these other zombie side projects feels too much like an albatross around my neck.
After much deliberation, I’ve decided to shut down two of my large side projects: BookCheaply and Smooth Bulletin. I really believe both of these projects could do someone some good, but they were both learning projects for me and I don’t see them moving forward anymore. Effective immediately I’m disabling their Apache configs, backing up their DBs, TARing it all together, and putting it somewhere safe. I’ll keep access to the Git repos, but eventually I’ll clone a copy of those out too and archive them. If I don’t get them out of the way completely, I feel like I’ll want to work on them too much.
Shuttering these projects marks a transition for me, where I move from using Python and Django on side projects to Node.js, Express, and Angular. While apprehensive about abandoning my go-to stack for side projects, I’m excited to learn the nooks and crannies of Node (and I still use Python/Django for my day job anyways).
Here’s to the future!
If you want to skip the restoration stuff, the final product can be found at https://re-cycledair.com/starwars/Entrance.html.
A long time ago in an era just before the first dot-com bust, I was a bright-eyed child with aspirations to make the greatest Star Wars web site of all time. With only Microsoft Front Page Express and a limited knowledge of HTML in hand, the 1999 version of me created the best Star Wars web site the world had ever seen.
Unfortunately I was hosting it on my ISPs free web hosting area. When that ISP was later gobbled up by another company, all traces of my website were gone… or so I thought. Recently I discovered that most of it still exists on the Way Back machine, however it’s in poor shape. Many of the assets don’t exist anymore, the linking is broke, and the frames just don’t work right in modern browsers. This is my journey to save this little piece of history while using minimal modern techniques. My goal is to have a legitimate 1999 website when I’m done and have it render nicely on most modern web browsers.
The Wayback machine at the Internet Archive has long been one of my favorite websites. I love getting nostalgic for the web of the past, my childhood, and history in general. It was awhile ago that I realized it still had some of my original Star Wars website. If you want, check out the capture from 2001. It doesn’t work much at all, but it gives you a good idea of what we’re working with here.
In addition to the main page I also had an “Entrance” page, because you know, every cool web site had an entrance page back then. This one is in slightly better shape, meaning the star field background survived intact.
My goal for this restoration is to make the website display great in modern web browsers with minimal code changes and without using too many modern techniques. At that point in time, CSS wasn’t really a thing that a ton of people used yet, and even if it was it was beyond my understanding, so I’m going to try and avoid it if at all possible. Javascript was available in most browsers by that time, but I didn’t understand it, so I didn’t use it. However, we do have FRAMES(!), which is going to make things super exciting for everyone still reading.
Now that we’ve laid down some ground rules, here is the order of attack:
Lets get started!
While it’s fantastic that the background star field has survived, the rest of the images didn’t. This is sort of a problem because I have to reach pretty far back into my memory to figure out what was there originally. Luckily, 1999 me wasn’t completely terrible at naming things.
From these file names and memory, I’m going to say that `tie.gif` was a rotating TIE fighter gif, and `starwars.gif` was the Star Wars logo that survived on the main section of the site.
And from this file name we get nothing. Luckily I remember this being an image of Tatooine, roughly 350px x 100px. This site came out a bit before Episode I, so I’m going to assume it was an image from one of the Episode I trailers. At least, thats how I remember it.
After a bit of digging, I found replacement images for everything but the Tatooine banner. In it’s place, I found a light saber gif that I definitely used on this site somewhere in the past. I also updated the star field to be a little more 1999.
Now that the entrance is starting to look like a real 1999 web site again, we can start to clean up the HTML and fix the broken link to enter the site. The entrance area contains 3 pages, 1 to bring the frames together, and 1 for each frame. They currently look like this:
The code isn’t too bad. With a bit of alignment help and some nesting, things are going to look pretty good. Now check out the final product:
Now that we have the entrance out of the way, I’ve have a pretty good idea of how this whole restoration will work for the rest of the site. Let’s start by fixing the background on the main site so things are a bit more workable.
Now we’re talking! There are few things that I noticed immediately after getting the background set.
Looking at the source, there are a few images that aren’t linking quite right.
After tracking down many of the original images that I used here, the home page now looks like this.
Looking good! Next up, code clean up.
The code for the main site is in pretty poor condition. There is a ton of code in there from the Internet Archive, it has poor indentation, and in general just isn’t anywhere near the quality of the entrance. Given the amount of code here, I thought using a Github Gist would be better.
Before Code: https://gist.github.com/vital101/56b2783b8e75b00fdac9
After Code: https://gist.github.com/vital101/2d1e6a44ac3ddbb1789d
As you can see I went ahead and re-aligned everything so that it’s more readable, took out all of the javascript and css added by the Internet Archive, and condensed the code a little bit so there wasn’t large bits of whitespace between elements.
The next step in the code cleanup is fixing URLs. If you look at the code, you’ll notice that all the links still point to the Internet Archive. That obviously isn’t going to work for us, so I simply need to replace “https://web.archive.org/web/20010811043323/http://my.voyager.net/~hands/star-wars/” with an empty string, making the link relative.
After fixing links, I found that many of the sub-pages have been lost to time, but there are still a few for us to look at including:
Fixing up these pages follows a similar pattern to the entrance page and the main site, so I’ll spare you the details. There were a few interesting things that popped up while I was doing the sub-page restoration though.
The final product can be found at https://re-cycledair.com/starwars/Entrance.html.
While going through this restoration I realized that this is when I knew what I wanted to do with my life. My parents had no idea what I was doing, but they realized that it stimulated my mind so they let me continue to do it anyways. Without that kind of encouragement and freedom to create, I probably wouldn’t be where I am today. This restoration has also re-kindled my respect for the old web. It was simple, but I’m extremely happy that the web has progressed to where it is today.
Ever since HealthCare.gov launched all I’ve been hearing about is how horrible the experience is. It never affected me directly, so I chalked it to a few vocal detractors trying to sway public opinion. I even went so far as to go to HealthCare.gov and use the logged-out experience to compare some plans. It looked pretty good actually, and I tweeted as such.
That was all about to change though. I recently made the move to work at a small startup that doesn’t yet have an employer-sponsored health plan. They do however offer a stipend of sorts to help you pay for insurance costs on HealthCare.gov. And with that, my journey began.
After discussing the healthcare situation with my wife, I ran the numbers and decided that it would be best for her to use her employer-sponsored healthcare and for me to use the exchange separately. At this point everything went as well as it could. Even though the UX on HealthCare.gov is pretty bad, I was able to figure out how to fill out an application, select health coverage, select dental coverage, and get things rolling. This all happened before December 23rd, which means I’d be getting coverage by January 1st! All was well… until I had to make a change.
As it turns out, I math’d wrong. When I was calculating the cost of healthcare for my wife at her job, I was off by something like $200 per month, which pushed her healthcare costs well into the “unreasonable for what I’m going to receive” range. With that in mind, I thought I could make a quick change on HealthCare.gov to get her on the same plan that I was. Boy was I wrong.
The process I had to take to get both my wife and myself health and dental insurance was ridiculous, and still hasn’t worked correctly. To add my wife to my plan, I had to delete the entire thing. Not only that, but I needed to delete my entire application and start over from scratch. Once I did that, I kept constantly running into javascript errors in their application which would force me to re-authenticate.
After finally getting through the application process, I get to the last step where I need to confirm. I press the button and… nothing. Not a goddamn thing. Being a software engineer, I do a little investigating and find out that the server is returning 500 errors to the client (in non-developer speak, this means that the server couldn’t process my request because of some error on their side). The error that is returned says that I should try to log-out, then resume the application. I do this, but when I try to resume my application I get:
Thats right, my application is locked. It doesn’t give any reason, which is especially confusing considering that I was asked to re-authenticate. But there is a little link that can “explain this task”, so I click it, and it naturally didn’t work.
At this point, I was beginning to realize that I would need to contact their support to get things resolved.
The first time I tried to use support, I clicked the helpful little “Live Chat” button. I then proceeded to wait for 25 minutes with no response. This was over the holidays, so I gave it another try after Christmas. This time I was connected to somebody, but no matter what question I asked I was given a canned response to contact the call center. My question is: Why have the “Live Chat” at all if you’re just going to tell me to contact the call center? It’s ridiculous and a waste of my time.
After my initial experience with HealthCare.gov, I’m not really excited to contact the call center. I honestly don’t have a ton of time to do it, and given the number of people that will be signing up for healthcare I’m sure the wait will be long. My likely next steps will be to cancel what remains of my application and start over. If it doesn’t work this time, then I’m going to cancel that application and contact my insurance provider of choice directly. In my case thats Blue Cross Blue Shield of Michigan, which happens to have a walk-in office about a block from where I work.
I create things all the time on the web. Its my chosen profession, so I know how hard it can be to make a good website when you need to integrate with a lot of different 3rd parties. However this sort of experience isn’t going to cut it. If I’m federally mandated to have health insurance, then the experience should be as painless as possible.
Because the system is broke I won’t have health coverage until February 1st now; a month long gap in coverage for my family. This just isn’t acceptable.
If you browse Hacker News enough you’ll start to see article trends.
Don’t get me wrong, some of this stuff is interesting, but most of it is just noise to me. The reason I come to HN is for the comments. Thats where the good stuff is. HN is one of the brightest internet communities I’ve ever had the pleasure to interact with. For instance, today I posted to HN asking for the community to review my startup Smooth Bulletin. Within an hour I had two very thoughtful comments about business and market ideas that I hadn’t even thought of. I expect I’ll probably get even more feedback as the day goes on, and that I’ll benefit from those just as much as the first comments.
Thats why I still have faith in Hacker News. Not because of the content, but because of the people. The people there are incredibly smart and willing to give their advice just so that maybe you can succeed one day. I’ve been a member of HN for several years now and this hasn’t changed at all since the time I joined, and I hope that it never does.
And yes, I posted this to Hacker news, so maybe I should add “Why I still have faith in X” to the list.
Over the past few years I’ve tried my hand at launching a couple of different start ups. Over this time period, I’ve collected a set of questions from many different sources (HN, YCombinator, TechStars, etc) that I ask myself before proceeding. They help be decided if this is a real or manufactured problem, if it can be profitable, and if people will buy what I’m selling. My latest project idea is the only idea I’ve had that passes all of these questions to my satisfaction. How do yours hold up?