End of the Monthish (Aug. 2016)

Every month, I ignore the reminder to do an end-of-month blog.  Another month has come and gone and again I didn't do it.  Well, I'm doing one today, dang it.  Not quite the end of the month, but it's labor day, it's a little slower and I'm going to write something.  It won't necessarily be great, but it'll be something.  I wish I wrote more often, but I guess it's not been a strong enough aspiration.  It definitely helps with a decent year-end blog post when I have lots of posts to go back and look at. Anyhow, a few thoughts...

Today is my Grandma Marjorie's birthday.  Well, she has passed away, but today's the day we celebrated with her.  She will forever be linked to my career - in 2014, I was let go at World Vision the day before her birthday.  My parents, my brother and his family and my family all traveled to Yakima to celebrate with her.  It was a wonderful time.  I didn't think much about the job loss that weekend, or probably more accurately, didn't talk about it much.  She did find out anyhow near the end of the trip and wanted to know why I hadn't mentioned it and I told her that it wasn't important, that the weekend was about her.  In 2015 I had just landed in L.A., a consultant employed by a company in Seattle assigned to ExactTarget/Salesforce who in turn assigned me to Activision.  They were bringing me along on a trip for a day-long strategy session with Activision.  And of course, we all know how that turned out.  Anyhow, so I had just landed in L.A. and there was a message from Lori letting me know that mom and dad had called, grandma had passed away in her sleep.  We knew it was coming and I just remember sitting there near baggage claim talking to mom.  I think we were all happy for grandma - she had lived her life the way she wanted to.  Until a day or two before, she had still lived in the home she and my grandfather had purchased together, nearly a decade after she first saw it and decided she wanted to someday live there.  And she had still been driving and working and volunteering at church.  But a few days before, her health had really started to decline and she allowed herself to be checked into a hospice.  She hadn't let dad put too many personal effects out because she wasn't planning on staying long.  She worried about dying but I think dad let her know that it was ok and she accepted this and went peacefully in her sleep.  Marjorie was really cool - she was well-loved and always in a really positive mood.  Quite inspirational.  Lori notes that she came across some handwritten notes recently - grandma was really good at that - always hand-written notes.

So, yeah, that other thing... two years ago yesterday I got let go from World Vision.  At the time I thought it was cowardly.  Now I'm not entirely sure, but I do know that it was a blessing.  Things had run their course and it was time.  I had contributed while I was there and hopefully left things in a better place than before I had been there.  But we all knew it was time.  I had a few interviews and had been looking for somewhere new, but unsuccessfully.  Because they were the ones who pulled the trigger, I ended up with severance, pension and a few other things I wouldn't have necessarily gotten had I acted first (though it wasn't for a lack of trying).  Recently it's been on my mind a lot to write Toby and Scott a thank you note, but that's probably not beneficial to anyone.

The city of Burbank has signed off on the renovations so now we're back in compliance.  I've been painting the garage floor the last few days and today we should be able to move most of the stuff back in.  There's a little more finish work to happen tomorrow, but no more living under the cloud of construction and worries about whether our changes are going to be approved.  Next, some foundational work under the house, saving up for the master suite renovation/expansion and designing the Little Free Library I want to add to the front fence.

And... I think that's a fine place to stop for now.  Phew.

Splitting Large File with Headers (Mac)

I was given a 3.6 gb CSV data file (43 million rows x ~12 columns). I have an automation that's supposed to intake and process those files automatically, but the file was too large. What I needed were smaller files, each with the same header.

I had to do a number of Google searches to figure out how to do this, so I thought I would bring together everything I learned into a single post in case it would help someone else with a similar scenario.

Experience: Moderate

Step 1: Split the file into multiple files, each 1,000,000 rows long.

split -l 1000000 filename.csv filename-

This will give you multiple files with filenames like filename-aa, filename-ab, filename-ac, etc.

Step 2: Move the large file (filename.csv) to another directory. You don't need it anymore, but you'll want to hang on to it until everything's as you want.

Step 3: Rename all the files as CSV files.

for i in *; do mv "$i" "$i.csv"; done

Step 4: Open filename-aa.csv with Excel. Copy the first row (the headers) into a new worksheet. Save as "header.csv" Close filename-aa.csv (don't save).

Step 5: Move filename-aa.csv to a different directory. We don't want to add the header to it a second time.

Step 6: Open header.csv with a text editor to make sure there's a "return" at the end of the row. (If you move to the last character in your header and press right-arrow, does it drop to a new line? If not, press return or enter and then save. If so, you're good.)

Step 7: Rename header.csv to header.txt - it will warn you, but it's ok.

Step 8: Add the header to all of the other files

for i in *.csv; do mv "$i" tmp; cat header.txt tmp > "$i"; done

Step 9: Move filename-aa.csv back into the directory

You should now have a bunch of files, each with a header and 1,000,000 rows.  (And a file called "tmp" that you can safely delete.)