Tuesday, March 30, 2010

Lazy weekend

Saturday, March 27: High:18F Low: 5F
Sunday, March 28: High: 29F Low: 25F

It's too windy on Saturday and the airport is closed on Sunday, so we take the weekend off. I'm glad for the chance to catch my breath and get caught up on the work.

Saturday is also the First Light Festival. Since there's total darkness here from November through February, the base has a party when the sun returns. The party has a western theme, and they serve a free dinner with ribs and corn bread. At one of the hangars on the base, they have a cash bar and the band Clazz plays mostly classic rock and roll, and they aren't half bad. Because of the theme, they attempt to play what they called "country", but you could tell they were a rock and roll band. There's also a contest for the best western costume, and they sell chili to raise money for the local Greenlandic children. I have a couple of beers and enjoy the band, but they allow smoking in the hangar, so I have to head home after only about an hour.

The bunch of the crew is hanging out in the common room. We watch the end of the Butler/Kansas State game, and I'm shocked to see little 'ol Butler go to the Final Four. I'm betting Indianapolis was a zoo Saturday night. I talk to several of the crew including one of the pilots. This really is a fun group of people, and I feel lucky to be hanging out with them.

The days are long here, and in fact, it never does get really dark this time of the year. The sun sets around 9pm but it never gets dark enough to see stars. It rises at 6am and the day is getting 15 minutes longer every day. By the time we leave in April, there will be 24 hours of daylight. I was hoping to do some star gazing, but not at this time of the year. In fact, probably not any time of the year around here because by the time they reach astronomical darkness, it's probably too frigid to be outside. Maybe when we get to Fairbanks, I'll be able to do some star gazing.

Sunday is a slow day. I finish up all my backups. I also move my access point around in order to cover the hallways the best I can. I find a good spot that seems to cover just about everybody. Everyone tells me how well it's working. Like one of my managers at work says, "the easiest way to make friends is to provide a good networking service."

It's really warm outside compared to the previous week. The warm weather is due to the storms that have popped up. These are mostly wind storms, but it does spit snow here and there. Greenland can get up passed 60F in July, and when everything thaws, the base gets super muddy. I come to realize that March in Greenland may be the best time to visit. All the other times seem to be filled with frigid cold, total darkness, 24 hour sun, or mud.

During the Sunday briefing meeting, it's thought that we may go to Fairbanks on Monday. So, again, I pack up our equipment and my luggage. I try to head to bed early so I can get up early enough to see if we are indeed going to Fairbanks. However, my sleep has been thoroughly disrupted and although I go to be at 10:30pm, I don't fall asleep until midnight.

Thursday, March 25, 2010

A blustery day

Thursday, March 25: High: 14F Low: -2F
Friday, March 26: High: 8F Low: 0F

The wind wakes me up around Thursday, 9:30am, and I quickly check on all my backups. I discover that some of the data is corrupt, and we're happy to have the down day so that we can analyze what might have happened. I also make a change to my procedures to double check the data so we can avoid any more data loss. The good news is that the important data is there, so no harm no foul. But it does mean another day of data copying and verifying to make sure everything is in order.

The weather condition is alpha for most of the day and turns to bravo at points. Alpha means just be aware of the weather and bravo means you can't leave the base without gear and a partner. I figure where would I go anyway? In fact, the clouds that have rolled in with the weather system make is warmer.

Michelle and Bryan from the LVIS crew are heading out tomorrow on the rotator flight. Every Thursday, a DC-8 brings supplies and people to the base, and flies anyone out the next day. Colleen shows up to replace Michelle and Bryan, and we make plans to have a beer before they leave.

We go to the Top of the World club, or TOW club, which is a bar on the base. It actually has beer on tap and you can get food. I missed dinner because I was working, so I'm happy to get some fried shrimp with mixed vegetables. Michelle buys me a beer and Bryan reminds her she owes me a case. I really like this team of people. They work well together, are serious and proud of their work, and they have fun at the same time. Luckily, Michelle and Bryan will be returning in two weeks, so I can hang out more with them then.

Most everyone goes to bed early because tomorrow is a fly day. I stay up until 3am moving data. It really does seem like a 4pm to 3am is going to be a standard day for me for most of the trip.

Friday I wake up around 10:30am and pretty much everyone is out flying a mission. On Wednesday they flew over the Petermann glacierPetermann Glacier. Today they fly over the sea ice to the north and east of Greenland. I spend the day copying data and doing my laundry.

Around 5:30pm we have our meeting about the next mission. It's decided to take the weekend off due to another report of high winds on Saturday and the fact that the runway is closed on Sunday. It also gives the flying staff a few days down time which means they get to rest. They are a great flight crew and fly these defined flight plans with precision. Everyone here talks about how good the crew is and knows how lucky this project is to have such an experience and caring team.

Around 10pm, I'm downstairs in the bathroom getting my last load out of the dryer when a local Greenlandic woman walks in. She's really tiny - maybe 4' 10" tall, and she doesn't weight but 90 lbs. To my surprise, she walks up to me and hands me a little bottle of Vodka. With hesitation I say, "thank you", but not wanting it, I try to give it back to her. She then says "open" and the light bulb clicks on in my head. I open the bottle and hand it back not thinking anything of it. About midnight, I come back downstairs and she's now stumbling about the hallway, moving like a pinball between the walls. She keeps looking at people saying "Sssshhhh!" which I guess means she doesn't want anyone to know she's there. Then about an hour later I see her again, but this time she's turned mean. She glares at me and starts moving towards me, all the while talking loudly in some foreign language. Up here the natives speak Greenlandic, but there's a lot of Danish as well since Greenland is governed by Denmark. Her anger spooks me, and I say to her "Sorry, I don't understand you" and then quickly dart into my room and lock the door. At this point I call security. By the time they show up, she's sitting in a chair in the lobby passed out. She's so out that they can't wake her up even though the shake her. The security guard was very nice, and chats a bit with me about the NASA mission. He says that he's trying to get on one of the flights as a guest. Apparently NASA will fly the local military personnel on the missions if they have the seats. I've been told that these missions are pretty popular among the local personnel. I guess when there isn't a lot to do in Thule, things like this are pretty exciting.

Data, data, and more data

Wednesday, March 24: High: 8F Low: -8F

I get up around noon, take a shower, and get to the mess hall just before it closes. Again the CRESIS team is flying a local flight and I'm now just waiting for the data to come back so I can start working. This really is a second/third shift job beginning around 4pm and going until 3am. I'm seriously getting some comp time when I get home.

While waiting for the flight to return, one of the NASA crew complains about not being able to watch the plane tracks on Google Earth because he can't connect to the server back at home base in California. Our mission commander claims it's a problem at the base in California, and the California base blames us. The crew asks me to look into it. I do a couple of traceroutes and discover that I can access the site in question from my workstation at Indiana University, but I can't access it from Thule. But in both cases I can get to a NASA router. So a quick look in the whois database, and I get a number for the NASA NOC. A young gentleman by the name of Sedrick answers the phone and I send him my traces. He says he'll look into it. Well, it ends up not being a NOC issue, but with Sedrick's help, we are able to get the California base to take the issue seriously. Eventually they "discover" that indeed the California base is blocking our IP range. Why, I'm not sure. But I hand the phone to the safety officer who tells the Californians that we need to see the real-time flight data because there are 40 people on board an airplane in the middle of *Greenland* for heaven's sake with very few populated areas around. That real-time data may be the only way we know that the plane has crashed and also *where* it crashed if it does. A little more arm twisting later, and the Californians agree to open the firewall to us. You would think one base wouldn't block another. It must be an international thing. Chalk up more brownie points for me!

The flight returns and the CRESIS team hands me about 4.5 TB of data to back up. I begin the eleven hour process of copying the data. I'm still running into some disk failure problems, but I'm able to get done what I need to get done. Also, it looks like most of the drive failures are in one particular case. I'm betting that case got manhandled at some point by someone who was angry about all the equipment we brought to Greenland.

During the daily crew briefing, we're told that the wind is going to pick up tomorrow. This means we cannot fly. Also it means that we need to be aware of the weather condition. They encourage us to check out the cold weather gear offered by the base. In fact, the gear is required if we want to leave the base tomorrow. I guess you wouldn't want to be caught outside in Greenland with no snow pants. Or something like that.

I'm glad tomorrow's a no fly day. It'll be good to have a day to get things in order.

Wednesday, March 24, 2010

Setting up

Tuesday, March 23: High 1F Low -12F

I spent most of the day setting up the room we have for the equipment. We got a second dorm room and set the bed up against the wall for more room. I set the desk near the windows so that we could straddle the two walls hoping that the power from each wall was on a different circuit. We brought an IBM 3650 M2 and a 16-disk SATA array. Both have dual power supplies, so hopefully we have spread the power out. I have a "Do Not Disturb" sign on the door not only to keep the maid from freaking out when she opens the door, but also to keep her from tripping the circuit breakers when she plugs in her vacuum cleaner. Apparently this has happened before on other missions. So far, the power has been good. First hurdle has been overcome.

I unload some of the cases and sort through the stuff. The CRESIS team left on flight mission at 8am and weren't due back until 4pm, so I had plenty of time to get everything in order. I began populating the disk array when I discovered that several of the disks had failed in transit. My failure rate was around 30% which was slightly frustrating. But all in all, setup goes smoothly and I'm all ready for the team when they arrive.

I continued talking with the LVIS team. Michelle has a MySQL problem she's has been banging her head on all morning. I show her how to solve it in 5 minutes and Michelle promises me a case of beer. Her husband, Bryan, says "She's from the UK - a bottle of beer, a case of beer, what do they know?" They show me a website where they are plotting all the DC-8 flights on Google Earth. If you have Google Earth and are interested in seeing the flight tracks, send me a note and I'll give you the login information.

I learn that there are weather alerts here. Normal, Alpha, Bravo, Charlie, and Delta are the levels. At Delta, you can't leave the building you are in - period. If you have a heart attack, we have to deal with it in the building. Also, the base offers to lend us storm gear. This gear will protect us in case we are out and about and a sudden storm comes up. At least the gear will help us get back to a building if we go out walking. It's suggested you get the gear even if you are walking around on base and it's required gear if you want to leave the base on foot. Wow.

Once the team gets back, they tell me that we are possibly going to Fairbanks, Alaska in the morning. This trip could involve 3 data runs - one on the way there, one local one in Fairbanks, and one on the return trip. We need to be able to handle the data from the run there and the possible local run in Alaska. This means taking laptops, cables, and power equipment for doing this without the server. Luckily for today's run, they only have a small amount of data to back up - around 350GB. This only takes a couple of hours, but I spend several more hours going through disks finding 20 good disks to take to Alaska and packing up the cases for the trip. I also hook up my ddwrt router so I can get internet in my room. This works brilliantly and also allows several other people to get internet in their rooms as well. I feel it's a huge success.

By the time I finish, it's 3am. The decision to go to Alaska will be around 7am, so I figure I'll just stay up. Around 3:30am, the mission leader, Kent, comes out and complains about the Internet connection. He's having difficulty accessing it, and I have to admit the connection is super slow. He sees my router and asks to remove it. I of course say yes. I don't want to cause any trouble. However, the installed AP will not allow me to connect for more that 10 or so seconds. Thus I'm now relegated to the common room with a wired connection to the AP. Ugh. However, Kent leaves on Friday and we'll rewire things to our liking at that time.

I eat breakfast around 5:30am and then take a shower. Around 7am, someone comes and says that they are not going to Alaska after all and I crawl into my bed and pass out. Thanks goodness for the blackout shades in our rooms.

If you want to check on the weather around here, use the Pituffik, Greenland weather station. And if you need to know exactly where I am, search for Pituffik, Avannaarsua, Greenland on Google Maps.

Tuesday, March 23, 2010

And we're off!

A few months ago, my work asked me to travel to Greenland to provide technical support for the PolarGrid project. The PG project is a joint effort between the Elizabeth City State University and Indiana University to provide technical support for projects that study glacier melting. This means that we provide computing clusters for data processing and we also go into the field with the teams whenever needed. These field missions have gone to Antartica and Chile, and the current field mission is at the Thule Air Field in Greenland.

The trip would entail 5 weeks in Greenland with the CRESIS team, and so many things had to be done before I could go. I had mounds of paperwork to do for travel and for NASA who is providing all the flight infrastructure for the Operation Ice Bridge mission. I had to find someone to watch the boys. Luckily their dad, with a blessing from Purdue, agreed to come to Bloomington to stay with them while I was gone. I had to pay my taxes and other end of the fiscal year type stuff. All this while other projects were ramping up at work meant I was wicked busy for the month leading up to the trip. Plus I was spending the week before my trip in Oneonta, NY visiting my friends Jason and Adriane, their toddler son Rowan, and their new born daughter Amelie. I didn't have much time to prepare before the trip was looming large.

So after kissing the boys goodbye and a week in NY, I headed to Palmdale, California. I had 6 bottles of Ommegang beer, three books from Terry Pratchet's Disc World series and a copy of Stephenson's Anathem, some movies, and my son Nick's Nintendo DS and a half a dozen games to keep me busy. I went to the Dryden Flight Research Center where a DC-8 awaited along with tons of scientific instruments to whisk us off to Greenland. I got there around 2pm and found no one else from the mission at the airfield, and so had to turn around and spend the day at the Palmdale Mall. Luckily a lovely lady at the Hilton Garden Inn watched my luggage while I went and drank beer at BJ's Restaurant Brewhouse. Purdue was on the TV playing Texas A&M and I think I was the only person in the bar interested in the game. Finally someone from the mission contacted me, and I headed back to the air field. I had to leave before the end of the game, but heard later that Purdue won. I left girl scout cookies for the ladies at the Hilton, and took a taxi to the airfield.

I was escorted inside the hangar after passing through security which only consisted of signing my name to a roster. Inside the hangar was the DC-8 all loaded up with equipment. Most of the seats had been removed, and periodically through out the plane, racks of equipment were bolted to the floor. Several instruments were installed in the cargo area of the plane including CRESIS's radar and a lidar. The seats that were in the airplane were first class seats that had 5-point harnesses instead of seat belts. Each seat also had a headset that anyone could use to communicate with the rest of the team, but mostly was used by the pilot to communicate with the researchers. After a short safety brief which included a description of a plane crash as a field of hot razor blades, we left around midnight.

Unlike the researchers on the plane who were running their instruments, I had no responsibilities, and I fell asleep immediately. The seats leaned back super far, and other than being cold, I slept very comfortably. I woke up just at dawn, and went up into the cockpit to watch the sun come up over the horizon. I had never been in the cockpit of such a large plane before, and the flight team was very friendly and let me hang out as long as I wanted. We were over the ocean, and it was completely frozen, but the ice wasn't flat. It was full of cracks, flows, cliffs, and mountains. Several of the researchers were madly taking data while we were over the ice, and I wandered around a bit and talked to several of the other teams. Michelle was with the LVIS project and showed me how they were using lidar to map the surface of the ice. Her husband Brian told me about the GRACE project which measures gravity so accurately that it can determine changes in the mass of the glaciers.

We landed in Thule around 2pm, and it was super cold. The temperatures are in the single digits this time of year. We began the long process of unloading all the gear. Luckily there is a free taxi service so it wasn't too terrible getting all the gear over to the dorm. I got a first floor room to setup as our machine room for all our computer gear, and a second floor room for sleeping - no one wants to sleep in the same room as a server and disk array. The dorm has fairly decent rooms, and each room has a TV, VCR, dorm refrigerator, microwave, and a coffee maker. The dorm also has a community kitchen, and the community bathrooms are nice as well with shower stalls that actually have locking doors. There's a washer and dryer in each bathroom and the dorm even provides the laundry soap. After taking a much needed shower, we went to dinner in the mess hall which had a large selection of good food. We also visited the supermarket which actually had green leaf lettuce, cucumbers, bell peppers, and strawberries. Then it was back to the dorm where after calling home, I spent the night installing ddwrt on a Cisco wireless router. There is internet in the common room/kitchen area, but it doesn't reach our rooms. I'm hoping to change that by boosting the antenna strength of my router. I went to sleep around 2am exhausted, and although I'm 950 miles above the Arctic Circle, I'm amazingly comfortable in this really inhospitable place.