Stone Aerospace

West Lake Bonney, Taylor Valley, Antarctica
Reporting from East Lake Bonney Basecamp

Most people did not get to bed until 3 or 4 am so it was a welcome sight to find John Priscu up at noon cheerily making crepes for everyone who walked in the door of the Jamesway at East Lake Bonney. Today was an off day. Many people caught up on their blogs… a luxury since our science days here are excessively long and most people are dog-tired by the time dinner is over. Somehow sitting down to write at 2am after 18 hours on your feet is difficult. A tent and sleeping bag seem more appealing. It was also a chance to look through data from the past several days at a less break-neck pace.

John Priscu hands out crepes in the Jamesway.

Shilpa, John, and Emma eventually went on a hike up to the ventifact field some 600 meters vertically above camp. Kristof went for a run along shore to the east. Chris built two snowmen (given the rare snow storm of yesterday). Vickie read Memoirs of a Geisha. In training, Bill did 155 pushups. Most watched an “Austin Powers” film following dinner.

Shilpa, Emma, Chris, Bart, Kristof, Vickie, and Chris’s creations “Frostie Boy” (left) and “Dr. Stone.” By the time Chris got to work most of yesterday’s snow storm had ablated.

Reporting by Bill Stone

West Lake Bonney, Taylor Valley, Antarctica
Reporting from East Lake Bonney Basecamp

We awoke to news that McMurdo was in Condition 1 (no travel). Here in the valleys it snowed during the evening at the higher peak levels but down low it was still dry.

During the routine (but extensive) pre-mission checklist we found that one of the six thrusters was showing a low level of pressure balance volume so it was taken out for maintenance and a spare swapped in.

Rachel completes the swap out of the #2 thruster for maintenance.

Prior to yesterday we had planned to incrementally (and conservatively) work our way out towards the “narrows” separating east and west Lake Bonney some 1.5 kilometers east of the lab. Using the revised mission planner and the improved battery behavior we realized we could acquire all of the remaining east end sonde casts in one mission. A year ago we had felt it would be required to have a secondary melt hole near the east end for “inflight refueling” and had re-designed the charging system on the vehicle for that purpose. Now with the improved vehicle travel efficiency and double power onboard it appeared we could run significant east end forays.

Our plan for today was to conduct sonde casts at E17, E18, E19, D21, E20, F19, F18, F17, E16, E15, E14, F14, and F6. Total mission length would be 3.2 km.

The mission began at 3:52 pm and the vehicle returned to the melt hole at 10:30pm with all sonde points acquired. Importantly, we had conducted a detailed multi-beam side-looking survey at two locations at western head of the Narrows. Multibeam was able to image all the way to shore on both sides. Depth at lake center in this area was 11m under keel, leaving enough to consider a more advanced mission in which the bot actually drove through the narrows and into East Lobe. At D21 (our furthest point today) we were just out of sonar range to image the location measured by Scott.

Marking the furthest exploratory mission for ENDURANCE, Vickie tracks the bot at 1.5 kilometers from the melt hole. Behind her is the “narrows” between west and east Lake Bonney, looking into East Lobe. Scott’s team took measurements here December 17, 1903.

Shortly after turn around (although still with more than half the sonde casts remaining to be performed) we had our only technical glitch for the day—at F19 communications dropped out for an unnerving 30 seconds. Then it suddenly came back. Shilpa analyzed the data and began searching a section of the onboard code she had been suspicious of for some time, ever since the earlier data link drop outs. With nearly 100,000 lines of software running the bot this would have been an insurmountable task during a mission, but Shilpa had spent a lot of time tracking this over the last 10 days and now found it—a “memory leak” in a routine designed to track the status of certain vehicle systems. Unchecked on a long mission, the errant process filled up memory and died. When restarted, it freed up memory and the system came back. The fix, until a more optimized re-write of that section could be performed, was to pre-empt and periodically re-boot that process—one of scores running the vehicle. Given the location of the bot at the time (1.5 kilometers from the hole) this was an impressive bit of thinking under pressure on the part of Shilpa.

At 8pm, still more than a kilometer from the lab, the grapel (1mm diameter micro hail) storm began with persistent 25 knot up-valley winds. The lab was already invisible from station F14 and we joked about having to navigate the tracking team back to the lab by following the bot home under the ice. By the time the tracking team returned there had been an accumulation of 1 inch of snow by completion of mission.

The grapel storm turned into a blizzard, obscuring the lab from 500 meters out in a white out.

Vickie gives Rachel a break from fiber tending following the 6 hour mission.

By 11pm the bot was on its sled at the lab. Power reserve was 21% when the it reached the melt hole, following a total of 7 hours 3 minutes in the water. At end of mission we found the bot very positively buoyant—adding the 8 kg dive bag we use to lower it each day would not get it underwater. Heavy micro-bubble formation had occurred all over the bot. There were also micro-bubbles all over the sonde (down-looking) camera window.

Shilpa, Bart, and Chris watch the new situational awareness visualizer at mission control as the bot “discovers” the melt hole on return from its long trip.

Status of all sonde casts (green = completed) as of November 14, 2009.

We drove home to East Lobe camp at 1am.

Reporting by Bill Stone

West Lake Bonney, Taylor Valley, Antarctica
Reporting from East Lake Bonney Basecamp

The lead team of Vickie, Kristof, and Bill was up at 6:40am and off to the bot garage by 8am. They reloaded both battery stacks into their housings and into the bot, then reconnected power lines and re-installed ballast lead. By 10am the system was powered up and showing 100% power charge.

By noon the bot was over the melt hole for the sonde instrument calibration. Outside, winds began to pick up steadily from South Pole (katabatics) to 30 knots gusting to 35 knots. Wind gusts hitting the north end of the lab were violently shaking it—every time it moved it blew out the heater. The air temperature at the start of the mission was a warm 32F but dropping.

We shortly received a radio briefing that helicopters were grounded at McMurdo for at least two days, preventing arrival of chief project scientist Peter Doran. At 12:20pm we performed a sonde test drop to 18m depth to verify changes made by Chris and John to the sonde instrument parameters—all looked good on the live sonde monitor at mission control so we proceeded. By 12:30pm the bot was down at 5 meters depth and performing a test of ice picking behavior. This time, due to variations in temperature, the vehicle was proving to be too heavy so we recalled it to the melt hole and removed 500 grams of ballast.

The ambitious 3.2 kilometer sonde mission planned for November 13, including 19 sonde casts.

The wind was blowing down valley strongly for the entire mission without let up. Going eastward the tracking team was being forcefully pushed ahead; coming back was a fight. Taking tracking beacon readings were difficult — the wind was blowing the coils and the wind drone made it difficult to hear the tone.

The mission was successfully concluded at 7:15pm with an automated ascent up through the melt hole and a 25% remaining power reserve. We had completed 20 sonde drops today: G12, G13, F14, F15, F16, G16, G15, G14, H13, H12, H11, F13, F12, F11, F9, E7, D7, D6, E6, F6.

The total mission length was 3400 meters, including additional post-return navigation testing that was performed in the vicinity of the melt hole involving the USBL secondary localizer. Following the additional navigation tests the power was down to 12% and still usable, validating Bart’s hunch of several days ago regarding the battery balancing.

The team returned to camp at 9:20pm (an early night for us) and discovered to everyone’s dismay that the winds (which were still driving hard) had uprooted Shilpa’s and Chris’s tents. After dinner the entire team broke out replacement tents and helped set new sites up, these tied down to bigger rocks.

Status of all sonde casts (green = completed) as of November 13, 2009.

Reporting by Bill Stone

West Lake Bonney, Taylor Valley, Antarctica
Reporting from East Lake Bonney Basecamp

The mission today was clear: find the cause of the communications system failure of yesterday. We had already tested all the elements of the external system yesterday (fiber, data converters) and the internal data hub itself proved to be functioning correctly when run off an external power supply dialed to the appropriate drive voltage. Similarly, the onboard power supplies were also working correctly. It took eight hours to methodically go down the list of likely contenders. In the end, the theory that wire terminations in a screw-type termination block had oxidized and thereby reduced the voltage getting to the communications hub had proved correct. In effect there had been a power “brown out” that was enough to drop the hub for the remainder of the mission after the initial failure at G11. There were parallel communications (via internet) going on with several members of the SAS engineering team in the U.S. and elsewhere to confer on this subject. Ultimately, we pulled the wires leading to the data hub, trimmed them, reterminated and soldered them and then re-installed into the terminal block. The system powered up normally. The general conjecture was that the bot had sat in the MEC engineering center in McMurdo station for 10 months between the last power up and that this had given time for a loose connection to oxidize enough to manifest itself as a reduced voltage coming out of the power supplies. We pre-empted at this point and re-terminated all the other wires going into this same power block.

Vickie connects a local computer interface to the Profiler housing. Using this and an external laptop computer we were able to respool the instrument sonde.

Chris checks voltages on a power supply terminal strip in the main computer processor housing.

The suspect power terminal block.

With refurbished connections, Chris successfully logs into the bot through the wireless interface.

Rachel reassembles the main computer. This is actually a stack of several processors. Overall there are 45 computer processors in ENDURANCE.

Meanwhile, Bart used the down time opportunity to play out a hunch regarding the 20% “stranded” power in the batteries. the new design, completed just in time for this season, used two stacks of batteries in series along with custom circuitry to balance the voltages. But that circuitry had to get its power somewhere and it was connected to the first half of the stack, slowly causing the voltage there to drop below the other stack. Safeguard circuitry looked for the lowest cell in the system and if it fell below a certain value (for fire safety) it disconnected the battery. What Bart discovered was a significantly larger power draw by the monitoring circuitry than was needed to run it. So a modification was made to the boards to reduce this demand by a factor of ten. Some test calculations indicated that at the new level of power draw from the monitoring circuits that the overall voltage balancing system would succeed (it was not designed for differential power draw). With the two halves of each of the power stacks (the vehicle uses two sets of two stacks) now balanced (manually) and the control circuitry installed, we anticipated a significant change in available power. Tomorrow’s mission will tell.

Reporting by Bill Stone

West Lake Bonney, Taylor Valley, Antarctica
Reporting from East Lake Bonney Basecamp

Today was to be the first long range mission, going 300 meters further east than we had reached in 2008. In preparation for this we re-spooled a new 2 kilometer data fiber we had commissioned for 2009. This, plus mission planning and software upgrades keep everyone busy until 2:45pm when the mission got fully underway, targeted at achieving 19 sonde drops. We had sunny weather with a persistent 10 knot wind coming up the valley from McMurdo Sound.

Vickie “flakes out” the 2 kilometer data fiber spool so that it will deploy without hockling (a tendency of coiled ropes, fibers, and cables to form loops).

We had already logged the first sonde cast (F10) and were localizing the vehicle at G11 when at 4:45PM mission control called to indicate communications went down on the bot. The team at mission control tried rebooting; changing the short fiber from the main targeting computer; changing the fiber optic to Ethernet conversion boxes, but no go. They then began pulling fiber in to test if communications would come back—due to a kink straightening out—but also no go. So Bart and Kristof began pulling it back using the fiber with a slow, steady 2 kilogram force—enough to gently get it moving under the ice cap. Vickie and Bill tracked it for 50m to assure it was pulling straight back towards the bot house before packing up and returning the remaining 400 meters on foot. The vehicle was in the middle of a sonde caste when the communications had gone dead so it was unclear whether the sonde mission completed and the instrument package homed to the bot.

Bill operates an instrument for measuring the surface albedo at station F10. These data will be used to cross correlate PAR (photosynthetic active radiation) readings from the vehicle sonde sensors with surface illumination.

Vickie grabs a quick bite of lunch while waiting at station G11 for a decision from Mission Control (450 meters away towards the glacier) on the status of the vehicle.

The vehicle successfully returned to the melt hole at 7:44pm, whereupon an extensive investigation began that would last another day. It was floating neutrally in the water, an important sign that there had been no housing leaks. We made a direct fiber optic connection (bypassing the 2 kilometer spool) but still no connection. Similarly, a wireless link that was activated when the bot was out of the water also would not work. We discovered when the bot was hoisted out of the water that the sonde had not reeled back in—indicating a significant communications problem had affected all systems at once. Vickie connected a diagnostic serial communications cable directly into the Profiler housing and Chris was able to verify that the bot had significant reserve power and was responsive to direct local commands—and we thus respooled the sonde—about 26 meters had been paid out at the time of failure. Further investigations through other independent instrumentation data ports showed that the main processor stack data router was down. We were able to talk to the cameras and several other key instruments but it was the main processor stack that issued commands to operate the various parts of the vehicle. The batteries had more than 50% power remaining. An independent test of the 2 kilometer fiber showed that to be operating as expected.

At 9pm we recycled vehicle power and the wireless link came up temporarily then crashed. We then opened up the main processor housing. There were no “smoking guns” that would suggest a shock-induced failure (e.g. a loose connector from impacting the underside of the ice during ice picking). We held a team meeting to discuss and the general consensus was that oxidation on wires from internal power supply to router may be the problem—will investigate tomorrow.

Reporting by Bill Stone